ibm gpfs 筆記

因為有參考到不同的文件,所以有可能指令下法各有不同

1: 創建cluster
mmcrcluster -N node01:quorum-manager,node02:quorum-manager -r /usr/bin/ssh -R /usr/bin/scp  <---我用這個


mmcrcluster -N "argentina-gpfs:manager-quorum;australia-gpfs:quorum;denmark-gpfs:quorum" -p argentina-gpfs -r /usr/bin/ssh -R /usr/bin/scp -C ThreeNode


錯誤訊息:
mmcrcluster: Performing preliminary node verification ...
mmcrcluster: Incorrect node client1 specified for command.
mmcrcluster: Error found while checking node descriptor client1:quorum-client
mmcrcluster: Command failed. Examine previous error messages to determine cause.

原因是找不到該node , 因為node已經在hosts裡面做名稱取代
#vim /etc/hosts

將node名稱改為上面的,可以自行加入想要的到hosts , 上面已經透過
#ssh denmark date
確認是可以通過的


2:檢查創建結果
#mmlscluster

3:設定license mode給其他的node,在這段可以設定client
mmchlicense server --accept -N node01,node02
mmchlicense server --accept -N argentina-gpfs,australia-gpfs,denmark-gpfs
mmchlicense client --accept -N gpfs-clinet1

檢查
#mmlslicense -L

4:start gpfs
mmstartup -a

5:create nsd
創建一個add_nsd.txt
格式為
%nsd: device=DiskName
  nsd=NsdName
  servers=ServerList
  usage={dataOnly | metadataOnly | dataAndMetadata | descOnly}
  failureGroup=FailureGroup
  pool=StoragePool


example1:
%nsd:
  device=/dev/sdc
  nsd=mynsd1
  usage=dataAndMetadata

%nsd:
  device=/dev/sdd
  nsd=mynsd2
  usage=dataAndMetadata


其中的/dev/sdx 是已經先透過iscsi做掛載的動作


還有另一種格式為
# DiskName:ServerList::DiskUsage:FailureGroup:DesiredName:StoragePool
/dev/sdb:argentina-gpfs::dataAndMetadata:3001:argentina-gpfsNSD1::
/dev/sdc:argentina-gpfs::dataAndMetadata:3001:argentina-gpfsNSD2::
/dev/sdd:argentina-gpfs::dataAndMetadata:3001:argentina-gpfsNSD3::
/dev/sde:argentina-gpfs::dataAndMetadata:3001:australia-gpfsNSD4::
/dev/sdb:australia-gpfs::dataAndMetadata:3002:australia-gpfsNSD1::


另一種格式
%nsd:
device=/dev/sdj
nsd=jynsd1
servers=spectrumscale2
usage=dataAndMetadata
failureGroup=1
pool=system


最後做創建的動作
mmcrnsd -F add_nsd.txt -v yes
或
mmcrnsd -F add_nsd.txt

檢查
#mmlsnsd

6:創建file system
mmshutdown -a
mmchconfig tiebreakerDisks="mynsd1"    <---你的nsd名稱
mmstartup -a


# mmcrfs gpfs1 -F stanza.txt -A yes -T /gpfs/fs1
創建一個file system , 名字為gpfs1並且掛載到/gpfs/fs1


# mmlsnsd     看到file system名稱 ,使用到下面的命令.
#mmmount fs1 -a














筆記:
1: 刪除cluster
# mmdelnode -f

2: 錯誤訊息
[root@spectrumscale2 jimmy]# mmaddnode -N gpfs-clinet1:client
Thu Jul  7 03:20:06 EDT 2016: mmaddnode: Processing node gpfs-clinet1
mmaddnode: Node gpfs-clinet1 was not added to the cluster.
   The node appears to already belong to a GPFS cluster.
mmaddnode: mmaddnode quitting.  None of the specified nodes are valid.
mmaddnode: Command failed. Examine previous error messages to determine cause.
這個原因是gpfs-clinet1是在另外一台pc-1 ,並且已經加入cluster , 所以只要在pc-1將該node從cluster刪除就可以加入了

3:錯誤訊息
[root@spectrumscale2 ~]# mmgetstate -a
 Node number  Node name        GPFS state
------------------------------------------
       1      spectrumscale    active
       2      spectrumscale2   arbitrating
       3      spectrumscale3   active
       4      gpfs-clinet1     down

可以發現總是 arbitrating
從  vim /var/adm/ras/mmfs.log.latest

Mon Jul 11 03:32:23.725 2016: [E] The key used by the cluster named 172.16.10.7 spectrumscale <c0p0> has changed.  Contact the administrator to obtain the new key and register it using "mmremotecluster update".

可以觀察到是有關於ssh key認證部分

所以解決方式為
0: # mmshutdown -a 

1: # mmauth genkey new

2: # mmremotecluster update cluster1 -k cluster1_id_rsa.pub

3:# mmauth genkey commit

4: # mmstartup -a

可以參考

http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adv.doc/bl1adv_keyren.htm


4:錯誤訊息
# vim /var/adm/ras/mmfs.log.latest
Tue Jul 12 15:50:33.292 2016: [E] The defined server spectrumscale for NSD nsd01 couldn't be resolved.
Tue Jul 12 15:50:33.297 2016: [N] mmfsd is shutting down.
Tue Jul 12 15:50:33.298 2016: [N] Reason for shutdown: Could not initialize network shared disks
Tue Jul 12 15:50:33 EDT 2016: mmcommon mmfsdown invoked.  Subsystem: mmfs Status: active

看起來是nsd nsd01的server 不通
 # ssh node date   測試看看,果然不通
修改vim /etc/hosts