關於 GPFS price 問題請參考：How is GPFS priced?
我這邊安裝的環境是：
kernel: 2.6.29-gentoo-r5
gpfs: 3.2.1-12
Step1.
安裝基本需要套件。
1.1. ksh
$ emerge -uD app-shells/ksh
1.2. rsh
$ emerge -uD net-misc/netkit-rsh
1.3. imake
$ emerge -uD x11-misc/imake
1.4. libstdc++.so.5
$ mkdir /usr/src/gpfs/
$ cd /usr/src/gpfs/
$ wget ftp://rpmfind.net/linux/fedora/releases/10/Everything/i386/os/Packages/compat-libstdc++-33-3.2.3-64.i386.rpm
$ rpm -ivh compat-libstdc++-33-3.2.3-64.i386.rpm
Step2.
下載安裝 GPFS.
$ wget ftp://ftp.software.ibm.com/software/server/gpfs/gpfs-3.2.1-12.i386.update.tar.gz
$ tar zxf gpfs-3.2.1-12.i386.update.tar.gz
$ rpm -ivh gpfs.msg.en_US-3.2.1-12.noarch.rpm
$ rpm -ivh gpfs.gpl-3.2.1-12.noarch.rpm
$ rpm -ivh gpfs.docs-3.2.1-12.noarch.rpm
$ rpm -ivh gpfs.base-3.2.1-12.i386.update.rpm --noscripts
取得 scripts.
$ rpm -qip --scripts gpfs.base-3.2.1-12.i386.update.rpm > scripts.sh
編譯相關 binary.
$ export SHARKCLONEROOT=/usr/lpp/mmfs/src
$ cd /usr/lpp/mmfs/src/config
$ cp site.mcr.proto site.mcr
確認 site.mcr 部分設定。
#define GPFS_ARCH_I386
LINUX_DISTRIBUTION = KERNEL_ORG_LINUX
#define LINUX_KERNEL_VERSION 2062999
$ cd /usr/lpp/mmfs/src
$ make World
$ make InstallImages
新增搜尋 PATH.
PATH = /usr/lpp/mmfs/bin:$PATH
Step3.
public key 設置。
$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
aa:bb:cc:dd:ee:ff:11:22:33:44:55:66:77:88:99:00 root@node147
The key's randomart image is:
+--[ RSA 2048]----+
| .o=*|
| o .=E+|
| o . .* |
| . +... .|
| S o .=. |
| . +. |
| o. |
| ... |
| .. |
+-----------------+
上傳 id_dsa.pub 至 所有主機 且重新命名為 authorized_keys.
Step4.
編輯主機節點設定。
$ cat mycluster.allnodes
node147:quorum
node148:quorum
Step5.
Create cluster.
$ mmcrcluster -N mycluster.allnodes -p node147 -r /usr/bin/ssh -R /usr/bin/scp -C mycluster
Mon Jun 8 22:19:24 CST 2009: mmcrcluster: Processing node node147
Mon Jun 8 22:19:24 CST 2009: mmcrcluster: Processing node node148
mmcrcluster: Command successfully completed
mmcrcluster: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
Step6.
顯示確認 cluster/node 狀態。
$ mmlscluster
GPFS cluster information
========================
GPFS cluster name: mycluster.node147
GPFS cluster id: 15226457667488521708
GPFS UID domain: mycluster.node147
Remote shell command: /usr/bin/ssh
Remote file copy command: /usr/bin/scp
GPFS cluster configuration servers:
-----------------------------------
Primary server: node147
Secondary server: (none)
Node Daemon node name IP address Admin node name Designation
-----------------------------------------------------------------------------------------------
1 node147 211.79.x.147 node147 quorum
2 node148 211.79.x.148 node148 quorum
$ mmlsnode -a
GPFS nodeset Node list
------------- -------------------------------------------------------
mycluster node147 node148
Step7.
啟動 GPFS.
$ mmstartup -a
Mon Jun 8 22:20:20 CST 2009: mmstartup: Starting GPFS ...
Step8.
檢查載入模組。
$ lsmod
Module Size Used by
mmfs 1048096 1
mmfslinux 174468 4 mmfs
tracedev 9888 3 mmfs,mmfslinux
Step9.
建立 Network shared disks (NSDs) 描述檔。
$ cat nodes.descfile
/dev/sda1:node147::dataAndMetadata::
/dev/sda1:node148::dataAndMetadata::
目前測試不支援 HP /dev/cciss/c0d0p1
Step10.
建立 NSD.
$ mmcrnsd -F nodes.descfile
mmcrnsd: Processing disk sda1
mmcrnsd: Processing disk sda1
mmcrnsd: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
建立完 NSD 後，nodes.descfile 會自動做些變更。
$ cat nodes.descfile
# /dev/sda1:node147::dataAndMetadata::
gpfs1nsd:::dataAndMetadata:4001::
# /dev/sda1:node148::dataAndMetadata::
gpfs2nsd:::dataAndMetadata:4002::
Step11.
確認節點硬碟狀況。
$ mmlsnsd -m
Disk name NSD volume ID Device Node name Remarks
---------------------------------------------------------------------------------------
gpfs1nsd D34F3E934A2D1FD6 /dev/sda1 node147 server node
gpfs2nsd D34F3E944A2D1FD7 /dev/sda1 node148 server node
Step12.
格式化檔案系統。
$ mmcrfs /gpfs gpfs0 -F nodes.descfile
The following disks of gpfs0 will be formatted on node node147:
gpfs1nsd: size 1220698993 KB
gpfs2nsd: size 1221269301 KB
Formatting file system ...
Disks up to size 10 TB can be added to storage pool 'system'.
Creating Inode File
Creating Allocation Maps
Clearing Inode Allocation Map
Clearing Block Allocation Map
Formatting Allocation Map for storage pool 'system'
Completed creation of file system /dev/gpfs0.
mmcrfs: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
程式會自動修改 /etc/fstab 檔案。
$ cat /etc/fstab
/dev/gpfs0 /gpfs gpfs rw,mtime,atime,dev=gpfs0,autostart 0 0
Step13.
掛載！測試！
$ mount /gpfs
$ time dd if=/dev/zero of=/gpfs/test1 bs=5120 count=1024000
1024000+0 records in
1024000+0 records out
5242880000 bytes (5.2 GB) copied, 23.5701 s, 222 MB/s
real 0m23.612s
user 0m0.329s
sys 0m11.902s
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/cciss/c0d0p3 136G 1.8G 134G 2% /
udev 10M 44K 10M 1% /dev
shm 8.0G 0 8.0G 0% /dev/shm
/dev/gpfs0 2.3T 5.6G 2.3T 1% /gpfs
$ md5sum /gpfs/test1 (on node147)
f0c4910bd1b40aecaad309d2a8999e66 test1
$ md5sum /gpfs/test1 (on node148)
f0c4910bd1b40aecaad309d2a8999e66 test1
其他：
Adding a new disk to an existing GPFS file system.
$ mmcrnsd -F descfile-node1-sda2
$ mmadddisk gpfs0 -F descfile-node1-sda2 -r -a
Delet a disk.
$ mmdeldisk gpfs0 gpfs6nsd -r -a
Delete all file systems with mmdelfs.
$ mmdelfs /dev/gpfs0
Delete all NSDs with mmdelnsd.
$ mmdelnsd 'gpfs1nsd;gpfs2nsd;gpfs3nsd'
Shutdown GPFS on all nodes.
$ mmshutdown -a
Tue Jun 9 15:18:09 CST 2009: mmshutdown: Starting force unmount of GPFS file systems
Tue Jun 9 15:18:14 CST 2009: mmshutdown: Shutting down GPFS daemons
node148: Shutting down!
node147: Shutting down!
node148: 'shutdown' command about to kill process 3461
node148: Unloading modules from /usr/lpp/mmfs/bin
node148: Unloading module mmfs
node148: Unloading module mmfslinux
node148: Unloading module tracedev
node147: 'shutdown' command about to kill process 3643
node147: Unloading modules from /usr/lpp/mmfs/bin
node147: Unloading module mmfs
node147: Unloading module mmfslinux
node147: Unloading module tracedev
Tue Jun 9 15:18:21 CST 2009: mmshutdown: Finished
Remove all nodes from nodeset.
$ mmdelnode -a
說明：
目前 node147 提供 1.2TB(sda1), node148 提供 1.2TB(sda1)，兩個 node 皆可看到 total: 2.3TB
目前有些東西編譯不出來，可是不知道影響到啥 XD 因為模組有正常載入 @_@
ERROR: "struct_module" [/usr/lpp/mmfs/src/gpl-linux/tracedev.ko] undefined!
ERROR: "struct_module" [/usr/lpp/mmfs/src/gpl-linux/mmfslinux.ko] undefined!
ERROR: "struct_module" [/usr/lpp/mmfs/src/gpl-linux/mmfs26.ko] undefined!
參考：
請參考 IBM GPFS 官方網站。
沒有留言:
張貼留言