PVFS install on Gentoo Linux

繼上一篇 Gluster install on Gentoo Linux 後，最近又試了另外一套 PVFS.
安裝上比 Gluster 步驟多很多，而且個人覺得編譯上不是很順利 ~_~
kernel: linux-2.6.28-gentoo-r5

Step1.
前置準備工作，安裝 PVFS 相依到的軟體。
1. Berkely DB (version 3 or 4)
$ emerge -uD sys-libs/db
2. aio support (provided by glibc and librt)
$ emerge -uD sys-libs/glibc
$ getconf GNU_LIBC_VERSION
glibc 2.8 (目前 glibc 版本用的是 2.8)
3. pthreads
$ getconf GNU_LIBPTHREAD_VERSION
NPTL 2.8
4. gcc 2.96 or newer
$ gcc -v
gcc version 4.3.2 (Gentoo 4.3.2-r3 p1.6, pie-10.1.5)
5. GNU Make
$ gmake -v
GNU Make 3.81

Step2.
下載 PVFS source code.
由於目前最新的 pvfs-2.8.1 在搭配 2.6.28 kernel 還有問題，我在 5/20/2009 checkout 新的 code 已經沒有問題了。
網路上有人遇到跟我相同的問題，在這有說明。
check out source code 方法可參考 Source Code Access
$ cd /usr/src
To check out the latest pvfs2 tree:
$ cvs -d :pserver:anonymous@cvs.parl.clemson.edu:/anoncvs login
(use any password)
$ cvs -d :pserver:anonymous@cvs.parl.clemson.edu:/anoncvs co pvfs2
$ cvs -d :pserver:anonymous@cvs.parl.clemson.edu:/anoncvs logout

Step3.
安裝 pvfs 套件。
$ cd /usr/src/pvfs2
$ ./configure --disable-karma --enable-epoll --with-kernel=/usr/src/linux-2.6.28-gentoo-r5
***** Displaying PVFS Configuration Information *****
------------------------------------------------------
PVFS2 configured to build karma gui : no
PVFS2 configured to perform coverage analysis : no
PVFS2 configured for aio threaded callbacks : yes
PVFS2 configured to use FUSE : no
PVFS2 configured for the 2.6.x kernel module : yes
PVFS2 configured for the 2.4.x kernel module : no
PVFS2 configured for using the mmap-ra-cache : no
PVFS2 will use workaround for redhat 2.4 kernels : no
PVFS2 will use workaround for buggy NPTL : no
PVFS2 server will be built : yes
$ make
$ make install

Step4.
設定 pvfs，在 PVFS 系統中，有幾個子系統，metadata server, I/O server, client.
我目前的組態設定是 4 台 metadata server, 同時這 4 台亦扮演 I/O server 的角色。

4.1. Server configuration
$ pvfs2-genconfig /etc/pvfs2-fs.conf
**********************************************************************
Welcome to the PVFS2 Configuration Generator:
This interactive script will generate configuration files suitable
for use with a new PVFS2 file system. Please see the PVFS2 quickstart
guide for details.
**********************************************************************
You must first select the network protocol that your file system will use.
The only currently supported options are "tcp", "gm", "mx", "ib", and "portals".
(For multi-homed configurations, use e.g. "ib,tcp".)
* Enter protocol type [Default is tcp]:
Choose a TCP/IP port for the servers to listen on. Note that this
script assumes that all servers will use the same port number.
* Enter port number [Default is 3334]:
Choose a directory for each server to store data in.
* Enter directory name: [Default is /pvfs2-storage-space]: /hd1
Choose a file for each server to write log messages to.
* Enter log file location [Default is /tmp/pvfs2-server.log]:
Next you must list the hostnames of the machines that will act as
I/O servers. Acceptable syntax is "node1, node2, ..." or "node{#-#,#,#}".
* Enter hostnames [Default is localhost]: node{25-28}
Use same servers for metadata? (recommended)
* Enter yes or no [Default is yes]:
Configured a total of 4 servers:
4 of them are I/O servers.
4 of them are Metadata servers.
* Would you like to verify server list (y/n) [Default is n]? y
****** I/O servers:
node27
node26
node25
node28
****** Metadata servers:
node27
node26
node25
node28
* Does this look ok (y/n) [Default is y]? y
Writing fs config file... done

4.2. 編輯 /etc/hosts 設定 host-ip map, 確認每台 node 都有彼此的 host-ip map.
140.110.x.24 node24
140.110.x.25 node25
140.110.x.26 node26
140.110.x.27 node27
140.110.x.28 node28
設定檔派送，因為每台 node 都需認得相同的 fsid, 因此建議直接 scp 過去。
$ for i in `seq 25 28`; do scp /etc/pvfs2-fs.conf node\${i}:/etc/

4.3. 儲存空間初始化。
$ pvfs2-server /etc/pvfs2-fs.conf -f
[S 05/21 08:12] PVFS2 Server version 2.8.1pre1-2009-05-21-000601 starting.
[D 05/21 08:13] PVFS2 Server: storage space created. Exiting.

4.4. 啟動 Server.
$ pvfs2-server /etc/pvfs2-fs.conf
[D 05/21 08:14] PVFS2 Server version 2.8.1pre1-2009-05-21-000601 starting.

4.5. Client configuration
$ mkdir /mnt/pvfs2
$ touch /etc/pvfs2tab
$ chmod a+r /etc/pvfs2tab
$ cat /etc/pvfs2tab
tcp://node25:3334/pvfs2-fs /mnt/pvfs2 pvfs2 defaults,noauto 0 0

4.6. 測試
$ pvfs2-ping -m /mnt/pvfs2
(1) Parsing tab file...
(2) Initializing system interface...
(3) Initializing each file system found in tab file: /etc/pvfs2tab...
PVFS2 servers: tcp://node25:3334
Storage name: pvfs2-fs
Local mount point: /mnt/pvfs2
/mnt/pvfs2: Ok
(4) Searching for /mnt/pvfs2 in pvfstab...
PVFS2 servers: tcp://node25:3334
Storage name: pvfs2-fs
Local mount point: /mnt/pvfs2
meta servers:
tcp://node25:3334
tcp://node26:3334
tcp://node27:3334
tcp://node28:3334
data servers:
tcp://node25:3334
tcp://node26:3334
tcp://node27:3334
tcp://node28:3334
(5) Verifying that all servers are responding...
meta servers:
tcp://node25:3334 Ok
tcp://node26:3334 Ok
tcp://node27:3334 Ok
tcp://node28:3334 Ok
data servers:
tcp://node25:3334 Ok
tcp://node26:3334 Ok
tcp://node27:3334 Ok
tcp://node28:3334 Ok
(6) Verifying that fsid 1234446610 is acceptable to all servers...
Ok; all servers understand fs_id 1234446610
(7) Verifying that root handle is owned by one server...
Root handle: 1048576
Ok; root handle is owned by exactly one server.
=============================================================
The PVFS2 filesystem at /mnt/pvfs2 appears to be correctly configured.

Step5.
PVFS Linux Kernel Interface
5.1. 安裝 kernel module
$ cd /usr/src/pvfs2
$ make kmod
$ make kmod_install
5.2. 載入 module 並確認
$ insmod /usr/src/pvfs2/src/kernel/linux-2.6/pvfs2.ko
$ lsmod
Module Size Used by
pvfs2 103280 0
$ dmesg grep pvfs
pvfs2: module version 2.8.1pre1-2009-05-21-000601 loaded

Step6.
啟動 Client.
$ cd /usr/src/pvfs2/src/apps/kernel/linux
$ ./pvfs2-client -f -p ./pvfs2-client-core
pvfs2-client starting
Spawning new child process
About to exec: ./pvfs2-client-core, with args: pvfs2-client-core --child -a 5 -n 5 --logtype file -L /tmp/pvfs2-client.log
Waiting on child with pid 29965

Step7.
掛載!最後測試!
$ mount -t pvfs2 tcp://node25:3334/pvfs2-fs /mnt/pvfs2
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/cciss/c0d0p3 33G 2.3G 31G 7% /
udev 10M 40K 10M 1% /dev
shm 506M 0 506M 0% /dev/shm
/dev/cciss/c0d1p1 34G 33M 34G 1% /hd1
tcp://node25:3334/pvfs2-fs 136G 132M 136G 1% /mnt/pvfs2
$ dd if=/dev/zero of=/mnt/pvfs2/test bs=4k count=10
10+0 records in
10+0 records out
$ ls -la /mnt/pvfs2/test
-rw-r--r-- 1 root root 40960 May 21 10:29 /mnt/pvfs2/test

參考：
詳細的安裝設定請參考 Documentation.