[讨论]nfsv4_pnfs_using_spnfsd_and_nfsv4_as_back-end
发表于 ： 2008-07-25 14:39
有没有人试试？NFSv4.1中的pNFS中的一种简单实现。
https://www.os3.nl/2007-2008/students/t ... s_back-end
spNFS (Simple Parallel Network File System, by NetApp) is a Linux pNFS server implementation from NetApp who use it to develop their pNFS client. Currently it is the only working Linux implementation using the latest code. More information can be found in their presentation: dmuntz-spnfs-cthon08.pdf
The setup in a virtualized environment (Xen) looks as follows:
The client uses pNFS with a file layout driver to mount an export from the MDS (Metadata server). The MDS uses a single physical connection to mount NFSv4 exports on the DSs (Data servers). When data from the client is written to the export, pNFS is used to stripe the data over the DSs without tranferring data through the MDS. Retrieving data works the same way. For example of a file of 230M, each DS in this setup stores 115M. Retrieving data uses both DSs network connections as well to retrieve the data.
The picture is just an abstract view of the situation, it could be that the pNFS client has for example a single very fast physical network connection such as 10G Ethernet which a regular NFS setup is unable to fully utilize. With pNFS the data comes from multiple servers at once.
Some personal notes from setting up spNFS:
% spNFS:
spNFS:
1.
→ /boot/
mkinitrd -o mkinitrd -o initrd.img-2.6.25-pnfs 2.6.25-pnfs
*
GRUB is installed. To automatically switch to new kernels, point your
*
default entry in menu.lst to /boot/vmlinuz-2.6.25-pnfs
%nfs-utils:
*do NOT install
%%% Setting things up:
On MDS:
Reconfigure for two DSs:
On the MD, the /spnfs will contain mount points (IP address as dir) to all DSs. The root is /pnfs thus on the DSs we export the /:
etc/exports
DSs mounts on MDS:
etc/fstab
/etc/exports
On MDS and DSs:
on MDS, start spnfsd:
** be sure to mount the /pnfs/145….. 's on the MDS
on client, load layout driver:
mount:
https://www.os3.nl/2007-2008/students/t ... s_back-end
spNFS (Simple Parallel Network File System, by NetApp) is a Linux pNFS server implementation from NetApp who use it to develop their pNFS client. Currently it is the only working Linux implementation using the latest code. More information can be found in their presentation: dmuntz-spnfs-cthon08.pdf
The setup in a virtualized environment (Xen) looks as follows:
The client uses pNFS with a file layout driver to mount an export from the MDS (Metadata server). The MDS uses a single physical connection to mount NFSv4 exports on the DSs (Data servers). When data from the client is written to the export, pNFS is used to stripe the data over the DSs without tranferring data through the MDS. Retrieving data works the same way. For example of a file of 230M, each DS in this setup stores 115M. Retrieving data uses both DSs network connections as well to retrieve the data.
The picture is just an abstract view of the situation, it could be that the pNFS client has for example a single very fast physical network connection such as 10G Ethernet which a regular NFS setup is unable to fully utilize. With pNFS the data comes from multiple servers at once.
Some personal notes from setting up spNFS:
% spNFS:
代码： 全选
apt-get install cogito git-core
git clone git://linux-nfs.org/~dmuntz/spnfs.git
git clone git://linux-nfs.org/~dmuntz/nfs-utils.git代码： 全选
make menuconfig
-.config
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=y
CONFIG_NFSD=y
CONFIG_NFS_V4_1=y
CONFIG_NFSD_V4_1=y
CONFIG_PNFS=y
CONFIG_PNFSD=y
CONFIG_SPNFS=y代码： 全选
make
make modules
make modules_install
make install→ /boot/
mkinitrd -o mkinitrd -o initrd.img-2.6.25-pnfs 2.6.25-pnfs
代码： 全选
name /lib/modules/2.6.25-pnfs ...
edit /boot/grub/menu.lstGRUB is installed. To automatically switch to new kernels, point your
*
default entry in menu.lst to /boot/vmlinuz-2.6.25-pnfs
%nfs-utils:
代码： 全选
apt-get install automake libtool pkg-config
export PKG_CONFIG_PATH=/usr/lib/pkgconfig
sh autogen.sh
./configure --disable-gss (avoid dependency problems, not required for this setup anyway)
make%%% Setting things up:
On MDS:
代码： 全选
cp /root/sPNFS/nfs-utils/utils/spnfsd/spnfsd.conf /etc/代码： 全选
"""
[General]
Verbosity = 1
Stripe-size = 8192
Dense-striping = 0
Pipefs-Directory = /var/lib/nfs/rpc_pipefs
DS-Mount-Directory = /spnfs
[DataServers]
NumDS = 2
DS1_IP = 145.100.106.107
DS1_PORT = 2049
DS1_ROOT = /pnfs
DS1_ID = 1
DS2_IP = 145.100.106.108
DS2_PORT = 2049
DS2_ROOT = /pnfs
DS2_ID = 2
"""代码： 全选
MD: mkdir -m 1777 /spnfs
DSs: mkdir -m 1777 /pnfs
mkdir /pnfs/145.100.106.107
mkdir /pnfs/145.100.106.108etc/exports
代码： 全选
/ *(rw,fsid=0,insecure,no_root_squash,sync,no_subtree_check)etc/fstab
代码： 全选
# spNFS DS mounts
145.100.106.107:/pnfs /spnfs/145.100.106.107 nfs defaults 1 2
145.100.106.108:/pnfs /spnfs/145.100.106.108 nfs defaults 1 2代码： 全选
/ *(rw,fsid=0,insecure,no_root_squash,sync,no_subtree_check)代码： 全选
/usr/sbin/rpc.statd
/usr/sbin/rpc.mountd
/usr/sbin/rpc.nfsd 8
exportfs -vr代码： 全选
"""
debian02:/usr/sbin# spnfsd
Verbosity: 1
Stripe size: 8192
Dense striping: 0
Number of data servers: 2
DS1 IP: 145.100.106.107
DS1 PORT: 2049
DS1 ROOT: /pnfs
DS1 DSID: 1
DS2 IP: 145.100.106.108
DS2 PORT: 2049
DS2 ROOT: /pnfs
DS2 DSID: 2
"""on client, load layout driver:
代码： 全选
modprobe nfslayoutdriver代码： 全选
mount -t nfs4 145.100.106.106:/ /mnt