Home‎ > ‎Server config‎ > ‎

Lustre config - creating MDS system with ZFS

Manual met beter detail: http://wiki.lustre.org/Category:Lustre_Systems_Administration 

Die sections oor MGS en MDS setup het baie gehelp! 


Manier wat gewerk het: 


- Install kernel* rpms (dit was fine….) 

- Install zfs met dkms metode (kmod metode moet ge-disable wees in repo file) uit zfs repo met yum (local zfs rpms wat ons ge-download het, het ongelukkig spl issues maar die yum repo een is fine) 

- Install lustre-dkms rpm 

- Install lustre-osd-zfs-mount-2.10.3-1.el7.x86_64.rpm 

- Install lustre rpm 


Ek het eers die zpools gaan skep met zfs se eie commands, en dan met lustre gaan format (mens kan dit toe so doen)


ZFS commands: 


zpool create -O canmount=off -o cachefile=none mgspool mirror /dev/disk/by-id/scsi-35000c500301e1f7f /dev/disk/by-id/scsi-35000c500301e1f8f 

zpool create -O canmount=off -o cachefile=none mdspool mirror /dev/disk/by-id/scsi-35000c5003021d057 /dev/disk/by-id/scsi-35000c5003021cfa3 mirror /dev/disk/by-id/scsi-35000c5003021cfa7 /dev/disk/by-id/scsi-35000c5003021d0a3 


Daar is dan ’n mgspool (2 drives) en ’n mdspool (2x2 drives). 


Lustre se commands om die zpools te format: 


mkfs.lustre --mgs --fsname=nlustre --backfstype=zfs mgspool/mgt 

mkfs.lustre --mdt --fsname=nlustre --backfstype=zfs --mgsnode=192.168.0.36@o2ib,137.215.75.120@tcp mdspool/mdt 


[root@nmds x86_64]# zfs list 

NAME USED AVAIL REFER MOUNTPOINT 

mdspool 5.26M 1.41T 96K /mdspool 

mdspool/mdt 4.78M 1.41T 4.78M /mdspool/mdt 

mgspool 3.87M 360G 96K /mgspool 

mgspool/mgt 3.45M 360G 3.45M /mgspool/mgt 



root@nmds x86_64]# zfs get all -s local 

NAME PROPERTY VALUE SOURCE 

mdspool canmount off local 

mdspool/mdt canmount off local 

mdspool/mdt xattr sa local 

mdspool/mdt lustre:mgsnode 192.168.0.36@o2ib,137.215.75.120@tcp local 

mdspool/mdt lustre:svname nlustre-MDT0000 local 

mdspool/mdt lustre:flags 33 local 

mdspool/mdt lustre:fsname nlustre local 

mdspool/mdt lustre:index 0 local 

mdspool/mdt lustre:version 1 local 

mgspool canmount off local 

mgspool/mgt canmount off local 

mgspool/mgt xattr sa local 

mgspool/mgt lustre:svname MGS local 

mgspool/mgt lustre:flags 36 local 

mgspool/mgt lustre:fsname nlustre local 

mgspool/mgt lustre:index 65535 local 

mgspool/mgt lustre:version 1 local 



Die volgende moet gemount word om lustre te start op die server: 


mount.lustre mgspool/mgt /mnt/mgt 

mount.lustre mdspool/mdt /mnt/mdt 

[root@nmds x86_64]# cat /etc/ldev.conf 

nmds.bi.up.ac.za - MGS zfs:mgspool/mgt 

nmds.bi.up.ac.za - nlustre-MDT0000 zfs:mdspool/mdt 



Check met lctl: 


[root@nmds x86_64]# lctl device_list 

0 UP osd-zfs MGS-osd MGS-osd_UUID 4 

1 UP mgs MGS MGS 4 

2 UP mgc MGC192.168.0.36@o2ib 67d080d8-256b-859a-e469-a945526ef827 4 

3 UP osd-zfs nlustre-MDT0000-osd nlustre-MDT0000-osd_UUID 6 

4 UP mds MDS MDS_uuid 2 

5 UP lod nlustre-MDT0000-mdtlov nlustre-MDT0000-mdtlov_UUID 3 

6 UP mdt nlustre-MDT0000 nlustre-MDT0000_UUID 4 

7 UP mdd nlustre-MDD0000 nlustre-MDD0000_UUID 3 

8 UP qmt nlustre-QMT0000 nlustre-QMT0000_UUID 3 

9 UP lwp nlustre-MDT0000-lwp-MDT0000 nlustre-MDT0000-lwp-MDT0000_UUID 4 



[root@nmds x86_64]# lctl list_nids 

192.168.0.36@o2ib 

137.215.75.120@tcp 



Note to self:


Lustre mount vir locking:


       flock  Enable full flock support, coherent across all client nodes.


       localflock

              Enable  local  flock  support,  using  only  client-local  flock

              (faster,  for  applications that require flock but do not run on

              multiple nodes).


       noflock

              Disable flock support entirely.  Applications calling flock will

              get an error.



Striping:


lfs setstripe -c 4 /mnt/lustre


of 


tunefs.lustre -fsname=[NAME] param lov.stripe_count=4Â