An overview of Lustre I wrote up to give a presentation to a sister
group within NACS to the one I work in: CCS Lustre presentation
To totally start over from scratch - IE wipe out all data and start
over again, follow this procedure
- Do this on esmfsn01 and esmfsn03 (the
OSS's, or "Object Storage Servers"). Make sure that the filesystems
lustre is going to format are not mounted.
- lconf --force --cleanup /etc/lustre/config.xml
- lconf --reformat /etc/lustre/config.xml
- Then do the same thing again, only this time on the MDS (metadata
server). This will take quite some time:
- lconf --force --cleanup /etc/lustre/config.xml
- lconf --reformat /etc/lustre/config.xml
- Then mount lustre on the client, esmft2:
- Then start up NFS on the lustre client/NFS server, esmft2:
- Then NFS mount lustre onto esmf04d
- mount esmft2:/mnt/lustre /mnt/lustre
- Verify that it's working, as root:
- cd /mnt/lustre
- df .
- touch foo
- rm foo
To reset the lustre system completely, (ideally) without loss of
data:
- Reboot the lustre OSS's, esmfsn01 and esmfsn03, and the lustre
MDS, esmfsn05.
- esmfsn01 and esmfsn03 should run lustre startup scripts
automatically.
- esmfsn05, the MDS, will require you to:
- cd /etc/rc5.d
- ./old.S99lustre
- Then mount lustre on the client, esmft2:
Note that this step hangs sometimes. Sometimes if you kill
the hanging mount process and try again, lustre will mount onto
the client though. Nic of ClusterFS regarded killing and trying
again to be best avoided though.
- Then start up NFS on the lustre client/NFS server, esmft2:
- Then NFS mount lustre onto esmf04d
- mount esmft2:/mnt/lustre /mnt/lustre
- Verify that it's working, as root:
- cd /mnt/lustre
- df .
- touch foo
- rm foo
Generating /etc/lustre/config.xml
- Apparently strange problems can ensue, if you take a working
lustre filesystem, rearrange the order in which the nodes are added
to the config.xml from the config.sh, and then tell the nodes to use
the new config.xml - because lmc wants to internally number the
nodes, and it numbers them in the order it sees them.
- This is the Lustre script that Duncan, Nate and I developed during
the Lustre training:
alias LMC="lmc -m config.xml"
LMC --add net --node guest24-uml5 --nettype tcp --nid guest24-uml5
LMC --add net --node guest24-uml4 --nettype tcp --nid guest24-uml4
LMC --add net --node guest24-uml3 --nettype tcp --nid guest24-uml3
LMC --add net --node guest24-uml2 --nettype tcp --nid guest24-uml2
LMC --add net --node guest24-uml1 --nettype tcp --nid guest24-uml1
LMC --add mds --mds mds1 --node guest24-uml2 --dev /tmp/mds1 --size 100000
LMC --add lov --lov lov1 --mds mds1 --stripe_cnt 2 --stripe_sz 65536
LMC --add ost --node guest24-uml3 --lov lov1 --dev /tmp/ost1 --size 400000
#LMC --add ost --node guest24-uml3 --lov lov1 --dev /tmp/ost2 --size 400000
LMC --add ost --node guest24-uml4 --lov lov1 --dev /tmp/ost1 --size 800000
#LMC --add ost --node guest24-uml4 --lov lov1 --dev /tmp/ost2 --size 800000
LMC --add net --node client --nettype tcp --nid "*"
LMC --add mtpt --node client --path /mnt/lustre --mds mds1 --lov lov1
Please note that it's very important that the strip_sz is a
multiple of the largest page size you are likely to encounter - a
good sized power of two is a good bet. The lustre doc claims that
their tools check for this, but they don't (Tue Sep 14 15:34:48
PDT 2004).
Here's the analogous script that ClusterFS used to create our test
config early on:
# Create nodes
rm config.xml
lmc -o config.xml --add net --node esmfsn05 --nid esmfsn05 --nettype tcp
lmc -m config.xml --add net --node esmfsn01 --nid esmfsn01 --nettype tcp
lmc -m config.xml --add net --node esmfsn03 --nid esmfsn03 --nettype tcp
lmc -m config.xml --add net --node esmft2 --nid esmft2 --nettype tcp
# Cofigure MDS
lmc -m config.xml --add mds --node esmfsn05 --mds mds1 --fstype ext3 --dev /dev/sda1
# Configures OST
lmc -m config.xml --add lov --lov lov1 --mds mds1 --stripe_sz 1048576 --stripe_cnt 2 --stripe_pattern 0
lmc -m config.xml --add ost --node esmfsn01 --lov lov1 --ost ost1 --fstype ext3 --dev /dev/sda
lmc -m config.xml --add ost --node esmfsn03 --lov lov1 --ost ost2 --fstype ext3 --dev /dev/sda
lmc -m config.xml --add ost --node esmfsn01 --lov lov1 --ost ost3 --fstype ext3 --dev /dev/sdb
lmc -m config.xml --add ost --node esmfsn03 --lov lov1 --ost ost4 --fstype ext3 --dev /dev/sdb
lmc -m config.xml --add ost --node esmfsn01 --lov lov1 --ost ost5 --fstype ext3 --dev /dev/sdc
lmc -m config.xml --add ost --node esmfsn03 --lov lov1 --ost ost6 --fstype ext3 --dev /dev/sdc
lmc -m config.xml --add ost --node esmfsn01 --lov lov1 --ost ost7 --fstype ext3 --dev /dev/sdd
lmc -m config.xml --add ost --node esmfsn03 --lov lov1 --ost ost8 --fstype ext3 --dev /dev/sdd
lmc -m config.xml --add ost --node esmfsn01 --lov lov1 --ost ost9 --fstype ext3 --dev /dev/sde
lmc -m config.xml --add ost --node esmfsn03 --lov lov1 --ost ost10 --fstype ext3 --dev /dev/sde
lmc -m config.xml --add ost --node esmfsn01 --lov lov1 --ost ost11 --fstype ext3 --dev /dev/sdf
lmc -m config.xml --add ost --node esmfsn03 --lov lov1 --ost ost12 --fstype ext3 --dev /dev/sdf
# Configure client
lmc -m config.xml --add mtpt --node esmft2 --path /mnt/lustre --mds mds1 --lov lov1
The --fstype ext3 is not arbitrary. Sep 2004. ClusterFS
only supports ext3 at this time.
Notes from the Lustre training I attended (taken on an old Palm
IIIxe, so they're full of abbreviation)
Lustre and quotas:
- "I expect to include this
feature in a production release in 2005, but the precise date or release
has not been decided."
3Ware RAID tuning on RHEL (not
really related to Lustre that much, but we happen to be combining the
two) Note that Lustre does not use the VM system (which is probably
how it gets by the 16 terrabyte limit common in linux filesystems),
so it's likely that these tweaks don't help Lustre any.
Expanding Lustre to include a new pair of OSS's
- Nic's instructions:
Here are the 'upgrade' instruction for getting esmfsn06 and esmfsn04
online when they are ready.
Install the RPMs in esmft2:/usr/local/src/current. There are kernel and
Lustre RPMs there.
Uncomment the lines in esmft2:/root/config.sh that correspond to the
host being brought up.
run 'sh config.sh' on esmft2. This will update the XML file with the new
system.
copy the new esmft2:/root/config.xml to all of the nodes
'/etc/lustre/config.xml'
go to the new OSS to be added and run 'lconf --refomat
/etc/lustre/config.xml'
---> this will take quite a while
Shutdown the Lustre Client and MDS - the MDS and client *must* be
shutdown cleanly for the next step. If need be, you can reboot the
machines and have the lustre init script not start to get to a clean state.
Update the configuration on the MDS:
esmfsn05: lconf --write_conf /etc/lustre/config.xml
Now you can start the MDS and client:
/etc/init.d/lustre start
You should now see the expanded disk space on the client.
Nic
Hopefully I'll put more info here later. :)