Lustre notes

An overview of Lustre I wrote up to give a presentation to a sister group within NACS to the one I work in: CCS Lustre presentation

To totally start over from scratch - IE wipe out all data and start over again, follow this procedure

Do this on esmfsn01 and esmfsn03 (the OSS's, or "Object Storage Servers"). Make sure that the filesystems lustre is going to format are not mounted.
- lconf --force --cleanup /etc/lustre/config.xml
- lconf --reformat /etc/lustre/config.xml
Then do the same thing again, only this time on the MDS (metadata server). This will take quite some time:
- lconf --force --cleanup /etc/lustre/config.xml
- lconf --reformat /etc/lustre/config.xml
Then mount lustre on the client, esmft2:

/etc/init.d/lustre start

Then start up NFS on the lustre client/NFS server, esmft2:
- /etc/init.d/nfs start
Then NFS mount lustre onto esmf04d
- mount esmft2:/mnt/lustre /mnt/lustre
Verify that it's working, as root:
- cd /mnt/lustre
- df .
- touch foo
- rm foo

To reset the lustre system completely, (ideally) without loss of data:

Reboot the lustre OSS's, esmfsn01 and esmfsn03, and the lustre MDS, esmfsn05.
esmfsn01 and esmfsn03 should run lustre startup scripts automatically.
esmfsn05, the MDS, will require you to:
- cd /etc/rc5.d
- ./old.S99lustre
Then mount lustre on the client, esmft2:

/etc/init.d/lustre start

Then start up NFS on the lustre client/NFS server, esmft2:
- /etc/init.d/nfs start
Then NFS mount lustre onto esmf04d
- mount esmft2:/mnt/lustre /mnt/lustre
Verify that it's working, as root:
- cd /mnt/lustre
- df .
- touch foo
- rm foo

Generating /etc/lustre/config.xml

Apparently strange problems can ensue, if you take a working lustre filesystem, rearrange the order in which the nodes are added to the config.xml from the config.sh, and then tell the nodes to use the new config.xml - because lmc wants to internally number the nodes, and it numbers them in the order it sees them.
This is the Lustre script that Duncan, Nate and I developed during the Lustre training:

alias LMC="lmc -m config.xml"
LMC --add net --node guest24-uml5 --nettype tcp --nid guest24-uml5
LMC --add net --node guest24-uml4 --nettype tcp --nid guest24-uml4
LMC --add net --node guest24-uml3 --nettype tcp --nid guest24-uml3
LMC --add net --node guest24-uml2 --nettype tcp --nid guest24-uml2
LMC --add net --node guest24-uml1 --nettype tcp --nid guest24-uml1
LMC --add mds --mds mds1 --node guest24-uml2 --dev /tmp/mds1 --size 100000
LMC --add lov --lov lov1 --mds mds1 --stripe_cnt 2 --stripe_sz 65536
LMC --add ost --node guest24-uml3 --lov lov1 --dev /tmp/ost1 --size 400000
#LMC --add ost --node guest24-uml3 --lov lov1 --dev /tmp/ost2 --size 400000
LMC --add ost --node guest24-uml4 --lov lov1 --dev /tmp/ost1 --size 800000
#LMC --add ost --node guest24-uml4 --lov lov1 --dev /tmp/ost2 --size 800000
LMC --add net --node client --nettype tcp --nid "*"
LMC --add mtpt --node client --path /mnt/lustre --mds mds1 --lov lov1

Here's the analogous script that ClusterFS used to create our test config early on:

# Create nodes

rm config.xml

lmc -o config.xml --add net --node esmfsn05 --nid esmfsn05 --nettype tcp
lmc -m config.xml --add net --node esmfsn01 --nid esmfsn01 --nettype tcp
lmc -m config.xml --add net --node esmfsn03 --nid esmfsn03 --nettype tcp
lmc -m config.xml --add net --node esmft2 --nid esmft2 --nettype tcp

# Cofigure MDS

lmc -m config.xml --add mds --node esmfsn05 --mds mds1 --fstype ext3 --dev /dev/sda1

# Configures OST

lmc -m config.xml --add lov --lov lov1 --mds mds1 --stripe_sz 1048576 --stripe_cnt 2 --stripe_pattern 0
lmc -m config.xml --add ost --node esmfsn01 --lov lov1 --ost ost1 --fstype ext3 --dev /dev/sda
lmc -m config.xml --add ost --node esmfsn03 --lov lov1 --ost ost2 --fstype ext3 --dev /dev/sda
lmc -m config.xml --add ost --node esmfsn01 --lov lov1 --ost ost3 --fstype ext3 --dev /dev/sdb
lmc -m config.xml --add ost --node esmfsn03 --lov lov1 --ost ost4 --fstype ext3 --dev /dev/sdb
lmc -m config.xml --add ost --node esmfsn01 --lov lov1 --ost ost5 --fstype ext3 --dev /dev/sdc
lmc -m config.xml --add ost --node esmfsn03 --lov lov1 --ost ost6 --fstype ext3 --dev /dev/sdc
lmc -m config.xml --add ost --node esmfsn01 --lov lov1 --ost ost7 --fstype ext3 --dev /dev/sdd
lmc -m config.xml --add ost --node esmfsn03 --lov lov1 --ost ost8 --fstype ext3 --dev /dev/sdd
lmc -m config.xml --add ost --node esmfsn01 --lov lov1 --ost ost9 --fstype ext3 --dev /dev/sde
lmc -m config.xml --add ost --node esmfsn03 --lov lov1 --ost ost10 --fstype ext3 --dev /dev/sde
lmc -m config.xml --add ost --node esmfsn01 --lov lov1 --ost ost11 --fstype ext3 --dev /dev/sdf
lmc -m config.xml --add ost --node esmfsn03 --lov lov1 --ost ost12 --fstype ext3 --dev /dev/sdf

# Configure client

lmc -m config.xml --add mtpt --node esmft2 --path /mnt/lustre --mds mds1 --lov lov1

not

Notes from the Lustre training I attended (taken on an old Palm IIIxe, so they're full of abbreviation)

day 1
day 2

Lustre and quotas:

"I expect to include this feature in a production release in 2005, but the precise date or release has not been decided."

3Ware RAID tuning on RHEL (not really related to Lustre that much, but we happen to be combining the two) Note that Lustre does not use the VM system (which is probably how it gets by the 16 terrabyte limit common in linux filesystems), so it's likely that these tweaks don't help Lustre any.

Expanding Lustre to include a new pair of OSS's

Nic's instructions:

Here are the 'upgrade' instruction for getting esmfsn06 and esmfsn04 
online when they are ready.

Install the RPMs in esmft2:/usr/local/src/current. There are kernel and 
Lustre RPMs there.

Uncomment the lines in esmft2:/root/config.sh that correspond to the 
host being brought up.

run 'sh config.sh' on esmft2. This will update the XML file with the new 
system.

copy the new esmft2:/root/config.xml to all of the nodes 
'/etc/lustre/config.xml'

go to the new OSS to be added and run 'lconf --refomat 
/etc/lustre/config.xml'
---> this will take quite a while

Shutdown the Lustre Client and MDS - the MDS and client *must* be 
shutdown cleanly for the next step. If need be, you can reboot the 
machines and have the lustre init script not start to get to a clean state.

Update the configuration on the MDS:
esmfsn05: lconf --write_conf /etc/lustre/config.xml

Now you can start the MDS and client:
/etc/init.d/lustre start

You should now see the expanded disk space on the client.

Nic

Hopefully I'll put more info here later. :)