• Apparently strange problems can ensue, if you take a working lustre filesystem, rearrange the order in which the nodes are added to the config.xml from the config.sh, and then tell the nodes to use the new config.xml - because lmc wants to internally number the nodes, and it numbers them in the order it sees them.
• This is the Lustre script that Duncan, Nate and I developed during the Lustre training:

alias LMC="lmc -m config.xml"
LMC --add net --node guest24-uml5 --nettype tcp --nid guest24-uml5
LMC --add net --node guest24-uml4 --nettype tcp --nid guest24-uml4
LMC --add net --node guest24-uml3 --nettype tcp --nid guest24-uml3
LMC --add net --node guest24-uml2 --nettype tcp --nid guest24-uml2
LMC --add net --node guest24-uml1 --nettype tcp --nid guest24-uml1
LMC --add mds --mds mds1 --node guest24-uml2 --dev /tmp/mds1 --size 100000
LMC --add lov --lov lov1 --mds mds1 --stripe_cnt 2 --stripe_sz 65536
LMC --add ost --node guest24-uml3 --lov lov1 --dev /tmp/ost1 --size 400000
#LMC --add ost --node guest24-uml3 --lov lov1 --dev /tmp/ost2 --size 400000
LMC --add ost --node guest24-uml4 --lov lov1 --dev /tmp/ost1 --size 800000
#LMC --add ost --node guest24-uml4 --lov lov1 --dev /tmp/ost2 --size 800000
LMC --add net --node client --nettype tcp --nid "*"
LMC --add mtpt --node client --path /mnt/lustre --mds mds1 --lov lov1
		

Please note that it's very important that the strip_sz is a multiple of the largest page size you are likely to encounter - a good sized power of two is a good bet. The lustre doc claims that their tools check for this, but they don't (Tue Sep 14 15:34:48 PDT 2004).
• Here's the analogous script that ClusterFS used to create our test config early on:

  # Create nodes
  
  rm config.xml
  
  lmc -o config.xml --add net --node esmfsn05 --nid esmfsn05 --nettype tcp
  lmc -m config.xml --add net --node esmfsn01 --nid esmfsn01 --nettype tcp
  lmc -m config.xml --add net --node esmfsn03 --nid esmfsn03 --nettype tcp
  lmc -m config.xml --add net --node esmft2 --nid esmft2 --nettype tcp
  
  # Cofigure MDS
  
  lmc -m config.xml --add mds --node esmfsn05 --mds mds1 --fstype ext3 --dev /dev/sda1
  
  # Configures OST
  
  lmc -m config.xml --add lov --lov lov1 --mds mds1 --stripe_sz 1048576 --stripe_cnt 2 --stripe_pattern 0
  lmc -m config.xml --add ost --node esmfsn01 --lov lov1 --ost ost1 --fstype ext3 --dev /dev/sda
  lmc -m config.xml --add ost --node esmfsn03 --lov lov1 --ost ost2 --fstype ext3 --dev /dev/sda
  lmc -m config.xml --add ost --node esmfsn01 --lov lov1 --ost ost3 --fstype ext3 --dev /dev/sdb
  lmc -m config.xml --add ost --node esmfsn03 --lov lov1 --ost ost4 --fstype ext3 --dev /dev/sdb
  lmc -m config.xml --add ost --node esmfsn01 --lov lov1 --ost ost5 --fstype ext3 --dev /dev/sdc
  lmc -m config.xml --add ost --node esmfsn03 --lov lov1 --ost ost6 --fstype ext3 --dev /dev/sdc
  lmc -m config.xml --add ost --node esmfsn01 --lov lov1 --ost ost7 --fstype ext3 --dev /dev/sdd
  lmc -m config.xml --add ost --node esmfsn03 --lov lov1 --ost ost8 --fstype ext3 --dev /dev/sdd
  lmc -m config.xml --add ost --node esmfsn01 --lov lov1 --ost ost9 --fstype ext3 --dev /dev/sde
  lmc -m config.xml --add ost --node esmfsn03 --lov lov1 --ost ost10 --fstype ext3 --dev /dev/sde
  lmc -m config.xml --add ost --node esmfsn01 --lov lov1 --ost ost11 --fstype ext3 --dev /dev/sdf
  lmc -m config.xml --add ost --node esmfsn03 --lov lov1 --ost ost12 --fstype ext3 --dev /dev/sdf
  
  # Configure client
  
  lmc -m config.xml --add mtpt --node esmft2 --path /mnt/lustre --mds mds1 --lov lov1
  

  The --fstype ext3 is not arbitrary. Sep 2004. ClusterFS only supports ext3 at this time.

• day 1
• day 2

• "I expect to include this feature in a production release in 2005, but the precise date or release has not been decided."

• Nic's instructions:

  Here are the 'upgrade' instruction for getting esmfsn06 and esmfsn04 
  online when they are ready.
  
  Install the RPMs in esmft2:/usr/local/src/current. There are kernel and 
  Lustre RPMs there.
  
  Uncomment the lines in esmft2:/root/config.sh that correspond to the 
  host being brought up.
  
  run 'sh config.sh' on esmft2. This will update the XML file with the new 
  system.
  
  copy the new esmft2:/root/config.xml to all of the nodes 
  '/etc/lustre/config.xml'
  
  go to the new OSS to be added and run 'lconf --refomat 
  /etc/lustre/config.xml'
  ---> this will take quite a while
  
  Shutdown the Lustre Client and MDS - the MDS and client *must* be 
  shutdown cleanly for the next step. If need be, you can reboot the 
  machines and have the lustre init script not start to get to a clean state.
  
  Update the configuration on the MDS:
  esmfsn05: lconf --write_conf /etc/lustre/config.xml
  
  Now you can start the MDS and client:
  /etc/init.d/lustre start
  
  You should now see the expanded disk space on the client.
  
  Nic
  

• Hopefully I'll put more info here later. :)

You can e-mail the author with questions or comments: