Rebooting portions of the ESMF

The only thing that's relevant really now is the AIX side, because both
GFS and Lustre are gone.

For info on rebooting things on the AIX side, see https://stromberg.dnsalias.org/~strombrg/Rebooting-an-IBM-SP2-cluster.html

If you need to reboot one or more of the Lustre storage nodes, read on.  :)

This is in two sections: Physical power cycling, and the
/usr/local/etc/lustre script.

Physical Powercycling:

The ESMF storage nodes are esmfsn0{1,2,3,4,5,6} and esmft{1,2}.

esmfsn0{1,2,3,4,5} and esmft{1,2} are located nearest the salvage racks
in the EG machine room.

esmfsn0{1,2,3,4,6} are 4U promicro units.  esmfsn04 is a 3U promicro unit.
esmft{1,2} are 1U IBM X335's.

esmfsn06 is across the room, in the row of DCS racks.

sn0[1-5] are labeled as sn1, sn2 through sn5, on both the front and the back.

To reboot sn0[1-4] and sn06, use a key on your keychain to hold in
the small red disk on the front of the machine for a few seconds.
The lights should go out.  Then press the red disk again with your key,
and the system in question should come back up.

Note: sn03 tends to take 5 minutes longer to reboot than the other
storage nodes - so don't panic if it takes a little longer.

To reboot sn05, which again is labeled sn5 from both the front and
the back, remove the grill on the front, using the thumb screws (no
screwdriver required).  Then, again using a key, hit the small power
button (IIRC, it's white, and has a semicircle with a vertical line
poking out the bottom).

On the promicro units with a red power button (sn0[1234] and sn06), if
the power button doesn't want to work, you can go around the back of the
unit in question, and pull all four power cords.  Then plug them back in.
If the powersupply lights all come back to green, you're in business.
If 1 to 3 come up green, there's a problem that needs to be taken up
with promicro (the ones with a red light may have failed).  If all 4
come up red, pull them out and put them back in a different arrangement,
EG 1->2, 2->3, 3->4, 4->1.  This usually makes them come up
all green on the first try.

To reboot esmft{1,2}, I usually just pull the power plug and plug it back in.

/usr/local/etc/lustre

This script accepts a variety of arguments like "start", "stop",
"restart", "reboot", "poweroff", "cleanup" and so on.

It's used to fire up/shut down things across all of the (currently)
relevant storage nodes.

esmfsn03 tends to take longer to reboot than the other storage nodes.