Buffer cache tweaks:
- Note: It is my belief that all of these are tuning
the linux buffercache, and Lustre does not use the
buffercache, so these tweaks most likely will not help Lustre,
much, if at all.
- It's possible not all of these are necessary.
- Do both:
- (echo "512" > /proc/sys/vm/max-readahead)
- (echo "512" > /proc/sys/vm/min-readahead)
- And then do one of the following:
- (sysctl -w "vm.bdflush=10 500 0 0 500 3000 10 20 0" )
- sysctl -w "vm.bdflush=0 500 0 0 500 3000 10 20 0"
Furthermore, 3Ware recommended that we upgrade to the latest 3w-xxxx
drivers on all systems using 3Ware RAID cards for the ESMF. This
gave better performance, and we're hoping it'll improve stability
with lustre as well. We configured the driver to upgrade the
firmware, again on 3Ware's recommendation, though this is not
generally a great idea - don't play with firmware if it ain't broke.
The versions of the driver and firmware we're at now are 1.02.00.037
(aka "771" on their website) and FE7S 1.05.00.065.
Here's
the latest 3ware driver
To check on the version of the 3ware driver:
- # grep 'Driver version' /proc/scsi/3w-xxxx/*
- # /proc/scsi/3w-xxxx/0:Driver version: 1.02.00.033
- # /proc/scsi/3w-xxxx/1:Driver version: 1.02.00.033
- # /proc/scsi/3w-xxxx/2:Driver version: 1.02.00.033
- # /proc/scsi/3w-xxxx/3:Driver version: 1.02.00.033
To make sure you get the right version of the 3ware driver, it isn't
necessarily enough to just put it under /lib/modules/`uname
-r`/kernel/drivers/scsi/3w-xxxx.o. You may also have to add it to the
initrd under /boot!
Version
info: driver, firmware, &c...
3ware's list
of low-level format utilities - we use Maxtors in the ESMF