Simple Solaris SDS/SVM Boot Disk Mirroring Howto.

Version 0.2 - Barbie LeVile - 14.12.2003

What we want to do:

1)   We want to create a mirrored systemdisk environment that can
     be booted from and survives a disk failure.
     In out example case we have 2 drives we want to use for that.
     A three way mirror can be done in much the similar way.


2) Some background on SDS concepts
 
     
     Metadbs:
     
       A metadb stores state informations of the metadevices, they are critical to SDS
       operation.
       In SDS metadbs must follow this rules to work:
       To be able to boot a SDS system, at least 2 and 50%+1 metadbs of all existing metadbs
       need to be in a ok state. Yes, both statements mut be true.
       For that reason metadbs are normaly placed on their own small slice on each drive.
       Less change for accidently deleting them.
       As a rule of thumb, 3 are placed on each drive in a 2 drive setup, and 2 in a 3 drive setup.
       However, a SDs system with only 2 disks can NEVER fullfill the minium requierments in case of diskfailure.
       There are two choices for such situations, 1) set the kernel to ignore the problem and keep booting.
       Or 2) fix the problem prior to getting back up manualy.
       
       In this example we will set the kernel to ignore it, since in most cases its the prefered mode
       of operation. On should however not forget to fix the problem ASAP.
       
       man metadb for details
       
       
     Metadevices:
     
       Metadevices are the building blocks of SDS.
       First one creats a metadevice out of a logical disk device like a slice, then assembles
       those into another metadevices.
       
       Example: We have 2 disks, each as 5 slices, each slice becomes a metadevice.
       In a mirror those are called submirrors, and then one from each disk gets assembled into
       actual mirror.
       Metadevices are manipulated via metainit.
       
       man metainit for details
       
       
     Metaroot:
     
       The metaroot command sets up a) the kernel to be able to handle a sds mirror boot device
       and b) creates a metadevice entry for the root filesystem in the /etc/vfstab.
       It can be used to revert the root device back to non metadevice mode too.
       
       man metaroot for details
       
     
     Hints:
   
       SDS likes to have drives of the same geometry.
       If thats not possible, make sure that the drive that holds your data has the smaller slices.
       You can always mirror a smaller slice to a bigger one, but not the other way.   


3) The example disklayout:
   
     We have 2 drives, c0t0d0 and c1t0d0, drive 1 is sliced as follows:
   
     c0t0d0s0	swap
     c0t0d0s1	/
     c0t0d0s2	backup (whole disk)
     c0t0d0s3	/var
     c0t0d0s4	/opt
     c0t0d0s5	/export/home
     c0t0d0s6	unused
     c0t0d0s7	here we place out metadbs
   
     We have no seperate /usr in this example, add it yourself if you want it.
     But generaly its not needed anymore.
     s7 should be around 50mb, more is not needed, even with 10s of metadbs.
   
     Now we plan how to name out metadevices.
   
     I usualy name my metadevices in the following sheme:
   
     d Slice 0 for the mirror, 1 for the submirror on disk 1, 2 for disk 2, 3 for disk 3,
     except for swap which lays on s0, there it just gets ommited
     Which gives us the following:
     
     Disk 1		Disk 2			Metadevices
     c0t0d0s0 -> d1	c1t0d0s0 -> d2		d1  + d2  -> d0
     c0t0d0s1 -> d11	c1t0d0s3 -> d12		d11 + d12 -> d10
     c0t0d0s3 -> d31	c1t0d0s3 -> d32		d31 + d32 -> d30
     c0t0d0s4 -> d41	c1t0d0s4 -> d42		d41 + d42 -> d40
     c0t0d0s5 -> d51	c1t0d0s5 -> d52		d51 + d52 -> d50
   

4) Time to get our hands dirty!

     The following steps should be done while in single usermode idealy.
     

4.1) Making both drives the same.
     
     We start with slicing the second drive in the same way as our first drive, the
     master.
     
     # prtvtoc /dev/rdsk/c0t0d0s2 | fmthard -s - /dev/rdsk/c1t0d0s2
     
     No need to newfs the second drive slices here, that will automaticaly done
     by the mirror syncing later.

     
4.2) Metadbs

     We can now setup our metadbs.
     
     # metadb -a -f -c3 c0t0d0s7 c1t0d0s7
     
     Since its the initial creation of the metadbs, we need to force it with -f
     -a adds the metadbs, and -c tells it how many to create.
     
     You can see the results with metadb -i.
     metadb -i is a very handy tool to determine the state of your metadb states.

     
4.3) Initializing the devices

     Now we go to setup the initial metadevices.
     
     # metainit -f d1 1 1 c0t0d0s0
     # metainit -f d11 1 1 c0t0d0s1 
     # metainit -f d31 1 1 c0t0d0s3    
     # metainit -f d41 1 1 c0t0d0s4 
     # metainit -f d51 1 1 c0t0d0s5 

     # metainit -f d2 1 1 c1t0d0s0  
     # metainit -f d12 1 1 c1t0d0s1    
     # metainit -f d32 1 1 c1t0d0s3     
     # metainit -f d42 1 1 c1t0d0s4     
     # metainit -f d52 1 1 c1t0d0s5     
     
     Like metadb metainit must be forced with -f, but this time not because its the initial creation, but
     because we work on mounted filesystems.
     So, here we create a 1 way concatenation of our actual slices and form the needed submirrors.
      
      
4.3) Mirroring fun part 1!

     # metainit d0 -m d1
     # metainit d10 -m d11     
     # metainit d30 -m d31     
     # metainit d40 -m d41     
     # metainit d50 -m d51 

     Here are the actualy mirroing intialized, the -m tells SDS that we want to build a mirror with the name
     in the first colum and consiting of the submirror in the third colum.
     We now have a one way mirror of our system drive, but its not active yet.
    

4.4) Setting the root device


     # cp /etc/vfstab /etc/vfstab_pre_sds
     # cp /etc/system /etc/system_pre_sds
     # metaroot d10
    
    
4.5) Setting up /etc/vfstab


     #device         	device          	mount           FS      fsck    mount   mount
     #to mount       	to fsck         	point           type    pass    at boot options
     #
     fd     		-			/dev/fd fd      -       no      -
     /proc   		-       		/proc   	proc    -       no      -
     #
     # sds drives
     #
     /dev/md/dsk/d0 	-       		-       	swap    -       no      -
     /dev/md/dsk/d10 /dev/md/rdsk/d10		/		ufs     1       no      logging
     /dev/md/dsk/d30 /dev/md/rdsk/d30		/var		ufs	1	no	logging,noatime
     /dev/md/dsk/d40 /dev/md/rdsk/d40		/opt    	ufs     2       yes     logging
     /dev/md/dsk/d50 /dev/md/rdsk/d50 		/export/home    ufs     2       yes     logging
     #
     # non sds drives
     #
     #/dev/dsk/c1t0d0s0	-       		-       	swap    -       no      -
     #/dev/dsk/c1t0d0s1	/dev/rdsk/c1t0d0s1	/       	ufs     1       no      logging
     #/dev/dsk/c1t0d0s3	/dev/rdsk/c1t0d0s2      /var		ufs     1       no      logging,noatime
     #/dev/dsk/c1t0d0s4	/dev/rdsk/c1t0d0s4      /opt		ufs     2       yes     logging
     #/dev/dsk/c1t0d0s5	/dev/rdsk/c1t0d0s5      /export/home    ufs     2       yes     logging
     #
     # swap slices
     #
     swap    		-       		/tmp		tmpfs   -       yes     -


4.5.1) Optional: Making sure we can boot in case of disk failure.

     USE ONLY WHERE DATA INTEGRETY IS LESS IMPORTANT THEN SERVICE AVILABILITY.
     THIS CAN LEAD TO FILE CORUPTION.

     To make sure we can boot in case a disk fails, we need to tell the kernel to ignore the
     quota on metadbs, otherwise we can't boot in a two disk setup, because we can't never
     fullfill the requirments.
     
     For that we add the following to /etc/system set md:mirrored_root_flag=1

     # echo "set md:mirrored_root_flag=1" >> /etc/system
    

4.6) Our first reboot!

     Bring down the system to the OBP, don't reboot yet fully.
     We need to setup the boot devices now.
     
     OK setenv boot device disk0:b disk1:b
     
     Make sure the devaliases of disk0 and disk1 are actualy pointing to the correct hardware decives.
     
     Now we boot our system up for the first time on the mirror.
     
     If all went well we are up and running in a few seconds.
        
     
4.7) Mirroring fun part 2!

     Now its time to hook up the second drive so we have actualy mirrored slices.
     
     # metattach d0 d2
     # metattach d10 d12     
     # metattach d30 d32     
     # metattach d40 d42     
     # metattach d50 d52     
          
     This will take considereble amount of time.
     Use metastat to check on the progress of the syncing         
     
     Example metastat d30     

     
5) All done, enjoy

6) Performance Tips

7) Troubleshooting