			IBM
          	RDAC Multi-Path Proxy Driver for Linux
          	Build and Installation Instructions
 
Updated: 08/15/08                            Copyright 2004-2008
================================================================


 
Contents
================================================================
 1. OS Support
 2. HBA Support
 3. Co-existence with QLogic and LSI Logic failover drivers
 4. Unpacking the driver source code
 5. Building and Installation
 6. Physical HBA Module options 
 7. Persistent Naming
 8. Package Uninstall and Reinstall
 9. Kernel Update
10. Physical HBA driver update
11. Initial Ramdisk image (initrd) 
12. Restrictions
13. GPL License
 
 
1. OS Support
================================================================
    The RDAC Multi-Path Proxy driver for Linux is supported 
    on the following Linux distributions for 2.6 kernel:

       SuSE SLES 10 SP1, SP2  ( x86, x86_64, ppc64, IA64, Itanium2  )

       RedHat RHEL 5, 5.1, 5.2 (x86, x86_64, ppc64, IA64, Itanium 2 )

NOTE : MPP Driver version for 2.6 Kernel for Redhat 4, SUSE SLES 9
       and SUSE SLES 10 is not supported by this driver. 
       Please download MPP Version 09.03.0Bxx.xxxx


NOTE : MPP Driver version for 2.4 Kernel is not supported by
       this driver. Please download MPP Version 09.00.xx.xx
 


2. HBA Support
================================================================
    The RDAC driver is supported on the following host 
    bus adapters:
	iSCSI Qlogic
	PCI-X :Qlogic 4050C, Qlogic 4052C
        PCI-E :Qlogic 4060C, Qlogic 4062C

	Fibre Channel Qlogic
	PCI-E : QLE2560, QLE2562  (8Gb) 
	PCI-E : QLE2460, QLE2462  (4Gb) 
	PCI-E : QLE236x (2Gb)
	PCI-X : QLA2460, QLA2462 (4Gb)
	PCI-X : QLA2310F, QLA2340, QLA2342 (2Gb) 
        PCI-X : QLA2000 (1Gb)

	SAS LSI Logic
	PCI-X : LSISAS3800-X, LSI SAS3442X-R, LSI SAS3442E-R  

	Fibre Channel LSI Logic:
	PCI-X : LSI7204XP-LC, LSI7404XP-LC, LSI7104XP, (4Gb) 
        PCI-X : LSI7202XP, LSI7402XP (2Gb)
	LSI7104EP, LSI7204EP
        PCI and PCI-X: LSIFC929X (2Gb)

	Fibre Channel IBM Branded Emulex
	PCI-X : IBM FC 5758, 5759 (4Gb)
        PCI-X : IBM FC 5716, (2Gb)
	PCI : IBM FC 6228 (1Gb) , 6239  (2Gb)

	Fibre Channel Emulex
	PCI-E : LPe12000, LPe12002 (8Gb)
	PCI-E : LPe11000, LPe11002, LPe1150 (4Gb)
	PCI-E : LP10000ExDC (2Gb)
	PCI-X : LP11000, LP11000DC, LP11002, LP1150 (4Gb)
        PCI-X : LP10000, LP10000DC, LP1050, LP9802, LP9802DC, LP982 (2Gb)
	PCI   : LP9002DC, LP952L, LP9002L, LP9000 (2Gb)

	Fibre Channel Brocade
	PCI-E : Brocade 815, Brocade 825 (8Gb)
 
    	Infiniband HCA Mellanox
	PCI-E : InfiniHost III Lx MHGS18-XTC, InfiniHost Ex MHGA28-XTC, ConnectX MHGH28-XTC

    	Infiniband HCA Voltaire
	PCI-E : HCA 400EX-D (MHGA28-XTC), HCA 410EX-D (MHGS18-XTC), HCA 500EX-D (MHGH28-XTC)

    	Infiniband HCA QLogic
	PCI-E : 7104-HCA-128LPX-DDR (MHGA28-1TC), 7104-HCA-LPX1P-DDR (MHGS18-XTC), 7104-HCA-LPX2P-DDR (MHGA28-XTC)
	 
 
3. Co-existence with QLogic and LSI failover drivers
================================================================
    The RDAC driver cannot co-exist with an HBA-level failover 
    driver such as the Emulex, QLogic or LSI Logic HBA failover drivers.
 
 
 
4. Unpack RDAC driver source
================================================================
    The RDAC driver is released as a source-code package in the
    format of a compressed tar file. To unpack the source
    package, type the following command and press Enter:

        tar -zxvf  rdac-LINUX-xx.xx.xxxx.xxxx-source.tar.gz

    where xx.xx.xxxx.xxxx is the release version of the RDAC driver.

    The source files will uncompress to the "linuxrdac-xx.xx.xxxx.xxxx" 
    directory. 
 
 
 
5. Build and Installation
================================================================
 
    5.1. Requirements before RDAC driver package installation
     -----------------------------------------------------------
        a. It's assumed that the host system that you are 
           installing the RDAC driver to has Supported HBA's.

        b. Although the system can have Fibre Channel HBAs from 
           multiple vendors or multiple models of HBAs from the 
           same vendor, only the same model of HBAs can be 
           connected to each Storage Array. 

        c. It is assumed that the low level HBA driver has been 
           properly built and installed before RDAC driver 
           installation. 

        d. The standard HBA driver needs to be loaded before 
           the RDAC driver installation. The HBA driver has to 
           be a non-failover driver.  
  
           For QLogic HBAs, the base driver is named "qla2xxx"
           and host driver is named "qla2300" .

           For LSI Logic HBAs, the port driver is named 
           mptbase and the host driver is named mptscsi or 
           mptscsih, for the mpt driver version earlier than
           3.02.xx. The 3.02.xx or later LSI mpt driver splits to three
           driver modules - mptspi, mptsas and mptfc for SCSI parallel 
           interface transport, SAS transport and Fibre Channel 
           transport respectively.

           For IBM Emulex HBAs, the base driver is named "lpfcdd"
           or "lpfc", depending on the driver version.

	   For Emulex HBAs, the base driver is named "lpfcdd"
           or "lpfc", depending on the driver version. 

           For Brocade HBAs, the base driver is named "bfa".


        e. On the SuSE operating system, it is assumed that the 
           low level HBA driver names are included in 
           INITRD_MODULES in /etc/sysconfig/kernel.

           For the systems that have QLogic HBA's, the modules 
           file should include a qla2xxx and qla2300 ( or qla2300). 
		
           For the systems that have LSI Logic HBAs, INITRD_MODULES 
           should include mptbase and mptscsi (or mptscsih) 
           for pre-3.02.xx version of LSI mpt driver. The INITRD_MODULES 
           should include mptbase, mptscsi, mptspi, mptsas and mptfc
           for 3.02.xx or later version of LSI mpt driver.

           For the systems that have IBM Emulex HBAs, INITRD_MODULES
           should include lpfcdd or lpfc.

	   For the systems that have Emulex HBAs, INITRD_MODULES
           should include lpfcdd or lpfc.

	   For the systems that have Brocade HBAs, INITRD_MODULES
           should include bfa.

        f. The kernel source tree for the kernel version to be 
           built against should already be installed before  
           RDAC driver package installation.
	
        g. The glibc-devel(64 bit) package is required for 64 bit machines.

    5.2. RDAC Driver Building Instructions
    -----------------------------------------------------------
        a. Change the directory to the ~/linuxrdac-xx.xx.xxxx.xxxx directory.

        b. To remove the old driver modules in that directory, 
           type the following and press Enter:
 
                make clean

        c. To compile all driver modules and utilities, type 
	     the following and press Enter:
 
                make
 

    5.3. RDAC Driver Installation
    -----------------------------------------------------------

        a. Uninstall the old RDAC Driver if previously installed, 
           type the following and press Enter:
 
                make uninstall
 
        b. Type the following and press Enter:
 
                make install

           This command will do the following things:
           * copy the driver modules to the kernel module tree.
           * build the new RAMdisk image (mpp-`uname -r`.img) which 
             includes the RDAC driver modules and all driver 
             modules that are needed at boot.

         c. Follow the instructions displayed at the end of of the 
            build process to add a new boot menu option which uses 
           /boot/mpp-`uname -r`.img as the initial RAMdisk image.
 
  	   The following is an example for /boot/grub/menu.lst (Note that it may vary
	   in accordance with different system setup configuration):
 
           title SUSE Linux  with MPP support
             kernel (hd0,1)/boot/vmlinuz root=/dev/sda2 vga=0x317 selinux=0 splash=silent console=tty0 resume=/dev/sda3 elevator=cfq showopts
             initrd (hd0,1)/boot/mpp-2.6.5-7.145-smp.img

	   The following is an example for /etc/elilo.conf (Note that it may vary
	   in accordance with different system setup configuration):
 
           image=vmlinuz-2.4.21-9.0.1.EL
             label=MPP
             initrd=mpp-2.4.21-9.0.1.EL.img
             read-only
             root=/dev/sda2

           The following is an example of /etc/yaboot.conf for 
           IBM eServer pSeries Architectures (ppc64) systems (Note entries
           may vary in accordance with your system setup configuration)

           image = /boot/vmlinux
             label = MPP
             root = /dev/sdb3
             append = "selinux=0 elevator=cfq"
             initrd = /boot/mpp-2.6.5-7.139-pseries64.img
 
    5.4. RDAC Driver Post-Installation
    -----------------------------------------------------------
        a. Reboot the system using the new boot menu option.

        b. Type in "lsmod" to verify that the driver stack is 
	     properly loaded:
           * On Redhat Platforms: scsi_mod, sd_mod, sg, 
             mppUpper, mppVhba, and low level HBA drivers 
             (e.g., qla2xxx, qla2300) are loaded.

           * On SuSE platforms: sg, mppUpper, mppVhba, and 
             low level HBA drivers (e.g., qla2xxx, qla2300),  are loaded.

        c. To verify that the RDAC driver discovered the 
           available physical LUNs and created virtual LUNs for
           them, type the following and press Enter:
 
                ls -lR /proc/mpp
 
        d. You can now issue I/Os to the LUNs.

        e. If you make any changes to the RDAC configuration 
           file (/etc/mpp.conf) or persistent binding file 
           (/var/mpp/devicemapping), please run "mppUpdate" to 
           re-build the RAMdisk image. This will ensure that
           the new configuration file (or persistent binding 
           file) can be used on the next system reboot.

        f. When an uninstall of RDAC is done, the old mpp.conf 
           is saved as /etc/mpp.conf.mppsave. This is to preserve
           the S2ToS3Key or hostId. On a reinstall of RDAC, a 
           default mpp.conf with default values is installed in 
           /etc/mpp.conf. If the user had specific values set, 
           these need to be copied from the old mpp.conf into 
           the new /etc/mpp.conf. After doing this "mppUpdate" 
           will also have to be run to re-build the MPP RAMdisk 
           image to pick up the changed mpp.conf.

	g. The file /etc/syslog.conf should have entries for
	   kern.debug, kern.notice, kern.info and kern.warning
	   in order to capture all debug and error messages
	   for MPP driver.

    5.5  RDAC Driver Unloading Instructions
    ----------------------------------------------------------
        a. The Unload order for the MPP driver stack is as
           follows.
           1. rmmod mppVhba
           2. rmmod (low-level HBA driver, this can be qla2xxx 
                  followed by qla2300
                  or mptscsih followed by mptbase
                  or lpfcdd or lpfc
           3. rmmod mppUpper



6. Physical HBA Module options
================================================================
    MPP Driver sets physical HBA moudle options during installation.     
    Preset options can be found in /opt/mpp/modprobe.conf.mppappend
    after MPP driver installation.

    Following are the recommendations for various HBA drivers:


    6.1  Qlogic HBA module options ( Module Name - qla2xxx )
    -----------------------------------------------------------

        ql2xfailover -> 0

	Disable failover support in Qlogic driver since failover
        is handled by MPP driver.


        ql2xretrycount -> 3

        Default retry count value is 20, we are setting it to a
        lower number and handle the IO return condition accordingly.
        This allows the IO to return to MPP driver from Qlogic driver
        before the MPP Driver's ControllerIOWaitTime ( Default 300 seconds )
        is expired. 


        ql2xprocessnotready -> 0

        Disable the handling of target not ready feature in Qlogic
        driver, MPP will control this. ( SK/ASC/ASCQ   02/04/01 )


        qlport_down_retry -> 35

        The above value needs to be set for any storage array types
        which have FC frond-end and back-end. The HBA driver doubles
        the value (70).
        This is to handle the port down timeout. According to our
        testing setting this value to 35 will buy us approximately
        70 seconds, after that qlogic driver returns DID_NO_CONNECT
        if the target is still gone. If the target comes back with
        in 70 seconds we do not need to perform a failover, else we
        will perform failover after qlogic responds with DID_NO_CONNECT
        after the 70 seconds.


        qlport_down_retry -> 72

        The above value needs to be set for any storage array types
        which have FC frond-end and SAS back-end. The HBA driver
        doubles the value (144).


    6.2  Emulex HBA module options ( Module Name - lpfc )
    -----------------------------------------------------------

        lpfc_nodev_tmo -> 60 	

        The above value needs to be set for any storage array types
        which have FC frond-end and back-end. This enables the lpfc
        driver to hold IO waiting for device to come back. This
        handles the target loss scenario. If the target comes back
        with in 60 seconds we do not need to perform a failover,
        else we will perform failover after lpfc responds with
        DID_NO_CONNECT after the 60 seconds.

        
        lpfc_nodev_tmo -> 144

        The above value needs to be set for any storage array types
        which have FC frond-end and SAS back-end.

    6.3  Brocade HBA module options ( Module Name - bfa )
    -----------------------------------------------------------

        rport_del_timeout -> 60 	

        The above value needs to be set for any storage array types
        which have FC frond-end and back-end. This enables the bfa
        driver to hold IO waiting for device to come back. This
        handles the target loss scenario. If the target comes back
        with in 60 seconds we do not need to perform a failover,
        else we will perform failover after bfa responds with
        DID_NO_CONNECT after the 60 seconds.

        
        rport_del_timeout -> 144

        The above value needs to be set for any storage array types
        which have FC frond-end and SAS back-end.

 
 
    6.4  LSI HBA module options    ( Module Name - mptscsih )
    -----------------------------------------------------------

        None. 	

    NOTE : These options are used in building the mpp initrd ramdisk.
           If you have to override the changes then module options
           can be specified in /etc/modprobe.conf for Redhat AS 4,
           and /etc/modprobe.conf.local for SUSE SLES 9.


 
7. Persistent Naming
================================================================
    Linux SCSI device names have the possibility of changing 
    when the host system reboots.  We recommend using a utility 
    such as devlabel to create user-defined device names that 
    will map devices based on a unique identifier, called a 
    UUID.

    As of this writing, devlabel is available as part of the
    Red Hat Enterprise Linux 3 distribution, or online at:
    http://www.lerhaupt.com/devlabel/devlabel.html.


 
8. RDAC Driver Package Uninstall and Reinstall/Update
================================================================
    To un-install the RDAC driver package, type the following 
    and press Enter:
 
          make uninstall
 
    To Reinstall/Update the RDAC driver package, you need to do
    a "make uninstall" followed by "make install" under the 
    /linuxrdac-xx.xx.xxxx.xxxx directory in the new RDAC driver source 
    code bundle.



9. Kernel Update
================================================================
    After applying the kernel update and booting into the new kernel, 
    please do the following things to build MPP Initrd image for the 
    new kernel:

    a. Change directory to  Linuxrdac source code directory.
	
    b. execute "make uninstall".

    c. Follow driver building and installation procedure described in  
       Section 5.3 and Section 5.4.
	

 
10. Physical HBA driver update
================================================================
    The following is the recommended procedure to update physical
    HBA driver :

    a. Get the latest hba drivers supported by MPP.

    b. install the drivers

    c. execute "mppUpdate" to create a new mpp initrd image.

    d. reboot the system to boot with the new initrd image.


    The following is the non-recommended procedure to update the 
    physical HBA driver using modprobe :

    a. unload MPP Virtual HBA Driver. execute "modprobe -r mppVhba".

    b. Unload the physical HBA driver stack. example "modprobe -r lpfc". 

    c. Unload MPP Upper Level Driver. execute "modprobe -r mppUpper".

    d. Get the latest hba drivers supported by MPP.

    e. compile / install the physical HBA driver.

    f. execute "mppUpdate" to create a new mpp initrd image.
       This step is important to pick up new physical HBA drivers 
       in the mpp initrd image for the next boot.
       

    g. load MPP Upper Level Driver. execute "modprobe mppUpper".

    h. execute "cat /opt/mpp/modprobe.conf.mppappend" to see the mpp 
       generated physical module parameters if any.

       example : options lpfc lpfc_nodev_tmo=60

    i. load the physical HBA driver stack using the module parameters. 
       example "modprobe lpfc lpfc_nodev_tmo=60". 

    j. load MPP Virtual HBA Driver. execute "modprobe mppVhba".
   

NOTE : Please see mpp supported module options in section "Physical HBA 
       Module options".

NOTE : Qlogic hba driver has a module parameter "ql2xfailover" 
       If the module parameter is not set to 0, qlogic driver acts 
       as a failover driver itself and hides the paths.




11. Initial Ramdisk image (initrd)
================================================================
    Initial RamDisk Image (initrd image) is automatically created 
    when the driver is installed using "make install". Boot loader
    Configuration file needs to have entry for this newly created 
    image. 
	
    The initrd image is located in boot partition  with file 
    name :'mpp-`uname -r`.img.
    { Example : mpp-2.4.21-9.ELsmp.img }

    For a Driver Update, If the system already has old entry for
    MPP, the system administrator has to just modify the existing
    MPP entry accordingly in the boot loader configuration file.
    In most of the cases no change is required if the kernel 
    version is same. 
    
    To create new 'initrd image' user has to run 'mppUpdate' which
    will overwrite old image file with the new image file.

    If 3rd party drivers needed to be added to initrd image, edit
    the /etc/sysconfig/kernel ( SUSE ) or /etc/modprobe.conf 
    (RedHat) with the 3rd party driver entires. Run 'mppUpdate' 
    to create new ramdisk image. 



12. Restrictions
================================================================
    a. Cluster support 
       The cluster support feature in the Linux RDAC driver is 
       only available for Sonoran 4 or later firmware. If a 
       SCSI-2 reserve/release command is addressed to a LUN on 
       an array running a firmware version lower than 05.40, a 
       check-condition 05-02-00 (key-asc-ascq) (illegal 
       request, invalid command operation code) will be  
       returned. 

    b. Fibre channel configuration 
       If a host has multiple HBA ports and each HBA port sees 
       both controllers (via a un-zoned switch), the Linux RDAC 
       driver may return I/O errors during controller failover.
 
       Recommendation: Use two switches or a zoned switch. 

    c. The RDAC driver reports I/O failures immediately after
       all paths are failed.
       When the RDAC driver detects that all paths to a storage 
       array are failed, it will report I/O failure immediately.
       This behavior is different from the QLogic failover 
       driver. The QLogic failover driver will wait for a 
       certain time period before reporting an I/O failure to 
       the host application. 

       Workaround: None.

    d. The RDAC driver stack, which includes the mppUpper, Physical 
       HBA driver and the mppVhba, should not be loaded 
       or unloaded using the modprobe kernel module utility. Any
       use of modprobe on the RDAC driver stack is entirely the 
       responsibility of the user. MPP does not recommend or
       support the use of modprobe with the RDAC driver stack. 

       Recommendation: Reboot the system whenever a need to unload
       the driver stack arises. Use of rmmod to remove all the 
       drivers in the RDAC driver stack, in order, is also not supported.

       Reasoning: MPP creates special initrd image with the required 
       driver modules and their module parameters.User should be aware
       of the correct module settings and order to perform the modprobe.
       Since the unloading of the mpp drivers and physical hba drivers
       will lose the connectivity to storage the best way is to reboot
       the system. 

       See section "Physical HBA driver update" for correct procedure.


13. GPL License
================================================================
       RDAC driver modules (mppUpper.ko and mppVhba.ko) 
       carry GPL License.

