How to connect Linux to a storage unit - Guillaume CRESTA

Feb 24, 2006 - Within the framework of this project, an integration of. System x has been effected. .... SVC_SpeederMan_EW 0042 50050768014003EA SVC.
1MB taille 2 téléchargements 405 vues
High Availability

Continuous Availability

Continuous Operations

How to connect Linux to a storage unit

PSSC - IBM Customer Center Montpellier Guillaume CRESTA

1

France

Table of Contents

1

Introduction ........................................................................................................................ 3

Presentation of the infrastructure .......................................................3 2

How to configure................................................................................................................ 4

2.1 2.2 2.3 2.4 2.5 2.6 3

Gathering the World Wide Name..............................................4 Configuring the unit storage......................................................5 Configuration at the SAN level: zoning....................................9 The operating system: Linux...................................................12 Script........................................................................................20 Dscli .........................................................................................21

Special Thanks ................................................................................................................. 23

2

1 Introduction In order to answer to some of customers’ main concerns and to provide them with answers based on real testing, we are implementing a two sites long distance disaster recovery solution between Montpellier (France) and Poughkeepsie (USA) that is 7000 km between the production site and the backup site. Within the framework of this project, an integration of System x has been effected. The purpose of this document is to describe the configuration of the System x with RedHat Enterprise Linux 4.0 and the storage unit DS8000 o DS6000 based on this real experience. In order to connect these two technologies, we ran different steps explained all over the document. The last part of this document is dedicated to the persons who help me.

Presentation of the infrastructure The main components of the architecture are:  IBM System p servers  IBM System x servers  IBM DS8000 and DS6000 disk subsystems  MCDATA SAN switches  MCDATA iFCP router 2027-R16 The hardware environment we are using for this test is dispatched in 2 sites (one site in Montpellier-France and one site in Poughkeepsie-US).

3

2 How to configure Different steps have to be done in order to attach a Linux (RedHat Enterprise Linux release 4.0) to a DS8000 storage unit: 

gather the World Wide Name, unique identifier of each fiber card and that permits the identification of the System x on storage unit



configure the storage



make configuration at SAN level



install sdd service



to configure Linux in order to show disks



make starting script and at the end



install the dscli tool in order to communicate with storage unit.

2.1 Gathering the World Wide Name First we have to see the compatibility matrix in order to verify if the Linux version installed is in

agreement

to

the

storage

unit

(http://www-

03.ibm.com/servers/storage/disk/ds8000/pdf/ds8000-matrix.pdf). As far as that goes, we have to know the Linux version and his kernel: [root@halinux Desktop]# more /etc/redhat-release Red Hat Enterprise Linux AS release 4 (Nahant Update 3) [root@halinux Desktop]# uname -a Linux halinux 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006 i686 i686 i386 GNU/Linux

Then, in order to make zoning, we need the WWPN of fiber channel card of the System x: [root@halinux Desktop]# cd /proc/scsi/qla2xxx/ [root@halinux qla2xxx]# ls 1 [root@halinux qla2xxx]# cat 1 QLogic PCI to Fibre Channel Host Adapter for QLA2310: Firmware version 3.03.18 IPX, Driver version 8.01.02-d4 ISP: ISP2300, Serial# D76762 Request Queue = 0x1bc0000, Response Queue = 0x37a00000 Request Queue count = 2048, Response Queue count = 512 Total number of active commands = 0 Total number of interrupts = 2781 Device queue depth = 0x20 Number of free request entries = 784 Number of mailbox timeouts = 0 Number of ISP aborts = 0 Number of loop resyncs = 0 Number of retries for empty slots = 0 Number of reqs in pending_q= 0, retry_q= 0, done_q= 0, scsi_retry_q= 0 Host adapter:loop state = , flags = 0x1a03

4

Dpc flags = 0x4000000 MBX flags = 0x0 Link down Timeout = 000 Port down retry = 030 Login retry count = 030 Commands retried with dropped frame(s) = 0 Product ID = 4953 5020 2020 0001 SCSI Device Information: scsi-qla0-adapter-node=200000e08b05babf; scsi-qla0-adapter-port=210000e08b05babf; scsi-qla0-target-0=50050763031080b7; scsi-qla0-target-1=50050763031bc0b7; FC Port Information: scsi-qla0-port-0=5005076303ffc0b7:50050763031080b7:740d13:81; scsi-qla0-port-1=5005076303ffc0b7:50050763031bc0b7:740f13:82; SCSI LUN Information: (Id:Lun) * - indicates lun is not registered with the OS. ( 0: 0): Total reqs 1674, Pending reqs 0, flags 0x0, 0:0:81 00 ( 1: 0): Total reqs 1587, Pending reqs 0, flags 0x0, 0:0:82 00 [root@halinux qla2xxx]#

The LUNs mapped to the server can be verified by checking the contains of the file /proc/scsi/scsi: [root@halinux ~]# cat /proc/scsi/scsi Host:SCSI0 Channel:00 Id:00 Lun:00 Vendor: IBM-ESXS Model:CBR036C31210ESFN Rev: DFQD Type: Direct-Access ANSI SCSI revision:04 Host:SCSI0 Channel:00 Id:08 Lun:00 Vendor: IBM Model:FT1V1S2 Rev: 0 Type: Processor ANSI SCSI revision:02 Host:SCSI1 Channel:00 Id:00 Lun:00 Vendor: IBM-ESXS Model:2107900 Rev: 201 Type: Direct-Access ANSI SCSI revision:05 Host:SCSI1 Channel:00 Id:01 Lun:00 Vendor: IBM-ESXS Model:2107900 Rev: 201 Type: Direct-Access ANSI SCSI revision:05

2.2 Configuring the unit storage The configuration of the unit storage can be done by GUI (web interface) or by CLI (commands line). I myself prefer working with commands line. The working is the same with a DS6000 or with a DS8000. The below extracts correspond to the creation on DS8000.

5

To create host on storage unit permitting relation between Linux and storage unit, the World Wide Port Name mentioned on Linux (210000E08B05BABF), the type of operating system used (LinuxRHEL) and the DS ports used (all in order to mention all available): dscli mkhostconnect -wwname 210000E08B05BABF -hosttype LinuxRHEL -ioport all 'HA1Linux' -cfg C:\Progra~1\IBM\dscli\profile\DS8000_HA.profile Date/Time: 4 ao?t 2006 14:25:30 CEST IBM DSCLI Version: 5.1.600.260 DS: IBM.2107-7506551 CMUC00012I mkhostconnect: Host connection 0013 successfully created. Date/Time: 08 04, 2006 14:25:39 Execution time : 12 Seconds.

The configuration made can be verified: dscli lshostconnect -l -cfg C:\Progra~1\IBM\dscli\profile\DS8000_HA.profile Date/Time: 25 juillet 2006 18:42:57 CEST IBM DSCLI Version: 5.1.600.260 DS: IBM.2107-7506551 Name ID WWPN HostType LBS addrDiscovery Profile portgrp volgrpID atchtopo ESSIOport =========================================================================== =========================================================================== =========================================================================== =========================================================================== ============================================================== BSBI_P53 0000 10000000C93EC6FE pSeries 512 reportLUN IBM pSeries - AIX 1 FC-AL all […] HA1Linux 0013 210000E08B05BABF LinuxRHEL 512 LUNPolling Intel - Linux RHEL 0 all […] SVC_SpeederMan_EW 0042 50050768014003EA SVC 512 reportLUN San Volume Controller 0 V2 all Date/Time: 07 25, 2006 18:43:06 Execution time : 11 Seconds.

Moreover, a space on storage unit called Volumes is created, specifying the capacity (5 Gb), the volume name (Vol1Halinux) and a number available (47000) and the extend pool on which it’s going to be created (P3). Before, we need to verify that the extend pool has allocated space. dscli lsextpool -l -cfg C:\Progra~1\IBM\dscli\profile\DS8000_HA.profile Date/Time: 4 ao?t 2006 14:29:36 CEST IBM DSCLI Version: 5.1.600.260 DS: IBM.2107-7506551 Name ID stgtype rankgrp status availstor (2^30B) %allocated available reserved numvols numranks =========================================================================== ============================= extpool_ha0 P0 fb 0 below 650 16 650 0 8 1 extpool_ha1 P1 fb 1 below 679 12 679 0 4 1 extpool_ha2 P2 fb 0 below 859 5 859 0 2 1

6

extpool_ha3 P3 fb 1 below 729 0 2 1 […] Date/Time: 08 04, 2006 14:29:51 Execution time : 17 Seconds.

729

6

dscli mkfbvol -extpool P3 -cap 5 -name 'Vol1Halinux' 4700 -cfg C:\Progra~1\IBM\dscli\profile\DS8000_HA.profile Date/Time: 4 ao?t 2006 14:30:45 CEST IBM DSCLI Version: 5.1.600.260 DS: IBM.2107-7506551 CMUC00025I mkfbvol: FB volume 4700 successfully created. Date/Time: 08 04, 2006 14:30:55 Execution time : 12 Seconds.

When the volumes are established they are grouped in a volume group. In this case, there’s only one volume. But, in order to connect the host to the volume, a volume group need to be created. When the volume group is created by command line interface, the type SCSIMASK o SCSIMap256 can be configured. By default, if you put SCSIMask, the relation between volume group and host configured as a RedHat EL is impossible in this case. In order to make a volume group, the type (scsimpa256 in case of a Linux RedHatEL), the volumes number gathered in the volume group (4700) and volume group name (VolGrpe1HALinux) have to be specified: dscli mkvolgrp -type scsimap256 -volume 4700 'Volgrpe1HALinux' -cfg C:\Progra~1\IBM\dscli\profile\DS8000_HA.profile Date/Time: 4 ao?t 2006 14:31:45 CEST IBM DSCLI Version: 5.1.600.260 DS: IBM.2107-7506551 CMUC00030I mkvolgrp: Volume group V4 successfully created. Date/Time: 08 04, 2006 14:31:53 Execution time : 10 Seconds.

The volume establishment by command line can be verified: dscli lsvolgrp -l -cfg C:\Progra~1\IBM\dscli\profile\DS8000_HA.profile Date/Time: 4 ao?t 2006 14:22:27 CEST IBM DSCLI Version: 5.1.600.260 DS: IBM.2107-7506551 Name ID Type ======================================== FC_OracleCode V0 SCSI Mask HAVG1 V1 SCSI Mask SVCDEMO_EW V2 SCSI Mask MED4PC12 V3 SCSI Map 256 VolGrp1_HALINUX V4 SCSI Map 256 sdvolume V5 SCSI Map 256 VGPC2 V6 SCSI Map 256 blue_aix_VG V7 SCSI Mask Volumes_for_AIX V8 SCSI Mask Date/Time: 08 04, 2006 14:22:36 Execution time : 12 Seconds.

7

The volume establishment can be verified and so that it is associated to the volume group V4: dscli lsfbvol -l -cfg C:\Progra~1\IBM\dscli\profile\DS8000_HA.profile Date/Time: 17 mai 2006 18:40:59 CEST IBM DSCLI Version: 5.1.600.260 DS: IBM.2107-7506551 Name ID accstate datastate configstate deviceMTM datatype extpool sam captype cap (2^30B) cap (10^9B) cap (blocks) volgrp =========================================================================== ================================================================ 03000 0300 Online Normal Normal 2107-900 FB 512 P7 Standard DS 2.0 4194304 V18 [...] Vol1Halinux 4700 Online Normal Normal 2107-900 FB 512 P3 Standard DS 5.0 10485760 V4 [...] volAIXLaurent FB00 Online Normal Normal 2107-900 FB 512 P7 Standard DS 1.0 2097152 Date/Time: 05 17, 2006 18:41:08 Execution time : 12 Seconds.

We need to associate the volume group to the host previously established, specifying the volume group number (V4) and the host one (0013): dscli chhostconnect -volgrp V4 0013 -cfg C:\Progra~1\IBM\dscli\profile\DS8000_HA.profile Date/Time: 4 ao?t 2006 14:32:43 CEST IBM DSCLI Version: 5.1.600.260 DS: IBM.2107-7506551 CMUC00013I chhostconnect: Host connection 0013 successfully modified. Date/Time: 08 04, 2006 14:32:51 Execution time : 10 Seconds.

It can be verified that this operation has been realized by watching the host configuration again: dscli lshostconnect -l -cfg C:\Progra~1\IBM\dscli\profile\DS8000_HA.profile Date/Time: 25 juillet 2006 18:42:57 CEST IBM DSCLI Version: 5.1.600.260 DS: IBM.2107-7506551 Name ID WWPN HostType LBS addrDiscovery Profile portgrp volgrpID atchtopo ESSIOport =========================================================================== =========================================================================== =========================================================================== =========================================================================== ============================================================== BSBI_P53 0000 10000000C93EC6FE pSeries 512 reportLUN IBM pSeries - AIX 1 FC-AL all […] HA1Linux 0013 210000E08B05BABF LinuxRHEL 512 LUNPolling Intel - Linux RHEL 0 V4 all […] SVC_SpeederMan_EW 0042 50050768014003EA SVC 512 reportLUN San Volume Controller 0 V2 all Date/Time: 07 25, 2006 18:43:06 Execution time : 11 Seconds.

8

2.3 Configuration at the SAN level: zoning In order to administrate the switches, a web application is available. Thanks to EFCM Basic, we can configure switches by accessing to the menu Configure > Zoning and the page figuring next page can be seen.

In the first column, all nodes are listed. The zone name is notified in the second column.

The World Wide Port Names have to be added by using the button >.

9

In the zone S1_LINUX, the System x World Wide Port Names and the unit storage World Wide Port Names can be verified to be there. When this inspection is done, this zone has to be joined to the zone set. The button > is used to realized it.

10

Now, the zone appears in the column Zone set. Thus, the modification is saved by the button Update and applied by Activate.

The created zones are editable using the button < and the steps above remade. The port used is configured. In this case, System x is connected on switch port 0. The fields associated to this port are informed.

11

Thereafter, we can access to the list of the switch ports (Product > Port List). Clicking on port 0, information linked to this port appear, for example le World Wide Port Name and the World Wide Node Name.

The zoning and the storage unit configuration can be done in parallel. The order done doesn’t matter.

2.4 The operating system: Linux  The integration of sdd service After the DS6/8000 configuration, the operating operation has to be restarted. It’s necessary to verify that the disks are in relation with the operating system using the fdisk -l command: login as: root Sent username "root" [email protected]'s password: Last login: Wed Aug 16 11:04:10 2006 from 10.8.1.1 [root@halinux ~]# fdisk -l Disk /dev/sda: 36.4 GB, 36401479680 bytes 255 heads, 63 sectors/track, 4425 cylinders

12

Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot /dev/sda1 * /dev/sda2

Start 1 14

End 13 4425

Blocks 104391 35439390

Id 83 8e

System Linux Linux LVM

Disk /dev/sdb: 5368 MB, 5368709120 bytes 166 heads, 62 sectors/track, 1018 cylinders Units = cylinders of 10292 * 512 = 5269504 bytes Disk /dev/sdb doesn't contain a valid partition table Disk /dev/sdc: 5368 MB, 5368709120 bytes 166 heads, 62 sectors/track, 1018 cylinders Units = cylinders of 10292 * 512 = 5269504 bytes Disk /dev/sdc doesn't contain a valid partition table

If this command doesn’t work, the disks mount through the Fiber Channel card can be forced by the command line: echo "scsi-qlascan" > /proc/scsi/qla2xxx/1. The System x linked to the DS8000 has got only one card, and when the zoning is made, two links are put to the storage unit. The disk also appears in double: /etc/sdb and /etc/sdc above. Whereas on System x linked to the DS6000 there are two cards each linked to the storage unit by two fibers, four disks appear as notified below (/etc/sdb, /etc/sdc, /etc/sdd and /etc/sde): login as: root Sent username "root" [email protected]'s password: Last login: Wed Aug 16 11:05:28 2006 from 10.8.1.1 [root@halinux2 ~]# fdisk -l Disk /dev/sda: 36.4 GB, 36401479680 bytes 255 heads, 63 sectors/track, 4425 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot /dev/sda1 * /dev/sda2

Start 1 14

End 13 4425

Blocks 104391 35439390

Id 83 8e

System Linux Linux LVM

Disk /dev/sdb: 5368 MB, 5368709120 bytes 166 heads, 62 sectors/track, 1018 cylinders Units = cylinders of 10292 * 512 = 5269504 bytes Disk /dev/sdb doesn't contain a valid partition table Disk /dev/sdc: 5368 MB, 5368709120 bytes 166 heads, 62 sectors/track, 1018 cylinders Units = cylinders of 10292 * 512 = 5269504 bytes Disk /dev/sdc doesn't contain a valid partition table Disk /dev/sdd: 5368 MB, 5368709120 bytes 166 heads, 62 sectors/track, 1018 cylinders Units = cylinders of 10292 * 512 = 5269504 bytes Disk /dev/sdd doesn't contain a valid partition table

13

Disk /dev/sde: 5368 MB, 5368709120 bytes 166 heads, 62 sectors/track, 1018 cylinders Units = cylinders of 10292 * 512 = 5269504 bytes Disk /dev/sde doesn't contain a valid partition table

Thus, the file IBM.sdd-1.6.1.0-3.i686.rhel4.rpm is installed.

[root@halinux ~]# rpm -ivh IBMsdd-1.6.0.1-11.1.i686.rhel4.rpm Preparing.... ####################################################[100%] 1:IBMsdd ####################################################[100%] Added following line to /etc/inittab: srv:345:respawn:/opt/IBMsdd/bin/sddsrv > /dev/null 2>&1

In order to verify if it is correctly installed, /etc/inittab can be verified: [root@halinux ~]# cat /etc/inittab | grep sdd #IBMsdd path recovery daemon: srv:345:respawn:/opt/IBMsdd/bin/sddsrv > /dev/null 2>&1

When done, you can verify the content of the package: [root@halinux ~]# rpm -ql IBMsdd /etc/cron.hourly/sddrv_log.sh /etc/logrotate.d/sddsrv_log.d /etc/sddsrv.conf /etc/vpath.conf /opt/IBMsdd /opt/IBMsdd/bin /opt/IBMsdd/bin/cfgvpath /opt/IBMsdd/bin/datapath /opt/IBMsdd/bin/lsvpcfg /opt/IBMsdd/bin/make_sddcrs /opt/IBMsdd/bin/pathtest /opt/IBMsdd/bin/rmvpath /opt/IBMsdd/bin/sdd_rcscript /opt/IBMsdd/bin/sddgetdata /opt/IBMsdd/bin/sddsrv /opt/IBMsdd/kernel_list /opt/IBMsdd/sdd-mod.ko-2.6.9-11.ELhugemen /opt/IBMsdd/sdd-mod.ko-2.6.9-11.ELsmp /opt/IBMsdd/sdd-mod.ko-2.6.9-22.0.1.ELhugemen /opt/IBMsdd/sdd-mod.ko-2.6.9-22.0.1.ELsmp /opt/IBMsdd/sdd-mod.ko-2.6.9-22.0.2.ELhugemen /opt/IBMsdd/sdd-mod.ko-2.6.9-22.0.2.ELsmp /opt/IBMsdd/sdd-mod.ko-2.6.9-22.ELhugemen /opt/IBMsdd/sdd-mod.ko-2.6.9-22.ELsmp /opt/IBMsdd/sdd-mod.ko-2.6.9-34.ELhugemen /opt/IBMsdd/sdd-mod.ko-2.6.9-34.ELsmp /usr/share/doc/IBMsdd-1.6.1.0/LICENSE /usr/share/doc/IBMsdd-1.6.1.0/README /usr/share/doc/IBMsdd-1.6.1.0/rd-linux.txt

This file is the last version of sdd for RedHat 4.0. When the file is installed, the status has to be verified: 14

[root@halinux ~]# sdd status IBMsdd status: You have new mail in /var/spool/mail/root

[FAILED]

If the status is FAILED, the sdd service has to be started: [root@halinux ~]# sdd start Starting IBMsdd driver load: [ OK ] Issuing killall sddsrv to trigger respawn... Starting IBMsdd configuration: [ OK ] [root@halinux ~]# sdd status IBMsdd status: [ OK ] 000 vpatha ( 252, 0) 75065514700 = 6005076303ffc0b70000000000004700 = /dev/sdb /dev/sdc

In order to make multipathing, sdd make a vpath going to take into account the two disks and manage the traffic between the two links.

 The visualization of the disks When an operating system Linux and especially sdd is configured, the configuration of the file /etc/lvm/lvm.conf has to be modified. Indeed, if you don’t modify it, you can’t create a vpath in order to establish multipathing. In order to resolve this problem two lines in this file have to be modified: filter = ["a/.*/"] types = ["fd", 16]

by the lines: filter = ["a|/dev/sda|", "r|/dev/sd|", "a/.*/"] types = ["vpath", 16]

These lines permit Linux server not to filter the other disk as sda (principal disk which is installed Linux). When this step is done, we can also show that the path is created: [root@halinux ~]#cfgvpath query /dev/sdb ( 8, 16) host=1 ch=0 id=0 lun=0 vid=IBM pid=2107900 serial=75065514700 lun_id=6005076303ffc0b70000000000004700 ctlr_flag=0 ctlr_nbr=0 df_ctlr=0 /dev/sdc ( 8, 32) host=1 ch=0 id=1 lun=0 vid=IBM pid=2107900 serial=75065514700 lun_id=6005076303ffc0b70000000000004700 ctlr_flag=0 ctlr_nbr=0 df_ctlr=0

When the scan disk is made, we can notice that a new disk is present and correspond to the path group: 15

[root@halinux ~]# fdisk -l …………………………… Disk /dev/vpatha: 5368 MB, 5368709120 bytes 166 heads, 62 sectors/track, 1018 cylinders Units = cylinders of 10292 * 512 = 5269504 bytes Disk /dev/vpatha doesn't contain a valid partition table

Then, we need to make a partition: [root@halinux ~]# fdisk /dev/vpatha Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel Building a new DOS disklabel. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable. Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite) Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-1018, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-1018, default 1018): Using default value 1018 Command (m for help): t Selected partition 1 Hex code (type L to list codes): l 0 Empty boot 1 FAT12 2 XENIX root (FAT3 XENIX usr (FAT4 FAT16