Configuring multipath devices for Oracle Clusterware 11g
Udev and Device Name Persistency
Unlike devlabel in the 2.4 kernel, udev (the 2.6 kernel device file naming scheme) dynamically creates device file names at boot time. However, this can give rise to the possibility that device file names may change – a device that may have once been named /dev/sdd say, may be renamed as /dev/sdf after reboot. Without specific configuration, if udev is left to dynamically name devices, the potential exists for devices referred to, or inadvertently accessed by, their arbitrary kernel-assigned name (e.g. Oracle Clusterware files; Cluster Registry, Voting disks, etc.) to fail to start or become corrupt.
Multipath and Udev
The creation of multipath devices is performed through the use of device-mapper-multipath, which although drawing upon device-mapper, is not completely integrated into the udev naming scheme. Significant modification of default udev rules is required to manipulate multipath device naming, therefore introduces potential supportability issues. Therefore, other means are required to configure device naming persistency and user-defined naming of multipath devices.
Before being able to configure udev to explicitly name devices, scsi_id(8) should first be configured to return their device identifiers. Modify the /etc/scsi_id.config file – add. or replace the existing ‘option=-b’ parameter/value pair (if exists), with ‘option=-g’, for example:
# grep -v ^# /etc/scsi_id.config
vendor="ATA",options=-p 0x80
options=-g
check the partitions names, these names are frequently gets change after each reboot.
[root@rac-node1 ~]# cat /proc/partitions
major minor #blocks name
104 0 143338560 cciss/c0d0
104 1 102400 cciss/c0d0p1
104 2 143236126 cciss/c0d0p2
8 0 20971520 sda
8 16 104857600 sdb
8 17 104856223 sdb1
8 32 104857600 sdc
8 33 104856223 sdc1
8 48 1048576 sdd
8 49 1044193 sdd1
8 64 1048576 sde
8 65 1044193 sde1
8 80 262144000 sdf
8 81 262140606 sdf1
8 96 20971520 sdg
8 112 104857600 sdh
8 113 104856223 sdh1
8 128 104857600 sdi
8 129 104856223 sdi1
8 144 1048576 sdj
8 145 1044193 sdj1
8 160 1048576 sdk
8 161 1044193 sdk1
8 176 262144000 sdl
8 177 262140606 sdl1
8 192 20971520 sdm
8 208 104857600 sdn
8 209 104856223 sdn1
8 224 104857600 sdo
8 225 104856223 sdo1
8 240 1048576 sdp
8 241 1044193 sdp1
65 0 1048576 sdq
65 1 1044193 sdq1
65 16 262144000 sdr
65 17 262140606 sdr1
65 32 20971520 sds
65 48 104857600 sdt
65 49 104856223 sdt1
65 64 104857600 sdu
65 65 104856223 sdu1
65 80 1048576 sdv
65 81 1044193 sdv1
65 96 1048576 sdw
65 97 1044193 sdw1
65 112 262144000 sdx
65 113 262140606 sdx1
253 0 20480000 dm-0
253 1 10223616 dm-1
253 2 30703616 dm-2
253 3 8192000 dm-3
253 4 10223616 dm-4
253 5 491520 dm-5
253 6 40960000 dm-6
253 7 8192000 dm-7
253 8 20971520 dm-8
253 9 104857600 dm-9
253 10 104857600 dm-10
253 11 1048576 dm-11
253 12 1048576 dm-12
253 13 262144000 dm-13
253 14 1044193 dm-14
253 15 104856223 dm-15
253 16 262140606 dm-16
253 17 1044193 dm-17
253 18 104856223 dm-18
[root@rac-node1 ~]#
using below script to know the disks with LUN number which is not changing on reboot
[root@rac-node1 ~]# for i in `cat /proc/partitions | awk '{print $4}' |grep sd`; do echo "### $i: `scsi_id -g -u -s /block/$i`"; done
### sda: 3600508b4000986c40000500000280000
### sdb: 3600508b4000986c400005000002c0000
### sdb1:
### sdc: 3600508b4000986c400005000002f0000
### sdc1:
### sdd: 3600508b4000986c40000500000350000
### sdd1:
### sde: 3600508b4000986c40000500000320000
### sde1:
### sdf: 3600508b4000986c40000500000500000
### sdf1:
### sdg: 3600508b4000986c40000500000280000
### sdh: 3600508b4000986c400005000002c0000
### sdh1:
### sdi: 3600508b4000986c400005000002f0000
### sdi1:
### sdj: 3600508b4000986c40000500000350000
### sdj1:
### sdk: 3600508b4000986c40000500000320000
### sdk1:
### sdl: 3600508b4000986c40000500000500000
### sdl1:
### sdm: 3600508b4000986c40000500000280000
### sdn: 3600508b4000986c400005000002c0000
### sdn1:
### sdo: 3600508b4000986c400005000002f0000
### sdo1:
### sdp: 3600508b4000986c40000500000350000
### sdp1:
### sdq: 3600508b4000986c40000500000320000
### sdq1:
### sdr: 3600508b4000986c40000500000500000
### sdr1:
### sds: 3600508b4000986c40000500000280000
### sdt: 3600508b4000986c400005000002c0000
### sdt1:
### sdu: 3600508b4000986c400005000002f0000
### sdu1:
### sdv: 3600508b4000986c40000500000350000
### sdv1:
### sdw: 3600508b4000986c40000500000320000
### sdw1:
### sdx: 3600508b4000986c40000500000500000
### sdx1:
[root@rac-node1 ~]#
Configure the multipath.conf file for raw and ASM disks
cat /etc/multipath.conf
multipaths {
multipath {
wwid 3600508b4000986c40000500000350000
alias voting
}
multipath {
wwid 3600508b4000986c40000500000320000
alias ocr
}
multipath {
wwid 3600508b4000986c400005000002f0000
alias data
}
multipath {
wwid 3600508b4000986c400005000002c0000
alias index
}
multipath {
wwid 3600508b4000986c40000500000500000
alias rman
}
}
confirm changes in the multipath.conf file
[root@rac-node1 ~]# multipath -ll index (3600508b4000986c400005000002c0000) dm-9 HP,HSV300 [size=100G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=100][active] \_ 0:0:1:2 sdh 8:112 [active][ready] \_ 1:0:0:2 sdn 8:208 [active][ready] \_ round-robin 0 [prio=20][enabled] \_ 0:0:0:2 sdb 8:16 [active][ready] \_ 1:0:1:2 sdt 65:48 [active][ready] mpath0 (3600508b4000986c40000500000280000) dm-8 HP,HSV300 [size=20G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=100][active] \_ 0:0:1:1 sdg 8:96 [active][ready] \_ 1:0:0:1 sdm 8:192 [active][ready] \_ round-robin 0 [prio=20][enabled] \_ 0:0:0:1 sda 8:0 [active][ready] \_ 1:0:1:1 sds 65:32 [active][ready] voting (3600508b4000986c40000500000350000) dm-11 HP,HSV300 [size=1.0G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=100][active] \_ 0:0:0:4 sdd 8:48 [active][ready] \_ 1:0:1:4 sdv 65:80 [active][ready] \_ round-robin 0 [prio=20][enabled] \_ 0:0:1:4 sdj 8:144 [active][ready] \_ 1:0:0:4 sdp 8:240 [active][ready] rman (3600508b4000986c40000500000500000) dm-13 HP,HSV300 [size=250G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=100][active] \_ 0:0:0:6 sdf 8:80 [active][ready] \_ 1:0:1:6 sdx 65:112 [active][ready] \_ round-robin 0 [prio=20][enabled] \_ 0:0:1:6 sdl 8:176 [active][ready] \_ 1:0:0:6 sdr 65:16 [active][ready] data (3600508b4000986c400005000002f0000) dm-10 HP,HSV300 [size=100G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=100][active] \_ 0:0:0:3 sdc 8:32 [active][ready] \_ 1:0:1:3 sdu 65:64 [active][ready] \_ round-robin 0 [prio=20][enabled] \_ 0:0:1:3 sdi 8:128 [active][ready] \_ 1:0:0:3 sdo 8:224 [active][ready] ocr (3600508b4000986c40000500000320000) dm-12 HP,HSV300 [size=1.0G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=100][active] \_ 0:0:1:5 sdk 8:160 [active][ready] \_ 1:0:0:5 sdq 65:0 [active][ready] \_ round-robin 0 [prio=20][enabled] \_ 0:0:0:5 sde 8:64 [active][ready] \_ 1:0:1:5 sdw 65:96 [active][ready] [root@rac-node1 ~]#
and
[root@rac-node1 ~]# dmsetup ls | sort data (253, 10) datap1 (253, 18) index (253, 9) indexp1 (253, 15) mpath0 (253, 8) ocr (253, 12) ocrp1 (253, 14) rman (253, 13) rmanp1 (253, 16) VolGroup00-boot (253, 5) VolGroup00-home (253, 6) VolGroup00-LogVol00 (253, 0) VolGroup00-LogVol01 (253, 7) VolGroup00-opt (253, 4) VolGroup00-tmp (253, 3) VolGroup00-user (253, 2) VolGroup00-var (253, 1) voting (253, 11) votingp1 (253, 17) [root@rac-node1 ~]#
Check permission and disk by id
[root@rac-node1 ~]# ls -l /dev/disk/by-id/ total 0 lrwxrwxrwx 1 root root 9 Jun 19 15:21 scsi-3600508b4000986c40000500000280000 -> ../../sdm lrwxrwxrwx 1 root root 9 Jun 19 15:21 scsi-3600508b4000986c400005000002c0000 -> ../../sdn lrwxrwxrwx 1 root root 10 Jun 19 15:22 scsi-3600508b4000986c400005000002c0000-part1 -> ../../sdt1 lrwxrwxrwx 1 root root 9 Jun 19 15:21 scsi-3600508b4000986c400005000002f0000 -> ../../sdo lrwxrwxrwx 1 root root 10 Jun 19 15:22 scsi-3600508b4000986c400005000002f0000-part1 -> ../../sdo1 lrwxrwxrwx 1 root root 9 Jun 19 15:21 scsi-3600508b4000986c40000500000320000 -> ../../sde lrwxrwxrwx 1 root root 10 Jun 19 15:22 scsi-3600508b4000986c40000500000320000-part1 -> ../../sdw1 lrwxrwxrwx 1 root root 9 Jun 19 15:21 scsi-3600508b4000986c40000500000350000 -> ../../sdj lrwxrwxrwx 1 root root 10 Jun 19 15:22 scsi-3600508b4000986c40000500000350000-part1 -> ../../sdp1 lrwxrwxrwx 1 root root 9 Jun 19 15:21 scsi-3600508b4000986c40000500000500000 -> ../../sdx lrwxrwxrwx 1 root root 10 Jun 19 15:22 scsi-3600508b4000986c40000500000500000-part1 -> ../../sdx1 lrwxrwxrwx 1 root root 9 Jun 19 15:21 usb-TEAC_DVD-ROM_DW-224EV_A60020000001 -> ../../sr0 [root@rac-node1 ~]#
check exact numbers of disks (LUN)
[root@rac-node1 ~]# ls -l /dev/dm-* brw-rw---- 1 root root 253, 10 Jun 19 15:21 /dev/dm-10 brw-rw---- 1 root root 253, 11 Jun 19 15:21 /dev/dm-11 brw-rw---- 1 root root 253, 12 Jun 19 15:21 /dev/dm-12 brw-rw---- 1 root root 253, 13 Jun 19 15:21 /dev/dm-13 brw-rw---- 1 root root 253, 14 Jun 19 15:21 /dev/dm-14 brw-rw---- 1 root root 253, 15 Jun 19 15:21 /dev/dm-15 brw-rw---- 1 root root 253, 16 Jun 19 15:21 /dev/dm-16 brw-rw---- 1 root root 253, 17 Jun 19 15:21 /dev/dm-17 brw-rw---- 1 root root 253, 18 Jun 19 15:21 /dev/dm-18 brw-rw---- 1 root root 253, 8 Jun 19 15:21 /dev/dm-8 brw-rw---- 1 root root 253, 9 Jun 19 15:21 /dev/dm-9 [root@rac-node1 ~]#
check exact numbers of disks (LUN)
[root@rac-node1 ~]# ls -l /dev/mpath/* lrwxrwxrwx 1 root root 7 Jun 19 15:21 /dev/mpath/3600508b4000986c40000500000280000 -> ../dm-8 lrwxrwxrwx 1 root root 8 Jun 19 15:21 /dev/mpath/data -> ../dm-10 lrwxrwxrwx 1 root root 8 Jun 19 15:21 /dev/mpath/datap1 -> ../dm-18 lrwxrwxrwx 1 root root 7 Jun 19 15:21 /dev/mpath/index -> ../dm-9 lrwxrwxrwx 1 root root 8 Jun 19 15:21 /dev/mpath/indexp1 -> ../dm-15 lrwxrwxrwx 1 root root 8 Jun 19 15:21 /dev/mpath/ocr -> ../dm-12 lrwxrwxrwx 1 root root 8 Jun 19 15:21 /dev/mpath/ocrp1 -> ../dm-14 lrwxrwxrwx 1 root root 8 Jun 19 15:21 /dev/mpath/rman -> ../dm-13 lrwxrwxrwx 1 root root 8 Jun 19 15:21 /dev/mpath/rmanp1 -> ../dm-16 lrwxrwxrwx 1 root root 8 Jun 19 15:21 /dev/mpath/voting -> ../dm-11 lrwxrwxrwx 1 root root 8 Jun 19 15:21 /dev/mpath/votingp1 -> ../dm-17
check permission on each LUN
[root@rac-node1 ~]# ls -l /dev/mapper/ total 0 crw------- 1 root root 10, 62 Jun 19 15:21 control brwxrwxr-x 1 oracle oinstall 253, 10 Jun 19 15:21 data brwxrwxr-x 1 oracle oinstall 253, 18 Oct 18 17:37 datap1 brwxrwxr-x 1 oracle oinstall 253, 9 Jun 19 15:21 index brwxrwxr-x 1 oracle oinstall 253, 15 Oct 18 17:37 indexp1 brw-rw---- 1 root disk 253, 8 Jun 19 15:21 mpath0 brwxrwxr-x 1 oracle oinstall 253, 12 Jun 19 15:21 ocr brw-rw---- 1 oracle oinstall 253, 14 Jun 19 15:21 ocrp1 brwxrwxr-x 1 oracle oinstall 253, 13 Jun 19 15:21 rman brwxrwxr-x 1 oracle oinstall 253, 16 Jun 19 15:21 rmanp1 brw-rw---- 1 root disk 253, 5 Jun 19 15:21 VolGroup00-boot brw-rw---- 1 root disk 253, 6 Jun 19 15:21 VolGroup00-home brw-rw---- 1 root disk 253, 0 Jun 19 15:21 VolGroup00-LogVol00 brw-rw---- 1 root disk 253, 7 Jun 19 15:21 VolGroup00-LogVol01 brw-rw---- 1 root disk 253, 4 Jun 19 15:21 VolGroup00-opt brw-rw---- 1 root disk 253, 3 Jun 19 15:21 VolGroup00-tmp brw-rw---- 1 root disk 253, 2 Jun 19 15:21 VolGroup00-user brw-rw---- 1 root disk 253, 1 Jun 19 15:21 VolGroup00-var brw-rw---- 1 oracle oinstall 253, 11 Jun 19 15:21 voting brw-rw---- 1 oracle oinstall 253, 17 Jun 19 15:21 votingp1
Add permission to rc.local file which will auto apply on reboot of the server
[root@rac-node1 ~]# cat /etc/rc.local
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.
touch /var/lock/subsys/local
modprobe hangcheck-timer hangcheck_tick=30 hangcheck_margin=180
###
#OCR
chown oracle:oinstall /dev/mapper/ocr
chmod 775 /dev/mapper/ocr
chown oracle:oinstall /dev/mapper/ocrp1
chmod 775 /dev/mappper/ocrp1
raw /dev/raw/raw1 /dev/mapper/ocrp1
sleep 2
chown oracle:oinstall /dev/raw/raw1
chmod 775 /dev/raw/raw1
###
#voting
chown oracle:oinstall /dev/mapper/voting**
chmod 660 /dev/mapper/voting*
raw /dev/raw/raw2 /dev/mapper/votingp1
sleep 2
chown oracle:oinstall /dev/raw/raw2
chmod 660 /dev/raw/raw2
###data
chown oracle:oinstall /dev/mapper/data*
chmod 775 /dev/mapper/data*
chown oracle:oinstall /dev/mapper/index*
chmod 775 /dev/mapper/index*
chown oracle:oinstall /dev/mapper/rman*
chmod 775 /dev/mapper/rman*
[root@rac-node1 ~]#
Comments
Post a Comment