Table of Contents
Test Enviromment
- OEL 6.4 with 11.2.0.4.0 RAC system
Map /dev/dm-? devices to the asm disk discovery string like /dev/asm
Display ISCSI partitions/Disks available for ASM # dmsetup ls --tree -o device vg_oel64-lv_swap (252:1) +- (8:3) +- (8:2) vg_oel64-lv_root (252:0) +- (8:2) grac41_disk1p1 (252:6) +-grac41_disk1 (252:2) +- (8:240) +- (65:32) grac41_disk2p1 (252:7) +-grac41_disk2 (252:3) +- (65:0) +- (65:48) grac41_disk3p1 (252:5) +-grac41_disk3 (252:4) +- (65:16) +- (65:64) What is Major number 252 ? # cat /proc/devices | egrep 'devices|252' .. Block devices: 252 device-mapper --> Driver for Device-Mapper Availabe partitions: # dmsetup ls | grep p1 grac41_disk1p1 (252:5) grac41_disk2p1 (252:7) grac41_disk3p1 (252:6) Map /dev/dm-? devices to /dev/asmdisk_OF-disk1 ( for example /dev/dm-5 to asmdisk_OF-disk1) Create/Add to 12-dm-permissions.rules in /etc/udev/rules.d following entries: ENV{DM_NAME}=="grac41_disk1p1",NAME="asmdisk_OF-disk1", OWNER:="grid", GROUP:="asmadmin", MODE:="660" ENV{DM_NAME}=="grac41_disk2p1",NAME="asmdisk_OF-disk2", OWNER:="grid", GROUP:="asmadmin", MODE:="660" ENV{DM_NAME}=="grac41_disk3p1",NAME="asmdisk_OF-disk3", OWNER:="grid", GROUP:="asmadmin", MODE:="660"
Change the protections for underlying sdX disks used by Device Mapper
- Note : Don’t use Name in the 99-oracle-asmdevices.rules file as we get 2 disks for a single WWID.
- Here we change group, owner and protection of the /dev/sdX devices created by ISCSI
Check WWID of ISCSI disks # multipath -ll | grep dm- grac41_disk3 (14f504e46494c45525237556f4c352d574b4e6c2d446e736b) dm-4 OPNFILER,VIRTUAL-DISK grac41_disk2 (14f504e46494c4552786431396d6c2d6d4950492d52517178) dm-2 OPNFILER,VIRTUAL-DISK grac41_disk1 (14f504e46494c45523264556151442d5261336d2d566e4250) dm-3 OPNFILER,VIRTUAL-DISK Create/Add to 99-oracle-asmdevices.rules in /etc/udev/rules.d following entries KERNEL=="sd*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="14f504e46494c45525237556f4c352d574b4e6c2d446e736b", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="14f504e46494c4552786431396d6c2d6d4950492d52517178", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="14f504e46494c45523264556151442d5261336d2d566e4250", OWNER="grid", GROUP="asmadmin", MODE="0660" Reload UDEV ( OEL 6 stile ) # udevadm control --reload-rules # start_udev Verify ASM disk protection # ls -l /dev/asmdisk_OF* brw-rw----. 1 grid asmadmin 252, 5 Feb 28 10:06 /dev/asmdisk_OF-disk1 brw-rw----. 1 grid asmadmin 252, 7 Feb 28 10:06 /dev/asmdisk_OF-disk2 brw-rw----. 1 grid asmadmin 252, 6 Feb 28 10:06 /dev/asmdisk_OF-disk3 Find all attached sd disks used by our multipath configuration # multipath -ll | grep sd |- 21:0:0:0 sdt 65:48 active ready running `- 22:0:0:0 sdu 65:64 active ready running |- 17:0:0:0 sdp 8:240 active ready running `- 18:0:0:0 sdr 65:16 active ready running |- 20:0:0:0 sdq 65:0 active ready running `- 19:0:0:0 sds 65:32 active ready running Verify SD disk protection # ls -l /dev/sdt /dev/sdu /dev/sdp /dev/sdr /dev/sdq /dev/sds brw-rw----. 1 grid asmadmin 8, 240 Feb 28 10:55 /dev/sdp brw-rw----. 1 grid asmadmin 65, 0 Feb 28 10:55 /dev/sdq brw-rw----. 1 grid asmadmin 65, 16 Feb 28 10:55 /dev/sdr brw-rw----. 1 grid asmadmin 65, 32 Feb 28 10:55 /dev/sds brw-rw----. 1 grid asmadmin 65, 48 Feb 28 10:55 /dev/sdt brw-rw----. 1 grid asmadmin 65, 64 Feb 28 10:55 /dev/sdu - The major number changes as there are eight major numbers reserved ( or even more ) for SCSI devices - Those major numbers are 8, 65, 66, 67, 68, 79, 70, and 71 Verify your ASM disks on all nodes before you use asmca to create an OPENFILER diskgroup