Category: ASM and UDEV

Test Enviromment

• OEL 6.4 with 11.2.0.4.0 RAC system

Map /dev/dm-? devices to the asm disk discovery string like /dev/asm

Display ISCSI partitions/Disks available for ASM
# dmsetup ls --tree -o device
vg_oel64-lv_swap (252:1)
 +- (8:3)
 +- (8:2)
vg_oel64-lv_root (252:0)
 +- (8:2)
grac41_disk1p1 (252:6)
 +-grac41_disk1 (252:2)
    +- (8:240)
    +- (65:32)
grac41_disk2p1 (252:7)
 +-grac41_disk2 (252:3)
    +- (65:0)
    +- (65:48)
grac41_disk3p1 (252:5)
 +-grac41_disk3 (252:4)
    +- (65:16)
    +- (65:64)

What is  Major number 252 ?
#   cat /proc/devices |  egrep 'devices|252'
..
Block devices:
252 device-mapper
-->  Driver for Device-Mapper

Availabe partitions: 
# dmsetup ls | grep p1
grac41_disk1p1    (252:5)
grac41_disk2p1    (252:7)
grac41_disk3p1    (252:6)

Map /dev/dm-? devices to /dev/asmdisk_OF-disk1 ( for example /dev/dm-5 to asmdisk_OF-disk1)
Create/Add to  12-dm-permissions.rules in /etc/udev/rules.d following entries:
ENV{DM_NAME}=="grac41_disk1p1",NAME="asmdisk_OF-disk1", OWNER:="grid", GROUP:="asmadmin", MODE:="660"
ENV{DM_NAME}=="grac41_disk2p1",NAME="asmdisk_OF-disk2", OWNER:="grid", GROUP:="asmadmin", MODE:="660"
ENV{DM_NAME}=="grac41_disk3p1",NAME="asmdisk_OF-disk3", OWNER:="grid", GROUP:="asmadmin", MODE:="660"

Change the protections for underlying sdX disks used by Device Mapper

• Note : Don’t use Name in the 99-oracle-asmdevices.rules file as we get 2 disks for a single WWID.
• Here we change group, owner and protection of the /dev/sdX devices created by ISCSI

Check WWID of ISCSI disks
#  multipath -ll |      grep dm-
grac41_disk3 (14f504e46494c45525237556f4c352d574b4e6c2d446e736b) dm-4 OPNFILER,VIRTUAL-DISK
grac41_disk2 (14f504e46494c4552786431396d6c2d6d4950492d52517178) dm-2 OPNFILER,VIRTUAL-DISK
grac41_disk1 (14f504e46494c45523264556151442d5261336d2d566e4250) dm-3 OPNFILER,VIRTUAL-DISK

Create/Add to  99-oracle-asmdevices.rules in /etc/udev/rules.d following entries
KERNEL=="sd*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="14f504e46494c45525237556f4c352d574b4e6c2d446e736b", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="14f504e46494c4552786431396d6c2d6d4950492d52517178", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="14f504e46494c45523264556151442d5261336d2d566e4250", OWNER="grid", GROUP="asmadmin", MODE="0660"

Reload UDEV ( OEL 6 stile )
# udevadm control --reload-rules
# start_udev

Verify ASM disk protection 
# ls -l /dev/asmdisk_OF*
brw-rw----. 1 grid asmadmin 252, 5 Feb 28 10:06 /dev/asmdisk_OF-disk1
brw-rw----. 1 grid asmadmin 252, 7 Feb 28 10:06 /dev/asmdisk_OF-disk2
brw-rw----. 1 grid asmadmin 252, 6 Feb 28 10:06 /dev/asmdisk_OF-disk3

Find all attached sd disks used by our multipath configuration 
# multipath -ll | grep sd
  |- 21:0:0:0 sdt 65:48 active ready running
  `- 22:0:0:0 sdu 65:64 active ready running
  |- 17:0:0:0 sdp 8:240 active ready running
  `- 18:0:0:0 sdr 65:16 active ready running
  |- 20:0:0:0 sdq 65:0  active ready running
  `- 19:0:0:0 sds 65:32 active ready running

Verify SD disk protection
# ls -l /dev/sdt /dev/sdu /dev/sdp /dev/sdr /dev/sdq /dev/sds
brw-rw----. 1 grid asmadmin  8, 240 Feb 28 10:55 /dev/sdp
brw-rw----. 1 grid asmadmin 65,   0 Feb 28 10:55 /dev/sdq
brw-rw----. 1 grid asmadmin 65,  16 Feb 28 10:55 /dev/sdr
brw-rw----. 1 grid asmadmin 65,  32 Feb 28 10:55 /dev/sds
brw-rw----. 1 grid asmadmin 65,  48 Feb 28 10:55 /dev/sdt
brw-rw----. 1 grid asmadmin 65,  64 Feb 28 10:55 /dev/sdu

- The major number changes as there are eight major numbers reserved  ( or even more ) for SCSI devices  
- Those major numbers are 8, 65, 66, 67, 68, 79, 70, and 71

Verify your ASM disks on all nodes before you use asmca to create an OPENFILER diskgroup 

Reference:

• http://orainternals.wordpress.com/2012/08/29/do-you-need-asmlib
• http://fritshoogland.wordpress.com/2012/07/23/using-udev-on-rhel-6-ol-6-to-change-disk-permissions-for-asm/

 

Tested Environment

• OEL 6.4 with RAC 11.2.0.4

Key Facts

• Don’t use /dev/sdX or /dev/dm-XX as device name as this device names can change after a reboot
• Use the WWID for identifying devices
• The World Wide Identifier (WWID) can be used to reliably identifying disk devices.
• WWID is a persistent, system-independent ID that the SCSI Standard requires from all SCSI devices.

Display all available WWID on your system

Shell script to display WWID on your system
#!/bin/bash
#
#Usage:  As root user run : ./check_wwid.sh 
#
for FILE in `find /dev -name "sd*" | sort`
   do
     WWID=`scsi_id --whitelisted --replace-whitespace --device=$FILE `
     echo $FILE " WWID:  "  $WWID
   done

$ sudo ./check_wwid.sh
..
/dev/sdb   WWID:   1ATA_VBOX_HARDDISK_VB81c4f844-94203a15
/dev/sdb1  WWID:   1ATA_VBOX_HARDDISK_VB81c4f844-94203a15
/dev/sdc   WWID:   1ATA_VBOX_HARDDISK_VB2ed83c09-e66dbcfb
/dev/sdc1  WWID:   1ATA_VBOX_HARDDISK_VB2ed83c09-e66dbcfb
..
--> Of course WWIDs for partitions and for the complete disk are identical 

 

Create UDEV for a WWID identified SCSI device

• Note in this sample we only change owner, group and protection for a disk device /dev/sdX

# cat 99-oracle-asmdevices.rules
KERNEL=="sd*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="14f504e46494c45523264556151442d5261336d2d566e4250", OWNER="grid", GROUP="asmadmin", MODE="0660"

# ls -ls /dev/sdr
0 brw-rw----. 1 grid asmadmin 65, 16 Mar 19 14:42 /dev/sdr

Reload and Restart the udev rules 
# udevadm control --reload-rules
# start_udev
Starting udev:                                             [  OK  ]
Verify disk protections:
# ls -ltr /dev/asm*
brw-rw----. 1 grid asmadmin   8,  17 Mar 15 10:03 /dev/asmdisk1_udev_sdb1
brw-rw----. 1 grid asmadmin   8,  33 Mar 15 10:03 /dev/asmdisk2_udev_sdc1

Redistribute   99-oracle-asmdevices.rules file to all cluster nodes and restat udev echo system on these nodes
# scp 99-oracle-asmdevices.rules grac42:/etc/udev/rules.d
# scp 99-oracle-asmdevices.rules grac43:/etc/udev/rules.d

On each RAC node
# udevadm control --reload-rules
# start_udev
# ls -ltr /dev/asm*
brw-rw----. 1 grid asmadmin   8,  17 Mar 15 10:03 /dev/asmdisk1_udev_sdb1
brw-rw----. 1 grid asmadmin   8,  33 Mar 15 10:03 /dev/asmdisk2_udev_sdc1

UDev rules for a disk partition using /sbin/scsi_id ( prefered mothode as we can change the NAME )

• Note this sample maps:   /dev/sdb1 to /dev/asmdisk1_udev_sdb1, …
  

# cat 99-oracle-asmdevices.rules
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB81c4f844-94203a15", NAME=
"asmdisk1_udev_sdb1", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB2ed83c09-e66dbcfb", NAME=
"asmdisk1_udev_sdc1", OWNER="grid", GROUP="asmadmin", MODE="0660"

# ls -l /dev/asmdisk1_udev_sdb1 /dev/asmdisk1_udev_sdc1
brw-rw----. 1 grid asmadmin 8, 17 Mar 19 13:53 /dev/asmdisk1_udev_sdb1
brw-rw----. 1 grid asmadmin 8, 33 Mar 19 13:53 /dev/asmdisk1_udev_sdc1

 

Reference:   

•  How To Setup Partitioned Linux Block Devices Using UDEV (Non-ASMLIB) And Assign Them To ASM? (Doc ID 1528148.1)
• Do you need asmlib?