Check current status
Try manually mount diskgroup
ASMCMD> mount -a
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "1" is missing from group number "1"
Check ASM alert.log
SQL> alter diskgroup ACFS mount
NOTE: cache registered group ACFS number=1 incarn=0x09884abc
NOTE: cache began mount (first) of group ACFS number=1 incarn=0x09884abc
NOTE: Assigning number (1,0) to disk (/dev/oracleasm/disks/ACFS_DATA)
Mon Aug 19 10:29:24 2013
NOTE: GMON heartbeating for grp 1
GMON querying group 1 at 334 for pid 31, osid 8882
NOTE: Assigning number (1,1) to disk ()
GMON querying group 1 at 335 for pid 31, osid 8882
NOTE: cache dismounting (clean) group 1/0x09884ABC (ACFS)
NOTE: messaging CKPT to quiesce pins Unix process pid: 8882, image: oracle@grac1.example.com (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: lgwr not being msg'd to dismount
NOTE: cache dismounted group 1/0x09884ABC (ACFS)
NOTE: cache ending mount (fail) of group ACFS number=1 incarn=0x09884abc
NOTE: cache deleting context for group ACFS 1/0x09884abc
GMON dismounting group 1 at 336 for pid 31, osid 8882
NOTE: Disk in mode 0x8 marked for de-assignment
NOTE: Disk in mode 0x8 marked for de-assignment
ERROR: diskgroup ACFS was not mounted
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "1" is missing from group number "1"
ERROR: alter diskgroup ACFS mount
Mon Aug 19 10:29:27 2013
ASM Health Checker found 1 new failures
Display even dismounted diskgroups
$ asmcmd lsdg --discovery
State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
DISMOUNTED N 0 4096 0 0 0 0 0 0 N ACFS/
MOUNTED NORMAL N 512 4096 1048576 15342 9932 5114 2409 0 N DATA/
MOUNTED NORMAL N 512 4096 1048576 6141 5217 2047 1585 0 Y OCR/
--> diskgroup ACFS is still dismounted
Try to mount the ASM diskgroup with force option and check the available disks
As we have only a single disk available for NORMAL redundancy the mount fails and we need to use the force option
SQL> alter diskgroup ACFS mount force;
Diskgroup altered.
Verify the disk status after mount force command.
$ asmcmd lsdsk -p -G ACFS
Group_Num Disk_Num Incarn Mount_Stat Header_Stat Mode_Stat State Path
1 1 3915954601 MISSING UNKNOWN OFFLINE NORMAL
1 0 3915954600 CACHED MEMBER ONLINE NORMAL /dev/oracleasm/disks/ACFS_DATA
$ asmcmd lsdg
State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED NORMAL N 512 4096 1048576 1019 122 0 61 1 N ACFS/
MOUNTED NORMAL N 512 4096 1048576 15342 9932 5114 2409 0 N DATA/
MOUNTED NORMAL N 512 4096 1048576 6141 5217 2047 1585 0 Y OCR/
1585 0 Y OCR/
Try to read and verify ASM disk header ( on all instances )
# $GRID_HOME/bin/kfed read /dev/sdj1
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 0 ; 0x001: 0x00
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt: 0 ; 0x003: 0x00
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 0 ; 0x008: file=0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
7FB74EC83400 00000000 00000000 00000000 00000000 [................]
Repeat 63 times
7FB74EC83800 0000FF00 0003FBB8 000032FC 0003DA6D [.........2..m...]
7FB74EC83810 0000FEF5 00000000 00000002 00000002 [................]
7FB74EC83940 00000000 00000000 00000000 01000000 [................]
7FB74EC83950 00000000 00000000 00000000 001C001C [................]
7FB74EC83960 00000001 00000000 00000000 00000000 [................]
7FB74EC83970 00000000 00000004 00008196 00000000 [................]
7FB74EC83980 00000000 00000000 00000000 00000000 [................]
Repeat 167 times
KFED-00322: file not found; arguments: [kfbtTraverseBlock] [Invalid OSM block type] [] [0]
--> This is not a valid ASM header - let's create new ASM disk: