root.sh fails with CLSRSC-366 after adding a node back to an ASM cluster

Before rerunning of root.sh script, we must deconfigure it.

[root@gract3 ~]#  $GRID_HOME/crs/install/rootcrs.pl -deconfig
Using configuration parameter file: /u01/app/12102/grid/crs/install/crsconfig_params2014/08/16 07:31:41 
CLSRSC-38: Oracle Clusterware stack is not active on this node2014/08/16 07:31:41 CLSRSC-311: 
Failed to verify resourcesDied at /u01/app/12102/grid/crs/install/crsdeconfig.pm line 333.

Use -force switch as CRS is not up 
[root@gract3 ~]#  $GRID_HOME/crs/install/rootcrs.pl -deconfig -force

Configuration

  • Upgrade the flex cluster and db to 12.1.0.2 smoothly.
  • Delete one hub node(gract3) from the clusterware.
  • Add the node back, but it was failed running root.sh with  CLSRSC-366

Review traces from  failed running of  root.sh  – CLSRSC-366 error

Locate any files with  CLSRSC-366 error
[grid@gract3 grid]$ find  $GRID_HOME/cfgtoollogs/crsconfig -exec grep -l  CLSRSC-366 {} \;
/u01/app/12102/grid/cfgtoollogs/crsconfig/rootcrs_gract3_2014-08-16_07-22-08AM.log

Find error details 
[grid@gract3 grid]$  find  $GRID_HOME/cfgtoollogs/crsconfig -exec grep CLSRSC-366 {} \;
>  CLSRSC-366: Failed to import credentials for ASM 
>  CLSRSC-366: Failed to import credentials for ASM 
2014-08-16 07:24:16: CLSRSC-366: Failed to import credentials for ASM

Check trace file more in detail:
2014-08-16 07:24:15: s_run_as_user2: Running /bin/su grid -c ' echo CLSRSC_START; 
   /u01/app/12102/grid/bin/kfod op=credimport wrap=/u01/app/12102/grid/gpnp/seed/asm/credentials.xml olr=TRUE force=TRUE '
...
2014-08-16 07:24:16: kfod op=credimport rc: 139
2014-08-16 07:24:16: Failed to enable flex ASM on local node, error: bash: line 1: 27788 Segmentation fault      (core dumped) 
                    /u01/app/12102/grid/bin/kfod op=credimport wrap=/u01/app/12102/grid/gpnp/seed/asm/credentials.xml olr=TRUE force=TRUE

2014-08-16 07:24:16: Executing cmd: /u01/app/12102/grid/bin/clsecho -p has -f clsrsc -m 366
2014-08-16 07:24:16: Command output:
>  CLSRSC-366: Failed to import credentials for ASM
--> kfod dump core :

Find related traces:
[root@gract3 log]# find $GRID_HOME/log  -type f -printf "%CY-%Cm-%Cd %CH:%CM:%CS  %h/%f\n" | sort -n  | more
2014-08-16 07:24:15.6642347810  /u01/app/12102/grid/log/gract3/client/GIPC_27788.log
2014-08-16 07:24:15.7582377850  /u01/app/12102/grid/log/diag/clients/user_grid/host_1786982425_82/lck/AM_1762783_4031814035.lck
2014-08-16 07:24:15.7592378170  /u01/app/12102/grid/log/diag/clients/user_grid/host_1786982425_82/metadata/INCIDENT.ams
2014-08-16 07:24:15.7652380080  /u01/app/12102/grid/log/diag/clients/user_grid/host_1786982425_82/trace/ora_27788_140265833375296.trc
2014-08-16 07:24:15.7652380080  /u01/app/12102/grid/log/diag/clients/user_grid/host_1786982425_82/trace/ora_27788_140265833375296.trm
2014-08-16 07:24:15.7662380400  /u01/app/12102/grid/log/diag/clients/user_grid/host_1786982425_82/alert/log.xml
2014-08-16 07:24:15.7662380400  /u01/app/12102/grid/log/diag/clients/user_grid/host_1786982425_82/sweep/stg1_9_inc.swp
2014-08-16 07:24:15.7662380400  /u01/app/12102/grid/log/diag/clients/user_grid/host_1786982425_82/trace/sqlnet.log
2014-08-16 07:24:15.7712381990  /u01/app/12102/grid/log/diag/clients/user_grid/host_1786982425_82/incident/incdir_9/ora_27788_140265833375296_i9.trc
2014-08-16 07:24:15.7712381990  /u01/app/12102/grid/log/diag/clients/user_grid/host_1786982425_82/incident/incdir_9/ora_27788_140265833375296_i9.trm
2014-08-16 07:24:15.7712381990  /u01/app/12102/grid/log/diag/clients/user_grid/host_1786982425_82/sweep/stg1_9_inc2.swp

root@gract3 log]# cat  /u01/app/12102/grid/log/diag/clients/user_grid/host_1786982425_82/trace/ora_27788_140265833375296.trc
Trace file /u01/app/12102/grid/log/diag/clients/user_grid/host_1786982425_82/trace/ora_27788_140265833375296.trc
DDE: Flood control is not active
Incident 9 created, dump file: /u01/app/12102/grid/log/diag/clients/user_grid/host_1786982425_82/incident/incdir_9/ora_27788_140265833375296_i9.trc
oci-24550 [11] [[si_signo=11] [si_errno=0] [si_code=1] [si_int=15332088] [si_ptr=0xe9f2f8] [si_addr=(nil)]] [] [] [] [] [] [] [] [] [] []

[root@gract3 log]#  cat  /u01/app/12102/grid/log/diag/clients/user_grid/host_1786982425_82/incident/incdir_9/ora_27788_140265833375296_i9.trc
Dump file /u01/app/12102/grid/log/diag/clients/user_grid/host_1786982425_82/incident/incdir_9/ora_27788_140265833375296_i9.trc
[TOC00000]
Jump to table of contents
Dump continued from file: /u01/app/12102/grid/log/diag/clients/user_grid/host_1786982425_82/trace/ora_27788_140265833375296.trc
[TOC00001]
oci-24550 [11] [[si_signo=11] [si_errno=0] [si_code=1] [si_int=15332088] [si_ptr=0xe9f2f8] [si_addr=(nil)]] [] [] [] [] [] [] [] [] [] []
[TOC00001-END]
[TOC00002]
========= Dump for incident 9 (oci 24550 [11]) ========
Tracing is in restricted mode!
[TOC00003]
----- Short Call Stack Trace -----
dbgexPhaseII()+878<-dbgexProcessError()+2539<-dbgeExecuteForError()+110<-dbgePostErrorDirect()+2230<-kpeDbgSignalHandler()
    +302<-skgesig_sigactionHandler()+229<-__sighandler()<-kgfz_importCluster()+2685<-kfodCredImport()+173<-kfodExecute()+1924
    <-kfod_main()+899<-lpmcall()+680<-lpmpmai()+420<-main()+458<-__libc_start_main()+253[TOC00003-END]
-->kfod executable crashes with following stack  
    kgfz_importCluster() 
    kfodCredImport() 
    kfodExecute()

Bug Reference

  •  Bug 18406774 : CLSXWRAP_WRAPSECTGETPROP() RETURNS SUCCESS EVEN WHEN PROPERTY IS MISSING
  •  Bug 19234452 : LNX64-12C-UD:FLEX CLUSTER ADD NODE, FAILED TO IMPORT CREDENTIALS FOR ASM
  
 Stack trace from  Bug 19234452  
  (gdb) where
   #0  0x0000003e4d278900 in strlen () from /lib64/libc.so.6
   #1  0x00002aaaafffe3ce in kgfz_importCluster (ctxt=0x6f3d90, 
      wfile=0x639814 "/u01/app/12.1.0/grid_1/gpnp/seed/asm/credentials.xml", 
      flags=1) at kgfz.c:3316
   #2  0x0000000000407eed in kfodCredImport
   #3  0x0000000000408c84 in kfodExecute 
   #4  0x0000000000406ea3 in kfod_main

Bug 18406774 - Basebug of BUG 19234452  - is fixed by upcomming October Patch Bundle 
-->  Will test asap !

Check ASM disk status and fix problems

Check ASM disk status

Check ASM disk status 

SQL> select    g.name disk_grp, o.group_number, operation , est_minutes from gv$asm_operation o,    gv$asm_diskgroup g
         where g.group_number = o.group_number and o.inst_id = g.inst_id;

no rows selected --> No rebalance operation in progress 

SQL> select d.inst_id, dg.name dg_name,  dg.state dg_state,    dg.type,d.name, d.DISK_NUMBER dsk_no, d.MOUNT_STATUS, d.HEADER_STATUS, d.MODE_STATUS,
             d.STATE, d. PATH, d.FAILGROUP  FROM GV$ASM_DISK d,  gv$asm_diskgroup dg
          where dg.group_number(+)=d.group_number and d.inst_id = dg.inst_id order by d.inst_id, d.group_number;
   INST_ID DG_NAME    DG_STATE     TYPE    NAME              DSK_NO MOUNT_S HEADER_STATU MODE_ST STATE    PATH                     FAILGROUP
---------- ---------- ---------- ------ -------------------- ------- ------- ------------ ------- -------- ---------------------------------------- ---------------
     1 OCR          MOUNTED     NORMAL OCR_0000           0 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk1_udev_sdf1            OCR_0000
     1 OCR          MOUNTED     NORMAL OCR_0002           2 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk1_udev_sdh1            OCR_0002
     1 OCR          MOUNTED     NORMAL OCR_0001           1 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk1_udev_sdg1            OCR_0001
     1 FRA          MOUNTED     EXTERN FRA_0002           2 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk_fra2                FRA_0002
     1 FRA          MOUNTED     EXTERN FRA_0001           1 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk_fra1                FRA_0001
     1 DATA       MOUNTED     NORMAL DATA_0001           1 CACHED  CANDIDATE      ONLINE  NORMAL   ORCL:DATA2                    DATA_0001
     1 DATA       MOUNTED     NORMAL DATA_0002           2 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk1_udev_sdd1            DATA_0002
     1 DATA       MOUNTED     NORMAL DATA_0003           3 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk1_udev_sde1            DATA_0003
     1 SSD          MOUNTED     NORMAL SSD_0002           2 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk8_ssd3                SSD_0002
     1 SSD          MOUNTED     NORMAL SSD_0000           0 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk8_ssd1                SSD_0000
     1 SSD          MOUNTED     NORMAL SSD_0001           1 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk8_ssd2                SSD_0001
     2 OCR          MOUNTED     NORMAL OCR_0002           2 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk1_udev_sdh1            OCR_0002
     2 OCR          MOUNTED     NORMAL OCR_0000           0 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk1_udev_sdf1            OCR_0000
     2 OCR          MOUNTED     NORMAL OCR_0001           1 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk1_udev_sdg1            OCR_0001
     2 FRA          MOUNTED     EXTERN FRA_0002           2 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk_fra2                FRA_0002
     2 FRA          MOUNTED     EXTERN FRA_0001           1 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk_fra1                FRA_0001
     2 DATA       MOUNTED     NORMAL DATA_0001           1 CACHED  CANDIDATE      ONLINE  NORMAL   /dev/asmdisk1_udev_sdc1            DATA_0001
     2 DATA       MOUNTED     NORMAL DATA_0003           3 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk1_udev_sde1            DATA_0003
     2 DATA       MOUNTED     NORMAL DATA_0002           2 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk1_udev_sdd1            DATA_0002
     2 SSD          MOUNTED     NORMAL SSD_0002           2 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk8_ssd3                SSD_0002
     2 SSD          MOUNTED     NORMAL SSD_0001           1 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk8_ssd2                SSD_0001
     2 SSD          MOUNTED     NORMAL SSD_0000           0 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk8_ssd1                SSD_0000
     2 ASMLIB_DG  MOUNTED     NORMAL ASMLIB_DG_0002           2 CACHED  CANDIDATE      ONLINE  NORMAL   /dev/oracleasm/disks/ASMLIB_DISK3        ASMLIB_DG_0002
     2 ASMLIB_DG  MOUNTED     NORMAL ASMLIB_DG_0001           1 CACHED  CANDIDATE      ONLINE  NORMAL   /dev/oracleasm/disks/ASMLIB_DISK2        ASMLIB_DG_0001
     2 ASMLIB_DG  MOUNTED     NORMAL ASMLIB_DG_0000           0 CACHED  CANDIDATE      ONLINE  NORMAL   /dev/oracleasm/disks/ASMLIB_DISK1        ASMLIB_DG_0000
     3 OCR          MOUNTED     NORMAL OCR_0001           1 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk1_udev_sdg1            OCR_0001
     3 OCR          MOUNTED     NORMAL OCR_0002           2 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk1_udev_sdh1            OCR_0002
     3 OCR          MOUNTED     NORMAL OCR_0000           0 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk1_udev_sdf1            OCR_0000
     3 FRA          MOUNTED     EXTERN FRA_0001           1 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk_fra1                FRA_0001
     3 FRA          MOUNTED     EXTERN FRA_0002           2 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk_fra2                FRA_0002
     3 DATA       MOUNTED     NORMAL DATA_0003           3 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk1_udev_sde1            DATA_0003
     3 DATA       MOUNTED     NORMAL DATA_0001           1 CACHED  CANDIDATE      ONLINE  NORMAL   /dev/asmdisk1_udev_sdc1            DATA_0001
     3 DATA       MOUNTED     NORMAL DATA_0002           2 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk1_udev_sdd1            DATA_0002
     3 SSD          MOUNTED     NORMAL SSD_0002           2 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk8_ssd3                SSD_0002
     3 SSD          MOUNTED     NORMAL SSD_0000           0 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk8_ssd1                SSD_0000
     3 SSD          MOUNTED     NORMAL SSD_0001           1 CACHED  MEMBER      ONLINE  NORMAL   /dev/asmdisk8_ssd2                SSD_0001
--> Here we have some problems 
    1. ASMLIB_DG is only mounted on Node 2 - disks status CANDIDATE 
    2. Disk DATA_0001 is handled by ASMLIB at Node 1 but is using UDEV at Node2 , Node 3

Generic ASM Overview

Key Facts

  • The RDBMS instance ( ASMB process )  connects to the ASM instance ( UFG process ) via BEQ as SYSDBA ( CSS provides this info )
  • ASM tracks all the file an RDBMS instance has open
  • During RDBMS startup ASM sends an extend map of the first 60 AUs to the RDBMS ( note SGA is not  initialized yet )
  • RDBMS ( LGWR, DBWR ) performs I/O directly the the ASM disks
  • Open/close and create/delete operations  of ASM files are maintained by ASM
  • ASM doesn’t balance I/O based on I/O statistics and if disk size is different the space allocation is different 
  • An read error in a RDBMs inacte never causes a disk to go OFFLINE
  • If RDBMs instance gets I/O error reading from primary extend it reads from mirrors extend and fixes the I/O problem by writing the fixed data to the primary extend
  • ASM uses bad block replacement
  • ASM takes the disk offline on write errors
  • ASM provides better data integrity during DISK resilvering  after incomplete/lost writes

Overview ASM and The Oracle Grid Infrastructure

Oracle Clusterware, ASM, Oracle ACFS, Kernel Services Driver (OKS), and Oracle ADVM software components. 
These components are installed into the Grid Infrastructure home using the Oracle Universal Installation (OUI) tool. 
Following a Grid Infrastructure installation on Linux and UNIX platforms, a root script is executed that copies 
the Oracle ACFS, Oracle ADVM, and OKS drivers and command-line tools into operating system-specific locations. 
On Linux systems, these three drivers are copied into the /lib/modules/Linux_kernel_version/kernel/drivers 
directory and the command-line tools are copied into the Linux /sbin directory.

On Windows, the three drivers are copied into the %systemroot%\system32\drivers folder and the command-line tools 
are copied into the %systemroot%\Program Files\Oracle folder. For single-instance homes, Oracle ACFS,
Kernel Services Driver (OKS), and Oracle ADVM drivers are installed and initially loaded into the operating 
system kernel memory during execution of the Oracle Grid Infrastructure root script. However, if the server 
is restarted, the three drivers must be manually loaded into the operating system kernel memory by executing 
the driver load command acfsload as shown below. To execute the command, the user must have root privileges. 
$ORACLE_HOME/bin/acfsload start –s Oracle Restart is part of the single server Grid Infrastructure installation 
 but is covered in a different eStudy.  

Grid Infrastructure Home The Grid Infrastructure home directory must be separate from any database home directory. 
Even though ASM and Clusterware are separate products they are installed in the same home directory. By separating 
the Grid Infrastructure home directory from the database home directory, it is possible to have job role separation
for the ASM administrator from database administrators.  This change facilitates patching, but has the side effect 
that clusterware and ASM must be patched together.  If ASM is already installed, a new ORACLE_HOME directory must 
be created for the 11.2 software. ASM and Clusterware software will be installed into the new directory.

ASM Privileges

ASM Privileges There are three defined privileges for ASM: SYSASM, SYSDBA, and SYSOPER.  The SYSASM privilege provides 
full administrative privileges for the ASM instance. Users with the SYSASM privilege can start and stop the ASM 
instance and manage ASM disks and diskgroups.  In the ASM instance the SYS user has the SYSASM privilege.  
The SYSDBA privilege on the ASM instance grants access to data stored on ASM. To use SQL*Plus or ASMCMD commands 
to manage ASM components associated with the database connect AS SYSDBA to the database instance rather than the 
ASM instance. Users connected AS SYSDBA can: Create and delete files, aliases, directories, and templates, examine 
various ASM instance views, operate on files that were created by this user ....        

During installation, the ASMSNMP user with SYSDBA privileges is created for monitoring the Oracle ASM instance.  
Users granted the SYSOPER privilege on the ASM instance, are allowed to startup, shutdown, mount, dismount, and 
check disk groups (but not repair). Other operations, such as CREATE DISKGROUP, and ADD/DROP/RESIZE DISK, 
requires SYSASM privilege and are not allowed with SYSOPER privilege. Note: the SYSOPER privilege does not 
allow access to system views, e.g. v$asm_*.

ASM Privileged OS Groups Prior to Oracle Database 11g Release 2 there were only three privileged groups. 
The oraInventory group is defined as the group that owns the oracle software installation. This group is often 
called oinstall. The OSDBA group for the database, often called dba, owns the database files, and has all privileges 
on the database. The OSOPER group for the database, called oper, has limited privileges to start, stop, and perform 
limited management operations.

Users connected with the SYSDBA privilege cannot create or resize a disk group when connected with the SYSDBA privilege.
During installation, the ASMSNMP user with SYSDBA privileges is created for monitoring the Oracle ASM instance.  
Users granted the SYSOPER privilege on the ASM instance, are allowed to startup, shutdown, mount, dismount, and check 
disk groups (but not repair). 
Other operations, such as CREATE DISKGROUP, and ADD/DROP/RESIZE DISK, requires SYSASM privilege and are not allowed 
with SYSOPER privilege. Note: the SYSOPER privilege does not allow access to system views, e.g. v$asm_*. 

On short : SYSASM = SYSDBA + SYSOPER priv             

Group                 For         OS Group   Priv       New on 11.2?
oraInventrory Group   DB and ASM  oinstall              no
OSDBA                 DB          dba        SYSDBA     no
OSOPER                DB          oper       SYSOPER    no
OSASM                 ASM         asmadmin   SYSASM     yes
OSDBA                 ASM         asmdba     SYSDBA     yes
OSOPER                ASM         asmoper    SYSOPER    yes

Group Sepration
User       Privs
grid       OSASM/asmadmin/SYSASM
           OSDBA/asmdba/SYSDBA    
           OSOPER/asmoper/SYSOPER 
disk owner : grid/asmadmin         
oracle1    asmdba/SYSDBA
           dba1  / SYSDBA for db1
           oper1 / SYSOPERfor db1
oracle2    asmdba/SYSDBA
           dba2  / SYSDBA for db2
           oper2 / SYSOPERfor db2         
                           #
SYSASM priv          
  - The owner of the Grid Infrastructure software installation must be a member of the OSASM group, 
    and automatically receives the SYSASM privilege. 
  - The SYSASM privilege is required to create, drop diskgroups,and to add and delete disks from disk groups.
  - The SYSASM privilege is not granted to database owners.
  - Database software owners must have the SYSDBA privilege in the ASM instance.
  - The ASMSNMP user is created during install with the SYSDBA privilege

ASM and ACLs

 - Access Control Lists are not enabled by default.
 - ACLs do prevent a database owner from inadvertently deleting an ASM file in the same disk group belonging 
   to another database when properly configured.
 - ASM Compatibility must be set to 11.2 or higher
 - Job role separation must be implemented for ACls. The ACL uses the OS id and groups to control file access privileges
 - ACLs can be managed with SQL or ASMCMD commands, or through Enterprise Manager

ACFS

  - ACFS is a general purpose file system that can be used to hold any type of file. 
    But Grid binaries required to access ACFS cannot be stored in an ACFS file system.
  - ACFS does not support Direct IO.  For best performance, database files should be placed directly in ASM.
  - ACFS can be used in clustered and non-clustered env
  - With ACFS you can search a snapshot for older version of a file
  - Copy-On-Write saves changed blocks to the ACFS snapshot
  - You can backup and restore ACFS snapshots

ASM and 4K Sector Disk

  - You must use 4 KB log files on 4 KB native mode disks
  - Intellgent Data Properties ( Hot / cold regions ) will take place after next rebalance operation
  - Changing  Intellgent Data Properties will not trigger a rebalance operation
  - Only 1 voting file can be stored per fail group in the same disk group :1

ASM versons dependencies 11.2 ( COMPATIBLE.ASM  COMPATIBLE.RDBMS COMPATIBLE.ADVM )

Attribute parameter 
Compatibility
  COMPATIBLE.ASM     determines the minimum software version for an ASM instance that can mount a disk group
                     This setting also affects the format of the data structures for the Oracle ASM metadata 
                     on the disk.
  COMPATIBLE.RDBMS   determines the minimum COMPATIBLE parameter setting for any database instance that is 
                     allowed to open this DG
  COMPATIBLE.ASM     will always be greater than COMPATIBLE.RDBMS 

COMPATIBLE.ASM  COMPATIBLE.RDBMS     COMPATIBLE.ADVM     ASM Instance Version     COMPATIBLE Setting 
                                                                                  for RDBMS Instance
10.1            10.1                    n/a                     >= 10.1                 >= 10.1 
11.1            10.1                    n/a                     >= 11.1                 >= 10.1
11.2            11.1                    11.2                    >= 11.2                 >= 11.1 
11.2            11.2                    11.2                    >= 11.2                 >= 11.2

Features enabled by disk group compatibility attribute settings
Disk Group Features Enabled                                     COMPATIBLE.ASM     COMPATIBLE.RDBMS COMPATIBLE.ADVM
Support for larger AU sizes (32 or 64 MB)                        >= 11.1        >= 11.1         n/a 
Attributes are displayed in the V$ASM_ATTRIBUTE view             >= 11.1        n/a              n/a 
Fast mirror resync                                               >= 11.1        >= 11.1          n/a 
Variable size extents                                            >= 11.1        >= 11.1          n/a 
Exadata storage                                                  >= 11.1.0.7    >= 11.1.0.7      n/a 
Intelligent Data Placement                                       >= 11.2        >= 11.2          n/a 
OCR and voting files in a disk group                             >= 11.2        n/a              n/a 
Sector size set to nondefault value                              >= 11.2        >= 11.2          n/a 
Oracle ASM SPFILE in a disk group i                              >= 11.2        n/a              n/a 
Oracle ASM File Access Control                                   >= 11.2        >= 11.2          n/a 
Volumes in disk groups                                           >= 11.2        n/a              >= 11.2 
ASM_POWER_LIMIT value up to 1024                                 >= 11.2.0.2    n/a              n/a 
Encryption, replication, security, tagging on Linux systems      >= 11.2.0.2    n/a              >= 11.2.0.2 
Encryption, replication, security, tagging on Windows systems    >= 11.2.0.3    n/a              >= 11.2.0.3 
Read-write snapshots                                             >= 11.2.0.3    n/a              >= 11.2.0.3

ASM tools used by Support : KFOD, KFED, AMDU (Doc ID 1485597.1)

KFOD – Kernel Files OSM Disk

KFOD utility is used to simulate the disk discovery from the operating system level and it can be found in $GRID_HOME/bin directory (or ASM_HOME/bin in older releases).

Note:
1) KFOD is used at installation time as well (by OUI,DBCA or ASMCA) in order to perform disk discovery.
2) In case of a failed installation (e.g no $GRID_HOME/bin existing yet) KFOD can be found under your stage folder: <stage_folder>/grid/stage/ext/bin/
In such cases you might need to set LD_LIBRARY_PATH to <stage_folder>/grid/stage/ext/lib.

Using kfod 
[grid@grac41 ]$ kfod status=TRUE asm_diskstring='/dev/asm*' disk=ALL 
--------------------------------------------------------------------------------
 Disk          Size Header    Path                                     User     Group   
================================================================================
   1:      10236 Mb MEMBER    /dev/asm_data_10G_disk0                  grid     asmadmin
   2:      10236 Mb MEMBER    /dev/asm_data_10G_disk1                  grid     asmadmin
   3:      10236 Mb MEMBER    /dev/asm_data_10G_disk2                  grid     asmadmin
   4:      10236 Mb MEMBER    /dev/asm_data_10G_disk3                  grid     asmadmin
   5:       2047 Mb MEMBER    /dev/asm_ocr_11204_2G_disk1              grid     asmadmin
   6:       2047 Mb MEMBER    /dev/asm_ocr_11204_2G_disk2              grid     asmadmin
   7:       2047 Mb MEMBER    /dev/asm_ocr_11204_2G_disk3              grid     asmadmin
   8:       1023 Mb MEMBER    /dev/asm_test_1G_disk1                   grid     asmadmin
   9:       1023 Mb MEMBER    /dev/asm_test_1G_disk2                   grid     asmadmin
  10:      20473 Mb MEMBER    /dev/asmdisk_fra1                        grid     asmadmin
  11:      20479 Mb MEMBER    /dev/asmdisk_fra2                        grid     asmadmin
--------------------------------------------------------------------------------
ORACLE_SID ORACLE_HOME                                                          
================================================================================
     +ASM1 /u01/app/11204/grid                                                  
     +ASM3 /u01/app/11204/grid                                                  
     +ASM2 /u01/app/11204/grid

KFED – Kernel Files metadata EDitor

KFED is a useful tool which allows to analyze ASM disk header information when ASM diskgroups can not be mounted.

NOTE:
KFED executable comes with installation from 11.1 onwards; for the older version you must build it.
But it is not included  the source of the software, therefore it is not available until GI installation completes.  
If you need to use it before GI being installed, please refer

root@grac41 Desktop]# kfed read  /dev/asm_test_1G_disk1 | egrep 'name|size|type'
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD   <-- ASM disk header
kfdhdb.dskname:               TEST_0000 ; 0x028: length=9          <-- ASM disk name
kfdhdb.grpname:                    TEST ; 0x048: length=4          <-- ASM DG name
kfdhdb.fgname:                TEST_0000 ; 0x068: length=9          <-- ASM Failgroup
kfdhdb.capname:                         ; 0x088: length=0
kfdhdb.secsize:                     512 ; 0x0b8: 0x0200            <-- Disk sector size   
kfdhdb.blksize:                    4096 ; 0x0ba: 0x1000            <-- ASM block size
kfdhdb.ausize:                  1048576 ; 0x0bc: 0x00100000        <-- AU size : 1 Mbyte 
kfdhdb.dsksize:                    1023 ; 0x0c4: 0x000003ff        <-- ASM disk size : 1 GByte

 

AMDU – ASM Metadata Dump Utility

AMDU is an utility to extract all the available metadata from one or more ASM disks and/or generate formatted printouts of individual blocks.
The utility does not require that any diskgroup to be mounted.

Key features 
 - Dumps metadata for ASM disks
 - Extract the content of ASM files even DG isn't mounted

Oracle Support may ask you to provide AMDU output, especially when facing internal errors related to the ASM metadata

Locate all DG and dump FRA2 DG 
$  asmcmd lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  NORMAL  N         512   4096  1048576     40944    18531            10236            4147              0             N  DATA/
MOUNTED  EXTERN  N         512   4096  1048576     40952    31315                0           31315              0             N  FRA2/
MOUNTED  NORMAL  N         512   4096  4194304      6132     4960             2044            1458              0             Y  OCR/
MOUNTED  NORMAL  N         512   4096  1048576      2046     1746                0             873              0             N  TEST/
[grid@grac41 ]$ amdu -diskstring '/dev/asm*' -dump 'FRA2'
    amdu_2014_07_15_11_20_42/

$  cd amdu_2014_07_15_11_20_42/
[grid@grac41 ]$ ls -l
total 144432
-rw-r--r--. 1 grid oinstall 147865600 Jul 15 11:20 FRA2_0001.img
-rw-r--r--. 1 grid oinstall     11920 Jul 15 11:20 FRA2.map
-rw-r--r--. 1 grid oinstall     12468 Jul 15 11:20 report.txt
--> report.txt
************************** SCANNING DISKGROUP FRA2 ***************************
            Creation Time: 2014/07/10 17:16:26.860000
         Disks Discovered: 2
               Redundancy: 1
                  AU Size: 1048576 bytes
      Metadata Block Size: 4096 bytes
     Physical Sector Size: 512 bytes
          Metadata Stride: 113792 AU
   Duplicate Disk Numbers: 0
---------------------------- SCANNING DISK N0010 -----------------------------
Disk N0010: '/dev/asmdisk_fra1'
           Allocated AU's: 4818
                Free AU's: 15655
       AU's read for dump: 73
       Block images saved: 17669
        Map lines written: 73
          Heartbeats seen: 0
  Corrupt metadata blocks: 0
        Corrupt AT blocks: 0
---------------------------- SCANNING DISK N0011 -----------------------------
Disk N0011: '/dev/asmdisk_fra2'
           Allocated AU's: 4819
                Free AU's: 15660
       AU's read for dump: 76
       Block images saved: 18431
        Map lines written: 76
          Heartbeats seen: 0
  Corrupt metadata blocks: 0
        Corrupt AT blocks: 0
------------------------- SUMMARY FOR DISKGROUP FRA2 -------------------------
           Allocated AU's: 9637
                Free AU's: 31315
       AU's read for dump: 149
       Block images saved: 36100
        Map lines written: 149
          Heartbeats seen: 0
  Corrupt metadata blocks: 0
        Corrupt AT blocks: 0
--> FRA2_0001.img  FRA2.map  are binary files

 

Reference

  • KFOD, KFED, AMDU (Doc ID 1485597.1)
  • Note 1180491.1 – KFED Tool For Windows OS
  • Note 1346190.1 – KFED.PL for diagnosing – ORA-15036 ORA-15042 ORA-15020 ORA-15033
  • Note 1505005.1 – Where to find kfed utility before Oracle Grid Infrastructure is installed

ASM doesn’t start due to CRS-1714 error

Startup and check CRS and verify your disk groups 
# $GRID_HOME/bin/crsctl  start crs
CRS-4123: Oracle High Availability Services has been started.

# $GRID_HOME/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager

$ asmcmd lsdg
Connected to an idle instance.
ASMCMD-8102: no connection to Oracle ASM; command requires Oracle ASM to run

--> From GRID alert.log 
[cssd(10157)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; 
Details at (:CSSNM00070:) in /u01/app/11204/grid/log/grac42/cssd/ocssd.log

$  /etc/init.d/oracleasm listdisks
--> No ASM disks configured - Installation uses udev devices!

Get your disk discovery string and verify your /dev/asm directory 
$GRID_HOME/bin/gpnptool get
DiscoveryString="/dev/asm*
SPFile="+DATA/grac4/asmparameterfile/registry.253.825944079"

$  ls -l /dev/asm*
brw-rw----. 1 grid asmadmin 252, 5 Mar 18 13:42 /dev/asmdisk_OF-disk1
brw-rw----. 1 grid asmadmin 252, 6 Mar 18 13:42 /dev/asmdisk_OF-disk2
brw-rw----. 1 grid asmadmin 252, 7 Mar 18 13:42 /dev/asmdisk_OF-disk3

$ kfed read /dev/asmdisk_OF-disk1 | egrep 'dskname|grpname'
kfdhdb.dskname:       OPENFILER_DG_0000 ; 0x028: length=17
kfdhdb.grpname:            OPENFILER_DG ; 0x048: length=12
[grid@grac42 grac42]$  kfed read /dev/asmdisk_OF-disk2 | egrep 'dskname|grpname'
kfdhdb.dskname:       OPENFILER_DG_0001 ; 0x028: length=17
kfdhdb.grpname:            OPENFILER_DG ; 0x048: length=12
[grid@grac42 grac42]$ kfed read /dev/asmdisk_OF-disk3  | egrep 'dskname|grpname'
kfdhdb.dskname:       OPENFILER_DG_0002 ; 0x028: length=17
kfdhdb.grpname:            OPENFILER_DG ; 0x048: length=12

Verify  voting file location locally
# more /etc/oracle/ocr.loc
ocrconfig_loc=+OCR
local_only=false

Verify voting file location on a working instance
$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   b0e94e5d83054fe9bf58b6b98bfacd65 (/dev/asmdisk5_udev_sdf1) [OCR]
 2. ONLINE   88c2a08b4c8c4f85bf0109e0990388e4 (/dev/asmdisk6_udev_sdg1) [OCR]
 3. ONLINE   1108f9a41e814fb2bfed879ff0039dd0 (/dev/asmdisk7_udev_sdh1) [OCR]

Check ASM devices 
ON Working Node
[root@grac41 rules.d]# ls /dev/asm*
/dev/asmdisk10_ssd3      /dev/asmdisk2_test       /dev/asmdisk4_udev_sde1  /dev/asmdisk8_ssd1     /dev/asmdisk_OF-disk3
/dev/asmdisk1_fra        /dev/asmdisk2_udev_sdc1  /dev/asmdisk5_udev_sdf1  /dev/asmdisk9_ssd2
/dev/asmdisk1_test       /dev/asmdisk3_test       /dev/asmdisk6_udev_sdg1  /dev/asmdisk_OF-disk1
/dev/asmdisk1_udev_sdb1  /dev/asmdisk3_udev_sdd1  /dev/asmdisk7_udev_sdh1  /dev/asmdisk_OF-disk2

Failing Node
grid@grac42 grac42]$  ls -l /dev/asm*
brw-rw----. 1 grid asmadmin 252, 5 Mar 18 13:42 /dev/asmdisk_OF-disk1
brw-rw----. 1 grid asmadmin 252, 6 Mar 18 13:42 /dev/asmdisk_OF-disk2
brw-rw----. 1 grid asmadmin 252, 7 Mar 18 13:42 /dev/asmdisk_OF-disk3

--> All disks for  Diskgroup OCR are missing from /dev/asm* 

Solution : Fix udev file and verify that our OCR diskgroup can be mounted
$ udevadm control --reload-rules
$ start_udev

Verify ASM disks:
[root@grac43 rules.d]#  $GRID_HOME/bin/kfed read  /dev/asmdisk1_udev_sdf1| egrep 'dskname|grpname'
kfdhdb.dskname:                OCR_0000 ; 0x028: length=8
kfdhdb.grpname:                     OCR ; 0x048: length=3
[root@grac43 rules.d]# $GRID_HOME/bin/kfed read  /dev/asmdisk1_udev_sdg1| egrep 'dskname|grpname'
kfdhdb.dskname:                OCR_0001 ; 0x028: length=8
kfdhdb.grpname:                     OCR ; 0x048: length=3
[root@grac43 rules.d]#  $GRID_HOME/bin/kfed read  /dev/asmdisk1_udev_sdh1| egrep 'dskname|grpname'
kfdhdb.dskname:                OCR_0002 ; 0x028: length=8
kfdhdb.grpname:                     OCR ; 0x048: length=3

Restart CRS ( as root ) and verify ASM status
# $GRID_HOME/bin/crsctl  stop crs -f
# $GRID_HOME/bin/crsctl start crs

$ $GRID_HOME/bin/asmcmd lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  NORMAL  N         512   4096  1048576     40944    20431            10236            5097              0             N  DATA/
MOUNTED  EXTERN  N         512   4096  1048576    204797   173814                0          173814              0             N  FRA/
MOUNTED  NORMAL  N         512   4096  1048576      6141     5091             2047            1522              0             Y  OCR/
MOUNTED  EXTERN  N         512   4096  1048576      5739     4565                0            4565              0             N  OPENFILER_DG/
MOUNTED  NORMAL  N         512   4096  1048576      3057     1995             1019             488              0             N  SSD/

 

Mapping Partitions with UDEV:

Mapping Disks with UDEV:

ORA-15040, ORA-15042 errors mounting a diskgroup

Check current status

Try manually mount diskgroup
ASMCMD> mount -a
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "1" is missing from group number "1" 

Check ASM alert.log
SQL> alter diskgroup ACFS mount 
NOTE: cache registered group ACFS number=1 incarn=0x09884abc
NOTE: cache began mount (first) of group ACFS number=1 incarn=0x09884abc
NOTE: Assigning number (1,0) to disk (/dev/oracleasm/disks/ACFS_DATA)
Mon Aug 19 10:29:24 2013
NOTE: GMON heartbeating for grp 1
GMON querying group 1 at 334 for pid 31, osid 8882
NOTE: Assigning number (1,1) to disk ()
GMON querying group 1 at 335 for pid 31, osid 8882
NOTE: cache dismounting (clean) group 1/0x09884ABC (ACFS) 
NOTE: messaging CKPT to quiesce pins Unix process pid: 8882, image: oracle@grac1.example.com (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: lgwr not being msg'd to dismount
NOTE: cache dismounted group 1/0x09884ABC (ACFS) 
NOTE: cache ending mount (fail) of group ACFS number=1 incarn=0x09884abc
NOTE: cache deleting context for group ACFS 1/0x09884abc
GMON dismounting group 1 at 336 for pid 31, osid 8882
NOTE: Disk  in mode 0x8 marked for de-assignment
NOTE: Disk  in mode 0x8 marked for de-assignment
ERROR: diskgroup ACFS was not mounted
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "1" is missing from group number "1" 
ERROR: alter diskgroup ACFS mount
Mon Aug 19 10:29:27 2013
ASM Health Checker found 1 new failures

Display even dismounted diskgroups
$  asmcmd lsdg --discovery
State       Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
DISMOUNTED          N           0   4096        0         0        0                0               0              0             N  ACFS/
MOUNTED     NORMAL  N         512   4096  1048576     15342     9932             5114            2409              0             N  DATA/
MOUNTED     NORMAL  N         512   4096  1048576      6141     5217             2047            1585              0             Y  OCR/
--> diskgroup ACFS is still dismounted  
Try to mount the ASM diskgroup with force option and check the available disks
As we have only a single disk available for NORMAL redundancy the mount fails and we need to use the force option  
SQL> alter diskgroup ACFS mount force;
Diskgroup altered.

Verify the disk status after mount force command. 
$ asmcmd lsdsk -p -G ACFS
Group_Num  Disk_Num      Incarn  Mount_Stat  Header_Stat  Mode_Stat  State   Path
        1         1  3915954601  MISSING     UNKNOWN      OFFLINE    NORMAL  
        1         0  3915954600  CACHED      MEMBER       ONLINE     NORMAL  /dev/oracleasm/disks/ACFS_DATA
$ asmcmd lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  NORMAL  N         512   4096  1048576      1019      122                0              61              1             N  ACFS/
MOUNTED  NORMAL  N         512   4096  1048576     15342     9932             5114            2409              0             N  DATA/
MOUNTED  NORMAL  N         512   4096  1048576      6141     5217             2047            1585              0             Y  OCR/
           1585              0             Y  OCR/
Try to read and verify ASM disk header ( on all instances )
# $GRID_HOME/bin/kfed read  /dev/sdj1
kfbh.endian:                          0 ; 0x000: 0x00
kfbh.hard:                            0 ; 0x001: 0x00
kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt:                          0 ; 0x003: 0x00
kfbh.block.blk:                       0 ; 0x004: blk=0
kfbh.block.obj:                       0 ; 0x008: file=0
kfbh.check:                           0 ; 0x00c: 0x00000000
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
7FB74EC83400 00000000 00000000 00000000 00000000  [................]
        Repeat 63 times
7FB74EC83800 0000FF00 0003FBB8 000032FC 0003DA6D  [.........2..m...]
7FB74EC83810 0000FEF5 00000000 00000002 00000002  [................]
7FB74EC83940 00000000 00000000 00000000 01000000  [................]
7FB74EC83950 00000000 00000000 00000000 001C001C  [................]
7FB74EC83960 00000001 00000000 00000000 00000000  [................]
7FB74EC83970 00000000 00000004 00008196 00000000  [................]
7FB74EC83980 00000000 00000000 00000000 00000000  [................]
  Repeat 167 times
KFED-00322: file not found; arguments: [kfbtTraverseBlock] [Invalid OSM block type] [] [0]
--> This is not a valid  ASM header - let's create new ASM disk:

Useful ASM Commands

Useful ASM commands in a RAC environment

Check diskgroup including the dismounted DGs
$  asmcmd lsdg -g  --discovery 
Inst_ID  State       Type  Rebal  Sector  Block  AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
      3  DISMOUNTED        N           0   4096   0         0        0                0               0              0             N  ACFS/
      2  DISMOUNTED        N           0   4096   0         0        0                0               0              0             N  ACFS/
      1  DISMOUNTED        N           0   4096   0         0        0                0               0              0             N  ACFS/

Check ASM disk status
$ asmcmd lsdsk -k -g 
Inst_ID  Total_MB  Free_MB  OS_MB  Name       Failgroup  Failgroup_Type  Library  Label  UDID  Product  Redund   Path
      3     10236     4316  10236  DATA_0000  DATA_0000  REGULAR         System                         UNKNOWN  /dev/asm_data_10g_disk1
      1     10236     4316  10236  DATA_0000  DATA_0000  REGULAR         System                         UNKNOWN  /dev/asm_data_10g_disk1
      2     10236     4316  10236  DATA_0000  DATA_0000  REGULAR         System                         UNKNOWN  /dev/asm_data_10g_disk1
..

Check ASM disk status for candidate disk 
$  asmcmd lsdsk -k -g --candidate
Inst_ID  Total_MB  Free_MB  OS_MB  Name       Failgroup  Failgroup_Type  Library  Label  UDID  Product  Redund   Path
      3         0        0   1019                        REGULAR         System                         UNKNOWN  /dev/asm_test_1G_disk1
      2         0        0   1019                        REGULAR         System                         UNKNOWN  /dev/asm_test_1G_disk1
      1         0        0   1019                        REGULAR         System                         UNKNOWN  /dev/asm_test_1G_disk1
      3         0        0   1019                        REGULAR         System                         UNKNOWN  /dev/asm_test_1G_disk2
      2         0        0   1019                        REGULAR         System                         UNKNOWN  /dev/asm_test_1G_disk2
      1         0        0   1019                        REGULAR         System                         UNKNOWN  /dev/asm_test_1G_disk2

Clients accessing ASM instance:
$ asmcmd lsct
DB_Name  Status     Software_Version  Compatible_version  Instance_Name  Disk_Group
+ASM     CONNECTED        11.2.0.3.0          11.2.0.3.0  +ASM1          OCR
GRACE2   CONNECTED        11.2.0.3.0          11.2.0.0.0  GRACE21        DATA

Files opened by ASM instance:
$ asmcmd lsof
DB_Name  Instance_Name  Path
+ASM     +ASM1          +ocr.255.4294967295
GRACE2   GRACE21        +data/grace2/controlfile/current.260.822058257
GRACE2   GRACE21        +data/grace2/datafile/example.264.822058263
GRACE2   GRACE21        +data/grace2/datafile/sysaux.257.822058183
GRACE2   GRACE21        +data/grace2/datafile/system.256.822058183
GRACE2   GRACE21        +data/grace2/datafile/undotbs1.258.822058183
GRACE2   GRACE21        +data/grace2/datafile/undotbs2.265.822058387
GRACE2   GRACE21        +data/grace2/datafile/users.259.822058183
GRACE2   GRACE21        +data/grace2/onlinelog/group_1.261.822058257
GRACE2   GRACE21        +data/grace2/onlinelog/group_2.262.822058259
GRACE2   GRACE21        +data/grace2/onlinelog/group_3.266.822058427
GRACE2   GRACE21        +data/grace2/onlinelog/group_4.267.822058427
GRACE2   GRACE21        +data/grace2/tempfile/temp.263.822058261

 

Use kfed, kfod to dump ASM headers and display the current ASM status

Use kfed to display the ASM disk header status
$ $GRID_HOME/bin/kfed  read /dev/oracleasm/disks/DATA2  | grep name
kfdhdb.dskname:               DATA_0001 ; 0x028: length=9
kfdhdb.grpname:                    DATA ; 0x048: length=4
kfdhdb.fgname:                DATA_0001 ; 0x068: length=9
kfdhdb.capname:                         ; 0x088: length=0

Use kfod to get an idea about the ASM disk status
$  kfod asm_diskstring='/dev/oracleasm/disks/*' nohdr=true verbose=true disks=all status=true op=disks
5114 CANDIDATE /dev/oracleasm/disks/DATA1 grid     asmadmin
5114 MEMBER /dev/oracleasm/disks/DATA2 grid     asmadmin
5114 MEMBER /dev/oracleasm/disks/DATA3 grid     asmadmin
2047 CANDIDATE /dev/oracleasm/disks/OCR1 grid     asmadmin
2047 CANDIDATE /dev/oracleasm/disks/OCR2 grid     asmadmin
2047 CANDIDATE /dev/oracleasm/disks/OCR3 grid     asmadmin

 

Use -g switch to check whether all diskgroups and disks are mounted clusterwide

$ asmcmd lsdg -g
Inst_ID  State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
      1  MOUNTED  NORMAL  N         512   4096  1048576      6141     5089             2047            1521              0             Y  OCR/
      3  MOUNTED  NORMAL  N         512   4096  1048576      6141     5089             2047            1521              0             Y  OCR/
      2  MOUNTED  NORMAL  N         512   4096  1048576      6141     5089             2047            1521              0             Y  OCR/
      3  MOUNTED  EXTERN  N         512   4096  1048576      5739     4565                0            4565              0             N  OPENFILER_DG/
      2  MOUNTED  EXTERN  N         512   4096  1048576      5739     4565                0            4565              0             N  OPENFILER_DG/
      1  MOUNTED  NORMAL  N         512   4096  1048576      3057     1995             1019             488              0             N  SSD/
      3  MOUNTED  NORMAL  N         512   4096  1048576      3057     1995             1019             488              0             N  SSD/
      2  MOUNTED  NORMAL  N         512   4096  1048576      3057     1995             1019             488              0             N  SSD/
--> OPENFILER diskgroup not mount on instance 1

# asmcmd lsdsk -k -g
Inst_ID  Total_MB  Free_MB   OS_MB  Name               Failgroup          Failgroup_Type  Library  Label  UDID  Product  Redund   Path
      1    204797   173589  204797  FRA_0000           FRA_0000           REGULAR         System                         UNKNOWN  /dev/asmdisk1_fra
      2    204797   173589  204797  FRA_0000           FRA_0000           REGULAR         System                         UNKNOWN  /dev/asmdisk1_fra
      3    204797   173589  204797  FRA_0000           FRA_0000           REGULAR         System                         UNKNOWN  /dev/asmdisk1_fra
...
      2      1913     1521    1913  OPENFILER_DG_0000  OPENFILER_DG_0000  REGULAR         System                         UNKNOWN  /dev/asmdisk_OF-disk1
      3      1913     1521    1913  OPENFILER_DG_0000  OPENFILER_DG_0000  REGULAR         System                         UNKNOWN  /dev/asmdisk_OF-disk1
      2      1913     1522    1913  OPENFILER_DG_0001  OPENFILER_DG_0001  REGULAR         System                         UNKNOWN  /dev/asmdisk_OF-disk2
      3      1913     1522    1913  OPENFILER_DG_0001  OPENFILER_DG_0001  REGULAR         System                         UNKNOWN  /dev/asmdisk_OF-disk2
      2      1913     1522    1913  OPENFILER_DG_0002  OPENFILER_DG_0002  REGULAR         System                         UNKNOWN  /dev/asmdisk_OF-disk3
      3      1913     1522    1913  OPENFILER_DG_0002  OPENFILER_DG_0002  REGULAR         System                         UNKNOWN  /dev/asmdisk_OF-disk3
-->  Disks OPENFILER_DG_0000, OPENFILER_DG_0001, OPENFILER_DG_0002 not visible on Instance 1 
     Action: check UDEV rules on instance 1 and verify diskheader with kfed

Dismount a diskgroup on a local instance

ASMCMD> lsdg -g OPENFILER_DG 
Inst_ID  State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
      1  MOUNTED  EXTERN  N         512   4096  1048576      5739     4565                0            4565              0             N  OPENFILER_DG/
      3  MOUNTED  EXTERN  N         512   4096  1048576      5739     4565                0            4565              0             N  OPENFILER_DG/
      2  MOUNTED  EXTERN  N         512   4096  1048576      5739     4565                0            4565              0             N  OPENFILER_DG/
--> Diskgroup mounted on all instances 

ASMCMD> umount  OPENFILER_DG
ORA-15032: not all alterations performed
ORA-15027: active use of diskgroup "OPENFILER_DG" precludes its dismount (DBD ERROR: OCIStmtExecute)

ASMCMD> umount -f OPENFILER_DG
ASMCMD>  lsdg -g OPENFILER_DG 
Inst_ID  State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
      3  MOUNTED  EXTERN  N         512   4096  1048576      5739     4565                0            4565              0             N  OPENFILER_DG/
      2  MOUNTED  EXTERN  N         512   4096  1048576      5739     4565                0            4565              0             N  OPENFILER_DG/
--> Diskgroup OPENFILER dismounted on instance 1 

 

Manually add an ASM instance after Oracle Restart installation  failed

Check resources - you may need to drop the ASM resource if already available
[root@orars4 Desktop]# crs
*****  Local Resources: *****
Rescource NAME                 TARGET     STATE           SERVER       STATE_DETAILS                       
-------------------------      ---------- ----------      ------------ ------------------                  
ora.LISTENER.lsnr              ONLINE     ONLINE          orars4          
ora.ons                        OFFLINE    OFFLINE         orars4          
*****  Cluster Resources: *****
Resource NAME               INST   TARGET       STATE        SERVER          STATE_DETAILS
--------------------------- ----   ------------ ------------ --------------- -----------------------------------------
ora.cssd                       1   OFFLINE      OFFLINE                        
ora.diskmon                    1   OFFLINE      OFFLINE                        
ora.evmd                       1   ONLINE       ONLINE       orars4   

--> Run GUI tool asmca and create DG 

Check resourses again asm resoruce is add and up and running 
[root@orars4 Desktop]# crs
*****  Local Resources: *****
Rescource NAME                 TARGET     STATE           SERVER       STATE_DETAILS                       
-------------------------      ---------- ----------      ------------ ------------------                  
ora.DATA.dg                    ONLINE     ONLINE          orars4          
ora.LISTENER.lsnr              ONLINE     ONLINE          orars4          
ora.asm                        ONLINE     ONLINE          orars4       Started   
ora.ons                        OFFLINE    OFFLINE         orars4          
*****  Cluster Resources: *****
Resource NAME               INST   TARGET       STATE        SERVER          STATE_DETAILS
--------------------------- ----   ------------ ------------ --------------- -----------------------------------------
ora.cssd                       1   ONLINE       ONLINE       orars4            
ora.diskmon                    1   OFFLINE      OFFLINE                        
ora.evmd                       1   ONLINE       ONLINE       orars4