Recovering corrupted OLR after PROCL-5 PROCL-26 error

Backup current OLR and all available OLR backups  – don’t miss this step

# ocrconfig -local -manualbackup
grac41     2014/05/20 12:22:20     /u01/app/11204/grid/cdata/grac41/backup_20140520_122220.olr

Verify available backups
# ocrconfig -local -showbackup
grac41     2014/05/20 12:22:20     /u01/app/11204/grid/cdata/grac41/backup_20140520_122220.olr
..
# ls -l /u01/app/11204/grid/cdata/grac41/*
-rw-------. 1 root root 6803456 Sep 12  2013 /u01/app/11204/grid/cdata/grac41/backup_20130912_130457.olr
-rw-------. 1 root root 7016448 May 20 09:08 /u01/app/11204/grid/cdata/grac41/backup_20140520_090842.olr
-rw-------. 1 root root 6807552 May 20 12:22 /u01/app/11204/grid/cdata/grac41/backup_20140520_122220.olr

Restore a  OLR from backup

Shutdown CRS and check that ohasd.bin process is not running and restore OLR
# crsctl stop crs
# ps -elf | egrep "PID|d.bin|ohas|oraagent|orarootagent|cssdagent|cssdmonitor" | grep -v grep
F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
4 S root      4494     1  0  80   0 -  2846 wait   10:04 ?        00:00:00 /bin/sh /etc/init.d/init.ohasd run

# ocrconfig -local -restore  /u01/app/11204/grid/cdata/grac41/backup_20140520_090842.olr
# crsctl stop crs

Recover OLR after PROCL-26. PROCL-5 error and no OLR backups available

Tracefile:
# crsctl start crs
TraceFileName: ./client/ocrcheck_27844.log
2014-05-20 09:38:48.918: [OCRCHECK][2768283392]Failed to access OCR repository. Retval [26] Error [PROCL-26: Error while accessing the physical storage]

TraceFileName: ./gpnpd/gpnpd_1.log
2014-05-19 13:23:19.100: [    GPNP][85341984]procr_open_key_ext: OLR error message : PROCL-5: User does not have permission to perform a local registry operation on this key. Authentication error 
[User does not have permission to perform this operation] [0]

Details of PROCL-5 from  ./gpnpd/gpnpd_1.log
..
[   CLWAL][3606726432]clsw_Initialize: OLR initlevel [70000]
[  OCRAPI][3606726432]a_init:10: AUTH LOC [/u01/app/11204/grid/srvm/auth]
[  OCRMSG][3606726432]prom_init: Successfully registered comp [OCRMSG] in clsd.
2014-05-20 07:57:51.118: [  OCRAPI][3606726432]a_init:11: Messaging init successful.
[  OCRCLI][3606726432]oac_init: Successfully registered comp [OCRCLI] in clsd.
2014-05-20 07:57:51.118: [  OCRCLI][3606726432]proac_con_init: Local listener using IPC. [(ADDRESS=(PROTOCOL=ipc)(KEY=procr_local_conn_0_PROL))]
2014-05-20 07:57:51.119: [  OCRCLI][3606726432]proac_con_init: Successfully connected to the server
2014-05-20 07:57:51.119: [  OCRCLI][3606726432]proac_con_init: Post sema. Con count [1]
2014-05-20 07:57:51.120: [  OCRAPI][3606726432]a_init:12: Client init successful.
2014-05-20 07:57:51.120: [  OCRAPI][3606726432]a_init:21: OCR init successful. Init Level [7]
2014-05-20 07:57:51.120: [  OCRAPI][3606726432]a_init:2: Init Level [7]
2014-05-20 07:57:51.132: [  OCRCLI][3606726432]proac_con_init: Post sema. Con count [2]
[  clsdmt][3595089664]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=grac41DBG_GPNPD))
2014-05-20 07:57:51.133: [  clsdmt][3595089664]PID for the Process [31034], connkey 10 
2014-05-20 07:57:51.133: [  clsdmt][3595089664]Creating PID [31034] file for home /u01/app/11204/grid host grac41 bin gpnp to /u01/app/11204/grid/gpnp/init/
2014-05-20 07:57:51.133: [  clsdmt][3595089664]Writing PID [31034] to the file [/u01/app/11204/grid/gpnp/init/grac41.pid] 
2014-05-20 07:57:52.108: [    GPNP][3606726432]clsgpnpd_validateProfile: [at clsgpnpd.c:2919] GPnPD taken cluster name 'grac4'
2014-05-20 07:57:52.108: [    GPNP][3606726432]clsgpnpd_openLocalProfile: [at clsgpnpd.c:3477] Got local profile from file cache provider (LCP-FS).
2014-05-20 07:57:52.111: [    GPNP][3606726432]clsgpnpd_openLocalProfile: [at clsgpnpd.c:3532] Got local profile from OLR cache provider (LCP-OLR).
2014-05-20 07:57:52.113: [    GPNP][3606726432]procr_open_key_ext: OLR api procr_open_key_ext failed for key SYSTEM.GPnP.profiles.peer.pending
2014-05-20 07:57:52.113: [    GPNP][3606726432]procr_open_key_ext: OLR current boot level : 7
2014-05-20 07:57:52.113: [    GPNP][3606726432]procr_open_key_ext: OLR error code    : 5
2014-05-20 07:57:52.126: [    GPNP][3606726432]procr_open_key_ext: OLR error message : PROCL-5: User does not have permission to perform a local registry operation on this key. Authentication error 
                         [User does not have permission to perform this operation] [0]
2014-05-20 07:57:52.126: [    GPNP][3606726432]clsgpnpco_ocr2profile: [at clsgpnpco.c:578] Result: (58) CLSGPNP_OCR_ERR. Failed to open requested OLR Profile.
2014-05-20 07:57:52.127: [    GPNP][3606726432]clsgpnpd_lOpen: [at clsgpnpd.c:1734] Listening on ipc://GPNPD_grac41
2014-05-20 07:57:52.127: [    GPNP][3606726432]clsgpnpd_lOpen: [at clsgpnpd.c:1743] GIPC gipcretFail (1) gipcListen listen failure on 
2014-05-20 07:57:52.127: [ default][3606726432]GPNPD failed to start listening for GPnP peers. 
2014-05-20 07:57:52.135: [    GPNP][3606726432]clsgpnpd_term: [at clsgpnpd.c:1344] STOP GPnPD terminating. Closing connections...
2014-05-20 07:57:52.137: [ default][3606726432]clsgpnpd_term STOP terminating.
2014-05-20 07:57:53.136: [  OCRAPI][3606726432]a_terminate:1:current ref count = 1
2014-05-20 07:57:53.136: [  OCRAPI][3606726432]a_terminate:1:current ref count = 0

Check OLR configuration 
[root@grac41 Desktop]#   ocrconfig -local -showbackup
--> No backup 

[root@grac41 Desktop]#  ocrcheck -local
Status of Oracle Local Registry is as follows :
     Version                  :          3
     Total space (kbytes)     :     262120
     Used space (kbytes)      :       2704
     Available space (kbytes) :     259416
     ID                       : 1855884304
     Device/File Name         : /u01/app/11204/grid/cdata/grac41.olr
                                    Device/File integrity check succeeded
     Local registry integrity check succeeded
     Logical corruption check succeeded

Dump OLR and display the failing resource:  SYSTEM.GPnP.profiles.peer.pending 
[root@grac41 Desktop]# ocrdump -local
-->
[SYSTEM.GPnP]
UNDEF :
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_NONE, OTHER_PERMISSION : PROCR_NONE, USER_NAME : grid, GROUP_NAME : oinstall}
[SYSTEM.GPnP.profiles]
BYTESTREAM (16) :
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_NONE, OTHER_PERMISSION : PROCR_NONE, USER_NAME : grid, GROUP_NAME : oinstall}
[SYSTEM.GPnP.profiles.peer]
UNDEF :
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_NONE, OTHER_PERMISSION : PROCR_NONE, USER_NAME : grid, GROUP_NAME : oinstall}
[SYSTEM.GPnP.profiles.peer.best]
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_NONE, OTHER_PERMISSION : PROCR_NONE, USER_NAME : grid, GROUP_NAME : oinstall}
[SYSTEM.GPnP.profiles.peer.pending]
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_NONE, OTHER_PERMISSION : PROCR_NONE, USER_NAME : root, GROUP_NAME : root}
--> Why do we have root permissions for resource  SYSTEM.GPnP.profiles.peer.pending] ? 
    Maybe GPnP process was started as root and created above entry ? 

Anyway before spending to much time in debugging we need to restore OLR from backup or recreate OLR.
Assuming we don't have a backup recreating OLR  by following Note :  1193643.1 
Recreating OLR  by create an empty OLR file.
# cd  /u01/app/11204/grid/cdata/
# touch /u01/app/11204/grid/cdata/grac41.olr
# chown  grid:oinstall /u01/app/11204/grid/cdata/grac41.olr
# chmod 600  /u01/app/11204/grid/cdata/grac41.olr
# ls -l /u01/app/11204/grid/cdata/grac41.olr
-rw-------. 1 grid oinstall 0 May 20 09:14 /u01/app/11204/grid/cdata/grac41.olr
-> But doesn't work get PROCL-26 during clusterware restart

Recreating OCR and OLR by following Doc ID 1539020.1 :
# crsctl stop crs -f 
# $GRID_HOME/crs/install/rootcrs.pl -deconfig -force
# $GRID_HOME/root.sh

After less than 10 minutes CW is up and running again !

Reference:

• How to Recreate OLR When OLR Backup does not Exist or Corrupted (Doc ID 1539020.1)
• How to backup or restore OLR in 11.2 Grid Infrastructure (Doc ID 1193643.1)