Table of Contents
Current Cluster Configuration
ASM Configuration [root@gract3 ~]# asmcmd showclustermode ASM cluster : Flex mode enabled [root@gract3 ~]# srvctl status asm -detail ASM is running on gract2,gract1 ASM is enabled. [root@gract3 ~]# crsctl status resource ora.asm -f | grep CARDINALITY= CARDINALITY=2 RAC configuration [root@gract3 ~]# olsnodes -c gract [root@gract3 ~]# crsctl get node role status -all Node 'gract1' active role is 'hub' Node 'gract2' active role is 'hub' Node 'gract3' active role is 'hub' Summary : - This cluster is running 3 HUB nodes where gract2 and gract1 are currently hosting our 2 ASM FLEX instances Following CRS resource output reflects the current status [root@gract1 Desktop]# crs ***** Local Resources: ***** Rescource NAME TARGET STATE SERVER STATE_DETAILS ------------------------- ---------- ---------- ------------ ------------------ ora.ACFS_DG1.ACFS_VOL1.advm ONLINE ONLINE gract1 Volume device /dev/a sm/acfs_vol1-443 isonline,STABLE ora.ACFS_DG1.ACFS_VOL1.advm ONLINE ONLINE gract2 Volume device /dev/a sm/acfs_vol1-443 isonline,STABLE ora.ACFS_DG1.ACFS_VOL1.advm ONLINE OFFLINE gract3 Unable to connect toASM,STABLE ora.ACFS_DG1.dg ONLINE ONLINE gract1 STABLE ora.ACFS_DG1.dg ONLINE ONLINE gract2 STABLE ora.ACFS_DG1.dg ONLINE OFFLINE gract3 STABLE ora.ASMNET1LSNR_ASM.lsnr ONLINE ONLINE gract1 STABLE ora.ASMNET1LSNR_ASM.lsnr ONLINE ONLINE gract2 STABLE ora.ASMNET1LSNR_ASM.lsnr ONLINE OFFLINE gract3 STABLE ora.DATA.dg ONLINE ONLINE gract1 STABLE ora.DATA.dg ONLINE ONLINE gract2 STABLE ora.DATA.dg ONLINE OFFLINE gract3 STABLE ora.LISTENER.lsnr ONLINE ONLINE gract1 STABLE ora.LISTENER.lsnr ONLINE ONLINE gract2 STABLE ora.LISTENER.lsnr ONLINE OFFLINE gract3 STABLE ora.acfs_dg1.acfs_vol1.acfs ONLINE ONLINE gract1 mounted on /u01/acfs /acfs-vol1,STABLE ora.acfs_dg1.acfs_vol1.acfs ONLINE ONLINE gract2 mounted on /u01/acfs /acfs-vol1,STABLE ora.acfs_dg1.acfs_vol1.acfs ONLINE OFFLINE gract3 (1) volume /u01/acfs /acfs-vol1 offline,STABLE ora.net1.network ONLINE ONLINE gract1 STABLE ora.net1.network ONLINE ONLINE gract2 STABLE ora.net1.network ONLINE ONLINE gract3 STABLE ora.ons ONLINE ONLINE gract1 STABLE ora.ons ONLINE ONLINE gract2 STABLE ora.ons ONLINE OFFLINE gract3 STABLE ora.proxy_advm ONLINE ONLINE gract1 STABLE ora.proxy_advm ONLINE ONLINE gract2 STABLE ora.proxy_advm ONLINE OFFLINE gract3 STABLE ***** Cluster Resources: ***** Resource NAME INST TARGET STATE SERVER STATE_DETAILS --------------------------- ---- ------------ ------------ --------------- ----------------------------------------- MyTestVIP 1 ONLINE OFFLINE - STABLE My_apache 1 ONLINE OFFLINE - STABLE ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE gract1 STABLE ora.LISTENER_SCAN2.lsnr 1 ONLINE ONLINE gract2 STABLE ora.LISTENER_SCAN3.lsnr 1 ONLINE ONLINE gract1 STABLE ora.MGMTLSNR 1 ONLINE ONLINE gract1 169.254.235.72 192.1 68.2.111,STABLE ora.asm 1 ONLINE ONLINE gract1 STABLE ora.asm 3 ONLINE ONLINE gract2 STABLE ora.cvu 1 ONLINE ONLINE gract1 STABLE ora.dw.db 1 ONLINE ONLINE gract2 Open,STABLE ora.dw.db 2 ONLINE OFFLINE - STABLE ora.erp.db 1 ONLINE ONLINE gract3 Open,STABLE ora.erp.db 2 OFFLINE OFFLINE - Instance Shutdown,ST ABLE ora.erp.db 3 ONLINE ONLINE gract1 Open,STABLE ora.gns 1 ONLINE ONLINE gract1 STABLE ora.gns.vip 1 ONLINE ONLINE gract1 STABLE ora.gract1.vip 1 ONLINE ONLINE gract1 STABLE ora.gract2.vip 1 ONLINE ONLINE gract2 STABLE ora.gract3.vip 1 ONLINE INTERMEDIATE gract2 FAILED OVER,STABLE ora.mgmtdb 1 ONLINE ONLINE gract1 Open,STABLE ora.oc4j 1 ONLINE ONLINE gract1 STABLE ora.scan1.vip 1 ONLINE ONLINE gract1 STABLE ora.scan2.vip 1 ONLINE ONLINE gract2 STABLE ora.scan3.vip 1 ONLINE ONLINE gract1 STABLE Following instances ------ RAC Database Info --------- Database name: dw Server pools: STANDARD_PRIORITY Instance dw_1 is running on node gract2 Database name: ERP Server pools: TOP_PRIORITY Instance ERP_1 is running on node gract3 Instance ERP_3 is running on node gract1 ------ Serverpool Info Config --------- NAME=Free IMPORTANCE=0 MIN_SIZE=0 MAX_SIZE=-1 SERVER_CATEGORY= NAME=ora.TOP_PRIORITY IMPORTANCE=5 MIN_SIZE=1 MAX_SIZE=2 SERVER_CATEGORY=ora.hub.category NAME=ora.STANDARD_PRIORITY IMPORTANCE=5 MIN_SIZE=1 MAX_SIZE=2 SERVER_CATEGORY=ora.hub.category ------ Serverpool Info --------- Server pool name: Free Active servers count: 0 Active server names: Server pool name: TOP_PRIORITY Active servers count: 2 Active server names: gract1,gract3 NAME=gract1 STATE=ONLINE NAME=gract3 STATE=ONLINE Server pool name: STANDARD_PRIORITYActive servers count: 1 Active server names: gract2 NAME=gract2 STATE=ONLINE
What Rescource Changes can we expect when server crashes or a CRS stack shutdown occurs
Run crsctl eval to get a rought idea what is going on [root@gract3 ~]# crsctl eval delete server gract2 -> Server 'gract2' will be removed from pools [ora.STANDARD_PRIORITY] As crs is down gract2 need to leave the pool Server 'gract3' will be moved from pools [ora.TOP_PRIORITY] to gract2 needs to move to STANDARD_PRIORITY pools [ora.STANDARD_PRIORITY] pool to satisfiy MIN_SIZE=1 pool settings Resource 'ora.DATA.dg' (gract2) will be in state [OFFLINE] As our FLEX ASM runs on gract2 DG are going OFFLINE Resource 'ora.asm' (3/1) will be in state [OFFLINE] After DGs are OFFLINE we can stop the ASM server Resource 'ora.dw.db' (1/1) will be in state [OFFLINE] DW instance needs to go OFFLINE as server gract2 leaves pool STANDARD_PRIORITY Resource 'ora.erp.db' (1/1) will be in state [OFFLINE] ERP_1 needs to set OFFLINE as server gract3 leaves that pool Resource 'ora.DATA.dg' (gract3) will be in state [ONLINE|INTERMEDIATE] Flex ASM restarted on gract3 on server [gract3] It seems no all events are displayed - but it is a good test to get a rough idea what is going on. The startup of DW instance was missing for crsctl eval delete server gract2 command. Afer stopping CRS on server gract2 by runinng [root@gract2 ~]# crsctl stop crs The following changes are active now ------ RAC Database Info --------- Database name: dw Server pools: STANDARD_PRIORITY Instance dw_2 is running on node gract3 Database name: ERP Server pools: TOP_PRIORITY Instance ERP_3 is running on node gract1 [grid@gract1 ~/PM]$ srvctl status asm -detail ASM is running on gract3,gract1 ASM is enabled.
Summary
- crsctl eval is a good tool to verify clusterware actions happening for a certain command
- Due to serverpool operations even instances not running on affected server can go OFFLINE
- Use crsctl eval delete server .. to track changes for a CRS stack outage
Full output from running: crsctl eval delete server ..
[root@gract2 ~]# crsctl eval delete server gract3 Stage Group 1: -------------------------------------------------------------------------------- Stage Number Required Action -------------------------------------------------------------------------------- 1 Y Server 'gract2' will be moved from pools [ora.TOP_PRIORITY] to pools [ora.STANDARD_PRIORITY] Y Server 'gract3' will be removed from pools [ora.STANDARD_PRIORITY] Y Resource 'ora.ACFS_DG1.ACFS_VOL1.advm' (gract3) will be in state [OFFLINE] Y Resource 'ora.ACFS_DG1.dg' (gract3) will be in state [OFFLINE] Y Resource 'ora.ASMNET1LSNR_ASM.lsnr' (gract3) will be in state [OFFLINE] Y Resource 'ora.DATA.dg' (gract3) will be in state [OFFLINE] Y Resource 'ora.LISTENER.lsnr' (gract3) will be in state [OFFLINE] Y Resource 'ora.LISTENER_SCAN1.lsnr' (1/1) will be in state [OFFLINE] Y Resource 'ora.acfs_dg1.acfs_vol1.acfs' (gract3) will be in state [OFFLINE] Y Resource 'ora.asm' (3/1) will be in state [OFFLINE] Y Resource 'ora.dw.db' (1/1) will be in state [OFFLINE] Y Resource 'ora.gract3.vip' (1/1) will be in state [OFFLINE] Y Resource 'ora.net1.network' (gract3) will be in state [OFFLINE] Y Resource 'ora.ons' (gract3) will be in state [OFFLINE] Y Resource 'ora.proxy_advm' (gract3) will be in state [OFFLINE] Y Resource 'ora.scan1.vip' (1/1) will be in state [OFFLINE] 2 N Create new group (Stage Group = 2) Y Resource 'ora.asm' (3/1) will be in state [ONLINE|INTERMEDIATE] on server [gract2] Y Resource 'ora.gract3.vip' (1/1) will be in state [ONLINE|INTERMEDIATE] on server [gract2] Y Resource 'ora.scan1.vip' (1/1) will be in state [ONLINE] on server [gract1] 3 Y Resource 'ora.LISTENER_SCAN1.lsnr' (1/1) will be in state [ONLINE|INTERMEDIATE] on server [gract1] Y Resource 'ora.erp.db' (1/1) will be in state [OFFLINE] -------------------------------------------------------------------------------- Stage Group 2: -------------------------------------------------------------------------------- Stage Number Required Action -------------------------------------------------------------------------------- 1 N Resource 'ora.DATA.dg' (gract2) will be in state [ONLINE|INTERMEDIATE] on server [gract2] --------------------------------------------------------------------------------
crsctl commands supporting eval switch
crsctl eval start resource {<resname>|-w <filter>}[-n server] [-f]
crsctl eval stop resource {<resname>|-w <filter>} [-f]
crsctl eval relocate resource {<resName> | {<resName>|-all} -s <server> | -w <filter>} {-n <server>} [-f]
crsctl eval add serverpool <spName> [-file <filePath> | -attr "<attrName>=<value>[,...]"] [-f] [-admin [-l <level>] [-x] [-a]]
crsctl eval modify serverpool <spName> {-file <filePath> | -attr "<attrName>=<value>[,...]"} [-f] [-admin [-l <level>] [-x] [-a]]
crsctl eval delete serverpool <spName> [-admin [-l <level>] [-x] [-a]]
crsctl eval add server <serverName> [-file <filePath> | -attr "<attrName>=<value>[,...]"] [-f] [-admin [-l <level>] [-x] [-a]]
crsctl eval relocate server <serverName> -to <toPool> [-f] [-admin [-l <level>] [-x] [-a]]
crsctl eval delete server <serverName> [-f] [-admin [-l <level>] [-x] [-a]]
crsctl eval add resource <resName> -type <typeName> [-file <filePath> | -attr "<attrName>=<value>[,...]"] [-f]
crsctl eval modify resource <resName> -attr "<attrName>=<value>[,...]" [-f]
crsctl eval fail resource {<resname> | -w <filter>} [-n <server>]
crsctl eval activate policy <policyName> [-f] [-admin [-l <level>] [-x] [-a]]
Reference
Hi Admin.
Can you please share your crs script
Either you use the menu: RAC -> RAC Generic -> RAC scripts
or go directly to http://www.hhutzler.de/blog/rac-scripts/
what will we do if we find an error?