Table of Contents
Overview
Note: - The information in this posts is not validated by Oracle, is not supported by Oracle, and should only be used at your own risk; it is for educational purposes only. - Backup your OCR before running any commands found below - Most parameter are case sensitive - Database and Instances are uppercase - where hostnames are lower case - be carefull with that or you can easily corrupt your OCR again
Step 1: Cleanup OCR resources for a specific node which needs to be deleted ( ract3 )
Stop CRS stack on ract3 [root@ract3 ~]# crsctl stop crs Remove Instance from node ract3 [oracle@ract1 ~]$ srvctl stop instance -d RACT -i RACT3 [oracle@ract1 ~]$ srvctl remove instance -d RACT -i RACT3 Remove instance RACT3 from the database ract? (y/[n]) y Remove ASM from node ract3 $ srvctl stop asm -n ract3 $ srvctl remove asm -f -n ract3 Remove Listener from node ract3 [root@ract3 ract3]# lsnrctl stop LISTENER_RACT3 [root@ract3 ract3]# crs_unregister ora.ract3.LISTENER_RACT3.lsnr Stop and remove nodeapps from node ract3 [root@ract1 install]# srvctl stop nodeapps -n ract3 CRS-0210: Could not find resource 'ora.ract3.LISTENER_RACT3.lsnr'. [root@ract1 install]# srvctl remove nodeapps -n ract3 Please confirm that you intend to remove the node-level applications on node ract3 (y/[n]) y Verify CRS resources : [root@ract1 install]# crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....T1.inst application ONLINE ONLINE ract1 ora....T2.inst application ONLINE ONLINE ract2 ora.RACT.db application ONLINE ONLINE ract2 ora....SM1.asm application ONLINE ONLINE ract1 ora....T1.lsnr application ONLINE ONLINE ract1 ora.ract1.gsd application ONLINE ONLINE ract1 ora.ract1.ons application ONLINE ONLINE ract1 ora.ract1.vip application ONLINE ONLINE ract1 ora....SM2.asm application ONLINE ONLINE ract2 ora....T2.lsnr application ONLINE ONLINE ract2 ora.ract2.gsd application ONLINE ONLINE ract2 ora.ract2.ons application ONLINE ONLINE ract2 ora.ract2.vip application ONLINE ONLINE ract2 --> All resource using ract3 are deleted now Verify that node ract3 is still part of our cluster [root@ract1 install]# olsnodes -n ract1 1 ract2 2 ract3 3 Remove node ract3 from OCR [root@ract1 install]# cd $ORA_CRS_HOME/install [root@ract1 install]# ./rootdeletenode.sh ract3,3 CRS-0210: Could not find resource 'ora.ract3.LISTENER_RACT3.lsnr'. CRS-0210: Could not find resource 'ora.ract3.ons'. CRS-0210: Could not find resource 'ora.ract3.vip'. CRS-0210: Could not find resource 'ora.ract3.gsd'. CRS-0210: Could not find resource ora.ract3.vip. CRS nodeapps are deleted successfully clscfg: EXISTING configuration version 3 detected. clscfg: version 3 is 10G Release 2. Successfully deleted 14 values from OCR. Key SYSTEM.css.interfaces.noderact3 marked for deletion is not there. Ignoring. Successfully deleted 5 keys from OCR. Node deletion operation successful. 'ract3,3' deleted successfully [root@ract1 install]# olsnodes -n ract1 1 ract2 2
Add resources again by manually calling commands found in rootaddnode.sh
Comands executed by rootaddnode.sh on ract1 Configure cluster topology for ract3 # /u01/app/oracle/product/crs/bin/clscfg -add -nn ract3,3 -pn ract3int,3 -hn ract3,3 clscfg: EXISTING configuration version 3 detected. clscfg: version 3 is 10G Release 2. Attempting to add 1 new nodes to the configuration Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node <nodenumber>: <nodename> <private interconnect name> <hostname> node 3: ract3 ract3int ract3 Creating OCR keys for user 'root', privgrp 'root'.. Add nodeapps for node ract3 [root@ract1 install]# /u01/app/oracle/product/crs/bin/srvctl add nodeapps -n ract3 -A ract3vip.example.com/255.255.255.0/eth1 -o /u01/app/oracle/product/crs Verify CRS objecs for node ract3 [root@ract1 install]# crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.ract3.gsd application OFFLINE OFFLINE ora.ract3.ons application OFFLINE OFFLINE ora.ract3.vip application OFFLINE OFFLINE Restart CRS stack on node ract3 root@ract3 ract3]# crsctl start crs root@ract3 ract3]# crsctl check crs CSS appears healthy CRS appears healthy EVM appears healthy [root@ract3 ract3]# crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.ract3.gsd application ONLINE ONLINE ract3 ora.ract3.ons application ONLINE ONLINE ract3 ora.ract3.vip application ONLINE ONLINE ract3 Configure listener on node ract3 Run netca and delete old ract3 configuration - add the listener again after deletion [oracle@ract1 ~]$ netca & Select Nodes to configure : ract3 -> Delete ract3... Deleted listener: LISTENER_RACT3 Select Nodes to configure : ract3 -> Add ract3... Listener configuration complete. [oracle@ract1 ~]$ crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....T3.lsnr application ONLINE ONLINE ract3 ora.ract3.gsd application ONLINE ONLINE ract3 ora.ract3.ons application ONLINE ONLINE ract3 ora.ract3.vip application ONLINE ONLINE ract3 Configure ASM Retrieve the needed parameter from a working node [oracle@ract1 ~]$ srvctl config asm -n ract1 +ASM1 /u01/app/oracle/product/10.2/rac_db1 [oracle@ract1 ~]$ srvctl add asm -n ract3 -i +ASM3 -o /u01/app/oracle/product/10.2/rac_db1 [oracle@ract1 ~]$ srvctl config asm -n ract3 +ASM3 /u01/app/oracle/product/10.2/rac_db1 [oracle@ract1 ~]$ srvctl status asm -n ract3 ASM instance +ASM3 is not running on node ract3. [oracle@ract1 ~]$ srvctl start asm -n ract3 [oracle@ract1 ~]$ srvctl status asm -n ract3 ASM instance +ASM3 is running on node ract3. [oracle@ract1 ~]$ crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....SM3.asm application ONLINE ONLINE ract3 ora....T3.lsnr application ONLINE ONLINE ract3 ora.ract3.gsd application ONLINE ONLINE ract3 ora.ract3.ons application ONLINE ONLINE ract3 ora.ract3.vip application ONLINE ONLINE ract3 Configure Instance RACT3 on node ract3 Get Database status [oracle@ract1 ~]$ srvctl status database -d RACT Instance RACT1 is running on node ract1 Instance RACT2 is running on node ract2 Get configugred instances [oracle@ract1 ~]$ srvctl config database -d RACT ract1 RACT1 /u01/app/oracle/product/10.2/rac_db1 ract2 RACT2 /u01/app/oracle/product/10.2/rac_db1 --> Node names are case sensitve - at least for 10.2.0.1 ! Check how the currently running instances are configured [oracle@ract1 ~]$ crs_stat | grep inst NAME=ora.RACT.RACT1.inst NAME=ora.RACT.RACT2.inst [oracle@ract1 ~]$ crs_stat ora.RACT.RACT1.inst NAME=ora.RACT.RACT1.inst TYPE=application TARGET=ONLINE STATE=ONLINE on ract1 --> Database ( RACT ) and Instance ( RACT1 ) are upper case - hostname ( ract1 ) are lower case Recreate Instance [oracle@ract1 ~]$ srvctl add instance -d RACT -i RACT3 -n ract3 [oracle@ract1 ~]$ srvctl start instance -d RACT -i RACT3 [oracle@ract1 ~]$ srvctl status database -d RACT Instance RACT1 is running on node ract1 Instance RACT2 is running on node ract2 Instance RACT3 is running on node ract3 [oracle@ract1 install]$ crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....T3.inst application ONLINE ONLINE ract3 ora....SM3.asm application ONLINE ONLINE ract3 ora....T3.lsnr application ONLINE ONLINE ract3 ora.ract3.gsd application ONLINE ONLINE ract3 ora.ract3.ons application ONLINE ONLINE ract3 ora.ract3.vip application ONLINE ONLINE ract3
Potential error: CRS-210 due mixing lower case and upper case names
Note we use ract3 for our instance name and ract for datbase name but we should use RACT and RACT3 [oracle@ract1 ~]$ srvctl add instance -d ract -i ract3 -n ract3 [oracle@ract1 ~]$ srvctl status database -d RACT Instance RACT1 is running on node ract1 Instance RACT2 is running on node ract2 PRKO-2015 : Error in checking condition of instance on node: ract3 [oracle@ract1 ~]$ srvctl start instance -d ract -i ract3 CRS-0210: Could not find resource ora.RACT.ract3.inst. --> Names are case sensitve - Restore you OCR to fix ORA-210
Major commands found in rootaddnode.sh and root.sh
rootaddnode.sh on ract1 runs following major commands # /u01/app/oracle/product/crs/bin/clscfg -add -nn ract3,3 -pn ract3int,3 -hn ract3,3 # /u01/app/oracle/product/crs/bin/srvctl add nodeapps -n ract3 -A ract3vip.example.com/255.255.255.0/eth1 -o /u01/app/oracle/product/crs # /bin/su -l oracle -c "/u01/app/oracle/product/crs/bin/cluutil -sourcefile /etc/oracle/ocr.loc -destfile /u01/app/oracle/product/crs/srvm/admin/ocr.loc -nodelist ract3" --> only first 2 commands are need to recreate nodeapps root.sh on ract3 runs following major commands # /u01/app/oracle/product/crs/bin/crsctl create scr oracle # /u01/app/oracle/product/crs/bin/crsctl check install -wait 600 Running vipca(silent) for configuring nodeapps # /u01/app/oracle/product/crs/bin/vipca -silent -nodelist ract1,ract2 -nodevips ract1/ract1vip/255.255.255.0/eth1,ract2/ract2vip/255.255.255.0/eth1 --> For recreating nodeaapps none of these commands are needed
Syntax for clscfg and cluutil utilities
[root@ract1 install]# clscfg -h clscfg: EXISTING configuration version 3 detected. clscfg: version 3 is 10G Release 2. clscfg -- Oracle cluster configuration tool This tool is typically invoked as part of the Oracle Cluster Ready Services install process. It configures cluster topology and other settings. Use -help for information on any of these modes. Use one of the following modes of operation. -install - creates a new configuration -upgrade - upgrades an existing configuration -downgrade - downgrades an existing configuration -add - adds a node to the configuration -delete - deletes a node from the configuration -local - creates a special single-node configuration for ASM -concepts - brief listing of terminology used in the other modes -trace - may be used in conjunction with any mode above for tracing WARNING: Using this tool may corrupt your cluster configuration. Do not use unless you positively know what you are doing. [root@ract1 install]# clscfg -add -h clscfg: EXISTING configuration version 3 detected. clscfg: version 3 is 10G Release 2. clscfg -add -nn nameA,numA,nameB,numB,... -pn privA,numA,privB,numB,... [-hn hostA,numA,hostB,numB,...] [-t p1,p2,p3,p4] -nn specifies nodenames in the same fashion as -nn in -install mode -pn specifies private interconnect names as -pn in -install mode -hn specifies hostnames in the same fashion as -hn in -install mode -t specifies port numbers to be used by CRS daemons on the new node(s) default ports: 49895,49896,49897,49898 WARNING: Using this tool may corrupt your cluster configuration. Do not use unless you positively know what you are doing. [root@ract1 install]# /u01/app/oracle/product/crs/bin/cluutil -h usage: java ClusterUtil -sourcefile <source file location> -destFile <destination file location> -nodelist node1[,node2,..]