Table of Contents
Cleanup OCR
Backup your OCR [root@ract1 ~]# $ORA_CRS_HOME/bin/ocrconfig -export ocr_before_node_removal.exp Verify and drop the failing instance ract3 --> Note ract3 is not available anymore Check current CRS status Name Type Target State Host ------------------------------------------------------------ ora....T3.inst application ONLINE OFFLINE ora....SM3.asm application ONLINE OFFLINE ora....T3.lsnr application ONLINE OFFLINE ora.ract3.gsd application ONLINE OFFLINE ora.ract3.ons application ONLINE OFFLINE ora.ract3.vip application ONLINE ONLINE ract1 --> After crash only ora.ract3.vip failed over to ract1 - other resources are OFFLINE Stop all resources running on ract3 [root@ract1 ~]# crs_stat -t | egrep 'Name|T3|t3|M3' Name Type Target State Host ora....T3.inst application ONLINE OFFLINE ora....SM3.asm application ONLINE OFFLINE ora....T3.lsnr application ONLINE OFFLINE ora.ract3.gsd application ONLINE OFFLINE ora.ract3.ons application ONLINE OFFLINE ora.ract3.vip application ONLINE ONLINE ract1 [root@ract1 ~]# crs_stop ora.RACT.RACT3.inst Target set to OFFLINE for `ora.RACT.RACT3.inst` [root@ract1 ~]# crs_stop ora.ract3.ASM3.asm Target set to OFFLINE for `ora.ract3.ASM3.asm [root@ract1 ~]# crs_stop ora.ract3.LISTENER_RACT3.lsnr Target set to OFFLINE for `ora.ract3.LISTENER_RACT3.lsnr` [root@ract1 ~]# crs_stop ora.ract3.gsd Target set to OFFLINE for `ora.ract3.gsd` [root@ract1 ~]# crs_stop ora.ract3.ons Target set to OFFLINE for `ora.ract3.ons` [root@ract1 ~]# crs_stop ora.ract3.vip Attempting to stop `ora.ract3.vip` on member `ract1` Stop of `ora.ract3.vip` on member `ract1` succeeded. [root@ract1 ~]# crs_stat -t | egrep 'Name|T3|t3|M3' Name Type Target State Host ora....T3.inst application OFFLINE OFFLINE ora....SM3.asm application OFFLINE OFFLINE ora....T3.lsnr application OFFLINE OFFLINE ora.ract3.gsd application OFFLINE OFFLINE ora.ract3.ons application OFFLINE OFFLINE ora.ract3.vip application OFFLINE OFFLINE Remove RDBMS instance [oracle@ract1 ~]$ srvctl remove instance -d ract -i RACT3 Remove instance RACT3 from the database ract? (y/[n]) y Remove ASM instance $ srvctl remove asm -f -n ract3 Remove RAC listener The only way to remove the listener resources is to use the command 'crs_unregister', please use this command only in this particular scenario: [root@ract1 ~]# crs_stat | grep lsnr NAME=ora.ract1.LISTENER_RACT1.lsnr NAME=ora.ract2.LISTENER_RACT2.lsnr NAME=ora.ract3.LISTENER_RACT3.lsnr [root@ract1 ~]# crs_unregister ora.ract3.LISTENER_RACT3.lsnr Remove remaining nodeapps [oracle@ract1 ~]$ crs_stat -t | egrep 'Name|T3|t3|M3' Name Type Target State Host ora.ract3.gsd application OFFLINE OFFLINE ora.ract3.ons application OFFLINE OFFLINE ora.ract3.vip application OFFLINE OFFLINE [root@ract1 ~]# srvctl remove nodeapps -n ract3 Please confirm that you intend to remove the node-level applications on node ract3 (y/[n]) y Verify that all resource linked to ract3 are removed [root@ract1 ~]# crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....T1.inst application ONLINE ONLINE ract1 ora....T2.inst application ONLINE ONLINE ract2 ora.RACT.db application ONLINE ONLINE ract2 ora....SM1.asm application ONLINE ONLINE ract1 ora....T1.lsnr application ONLINE ONLINE ract1 ora.ract1.gsd application ONLINE ONLINE ract1 ora.ract1.ons application ONLINE ONLINE ract1 ora.ract1.vip application ONLINE ONLINE ract1 ora....SM2.asm application ONLINE ONLINE ract2 ora....T2.lsnr application ONLINE ONLINE ract2 ora.ract2.gsd application ONLINE ONLINE ract2 ora.ract2.ons application ONLINE ONLINE ract2 ora.ract2.vip application ONLINE ONLINE ract2 Remove node by running ./rootdeletenode.sh [root@ract1 ~]# olsnodes -n ract1 1 ract2 2 ract3 3 [root@ract1 ~]# cd $ORA_CRS_HOME/install [root@ract1 install]# ./rootdeletenode.sh ract3,3 CRS-0210: Could not find resource 'ora.ract3.LISTENER_RACT3.lsnr'. CRS-0210: Could not find resource 'ora.ract3.ons'. CRS-0210: Could not find resource 'ora.ract3.vip'. CRS-0210: Could not find resource 'ora.ract3.gsd'. CRS-0210: Could not find resource ora.ract3.vip. CRS nodeapps are deleted successfully clscfg: EXISTING configuration version 3 detected. clscfg: version 3 is 10G Release 2. Successfully deleted 14 values from OCR. Key SYSTEM.css.interfaces.noderact3 marked for deletion is not there. Ignoring. Successfully deleted 5 keys from OCR. Node deletion operation successful. 'ract3,3' deleted successfully [root@ract1 install]# olsnodes -n ract1 1 ract2 2 Cleanup CRS and Rdbms Inventory $ cd $ORA_CRS_HOME/oui/bin $ ./runInstaller Installed products OraCrs10g_home Cluster Nodes ract1 ract2 ract3 OraDb10g_home1 Cluster Nodes ract1 ract2 ract3 [oracle@ract1 bin]$ cd $ORA_CRS_HOME/oui/bin [oracle@ract1 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=$ORA_CRS_HOME "CLUSTER_NODES={ract1,ract2}" CRS=true Starting Oracle Universal Installer... No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed. 'UpdateNodeList' was successful. [oracle@ract1 bin]$ cd $ORACLE_HOME/oui/bin [oracle@ract1 bin]$ $ORACLE_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={ract1,ract2}" Starting Oracle Universal Installer... No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed. 'UpdateNodeList' was successful. Verify Cluster Nodes $ ./runInstaller Installed products OraCrs10g_home Cluster Nodes ract1 ract2 OraDb10g_home1 Cluster Nodes ract1 ract2
Cleanup crashed Node ( ract3 ) after reboot
Be carefull when deleting files - depending on your setup you may damage a 2.nd database install # To remove RDBMS and CRS software run ( both crs and rdbms are installed under /u01/app/oracle/ ) rm -rf /u01/app/oracle/* <-- dangerous !! # rm /etc/oracle/* rm -f /etc/init.d/init.cssd rm -f /etc/init.d/init.crs rm -f /etc/init.d/init.crsd rm -f /etc/init.d/init.evmd rm -f /etc/rc2.d/K96init.crs rm -f /etc/rc2.d/S96init.crs rm -f /etc/rc3.d/K96init.crs rm -f /etc/rc3.d/S96init.crs rm -f /etc/rc5.d/K96init.crs rm -f /etc/rc5.d/S96init.crs rm -Rf /etc/oracle/scls_scr rm -f /etc/inittab.crs cp /etc/inittab.orig /etc/inittab --> reboot cleanup node again and check that no cluster process has survived the cleanup ! Verify that no CRS process survived the cleanup # ps -elf |grep d.bin
Error: srvctl remove instance fails IF INSTANCE: TARGET=ONLINE AND STATE=OFFLINE
[root@ract1 ~]# srvctl remove instance -d ract -i RACT3 root@ract1 ~]# crs_stat -t | egrep 'Name|t3|T3' Name Type Target State Host ora....T3.inst application ONLINE OFFLINE --> Target for Instance RACT3 still ONLINE !! See BUG 4423294 : DELETE INSTANCE THROUGH SRVM FAILS IF INSTANCE: TARGET=ONLINE AND STATE=OFFLINE crs_stop should fix ! [root@ract1 ~]# crs_stop ora.RACT.RACT3.inst Target set to OFFLINE for `ora.RACT.RACT3.inst` [root@ract1 ~]# crs_stat -t | egrep 'Name|t3|T3' Name Type Target State Host ora....T3.inst application OFFLINE OFFLINE Now srvctl remove instance should work [root@ract1 ~]# srvctl remove instance -d ract -i RACT3
Reference
- Steps to Remove Node from Cluster When the Node Crashes Due to OS/Hardware Failure and cannot boot up (Doc ID 466975.1)
- http://www.idevelopment.info/data/Oracle/DBA_tips/Oracle10gRAC/CLUSTER_23.shtml
I’m extremely impressed along with your writing abilities and also with the
format for your blog. Is that this a paid theme or did
you modify it yourself? Either way stay up the nice high quality writing, it is uncommon to
peer a nice blog like this one these days..
For an incredible solution please click the link to this page
:: http://cnn.com (Tabitha)
Hi Tabitha,
thx for your nice reply.
I have used and modified default wordpress theme : Twenty Twelve
br
Helmut