Table of Contents
Overview
- Clusterware will have an active version and software version. The active version in my case is the 10.2.0,1 base release, and until the patch hasn’t completed (i.e. root.sh executed on last node) on all nodes that will remain the original. When root102.sh completes on the last node, the active version will match the software version
- CRS is upgraded running a rolling upgrade
Check current CRS and RDBMS status
[oracle@ract2 ~]$ crsctl query crs activeversion CRS active version on the cluster is [10.2.0.1.0] [oracle@ract2 ~]$ crsctl query crs softwareversion CRS software version on node [ract2] is [10.2.0.1.0] [oracle@ract2 ~]$ crsctl check crs CSS appears healthy CRS appears healthy EVM appears healthy Before applying the 10.2.0.5 patch set to the Clusterware home, run the following command on all the cluster nodes # chmod -R +w $ORA_CRS_HOME/inventory/Templates/* Check for invalid objects SQL> select object_name,status from dba_objects where status='INVALID'; no rows selected Check for timezone version SQL> SELECT version FROM v$timezone_file; VERSION ---------- 2 As version != 4 you need to follow Actions for the DSTv4 update in the 10.2.0.5 patchset (Doc ID 1086400.1)
Verify RAC status with cluvfy ( always use newest cluvfy version and version switch like -r 10gR2 )
oracle@ract1 cluvfy12]$ ./bin/cluvfy -version 12.1.0.1.0 Build 112713x8664 [oracle@ract1 cluvfy12]$ ./bin/cluvfy stage -pre crsinst -r 10gR2 -n ract1,ract2,ract3
Prepare Clusterware for a Rolling Upgrade
Stop all Processes in node ract1 [oracle@ract1 ~]$ emctl stop dbconsole [oracle@ract1 ~]$ srvctl stop instance -d RACT -i RACT1 -o immediate [oracle@ract1 ~]$ lsnrctl stop LISTENER_RACT1 [oracle@ract1 ~]$ srvctl stop asm -n ract1 [root@ract1 admin]# srvctl stop nodeapps -n ract1 If database is running on the same node relocate DB [root@ract2 network-scripts]# crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.RACT.db application ONLINE ONLINE ract1 [root@ract2 network-scripts]# crs_relocate ora.RACT.db Attempting to stop `ora.RACT.db` on member `ract1` Stop of `ora.RACT.db` on member `ract1` succeeded. Attempting to start `ora.RACT.db` on member `ract2` Start of `ora.RACT.db` on member `ract2` succeeded.
Backup your data
• Verify OCR and Backup OCR # ocrcheck Status of Oracle Cluster Registry is as follows : Version : 2 Total space (kbytes) : 1044104 Used space (kbytes) : 5380 Available space (kbytes) : 1038724 ID : 1065530224 Device/File Name : /dev/raw/raw1 Device/File integrity check succeeded Device/File Name : /dev/raw/raw2 Device/File integrity check succeeded Cluster registry integrity check succeeded Export OCR # ocrconfig -export /restore/crs/ocr_backup.ocr -s online • Voting Disk Backup # crsctl query css votedisk 0. 0 /dev/raw/raw3 1. 0 /dev/raw/raw4 2. 0 /dev/raw/raw5 # dd if=/dev/raw/raw3 of=/restore/crs/votedisk1.dmp # dd if=/dev/raw/raw4 of=/restore/crs/votedisk2.dmp # dd if=/dev/raw/raw5 of=/restore/crs/votedisk3.dmp Backup CLUSTER HOME, ORACLE HOME and OraInventory on all nodes • Cluster Home and Oracle Homes [ALL Nodes] # tar -czvf /restore/ORA_CRS_HOME.tar.gz /u01/app/oracle/product/crs # tar -czvf /restore/ORACLE_HOME.tar.gz /u01/app/oracle/product/10.2/rac_db1 Locate OraInventory # cat /etc/oraInst.loc inventory_loc=/home/oracle/oraInventory inst_group=oinstall # tar -czvf /restore/ORA_INVENTORY.tar.gz /home/oracle/oraInventory -> Note if your ASM is not installed with the RDBMS you need to backup your ASM ORACLE_HOME too • Backup /etc/oracle # tar -czvf /restore/ETC_ORACLE.tar.gz /etc/oracle and save: /etc/init.d/init.cssd /etc/init.d/init.crs /etc/init.d/init.crsd /etc/init.d/init.evmd /etc/inittab For more details check: 10gR2/11gR1: What Files To Backup In Oracle Clusterware (CRS) Installation (Doc ID 754369.1) --> In any case you should have a full OS backup before applying the patch Verify CRS a last time ot@ract1 ~]# crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....T1.inst application OFFLINE OFFLINE ora....T2.inst application ONLINE ONLINE ract2 ora....T3.inst application ONLINE ONLINE ract3 ora.RACT.db application ONLINE ONLINE ract2 ora....SM1.asm application OFFLINE OFFLINE ora....T1.lsnr application OFFLINE OFFLINE ora.ract1.gsd application OFFLINE OFFLINE ora.ract1.ons application OFFLINE OFFLINE ora.ract1.vip application OFFLINE OFFLINE ora....SM2.asm application ONLINE ONLINE ract2 ora....T2.lsnr application ONLINE ONLINE ract2 ora.ract2.gsd application ONLINE ONLINE ract2 ora.ract2.ons application ONLINE ONLINE ract2 ora.ract2.vip application ONLINE ONLINE ract2 ora....SM3.asm application ONLINE ONLINE ract3 ora....T3.lsnr application ONLINE ONLINE ract3 ora.ract3.gsd application ONLINE ONLINE ract3 ora.ract3.ons application ONLINE ONLINE ract3 ora.ract3.vip application ONLINE ONLINE ract3 --> Now its time to backup your database using RMAN
Install 10.2.0.5 clusterware software
[oracle@ract1 Disk1]$ ./runInstaller --> Select your CRS Oracle_HOME OUI prompts: The installer has detected that your Cluster Ready Services (CRS) installation is distributed across the following nodes: ract1 ract2 ract3 Because the software consists of local identical copies distributed across each of the nodes in the cluster, it is possible to patch your CRS installation in a rolling manner, one node at a time. To complete the installation of this patchset, you must perform the following tasks on each node: 1. Log in as the root user. 2. As the root user, perform the following tasks: a. Shutdown the CRS daemons by issuing the following command: /u01/app/oracle/product/crs/bin/crsctl stop crs b. Run the shell script located at: /u01/app/oracle/product/crs/install/root102.sh This script will automatically start the CRS daemons on the patched node upon completion. 3. After completing this procedure, proceed to the next node and repeat. Start with node ract1 [root@ract1 ~]# /u01/app/oracle/product/crs/bin/crsctl stop crs Stopping resources. Successfully stopped CRS resources Stopping CSSD. Shutting down CSS daemon. Shutdown request successfully issued. [root@ract1 ~]# /u01/app/oracle/product/crs/install/root102.sh Creating pre-patch directory for saving pre-patch clusterware files Completed patching clusterware files to /u01/app/oracle/product/crs Relinking some shared libraries. Relinking of patched files is complete. WARNING: directory '/u01/app/oracle/product' is not owned by root WARNING: directory '/u01/app/oracle' is not owned by root WARNING: directory '/u01/app' is not owned by root WARNING: directory '/u01' is not owned by root Preparing to recopy patched init and RC scripts. Recopying init and RC scripts. Startup will be queued to init within 30 seconds. Starting up the CRS daemons. Waiting for the patched CRS daemons to start. This may take a while on some systems. . 10205 patch successfully applied. clscfg: EXISTING configuration version 3 detected. clscfg: version 3 is 10G Release 2. Successfully deleted 1 values from OCR. Successfully deleted 1 keys from OCR. Successfully accumulated necessary OCR keys. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node <nodenumber>: <nodename> <private interconnect name> <hostname> node 1: ract1 ract1int ract1 Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. clscfg -upgrade completed successfully Creating '/u01/app/oracle/product/crs/install/paramfile.crs' with data used for CRS configuration Setting CRS configuration values in /u01/app/oracle/product/crs/install/paramfile.crs Verify CRS version on ract1 [root@ract1 ~]# crsctl query crs activeversion CRS active version on the cluster is [10.2.0.1.0] [root@ract1 ~]# crsctl query crs softwareversion CRS software version on node [ract1] is [10.2.0.5.0] Run root102.sh on ract2 [root@ract2 ~]# /u01/app/oracle/product/crs/bin/crsctl stop crs [root@ract2 ~]# /u01/app/oracle/product/crs/install/root102.sh Check CRS status after running root102.sh on ract2 [root@ract1 ~]# crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....T1.inst application ONLINE ONLINE ract1 ora....T2.inst application ONLINE ONLINE ract2 ora....T3.inst application ONLINE ONLINE ract3 ora.RACT.db application ONLINE ONLINE ract3 ora....SM1.asm application ONLINE ONLINE ract1 ora....T1.lsnr application ONLINE ONLINE ract1 ora.ract1.gsd application ONLINE ONLINE ract1 ora.ract1.ons application ONLINE ONLINE ract1 ora.ract1.vip application ONLINE ONLINE ract1 ora....SM2.asm application ONLINE ONLINE ract2 ora....T2.lsnr application ONLINE ONLINE ract2 ora.ract2.gsd application ONLINE ONLINE ract2 ora.ract2.ons application ONLINE ONLINE ract2 ora.ract2.vip application ONLINE ONLINE ract2 ora....SM3.asm application ONLINE ONLINE ract3 ora....T3.lsnr application ONLINE ONLINE ract3 ora.ract3.gsd application ONLINE ONLINE ract3 ora.ract3.ons application ONLINE ONLINE ract3 ora.ract3.vip application ONLINE ONLINE ract3 --> Again after running root102.sh all nodes are active .... Now running root102.sh on the last node [root@ract3 ~]# /u01/app/oracle/product/crs/bin/crsctl stop crs [root@ract3 ~]# /u01/app/oracle/product/crs/install/root102.sh [root@ract3 ~]# crsctl query crs activeversion CRS active version on the cluster is [10.2.0.5.0] [root@ract3 ~]# crsctl query crs softwareversion CRS software version on node [ract3] is [10.2.0.5.0] --> CRS activersion is now 10.2.0.5.0 as root102.sh was run sucessfully on our last node!
Install 10.2.0.5 RDBMS software
Stopping All Processes for an Oracle RAC Installation # srvctl stop database -d RACT # srvctl stop asm -n ract1 # srvctl stop asm -n ract2 # srvctl stop asm -n ract3 # srvctl stop listener -n ract1 # srvctl stop listener -n ract2 # srvctl stop listener -n ract3 Check CRS status [root@ract1 ~]# crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....T1.inst application OFFLINE OFFLINE ora....T2.inst application OFFLINE OFFLINE ora....T3.inst application ONLINE OFFLINE ora.RACT.db application OFFLINE OFFLINE ora....SM1.asm application OFFLINE OFFLINE ora....T1.lsnr application OFFLINE OFFLINE ora.ract1.gsd application ONLINE ONLINE ract1 ora.ract1.ons application ONLINE ONLINE ract1 ora.ract1.vip application ONLINE ONLINE ract1 ora....SM2.asm application OFFLINE OFFLINE ora....T2.lsnr application OFFLINE OFFLINE ora.ract2.gsd application ONLINE ONLINE ract2 ora.ract2.ons application ONLINE ONLINE ract2 ora.ract2.vip application ONLINE ONLINE ract2 ora....SM3.asm application OFFLINE OFFLINE ora....T3.lsnr application OFFLINE OFFLINE ora.ract3.gsd application ONLINE ONLINE ract3 ora.ract3.ons application ONLINE ONLINE ract3 ora.ract3.vip application ONLINE ONLINE ract3 Run OUI [oracle@ract1 Disk1]$ ./runInstaller Select : OraDb10g_home1 -> /u01/app/oracle/product/10.2/rac_db1
Upgrade Database to 10.2.0.5
[oracle@ract1 ~]$ sqlplus / as sysdba SQL> STARTUP NOMOUNT SQL> ALTER SYSTEM SET CLUSTER_DATABASE=FALSE SCOPE=spfile; SQL> SHUTDOWN SQL> STARTUP UPGRADE SQL> SPOOL patch.log SQL> @?/rdbms/admin/catupgrd.sql SQL> SPOOL OFF Check patch.log for errors: [root@ract1 Desktop]# grep ORA- -/patch.log DOC> The following statement will cause an "ORA-01722: invalid number" DOC> The following statement will cause an "ORA-01722: invalid number" DOC> The following statement will cause an "ORA-01722: invalid number" DOC> The following statements will cause an "ORA-01722: invalid number" DOC> The following statement will cause an "ORA-01722: invalid number" DOC> The following PL/SQL block will cause an ORA-20000 error and DOC> The following PL/SQL block will cause an ORA-20000 error and ORA-29558: JAccelerator (NCOMP) not installed. Refer to Install Guide for ORA-06512: at "SYS.DBMS_JAVA", line 236 SQL> !grep ORA- patch.log ORA-00923: FROM keyword not found where expected If you get a similiar output like above output you are save. If not rerun the catupgrd.sql script after correcting any problems. Restart the database: SQL> SHUTDOWN IMMEDIATE SQL> STARTUP Run the utlrp.sql script to recompile all invalid PL/SQL packages SQL> @?/rdbms/admin/utlrp.sql Check JAVA/PLSQL package status : SQL> select object_name,status from dba_objects where status='INVALID'; no rows selected SQL> select COMP_NAME,VERSION,STATUS from dba_registry; COMP_NAME VERSION STATUS ----------------------------------- ------------------------------ ----------- Oracle Enterprise Manager 10.2.0.5.0 VALID Spatial 10.2.0.5.0 VALID Oracle interMedia 10.2.0.5.0 VALID OLAP Catalog 10.2.0.5.0 VALID Oracle XML Database 10.2.0.5.0 VALID Oracle Text 10.2.0.5.0 VALID Oracle Expression Filter 10.2.0.5.0 VALID Oracle Rule Manager 10.2.0.5.0 VALID Oracle Workspace Manager 10.2.0.5.0 VALID Oracle Data Mining 10.2.0.5.0 VALID Oracle Database Catalog Views 10.2.0.5.0 VALID Oracle Database Packages and Types 10.2.0.5.0 VALID JServer JAVA Virtual Machine 10.2.0.5.0 VALID Oracle XDK 10.2.0.5.0 VALID Oracle Database Java Packages 10.2.0.5.0 VALID OLAP Analytic Workspace 10.2.0.5.0 VALID Oracle OLAP API 10.2.0.5.0 VALID Oracle Real Application Clusters 10.2.0.5.0 VALID Set the CLUSTER_DATABASE initialization parameter to TRUE: SQL> ALTER SYSTEM SET CLUSTER_DATABASE=TRUE SCOPE=spfile; Restart the database: SQL> SHUTDOWN IMMEDIATE SQL> STARTUP Start listener , ASM and RDBMS instances as needed [oracle@ract1 ~]$ srvctl start listener -n ract2 [oracle@ract1 ~]$ srvctl start asm -n ract2 [oracle@ract1 ~]$ srvctl start instance -d RACT -i RACT2 [oracle@ract1 ~]$ srvctl start listener -n ract3 [oracle@ract1 ~]$ srvctl start asm -n ract3 [oracle@ract1 ~]$ srvctl start instance -d RACT -i RACT3 Check CRS status after RDBMS upgrade [oracle@ract1 ~]$ crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....T1.inst application ONLINE ONLINE ract1 ora....T2.inst application ONLINE ONLINE ract2 ora....T3.inst application ONLINE ONLINE ract3 ora.RACT.db application ONLINE ONLINE ract2 ora....SM1.asm application ONLINE ONLINE ract1 ora....T1.lsnr application ONLINE ONLINE ract1 ora.ract1.gsd application ONLINE ONLINE ract1 ora.ract1.ons application ONLINE ONLINE ract1 ora.ract1.vip application ONLINE ONLINE ract1 ora....SM2.asm application ONLINE ONLINE ract2 ora....T2.lsnr application ONLINE ONLINE ract2 ora.ract2.gsd application ONLINE ONLINE ract2 ora.ract2.ons application ONLINE ONLINE ract2 ora.ract2.vip application ONLINE ONLINE ract2 ora....SM3.asm application ONLINE ONLINE ract3 ora....T3.lsnr application ONLINE ONLINE ract3 ora.ract3.gsd application ONLINE ONLINE ract3 ora.ract3.ons application ONLINE ONLINE ract3 ora.ract3.vip application ONLINE ONLINE ract3 SQL> select inst_id, instance_name, host_name, status, to_char(startup_time, 'DD-MON-YYYY HH24:MI:SS') startup_time, version from gv$instance order by inst_id; INST_ID INSTANCE_NAME HOST_NAME STATUS STARTUP_TIME VERSION ---------- ---------------- -------------------- ------------ -------------------- -------------------- 1 RACT1 ract1.example.com OPEN 15-APR-2014 19:35:35 10.2.0.5.0 2 RACT2 ract2.example.com OPEN 15-APR-2014 19:36:36 10.2.0.5.0 5 RACT3 ract3.example.com OPEN 15-APR-2014 19:38:12 10.2.0.5.0
Reference
- 10gR2/11gR1: What Files To Backup In Oracle Clusterware (CRS) Installation (Doc ID 754369.1)
Dear Helmut,
Very nicely explained article.
I have one doubt though. in the section “Install 10.2.0.5 clusterware software”, have you performed rolling upgrade ?
I am not sure if it can be called rolling upgrade as you are installing patchset in all the nodes in one go.
Please clarify.
Thanks.