Table of Contents
Overiew
- rootupgrade.sh (and root.sh for that matter) are restartable since 11.2.0.2 at least
- A great deal of logging is available in $GRID_HOME/cfgtoollogs/crsconfig/rootcrs_hostname.log
- Checkpoint file : /u01/app/grid/Clusterware/ckptGridHA_grac31.xml
- Log File : $GRID_HOME/cfgtoollogs/crsconfig/rootcrs_grac31.log
Note : This article simulates and fix a failed upgrade by having account qosadmim deactivated –
( see Note 1577072.1 for details )
Verify that we are ready to UPGRADE with orachk
[oracle@grac31 ORACHECK225]$ ./orachk -u -o pre Enter upgrade target version (valid versions are 11.2.0.3.0, 11.2.0.4.0, 12.1.0.1.0, 12.1.0.2.0):- 11.2.0.4.0 CRS stack is running and CRS_HOME is not set. Do you want to set CRS_HOME to /u01/app/11203/grid?[y/n][y] Checking ssh user equivalency settings on all nodes in cluster Data collections completed. Checking best practices on grac31. -------------------------------------------------------------------------------------- INFO => Prior to Upgrade Verify NO Materialized Views Being Refreshed (Automatically or Manually) for on11203 INFO => Special Considerations for SCAN and HAIP of 11.2.0.2 and above Grid Infrastructure INFO => Minimizing Planned Downtime for Upgrades and Patching INFO => Upgrade Related References INFO => Lifetime Support Policy and Product Certification INFO INFO => Capture performance baseline, backup important configuration files and batch, cron, DBMS_JOBS and DBMS_SCHEDULER jobs INFO => Known Issues Integrating pre-11gR2 Databases with Grid Infrastructure INFO => Be Aware of New Instant Client for Simplified Deployment INFO => Oracle E-Business Suite interoperability and migration resources INFO => Oracle Software Download Sites WARNING => One or More Column Names in ALL_TAB_COLUMNS table are Reserved Words for on11203 WARNING => Some Users Needing Network ACLs for Oracle Utility Packages Found for on11203 WARNING => Package unixODBC-2.2.14-11.el6-i686 is recommended but NOT installed WARNING => Package unixODBC-devel-2.2.14-11.el6-i686 is recommended but NOT installed INFO => Steps to execute rootupgrade.sh INFO => Save "Oracle Enterprise Manager Database Control" files and data with the emdwgrd Utility before upgrading database. INFO => Information about ASM process parameter when its not set to default value INFO => Information about Grid Infrastructure software installation directory WARNING => Package unixODBC-devel-2.2.14-11.el6-x86_64 is recommended but NOT installed --------------------------------------------------------------------------------- CLUSTERWIDE CHECKS --------------------------------------------------------------------------------- Detailed report (html) - /home/oracle/ORACHECK225/orachk_grac31_on11203_092214_152522/orachk_grac31_on11203_092214_152522.html UPLOAD(if required) - /home/oracle/ORACHECK225/orachk_grac31_on11203_092214_152522.zip Check that qosadmin account is disabled which aborts/stops the upgrade process ( see Doc ID 1577072.1 ) [root@grac31 Desktop]# qosctl qosadmin -listusers AbstractLoginModule password: <-- oracle112 Sep 22, 2014 7:05:48 PM oracle.security.jazn.spi.xml.XMLRealmUser authenticate INFO: User(jazn.com/qosadmin) is deactivated. AUTH FAILURE. Sep 22, 2014 7:05:48 PM oracle.security.jazn.login.module.RealmLoginModule authenticate SEVERE: [RealmLoginModule] authentication failed Note : The above error will abort the rootupgrade.sh script later on
Prepare and Install new 11.2.0.4 GRID_HOME
[grid@grac31 ~]$ env | grep -i ORAC ORACLE_BASE=/u01/app/grid ORACLE_HOME=/u01/app/11204_upg/grid [grid@grac31 ~]$ ls -ld /u01/app/oracle drwxrwxr-x. 7 oracle oinstall 4096 Sep 19 17:37 /u01/app/oracle Create the GRID_HOME directory [root@grac31 Desktop]# mkdir -p /u01/app/11204_upg/grid [root@grac31 Desktop]# chown grid:oinstall /u01/app/11204_upg/grid [root@grac31 Desktop]# chmod 775 /u01/app/11204_upg/grid [root@grac31 Desktop]# ls -ld /u01/app/11204_upg/grid drwxrwxr-x. 2 grid oinstall 4096 Sep 22 19:12 /u01/app/11204_upg/grid Invoke Installer [grid@grac31 grid]$ ./runInstaller --> Upgrade GRID Infrastructure or Oracle ASM [root@grac31 Desktop]# /u01/app/11204_upg/grid/rootupgrade.sh .. Database Configuration Assistant when a database is created Finished running generic part of root script. Now product-specific root actions will be performed. Using configuration parameter file: /u01/app/11204_upg/grid/crs/install/crsconfig_params Creating trace directory Installing Trace File Analyzer ASM upgrade has started on first node. ASM upgrade has started on first node. CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'grac31' CRS-2673: Attempting to stop 'ora.crsd' on 'grac31' CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'grac31' CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'grac31' .. CRS-2677: Stop of 'ora.drivers.acfs' on 'grac31' succeeded CRS-2677: Stop of 'ora.gipcd' on 'grac31' succeeded CRS-2673: Attempting to stop 'ora.gpnpd' on 'grac31' CRS-2677: Stop of 'ora.gpnpd' on 'grac31' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'grac31' has completed CRS-4133: Oracle High Availability Services has been stopped. OLR initialization - successful Replacing Clusterware entries in upstart clscfg: EXISTING configuration version 5 detected. clscfg: version 5 is 11g Release 2. Successfully accumulated necessary OCR keys. Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. Started to upgrade the Oracle Clusterware. This operation may take a few minutes. Started to upgrade the CSS. Started to upgrade the CRS. The CRS was successfully upgraded. Successfully upgraded the Oracle Clusterware. Oracle Clusterware operating version was successfully set to 11.2.0.4.0 ASM upgrade has finished on last node. Failed to perform J2EE (OC4J) Container Resource upgrade at /u01/app/11204_upg/grid/crs/install/crsconfig_lib.pm line 9323. /u01/app/11204_upg/grid/perl/bin/perl -I/u01/app/11204_upg/grid/perl/lib -I/u01/app/11204_upg/grid/crs/install /u01/app/11204_upg/grid/crs/install/rootcrs.pl execution failed Only ora.oc4j resource is OFFLINE [root@grac31 Desktop]# crs NAME TARGET STATE SERVER STATE_DETAILS ------------------------- ---------- ---------- ------------ ------------------ ora.oc4j OFFLINE OFFLINE
Fix the rootupgrade problem by following Note 1577072.1
For Case I - Upgrading Grid Infrastructure from 11.2.0.3 GI PSU 6 or higher to 11.2.0.4 Please perform the following workaround as grid user either before rootupgrade.sh is issued or after rootupgrade.sh fails with above error: 1. Activate the qosadmin user from 11.2.0.3 GRID_HOME as grid user: grid@grac31 grid]$ env | grep GRID GRID_HOME=/u01/app/11203/grid [grid@grac31 grid]$ env | grep PATH PATH=/u01/app/11203/grid/bin:.:/u01/app/11203/grid/bin: Change qosadmin password to neworacle112 [grid@grac31 grid]$ qosctl qosadmin -setpasswd qosadmin oracle112 neworacle112 AbstractLoginModule password: <<< enter oracle112 Please re-enter the password entered on the command line above Enter Password: <<< enter oracle112 [grid@grac31 grid]$ qosctl qosadmin -listusers AbstractLoginModule password: <<< enter neworacle112 oc4jadmin JtaAdmin qosadmin The next steps are not needed but I want to have back ther original password in my test env Change qosadmin password back to oracle112 ( here we use the default password again ) [grid@grac31 grid]$ qosctl qosadmin -setpasswd qosadmin neworacle112 oracle112 AbstractLoginModule password: <<< enter neworacle112 Please re-enter the password entered on the command line above Enter Password: <<< enter neworacle112 [grid@grac31 grid]$ qosctl qosadmin -listusers AbstractLoginModule password: <<< enter oracle112 oc4jadmin JtaAdmin qosadmi Rerun rootupgrade.sh from updated 11.2.0.4 CRS Home /u01/app/11204_upg/grid [root@grac31 grid]# pwd /u01/app/11204_upg/grid [root@grac31 grid]# ./rootupgrade.sh .. Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root script. Now product-specific root actions will be performed. Using configuration parameter file: /u01/app/11204_upg/grid/crs/install/crsconfig_params Installing Trace File Analyzer clscfg: EXISTING configuration version 5 detected. clscfg: version 5 is 11g Release 2. Successfully accumulated necessary OCR keys. Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. CRS-1115: Oracle Clusterware has already been upgraded. ASM upgrade has finished on last node. Configure Oracle Grid Infrastructure for a Cluster ... succeeded Related rootcrs_grac31.log entries: > Oracle Clusterware version on node [grac31] is [11.2.0.4.0] >End Command output 2014-09-23 08:38:29: Version String passed is: Oracle Clusterware version on node [grac31] is [11.2.0.4.0] 2014-09-23 08:38:29: Version Info returned is : 11.2.0.4.0 2014-09-23 08:38:29: Got CRS softwareversion for grac31: 11.2.0.4.0 2014-09-23 08:38:29: The software version on grac31 is 11.2.0.4.0 2014-09-23 08:38:29: Invoking "/u01/app/11204_upg/grid/bin/srvctl upgrade model -s 11.2.0.3.0 -d 11.2.0.4.0 -p first" 2014-09-23 08:38:29: trace file=/u01/app/11204_upg/grid/cfgtoollogs/crsconfig/srvmcfg0.log 2014-09-23 08:38:29: Executing /u01/app/11204_upg/grid/bin/srvctl upgrade model -s 11.2.0.3.0 -d 11.2.0.4.0 -p first 2014-09-23 08:38:29: Executing cmd: /u01/app/11204_upg/grid/bin/srvctl upgrade model -s 11.2.0.3.0 -d 11.2.0.4.0 -p first 2014-09-23 08:38:31: srvctl upgrade model -first ... passed 2014-09-23 08:38:31: Running as user grid: /u01/app/11204_upg/grid/bin/srvctl stop oc4j 2014-09-23 08:38:31: s_run_as_user2: Running /bin/su grid -c ' /u01/app/11204_upg/grid/bin/srvctl stop oc4j ' 2014-09-23 08:38:32: Removing file /tmp/filefdfk1C 2014-09-23 08:38:32: Successfully removed file: /tmp/filefdfk1C 2014-09-23 08:38:32: /bin/su exited with rc=2 2014-09-23 08:38:32: J2EE (OC4J) Container Resource Stop ... passed ... 2014-09-23 08:38:32: Running as user grid: /u01/app/11204_upg/grid/bin/srvctl disable oc4j 2014-09-23 08:38:32: s_run_as_user2: Running /bin/su grid -c ' /u01/app/11204_upg/grid/bin/srvctl disable oc4j ' 2014-09-23 08:38:32: Removing file /tmp/file5FS4ZR 2014-09-23 08:38:32: Successfully removed file: /tmp/file5FS4ZR 2014-09-23 08:38:32: /bin/su exited with rc=1 2014-09-23 08:38:32: J2EE (OC4J) Container Resource Disable ... passed 2014-09-23 08:38:32: J2EE (OC4J) Container Resource stopped and disabled 2014-09-23 08:38:32: isLastNodeToUpgrade... 2014-09-23 08:38:32: isLastNodeToUpgrade: 1 2014-09-23 08:38:32: Executing /u01/app/11204_upg/grid/bin/crsctl set crs activeversion 2014-09-23 08:38:32: Executing cmd: /u01/app/11204_upg/grid/bin/crsctl set crs activeversion 2014-09-23 08:38:32: Command output: > CRS-1115: Oracle Clusterware has already been upgraded. >End Command output 2014-09-23 08:38:32: /u01/app/11204_upg/grid/bin/crsctl set crs activeversion ... passed 2014-09-23 08:39:32: Rolling upgrade is set to 1 2014-09-23 08:39:32: End ASM rolling upgrade 2014-09-23 08:39:32: Executing as grid: /u01/app/11204_upg/grid/bin/asmca -silent -upgradeLocalASM -lastNode /u01/app/11203/grid 2014-09-23 08:39:32: Running as user grid: /u01/app/11204_upg/grid/bin/asmca -silent -upgradeLocalASM -lastNode /u01/app/11203/grid 2014-09-23 08:39:32: Invoking "/u01/app/11204_upg/grid/bin/asmca -silent -upgradeLocalASM -lastNode /u01/app/11203/grid" as user "grid" 2014-09-23 08:39:32: Executing /bin/su grid -c "/u01/app/11204_upg/grid/bin/asmca -silent -upgradeLocalASM -lastNode /u01/app/11203/grid" 2014-09-23 08:39:32: Executing cmd: /bin/su grid -c "/u01/app/11204_upg/grid/bin/asmca -silent -upgradeLocalASM -lastNode /u01/app/11203/grid" 2014-09-23 08:39:45: Command output: > > ASM upgrade has finished on last node. ..
Rerun ./rootupgrade.sh a 3rd time ( now we don’t see any CRS-1115 )
[root@grac31 grid]# ./rootupgrade.sh .. Database Configuration Assistant when a database is created Finished running generic part of root script. Now product-specific root actions will be performed. Using configuration parameter file: /u01/app/11204_upg/grid/crs/install/crsconfig_params Installing Trace File Analyzer clscfg: EXISTING configuration version 5 detected. clscfg: version 5 is 11g Release 2. Successfully accumulated necessary OCR keys. Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Reference
- GI Upgrade from 11.2.0.3.6+ to 11.2.0.4 or 12.1.0.1 Fails with User(qosadmin) is deactivated. AUTH FAILURE. (Doc ID 1577072.1)
- How to Downgrade 11.2 Grid Infrastructure Standalone (Oracle Restart) After Successful or Failed Upgrade (Doc ID 1364412.1)