Upgrading from 12.1.0.1 to 12.1.0.2 using FLEX cluster hangs with progress bar frozzen at at 7 %

OUI hangs testing prerequisites steps  with progress level frozzen at 7%

OUI Step to progress : Checking Verify that the ASM instance was configure using an existing ASM paramter file

Verify that  asmcmd spget works 
If  asmcmd spget hangs try to kill process with is running asmcmd spget in parallel 
[root@gract1 /]# ps -elf  | grep asmcmd
0 S grid     15820     1  1  80   0 - 351661 pipe_w 16:34 ?       00:00:00 oracle+ASM1_asmcmd (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))]
[root@gract1 /]# kill -9 15820
After that  asmcmd spget should work

Strace output from above hang scenario 
[grid@gract1 ~/DBSupportBundle]$ asmcmd spget
+DATA/gract/ASMPARAMETERFI LE/registry.253.827098053
stat("/tmp/pipe_501_sysasm", {st_mode=S_IFIFO|0777, st_size=0, ...}) = 0
open("expor/tmp/pipe_501_sysasm", O_WRONLY|O_CREAT|O_TRUNC, 0666^C <unfinished ...>

If installation still hange try as next step to test your installation env by running
$ /u01/app/121/grid/perl/bin/perl -w -I /u01/app/121/grid/perl/lib/5.14.1 -I /u01/app/121/grid/perl/lib/site_perl/5.14.1 
   -I /u01/app/121/grid/lib -I /u01/app/121/grid/lib/asmcmd -I /u01/app/121/grid/rdbms/lib/asmcmd /u01/app/121/grid/bin/asmcmdcore spget
--> You may fix here errors like wrong settings for LD_LIBRARY_PATH , perl problems ....

If OUI still hangs check for defunct asmcmd daemon processed
[root@gract1 log]# ps -elf | grep asmcmd
0 S grid      9087  3028  0  80   0 - 65796 hrtime 11:43 pts/8    00:00:00 /u01/app/121/grid/perl/bin/perl -w -I /u01/app/121/grid/perl/lib/5.14.1 
                                                                            -I /u01/app/121/grid/perl/lib/site_perl/5.14.1 -I /u01/app/121/grid/lib 
                                                                            -I /u01/app/121/grid/lib/asmcmd -I /u01/app/121/grid/rdbms/lib/asmcmd 
                                                                             /u01/app/121/grid/bin/asmcmdcore spget
1 Z grid      9092  9087  0  80   0 -     0 exit   11:43 ?        00:00:00 [asmcmd daemon] <defunct>
0 S root     10571 25378  0  80   0 - 25824 pipe_w 12:05 pts/11   00:00:00 grep asmcmd
[root@gract1 log]# kill -9 9087
After killing process 4842 OUI should go  forward processing next prerequisites checks !

After all  prerequisites checks have been finished run followíng command on all nodes verifying that ASM parameter file is valid
[grid@gract1 ~]$ /u01/app/121/grid/perl/bin/perl -w -I /u01/app/121/grid/perl/lib/5.14.1 -I /u01/app/121/grid/perl/lib/site_perl/5.14.1 
   -I /u01/app/121/grid/lib -I /u01/app/121/grid/lib/asmcmd -I /u01/app/121/grid/rdbms/lib/asmcmd /u01/app/121/grid/bin/asmcmdcore spget
+DATA/gract/ASMPARAMETERFILE/registry.253.827098053

Relatad Installer error:
PRVG-4568 : An ASM instance was found to be configured but the ASM parameter file does not exist at location "/u01/app/121/grid/dbs/initnull.ora" 
            on the node "gract1" on which upgrade is requested.  
           Cause:  The indicated ASM parameter file did not exist at the identified location.  
          Action:  Ensure that the ASM instance is configured using an existing ASM parameter file, SPFILE or PFILE, on the indicated node

Summary 
  - Don't run any asmcmd commands in parallel with 12.1.0.1
  - Check for any outstanding asmcmd commands with <defunct>  for hanging asmcmd processes ( kill them ) 
  - After killing that process OUI will report an error after finishing prerequisites a
  - Test ASM spfile usage manually after OUI prerequisites testing was finished  and press ignore prerequisite test
    to continue installation

 

Reference

  • Bug 16875041 : ASMCMD -P LS COMMAND HANG OR FAILED WITH UNEXPECTED EOF ERROR
  • Bug 18630276 : ASMCMD MKDG HANG AND ASMCMD PERFORMANCE ISSUE WHILE SWINGBENCH WORKLOAD RUNNING

Leave a Reply

Your email address will not be published. Required fields are marked *