Table of Contents
Key Facts:
- Recreate the GI Management Repository database – fails with 12.1.0.1.5
- Erorr: PLS-00801: internal error [*** ASSERT at file pdw1.c, line 3491; Failed To Get”
- Reference: BUG 20125473 – PLS-00801 ERROR OCCURRED WHEN APPLYING PATCH 19392372
Delete old MGMTDB database
If needed start MGMTDB with a pfile SQL> create pfile='/tmp/m.ora' from spfile='+DATA/_MGMTDB/spfile-MGMTDB.ora'; SQL> startup pfile='/tmp/m.ora' On each node, as root user: # $GRID_HOME/bin/crsctl stop res ora.crf -init # $GRID_HOME/bin/crsctl modify res ora.crf -attr ENABLED=0 -init As Grid User on the node where ora.mgmtdb resource is running execute: $ $GRID_HOME/bin/dbca -silent -deleteDatabase -sourceDB -MGMTDB If the above dbca -silent -deleteDatabase not work you may need to use asmcmd rm -rf command
Recreate the 12.1.0.1 MGMTDB database
Note that there are different commands for 12.1.0.1 and 12.1.0.2 (see Doc ID 1589394.1) [grid@gract1 ~]$ $GRID_HOME/bin/dbca -silent -createDatabase -templateName MGMTSeed_Database.dbc -sid -MGMTDB -gdbName _mgmtdb -storageType ASM -diskGroupName +DATA -datafileJarLocation $GRID_HOME//assistants/dbca/templates -characterset AL32UTF8 -autoGeneratePasswords -oui_internal Copying database files 1% complete 3% complete 10% complete 17% complete 24% complete 31% complete 35% complete Creating and starting Oracle instance 37% complete 42% complete 47% complete 52% complete 53% complete 56% complete 58% complete Registering database with Oracle Grid Infrastructure 64% complete Completing Database Creation 68% complete 79% complete 90% complete 100% complete Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb/_mgmtdb1.log" for further details. --> Your really should check that log file. Even dbca reports no error there can be an error Review your database creation log file very carefully ! Reported error: PRCD-1000 : Database _mgmtdb already exists PRCR-1086 : resource ora.mgmtdb is already registered DBCA_PROGRESS : 64% Completing Database Creation DBCA_PROGRESS : 68% ORA-06550: line 1, column 7: PLS-00801: internal error [*** ASSERT at file pdw1.c, line 3491; Failed To Get DBCA_PROGRESS : 79% DBCA_PROGRESS : 90% DBCA_PROGRESS : 100% Database creation complete. For details check the logfiles at: /u01/app/grid/cfgtoollogs/dbca/_mgmtdb. Database Information: Global Database Name:_mgmtdb
Delete and recreate the MGMT resources
[root@gract1 _mgmtdb]# srvctl stop mgmtlsnr [root@gract1 _mgmtdb]# srvctl remove mgmtdb Remove the database _mgmtdb? (y/[n]) y [root@gract1 _mgmtdb]# srvctl remove mgmtlsnr [root@gract1 gract1]# srvctl add mgmtlsnr [root@gract1 gract1]# srvctl add mgmtdb [root@gract1 gract1]# srvctl modify mgmtdb -spfile '+DATA/_MGMTDB/spfile-MGMTDB.ora' [root@gract1 gract1]# srvctl stop mgmtdb PRCC-1016 : _mgmtdb was already stopped [root@gract1 gract1]# srvctl start mgmtdb ***** Cluster Resources: ***** Resource NAME INST TARGET STATE SERVER STATE_DETAILS --------------------------- ---- ------------ ------------ --------------- ----------------------------------------- ora.MGMTLSNR 1 ONLINE ONLINE gract1 169.254.235.72 192.1 68.2.111,STABLE ora.mgmtdb 1 ONLINE ONLINE gract1 Open,STABLE Enable and start ora.crf resource. On each node, as root user: # $GRID_HOME/bin/crsctl modify res ora.crf -attr ENABLED=1 -init # $GRID_HOME/bin/crsctl start res ora.crf -init ***** Local Resources: ***** Resource NAME INST TARGET STATE SERVER STATE_DETAILS --------------------------- ---- ------------ ------------ --------------- ----------------------------------------- ora.crf 1 ONLINE ONLINE gract1 STABLE
Verifying oclumon functionality
[grid@gract2 ~]$ oclumon dumpnodeview -allnodes CRS-9118-Grid Infrastructure Management Repository connection error ORA-12514: TNS:listener does not currently know of service requested in connect descriptor [grid@gract1 ~]$ lsnrctl status MGMTLSNR Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=MGMTLSNR))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.2.111)(PORT=1531))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=169.254.235.72)(PORT=1531))) The listener supports no services The command completed successfully --> Service _mgmtdb not registered by MGMTLSNR [grid@gract1 ~]$ lsnrctl status Service "_mgmtdb" has 1 instance(s). Instance "-MGMTDB", status READY, has 1 handler(s) for this service... --> Service _mgmtdb is configured with default listener Local listener not defined SQL> show parameter local_listener NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ local_listener string Change LISTENER_NETWORKS ( from Bug 19313202 ) SQL> ALTER SYSTEM SET LISTENER_NETWORKS = '((NAME=private_network) (LOCAL_LISTENER="(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP) (HOST=169.254.235.72)(PORT=1531))(ADDRESS=(PROTOCOL=TCP) (HOST=192.168.2.111)(PORT=1531)))"))' SCOPE=MEMORY SID='-MGMTDB'; SQL> show parameter listener NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ listener_networks string ((NAME=private_network) (LOCAL_LISTENER="(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST 169.254.235.72)(PORT=1531))(ADDRESS=(PROTOCOL=TCP)(HOST=192. 168.2.111)(PORT=1531)))")) local_listener string remote_listener string [grid@gract1 ~]$ lsnrctl status MGMTLSNR .. Services Summary... Service "_mgmtdb" has 1 instance(s). Instance "-MGMTDB", status READY, has 1 handler(s) for this service. [grid@gract1 ~]$ lsnrctl status .. Service "_mgmtdb" has 1 instance(s). Instance "-MGMTDB", status READY, has 1 handler(s) for this service... --> Service "_mgmtdb" is not registered for both Listeners MGMTLSNR and default LISTENER [grid@gract1 ~]$ oclumon dumpnodeview -allnodes ---------------------------------------- Node: gract1 Clock: '14-10-28 08.02.01' SerialNo:8758 ---------------------------------------- SYSTEM: #pcpus: 1 #vcpus: 1 cpuht: N chipname: Intel(R) cpu: 17.76 cpuq: 16 physmemfree: 218160 physmemtotal: 4354416 mcache: 2374288 swapfree: 5053196 swaptotal: 5210108 hugepagetotal: 0 hugepagefree: 0 hugepagesize: 2048 ior: 95 iow: 207 ios: 30 swpin: 0 swpout: 0 pgin: 95 pgout: 196 netr: 35.711 netw: 32.879 procs: 340 rtprocs: 10 #fds: 12832 #sysfdlimit: 6815744 #disks: 18 #nics: 5 nicErrors: 0 TOP CONSUMERS: topcpu: 'mdb_vktm_-mgmtd(13626) 3.40' topprivmem: 'ocssd.bin(11228) 98072' topshm: 'ora_mman_erp_1(15523) 229804' topfd: 'ocssd.bin(11228) 305' topthread: 'console-kit-dae(3598) 64' .... [grid@gract3 ~]$ oclumon dumpnodeview -allnodes --> works for all 3 nodes [grid@gract2 ~]$ oclumon dumpnodeview -allnodes --> works for all 3 nodes [grid@gract1 ~]$ oclumon dumpnodeview -allnodes ---------------------------------------- Node: gract1 Clock: '14-10-28 08.19.56' SerialNo:8973 ---------------------------------------- --> Works only on local node gract1 - need to investigate
Checking patch status of -MGMTDB:
[grid@gract1 ~]$ env | grep SID ORACLE_SID=-MGMTDB SQL> select * from dba_registry_history; ACTION_TIME ACTION NAMESPACE VERSION ------------------------------ ------------------------------ ------------------------------ ------------------------------ ID BUNDLE_SERIES COMMENTS ---------- ------------------------------ ---------------------------------------- 24-MAY-13 12.21.12.617091 PM APPLY SERVER 12.1.0.1 0 PSU Patchset 12.1.0.0.0 27-OCT-14 07.34.33.119553 PM APPLY SERVER 12.1.0.1 5 PSU PSU 12.1.0.1.5 SQL> select * from dba_registry_sqlpatch ; no rows selected --> Datapatch not yet installed - Installing [grid@gract1 ~]$ $GRID_HOME/OPatch/datapatch -verbose SQL Patching tool version 12.1.0.1.0 on Tue Oct 28 06:56:35 2014 Copyright (c) 2014, Oracle. All rights reserved. Connecting to database...OK Determining current state... Currently installed SQL Patches: Currently installed C Patches: 19121550 Adding patches to installation queue and performing prereq checks... Installation queue: Nothing to roll back The following patches will be applied: 19121550 Installing patches... Patch installation complete. Total patches installed: 1 Validating logfiles... Patch 19121550 apply: WITH ERRORS logfile: /u01/app/121/grid/sqlpatch/19121550/19121550_apply__MGMTDB_2014Oct28_06_57_09.log (no errors) catbundle generate logfile: /u01/app/grid/cfgtoollogs/catbundle/catbundle_PSU__MGMTDB__mgmtdb_GENERATE_2014Oct28_06_57_11.log (no errors) catbundle apply logfile: /u01/app/grid/cfgtoollogs/catbundle/catbundle_PSU__MGMTDB__mgmtdb_APPLY_2014Oct28_06_57_14.log (errors) Error at line 102: ORA-06550: line 1, column 7: Error at line 103: PLS-00801: internal error [*** ASSERT at file pdw1.c, line 3491; Failed To Get See support note 1609718.1 for information on how to resolve the above errors --> The above problems seems to be related to BUG 20125473 - PLS-00801 ERROR OCCURRED WHEN APPLYING PATCH 19392372 Please open a SR with Oracle Support to get details about a potential WA.
Reference
- BUG 20125473 – PLS-00801 ERROR OCCURRED WHEN APPLYING PATCH 19392372
- DBCA is not failing if Management Database Resource Creation Fails (Doc ID 1631311.1)
- Bug 19313202 : MGMTDB HAS ALL LISTENERS IN LOCAL_LISTENER PARAMETER
Hi Helmut
I had exactly the same error applying PSU 12.1.0.1.5
” Error at line 103: PLS-00801: internal error [*** ASSERT at file pdw1.c, line 3491; Failed To Get”
How did you resolve it?
Thanks in advance.
regards,
Lawrence
Hi Lawrence,
please ask Oracle Support for the WA explained in
BUG 20125473 – PLS-00801 ERROR OCCURRED WHEN APPLYING PATCH 19392372
br
Helmut
Thanks Helmut, much appreciated. When I applied this patch on 17th October, it was only 3 days old having been released on 14th so when we reported the error Oracle didn’t have an answer. Thanks for a very helpful blog.
regards
Lawrence