–> Disclaimer – Must read !!
Table of Contents
– DEBUGGING related ARTICLES_______
- Debug RAC listener status change from ONLINE to INTERMEDIATE CHECK TIMED OUT,STABLE
- Debugging Node Eviction using ISDE and TFA WEB ( password protected )
- Debugging Node Eviction triggered by low CPU and/or Paging/Swapping by using TFA, CHM, OSWatcher
- Troubleshooting Clusterware startup problems with detailed debugging info
- Debugging DRM
- DEBUG Install problems
- Debug Upgrade Problems
- Debugging a RAC Node Eviction by reading LMBH trace file
- Analyze CRSD logfiles using TINTs
- Tracing LMS processes using DBMS_SYSTEM.set_ev
– 12c/11g NEW FEATURES ______
- RAC 12c : Pluggable Databases
- 12c feature: FLEX ASM
- 12c Feature: Testing VIP/Application failover on LEAF Nodes
- 12c feature: ACFS/HANFS
- 12c/11g feature : Using Server pools
- 12c feature: Using Management Repository
- 12c feature : What-If Command Evaluation with serverpool
– ASM related ARTICLES ______
+++ ASM Generic ( Overview / Errors / Diag )
- Generic ASM Overview
- ASM tools used by Support : KFOD, KFED, AMDU (Doc ID 1485597.1)
- ASM doesn’t start due to CRS-1714 error
- ORA-15040, ORA-15042 errors mounting a diskgroup
- Useful ASM Commands
- Add/Drop ASM disk ( RAC 12.1 )
- Why is Usable_file_MB negative in v$asm_disk view ?
- Recover from a lost ASM disk with a voting disk
+++ ASM Operations
- ASM: Backup and restore DG metadata
- ASM: Create/Drop Diskgroup
- ASM : Cancel a DROP Disk operation
- ASM Operation: Rename an ASM disk
- ASM operation : Rename Diskgroup
+++ ASM Recovering +FRA +OCR Diskgroup
- Recovering OCR DG when all disks are lost
- Recovering OCR DG when missing a single disk
- Recover RAC database after loosing FRA diskgroup
+++ ASM Block/Disk corruptions
- Locate ASM AU blocks at OS level
- How ASM fixes block corruption with NORMAL redundancy ?
- Using kfed/dd to repair ASM disk header corruption
+++ ASM and UDEV
+++ ASM SPFILE
+++ ACFS
- ACFS – A closer look
- 12.1 ACFS installation and usage
- ACFS install on top of GRID 11.2.0.3.4
- Working with ACFS Snapshots on top of Clusterware 11.2
- Debugging ACFS mount errors: ACFS-02017
+++ DBFS
- Setup DBFS on RAC 11.2.0.4.3, UEK3.13 using fuse-2.8.3
- Debugging problems when mounting a DBFS filesystem
+++ DNFS / Openfiler
- DNFS – Direct NFS setup for 11.2.0.4 3-node RAC cluster on OEL 6.4
- RAC 11.2.0.4 setup using OPENFILER with Multipath ISCSI disks
– CLUSTERWARE related ARTICLES _____
+++ GPNP – Global Plug and Play
- Fix a wrong entry profile.xml and redistribute the fixed profile with gpnptool put
- Grid Plug and Play ( GPnP ) – new 11.2 feature
+++ Clusterware Resources
- Managing Resources
- Using Oracle Clusterware 11.2 to Protect 3rd Party Applications
- Recreate 10.2 Nodeapps,RAC Listener, ASM and DB instance after an OCR corruption – not supported
+++ CRS startup problems
- CRS does not start GIPC error: [29] msg [gipcretConnectionRefused]
- Troubleshooting Clusterware startup problems with detailed debugging info
+++ CRS errors/corruptions
- Startup a policy managed database fails with CRS-2643
- Dropping an already deleted diskgroup returns CRS-222 CRS-2730
- CRS-210 error and OCR corruption – Error: could not find ora.ract3.vip
- DBCA 10.2.0.1 fails with ORA-12519 on adding a new RAC instance
+++ OLR, OCR and vote disk related articles
- Recovering corrupted OLR after PROCL-5 PROCL-26 error
- Relocate OCR and Voting Disks to a different ASM diskgroup ( 11.2.0.4 )
- Cleanup root.sh configuration after a failed CRS 10g, 11.1 installation
- Rebuild OCR for a single RAC Note – RAC 11.2
- Find OCR Master Node
– RAC related ARTICLES ___
Errors/Using SPfile/Events
+++ Policy Managed Database
- Impact loosing a cluster member used by a Server Pool
- RAC 12c policy managed database and Serverpools
- RAC 12c : Pluggable Databases and Serverpools
- Managing Server Pools
- Convert Admin Managed database to Policy Managed
- ORA-1031 connect as sysdba using password authentification after converting
+++ Cache Fusion and Dynamic Resource Mastering ( DRM )
- A closer look into DRM ( Dynamic Resource Management )
- A closer look at Cache Fusion II
- Mapping between resources and datablocks by using ROWID
+++ RAC and CRS UPGRADE / Install / Deinstallation / Cloning Addnote.sh
UPGRADE
- Debug Upgrade Problems
- Upgrade FLEX ASM cluster fom 12.1.0.1 to 12.1.0.2
- Upgrade 12.1.0.1 Oracle Multitenant database to 12.1.0.2
- Upgrading from 12.1.0.1 to 12.1.0.2 using FLEX cluster hangs with progress bar frozzen at at 7 %
INSTALL
- DEBUG Install problems
- Install RAC One Node database ( 12c )
- Install Oracle RAC 12.1,OEL 6.4 and Virtualbox 4.2 with GNS and UDEV
- Install Oracle RAC 12.1, OEL 6.3 and Virtualbox 4.2 with GNS
- Install Oracle RAC 11.2.0.4,OEL 6.4 and Virtualbox 4.2 with GNS and UDEV
- Installing Oracle RAC 11.2.0.3, OEL 6.3 and Virtualbox 4.2 with GNS
- Install CRS 10.2.0.1 on top of OEL 5.10 / Virtualbox 4.2
- Install RAC 10.2.0.1 on top of OEL 5.10 and Virtualbox 4.2
CLONING/AddNote
- Cloning CRS 11.2.0.3.4 using config.sh
- Add a new Node to 11.2.0.3.4 Clusterware
- Clone Clusterware with active GNS to new node ( Single Node )
CLEANUP
- Delete a CRS installation manually
- Cleanup steps for failed/missing/deleted CRS installation ( 11.2.0.3)
- Cleanup steps for cluster node removal for following scenarios
- Cleanup / Removing a crashed node from OCR – RAC 10.2.0.1
Kernel UPGRADE
+++ RAC/GRID patching
- Install Grid Infrastructure Patch Set Update : 11.2.0.3.4 (14275572) using opatch auto
- Generic PSU patching considerations
- Installation of GRID PSU 11.2.0.3.8 – patch 17272731 – on top of OEL 6.4 using opatch auto
- Opatch auto is generating 30 Steps to install PSU 11.2.0.3.8 on a RAC instance
- Deinstallation of PSU 11.2.0.3.8 – patch 17272731
- Tracking history of patch Sets and PSU patches
+++ Node Eviction
- Eviction Overview
- Debugging Node Eviction triggered by low CPU and/or Paging/Swapping by using TFA, CHM, OSWatcher
- Check following Log files after a Node Eviction
- Node Eviction due to failed Network Heartbeat (NHB )
- Node Eviction due to failed Disk Heartbeat ( DHB )
- Member Kill Eviction
- Eviction due to CSSD hang ( leads to a system reboot )
- Using oclumon to detect potential root causes for node evictions
- Tracefiles updates generated by an instance node eviction ( ordered by timestamps )
- Known Bugs leading to Node Eviction
+++ ADMIN TASKS
- Convert a 3-Node 11.2.0.3 RAC cluster to use HUGEPAGES ( OEL 6.4 )
- How to move REDO logs to a SSD Disk in a RAC/ASM env ?
- Why is my RAC or TAF service not relocated ?
+++ RAC Backup and Recovery
– RAC PERFORMANCE _______
+++ RAC Performance and Locks____
- RAC Performance Overview
- RAC Wait Events Summary
- RAC 12c Practical Performance Management ( Markus Michalewicz, Michael Zoll ) Must READ !
+++ GES Locks
+++ AWR/ASH related articles
- RAC AWR sample
- RAC and ASH reports
- Create and Analyze ASH dumps
- RAC Perfomance tuning with AWR/EM reports
+++ GCS/RAC related Wait Events
- Gc buffer busy release /Gc buffer busy acquire due to Right-Hand-Side Index Growth
- GC CR Block 3-Way/GC CR buffer busy Wait Event
- Sequence – SV contention/Row Cache lock
- RAC Wait Event: gcs log flush sync
+++ LOG FILE SYNC related articles
- Testing Cluster Redo Log Write Performance with dd
- Debug log file sync wait event with lfsdiag.sql/iostat
- Log File Sync wait event and SSD disks
- Log File Sync wait event and CPU starvation
– RAC NETWORKING RELATED ARCTICLES_____
HAIP
SCAN Listeners
– RAC TOOLS ( CHM,OSWATHER.OUI,CLUVFY)_______
OSWatcher
Cluvfy
Cluster Health Monitor
- Overview Cluster Health Monitor
- Debugging a RAC Node Eviction with CHM
- Using oclumon to display CHM data
- Convert CHM data to more human readable format – like vmstat reports
Oracle Installer OUI
- Oracle Installer and RAC installation
- Install newest Opatch version
- Opatch fails with version check error when installing 11.2.0.4.3 with OPatch 11.2.0.3.6
Oratop/Racheck/Procwatcher/TFA
- RAC realtime monitoring with ORATOP
- Procwatcher Setup and Usage
- RACCheck/ORACheck
- Trace File Analyzer Collector – TFA Collector Usage and installation
great blog
great articles
much kudos
Thank you for sharing wonderfully impressive knowledge. Do you mentor junior people in your field?
sehr gute Seiten
Hello Sir –
Can you share the password for this debugging article please ?
“Protected: Debugging Node Eviction using ISDE and TFA WEB”
Thanks
Raj
Sorry Ray,
I can’t share this info as this info is for Oracle Support people ONLY
These articles are based on Oracle confidential details
Hi Admin, i want to open link “Diagnostics for ASM Basic Issues” but it asking for password, how can get its password ?
Hi Helmut!
How are you doing? Are you still with Oracle?
Joakim
Hi Joakim,
No I’m not working for Oracle anymore. As you may know support was closde in Europe and
I don’ want to switch to Oracle Dev [ got 2 jobs offered ]. I just took the package worked about 2 years
as freelancer and has started a new a career as a Full-Stack Developer at a University
in Nuremberg 2 weeks ago . This is much fun and I’m learning a lot of new things.
You can reach me at : Helmut.Hutzler@gmail.com
How are you doing? Are you still with Oracle?
Helmut
Hi Helmut,
I am following your works and my hope is becoming a man like you, knowing everything wishes the best for you.
Thanks for sharing, it was informative. We play a small role in upskilling people providing the latest tech courses. Join us to upgradeORACLE RAC ONLINE TRAINING