Table of Contents
Overview
To add node to 10.2 RAC database is done by using following steps
- Verify OS settings for the new Node ract3
- Run ./addNode.sh on ract1 from $ORA_CRS_HOME to install CW sofware and verify CW Installation
- Run ./addNode.sh ract1 from $ORACLE_HOME to install RDBMS sofware
- Use netca on cluster node ract1 and configure ract3 listener
- Use dbca on ract1 and finally add ASM and Rdbms Instance on ract3
Note OUI writes addNode.sh logs to /home/oracle/oraInventory/logs/
OS Verification for node ract3
Verify DNS [root@ract3 ~]# nslookup ract3 Name: ract3.example.com Address: 192.168.1.133 # nslookup ract3int Name: ract3int.example.com Address: 192.168.2.133 Reverse Address resolution : # nslookup 192.168.1.133 133.1.168.192.in-addr.arpa name = ract3.example.com. # nslookup 192.168.2.133 133.2.168.192.in-addr.arpa name = ract3int.example.com. Verify Network devices # ifconfig | egrep 'eth|inet addr' eth0 Link encap:Ethernet HWaddr 08:00:27:CE:53:81 inet addr:10.0.2.15 Bcast:10.0.2.255 Mask:255.255.255.0 eth1 Link encap:Ethernet HWaddr 08:00:27:F4:CF:0C inet addr:192.168.1.133 Bcast:192.168.1.255 Mask:255.255.255.0 eth2 Link encap:Ethernet HWaddr 08:00:27:CA:C8:09 inet addr:192.168.2.133 Bcast:192.168.2.255 Mask:255.255.255.0 inet addr:127.0.0.1 Mask:255.0.0.0 Verify RAW devices # raw -qa /dev/raw/raw1: bound to major 8, minor 17 /dev/raw/raw2: bound to major 8, minor 33 /dev/raw/raw3: bound to major 8, minor 49 /dev/raw/raw4: bound to major 8, minor 65 /dev/raw/raw5: bound to major 8, minor 81 # ls -l /dev/raw/ra* crw-r----- 1 root oinstall 162, 1 Apr 9 13:24 /dev/raw/raw1 crw-r----- 1 root oinstall 162, 2 Apr 9 13:24 /dev/raw/raw2 crw-r--r-- 1 oracle oinstall 162, 3 Apr 9 13:24 /dev/raw/raw3 crw-r--r-- 1 oracle oinstall 162, 4 Apr 9 13:24 /dev/raw/raw4 crw-r--r-- 1 oracle oinstall 162, 5 Apr 9 13:24 /dev/raw/raw5 Verify ASM disks # /usr/sbin/oracleasm scandisks Reloading disk partitions: done Cleaning any stale ASM disks... Scanning system for ASM disks... # /usr/sbin/oracleasm listdisks ASM_DATA01 ASM_DATA02 ASM_DATA03 ASM_DATA04 Verify SSH connectivity [root@ract3 ~]# su - oracle [oracle@ract3 ~]$ ssh ract1 date Wed Apr 9 13:26:46 CEST 2014 [oracle@ract3 ~]$ ssh ract2 date Wed Apr 9 13:26:55 CEST 2014 Run cluvfy: [oracle@ract3 cluvfy12]$ ./bin/cluvfy comp sys -p crs -r 10gR2 -n ract3 Verifying system requirement Total memory check passed Available memory check passed Swap space check passed Free disk space check passed for "ract3:/tmp" Check for multiple users with UID value 500 passed User existence check passed for "oracle" Check for multiple users with UID value 99 passed User existence check passed for "nobody" Group existence check passed for "oinstall" Group existence check passed for "dba" Membership check for user "oracle" in group "oinstall" [as Primary] passed Membership check for user "oracle" in group "dba" passed Hard limits check passed for "maximum open file descriptors" Soft limits check passed for "maximum open file descriptors" Hard limits check passed for "maximum user processes" Soft limits check passed for "maximum user processes" System architecture check passed Kernel version check passed Kernel parameter check passed for "semmsl" Kernel parameter check passed for "semmns" Kernel parameter check passed for "semopm" Kernel parameter check passed for "semmni" Kernel parameter check passed for "shmmax" Kernel parameter check passed for "shmmni" Kernel parameter check passed for "shmall" Kernel parameter check passed for "file-max" Kernel parameter check passed for "ip_local_port_range" Kernel parameter check passed for "rmem_default" Kernel parameter check passed for "rmem_max" Kernel parameter check passed for "wmem_default" Kernel parameter check passed for "wmem_max" Kernel parameter check passed for "aio-max-nr" Package existence check passed for "make" Package existence check passed for "binutils" WARNING: PRVF-7584 : Multiple versions of package "control-center" found on node ract3: control-center(x86_64)-2.16.0-16.el5,control-center(i386)-2.16.0-16.el5 Package existence check passed for "control-center" Package existence check passed for "gcc" WARNING: PRVF-7584 : Multiple versions of package "libstdc++" found on node ract3: libstdc++(x86_64)-4.1.2-54.el5,libstdc++(i386)-4.1.2-54.el5 Package existence check passed for "libstdc++" Package existence check passed for "libstdc++-devel" Package existence check passed for "sysstat" Package existence check passed for "setarch" WARNING: PRVF-7584 : Multiple versions of package "glibc" found on node ract3: glibc(x86_64)-2.5-118.el5_10.2,glibc(i686)-2.5-118.el5_10.2 Package existence check passed for "glibc" Package existence check passed for "glibc-common" Check for multiple users with UID value 0 passed Starting check for consistency of primary group of root user Check for consistency of root user's primary group passed Time zone consistency check passed Verification of system requirement was successful. If getting fixable errors run cluvfy with -verbose -fixup [oracle@ract3 ract3]$ ./bin/cluvfy comp sys -p crs -r 10gR2 -n ract3 -verbose -fixup Now check CRS readiness with cluvfy [oracle@ract3 cluvfy12]$ ./bin/cluvfy stage -pre crsinst -n ract3 -r 10gR2 ... Check for consistency of root user's primary group passed Time zone consistency check passed Verification of system requirement was successful. Ignore potential Error PRVG-5745 for cluvfy stage -pre nodeadd [oracle@ract3 cluvfy12]$ ./bin/cluvfy stage -pre nodeadd -n ract3 Performing pre-checks for node addition ERROR: PRVG-5745 : CRS Configuration detected, Restart configuration check not valid in this environment Verification cannot proceed Pre-check for node addition was unsuccessful on all the nodes. ---> Run cluvfy with nodeadd option not from the node to be added ( ract3 ) - just run the command from an alreday installed node ( ract2 ) [oracle@ract2 ~]$ cluvfy stage -pre nodeadd -n ract3 For details check following Bugs Bug 12705949 : CLUVFY COMP NODECON FAILING WHEN THERE IS NO CRS INSTALLED Bug 13343726 : LNX64-12.1-CVU: MISLEADING PRVG-5745 MESSAGE BEFORE CRS FRESH INSTALLATION
Install CRS software
Start AddNode.sh script from an already installed RAC instance First testing ssh connectivity to our newly connecting node [oracle@ract1 ~]$ ssh ract3 date Wed Apr 9 14:38:27 CEST 2014 [oracle@ract1 bin]$ cd /u01/app/oracle/product/crs/oui/bin [oracle@ract1 bin]$ ./addNode.sh Starting Oracle Universal Installer... Select Node : ract3 Cluster-Interconnect : ract3int VIP : ract3vip Run rootaddnode.sh on ract1: [root@ract1 etc]# /u01/app/oracle/product/crs/install/rootaddnode.sh clscfg: EXISTING configuration version 3 detected. clscfg: version 3 is 10G Release 2. Attempting to add 1 new nodes to the configuration Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node <nodenumber>: <nodename> <private interconnect name> <hostname> node 3: ract3 ract3int ract3 Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. Failure -2 opening file handle for (raw5) Failed to update the voting device raw5 with addnode info. 1 /u01/app/oracle/product/crs/bin/srvctl add nodeapps -n ract3 -A ract3vip.example.com/255.255.255.0/eth1 -o /u01/app/oracle/product/crs Check OCR after running rootaddnode.sh on ract1 [root@ract1 ~]# crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.ract3.gsd application OFFLINE OFFLINE ora.ract3.ons application OFFLINE OFFLINE ora.ract3.vip application OFFLINE OFFLINE Now run root.sh on ract3 [root@ract3 ~]# /u01/app/oracle/product/crs/root.sh WARNING: directory '/u01/app/oracle/product' is not owned by root WARNING: directory '/u01/app/oracle' is not owned by root WARNING: directory '/u01/app' is not owned by root WARNING: directory '/u01' is not owned by root Checking to see if Oracle CRS stack is already configured Oracle CRS stack is already configured and will be running under init(1M) [root@ract3 ~] crsctl check crs CSS appears healthy CRS appears healthy EVM appears healthy [root@ract3 ~]# crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.ract3.gsd application ONLINE ONLINE ract3 ora.ract3.ons application ONLINE ONLINE ract3 ora.ract3.vip application ONLINE ONLINE ract3 --> CRS is up and running - need to add listener and ASM and RAC instances
Install database software
[oracle@ract1 ~]$ cd $ORACLE_HOME/oui/bin/ [oracle@ract1 bin]$ ./addNode.sh Starting Oracle Universal Installer... --> Select new node : ract3
Configure listener on node ract3
Start netca on ract1 $ netca & Select the Type of Oracle Net Services Configuration Select Cluster configuration Select the nodes to configure Only select the new Oracle RAC node: ract3 Type of Configuration Select Listener configuration. Listener Configuration Next 6 Screens The following screens are now like any other normal listener configuration. What do you want to do: Add Listener name: LISTENER Selected protocols: TCP Port number: 1521 Configure another listener: No Listener configuration complete! [ Next ] Verify listener configuration [oracle@ract3 rac_db1]$ crs_stat ora.ract3.LISTENER_RACT3.lsnr NAME=ora.ract3.LISTENER_RACT3.lsnr TYPE=application TARGET=ONLINE STATE=ONLINE on ract3 [oracle@ract3 rac_db1]$ lsnrctl status LISTENER_RACT3 LSNRCTL for Linux: Version 10.2.0.1.0 - Production on 09-APR-2014 18:37:45 Copyright (c) 1991, 2005, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=ract3vip.example.com)(PORT=1521)(IP=FIRST))) .. Service "RACT" has 2 instance(s). Instance "RACT1", status READY, has 1 handler(s) for this service... Instance "RACT2", status READY, has 1 handler(s) for this service... .. [oracle@ract3 admin]$ crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....T3.lsnr application ONLINE ONLINE ract3 ora.ract3.gsd application ONLINE ONLINE ract3 ora.ract3.ons application ONLINE ONLINE ract3 ora.ract3.vip application ONLINE ONLINE ract3
Run dbca to configure ASM and RDBMS instance
$ dbca Welcome Screen Select Oracle Real Application Clusters database. Operations Select Instance Management. Instance Management Select Add an instance. ... DBCA verifies the new node ract3, and as the database is configured to use ASM, prompts with the message “ASM is present on the cluster but needs to be extended to the following nodes: [ract3]. Do you want ASM to be extended?” Click on Yes to add ASM to the new instance. [oracle@ract3 admin]$ crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....T3.inst application ONLINE ONLINE ract3 ora....SM3.asm application ONLINE ONLINE ract3 ora....T3.lsnr application ONLINE ONLINE ract3 ora.ract3.gsd application ONLINE ONLINE ract3 ora.ract3.ons application ONLINE ONLINE ract3 ora.ract3.vip application ONLINE ONLINE ract3 SQL> select inst_id, instance_name, host_name, status, to_char(startup_time, 'DD-MON-YYYY HH24:MI:SS') from gv$instance order by inst_id; INST_ID INSTANCE_NAME HOST_NAME STATUS TO_CHAR(STARTUP_TIME ---------- ---------------- -------------------- ------------ -------------------- 1 RACT1 ract1.example.com OPEN 09-APR-2014 14:06:17 2 RACT2 ract2.example.com OPEN 09-APR-2014 10:38:20 3 RACT3 ract3.example.com OPEN 09-APR-2014 18:41:52
Potential CRS-Error
CRS-0223: Resource 'ora.ract3.gsd' has placement error. [root@ract3 client]# crs_start ora.ract3.gsd CRS-1028: Dependency analysis failed because of: 'Resource in UNKNOWN state: ora.ract3.gsd' CRS-0223: Resource 'ora.ract3.gsd' has placement error. Fix : Try to manually restart the resource [root@ract3 client]# crs_stop ora.ract3.gsd Attempting to stop `ora.ract3.gsd` on member `ract3` Stop of `ora.ract3.gsd` on member `ract3` succeeded. [root@ract3 client]# crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.ract3.gsd application ONLINE ONLINE ract3 CRS-210 running rootaddnode.sh running srvctl add nodeapps - for details read following note.
Reference