Note :
You need to install your RAC Nameserver on a separate Virtualbox image/system as a NON-functional Nameserver can lead to a RAC hang scenario !!
- If using Virtualbox you may read the following article first : http://www.hhutzler.de/blog/configure-virtual-network-devices-rac/
Table of Contents
Install BIND / DHCP RPMs and learn the needed configuration commands
Install and verify BIND installation [ bind_libs and bind_utils should be arlready installed ] [root@hract21 Desktop]# yum install bind bind-utils bind-libs [root@hract21 Desktop]# rpm -qa |grep '^bind' bind-utils-9.8.2-0.30.rc1.el6_6.1.x86_64 bind-libs-9.8.2-0.30.rc1.el6_6.1.x86_64 bind-9.8.2-0.30.rc1.el6_6.1.x86_64 Install and verify DHCP setup Download and install dcping utility; Download location: http://pkgs.repoforge.org/dhcping following package : dhcping-1.2-2.2.el6.rf.x86_64.rpm 11-Nov-2010 07:31 16K RHEL6 and CentOS-6 x86 64bit [root@ns1 ~]# rpm -i Downloads/dhcping-1.2-2.2.el6.rf.x86_64.rpm [root@hract21 Desktop]# yum install dhcp.x86_64 Total download size: 1.2 M Is this ok [y/N]: y Downloading Packages: (1/3): dhclient-4.1.1-43.P1.0.1.el6_6.1.x86_64.rpm | 318 kB 00:00 (2/3): dhcp-4.1.1-43.P1.0.1.el6_6.1.x86_64.rpm | 819 kB 00:00 (3/3): dhcp-common-4.1.1-43.P1.0.1.el6_6.1.x86_64.rpm | 142 kB 00:00 [root@hract21 Desktop]# rpm -qa | grep -i DHCP dhcp-common-4.1.1-43.P1.0.1.el6_6.1.x86_64 dhcp-4.1.1-43.P1.0.1.el6_6.1.x86_64 Setup Files needed: : /etc/named.conf : /var/named/example.com.db : /var/named/192.168.2.db : /var/named/192.168.5.db : /etc/dhcp/dhcpd.conf : /etc/sysconfig/dhcpd : /etc/dhcp/dhcpd.conf --> For details how to configure DNS/DHCP please read the details the chapters below. Setup,test and configure BIND service # service named restart # nslookup google.de Server: 192.168.5.50 Address: 192.168.5.50#53 Non-authoritative answer: Name: google.de Address: 173.194.112.152 Name: google.de Address: 173.194.112.159 Name: google.de Address: 173.194.112.143 Name: google.de Address: 173.194.112.151 # chkconfig named on # chkconfig named --list named 0:off 1:off 2:on 3:on 4:on 5:on 6:off Setup,test and configure DHCP service # service dhcpd start Starting dhcpd: [ OK ] # chkconfig dhcpd on # chkconfig dhcpd --list dhcpd 0:off 1:off 2:on 3:on 4:on 5:on 6:off Verify DHCP setup with dhcping [root@hract21 Desktop]# dhcping -s 192.168.5.50 -c 192.168.5.197 Got answer from: 192.168.5.50
DNS Server Setup
Our DNS server should have configured the Virtualbox Network Devices eth0 -> Bridged Network : inet addr:192.168.1.XXX Bcast:192.168.1.255 [ Internet Access ] eth1 -> Internal Network : inet addr:192.168.5.50 Bcast:192.168.5.255 [ Public RAC Interface ] eth0 Link encap:Ethernet HWaddr 08:00:27:E6:71:54 inet addr:192.168.1.X Bcast:192.168.1.255 Mask:255.255.255.0 eth1 Link encap:Ethernet HWaddr 08:00:27:8D:8A:93 inet addr:192.168.5.50 Bcast:192.168.5.255 Mask:255.255.255.0 Setup files used by DNS : /etc/named.conf /var/named/example.com.db /var/named/192.168.2.db /var/named/192.168.5.db /etc/named.conf : options { listen-on port 53 { 192.168.5.50; 127.0.0.1; }; directory "/var/named"; dump-file "/var/named/data/cache_dump.db"; statistics-file "/var/named/data/named_stats.txt"; memstatistics-file "/var/named/data/named_mem_stats.txt"; allow-query { any; }; allow-recursion { any; }; recursion yes; dnssec-enable no; dnssec-validation no; }; logging { channel default_debug { file "data/named.run"; severity dynamic; }; }; zone "." IN { type hint; file "named.ca"; }; zone "5.168.192.in-addr.arpa" IN { // Reverse zone type master; file "192.168.5.db"; allow-transfer { any; }; allow-update { none; }; }; zone "2.168.192.in-addr.arpa" IN { // Reverse zone type master; file "192.168.2.db"; allow-transfer { any; }; allow-update { none; }; }; zone "example.com" IN { type master; notify no; file "example.com.db"; }; /var/named/example.com.db: $TTL 1H ; Time to live $ORIGIN example.com. @ IN SOA ns1.example.com. hostmaster.example.com. ( 2009011202 ; serial (todays date + todays serial #) 3H ; refresh 3 hours 1H ; retry 1 hour 1W ; expire 1 week 1D ) ; minimum 24 hour ; IN NS ns1 ; name server for example.com ns1 IN A 192.168.5.50 grac41 IN A 192.168.5.101 grac42 IN A 192.168.5.102 grac43 IN A 192.168.5.103 grac41int IN A 192.168.2.101 grac42int IN A 192.168.2.102 grac43int IN A 192.168.2.103 ; $ORIGIN grid4.example.com. @ IN NS gns4.grid4.example.com. ; NS grid4.example.com IN NS ns1.example.com. ; NS example.com gns4 IN A 192.168.5.54 ; glue record /var/named/192.168.5.db : $TTL 1H @ IN SOA ns1.example.com. root.domin.com. ( 2009011201 ; serial (todays date + todays serial #) 3H ; refresh 3 hours 1H ; retry 1 hour 1W ; expire 1 week 1D ) ; minimum 24 hour IN NS ns1 ns1 IN A 192.168.5.50 ; 50 PTR ns1.example.com. 54 PTR gns4.grid4.example.com. ; reverse mapping for GNS 101 PTR grac41.example.com. 102 PTR grac42.example.com. 103 PTR grac43.example.com. 201 PTR wls1.example.com. /var/named/192.168.2.db : $TTL 1H @ IN SOA ns1.example.com. hostmaster.example.com. ( 2009011201 ; serial (todays date + todays serial #) 3H ; refresh 3 hours 1H ; retry 1 hour 1W ; expire 1 week 1D ) ; minimum 24 hour IN NS ns1 ns1 IN A 192.168.5.50 ; 101 PTR grac41int.example.com. 102 PTR grac42int.example.com. 103 PTR grac43int.example.com. Verify zone files and restart named deamon [root@ns1 named]# named-checkconf /etc/named.conf [root@ns1 named]# named-checkzone example.com example.com.db zone example.com/IN: grid.example.com/NS 'gns.grid.example.com' (out of zone) has no addresses records (A or AAAA) zone example.com/IN: grid12c.example.com/NS 'gns12c.grid12c.example.com' (out of zone) has no addresses records (A or AAAA) zone example.com/IN: grid2.example.com/NS 'gns2.grid2.example.com' (out of zone) has no addresses records (A or AAAA) zone example.com/IN: grid3.example.com/NS 'gns3.grid3.example.com' (out of zone) has no addresses records (A or AAAA) zone example.com/IN: grid4.example.com/NS 'gns4.grid4.example.com' (out of zone) has no addresses records (A or AAAA) zone example.com/IN: loaded serial 2009011202 OK [root@ns1 named]# named-checkzone example.com 192.168.5.db zone example.com/IN: loaded serial 2009011201 OK [root@ns1 named]# named-checkzone example.com 192.168.2.db zone example.com/IN: loaded serial 2009011201 OK
Verify DNS Setup
[root@ns1 ~]# nslookup google.de Server: 192.168.5.50 Address: 192.168.5.50#53 Non-authoritative answer: Name: google.de Address: 173.194.67.94 [root@ns1 ~]# nslookup grac41 Server: 192.168.5.50 Address: 192.168.5.50#53 Name: grac41.example.com Address: 192.168.5.101 [root@ns1 ~]# ping -c 2 google.de PING google.de (173.194.67.94) 56(84) bytes of data. 64 bytes from wi-in-f94.1e100.net (173.194.67.94): icmp_seq=1 ttl=38 time=66.3 ms 64 bytes from wi-in-f94.1e100.net (173.194.67.94): icmp_seq=2 ttl=38 time=62.3 ms --- google.de ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1064ms rtt min/avg/max/mdev = 62.373/64.344/66.316/1.987 ms [root@ns1 ~]# ping -c 2 grac41 PING grac41.example.com (192.168.5.101) 56(84) bytes of data. 64 bytes from grac41.example.com (192.168.5.101): icmp_seq=1 ttl=64 time=0.200 ms 64 bytes from grac41.example.com (192.168.5.101): icmp_seq=2 ttl=64 time=0.293 ms --- grac41.example.com ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 999ms rtt min/avg/max/mdev = 0.200/0.246/0.293/0.049 ms [root@ns1 ~]# cat /etc/resolv.conf # Generated by NetworkManager search example.com grid4.example.com nameserver 192.168.5.50 [root@ns1 ~]# netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 eth0 192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 192.168.3.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2 192.168.5.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1 If the GNS server is running the following commands should work too ! [root@ns1 ~]# nslookup grac4-scan Server: 192.168.5.50 Address: 192.168.5.50#53 Non-authoritative answer: Name: grac4-scan.grid4.example.com Address: 192.168.5.167 Name: grac4-scan.grid4.example.com Address: 192.168.5.156 Name: grac4-scan.grid4.example.com Address: 192.168.5.153 [root@ns1 ~]# ping -c 2 grac4-scan PING grac4-scan.grid4.example.com (192.168.5.167) 56(84) bytes of data. 64 bytes from 192.168.5.167: icmp_seq=1 ttl=64 time=0.176 ms 64 bytes from 192.168.5.167: icmp_seq=2 ttl=64 time=0.203 ms --- grac4-scan.grid4.example.com ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1000ms rtt min/avg/max/mdev = 0.176/0.189/0.203/0.019 ms [root@ns1 ~]# dig @192.168.5.50 grac4-scan.grid4.example.com ; <<>> DiG 9.9.3-RedHat-9.9.3-P1.el6 <<>> @192.168.5.50 grac4-scan.grid4.example.com ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18529 ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 2, ADDITIONAL: 2 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;grac4-scan.grid4.example.com. IN A ;; ANSWER SECTION: grac4-scan.grid4.example.com. 94 IN A 192.168.5.167 grac4-scan.grid4.example.com. 94 IN A 192.168.5.156 grac4-scan.grid4.example.com. 94 IN A 192.168.5.153 ;; AUTHORITY SECTION: grid4.example.com. 3600 IN NS gns4.grid4.example.com. grid4.example.com. 3600 IN NS ns1.example.com. ;; ADDITIONAL SECTION: ns1.example.com. 3600 IN A 192.168.5.50 ;; Query time: 1 msec ;; SERVER: 192.168.5.50#53(192.168.5.50) ;; WHEN: Sun Jan 11 17:17:51 CET 2015 ;; MSG SIZE rcvd: 158 [root@ns1 ~]# dig @192.168.5.54 grac4-scan.grid4.example.com ; <<>> DiG 9.9.3-RedHat-9.9.3-P1.el6 <<>> @192.168.5.54 grac4-scan.grid4.example.com ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5071 ;; flags: qr aa; QUERY: 1, ANSWER: 3, AUTHORITY: 1, ADDITIONAL: 1 ;; QUESTION SECTION: ;grac4-scan.grid4.example.com. IN A ;; ANSWER SECTION: grac4-scan.grid4.example.com. 120 IN A 192.168.5.153 grac4-scan.grid4.example.com. 120 IN A 192.168.5.156 grac4-scan.grid4.example.com. 120 IN A 192.168.5.167 ;; AUTHORITY SECTION: grid4.example.com. 10800 IN SOA grac4-gns-vip.grid4.example.com. grac4-gns-vip.grid4.example.com. 264601876 10800 10800 30 120 ;; ADDITIONAL SECTION: grac4-gns-vip.grid4.example.com. 10800 IN A 192.168.5.54 ;; Query time: 2 msec ;; SERVER: 192.168.5.54#53(192.168.5.54) ;; WHEN: Sun Jan 11 17:17:59 CET 2015 ;; MSG SIZE rcvd: 160 If GNS is not configured or running you will get error: can't find grac4-scan: NXDOMAIN [grid@grac41 ~]$ srvctl stop gns [root@ns1 ~]# ping 192.168.5.54 PING 192.168.5.54 (192.168.5.54) 56(84) bytes of data. From 192.168.5.50 icmp_seq=2 Destination Host Unreachable From 192.168.5.50 icmp_seq=3 Destination Host Unreachable From 192.168.5.50 icmp_seq=4 Destination Host Unreachable ^C --- 192.168.5.54 ping statistics --- 4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3944ms pipe 3 [root@ns1 ~]# nslookup grac4-scan Server: 192.168.5.50 Address: 192.168.5.50#53 ** server can't find grac4-scan: NXDOMAIN
Verify subdomain delegation with cluvfy
Starting with Oracle Database 11g release 2 (11.2.0.2), use the cluvfy comp dns component verification command to verify that the Grid Naming Service (GNS) subdomain delegation has been properly set up in the Domain Name Service (DNS) server. Run cluvfy comp dns -server on one node of the cluster. On each node of the cluster run cluvfy comp dns -client to verify the DNS server setup for the cluster. Oh grac41: [root@grac41 ~]# cluvfy comp dns -server -domain grid4.example.com -vipaddress 192.168.5.54/255.255.255.0/eth1 -verbose Verifying DNS Check Starting the test DNS server on IP "192.168.5.54/255.255.255.0/eth1" listening on port 53 Started the IP address "192.168.5.54/255.255.255.0/eth1" on node "grac41" On grac42: [root@grac42 ~]# cluvfy comp dns -client -domain grid4.example.com -vip 192.168.5.54 Verifying DNS Check Checking if the IP address "192.168.5.54" is reachable The IP address "192.168.5.54" is reachable from local node Successfully connected to test DNS server Checking if the test DNS server started on address "192.168.5.54", listening on port 53 can be queried Check output of command "cluvfy comp dns -server" to see if it received IP address lookup for name "grac42.grid4.example.com" Successfully connected to the test DNS server started on address "192.168.5.54", listening on port 53 Checking DNS delegation for the GNS subdomain "grid4.example.com"... Check output of command "cluvfy comp dns -server" to see if it received IP address lookup for name "grac42.grid4.example.com" Successfully verified DNS delegation of the GNS subdomain "grid4.example.com" Verification of DNS Check was successful. --> Server should report Received IP address lookup query for name "grac42.grid4.example.com" Received IP address lookup query for name "grac42.grid4.example.com" On grac43: [root@grac43 ~]# cluvfy comp dns -client -domain grid4.example.com -vip 192.168.5.54 .. Verification of DNS Check was successful. --> Server should report Received IP address lookup query for name "grac43.grid4.example.com" Received IP address lookup query for name "grac43.grid4.example.com" On grac41 [root@grac41 Desktop]# cluvfy comp dns -client -domain grid4.example.com -vip 192.168.5.54 .. Verification of DNS Check was successful. --> Server should report Received IP address lookup query for name "grac41.grid4.example.com" Received IP address lookup query for name "grac41.grid4.example.com"
Setup DHCP server
DHCP configuration file /etc/dhcp/dhcpd.conf : ddns-update-style interim; ignore client-updates; subnet 192.168.5.0 netmask 255.255.255.0 { option routers 192.168.5.1; # Default gateway to be used by DHCP clients option subnet-mask 255.255.255.0; # Default subnet mask to be used by DHCP clients. option ip-forwarding off; # Do not forward DHCP requests. option broadcast-address 192.168.5.255; # Default broadcast address to be used by DHCP client. option domain-name-servers 192.168.5.50; # IP address of the DNS server. option time-offset -19000; # Central Standard Time option ntp-servers 192.168.5.50; # Default NTP server to be used by DHCP clients range 192.168.5.150 192.168.5.254; # Range of IP addresses that can be issued to DHCP client default-lease-time 21600; # Amount of time in seconds that a client may keep the IP address max-lease-time 43200; } /etc/sysconfig/dhcpd # Command line options here DHCPDARGS="eth1" Restart the DHCP server : [root@ns1 network-scripts]# service dhcpd restart
Verify DHCP setup with cluvfy
[root@grac41 ~]# $GRID_HOME/bin/cluvfy comp dhcp -clustername grac4 Verifying DHCP Check Checking if any DHCP server exists on the network... PRVG-5723 : Network CRS resource is configured to use DHCP provided IP addresses Verification of DHCP Check was unsuccessful on all the specified nodes. From Oracle docu : - You must run this command as root. - Do not run this check while the default network Oracle Clusterware resource, configured to use a DHCP-provided IP address, is online (because the VIPs get released and, since the cluster is online, DHCP has provided IP, so there is no need to double the load on the DHCP server). - Before running this command, ensure that the network resource is offline. Use the srvctl stop nodeapps command to bring the network resource offline, if necessary. As we are on a test cluster go ahead and stop the Nodeapps [root@grac41 Desktop]# srvctl stop nodeapps -f [root@grac41 ~]# $GRID_HOME/bin/cluvfy comp dhcp -clustername grac4 -verbose Verifying DHCP Check Checking if any DHCP server exists on the network... Checking if network CRS resource is configured and online Network CRS resource is offline or not configured. Proceeding with DHCP checks. CRS-10009: DHCP server returned server: 192.168.5.50, loan address : 192.168.5.165/255.255.255.0, lease time: 21600 At least one DHCP server exists on the network and is listening on port 67 Checking if DHCP server has sufficient free IP addresses for all VIPs... Sending DHCP "DISCOVER" packets for client ID "grac4-scan1-vip" CRS-10009: DHCP server returned server: 192.168.5.50, loan address : 192.168.5.165/255.255.255.0, lease time: 21600 ... CRS-10012: released DHCP server lease for client ID grac4-scan3-vip on port 67 CRS-10012: released DHCP server lease for client ID grac4-grac41-vip on port 67 DHCP server was able to provide sufficient number of IP addresses The DHCP server response time is within acceptable limits Verification of DHCP Check was successful. Note you can track the lease operation with following OS command [root@ns1 ~]# tail -f /var/lib/dhcpd/dhcpd.leases } lease 192.168.5.164 { starts 0 2015/01/11 17:29:10; ends 0 2015/01/11 17:29:10; tstp 0 2015/01/11 17:29:10; cltt 0 2015/01/11 17:29:10; binding state free; hardware ethernet 00:00:00:00:00:00; uid "\000grac4-grac41-vip"; }
Configure NTP
Configuration script : /etc/ntp.conf restrict default nomodify notrap noquery restrict 127.0.0.1 # -- CLIENT NETWORK ------- restrict 192.168.5.0 mask 255.255.255.0 nomodify notrap # --- OUR TIMESERVERS ----- can't reach NTP servers - build my own server server 0.pool.ntp.org iburst server 1.pool.ntp.org iburst server 127.127.1.0 # --- NTP MULTICASTCLIENT --- # --- GENERAL CONFIGURATION --- # Undisciplined Local Clock. fudge 127.127.1.0 stratum 9 # Drift file. driftfile /var/lib/ntp/drift broadcastdelay 0.008 # Keys file. keys /etc/ntp/keys Restart NTP daemon [root@ns1 network-scripts]# service ntpd restart Shutting down ntpd: [ OK ] Starting ntpd: [ OK ] Verify setup [root@ns1 network-scripts]# ntpq -p remote refid st t when poll reach delay offset jitter ============================================================================== foxtrot.zq1.de 122.227.206.195 3 u 2 64 1 68.504 4608.38 1.115 der.beste.tiger 159.173.11.127 3 u 1 64 1 38.195 4603.43 11.063 LOCAL(0) .LOCL. 9 l 2 64 1 0.000 0.000 0.000
Verify NTP setup with cluvfy
Verify NTP setup with cluvfy [grid@grac41 ~]$ cluvfy comp clocksync Verifying Clock Synchronization across the cluster nodes Checking if Clusterware is installed on all nodes... Check of Clusterware install passed Checking if CTSS Resource is running on all nodes... CTSS resource check passed Querying CTSS for time offset on all nodes... Query of CTSS for time offset passed Check CTSS state started... CTSS is in Observer state. Switching over to clock synchronization checks using NTP Starting Clock synchronization checks using Network Time Protocol(NTP)... NTP Configuration file check started... NTP Configuration file check passed Checking daemon liveness... Liveness check passed for "ntpd" Check for NTP daemon or service alive passed on all nodes NTP daemon slewing option check passed NTP daemon's boot time configuration check for slewing option passed NTP common Time Server Check started... Check of common NTP Time Server passed Clock time offset check from NTP Time Server started... Clock time offset check passed Clock synchronization check using Network Time Protocol(NTP) passed Oracle Cluster Time Synchronization Services check passed Verification of Clock Synchronization across the cluster nodes was successful.