Table of Contents
Generic RAC Portnumber Information
Default Port Port Range Protocol Used for Number CI only? Cluster Synchronization Service daemon (CSSD) 42424 Dynamic UDP Yes Oracle Grid Interprocess Communication (GIPCD) 42424 Dynamic UDP Yes Oracle HA Services daemon (OHASD) 42424 Dynamic UDP Yes Multicast Domain Name Service (MDNSD) 5353 Dynamic UDP/TCP No Oracle Grid Naming Service (GNSD) 53 53 (public) UDP/TCP No Oracle Notification Services (ONS) 6100 (local) Configured TCP No 6200 (remote) manually Port 42424 : CSSD : The Cluster Synchronization Service (CSS) daemon uses a fixed port for node restart advisory messages.This port is used on all interfaces that have broadcast capability. Broadcast occurs only when a node eviction restart is imminent. OHASD : The Oracle High Availability Services (OHAS) daemon starts the Oracle Clusterware stack. GIPCD : A support daemon that enables Redundant Interconnect Usage. Port 5353 : MDNSD : The mDNS process is a background process on Linux and UNIX, and a service on Window, and is necessary for Grid Plug and Play and GNS. Port 53: GNSD : The Oracle Grid Naming Service daemon provides a gateway between the cluster mDNS and external DNS servers. The gnsd process performs name resolution within the cluster. Port 6100/6200 : ONS : Port for ONS, used to publish and subscribe service for communicating information about Fast Application Notification (FAN) events. The FAN notification process uses system events that Oracle Database publishes when cluster servers become unreachable or if network interfaces fail. Use srvctl to modify ONS port Verify port usage at OS level As GNS runs only on a single node the cluster we need to Relocate GNS first : [root@hract21 ~]# srvctl relocate gns -n hract21 [root@hract21 Desktop]# netstat -taupen |grep ":42424 " udp 0 0 192.168.2.255:42424 0.0.0.0:* 0 10361774 11545/ohasd.bin udp 0 0 230.0.1.0:42424 0.0.0.0:* 0 10361773 11545/ohasd.bin udp 0 0 224.0.0.251:42424 0.0.0.0:* 0 10361772 11545/ohasd.bin udp 0 0 192.168.2.255:42424 0.0.0.0:* 501 10361732 11764/gipcd.bin udp 0 0 230.0.1.0:42424 0.0.0.0:* 501 10361731 11764/gipcd.bin udp 0 0 224.0.0.251:42424 0.0.0.0:* 501 10361730 11764/gipcd.bin udp 0 0 192.168.2.255:42424 0.0.0.0:* 501 10361722 11825/ocssd.bin udp 0 0 230.0.1.0:42424 0.0.0.0:* 501 10361721 11825/ocssd.bin udp 0 0 224.0.0.251:42424 0.0.0.0:* 501 10361720 11825/ocssd.bin [root@hract21 Desktop]# netstat -taupen |grep ":53 " udp 0 0 192.168.5.58:53 0.0.0.0:* 0 46593880 5261/gnsd.bin [root@hract21 Desktop]# netstat -taupen |grep ":5353 " udp 0 0 0.0.0.0:5353 0.0.0.0:* 501 1378331 11724/mdnsd.bin udp 0 0 0.0.0.0:5353 0.0.0.0:* 501 1378210 11724/mdnsd.bin udp 0 0 0.0.0.0:5353 0.0.0.0:* 501 1378209 11724/mdnsd.bin udp 0 0 0.0.0.0:5353 0.0.0.0:* 501 1378208 11724/mdnsd.bin [root@hract21 Desktop]# netstat -taupen |grep ":6100 " tcp 0 0 127.0.0.1:6100 0.0.0.0:* LISTEN 501 10419706 31762/ons ..
Prepare Test program JavaUDPServer.java
Source can be found here : Simple Java example of UDP Client/Server communication [root@hract21 JAVA]# javac JavaUDPServer.java Testing when a port is free and our program can successful listen to that port: [root@hract21 JAVA]# java JavaUDPServer 59 Listening on UDP Port: 59 --> press <cntrl>C to terminate the program Testing program when port is already in use [root@hract21 JAVA]# java JavaUDPServer 53 Listening on UDP Port: 53 Jan 31, 2015 4:57:29 PM JavaUDPServer main SEVERE: null java.net.BindException: Address already in use at java.net.PlainDatagramSocketImpl.bind0(Native Method) at java.net.PlainDatagramSocketImpl.bind(PlainDatagramSocketImpl.java:125) at java.net.DatagramSocket.bind(DatagramSocket.java:372)
Case I: Clusterware startup fails as Portnumber: 42424 is in use
Start our test program to block UPD port 42424 [root@hract21 JAVA]# java JavaUDPServer 42424 Listening on UDP Port: 42424 Start CRS and monitor local CRS stack [root@hract21 Desktop]# crsct start crs ***** Local Resources: ***** Resource NAME INST TARGET STATE SERVER STATE_DETAILS --------------------------- ---- ------------ ------------ --------------- ----------------------------------------- ora.asm 1 ONLINE OFFLINE - STABLE ora.cluster_interconnect.haip 1 ONLINE OFFLINE - STABLE ora.crf 1 ONLINE OFFLINE - STABLE ora.crsd 1 ONLINE OFFLINE - STABLE ora.cssd 1 ONLINE OFFLINE hract21 STARTING ora.cssdmonitor 1 ONLINE ONLINE hract21 STABLE ora.ctssd 1 ONLINE OFFLINE - STABLE ora.diskmon 1 OFFLINE OFFLINE - STABLE ora.drivers.acfs 1 ONLINE ONLINE hract21 STABLE ora.evmd 1 ONLINE INTERMEDIATE hract21 STABLE ora.gipcd 1 ONLINE ONLINE hract21 STABLE ora.gpnpd 1 ONLINE ONLINE hract21 STABLE ora.mdnsd 1 ONLINE ONLINE hract21 STABLE ora.storage 1 ONLINE OFFLINE - STABLE --> evmd process remain in status INTERMEDIATE . Local cluster stack doesn't up ! Investigate Trace files: alert.log: 2015-01-31 17:14:54.492 [CSSDAGENT(22642)]CRS-5818: Aborted command 'start' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:9:3} in /u01/app/grid/diag/crs/hract21/crs/trace/ohasd_cssdagent_root.trc. Sat Jan 31 17:14:59 2015 Errors in file /u01/app/grid/diag/crs/hract21/crs/trace/ocssd.trc (incident=1): CRS-8503 [] [] [] [] [] [] [] [] [] [] [] [] gipcd.trc: 2015-01-31 17:20:27.606277 :GIPCHTHR:812046080: gipchaWorkerCreateInterface: created local interface for node 'hract21', haName 'gipcd_ha_name', inf 'udp://192.168.2.121:28764' inf 0x7fef0c190b30 2015-01-31 17:20:27.606350 :GIPCXCPT:812046080: gipcmodNetworkProcessBind: failed to bind endp 0x7fef182d8230 [000000000001e71a] { gipcEndpoint : localAddr 'mcast://224.0.0.251:42424/192.168.2.121', remoteAddr '', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready 0, wobj (nil), sendp 0x7fef182da320 status 13flags 0x20000000, flags-2 0x0, usrFlags 0xc000 }, addr 0x7fef182d8cf0 [000000000001e71c] { gipcAddress : name 'mcast://224.0.0.251:42424/192.168.2.121', objFlags 0x0, addrFlags 0x5 } 2015-01-31 17:20:27.606358 :GIPCXCPT:812046080: gipcmodNetworkProcessBind: slos op : sgipcnMctBind 2015-01-31 17:20:27.606360 :GIPCXCPT:812046080: gipcmodNetworkProcessBind: slos dep : Address already in use (98) 2015-01-31 17:20:27.606361 :GIPCXCPT:812046080: gipcmodNetworkProcessBind: slos loc : bind 2015-01-31 17:20:27.606363 :GIPCXCPT:812046080: gipcmodNetworkProcessBind: slos info: Invalid argument 2015-01-31 17:20:27.606399 :GIPCXCPT:812046080: gipcBindF [gipcInternalEndpoint : gipcInternal.c : 468]: EXCEPTION[ ret gipcretAddressInUse (20) ] failed to bind endp 0x7fef182d8230 [000000000001e71a] { gipcEndpoint : localAddr 'mcast://224.0.0.251:42424/192.168.2.121', remoteAddr '', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready 0, wobj (nil), sendp 0x7fef182da320 status 13flags 0x20000000, flags-2 0x0, usrFlags 0xc000 }, addr 0x7fef182d9a20 [000000000001e721] { gipcAddress : name 'mcast://224.0.0.251:42424/192.168.2.121', objFlags 0x0, addrFlags 0x4 }, flags 0x8000 2015-01-31 17:20:27.606408 :GIPCXCPT:812046080: gipcInternalEndpoint: failed to bind address to endpoint name 'mcast://224.0.0.251:42424/192.168.2.121', ret gipcretAddressInUse (20) 2015-01-31 17:20:27.606426 :GIPCHTHR:812046080: gipchaWorkerUpdateInterface: EXCEPTION[ ret gipcretAddressInUse (20) ] failed to create local interface 'udp://192.168.2.121', 0x7fef0c190b30 { host '', haName 'gipcd_ha_name', local (nil), ip '192.168.2.121', subnet '192.168.2.0', mask '255.255.255.0', mac '08-00-27-4e-c9-bf', ifname 'eth2', numRef 0, numFail 0, idxBoot 0, flags 0x1841 }, hctx 0x10639b0 [0000000000000011] { gipchaContext : host 'hract21', name 'gipcd_ha_name', luid '8c45d6e7-00000000', name2 3aca-bf27-17d5-691e, numNode 0, numInf 1, maxPriority 0, clientMode 1, nodeIncarnation d64c9b7c-06451148 usrFlags 0x0, flags 0x2d65 } 2015-01-31 17:20:27.606432 :GIPCHGEN:812046080: gipchaInterfaceDisable: disabling interface 0x7fef0c190b30 { host '', haName 'gipcd_ha_name', local (nil), ip '192.168.2.121', subnet '192.168.2.0', mask '255.255.255.0', mac '08-00-27-4e-c9-bf', ifname 'eth2', numRef 0, numFail 0, idxBoot 0, flags 0x1841 } 2015-01-31 17:20:27.606438 :GIPCHDEM:812046080: gipchaWorkerCleanInterface: performing cleanup of disabled interface 0x7fef0c190b30 { host '', haName 'gipcd_ha_name', local (nil), ip '192.168.2.121', subnet '192.168.2.0', mask '255.255.255.0', mac '08-00-27-4e-c9-bf', ifname 'eth2', numRef 0, numFail 0, idxBoot 0, flags 0x1861 } 2015-01-31 17:20:27.60 Investigate the error more in detail : gipcmodNetworkProcessBind: slos dep : Address already in use (98) [root@hract21 Desktop]# cat /usr/include/asm-generic/errno.h | grep 98 #define EADDRINUSE 98 /* Address already in use */ Locate the port number : gipcEndpoint : localAddr 'mcast://224.0.0.251:42424/192.168.2.121 --> 42424 is the port --> CW can't listen on port 42424 ! Locate the blocking process at OS level [root@hract21 Desktop]# netstat -taupen |grep ":42424 " udp 0 0 :::42424 .... 22338/java [root@hract21 Desktop]# ps -elf | grep 22338 0 S root 22338 26783 0 80 0 - 438331 futex_ 17:04 pts/12 00:00:01 java JavaUDPServer 42424 --> Yep our java program blocks CW from comming up ! Kill java program and restart CW [root@hract21 Desktop]# kill -9 22338 [root@hract21 Desktop]# crsctl stop crs -f [root@hract21 Desktop]# crsctl start crs
Case II: Clusterware startup fails as Portnumber 5353 is in use
Start our test program and block MDSND port 5353 [root@hract21 JAVA]# java JavaUDPServer 5353 Listening on UDP Port: 5353 ***** Local Resources: ***** Resource NAME INST TARGET STATE SERVER STATE_DETAILS --------------------------- ---- ------------ ------------ --------------- ----------------------------------------- ora.asm 1 ONLINE ONLINE hract21 STABLE ora.cluster_interconnect.haip 1 ONLINE ONLINE hract21 STABLE ora.crf 1 ONLINE ONLINE hract21 STABLE ora.crsd 1 ONLINE ONLINE hract21 STABLE ora.cssd 1 ONLINE ONLINE hract21 STABLE ora.cssdmonitor 1 ONLINE ONLINE hract21 STABLE ora.ctssd 1 ONLINE ONLINE hract21 OBSERVER,STABLE ora.diskmon 1 OFFLINE OFFLINE - STABLE ora.drivers.acfs 1 ONLINE ONLINE hract21 STABLE ora.evmd 1 ONLINE ONLINE hract21 STABLE ora.gipcd 1 ONLINE ONLINE hract21 STABLE ora.gpnpd 1 ONLINE ONLINE hract21 STABLE ora.mdnsd 1 ONLINE INTERMEDIATE hract21 STABLE ora.storage 1 ONLINE ONLINE hract21 STABLE --> MDMSD daemon doesn' start mdnsd.trc : Oracle Database 12c Clusterware Release 12.1.0.2.0 - Production Copyright 1996, 2014 Oracle. All rights reserved. CLSB:2559100480: Argument count (argc) for this daemon is 1 CLSB:2559100480: Argument 0 is: /u01/app/121/grid/bin/mdnsd.bin 2015-01-31 17:40:17.131516 : CLSDMT:2554820352: PID for the Process [9863], connkey 9 2015-01-31 17:40:18.042329 : MDNS:2559100480: mdnsd interface eth0 (0x2 AF=2 f=0x1043 mcast=-1) 192.168.1.9 mask 255.255.255.0 FAILED. Error 98 (Address already in use) 2015-01-31 17:40:18.043191 : MDNS:2559100480: mdnsd interface eth1 (0x3 AF=2 f=0x1043 mcast=-1) 192.168.5.121 mask 255.255.255.0 FAILED. Error 98 (Address already in use) 2015-01-31 17:40:18.046952 : MDNS:2559100480: mdnsd interface eth1:1 (0x3 AF=2 f=0x1043 mcast=-1) 192.168.5.241 mask 255.255.255.0 FAILED. Error 98 (Address already in use) 2015-01-31 17:40:18.047574 : MDNS:2559100480: mdnsd interface eth1:2 (0x3 AF=2 f=0x1043 mcast=-1) 192.168.5.242 mask 255.255.255.0 FAILED. Error 98 (Address already in use) 2015-01-31 17:40:18.047597 : MDNS:2559100480: mdnsd interface eth2 (0x4 AF=2 f=0x1043 mcast=-1) 192.168.2.121 mask 255.255.255.0 FAILED. Error 98 (Address already in use) 2015-01-31 17:40:18.047612 : MDNS:2559100480: mdnsd interface eth2:1 (0x4 AF=2 f=0x1043 mcast=-1) 169.254.213.86 mask 255.255.0.0 FAILED. Error 98 (Address already in use) 2015-01-31 17:40:18.049171 : MDNS:2559100480: mdnsd interface eth3 (0x5 AF=2 f=0x1043 mcast=-1) 192.168.3.121 mask 255.255.255.0 FAILED. Error 98 (Address already in use) 2015-01-31 17:40:18.049222 : MDNS:2559100480: mdnsd interface lo (0x1 AF=2 f=0x49 mcast=-1) 127.0.0.1 mask 255.0.0.0 FAILED. Error 98 (Address already in use) 2015-01-31 17:40:18.049236 : MDNS:2559100480: Error! No valid netowrk interfaces found to setup mDNS. 2015-01-31 17:40:18.049240 : MDNS:2559100480: Oracle mDNSResponder ver. mDNSResponder-1076 (Jun 30 2014 19:39:45) , init_rv=-65537 2015-01-31 17:40:18.049335 : MDNS:2559100480: stopping --> Here we only get the error : Address already in use but info about the portnumber. We need to reference above list and remember that MSDNS is running on port 5353 Now we can locate the blocking process , kill that process and restart clusterware [root@hract21 Desktop]# netstat -taupen |grep ":5353 " udp 0 0 :::5353 ... 50111629 7252/java Again our java program prevents CW from startup. Kill the that process and resart CW. [root@hract21 Desktop]# kill -9 7252
Case III: Investigate GNS startup problem due to Error: Address already in use
Relocate GNS to a different host [root@hract21 Desktop]# srvctl relocate gns -n hract23 ora.gns 1 ONLINE ONLINE hract23 STABLE ora.gns.vip 1 ONLINE ONLINE hract23 STABLE Now occupy port 53 by running our JAVA program: [root@hract21 JAVA]# java JavaUDPServer 53 Listening on UDP Port: 53 Now try to bring back the GNS [root@hract21 Desktop]# srvctl relocate gns -n hract21 ***** Cluster Resources: ***** Resource NAME INST TARGET STATE SERVER STATE_DETAILS --------------------------- ---- ------------ ------------ --------------- ----------------------------------------- ora.gns 1 ONLINE OFFLINE hract21 STARTING ora.gns.vip 1 ONLINE ONLINE hract21 STABLE --> GNS is in status STARTING but doesn't come up gnsd.trc : 2015-01-31 18:09:13.518516 :GIPCXCPT:255158016: gipcmodNetworkProcessBind: slos op : sgipcnTcpBind 2015-01-31 18:09:13.518518 :GIPCXCPT:255158016: gipcmodNetworkProcessBind: slos dep : Address already in use (98) 2015-01-31 18:09:13.518520 :GIPCXCPT:255158016: gipcmodNetworkProcessBind: slos loc : bind 2015-01-31 18:09:13.518521 :GIPCXCPT:255158016: gipcmodNetworkProcessBind: slos info: addr '192.168.5.58:53' 2015-01-31 18:09:13.518577 :GIPCXCPT:255158016: gipcBindF [gipcInternalEndpoint : gipcInternal.c : 468]: EXCEPTION[ ret gipcretAddressInUse (20) ] failed to bind endp 0x7ff7000034c0 [0000000000001fc6] { gipcEndpoint : localAddr 'udp://192.168.5.58:53', remoteAddr '', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready 0, wobj (nil), sendp 0x7ff7000050f0 status 13flags 0x20008000, flags-2 0x0, usrFlags 0x24000 }, addr 0x7ff7000047f0 [0000000000001fcd] { gipcAddress : name 'udp://192.168.5.58:53', objFlags 0x0, addrFlags 0x4 }, flags 0x20000 2015-01-31 18:09:13.518589 :GIPCXCPT:255158016: gipcInternalEndpoint: failed to bind address to endpoint name 'udp://192.168.5.58:53', ret gipcretAddressInUse (20) 2015-01-31 18:09:13.518608 :GIPCXCPT:255158016: gipcEndpointF [clsgngipcCreateEndpointInternal : clsgngipc.c : 2008]: EXCEPTION[ ret gipcretAddressInUse (20) ] failed endp create ctx 0x7ff7196f3c80 [0000000000001e99] { gipcContext : traceLevel 2, fieldLevel 0x0, numDead 0, numPending 0, numZombie 0, numObj 4, numWait 0, numReady 0, wobj 0x7ff7196f1c10, hgid 0000000000001e9a, flags 0x1a, objFlags 0x0 }, name 'udp://192.168.5.58:53', flags 0x24000 2015-01-31 18:09:13.518728 : GNS:255158016: Resolve::clsgndnsCreateContainerCallback: (:CLSGN01163:) Error - Address in use: port 53 address "192.168.5.58". 1: clskec:has:CLSGN:208 2 args[has:CLSGN:208][udp://192.168.5.58:53] 2: clskec:has:gipc:20 1 args[has:gipc:20] 3: clskec:has:CLSU:910 4 args[has][mod=gipcInternalEndpoint][loc=473][msg=failed to bind address to endpoint name 'udp://192.168.5.58:53'] 2015-01-31 18:09:13.518769 : GNS:255158016: Resolve::clsgndnsCreateContainer: (:CLSGN00927:) failed to listen on all addresses - throwing error. default:255158016: listen failed with 1 errors 1: clskec:has:CLSGN:208 3 args[has:CLSGN:208][192.168.5.58][53] The following error messages tell use Linux errno code and the related portnumber : 2015-01-31 18:09:13.518518 :GIPCXCPT:255158016: gipcmodNetworkProcessBind: slos dep : Address already in use (98) 2015-01-31 18:09:13.518521 :GIPCXCPT:255158016: gipcmodNetworkProcessBind: slos info: addr '192.168.5.58:53' Again locate the port number and kill the process [root@hract21 Desktop]# netstat -taupen |grep ":53 " udp 16128 0 :::53 ... 51417680 23723/java Again kill the process which holds the port number and restart CW [root@hract21 Desktop]# kill -9 23723 Now test whether Relocated GNS works again [root@hract21 ~]# srvctl relocate gns -n hract21 ***** Cluster Resources: ***** Resource NAME INST TARGET STATE SERVER STATE_DETAILS --------------------------- ---- ------------ ------------ --------------- ----------------------------------------- ora.gns 1 ONLINE ONLINE hract21 STABLE ora.gns.vip 1 ONLINE ONLINE hract21 STABL
Complete Portnumber Usage of a working RAC system
[root@hract21 ~]# netstat -taupen |grep 192.168
tcp 0 0 192.168.5.242:1521 0.0.0.0:* LISTEN 501 50803141 17310/tnslsnr
tcp 0 0 192.168.5.241:1521 0.0.0.0:* LISTEN 501 50793916 17258/tnslsnr
tcp 0 0 192.168.5.121:1521 0.0.0.0:* LISTEN 501 50793894 17258/tnslsnr
tcp 0 0 192.168.2.121:1522 0.0.0.0:* LISTEN 501 50790436 17212/tnslsnr
tcp 0 0 192.168.2.121:61020 0.0.0.0:* LISTEN 0 50773311 16994/osysmond.bin
tcp 0 0 192.168.5.121:42942 0.0.0.0:* LISTEN 501 50724207 16454/gipcd.bin
tcp 0 0 192.168.5.58:39839 0.0.0.0:* LISTEN 0 51856456 27381/gnsd.bin
tcp 0 0 192.168.5.232:36063 0.0.0.0:* LISTEN 502 8145376 596/exectask
tcp 0 0 192.168.5.121:15043 0.0.0.0:* LISTEN 0 50730332 16281/ohasd.bin
tcp 0 0 192.168.5.121:42942 192.168.5.123:28657 ESTABLISHED 501 50841598 16454/gipcd.bin
tcp 0 0 192.168.5.241:1521 192.168.5.121:55119 ESTABLISHED 501 50829166 17258/tnslsnr
tcp 0 0 192.168.2.121:46847 192.168.2.122:1522 ESTABLISHED 0 50774509 17012/crsd.bin
tcp 0 0 192.168.2.121:1522 192.168.2.123:60331 ESTABLISHED 501 50795614 17212/tnslsnr
tcp 0 0 192.168.2.121:1522 192.168.2.121:16025 ESTABLISHED 501 50829535 17212/tnslsnr
tcp 0 0 192.168.2.121:1522 192.168.2.122:54611 ESTABLISHED 501 50796842 17212/tnslsnr
tcp 0 0 192.168.2.121:46865 192.168.2.122:1522 ESTABLISHED 501 50829527 17468/asm_lreg_+ASM
tcp 0 0 192.168.5.242:1521 192.168.5.121:61101 ESTABLISHED 501 50838159 17310/tnslsnr
tcp 0 0 192.168.5.121:42942 192.168.5.122:32304 ESTABLISHED 501 50841582 16454/gipcd.bin
tcp 1 0 192.168.1.9:39471 80.150.192.73:80 CLOSE_WAIT 0 50900520 4786/clock-applet
tcp 0 0 192.168.2.121:1522 192.168.2.121:16024 ESTABLISHED 501 50829534 17212/tnslsnr
tcp 0 0 192.168.2.121:16024 192.168.2.121:1522 ESTABLISHED 501 50829529 17468/asm_lreg_+ASM
tcp 0 0 192.168.2.121:16025 192.168.2.121:1522 ESTABLISHED 501 50829531 17468/asm_lreg_+ASM
tcp 0 0 192.168.2.121:28139 192.168.2.123:1522 ESTABLISHED 501 50829525 17468/asm_lreg_+ASM
tcp 0 0 192.168.5.121:64227 192.168.5.122:35547 ESTABLISHED 501 50790718 16454/gipcd.bin
tcp 0 0 192.168.5.121:21046 192.168.5.123:6200 ESTABLISHED 501 50787900 17215/ons
tcp 0 0 192.168.5.121:59844 192.168.5.50:22 ESTABLISHED 0 44509382 13726/ssh
tcp 0 0 192.168.5.121:61101 192.168.5.242:1521 ESTABLISHED 502 50838158 17721/ora_lreg_bank
tcp 0 0 192.168.5.58:39839 192.168.5.121:34266 TIME_WAIT 0 0 -
tcp 0 0 192.168.5.121:16432 192.168.5.122:6200 ESTABLISHED 501 50787901 17215/ons
tcp 0 0 192.168.2.121:39861 192.168.2.123:61021 ESTABLISHED 0 50769440 16994/osysmond.bin
tcp 0 0 192.168.5.121:55125 192.168.5.241:1521 ESTABLISHED 502 50837652 17721/ora_lreg_bank
tcp 0 0 192.168.5.121:55119 192.168.5.241:1521 ESTABLISHED 501 50829165 17468/asm_lreg_+ASM
tcp 0 0 192.168.5.241:1521 192.168.5.121:55125 ESTABLISHED 501 50837653 17258/tnslsnr
tcp 0 0 192.168.5.121:10242 192.168.5.123:17701 ESTABLISHED 501 50790723 16454/gipcd.bin
tcp 0 0 192.168.5.121:55728 192.168.5.123:22 ESTABLISHED 0 27679552 25184/ssh
udp 0 0 192.168.2.121:35570 0.0.0.0:* 0 50731287 16281/ohasd.bin
udp 0 0 192.168.2.121:51962 0.0.0.0:* 0 50751183 16922/octssd.bin
udp 0 0 192.168.2.255:42424 0.0.0.0:* 501 50734962 16537/ocssd.bin
udp 0 0 192.168.2.255:42424 0.0.0.0:* 0 50731290 16281/ohasd.bin
udp 0 0 192.168.2.255:42424 0.0.0.0:* 501 50725223 16454/gipcd.bin
udp 0 0 192.168.2.121:15891 0.0.0.0:* 501 50734959 16537/ocssd.bin
udp 0 0 192.168.2.121:12075 0.0.0.0:* 501 50782505 16408/evmd.bin
udp 0 0 192.168.5.58:53 0.0.0.0:* 0 51856599 27381/gnsd.bin
udp 0 0 192.168.5.58:123 0.0.0.0:* 38 51843931 1291/ntpd
udp 0 0 192.168.5.242:123 0.0.0.0:* 38 50803109 1291/ntpd
udp 0 0 192.168.5.241:123 0.0.0.0:* 38 50793859 1291/ntpd
udp 0 0 192.168.3.121:123 0.0.0.0:* 0 43573989 1291/ntpd
udp 0 0 192.168.2.121:123 0.0.0.0:* 0 43573987 1291/ntpd
udp 0 0 192.168.5.121:123 0.0.0.0:* 0 43573984 1291/ntpd
udp 0 0 192.168.1.9:123 0.0.0.0:* 0 43573983 1291/ntpd
udp 0 0 192.168.2.121:53498 0.0.0.0:* 0 50776026 17012/crsd.bin
udp 0 0 192.168.2.121:45379 0.0.0.0:* 501 50725220 16454/gipcd.bin
Troubleshooting hint CW startproblems due to Address already in use errors
Before CW startup verify the the following ports are not in use at all [root@hract21 Desktop]# netstat -taupen |grep ":42424 " [root@hract21 Desktop]# netstat -taupen |grep ":5353 " [root@hract21 Desktop]# netstat -taupen |grep ":53 " [root@hract21 Desktop]# netstat -taupen |egrep ":6100 |:6200" If you find any processes not belonging to the Oracle Clusterware stack you need to kill/stop these processes If having problem with Clusterware startup or CW components startup ( GSD, VIPs ) you may check your clusterware tracefils for "Address already in use" Error . Note the tracefile location has changed for RAC 12.1.0.2 : [grid@hract21 trace]$ grep -l "Address already in use" * gipcd.trc gnsd.trc mdnsd.trc ocssd.trc ohasd.trc Now find details: # grep "Address already in use" ohasd.trc mdnsd.trc ocssd.trc gnsd.trc gipcd.trc gnsd.trc | grep "2015-01-31 17" ohasd.trc:2015-01-31 17:30:16.613432 :GIPCXCPT:2420897536: gipcmodNetworkProcessBind: slos dep : Address already in use (98) mdnsd.trc:2015-01-31 17:40:18.049222 : MDNS:2559100480: mdnsd interface lo (0x1 AF=2 f=0x49 mcast=-1) 127.0.0.1 mask 255.0.0.0 FAILED. Error 98 (Address already in use) ocssd.trc:2015-01-31 17:04:56.013085 :GIPCXCPT:3986515712: gipcmodNetworkProcessBind: slos dep : Address already in use (98) gipcd.trc:2015-01-31 17:06:25.204775 :GIPCXCPT:812046080: gipcmodNetworkProcessBind: slos dep : Address already in use (98) gnsd.trc:2015-01-31 18:09:13.518518 :GIPCXCPT:2551580oblematic Portnumber and 16: gipcmodNetworkProcessBind: slos dep : Address already in use (98) For mdnsd.trc we already know the port number : 5353 For ohasd.trc, ocssd.trc, gipcd.trc the port number is : 42424 For GNS the tracefiles provides details about the problematic Portnumber and IP-adress [grid@hract21 trace]$ grep gipcmodNetworkProcessBind gnsd.trc 2015-01-31 18:09:13.518483 :GIPCXCPT:255158016: gipcmodNetworkProcessBind: failed to bind endp 0x7ff7000034c0 [0000000000001fc6] { gipcEndpoint : localAddr 'udp://192.168.5.58:53', remoteAddr '', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready 0, wobj (nil), sendp 0x7ff7000050f0 status 13flags 0x20008000, flags-2 0x0, usrFlags 0x24000 }, addr 0x7ff7000038b0 [0000000000001fc8] { gipcAddress : name 'udp://192.168.5.58:53', objFlags 0x0, addrFlags 0x5 } 2015-01-31 18:09:13.518516 :GIPCXCPT:255158016: gipcmodNetworkProcessBind: slos op : sgipcnTcpBind 2015-01-31 18:09:13.518518 :GIPCXCPT:255158016: gipcmodNetworkProcessBind: slos dep : Address already in use (98) 2015-01-31 18:09:13.518520 :GIPCXCPT:255158016: gipcmodNetworkProcessBind: slos loc : bind 2015-01-31 18:09:13.518521 :GIPCXCPT:255158016: gipcmodNetworkProcessBind: slos info: addr '192.168.5.58:53'