Tracing an unplanned status change for resource ora.LISTENER.lsnr
Check resource properties:
[grid@grac41 ~]$ crsctl status resource ora.LISTENER.lsnr -f
NAME=ora.LISTENER.lsnr
TYPE=ora.listener.type
STATE=ONLINE
TARGET=ONLINE
ACTION_SCRIPT=%CRS_HOME%/bin/racgwrap%CRS_SCRIPT_SUFFIX%
AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX%
ALIAS_NAME=ora.%CRS_CSS_NODENAME_LOWER_CASE%.LISTENER_%CRS_CSS_NODENAME_UPPER_CASE%.lsnr
AUTO_START=restore
CHECK_INTERVAL=60
--> The agent binary is : $GRID_HOME/bin/oraagent.bin and the status is checked every second 60 seconds
Every 60 seconds the listener status gets checked ( lsnrctl status LISTENER )
2014-04-25 14:15:22.372: [ora.LISTENER.lsnr][582964992]{1:51811:6685} [check] Utils:execCmd action = 3 flags = 38 ohome = (null) cmdname = lsnrctl.
2014-04-25 14:15:22.477: [ora.LISTENER.lsnr][582964992]{1:51811:6685} [check] execCmd ret = 0
..
2014-04-25 14:16:22.377: [ora.LISTENER.lsnr][582964992]{1:51811:6685} [check] Utils:execCmd action = 3 flags = 38 ohome = (null) cmdname = lsnrctl.
2014-04-25 14:16:22.481: [ora.LISTENER.lsnr][582964992]{1:51811:6685} [check] execCmd ret = 0
--> Return code is 0
You can verify/test the status by running :
[grid@grac41 ~]$ lsnrctl status LISTENER
...
Instance "grac41", status READY, has 1 handler(s) for this service...
Service "grac4XDB" has 1 instance(s).
Instance "grac41", status READY, has 1 handler(s) for this service...
The command completed successfully
[grid@grac41 ~]$ echo $status
0
Kill local listener
[root@grac41 oraagent_oracle]# ps -elf | grep tns
4 S daemon 13322 1 0 80 0 - 54027 poll_s 11:59 ? 00:00:00 /u01/app/11204/grid/bin/tnslsnr LISTENER -inherit
0 S grid 19482 1 0 80 0 - 54159 poll_s 08:14 ? 00:00:00 /u01/app/11204/grid/bin/tnslsnr LISTENER_SCAN2 -inherit
[root@grac41 oraagent_oracle]# kill -9 13322
--> This triggers an unplanned resource status change for resource ora.LISTENER.lsnr on grac41
..
2014-04-25 14:30:37.956: [ora.LISTENER.lsnr][582964992]{1:51811:6939} [check] Utils:execCmd action = 3 flags = 38 ohome = (null) cmdname = lsnrctl.
2014-04-25 14:30:39.683: [ora.LISTENER.lsnr][582964992]{1:51811:6939} [check] (:CLSN00010:)Utils:execCmd scls_process_join() uret 1
2014-04-25 14:30:39.683: [ora.LISTENER.lsnr][582964992]{1:51811:6939} [check] execCmd ret = 1
--> Listener status check returns 1 instead of 0 - Agent frame work immediate restart the listener resource
2014-04-25 14:30:39.684: [ AGFW][1088186112]{1:51811:6939} ora.LISTENER.lsnr grac41 1 state changed from: ONLINE to: OFFLINE
2014-04-25 14:30:39.684: [ AGFW][1088186112]{0:17:25} Generating new Tint for unplanned state change. Original Tint: {1:51811:6939}
--> Creating new TINT means we have a unplanned resoruce status change before
2014-04-25 14:30:39.684: [ AGFW][1088186112]{0:17:25} Agent sending message to PE: RESOURCE_STATUS[Proxy] ID 20481:6019
2014-04-25 14:30:39.707: [ AGFW][1088186112]{0:17:25} Agent received the message: RESOURCE_START[ora.LISTENER.lsnr grac41 1] ID 4098:37563
2014-04-25 14:30:39.707: [ AGFW][1088186112]{0:17:25} Preparing START command for: ora.LISTENER.lsnr grac41 1
2014-04-25 14:30:39.707: [ AGFW][1088186112]{0:17:25} ora.LISTENER.lsnr grac41 1 state changed from: OFFLINE to: STARTING
--> Agent frame work changes listener status from: ONLINE to: OFFLINE and restarts listener
Agent frame work changes listener status from: OFFLINE to: STARTING
..
2014-04-25 14:30:43.178: [ AGFW][582964992]{0:17:25} Command: start for resource: ora.LISTENER.lsnr grac41 1 completed with status: SUCCESS
2014-04-25 14:30:43.178: [ AGFW][1088186112]{0:17:25} Agent sending reply for: RESOURCE_START[ora.LISTENER.lsnr grac41 1] ID 4098:37563
..
2014-04-25 14:30:43.306: [ AGFW][1088186112]{0:17:25} ora.LISTENER.lsnr grac41 1 state changed from: STARTING to: ONLINE
2014-04-25 14:30:43.306: [ AGFW][1088186112]{0:17:25} Started implicit monitor for [ora.LISTENER.lsnr grac41 1] interval=60000 delay=60000
2014-04-25 14:30:43.306: [ AGFW][1088186112]{0:17:25} Agent sending last reply for: RESOURCE_START[ora.LISTENER.lsnr grac41 1] ID 4098:37563