Clusterware_for_3rd_party

Using Oracle Clusterware 11.2 to Protect 3rd Party Applications

To protect your standard application you may need the following:

  • Clustered filesystem like ACFS
  • Failover and automatic restart capabilities by avoiding any  single point of failure

In the sample below we will harden the standard Apache application by using the following Software Versions :

  • GRID: 11.2.0.3.4
  • OEL 6.3
  • VirtualBox 4.2.14
  •  Apache/2.2.15 (Unix)

Following Clusterware parts are used :

  • Create a VIP
  • Create a Server Pool
  • Create a Cluster Resource with CARDINALITY=1

Network VIP used for Apache Listener : 192.168.1.91 -Verify that ping returns an error 
# ping 192.168.1.91
PING 192.168.1.91 (192.168.1.91) 56(84) bytes of data.
From 192.168.1.61 icmp_seq=1 Destination Host Unreachable
From 192.168.1.61 icmp_seq=2 Destination Host Unreachable

Create our  VIP - by checking the  current Network and creating the related VIP
$ srvctl config network
Network exists: 1/192.168.1.0/255.255.255.0/eth0, type dhcp
# $GRID_HOME/bin/appvipcfg create -network=1 -ip=192.168.1.91 -vipname=MyTestVIP -user=root
Production Copyright 2007, 2008, Oracle.All rights reserved
2013-08-07 13:07:46: Creating Resource Type
2013-08-07 13:07:46: Executing /u01/app/11203/grid/bin/crsctl add type app.appvip_net1.type -basetype ora.cluster_vip_net1.type -file /u01/app/11203/grid/crs/template/appvip.type
2013-08-07 13:07:46: Executing cmd: /u01/app/11203/grid/bin/crsctl add type app.appvip_net1.type -basetype ora.cluster_vip_net1.type -file /u01/app/11203/grid/crs/template/appvip.type
2013-08-07 13:07:47: Create the Resource
2013-08-07 13:07:47: Executing /u01/app/11203/grid/bin/crsctl add resource MyTestVIP -type app.appvip_net1.type -attr "USR_ORA_VIP=192.168.1.91,START_DEPENDENCIES=hard(ora.net1.network) pullup(ora.net1.network),STOP_DEPENDENCIES=hard(ora.net1.network),ACL='owner:root:rwx,pgrp:root:r-x,other::r--,user:root:r-x',HOSTING_MEMBERS=grac1.example.com,APPSVIP_FAILBACK="
2013-08-07 13:07:47: Executing cmd: /u01/app/11203/grid/bin/crsctl add resource MyTestVIP -type app.appvip_net1.type -attr "USR_ORA_VIP=192.168.1.91,START_DEPENDENCIES=hard(ora.net1.network) pullup(ora.net1.network),STOP_DEPENDENCIES=hard(ora.net1.network),ACL='owner:root:rwx,pgrp:root:r-x,other::r--,user:root:r-x',HOSTING_MEMBERS=grac1.example.com,APPSVIP_FAILBACK="
# $GRID_HOME/bin/crsctl stat res MyTestVIP
NAME=MyTestVIP
TYPE=app.appvip_net1.type
TARGET=OFFLINE
STATE=OFFLINE
..
Start our VIP resource:
#  $GRID_HOME/bin/crsctl start res MyTestVIP
CRS-2672: Attempting to start 'MyTestVIP' on 'grac1'
CRS-2676: Start of 'MyTestVIP' on 'grac1' succeeded
#  $GRID_HOME/bin/crsctl stat res MyTestVIP -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
MyTestVIP
      1        ONLINE  ONLINE       grac1 

Verify that our VIP is working  with ping :
# ping 192.168.1.91
PING 192.168.1.91 (192.168.1.91) 56(84) bytes of data.
64 bytes from 192.168.1.91: icmp_seq=1 ttl=64 time=0.032 ms
64 bytes from 192.168.1.91: icmp_seq=2 ttl=64 time=0.031 ms
64 bytes from 192.168.1.91: icmp_seq=3 ttl=64 time=0.029 ms

Change Apache Listen address to our VIP and restart Apache 
# grep Listen   /etc/httpd/conf/httpd.conf
Listen 192.168.1.91:80
# apachectl restart
Test your apache by using the following URL: http://192.168.1.91/

Test VIP Relocation to  grac2
Stop apache on grac1
[root@grac1]# apachectl stop
--> Should now get Unable to connect for testing URL:  http://192.168.1.91/
Configure apache on grac2 and start it
# grep Listen /etc/httpd/conf/httpd.conf
Listen 192.168.1.91:80
# apachectl start
(99)Cannot assign requested address: make_sock: could not bind to address 192.168.1.91:80
no listening sockets available, shutting down
Unable to open logs
--> Need relocate VIP first to grac2
# $GRID_HOME/bin/crs_relocate MyTestVIP -c grac2
Attempting to stop `MyTestVIP` on member `grac1`
Stop of `MyTestVIP` on member `grac1` succeeded.
Attempting to start `MyTestVIP` on member `grac2`
Start of `MyTestVIP` on member `grac2` succeeded.
# $GRID_HOME/bin/crsctl stat res MyTestVIP -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
MyTestVIP
      1        ONLINE  ONLINE       grac2      
Now apache start should on grac2   
# apachectl start
Verify this by testing following URL http://192.168.1.91/ 
should display the Apache init page
Note as we use a VIP this URL should work from any system in your network

Create action script for our cluster resource on both Nodes: 
# cat /usr/local/bin/apache.scr
#!/bin/bash
HTTPDCONFLOCATION=/etc/httpd/conf/httpd.conf
WEBPAGECHECK=http://192.168.1.91:80/icons/apache_pb.gif
case $1 in
'start')
/usr/sbin/apachectl -k start -f $HTTPDCONFLOCATION
RET=$?
;;
'stop')
/usr/sbin/apachectl -k stop
RET=$?
;;
'clean')
/usr/sbin/apachectl -k stop
RET=$?
;;
'check')
/usr/bin/wget -q --delete-after $WEBPAGECHECK
RET=$?
;;
*)
RET=0
;;
esac
# 0: success; 1 : error
if [ $RET -eq 0 ]; then
exit 0
else
exit 1
fi

Test the check part of above  action script - Note only if apache is down on both nodes the check routine should return  failure code : 1
# chmod 755 /usr/local/bin/apache.scr
root@grac2 ~]#  /usr/local/bin/apache.scr check
[root@grac2 ~]# echo $?
0
[root@grac2 ~]#   /usr/local/bin/apache.scr stop
[root@grac2 ~]# echo $?
0
[root@grac2 ~]# /usr/local/bin/apache.scr check
[root@grac2 ~]#  echo $?
1
[root@grac2 ~]# /usr/local/bin/apache.scr start
[root@grac2 ~]#  echo $?
0
[root@grac2 ~]# /usr/local/bin/apache.scr check
[root@grac2 ~]#  echo $?
0
Repeat testing on grac1 - but relocate VIP first
# $GRID_HOME/bin/crs_relocate MyTestVIP -c grac1
Attempting to stop `MyTestVIP` on member `grac2`
Stop of `MyTestVIP` on member `grac2` succeeded.
Attempting to start `MyTestVIP` on member `grac1`
...
Create a Server pool named myApache_sp
$ crsctl  add serverpool  myApache_sp -attr "PARENT_POOLS=Generic,SERVER_NAMES=grac1 grac2"
$ crsctl status server -f
NAME=grac1
STATE=ONLINE
ACTIVE_POOLS=Generic ora.GRACE2 myApache_sp
STATE_DETAILS=
NAME=grac2
STATE=ONLINE
ACTIVE_POOLS=Generic ora.GRACE2 myApache_sp
STATE_DETAILS=
Create Cluster resource and verify this resource  - don't forget following parameters: 
START_DEPENDENCIES=hard(MyTestVIP) pullup(MyTestVIP) 
STOP_DEPENDENCIES=hard(intermediate:MyTestVIP)
CARDINALITY=1
# $GRID_HOME/bin/crsctl add resource myApache -type cluster_resource -attr "ACTION_SCRIPT=/usr/local/bin/apache.scr,PLACEMENT='restricted',SERVER_POOLS=myApache_sp,CHECK_INTERVAL='30',RESTART_ATTEMPTS='2',
START_DEPENDENCIES=hard(MyTestVIP) pullup(MyTestVIP), STOP_DEPENDENCIES=hard(intermediate:MyTestVIP),CARDINALITY=1 "
# $GRID_HOME/bin/crsctl status resource myApache -f
NAME=myApache
TYPE=cluster_resource
STATE=ONLINE
TARGET=ONLINE
ACL=owner:root:rwx,pgrp:root:r-x,other::r--
ACTION_FAILURE_TEMPLATE=
ACTION_SCRIPT=/usr/local/bin/apache.scr
ACTIVE_PLACEMENT=0
AGENT_FILENAME=%CRS_HOME%/bin/scriptagent
AUTO_START=restore
CARDINALITY=1
CARDINALITY_ID=0
CHECK_INTERVAL=30
CREATION_SEED=65
DEFAULT_TEMPLATE=
DEGREE=1
DESCRIPTION=
ENABLED=1
FAILOVER_DELAY=0
FAILURE_INTERVAL=0
FAILURE_THRESHOLD=0
HOSTING_MEMBERS=
ID=myApache
LOAD=1
LOGGING_LEVEL=1
NOT_RESTARTING_TEMPLATE=
OFFLINE_CHECK_INTERVAL=0
PLACEMENT=restricted
PROFILE_CHANGE_TEMPLATE=
RESTART_ATTEMPTS=2
SCRIPT_TIMEOUT=60
SERVER_POOLS=myApache_sp
START_DEPENDENCIES=hard(MyTestVIP) pullup(MyTestVIP)
START_TIMEOUT=0
STATE_CHANGE_TEMPLATE=
STOP_DEPENDENCIES=hard(intermediate:MyTestVIP)
STOP_TIMEOUT=0
UPTIME_THRESHOLD=1h

# $GRID_HOME/bin/crsctl start resource myApache
CRS-2672: Attempting to start 'myApache' on 'grac1'
CRS-2676: Start of 'myApache' on 'grac1' succeeded
# $GRID_HOME/bin/crsctl status resource myApache -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
myApache
      1        ONLINE  ONLINE       grac1 

Relocate cluster resource myApache:
# $GRID_HOME/bin/crsctl status resource myApache -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
myApache
      1        ONLINE  ONLINE       grac1                                        
# $GRID_HOME/bin/crsctl relocate  resource myApache -f
CRS-2673: Attempting to stop 'myApache' on 'grac1'
CRS-2677: Stop of 'myApache' on 'grac1' succeeded
CRS-2673: Attempting to stop 'MyTestVIP' on 'grac1'
CRS-2677: Stop of 'MyTestVIP' on 'grac1' succeeded
CRS-2672: Attempting to start 'MyTestVIP' on 'grac2'
CRS-2676: Start of 'MyTestVIP' on 'grac2' succeeded
CRS-2672: Attempting to start 'myApache' on 'grac2'
CRS-2676: Start of 'myApache' on 'grac2' succeeded
# $GRID_HOME/bin/crsctl status resource myApache -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
myApache
      1        ONLINE  ONLINE       grac2    

Reboot server grac2 to test resource failover and check resource status until available on grac1 
After reboot monitor  myApache cluster resource until it becomes ONLINE on note grac1.
# $GRID_HOME/bin/crsctl status resource myApache -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
myApache
      1        ONLINE  ONLINE       grac1  
Retest your apache application
$ /usr/local/bin/apache.scr check 
[grid@grac1 ~]$  echo $?
0
or use your browser with URL: http://192.168.1.91/icons/apache_pb.gif

Check cluster alert.log on grac1 for any cluster related operations:
[crsd(3799)]CRS-2773:Server 'grac2' has been removed from pool 'Generic'.
2013-08-07 18:02:00.124
[crsd(3799)]CRS-2773:Server 'grac2' has been removed from pool 'myApache_sp'.
2013-08-07 18:02:00.124
[crsd(3799)]CRS-2773:Server 'grac2' has been removed from pool 'ora.GRACE2'.
2013-08-07 18:03:37.653
[cssd(3170)]CRS-1601:CSSD Reconfiguration complete. Active nodes are grac1 grac2 .
2013-08-07 18:04:11.034
[crsd(3799)]CRS-2772:Server 'grac2' has been assigned to pool 'Generic'.
2013-08-07 18:04:11.034
[crsd(3799)]CRS-2772:Server 'grac2' has been assigned to pool 'ora.GRACE2'.
2013-08-07 18:04:11.034
[crsd(3799)]CRS-2772:Server 'grac2' has been assigned to pool 'myApache_sp'.
2013-08-07 18:04:11.473

References:
11gR2 VIP Does not Fail Over After Public Network Issue if Application VIP is ONLINE and Added With "appvipcfg create" (Doc ID 1457380.1)
How to create application VIP using appvipcfg for non-default network (Doc ID 1464080.1)
http://gjilevski.com/2011/11/13/build-ha-for-third-party-application-with-oracle-gi-11-2-0-3/

One thought on “Clusterware_for_3rd_party”

Leave a Reply

Your email address will not be published. Required fields are marked *