Framework for testing UCP Pool & RAC 12.1.0.2 with Wildfly 8.2

Overview

  • This Framework is easily extensible to test add. UCP features
  • Currently Implemented Features are : FCF, Standard Pool Operations, some Timeouts
  • Note: Using XA Datasoruce and UCP does not work with Wildfly !!
  • The Framework display Instance Status, Session Status, Pool Status and FCF Status
  • All top level function starting with jsf* should catch all excepetions and display these Exceptions via an Exception Window
  • getConnection() initializes the Pooled Datasource  and returns a Connection object
  • getConnection() stores the Pooled Datasource in a static variable and it is only loaded once per JVM during first usage of UCPPoolBean class
  • initPool() creates and starts the named UCP pool :   my_ucp_pool
  • Following UCP Pool Operations are supported : INIT  STOP  START  DESTROY  PURGE REFRESH
  • The JNDI functions creates its own JNDI entry named:  java:global/myUcpDs
  • The Code expects that you have created a Datasource named myRacDS
  • This Datasource is created via Annotations:  @Resource(lookup=”java:jboss/datasources/myRacDS”)  
  • Above mentioned datasource myRACDS is only used in jsfCheckDriver() function and works even without the UCP POOL
  • Take A first look on the Operation Panel – Click on the image to see details !UCP_Panel

Source Code:  getConnection() –  An important Code Detail

Central part of the Java Code is the getConnection() Function     
public class UcpPoolBean
    {
    private static oracle.ucp.jdbc.PoolDataSource pds = null;
    ..      
    public Connection getConnection() throws Exception
      {
        Connection c = null;
        if (pds == null )
          {
            setPoolStats("UCP POOL not yet initialized - creating  UCP pool in getConection()  !");
            initPool();
              // This is just an informational message that the Pooled Datasource was recreated !              
            genericExp("INFO: UCP pool created via getConnection() as Pooled DataSource == null ! ", null);
          }
        if ( pds == null )
            genericExp("FATAL: Pooled DataSource == null after init in getConnection()  ", null);       
        else
            c = pds.getConnection();
         return c;
      }

 

Framework: Generic Display Options

Instance status :  CheckInstance

11:51:32.479:    RAC DB: BANKA
Instance Name:bankA_1 - Host: hract21.example.com
Instance Name:bankA_2 - Host: hract22.example.com
Instance Name:bankA_1 - Session Count 5
Instance Name:bankA_2 - Session Count 2

UCP Pool status :  CheckUCPPool  
Press -> Check UCP Pool Status 
11:52:49.906: Intialized Pooled DS: oracle.ucp.jdbc.PoolDataSourceImpl@932eae2
11:52:49.907: Available Pools:  my_ucp_pool
11:52:49.907:   Connection Count : Total : 5 - Available: 5 - Borrowed: 0
11:52:49.907:   Pools Size       : Inital: 5 - Min      : 5 - Max     : 10
11:52:49.907:   PoolLifeCycle state:  Running
11:52:49.907:   InactiveConnectionTimeout: 30 - AbandonedConnectionTimeout: 10
     
Session Info :  CheckUCPSessions    
12:04:46.824: Available Connections in UCP POOL: 5
12:04:46.876: Instance Name:bankA_2 - Host: hract22.example.com
12:04:46.944: Instance Name:bankA_1 - Host: hract21.example.com
12:04:46.946: Instance Name:bankA_2 - Host: hract22.example.com
12:04:46.948: Instance Name:bankA_1 - Host: hract21.example.com
12:04:46.950: Instance Name:bankA_2 - Host: hract22.example.com

To display all of the above statistics press : CheckAll

Testing Abandon Connection Timeout

The abandoned connection timeout (ACT) enables borrowed connections to be
reclaimed back into the connection pool after a connection has not been used for a
specific amount of time. Abandonment is determined by monitoring calls to the
database. This timeout feature helps maximize connection reuse and conserves system
resources that are otherwise lost on maintaining borrowed connections that are no
longer in use.

Settings:
Set Connection Count         : 5   
Abandoned Connection Timeout : 10 [seconds ]
   AllocateConnections
UCP Pool Status:
12:31:41.341: Intialized Pooled DS: oracle.ucp.jdbc.PoolDataSourceImpl@932eae2
12:31:41.341: Available Pools:  my_ucp_pool
12:31:41.341:   Connection Count : Total : 5 - Available: 0 - Borrowed: 5
12:31:41.341:   Pools Size       : Inital: 5 - Min      : 5 - Max     : 10
12:31:41.341:   PoolLifeCycle state:  Running
12:31:41.341:   InactiveConnectionTimeout: 30 - AbandonedConnectionTimeout: 10
12:31:41.341: Connections in static ArrayList  jsfOpenConnections(): 5
12:31:41.341: jsfOpenConnections(): Connection - Opened:  5
-> 5 connections are now in use !

Session INFO :
12:31:41.335: jsfOpenConnections(): Connections newly Opened:  5
12:31:41.335: verifyConnections() - Connections in static ArrayList :
12:31:41.336: Instance Name:bankA_2 - Host: hract22.example.com
12:31:41.336: Instance Name:bankA_2 - Host: hract22.example.com
12:31:41.339: Instance Name:bankA_2 - Host: hract22.example.com
12:31:41.340: Instance Name:bankA_1 - Host: hract21.example.com
12:31:41.340: Instance Name:bankA_2 - Host: hract22.example.com

Now wait about 30 seconds and press : CheckUCPPool

12:35:27.493: Intialized Pooled DS: oracle.ucp.jdbc.PoolDataSourceImpl@932eae2
12:35:27.494: Available Pools:  my_ucp_pool
12:35:27.494:   Connection Count : Total : 5 - Available: 5 - Borrowed: 0
12:35:27.494:   Pools Size       : Inital: 5 - Min      : 5 - Max     : 10
12:35:27.494:   PoolLifeCycle state:  Running
12:35:27.494:   InactiveConnectionTimeout: 30 - AbandonedConnectionTimeout: 10
-> The connections are reclaimed by the UCP pool after reaching the timeout

Verify the connection status by pressing : VerifyConnections

Exceptions :
12:37:12.567: Error in retSessionInfo() - Failed SQL:
12:37:12.567: The connection is closed: The connection is closed
12:37:12.568: java.sql.SQLException: The connection is closed: The connection is closed
at oracle.ucp.util.UCPErrorHandler.newSQLException(UCPErrorHandler.java:464)
at oracle.ucp.util.UCPErrorHandler.newSQLException(UCPErrorHandler.java:448)
at oracle.ucp.jdbc.proxy.JDBCConnectionProxyFactory.invoke(JDBCConnectionProxyFactory.java:307)
at oracle.ucp.jdbc.proxy.ConnectionProxyFactory.invoke(ConnectionProxyFactory.java:50)
at com.sun.proxy.$Proxy17.createStatement(Unknown Source)
at UcpPool.UcpPoolBean.retSessionInfo(UcpPoolBean.java:1201)
at UcpPool.UcpPoolBean.verifyConnections(UcpPoolBean.java:1113)
-> The connections are reclaimed by the pool - Note : the above Exception is expected !

Release the stale Connections 
Finally Press ReleaseConnections  to cleanup the static Connection Pool Array

The Release Operation should report:
12:43:33.146: Connections in static ArrayList  jsfCloseConnections(): 0
12:43:33.146: jsfCloseConnections(): Connections Closed:  1

 

Testing Inactive Connection Timeout

The inactive connection timeout specifies how long an available connection can remain
idle before it is closed and removed from the pool. This timeout property is only
applicable to available connections and does not affect borrowed connections. This
property helps conserve resources that are otherwise lost on maintaining connections
that are no longer being used. The inactive connection timeout (together with the
Note: UCP for JDBC either cancels or rolls back connections that
have local transactions pending before reclaiming connections for
reuse.

Settings: 
Set Connection Count       :  4  
Inactive Connection Timeout:     30
Press 2x:  AllocateConnections
Press 2x:  ReleaseConnections

UCP Pool Status :     
12:54:32.053: Intialized Pooled DS: oracle.ucp.jdbc.PoolDataSourceImpl@359db23
12:54:32.054: Available Pools:  my_ucp_pool
12:54:32.054:   Connection Count : Total : 8 - Available: 8 - Borrowed: 0
12:54:32.054:   Pools Size       : Inital: 5 - Min      : 5 - Max     : 10
12:54:32.054:   PoolLifeCycle state:  Running
12:54:32.054:   InactiveConnectionTimeout: 30 - AbandonedConnectionTimeout: 10
12:54:32.054: Connections in static ArrayList  jsfCloseConnections(): 0
12:54:32.054: jsfCloseConnections(): Connections Closed:  4

Session Info :     
12:57:24.381: Available Connections in UCP POOL: 8
12:57:24.383: Instance Name:bankA_1 - Host: hract21.example.com
12:57:24.384: Instance Name:bankA_2 - Host: hract22.example.com
12:57:24.385: Instance Name:bankA_2 - Host: hract22.example.com
12:57:24.386: Instance Name:bankA_1 - Host: hract21.example.com
12:57:24.387: Instance Name:bankA_2 - Host: hract22.example.com
12:57:24.388: Instance Name:bankA_2 - Host: hract22.example.com
12:57:24.389: Instance Name:bankA_2 - Host: hract22.example.com
12:57:24.390: Instance Name:bankA_1 - Host: hract21.example.com
-> The UCP Connection Count is increased to 8 !

Now wait about 60 seconds:
UCP Pool status :     
12:58:19.972: Intialized Pooled DS: oracle.ucp.jdbc.PoolDataSourceImpl@359db23
12:58:19.972: Available Pools:  my_ucp_pool
12:58:19.972:   Connection Count : Total : 5 - Available: 5 - Borrowed: 0
12:58:19.972:   Pools Size       : Inital: 5 - Min      : 5 - Max     : 10
12:58:19.973:   PoolLifeCycle state:  Running
12:58:19.973:   InactiveConnectionTimeout: 30 - AbandonedConnectionTimeout: 10
-> UCP Pool has reduced the Connection Count to 5 from 8

 

Testing Pool Limits

Settings:
Set Connection Count       :  4  
Press  3x                  : AllocateConnections

UCP Pool status :     
13:00:08.544: Intialized Pooled DS: oracle.ucp.jdbc.PoolDataSourceImpl@359db23
13:00:08.544: Available Pools:  my_ucp_pool
13:00:08.544:   Connection Count : Total : 10 - Available: 0 - Borrowed: 10
13:00:08.544:   Pools Size       : Inital: 5 - Min      : 5 - Max     : 10
13:00:08.544:   PoolLifeCycle state:  Running
13:00:08.544:   InactiveConnectionTimeout: 30 - AbandonedConnectionTimeout: 10
13:00:08.544: Connections in static ArrayList  jsfOpenConnections(): 10
13:00:08.544: jsfOpenConnections(): Connection - Opened:  2   

Exceptions :
13:00:08.531: Error in openConnections() + Already opened Connections: 2
13:00:08.531: Exception occurred while getting connection: oracle.ucp.UniversalConnectionPoolException: 
              All connections in the Universal Connection Pool are in use
13:00:08.531: java.sql.SQLException: Exception occurred while getting connection: oracle.ucp.UniversalConnectionPoolException:
     All connections in the Universal Connection Pool are in use
    at oracle.ucp.util.UCPErrorHandler.newSQLException(UCPErrorHandler.java:479)
    at oracle.ucp.util.UCPErrorHandler.throwSQLException(UCPErrorHandler.java:154)
    at oracle.ucp.jdbc.PoolDataSourceImpl.getConnection(PoolDataSourceImpl.java:1127)
    at oracle.ucp.jdbc.PoolDataSourceImpl.getConnection(PoolDataSourceImpl.java:1031)
-> The lastest Allocate Connection request can only allocate 2 connections. 
   After that the get getConnection() fails with above Exception.

 

Testing UCP Purge Operation

UCP Pool status after Purge  :     PurgeUCP

13:04:23.530: Intialized Pooled DS: oracle.ucp.jdbc.PoolDataSourceImpl@57e31b7
13:04:23.530: Available Pools:  my_ucp_pool
13:04:23.531:   Connection Count : Total : 0 - Available: 0 - Borrowed: 0
13:04:23.531:   Pools Size       : Inital: 5 - Min      : 5 - Max     : 10
13:04:23.531:   PoolLifeCycle state:  Running
13:04:23.531:   InactiveConnectionTimeout: 30 - AbandonedConnectionTimeout: 10
13:04:23.531: Connections in static ArrayList after openConnections(): 0
13:04:23.531: Connections remaining in static ArrayList after closeConnections() : 0
13:04:23.531: checkSessionInPool(): Connection - Opened:  0 - Closed: 10       
 -> After a purge pool operation all connections get closed and the pool  
    Total Connection Count is : 0 even MinPoolSize is set to 5 !

Reallocate the connections :
Set Connection Count       :  5  
Press  1x                  : AllocateConnections
Press  1x                  : ReleaseConnections

UCP Pool Status :
13:07:02.775: Intialized Pooled DS: oracle.ucp.jdbc.PoolDataSourceImpl@57e31b7
13:07:02.775: Available Pools:  my_ucp_pool
13:07:02.775:   Connection Count : Total : 5 - Available: 5 - Borrowed: 0
13:07:02.775:   Pools Size       : Inital: 5 - Min      : 5 - Max     : 10
13:07:02.775:   PoolLifeCycle state:  Running
13:07:02.775:   InactiveConnectionTimeout: 30 - AbandonedConnectionTimeout: 10
13:07:02.775: Connections in static ArrayList  jsfCloseConnections(): 0
13:07:02.775: jsfCloseConnections(): Connections Closed:  5    

Testing UCP Refresh Operation

Current Pool Status
13:08:46.268: Intialized Pooled DS: oracle.ucp.jdbc.PoolDataSourceImpl@57e31b7
13:08:46.268: Available Pools:  my_ucp_pool
13:08:46.269:   Connection Count : Total : 5 - Available: 5 - Borrowed: 0
13:08:46.269:   Pools Size       : Inital: 5 - Min      : 5 - Max     : 10
13:08:46.269:   PoolLifeCycle state:  Running
13:08:46.269:   InactiveConnectionTimeout: 30 - AbandonedConnectionTimeout: 10
13:08:46.269: Connections in static ArrayList after openConnections(): 0
13:08:46.269: Connections remaining in static ArrayList after closeConnections() : 0
13:08:46.269: checkSessionInPool(): Connection - Opened:  5 - Closed: 5

Session Info :     
13:08:46.258: Available Connections in UCP POOL: 5
13:08:46.262: Instance Name:bankA_1 - Host: hract21.example.com
13:08:46.264: Instance Name:bankA_2 - Host: hract22.example.com
13:08:46.266: Instance Name:bankA_1 - Host: hract21.example.com
13:08:46.267: Instance Name:bankA_2 - Host: hract22.example.com
13:08:46.268: Instance Name:bankA_1 - Host: hract21.example.com

Press : RefreshUCP
13:10:03.940: Available Connections in UCP POOL: 5
13:10:03.940: Instance Name:bankA_2 - Host: hract22.example.com
13:10:03.942: Instance Name:bankA_2 - Host: hract22.example.com
13:10:03.943: Instance Name:bankA_2 - Host: hract22.example.com
13:10:03.945: Instance Name:bankA_1 - Host: hract21.example.com
13:10:03.945: Instance Name:bankA_2 - Host: hract22.example.com
-> Refresh UCP Pool has closed and recreated the physical Connections 
   The connections point now to different RAC instances.

 

Testing FCF functionality


Current UCP Pool status :     
11:42:07.415: Intialized Pooled DS: oracle.ucp.jdbc.PoolDataSourceImpl@44d2df43
11:42:07.415: Available Pools:  my_ucp_pool
11:42:07.416:   Connection Count : Total : 5 - Available: 5 - Borrowed: 0
11:42:07.416:   Pools Size       : Inital: 5 - Min      : 5 - Max     : 10
11:42:07.416:   PoolLifeCycle state:  Running
11:42:07.416:   InactiveConnectionTimeout: 30 - AbandonedConnectionTimeout: 10
11:42:07.416: Connections in static ArrayList after openConnections(): 0
11:42:07.416: Connections remaining in static ArrayList after closeConnections() : 0
11:42:07.416: checkSessionInPool(): Connection - Opened:  5 - Closed: 5

Session Info :     
11:42:07.313: Available Connections in UCP POOL: 5
11:42:07.317: Instance Name:bankA_2 - Host: hract22.example.com
11:42:07.319: Instance Name:bankA_2 - Host: hract22.example.com
11:42:07.321: Instance Name:bankA_2 - Host: hract22.example.com
11:42:07.413: Instance Name:bankA_3 - Host: hract21.example.com
11:42:07.415: Instance Name:bankA_3 - Host: hract21.example.com
-> All UCP session are  equally distributed on both available RAC instances 

Now stop one of the instances 
[oracle@hract21 ~]$ srvctl stop instance -db banka -n hract22 -o abort

Verify Session Info again  :     
11:44:39.005: Available Connections in UCP POOL: 5
11:44:39.124: Instance Name:bankA_3 - Host: hract21.example.com
11:44:39.127: Instance Name:bankA_3 - Host: hract21.example.com
11:44:39.129: Instance Name:bankA_3 - Host: hract21.example.com
11:44:39.130: Instance Name:bankA_3 - Host: hract21.example.com
11:44:39.132: Instance Name:bankA_3 - Host: hract21.example.com
-> As expected - all UCP sessions are now pointing to the surviving instance.
   Note: We don't need a server roundtrip for this operation - ONS does a great job here ! 
    
FCF Info:
11:44:39.133:  FCF status:  {Apr 11, 2015 11:44 AM SUCCESS      
    Connections:(Available=5 Opened=2 FailedToProcess=0 MarkedDown=0 Closed=2)(Borrowed=0 FailedToProcess=0 MarkedDown=0 Closed=0)}


Restart Instance  and Refresh the pool
[oracle@hract21 ~]$ srvctl start instance -db banka -n hract22

Press : RefreshUCP
11:57:14.914: Available Connections in UCP POOL: 5
11:57:14.915: Instance Name:bankA_2 - Host: hract22.example.com
11:57:14.916: Instance Name:bankA_3 - Host: hract21.example.com
11:57:14.917: Instance Name:bankA_2 - Host: hract22.example.com
11:57:14.918: Instance Name:bankA_3 - Host: hract21.example.com
11:57:14.919: Instance Name:bankA_2 - Host: hract22.example.com
-> All UCP session are now again equally redistributed on both RAC instances

 

Testing  JNDI

  • The Framework uses ctx.bind() to temporary create a JNDI entry on the fly
  • After ctx.bind() the Framework use ctx.lookup() to create the pooled Datasource
  • Run ctx.lookup() only once and store the created Pooled Datasource in a static variable for re-using.
  • Note: After a JVM reboot the JNDI resource/UCP Pool must be recreated running rebindPool() and lookupPool()
  • The JNDI functions create its own JNDI entry named: java:global/myUcpDs

Complete JAVA Code for JNDI lookup

private static oracle.ucp.jdbc.PoolDataSource pds = null;
   public void lookupPool() throws Exception
      {
        if ( pds != null)
          {
            setPoolStats(Tools.add_hmtl_pre_tag("Pooled Datasource already exist - don't run ctx.lookup() again  "));
            return;
          }    
        cleanAll();
        setPoolStats(Tools.add_hmtl_pre_tag("Testing UPC Pool via JNDI in progress JNDI name: " +  jndiName));
        ctx = new InitialContext();  
        pds = (PoolDataSource)ctx.lookup("java:global/myUcpDs");
        checkAll();
        setPoolStats(getPoolStats() + Tools.add_hmtl_pre_tag("lookupPool(): Pooled Datasource initialized: " + jndiName ));
      }
       
    public void rebindPool() throws Exception
      {
        pds =null;
        cleanAll();                                 
        UniversalConnectionPoolManager ucpMgr = createPoolManager();
        if ( ucpMgr == null)
          {
            setPoolStats(Tools.add_hmtl_pre_tag("Could not create UCP Manager "));
            return;
          }            
        setPoolStats(Tools.add_hmtl_pre_tag("Destroying UCP Pool in progres ..."));
                // We need to destroy the pool first if the pool exist
                // If we recreate the pool without destroying we will get a session leak at the RAC layer
        try
          {    
            ucpMgr.destroyConnectionPool(poolName);
            setPoolStats(getPoolStats() + Tools.add_hmtl_pre_tag("UCP Pool destroyed !"));
          } catch ( Exception e2)         
          {
            setPoolStats(getPoolStats () +Tools.add_hmtl_pre_tag("Exception during delete Pool - [ ignore the error]  "  ));  
                  // genericExp("Execption duringe deleteing pool - If pool does not exists ignore this exception ",  e2);
          }  
        ctx = new InitialContext();
        PoolDataSource lpds = setupPooledDS();
            // ucpMgr.createConnectionPool((UniversalConnectionPoolAdapter)pds);
        ctx.rebind(jndiName, lpds);
        setPoolStats(getPoolStats () +Tools.add_hmtl_pre_tag("JNDI context rebind ok for :  " + jndiName ));  
            // Disable JMX - if needed
             // ucpManager.setJmxEnabled(false);
            // ucpManager.createConnectionPool((UniversalConnectionPoolAdapter)pds);                                                                              
        return;
      }
        
   public  PoolDataSource setupPooledDS() throws Exception
      {
        
        PoolDataSource lpds = oracle.ucp.jdbc.PoolDataSourceFactory.getPoolDataSource();        
         //set the connection properties on the data source and pool properties
        String ONS_CONFIG = "nodes=hract21:6200,hract22:6200,hract23:6200";
        lpds.setONSConfiguration ( ONS_CONFIG);
        lpds.setFastConnectionFailoverEnabled( true);
        lpds.setUser("scott");
        lpds.setPassword("tiger");
        lpds.setURL("jdbc:oracle:thin:@ract2-scan.grid12c.example.com:1521/banka");
        lpds.setConnectionFactoryClassName("oracle.jdbc.pool.OracleDataSource"); 
        lpds.setInitialPoolSize(minPoolSize);
        lpds.setMinPoolSize(minPoolSize);
        lpds.setMaxPoolSize(maxPoolSize);
        lpds.setAbandonedConnectionTimeout(abandonedConnectionTimeout);
        lpds.setInactiveConnectionTimeout(inactiveConnectionTimeout);
        lpds.setTimeoutCheckInterval(10);
        lpds.setConnectionPoolName(poolName);
        return lpds;
      } 

JNDI: Use Case I – Destroy Pooled DS and recreate UCP Pool

Destroy UCP Pool :  my_ucp_pool   DestroyUCP         [  but don't run Unbind UCP Pool via JNDI ]    
Lookup UCP Pool via JNDI  :       Lookup
16:49:48.930: Intialized Pooled DS: oracle.ucp.jdbc.PoolDataSourceImpl@7ffa742a
16:49:48.930: Available Pools:  my_ucp_pool
16:49:48.930:   Connection Count : Total : 5 - Available: 5 - Borrowed: 0
16:49:48.930:   Pools Size       : Inital: 5 - Min      : 5 - Max     : 10
16:49:48.930:   PoolLifeCycle state:  Running
16:49:48.930:   InactiveConnectionTimeout: 30 - AbandonedConnectionTimeout: 10
16:49:48.930: Connections in static ArrayList after openConnections(): 0
16:49:48.930: Connections remaining in static ArrayList after closeConnections() : 0
16:49:48.930: checkSessionInPool(): Connection - Opened:  5 - Closed: 5
16:49:48.930: lookupPool(): Pooled Datasource initialized: java:global/myUcpDs

-> After destruction of UCP pool the pool can be re-created by runing lookupPool()
   Note: A destruction of the UCP pool does not delete the JNDI entry.

JNDI: Use Case II – Destropy Pooled DS, Unbind JNDI and recreate JNDI and UCP Pool

Destroy UCP Pool :  my_ucp_pool          DestroyUCP
Unbind JNDI      :  java:global/myUcpDs  JndiUnbindUCP
Lookup UCP Pool via JNDI  fails:         Lookup

16:28:12.673: Exception inside lookupPool() :  JNDI Lookup 
16:28:12.673: myUcpDs -- service jboss.naming.context.java.global.myUcpDs
16:28:12.673: javax.naming.NameNotFoundException: myUcpDs -- service jboss.naming.context.java.global.myUcpDs
    at org.jboss.as.naming.ServiceBasedNamingStore.lookup(ServiceBasedNamingStore.java:104)
    at org.jboss.as.naming.NamingContext.lookup(NamingContext.java:202)
    at org.jboss.as.naming.NamingContext.lookup(NamingContext.java:179)  
    
Solution: 
Rebind the Pooled Datasource :  JndiRebindUCP
16:29:22.214: Destroying UCP Pool in progres ...
16:29:22.214: Exception during delete Pool - [ ignore the error]  
16:29:22.215: JNDI context rebind ok for :  java:global/myUcpDs

Run JNDI Lookup again :                Lookup 
16:29:48.930: Intialized Pooled DS: oracle.ucp.jdbc.PoolDataSourceImpl@7ffa742a
16:29:48.930: Available Pools:  my_ucp_pool
16:29:48.930:   Connection Count : Total : 5 - Available: 5 - Borrowed: 0
16:29:48.930:   Pools Size       : Inital: 5 - Min      : 5 - Max     : 10
16:29:48.930:   PoolLifeCycle state:  Running
16:29:48.930:   InactiveConnectionTimeout: 30 - AbandonedConnectionTimeout: 10
16:29:48.930: Connections in static ArrayList after openConnections(): 0
16:29:48.930: Connections remaining in static ArrayList after closeConnections() : 0
16:29:48.930: checkSessionInPool(): Connection - Opened:  5 - Closed: 5
16:29:48.930: lookupPool(): Pooled Datasource initialized: java:global/myUcpDs
-> After a JVM reboot or after deletion of UPC Pool and JNDI entry the UCP pool must be 
   recreated by calling rebindPool() and lookupPool().

Problem : JNDI lookup fails when calling lookup a 2.nd time

If you re-running the following JNDI code without destryoing the UCP pool first you will get an Exception
    public String lookupPool()
      {  
        cleanAll();
        setPoolStats(Tools.add_hmtl_pre_tag("Testing UPC Pool via JNDI in progress  "));
        genericExp("INFO:  JNDI Lookup doesn't work yet", null);
        try
          {
          ctx = new InitialContext();  
          pds = (PoolDataSource)ctx.lookup("java:global/myUcpDs");
          setPoolStats(getPoolStats() + Tools.add_hmtl_pre_tag("Testing UPC Pool via JNDI - Pooled Datasource initialized "));
          checkAll();
          } catch ( Exception e1)
              {
              genericExp("Exception inside lookupPool() :  JNDI Lookup ", e1);
            
              return "index";
              }
       
        setPoolStats(getPoolStats() + "- JNDI lookup ok  ! ");
        return "index";
      } 


1.st Run ( Note UCP pool was already destroyed )

Instance status :     RAC DB: BANKA
Instance Name:bankA_3 - Host: hract21.example.com
Instance Name:bankA_2 - Host: hract22.example.com
Instance Name:bankA_3 - Session Count 35
Instance Name:bankA_2 - Session Count 35

UCP Pool status :     
15:42:36.773: Initialized Pooled DS: oracle.ucp.jdbc.PoolDataSourceImpl@2daf23a3
15:42:36.773: Available Pools:  my_ucp_pool
15:42:36.773:   Connection Count : Total : 5 - Available: 5 - Borrowed: 0
15:42:36.773:   Pools Size       : Inital: 5 - Min      : 5 - Max     : 10
15:42:36.773:   PoolLifeCycle state:  Running
-> Status: UCP pool was created and started by  JNDI lookup
    
2.nd Run ( Note: UCP pool is already active ! )    
Exceptions :
15:48:09.493: Exception inside lookupPool() :  JNDI Lookup 
15:48:09.493: Unable to start the Universal Connection Pool: oracle.ucp.UniversalConnectionPoolException: 
   Error during pool creation in Universal Connection Pool Manager MBean: oracle.ucp.UniversalConnectionPoolException: 
   Error during pool creation in Universal Connection Pool Manager: oracle.ucp.UniversalConnectionPoolException: 
   Universal Connection Pool already exists in the Universal Connection Pool Manager. 
   Universal Connection Pool cannot be added to the Universal Connection Pool Manager
    at oracle.ucp.util.UCPErrorHandler.newSQLException(UCPErrorHandler.java:479)
    at oracle.ucp.util.UCPErrorHandler.throwSQLException(UCPErrorHandler.java:154)
    at oracle.ucp.jdbc.PoolDataSourceImpl.startPool(PoolDataSourceImpl.java:730)
    at oracle.ucp.jdbc.PoolDataSourceImpl.getConnection(PoolDataSourceImpl.java:1093)
    at oracle.ucp.jdbc.PoolDataSourceImpl.getConnection(PoolDataSourceImpl.java:1031)

Instance Status     
15:49:47.032:    RAC DB: BANKA
Instance Name:bankA_2 - Host: hract22.example.com
Instance Name:bankA_3 - Host: hract21.example.com
Instance Name:bankA_2 - Session Count 38
Instance Name:bankA_3 - Session Count 37    
    
-> The 2.nd run leads to a databases session leak
    
WA : Store the Pooled DataSource in a static variable and run lookup only a single time!

Using XA Datasource with UCP and Wildfly fails with  loader constraint violation

Init XA Datasource:  InitUCPXA
12:20:50.557: Initializing XA UCP Pool in progress ...
12:20:50.561: INFO : Jar File     for XAResource.class:jar:file:/usr/local/wildfly-8.2.0.Final/modules/system/layers/base/javax/transaction/api/main/jboss-transaction-api_1.2_spec-1.0.0.Final.jar!/javax/transaction/xa/XAResource.class
12:20:50.561: INFO : Class Loader for XAResource.class :ModuleClassLoader for Module "javax.transaction.api:main" from local module loader @7fe9a140 (finder: local module finder @4359b25b (roots: /usr/local/wildfly-8.2.0.Final/modules,/usr/local/wildfly-8.2.0.Final/modules/system/layers/base))
12:20:50.784: XA Initializing UCP Pool failed !

Exceptions :
12:20:50.784: Runtime exception in jsfInitPoolXA() : 
12:20:50.785: loader constraint violation: loader (instance of ) previously initiated loading for a different type with name "javax/transaction/xa/XAResource"
12:20:50.787: java.lang.LinkageError: loader constraint violation: loader (instance of ) previously initiated loading for a different type with name "javax/transaction/xa/XAResource"
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2615)
    at java.lang.Class.privateGetPublicMethods(Class.java:2733)
    at java.lang.Class.getMethods(Class.java:1472)
    at sun.misc.ProxyGenerator.generateClassFile(ProxyGenerator.java:426)
    at sun.misc.ProxyGenerator.generateProxyClass(ProxyGenerator.java:323

For Details please check ... 
 Debugging UCP Connection Pool using XA Datasource pointing to a RAC database - http://www.hhutzler.de/blog/debugging-ucp-connection-pool-using-xa-datasource-pointing-to-a-rac-database/

 

Source Code

 JSF code:   index.xhtml      
 Java Code:  UcpPoolBean.java 

JAVADOC

Reference

How to integrate the Oracle JDBC driver as a Wildfly module ?

Purpose of this tutorial

  • Create and test a  RAC datasoure with WildFly jboss-cli.sh command utility
  • Deploy and integrate the Oracle JDBC driver as a Wildfly Module
  • Run a short Web application to display JDBC driver and Oracle Database version  

Deploying the Oracle JDBC Driver as a Wildfly module

Prepare module.xml and copy ojdbc7.jar to its destination
[oracle@wls1 ~]$ ls /usr/local/wildfly-8.2.0.Final/modules/com/oracle/ojdbc/main/
module.xml  ojdbc7.jar

[oracle@wls1 ~]$ cat /usr/local/wildfly-8.2.0.Final/modules/com/oracle/ojdbc/main/module.xml
?xml version="1.0" ?>
<module xmlns="urn:jboss:module:1.1" name="com.oracle.ojdbc">
    <resources>
        <resource-root path="ojdbc7.jar"/>
    </resources>
    <dependencies>
        <module name="javax.api"/>
        <module name="javax.transaction.api"/>
    </dependencies>
</module>

--> Restart Wildfly and your Datasource and verify the Oracle Driver :

[oracle@wls1 WILDFLY]$ $WILDFLY_HOME/bin/jboss-cli.sh --connect  /subsystem=datasources:installed-drivers-list
{
    "outcome" => "success",
    "result" => [
        {
            "driver-name" => "oracle",
            "deployment-name" => undefined,
            "driver-module-name" => "com.oracle.ojdbc",
            "module-slot" => "main",
            "driver-datasource-class-name" => "",
            "driver-xa-datasource-class-name" => "oracle.jdbc.xa.client.OracleXADataSource",
            "driver-class-name" => "oracle.jdbc.OracleDriver",
            "driver-major-version" => 12,
            "driver-minor-version" => 1,
            "jdbc-compliant" => true
        }

Setup RAC datasource by using the RAC SCAN address
[oracle@wls1 WILDFLY]$ cat add_ds.cli
if (outcome != success) of /subsystem=datasources/data-source=myRacDS:read-resource
    data-source add \
        --name=myRacDS \
        --driver-name=oracle \
        --jndi-name=java:jboss/datasources/myRacDS \
        --user-name="scott"\
        --password="tiger"\
        --use-java-context=true \
        --use-ccm=true \
        --min-pool-size=5 \
        --max-pool-size=10 \
        --pool-prefill=true \
        --allocation-retry=1 \
        --prepared-statements-cache-size=32 \
        --share-prepared-statements=true \
        --connection-url="jdbc:oracle:thin:@ract2-scan.grid12c.example.com:1521/banka"
end-if

[oracle@wls1 WILDFLY]$ $WILDFLY_HOME/bin/jboss-cli.sh --connect --file=add_ds.cli

Test the connection
[oracle@wls1 WILDFLY]$   $WILDFLY_HOME/bin/jboss-cli.sh --connect   /subsystem=datasources/data-source=myRacDS:test-connection-in-pool
{
    "outcome" => "success",
    "result" => [true]
}

Check DS properties

[oracle@wls1 WILDFLY]$ $WILDFLY_HOME/bin/jboss-cli.sh --connect   /subsystem=datasources/data-source=myRacDS:read-resource
{
    "outcome" => "success",
    "result" => {
        "allocation-retry" => 1,
        "allocation-retry-wait-millis" => undefined,
        "allow-multiple-users" => false,
        "background-validation" => undefined,
        "background-validation-millis" => undefined,
        "blocking-timeout-wait-millis" => undefined,
        "capacity-decrementer-class" => undefined,
        "capacity-decrementer-properties" => undefined,
        "capacity-incrementer-class" => undefined,
        "capacity-incrementer-properties" => undefined,
        "check-valid-connection-sql" => undefined,
        "connection-listener-class" => undefined,
        "connection-listener-property" => undefined,
        "connection-properties" => undefined,
        "connection-url" => "jdbc:oracle:thin:@ract2-scan.grid12c.example.com:1521/banka",
        "datasource-class" => undefined,
        "driver-class" => undefined,
        "driver-name" => "oracle",
        "enabled" => true,
        "exception-sorter-class-name" => undefined,
        "exception-sorter-properties" => undefined,
        "flush-strategy" => undefined,
        "idle-timeout-minutes" => undefined,
        "initial-pool-size" => undefined,
        "jndi-name" => "java:jboss/datasources/myRacDS",
        "jta" => true,
        "max-pool-size" => 10,
        "min-pool-size" => 5,
        "new-connection-sql" => undefined,
        "password" => "tiger",
        "pool-prefill" => true,
        "pool-use-strict-min" => undefined,
        "prepared-statements-cache-size" => 32,
        "query-timeout" => undefined,
        "reauth-plugin-class-name" => undefined,
        "reauth-plugin-properties" => undefined,
        "security-domain" => undefined,
        "set-tx-query-timeout" => false,
        "share-prepared-statements" => true,
        "spy" => false,
        "stale-connection-checker-class-name" => undefined,
        "stale-connection-checker-properties" => undefined,
        "track-statements" => "NOWARN",
        "transaction-isolation" => undefined,
        "url-delimiter" => undefined,
        "url-selector-strategy-class-name" => undefined,
        "use-ccm" => true,
        "use-fast-fail" => false,
        "use-java-context" => true,
        "use-try-lock" => undefined,
        "user-name" => "scott",
        "valid-connection-checker-class-name" => undefined,
        "valid-connection-checker-properties" => undefined,
        "validate-on-match" => false,
        "statistics" => {
            "jdbc" => undefined,
            "pool" => undefined
        }
    }
}

If you need to remove the DS run: 
 $WILDFLY_HOME/bin/jboss-cli.sh --connect   /subsystem=datasources/data-source=myRacDS:remove

 

Test  Oracle JDBC driver from a JBOSS WEB application

Run the following JAVA code: 
 try
          {
            String clname= "oracle.jdbc.OracleDriver";
            Class.forName (clname);  
            setJdbcInfo( StackTraceUtil.add_hmtl_pre_tag( "JDBC Driver Check - Loading Driver class ok : " + clname));
            java.util.Properties props = System.getProperties();
            java.util.Enumeration propNames = props.propertyNames();
            setJdbcInfo(getJdbcInfo()  + StackTraceUtil.add_hmtl_pre_tag("JDK Version: " + props.getProperty("java.version")));
            setJdbcInfo(getJdbcInfo() + StackTraceUtil.add_hmtl_pre_tag( "ClassPath  : "  + props.getProperty("java.class.path")));
            Connection c = ds1.getConnection();
            java.sql.DatabaseMetaData md = c.getMetaData();
            setJdbcInfo(getJdbcInfo()  + StackTraceUtil.add_hmtl_pre_tag( "Driver Name             : "+md.getDriverName()));
            setJdbcInfo(getJdbcInfo()  + StackTraceUtil.add_hmtl_pre_tag( "Driver Version          : "+md.getDriverVersion()));
            setJdbcInfo(getJdbcInfo()  + StackTraceUtil.add_hmtl_pre_tag( "Database Product Version: " +md.getDatabaseProductVersion()));
            c.close();
          
          } catch ( Exception e1)
..
Very likely
Class.forName (clname);   throws the following Exceptions  in checkDriver()

oracle.jdbc.OracleDriver from [Module "deployment.WF_JdbcDriverTest.war:main" from Service Module Loader]
java.lang.ClassNotFoundException: oracle.jdbc.OracleDriver from [Module "deployment.WF_JdbcDriverTest.war:main" from Service Module Loader]
    at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:213)
    at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:459)
    at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:408)
    at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:389)
    at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:134)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:191)
    at DriverTest.DriverTestBean.checkDriver(DriverTestBean.java:54)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.sun.el.parser.AstValue.invoke(AstValue.java:292)
    at com.sun.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:304)
    at com.sun.faces.facelets.el.TagMethodExpression.invoke(TagMethodExpression.java:105

Note if working with Datasources you may see following stack:
Caused by: oracle.ucp.UniversalConnectionPoolException: Error during pool creation in Universal Connection Pool Manager: java.sql.SQLException: Invalid Universal Connection Pool configuration: java.sql.SQLException: Unable to create factory class instance with provided factory class name: java.lang.ClassNotFoundException: oracle.jdbc.pool.OracleDataSource from [Module "deployment.WF_UCPPool.war:main" from Service Module Loader]
    at oracle.ucp.util.UCPErrorHandler.newUniversalConnectionPoolException(UCPErrorHandler.java:359)
    at oracle.ucp.util.UCPErrorHandler.throwUniversalConnectionPoolException(UCPErrorHandler.java:58)
    at oracle.ucp.admin.UniversalConnectionPoolManagerBase.createConnectionPool(UniversalConnectionPoolManagerBase.java:528)
    at UcpPool.UcpPoolBean.rebindPool(UcpPoolBean.java:289)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

Detailed Problem description:

  • Why is Module Loader looking for the JDBC driver ?
  • We expect that Wildfly picks up the Oracle JDBC Driver as we have installed the JDBC Driver as a Module

  Note if have configured the  Oracle JDBC driver as a library in your WAR file the above code works fine .

 

Solution to integrate the Oracle JDBC Driver with Wildfly

Add a dependency to the Oralce JDBC driver module by adding  jboss-deployment-structure.xml 
under your WEB_INF folder:   
<?xml version="1.0" encoding="UTF-8"?>
<!--
To change this license header, choose License Headers in Project Properties.
To change this template file, choose Tools | Templates
and open the template in the editor.
-->

<jboss-deployment-structure>
  <deployment>
    <dependencies>
       <module name="com.oracle.ojdbc" />
    </dependencies>
   </deployment>
</jboss-deployment-structure>

Rerun the test program - ths JSF output should display the following : 

JDBC Driver Check - Loading Driver class ok : oracle.jdbc.OracleDriver
JDK Version: 1.7.0_71
ClassPath  : /usr/local/wildfly-8.2.0.Final/jboss-modules.jar
Driver Name             : Oracle JDBC driver
Driver Version          : 12.1.0.2.0
Database Product Version: Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options

JSF code : index.xhtml

<?xml version='1.0' encoding='UTF-8' ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml"
xmlns:h="http://xmlns.jcp.org/jsf/html">
<h:head>
<title>JDBC Driver Test</title>
</h:head>
<h:body>
<h:form>
<h:commandButton id="cBnt1" value="JDBC Driver Check"  action="#{driverTestBean.checkDriver}"></h:commandButton>
<hr color="blue"  size="3" width="90%" align="center"/>
<h:panelGrid columns="6" border="3">
<h:outputText style="color: red"  value="JDBC Driver Status :" />
<h:outputText  id="optxt1"  style="color: blue"  value="#{driverTestBean.jdbcInfo}" escape="false" />
</h:panelGrid>
<hr color="blue"  size="3" width="90%" align="center"/>
<h:outputText style="color: red"  value="Exceptions : "  escape="false"  />
<br/>
<h:outputText  id="optxt2"  style="color: red"  value="#{driverTestBean.exceptInfo}" escape="false" />
</h:form>

</h:body>
</html>

JAVA Code:  DriverTestBean.java

package DriverTest;

import java.io.PrintWriter;
import java.io.StringWriter;
import java.io.Writer;
import java.sql.Connection;
import javax.annotation.Resource;
import javax.faces.bean.ManagedBean;
import javax.faces.bean.SessionScoped;
import javax.sql.DataSource;

/**
 *
 * @author Helmut Hutzler 
 */
 @ManagedBean
 @SessionScoped
public class DriverTestBean
    {
    private String exceptInfo="";
    private String jdbcInfo="  -->  please Press JDBC Driver Check Button";
    @Resource(lookup="java:jboss/datasources/myRacDS")
    private DataSource ds1;
      
    public String getJdbcInfo()
      {
        return jdbcInfo;
      }

    public void setJdbcInfo(String jdbcInfo)
      {
        this.jdbcInfo = jdbcInfo;
      }
     
    public String getExceptInfo()
      {
        return exceptInfo;
      }

    public void setExceptInfo(String exceptInfo)
      {
        this.exceptInfo = exceptInfo;
      }
    public String checkDriver() throws Exception
      {
        try
          {
            String clname= "oracle.jdbc.OracleDriver";
            Class.forName (clname);  
            setJdbcInfo( StackTraceUtil.add_hmtl_pre_tag( "JDBC Driver Check - Loading Driver class ok : " + clname));
            java.util.Properties props = System.getProperties();
            java.util.Enumeration propNames = props.propertyNames();
            setJdbcInfo(getJdbcInfo()  + StackTraceUtil.add_hmtl_pre_tag("JDK Version: " + props.getProperty("java.version")));
            setJdbcInfo(getJdbcInfo() + StackTraceUtil.add_hmtl_pre_tag( "ClassPath  : "  + props.getProperty("java.class.path")));
            Connection c = ds1.getConnection();
            java.sql.DatabaseMetaData md = c.getMetaData();
            setJdbcInfo(getJdbcInfo()  + StackTraceUtil.add_hmtl_pre_tag( "Driver Name             : "+md.getDriverName()));
            setJdbcInfo(getJdbcInfo()  + StackTraceUtil.add_hmtl_pre_tag( "Driver Version          : "+md.getDriverVersion()));
            setJdbcInfo(getJdbcInfo()  + StackTraceUtil.add_hmtl_pre_tag( "Database Product Version: " +md.getDatabaseProductVersion()));
            c.close();
          
          } catch ( Exception e1)
              {
                generic_exp("Error in checkDriver()", e1);
              }
      
       return "index";
      }
    
    public void generic_exp(String ip, Exception e1)
      {
        if ( ip != null)
            setExceptInfo( StackTraceUtil.add_hmtl_pre_tag(ip));
        if ( e1 != null)
          {
            setExceptInfo(getExceptInfo() +  StackTraceUtil.add_hmtl_pre_tag(e1.getMessage()));
            setExceptInfo(getExceptInfo() +  StackTraceUtil.add_hmtl_pre_tag( StackTraceUtil.getStackTrace(e1)));
      
          } 
      }  
    
    }

 class StackTraceUtil 
  {

    public static String getStackTrace(Throwable aThrowable) 
      {
        final Writer result = new StringWriter();
        final PrintWriter printWriter = new PrintWriter(result);
        aThrowable.printStackTrace(printWriter);
        return result.toString();
     }
    
    public static String add_hmtl_pre_tag (String ip)
      {
        String op =  "<pre>"  + ip + "</pre>" ; 
        return op;        
      }      
  }

Reference

Create an object factory to return a Singleton Java Object Instance

Purpose  of this tutorial

  • Create and deploy your custom Wildfly module
  • Create an Object Factory class which instantiates a Singleton Instance of our TestBean class
  • Build a Test client [ pure java SE program ] using remote JNDI to lookup our Remote Object Factory MyObjectFactory

Requirements

  • Set classpath :  $ export CLASSPATH=/usr/local/wildfly-8.2.0.Final/bin/client/jboss-client.jar:.
  • Tested with   Wildfly 8.0.2

JAVA Code [ Server Side ]

 
MyObjectFactory.java:

package hhu.object_factory;
/**
 * @author Helmut Hutzler
 */
import java.util.Enumeration;
import java.util.Hashtable;
import javax.naming.Context;
import javax.naming.Name;
import javax.naming.NamingException;
import javax.naming.RefAddr;
import javax.naming.Reference;import javax.naming.spi.ObjectFactory;
import hhu.object_factory.TestBean;
public class MyObjectFactory implements ObjectFactory
 {
    /*
       Note we need to restart the Wilfdly server when we change the  MyObjectFactory object !
       Making constructor  MyObjectFactory() as private to prevent access to outsiders
       The instance of  TestBean()  is created at the startup of the class and remains valid
         until someone shutdown the  JVM.
       Since TestBean is a static, it gets loaded and created during loading of the MyObjectFactory class.
       Calling getObjectInstance does not instantiate the TestBean class a 2.nd time
    */
    private static volatile TestBean bean =  new TestBean("City","Sattelmannsburg - ( believe me - this is not  in India)");
    
       public MyObjectFactory()
       {
            System.out.println(" NEW: [MyCustomObjectFactory] MyCustomObjectFactory initialized.");
       }

       public Object getObjectInstance(Object obj, Name name, Context nameCtx,Hashtable environment) throws Exception
       {
            // TestBean bean = new TestBean("City","Pune (India)");
            return bean;
       }
   }

TestBean.java:
package hhu.object_factory;
/**
 *
 * @author Helmut Hutzler 
 */
public class TestBean implements java.io.Serializable
  {
      private String name;
      private String value;
      private long itime;

    public long getItime()
      {
        return itime;
      }
      
      public TestBean(String name,String value)
       {
         this.name=name;
         this.value=value;
         this.itime=System.currentTimeMillis();
         System.out.println("[TestBean] TestBean initialized - itime: " + this.itime);
       }

      public String getName()
       {
          return name;
       }
      public String getValue()
       {
          return value;
       }
      
  }

Deploy our JAR file as a Wildfly Module

Assemble jar file WF_Object_Factory.jar  with following files 

[oracle@wls1 dist]$ pwd 
/home/oracle/NetBeansProjects/GIT/WF_Object_Factory/dist
[oracle@wls1 dist]$  jar tvf WF_Object_Factory.jar
     0 Fri Mar 13 17:50:52 CET 2015 META-INF/
   177 Fri Mar 13 17:50:50 CET 2015 META-INF/MANIFEST.MF
     0 Fri Mar 13 17:50:50 CET 2015 hhu/
     0 Fri Mar 13 17:50:50 CET 2015 hhu/object_factory/
  1135 Fri Mar 13 17:50:50 CET 2015 hhu/object_factory/MyObjectFactory.class
   818 Fri Mar 13 17:50:50 CET 2015 hhu/object_factory/TestBean.class

Deploy JAR file 
[oracle@wls1 dist]$ cd /home/oracle/NetBeansProjects/GIT/WF_Object_Factory/dist
[oracle@wls1 dist]$ mkdir -p $WILDFLY_HOME/modules/object_factory/main
[oracle@wls1 dist]$ cp /home/oracle/NetBeansProjects/GIT/WF_Object_Factory/dist/WF_Object_Factory.jar  
                       $WILDFLY_HOME/modules/object_factory/main 
[oracle@wls1 dist]$ cd    $WILDFLY_HOME/modules/object_factory/main 
[oracle@wls1 dist]$ pwd 
/usr/local/wildfly-8.2.0.Final/modules/object_factory_wf/main
[oracle@wls1 main]$ ls -l
total 8
-rw-rw-r-- 1 oracle oracle  265 Mar 12 16:03 module.xml
-rw-rw-r-- 1 oracle oracle 2819 Mar 12 15:52 Object_Factory_WF.jar

Create a module.xml with following content
<?xml version="1.0" encoding="UTF-8"?>
<module xmlns="urn:jboss:module:1.4" name="object_factory">
   <resources>
       <resource-root path="WF_Object_Factory.jar"/>
   </resources>
  <dependencies>
       <module name="javax.api"/>
  </dependencies>
</module>
Verify module.xml entries
[oracle@wls1 main]$ ls ../../object_factory
main
[oracle@wls1 main]$ ls WF_Object_Factory.jar
WF_Object_Factory.jar

Module Management - Load the module a first time 

[oracle@wls1 JNDI]$   $WILDFLY_HOME/bin/jboss-cli.sh --connect --file=obj_factory_jndi.cli
obj_factory_jndi.cli : 
/subsystem=naming/binding=java\:jboss\/exported\/object_factory/:add(binding-type=object-factory,module=object_factory,class=hhu.object_factory.MyObjectFactory

Server log should report: 
17:41:13,286 INFO  [stdout] (management-handler-thread - 4) [TestBean] TestBean initialized - itime: 1426351273285
17:41:13,286 INFO  [stdout] (management-handler-thread - 4)  NEW: [MyCustomObjectFactory] MyCustomObjectFactory initialized.
2015-03-14 17:41:13,286 INFO  [stdout] (management-handler-thread - 4) [TestBean] TestBean initialized - itime: 1426351273285
2015-03-14 17:41:13,286 INFO  [stdout] (management-handler-thread - 4)  NEW: [MyCustomObjectFactory] MyCustomObjectFactory initialized.

Note if you need to change the MyObjectFactory.class or TestBean.class you need to restart your Wildfly server 

In case you need to remove the module run :
[oracle@wls1 JNDI]$ cat rm_obj_factory_jndi.cli
/subsystem=naming/binding=java\:jboss\/exported\/object_factory/:remove

[oracle@wls1 JNDI]$   $WILDFLY_HOME/bin/jboss-cli.sh --connect --file=rm_obj_factory_jndi.cli
{
    "outcome" => "success",
    "response-headers" => {
        "operation-requires-reload" => true,
        "process-state" => "reload-required"
    }
}
--> restart Wildfly

Build and run Remote JNDI test program

BindJndiDemo.java:

import java.io.*;
import java.util.Hashtable;
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;

public class BindJndiDemo
{
public final static String JNDI_FACTORY="org.jboss.naming.remote.client.InitialContextFactory";

public static void main(String[] args) throws Exception
  {
     if (args.length != 1) {
         System.out.println("Usage: java BindJndiDemo URL");
         System.out.println("Example:  java BindJndiDemo  http-remoting://127.0.0.1:8180");
         return;
     } 

     InitialContext ic = getInitialContext(args[0]);
     BindJndiDemo demo = new BindJndiDemo();
     String jnid_lookup = "java:object_factory";
     hhu.object_factory.TestBean testBean=(hhu.object_factory.TestBean)ic.lookup(jnid_lookup);
     System.out.println("\t(hhu.object_factory.TestBean)ic.lookup(jnid_lookup) testBean = "+testBean);
     System.out.println("\tname="+testBean.getName()+"\tvalue="+testBean.getValue()+" - Object Inst. Time: " + testBean.getItime());
     System.out.println("");
  }

private static InitialContext getInitialContext(String url) throws NamingException
     {
        Hashtable env = new Hashtable();
        env.put(Context.INITIAL_CONTEXT_FACTORY, JNDI_FACTORY);
        env.put(Context.PROVIDER_URL, url);

        env.put(Context.SECURITY_PRINCIPAL, "oracle");
        env.put(Context.SECURITY_CREDENTIALS, "helmut11");
        InitialContext ic=new InitialContext(env);
        System.out.println("\t Got InitialContext ic: "+ic);
         return ic;
      }
}

Test Client: 
[oracle@wls1 JNDI]$  java BindJndiDemo  http-remoting://127.0.0.1:8180
Mar 14, 2015 5:47:17 PM org.xnio.Xnio <clinit>
INFO: XNIO version 3.3.0.Final
Mar 14, 2015 5:47:17 PM org.xnio.nio.NioXnio <clinit>
INFO: XNIO NIO Implementation Version 3.3.0.Final
Mar 14, 2015 5:47:17 PM org.jboss.remoting3.EndpointImpl <clinit>
INFO: JBoss Remoting version 4.0.6.Final
     Got InitialContext ic: javax.naming.InitialContext@5cef8a56
    (hhu.object_factory.TestBean)ic.lookup(jnid_lookup) testBean = hhu.object_factory.TestBean@7c91f442
    name=City    value=Sattelmannsburg - ( believe me - this is not  in India) - Object Inst. Time: 1426351273285

Rerunning Test client: 
[oracle@wls1 JNDI]$  java BindJndiDemo  http-remoting://127.0.0.1:8180
Mar 14, 2015 5:47:19 PM org.xnio.Xnio <clinit>
...
    (hhu.object_factory.TestBean)ic.lookup(jnid_lookup) testBean = hhu.object_factory.TestBean@7c91f442
    name=City    value=Sattelmannsburg - ( believe me - this is not  in India) - Object Inst. Time: 1426351273285

--> The Object Instantiation time is always 1426351273285 so we can sure it is the same object!

Reference

Testing UCP Connection Pool with FCF against a RAC 12.1.0.2 database

Overview

  • Set CLASSPATH : $ export CLASSPATH=$ORACLE_HOME/jdbc/lib/ojdbc7.jar:.:$ORACLE_HOME/ucp/lib/ucp.jar:$ORACLE_HOME//opmn/lib/ons.jar
  •  Always use the newest version of:  ojdbc7.jar ucp.jar ons.jar [ 12.1.0.2 ]
  • Download location for java test program:  UcpRacTest.java

Run java test program and monitor UCP pool after and before instance shutdown

Check current instance status 
[oracle@hract21 ~]$ srvctl status database  -db banka
Instance bankA_1 is running on node hract21
Instance bankA_2 is running on node hract22

Java code to create UCP connection pool supporting FCF 
 public UcpRacTest() throws SQLException
    {
    // Create pool-enabled data source instance.
    pds = PoolDataSourceFactory.getPoolDataSource();
        // PoolDataSource and UCP configuration
         //set the connection properties on the data source and pool properties
    String ONS_CONFIG = "nodes=hract21:6200,hract22:6200,hract23:6200";
    pds.setONSConfiguration ( ONS_CONFIG);
    pds.setFastConnectionFailoverEnabled( true);
    pds.setUser("scott");
    pds.setPassword("tiger");
    pds.setURL("jdbc:oracle:thin:@ract2-scan.grid12c.example.com:1521/banka");
    pds.setConnectionFactoryClassName("oracle.jdbc.pool.OracleDataSource");
    pds.setInitialPoolSize(5);
    pds.setMinPoolSize(5);
    pds.setMaxPoolSize(20);
    System.out.println("--> UCP Pool wtih FCF sucessfuly initialized !");
    }

Java code to check closed connections 
  public String getInstanceInfo (Connection c) throws SQLException
    {
        String query1= "select name from v$database";
        String query2 = "select host_name,instance_name from v$instance";
        String query = null;

        String inst_info ="";
      // Statement stmt = c.createStatement();
        try ( Statement stmt =  c.createStatement() ) 
        {
            inst_info ="   RAC DB: ";
            query = query1;
            ResultSet rset = stmt.executeQuery (query);
            rset.next ();
            inst_info = inst_info +  rset.getString (1) ;

            query = query2;
            rset = stmt.executeQuery (query);
            while( rset.next() )
                inst_info =  inst_info + " - Instance Name:" +  rset.getString (2)  + " - Host: " +  rset.getString(1) ;
        } catch ( SQLException e1)
          {
            throw e1;
          }
    return inst_info;
    }

   public void displayPoolDetails (Connection c) throws SQLException
     {
     // System.out.println("-----------");
     System.out.print("NumberOfAvailableConnections: " + pds.getAvailableConnectionsCount());
     System.out.println(" - BorrowedConnectionsCount: " + pds.getBorrowedConnectionsCount());
     if ( c != null )
         System.out.println(getInstanceInfo(c));
     }
               
Run test program: 
[oracle@hract21 UCP]$  java UcpRacTest
Started at Wed Mar 11 12:21:30 CET 2015
--> UCP Pool wtih FCF sucessfuly initialized !
--> Opening 5 connections to a RAC DB !
getConnection(): NumberOfAvailableConnections: 4 - BorrowedConnectionsCount: 1
   RAC DB: BANKA - Instance Name:bankA_1 - Host: hract21.example.com
getConnection(): NumberOfAvailableConnections: 3 - BorrowedConnectionsCount: 2
   RAC DB: BANKA - Instance Name:bankA_2 - Host: hract22.example.com
getConnection(): NumberOfAvailableConnections: 2 - BorrowedConnectionsCount: 3
   RAC DB: BANKA - Instance Name:bankA_2 - Host: hract22.example.com
getConnection(): NumberOfAvailableConnections: 1 - BorrowedConnectionsCount: 4
   RAC DB: BANKA - Instance Name:bankA_1 - Host: hract21.example.com
getConnection(): NumberOfAvailableConnections: 0 - BorrowedConnectionsCount: 5
   RAC DB: BANKA - Instance Name:bankA_1 - Host: hract21.example.com
--> Closing all opend  RAC connections !
--> Pool stats After Closing all opened connection: NumberOfAvailableConnections: 5 - BorrowedConnectionsCount: 0

--> Shutdown an Instance and press <CR>:

Now shutdown instance bankA_1 
[oracle@hract21 ~]$  srvctl stop instance -db banka -i bankA_1 -o abort
[oracle@hract21 ~]$  srvctl status database  -db banka
Instance bankA_2 is running on node hract22
Instance bankA_1 is not running on node hract21
and continue program by pressing <CR> :
--> Opening 5 connections to a RAC DB !
getConnection(): NumberOfAvailableConnections: 4 - BorrowedConnectionsCount: 1
   RAC DB: BANKA - Instance Name:bankA_2 - Host: hract22.example.com
getConnection(): NumberOfAvailableConnections: 3 - BorrowedConnectionsCount: 2
   RAC DB: BANKA - Instance Name:bankA_2 - Host: hract22.example.com
getConnection(): NumberOfAvailableConnections: 2 - BorrowedConnectionsCount: 3
   RAC DB: BANKA - Instance Name:bankA_2 - Host: hract22.example.com
getConnection(): NumberOfAvailableConnections: 1 - BorrowedConnectionsCount: 4
   RAC DB: BANKA - Instance Name:bankA_2 - Host: hract22.example.com
getConnection(): NumberOfAvailableConnections: 0 - BorrowedConnectionsCount: 5
   RAC DB: BANKA - Instance Name:bankA_2 - Host: hract22.example.com
--> Closing all opend  RAC connections !
--> Pool stats After Closing all opened connection: NumberOfAvailableConnections: 5 - BorrowedConnectionsCount: 0
Ended at Wed Mar 11 12:23:43 CET 2015

 

Test Summary

  • All connections are reconnected to  instance bankA_2 after shutdown of instance  bankA_1 immediately
  •   The failed RAC system generates a DOWN FAN EVENT and send a message to our UCP pool
  •    Note we see no errors when we reusing the staled connections from the UCP pool
  •    The RAC instance has notified the UCP Pool via ONS to close and reconnect the stale connections pointing to hract21

Install RAC 12.1.0.2 PM – Policy Managed

Feature Overview

  • FLEX ASM
  • Policy Managed Database ( PM)
  • GNS – requirement for Policy Managed Database
  • UDEV to manage ASM disks

 Versions used

  • VirtualBox 4.3.20
  • OEL 6.6
  • Oracle RAC  12.1.0.2 using UDEV, Policy Managed Database , Flex ASM  feature

 Virtualbox Images used

  • ns1      – Name Server / DHCP server running on 192.168.5.50 IP addresss
  • oel66    – Basic Linix Image which we use for cloning [ Vbox image we need to install first ]
  • hract21  – Cluster node 1
  • hract22  – Cluster node 2
  • hract23  – Cluster node 3

  For installing the nameserver please read the article mentioned  below :

Create VirtualBox Image OEL66 as our RAC Provisioning Server

  -Install OEL 6.6 base Linux System with all needed packages 
  -Setup Network, Groups, Users to allow this RAC Provisiong Server to used :
     For cloning 
     For quick reinstall 
     For adding a new node to our cluster 

Download OEL V52218-01.iso and bind this ISO image to your Boot CD boot !
During the Linux installation --> Use customize NOW 
The "Package Group Selection" screen allows you to select the required package groups, and i
ndividual packages within the details section. When you've made your selection, click the "Next" button. 
If you want the server to have a regular gnome desktop you need to include the following package 
groups from the "Desktops" section:

    Desktop
    Desktop Platform
    Fonts
    General Purpose Desktop
    Graphical Administration Tools
    X Windows System
For details see : http://oracle-base.com/articles/linux/oracle-linux-6-installation.php

Network Configurations 
Status : We assume DNS/DHCP server  are already runing  running !

Please read the following article to configure:  Virtualbox Network devices for RAC and Internet Access

Install the newest available Kernel package version : 
[root@hract21 Desktop]# yum update
Loaded plugins: refresh-packagekit, security
Setting up Update Process
public_ol6_UEKR3_latest                                  | 1.2 kB     00:00     
public_ol6_UEKR3_latest/primary                          |  11 MB     00:08     
public_ol6_UEKR3_latest                                                 288/288
public_ol6_latest                                        | 1.4 kB     00:00     
public_ol6_latest/primary                                |  45 MB     00:50     
public_ol6_latest                                                   29161/29161
Resolving Dependencies
--> Running transaction check
---> Package at.x86_64 0:3.1.10-43.el6_2.1 will be updated
---> Package at.x86_64 0:3.1.10-44.el6_6.2 will be an update
..
  xorg-x11-server-Xorg.x86_64 0:1.15.0-25.el6_6                                 
  xorg-x11-server-common.x86_64 0:1.15.0-25.el6_6                               
  yum-rhn-plugin.noarch 0:0.9.1-52.0.1.el6_6                                    
Complete!

Determine your current kernel versions 
[root@hract21 Desktop]# uname -a
Linux hract21.example.com 3.8.13-55.1.2.el6uek.x86_64 #2 SMP Thu Dec 18 00:15:51 PST 2014 x86_64 x86_64 x86_64 GNU/Linux

Find the related kernel packages for downloading 
[root@hract21 Desktop]# yum list | grep 3.8.13-55
kernel-uek.x86_64                    3.8.13-55.1.2.el6uek     @public_ol6_UEKR3_latest
kernel-uek-devel.x86_64              3.8.13-55.1.2.el6uek     @public_ol6_UEKR3_latest
kernel-uek-doc.noarch                3.8.13-55.1.2.el6uek     @public_ol6_UEKR3_latest
kernel-uek-firmware.noarch           3.8.13-55.1.2.el6uek     @public_ol6_UEKR3_latest
dtrace-modules-3.8.13-55.1.1.el6uek.x86_64
dtrace-modules-3.8.13-55.1.2.el6uek.x86_64
dtrace-modules-3.8.13-55.el6uek.x86_64
kernel-uek-debug.x86_64              3.8.13-55.1.2.el6uek     public_ol6_UEKR3_latest
kernel-uek-debug-devel.x86_64        3.8.13-55.1.2.el6uek     public_ol6_UEKR3_latest

[root@hract21 Desktop]# yum install kernel-uek-devel.x86_64
Loaded plugins: refresh-packagekit, security
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package kernel-uek-devel.x86_64 0:3.8.13-55.1.2.el6uek will be installed
--> Finished Dependency Resolution

Verify installed kernel sources
[root@hract21 Desktop]#  ls /usr/src/kernels
2.6.32-504.3.3.el6.x86_64  3.8.13-55.1.2.el6uek.x86_64

Rebuild VirtualBox non-DKMS kernel 
[root@hract21 Desktop]# /etc/init.d/vboxadd setup
Removing existing VirtualBox non-DKMS kernel modules       [  OK  ]
Building the VirtualBox Guest Additions kernel modules
Building the main Guest Additions module                   [  OK  ]
Building the shared folder support module                  [  OK  ]
Building the OpenGL support module                         [  OK  ]
Doing non-kernel setup of the Guest Additions              [  OK  ]
Starting the VirtualBox Guest Additions                    [  OK  ]

--> Reboot system and verify Virtualbox device drivers
[root@hract21 Desktop]# lsmod | grep vbox
vboxsf                 38015  0 
vboxguest             263369  7 vboxsf
vboxvideo               2154  1 
drm                   274140  2 vboxvideo

Verify Network setup 
[root@hract21 network-scripts]# cat /etc/resolv.conf
# Generated by NetworkManager
search example.com grid12c.example.com
nameserver 192.168.5.50

After configuring your network devices / Firewall status:
[root@hract21 network-scripts]# ifconfig 
eth0      Link encap:Ethernet  HWaddr 08:00:27:31:B8:A0  
          inet addr:192.168.1.8  Bcast:192.168.1.255  Mask:255.255.255.0
  
eth1      Link encap:Ethernet  HWaddr 08:00:27:7B:E2:09  
          inet addr:192.168.5.121  Bcast:192.168.5.255  Mask:255.255.255.0

eth2      Link encap:Ethernet  HWaddr 08:00:27:C8:CA:AD  
          inet addr:192.168.2.121  Bcast:192.168.2.255  Mask:255.255.255.0

eth3      Link encap:Ethernet  HWaddr 08:00:27:2A:0B:EC  
          inet addr:192.168.3.121  Bcast:192.168.3.255  Mask:255.255.255.0

Note : eth0 is a DHCP address from our router whereas eth1, eht2 and eth3 are fixed addresses.

At this point DNS name resoluiton and ping  even for Internet hosts  should work :  
[root@hract21 network-scripts]#  ping google.de
PING google.de (173.194.112.191) 56(84) bytes of data.
64 bytes from fra07s32-in-f31.1e100.net (173.194.112.191): icmp_seq=1 ttl=47 time=40.6 ms
64 bytes from fra07s32-in-f31.1e100.net (173.194.112.191): icmp_seq=2 ttl=47 time=40.2 ms

For add. info you may read: 
 - Configure DNS, NTP and DHCP  for a mixed RAC/Internet usage  

Setup NTP ( this in the NTP setup for the nameserver !) 
As we have Internet connection we can use the default configure NTP servers 
[root@hract21 Desktop]#  more /etc/ntp.conf
server 0.rhel.pool.ntp.org iburst
server 1.rhel.pool.ntp.org iburst
server 2.rhel.pool.ntp.org iburst
server 3.rhel.pool.ntp.org iburst

# service ntpd stop
Shutting down ntpd:                                        [  OK  ]

If your RAC is going to be permanently connected to your main network and you want to use NTP, you must add 
the "-x" option into the following line in the "/etc/sysconfig/ntpd" file.

    OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid"

Then restart NTP and verify your setup 
Note NTP must run on all custer nodes and of course on your nameservr
# service ntpd restart

[root@hract21 Desktop]# ntpq -p
     remote           refid     st t when poll reach   delay   offset  jitter
==============================================================================
*ridcully.episod 5.9.39.5         3 u    -   64    1   47.564  -66.443   3.231
 alvo.fungus.at  193.170.62.252   3 u    1   64    1   52.608  -68.424   2.697
 main.macht.org  192.53.103.103   2 u    1   64    1   56.734  -71.436   3.186
 formularfetisch 131.188.3.223    2 u    -   64    1   37.748  -78.676  13.875

Add to rc.local 
service ntpd stop
ntpdate -u 192.168.5.50
service ntpd start


OS setup : 
Turn off and disable the firewall IPTables , disable SELinux and disable AVAHI daemon
# service iptables stop
iptables: Flushing firewall rules:                         [  OK  ]
iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
iptables: Unloading modules:                               [  OK  ]
# chkconfig iptables off
# chkconfig --list iptables
iptables        0:off   1:off   2:off   3:off   4:off   5:off   6:off

[root@hract21 Desktop]# service iptables status
iptables: Firewall is not running.

Disable SELinux. Open the config file and change the SELINUX variable from enforcing to disabled.
[root@hract21 Desktop]#   cat /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
#     targeted - Targeted processes are protected,
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

Disable AVAHI daemon  [ only if running ]
# /etc/init.d/avahi-daemon stop
To disable it: 
# /sbin/chkconfig  avahi-daemon off

Install some helpful packages
[root@hract21 network-scripts]# yum install wireshark
[root@hract21 network-scripts]# yum install wireshark-gnome
Install X11 applications like xclock
[root@hract21 network-scripts]# yum install xorg-x11-apps
[root@hract21 network-scripts]# yum install telnet 

[grid@hract21 rpm]$ cd /media/sf_kits/Oracle/12.1.0.2/grid/rpm
[root@hract21 rpm]#  rpm -iv cvuqdisk-1.0.9-1.rpm
Preparing packages for installation...
Using default group oinstall to install package
cvuqdisk-1.0.9-1

[root@hract21 network-scripts]# yum list | grep oracle-rdbms-server
oracle-rdbms-server-11gR2-preinstall.x86_64
oracle-rdbms-server-12cR1-preinstall.x86_64
[root@hract21 network-scripts]# yum install oracle-rdbms-server-12cR1-preinstall.x86_64
Loaded plugins: refresh-packagekit, security
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package oracle-rdbms-server-12cR1-preinstall.x86_64 0:1.0-12.el6 will be installed

Create Users and Groups
Add to grid .bashrc 
export ORACLE_BASE=/u01/app/grid
export ORACLE_SID=+ASM1
export GRID_HOME=/u01/app/121/grid
export ORACLE_HOME=$GRID_HOME
export PATH=$ORACLE_HOME/bin:.:$PATH
export HOST=`/bin/hostname`
alias h=history
unalias ls 
alias sys='sqlplus / as sysdba'
alias sql='sqlplus scott/tiger'

Add to oracle .bashrc 
export ORACLE_BASE=/u01/app/grid
export ORACLE_SID=ract2
export GRID_HOME=/u01/app/121/grid
export ORACLE_HOME=/u01/app/oracle/product/121/rac121
export PATH=$ORACLE_HOME/bin:.:$PATH
export  LD_LIBRARY_PATH=$ORACLE_HOME/lib:.
export HOST=`/bin/hostname`
alias h=history
unalias ls 
alias sys='sqlplus / as sysdba'
alias sql='sqlplus scott/tiger'

export ORACLE_SID=`ps -elf | grep ora_smon | grep -v grep | awk ' { print  substr( $15,10) }' `
export CLASSPATH=$ORACLE_HOME/jdbc/lib/ojdbc6_g.jar:.
echo  "-> Active ORACLE_SID:  " $ORACLE_SID 

alias h=history 
alias oh='cd $ORACLE_HOME'
alias sys1='sqlplus sys/sys@ract2_1 as sysdba'
alias sys2='sqlplus sys/sys@ract2_2 as sysdba'
alias sys3='sqlplus sys/sys@ract2_3 as sysdba'
alias sql1='sqlplus scott/tiger@ract1'
alias sql2='sqlplus scott/tiger@ract2'
alias sql3='sqlplus scott/tiger@ract3'
alias trc1='cd /u01/app/oracle/diag/rdbms/ract2/ract2_1/trace'
alias trc2='cd /u01/app/oracle/diag/rdbms/ract2/gract2_2/trace'
alias trc3='cd /u01/app/oracle/diag/rdbms/ractr/gract2_3/trace'

Create groups: 
[root@hract21 network-scripts]# /usr/sbin/groupadd -g 501 oinstall
groupadd: group 'oinstall' already exists
[root@hract21 network-scripts]# /usr/sbin/groupadd -g 502 dba
groupadd: group 'dba' already exists
[root@hract21 network-scripts]# /usr/sbin/groupadd -g 504 asmadmin
[root@hract21 network-scripts]# /usr/sbin/groupadd -g 506 asmdba
[root@hract21 network-scripts]# /usr/sbin/groupadd -g 507 asmoper
[root@hract21 network-scripts]# /usr/sbin/useradd -u 501 -g oinstall -G asmadmin,asmdba,asmoper grid
[root@hract21 network-scripts]# /usr/sbin//userdel oracle
[root@hract21 network-scripts]# /usr/sbin/useradd -u 502 -g oinstall -G dba,asmdba oracle

[root@hract21 network-scripts]#  su - oracle 
[oracle@hract21 ~]$ id
uid=502(oracle) gid=54321(oinstall) groups=54321(oinstall),506(asmdba),54322(dba) 

[root@hract21 network-scripts]# su - grid
[grid@hract21 ~]$ id
uid=501(grid) gid=54321(oinstall) groups=54321(oinstall),504(asmadmin),506(asmdba),507(asmoper) 

For the C shell (csh or tcsh), add the following lines to the /etc/csh.login file:
  if ( $USER = "oracle" || $USER = "grid" ) then
  limit maxproc 16384
  limit descriptors 65536
  endif

Modify  /etc/security/limits.conf
oracle   soft   nofile    1024
oracle   hard   nofile    65536
oracle   soft   nproc    2047
oracle   hard   nproc    16384
oracle   soft   stack    10240
oracle   hard   stack    32768
grid     soft   nofile    1024
grid     hard   nofile    65536
grid     soft   nproc    2047
grid     hard   nproc    16384
grid     soft   stack    10240
grid     hard   stack    32768

Create Directories:
 - Have a separate ORACLE_BASE for both GRID and RDBMS install !
Create the Oracle Inventory Directory
To create the Oracle Inventory directory, enter the following commands as the root user:
  # mkdir -p /u01/app/oraInventory
  # chown -R grid:oinstall /u01/app/oraInventory

Creating the Oracle Grid Infrastructure Home Directory
To create the Grid Infrastructure home directory, enter the following commands as the root user:
  # mkdir -p /u01/app/grid
  # chown -R grid:oinstall /u01/app/grid
  # chmod -R 775 /u01/app/grid
  # mkdir -p /u01/app/121/grid
  # chown -R grid:oinstall /u01//app/121/grid
  # chmod -R 775 /u01/app/121/grid

Creating the Oracle Base Directory
  To create the Oracle Base directory, enter the following commands as the root user:
  # mkdir -p /u01/app/oracle
  # chown -R oracle:oinstall /u01/app/oracle
  # chmod -R 775 /u01/app/oracle

Creating the Oracle RDBMS Home Directory
  To create the Oracle RDBMS Home directory, enter the following commands as the root user:
  # mkdir -p /u01/app/oracle/product/121/rac121
  # chown -R oracle:oinstall /u01/app/oracle/product/121/rac121
  # chmod -R 775 /u01/app/oracle/product/121/rac121

Download cluvfy from : http://www.oracle.com/technetwork/database/options/clustering/downloads/index.html

Cluster Verification Utility Download for Oracle Grid Infrastructure 12c 
Note: The latest CVU version (July 2013) can be used with all currently supported Oracle RAC versions, including Oracle RAC 10g, 
      Oracle RAC 11g and Oracle RAC 12c.

Unzip cluvfy:
[grid@hract21 CLUVFY]$ unzip /tmp/cvupack_Linux_x86_64.zip
[grid@hract21 CLUVFY]$ pwd
/home/grid/CLUVFY
[grid@hract21 CLUVFY]$ ls
bin  clone  crs  css  cv  deinstall  diagnostics  has  install  jdbc  jdk  jlib  lib  network  nls  oracore  oui  srvm  utl  xdk
[grid@hract21 CLUVFY]$ bin/cluvfy -version
12.1.0.1.0 Build 112713x8664

Run cluvfy to verify the OS current installation  
Verify OS setup :
As Grid user
$ ./bin/cluvfy comp sys -p crs -n gract2 -verbose -fixup
--> If needed run the fix script and/or fix underlying problems 
As root user verify DHCP setup 

Verify DHCP setup :
[root@hract21 CLUVFY]#  ./bin/cluvfy comp dhcp -clustername  ract2 -verbose
Verifying DHCP Check 
Checking if any DHCP server exists on the network...
DHCP server returned server: 192.168.5.50, loan address: 192.168.5.218/255.255.255.0, lease time: 21600
At least one DHCP server exists on the network and is listening on port 67
Checking if DHCP server has sufficient free IP addresses for all VIPs...
Sending DHCP "DISCOVER" packets for client ID "ract2-scan1-vip"
DHCP server returned server: 192.168.5.50, loan address: 192.168.5.218/255.255.255.0, lease time: 21600
Sending DHCP "REQUEST" packets for client ID "ract2-scan1-vip"
.. 

Verify GNS setup : 
[grid@hract21 CLUVFY]$  ./bin/cluvfy comp gns -precrsinst -domain grid12c.example.com  -vip 192.168.5.58 
Verifying GNS integrity 
Checking GNS integrity...
The GNS subdomain name "grid12c.example.com" is a valid domain name
GNS VIP "192.168.5.58" resolves to a valid IP address
GNS integrity check passed
Verification of GNS integrity was successful. 
--> Note you may get the PRVF-5229 warning if this address is in use [ maybe be a different GNS VIP ]

Add this time we have created a base system which we will now clone 3x for our rac nodes

Clone base system 

You man change in File-> Preference the default machine path first  
M:\VM\RAC_OEL66_12102

Cloning ract21 :
Now cleanly shutdown your Reference/Clone system 
Virtualbox -> Clone [ Name clone ract21 ]  ->  add new Network Addresses -> Full Clone 

Boot system a first time - retrieve the new MAC addresses 
[root@hract21 Desktop]# dmesg |grep eth
e1000 0000:00:03.0 eth0: (PCI:33MHz:32-bit) 08:00:27:e7:c0:6b
e1000 0000:00:03.0 eth0: Intel(R) PRO/1000 Network Connection
e1000 0000:00:08.0 eth1: (PCI:33MHz:32-bit) 08:00:27:7d:8e:49
e1000 0000:00:08.0 eth1: Intel(R) PRO/1000 Network Connection
e1000 0000:00:09.0 eth2: (PCI:33MHz:32-bit) 08:00:27:4e:c9:bf
e1000 0000:00:09.0 eth2: Intel(R) PRO/1000 Network Connection
e1000 0000:00:0a.0 eth3: (PCI:33MHz:32-bit) 08:00:27:3b:89:bf
e1000 0000:00:0a.0 eth3: Intel(R) PRO/1000 Network Connection

[root@hract21 network-scripts]# egrep 'HWADDR|IP' ifcfg-eth*
ifcfg-eth0:HWADDR=08:00:27:e7:c0:6b
ifcfg-eth1:HWADDR=08:00:27:7d:8e:49
ifcfg-eth1:IPADDR=192.168.5.121
ifcfg-eth2:HWADDR=08:00:27:4e:c9:bf
ifcfg-eth2:IPADDR=192.168.2.121
ifcfg-eth3:HWADDR=08:00:27:3b:89:bf
ifcfg-eth3:IPADDR=192.168.3.121
 
Remove file 
[root@hract21 Desktop]# rm  /etc/udev/rules.d/70-persistent-net.rules
rm: remove regular file `/etc/udev/rules.d/70-persistent-net.rules'? y

Change hostname in the “/etc/sysconfig/network” 
[root@gract21 network-scripts]# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=hract21.example.com
NTPSERVERARGS=iburst
# oracle-rdbms-server-12cR1-preinstall : Add NOZEROCONF=yes
NOZEROCONF=yes

-> And finally  reboot system and verify network setup

[root@hract21 network-scripts]# ifconfig | egrep 'eth|inet addr'
eth0      Link encap:Ethernet  HWaddr 08:00:27:E7:C0:6B  
          inet addr:192.168.1.14  Bcast:192.168.1.255  Mask:255.255.255.0
eth1      Link encap:Ethernet  HWaddr 08:00:27:7D:8E:49  
          inet addr:192.168.5.121  Bcast:192.168.5.255  Mask:255.255.255.0
eth2      Link encap:Ethernet  HWaddr 08:00:27:4E:C9:BF  
          inet addr:192.168.2.121  Bcast:192.168.2.255  Mask:255.255.255.0
eth3      Link encap:Ethernet  HWaddr 08:00:27:3B:89:BF  
          inet addr:192.168.3.121  Bcast:192.168.3.255  Mask:255.255.255.0

root@hract21 network-scripts]# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG        0 0          0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 eth0
192.168.2.0     0.0.0.0         255.255.255.0   U         0 0          0 eth2
192.168.3.0     0.0.0.0         255.255.255.0   U         0 0          0 eth3
192.168.5.0     0.0.0.0         255.255.255.0   U         0 0          0 eth1

Repeat steps for ract22, ract23 !

Create Asm Disks

cd M:\VM\RAC_OEL66_12102

VBoxManage createhd --filename M:\VM\RAC_OEL66_12102\asm1_12102_10G.vdi --size 10240 --format VDI --variant Fixed
VBoxManage createhd --filename M:\VM\RAC_OEL66_12102\asm2_12102_10G.vdi --size 10240 --format VDI --variant Fixed
VBoxManage createhd --filename M:\VM\RAC_OEL66_12102\asm3_12102_10G.vdi --size 10240 --format VDI --variant Fixed
VBoxManage createhd --filename M:\VM\RAC_OEL66_12102\asm4_12102_10G.vdi --size 10240 --format VDI --variant Fixed

VBoxManage modifyhd  asm1_12102_10G.vdi  --type shareable
VBoxManage modifyhd  asm2_12102_10G.vdi  --type shareable
VBoxManage modifyhd  asm3_12102_10G.vdi  --type shareable
VBoxManage modifyhd  asm4_12102_10G.vdi  --type shareable

VBoxManage storageattach ract21 --storagectl "SATA" --port 1 --device 0 --type hdd --medium asm1_12102_10G.vdi --mtype shareable
VBoxManage storageattach ract21 --storagectl "SATA" --port 2 --device 0 --type hdd --medium asm2_12102_10G.vdi --mtype shareable
VBoxManage storageattach ract21 --storagectl "SATA" --port 3 --device 0 --type hdd --medium asm3_12102_10G.vdi --mtype shareable
VBoxManage storageattach ract21 --storagectl "SATA" --port 4 --device 0 --type hdd --medium asm4_12102_10G.vdi --mtype shareable
   
VBoxManage storageattach ract22 --storagectl "SATA" --port 1 --device 0 --type hdd --medium asm1_12102_10G.vdi --mtype shareable
VBoxManage storageattach ract22 --storagectl "SATA" --port 2 --device 0 --type hdd --medium asm2_12102_10G.vdi --mtype shareable
VBoxManage storageattach ract22 --storagectl "SATA" --port 3 --device 0 --type hdd --medium asm3_12102_10G.vdi --mtype shareable
VBoxManage storageattach ract22 --storagectl "SATA" --port 4 --device 0 --type hdd --medium asm4_12102_10G.vdi --mtype shareable

VBoxManage storageattach ract23 --storagectl "SATA" --port 1 --device 0 --type hdd --medium asm1_12102_10G.vdi --mtype shareable
VBoxManage storageattach ract23 --storagectl "SATA" --port 2 --device 0 --type hdd --medium asm2_12102_10G.vdi --mtype shareable
VBoxManage storageattach ract23 --storagectl "SATA" --port 3 --device 0 --type hdd --medium asm3_12102_10G.vdi --mtype shareable
VBoxManage storageattach ract23 --storagectl "SATA" --port 4 --device 0 --type hdd --medium asm4_12102_10G.vdi --mtype shareable

Check newly created disk devices after RAC node reboot
[root@hract21 Desktop]# ls /dev/sd*
/dev/sda  /dev/sda1  /dev/sda2  /dev/sdb  /dev/sdc  /dev/sdd  /dev/sde

Run fdisk to partttion the new disk ( we only want a single partition )
[root@hract21 Desktop]# fdisk /dev/sdb
Command (m for help): p
Disk /dev/sdb: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xf9bddbc6
   Device Boot      Start         End      Blocks   Id  System
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1305, default 1): 1
Last cylinder, +cylinders or +size{K,M,G} (1-1305, default 1305): 
Using default value 1305
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
--> Repeat above step for  /dev/sdc  /dev/sdd  /dev/sde and verify the created devices.
[root@hract21 Desktop]# ls /dev/sd*
/dev/sda  /dev/sda1  /dev/sda2  /dev/sdb  /dev/sdb1  /dev/sdc  /dev/sdc1  /dev/sdd  /dev/sdd1  /dev/sde  /dev/sde1

Use following bash script to return the WWID disk IDs  : http://www.hhutzler.de/blog/configure-udev-rules-for-asm-devices/

[root@hract21 ~]# ./check_wwid.sh
/dev/sda  WWID:    1ATA_VBOX_HARDDISK_VB98f7f6e6-e47cb456
/dev/sda1  WWID:   1ATA_VBOX_HARDDISK_VB98f7f6e6-e47cb456
/dev/sda2  WWID:   1ATA_VBOX_HARDDISK_VB98f7f6e6-e47cb456
/dev/sdb  WWID:    1ATA_VBOX_HARDDISK_VBe7363848-cbf94b0c
/dev/sdb1  WWID:   1ATA_VBOX_HARDDISK_VBe7363848-cbf94b0c
/dev/sdc  WWID:    1ATA_VBOX_HARDDISK_VBb322a188-b4771866
/dev/sdc1  WWID:   1ATA_VBOX_HARDDISK_VBb322a188-b4771866
/dev/sdd  WWID:    1ATA_VBOX_HARDDISK_VB00b7878b-c50d45f4
/dev/sdd1  WWID:   1ATA_VBOX_HARDDISK_VB00b7878b-c50d45f4
/dev/sde  WWID:    1ATA_VBOX_HARDDISK_VB7a3701f8-f1272747
/dev/sde1  WWID:   1ATA_VBOX_HARDDISK_VB7a3701f8-f1272747

Create 99-oracle-asmdevices.rules - change the RESULT values by using the output of our  ./check_wwid.sh script :
[root@hract21 rules.d]#  cd /etc/udev/rules.d
[root@hract21 rules.d]#  cat  99-oracle-asmdevices.rules
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VBe7363848-cbf94b0c", NAME= "asmdisk1_10G", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VBb322a188-b4771866", NAME= "asmdisk2_10G", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB00b7878b-c50d45f4", NAME= "asmdisk3_10G", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB7a3701f8-f1272747", NAME= "asmdisk4_10G", OWNER="grid", GROUP="asmadmin", MODE="0660"

[root@hract21 ~]# udevadm control --reload-rules
[root@hract21 ~]# start_udev
Starting udev: udevd[14512]: GOTO 'pulseaudio_check_usb' has no matching label in: '/lib/udev/rules.d/90-pulseaudio.rules'
                                                           [  OK  ]
[root@hract21 ~]#  ls -ltr /dev/asmd*
brw-rw---- 1 grid asmadmin 8, 17 Jan 29 09:33 /dev/asmdisk1_10G
brw-rw---- 1 grid asmadmin 8, 49 Jan 29 09:33 /dev/asmdisk3_10G
brw-rw---- 1 grid asmadmin 8, 33 Jan 29 09:33 /dev/asmdisk2_10G
brw-rw---- 1 grid asmadmin 8, 65 Jan 29 09:33 /dev/asmdisk4_10G

Copy newly created rules file to the remaining rac nodes and restart udev
[root@hract21 rules.d]#  scp 99-oracle-asmdevices.rules hract22:/etc/udev/rules.d
[root@hract21 rules.d]#  scp 99-oracle-asmdevices.rules hract23:/etc/udev/rules.d
and run  following bash script to restart udev

Bash script: restart_udev.sh  
#!/bin/bash 
udevadm control --reload-rules
start_udev
ls -ltr /dev/asm*

Note the ls output on hract22 and hract23 should be indentical to the output on hract21 !

Here you may add oracle and grid user to the vboxsf group. 
This allows us to use the mounted/shared VBOX devices !
[root@hract21 ~]#  usermod -G vboxsf oracle
[root@hract21 ~]#  usermod -G vboxsf grid

Setup ssh connectivity
[grid@hract21 ~]$  cp /media/sf_kits/Oracle/12.1.0.2/grid/sshsetup/sshUserSetup.sh .
[grid@hract21 ~]$  ./sshUserSetup.sh -user grid -hosts "hract21  hract22 hract23" -noPromptPassphrase

[grid@hract21 ~]$  /usr/bin/ssh -x -l grid hract21 date
Thu Jan 29 11:06:55 CET 2015
[grid@hract21 ~]$  /usr/bin/ssh -x -l grid hract22 date
Thu Jan 29 11:06:56 CET 2015
[grid@hract21 ~]$  /usr/bin/ssh -x -l grid hract21 date
Thu Jan 29 11:07:01 CET 2015

NTP setup on all RAC nodes
Note only our Name server is getting the time from the Internet 

For the RAC nodes add to ntp.conf only a single server ( which is our nameserver ) 
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
server 192.168.5.50 
[root@hract21 etc]#  service ntp restart
ntp: unrecognized service
[root@hract21 etc]# service ntpd restart
Shutting down ntpd:                                        [  OK  ]
Starting ntpd:                                             [  OK  ]
[root@hract21 etc]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 ns1.example.com 131.188.3.220   10 u    5   64    1    0.149  -170.05   0.000

Run cluvfy  and install GRID software

Run Cluvfy 
[grid@hract21 CLUVFY]$ ./bin/cluvfy stage -pre crsinst -asm -presence local -asmgrp asmadmin  \
    -asmdev /dev/asmdisk1_10G,/dev/asmdisk2_10G,/dev/asmdisk3_10G,/dev/asmdisk4_10G    \
    -networks eth1:192.168.5.0:PUBLIC/eth2:192.168.2.0:cluster_interconnect  \
    -n hract21,hract22,hract23 | egrep 'PRVF|fail'
Node reachability check failed from node "hract21"
Total memory check failed
Check failed on nodes: 
PRVF-9802 : Attempt to get udev information from node "hract21" failed
PRVF-9802 : Attempt to get udev information from node "hract23" failed
UDev attributes check failed for ASM Disks 
--> The PRVFV-9802 error is explained in following article 
    The Memory check failed as I had reduced RAC Vbox images to 4 GByte 
    For other cluvfy errors you may check this article 

Installing  GRID software 
[grid@hract21 CLUVFY]$ cd /media/sf_kits/Oracle/12.1.0.2/grid

$ cd grid
$ ls
install  response  rpm    runcluvfy.sh  runInstaller  sshsetup  stage  welcome.html
$ ./runInstaller 
-> Configure a standard cluster
-> Advanced Installation
   Cluster name : ract2
   Scan name    : ract2-scan.grid12c.example.com
   Scan port    : 1521
   -> Create New GNS
      GNS VIP address: 192.168.1.58
      GNS Sub domain : grid12c.example.com
  Public Hostname           Virtual Hostname 
  hract21.example.com        AUTO
  hract22.example.com        AUTO
  hract23.example.com        AUTO

-> Test and Setup SSH connectivity
-> Setup network Interfaces
   eth0: don't use
   eth1: PUBLIC                              192.168.5.X
   eht2: Private Cluster_Interconnect,ASM    192.168.2.X
 
-> Configure GRID Infrastruce: YES
-> Use standard ASM for storage
-> ASM setup
   Diskgroup         : DATA
   Disk discover PATH: /dev/asm*
--> Don't use IPMI

Run root scritps:
[root@hract21 etc]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.

[root@hract21 etc]# /u01/app/121/grid/root.sh
Performing root user operation.
The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/121/grid
..
LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 30-JAN-2015 12:39:53
Copyright (c) 1991, 2014, Oracle.  All rights reserved.
CRS-5014: Agent "ORAAGENT" timed out starting process "/u01/app/121/grid/bin/lsnrctl" for action "check": details at "(:CLSN00009:)" in "/u01/app/grid/diag/crs/hract21/crs/trace/crsd_oraagent_grid.trc"
CRS-5017: The resource action "ora.MGMTLSNR check" encountered the following error: 
(:CLSN00009:)Command Aborted. For details refer to "(:CLSN00109:)" in "/u01/app/grid/diag/crs/hract21/crs/trace/crsd_oraagent_grid.trc".
CRS-2664: Resource 'ora.DATA.dg' is already running on 'hract21'
CRS-6017: Processing resource auto-start for servers: hract21
CRS-2672: Attempting to start 'ora.oc4j' on 'hract21'
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'hract21'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'hract21' succeeded
CRS-2676: Start of 'ora.oc4j' on 'hract21' succeeded
CRS-6016: Resource auto-start has completed for server hract21
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2015/01/30 12:41:03 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Now run scripts below on hrac22 and hrac23
# /u01/app/oraInventory/orainstRoot.sh
# /u01/app/121/grid/root.sh

Verify CW status 
[root@hract21 ~]# crs
*****  Local Resources: *****
Rescource NAME                 TARGET     STATE           SERVER       STATE_DETAILS                       
-------------------------      ---------- ----------      ------------ ------------------                  
ora.ASMNET1LSNR_ASM.lsnr       ONLINE     ONLINE          hract21      STABLE   
ora.ASMNET1LSNR_ASM.lsnr       ONLINE     ONLINE          hract22      STABLE   
ora.ASMNET1LSNR_ASM.lsnr       ONLINE     ONLINE          hract23      STABLE   
ora.DATA.dg                    ONLINE     ONLINE          hract21      STABLE   
ora.DATA.dg                    ONLINE     ONLINE          hract22      STABLE   
ora.DATA.dg                    ONLINE     ONLINE          hract23      STABLE   
ora.LISTENER.lsnr              ONLINE     ONLINE          hract21      STABLE   
ora.LISTENER.lsnr              ONLINE     ONLINE          hract22      STABLE   
ora.LISTENER.lsnr              ONLINE     ONLINE          hract23      STABLE   
ora.net1.network               ONLINE     ONLINE          hract21      STABLE   
ora.net1.network               ONLINE     ONLINE          hract22      STABLE   
ora.net1.network               ONLINE     ONLINE          hract23      STABLE   
ora.ons                        ONLINE     ONLINE          hract21      STABLE   
ora.ons                        ONLINE     ONLINE          hract22      STABLE   
ora.ons                        ONLINE     ONLINE          hract23      STABLE   
*****  Cluster Resources: *****
Resource NAME               INST   TARGET       STATE        SERVER          STATE_DETAILS
--------------------------- ----   ------------ ------------ --------------- -----------------------------------------
ora.LISTENER_SCAN1.lsnr        1   ONLINE       ONLINE       hract22         STABLE  
ora.LISTENER_SCAN2.lsnr        1   ONLINE       ONLINE       hract23         STABLE  
ora.LISTENER_SCAN3.lsnr        1   ONLINE       ONLINE       hract21         STABLE  
ora.MGMTLSNR                   1   ONLINE       ONLINE       hract21         169.254.213.86 192.1 68.2.121,STABLE
ora.asm                        1   ONLINE       ONLINE       hract21         Started,STABLE  
ora.asm                        2   ONLINE       ONLINE       hract22         Started,STABLE  
ora.asm                        3   ONLINE       ONLINE       hract23         Started,STABLE  
ora.cvu                        1   ONLINE       ONLINE       hract21         STABLE  
ora.gns                        1   ONLINE       ONLINE       hract21         STABLE  
ora.gns.vip                    1   ONLINE       ONLINE       hract21         STABLE  
ora.hract21.vip                1   ONLINE       ONLINE       hract21         STABLE  
ora.hract22.vip                1   ONLINE       ONLINE       hract22         STABLE  
ora.hract23.vip                1   ONLINE       ONLINE       hract23         STABLE  
ora.mgmtdb                     1   ONLINE       ONLINE       hract21         Open,STABLE  
ora.oc4j                       1   ONLINE       ONLINE       hract21         STABLE  
ora.scan1.vip                  1   ONLINE       ONLINE       hract22         STABLE  
ora.scan2.vip                  1   ONLINE       ONLINE       hract23         STABLE  
ora.scan3.vip                  1   ONLINE       ONLINE       hract21         STABLE


Verify GNS SETUP / Network Setup 
[root@hract21 ~]# sh -x  check_net_12c.sh
+ dig @192.168.5.50 ract2-scan.grid12c.example.com
;; QUESTION SECTION:
;ract2-scan.grid12c.example.com.    IN    A

;; ANSWER SECTION:
ract2-scan.grid12c.example.com.    34 IN    A    192.168.5.236
ract2-scan.grid12c.example.com.    34 IN    A    192.168.5.220
ract2-scan.grid12c.example.com.    34 IN    A    192.168.5.218

;; AUTHORITY SECTION:
grid12c.example.com.    3600    IN    NS    gns12c.grid12c.example.com.
grid12c.example.com.    3600    IN    NS    ns1.example.com.

;; ADDITIONAL SECTION:
ns1.example.com.    3600    IN    A    192.168.5.50


+ dig @192.168.5.58 ract2-scan.grid12c.example.com

;; QUESTION SECTION:
;ract2-scan.grid12c.example.com.    IN    A

;; ANSWER SECTION:
ract2-scan.grid12c.example.com.    120 IN    A    192.168.5.218
ract2-scan.grid12c.example.com.    120 IN    A    192.168.5.220
ract2-scan.grid12c.example.com.    120 IN    A    192.168.5.236

;; AUTHORITY SECTION:
grid12c.example.com.    10800    IN    SOA    hract22. hostmaster.grid12c.example.com. 46558097 10800 10800 30 120

;; ADDITIONAL SECTION:
ract2-gns-vip.grid12c.example.com. 10800 IN A    192.168.5.58


+ nslookup ract2-scan
Server:        192.168.5.50
Address:    192.168.5.50#53
Non-authoritative answer:
Name:    ract2-scan.grid12c.example.com
Address: 192.168.5.236
Name:    ract2-scan.grid12c.example.com
Address: 192.168.5.218
Name:    ract2-scan.grid12c.example.com
Address: 192.168.5.220

+ ping -c 2 google.de
PING google.de (173.194.65.94) 56(84) bytes of data.
64 bytes from ee-in-f94.1e100.net (173.194.65.94): icmp_seq=1 ttl=38 time=177 ms
64 bytes from ee-in-f94.1e100.net (173.194.65.94): icmp_seq=2 ttl=38 time=134 ms
..

+ ping -c 2 hract21
PING hract21.example.com (192.168.5.121) 56(84) bytes of data.
64 bytes from hract21.example.com (192.168.5.121): icmp_seq=1 ttl=64 time=0.013 ms
64 bytes from hract21.example.com (192.168.5.121): icmp_seq=2 ttl=64 time=0.024 ms
..

+ ping -c 2 ract2-scan.grid12c.example.com
PING ract2-scan.grid12c.example.com (192.168.5.220) 56(84) bytes of data.
64 bytes from 192.168.5.220: icmp_seq=1 ttl=64 time=0.453 ms
64 bytes from 192.168.5.220: icmp_seq=2 ttl=64 time=0.150 ms
..

+ cat /etc/resolv.conf
# Generated by NetworkManager
search example.com grid12c.example.com
nameserver 192.168.5.50

Run Cluvfy and Install RDBMS software

Verify that your .bashrc doens't read/write any data from/to stdin/stdout 
Setup ssh connectivity :
[oracle@hract21 ~]$  ./sshUserSetup.sh -user oracle -hosts "hract21  hract22 hract23" -noPromptPassphrase
Verify ssh connectivity ( run this on hract22 and hract23 too )
[oracle@hract21 ~]$ ssh -x -l oracle hract21 date
Fri Jan 30 15:40:45 CET 2015
[oracle@hract21 ~]$ ssh -x -l oracle hract22 date
Fri Jan 30 15:40:50 CET 2015
[oracle@hract21 ~]$ ssh -x -l oracle hract23 date
Fri Jan 30 15:40:52 CET 2015

[grid@hract21 CLUVFY]$./bin/cluvfy stage -pre  dbinst  -n hract21,hract22,hract23 -d /u01/app/oracle/product/121/rac121 -fixup
Performing pre-checks for database installation 
Checking node reachability...
Node reachability check passed from node "hract21"
Checking user equivalence...
User equivalence check passed for user "grid"
ERROR: 
PRVG-11318 : The following error occurred during database operating system groups check. "PRCT-1005 :
 Directory /u01/app/oracle/product/121/rac121/bin does not exist"
 --> You can ignore this as we haven't installed a RAC DB software yet 

Checking node connectivity...
Checking hosts config file...
Verification of the hosts config file successful
Check: Node connectivity using interfaces on subnet "192.168.5.0"
Node connectivity passed for subnet "192.168.5.0" with node(s) hract22,hract23,hract21
TCP connectivity check passed for subnet "192.168.5.0"
Check: Node connectivity using interfaces on subnet "192.168.2.0"
Node connectivity passed for subnet "192.168.2.0" with node(s) hract22,hract23,hract21
TCP connectivity check passed for subnet "192.168.2.0"
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.5.0".
Subnet mask consistency check passed for subnet "192.168.2.0".
Subnet mask consistency check passed.
Node connectivity check passed
Checking multicast communication...
Checking subnet "192.168.2.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "192.168.2.0" for multicast communication with multicast group "224.0.0.251" passed.
Check of multicast communication passed.
Total memory check passed
Available memory check passed
Swap space check passed
Free disk space check passed for "hract23:/u01/app/oracle/product/121/rac121,hract23:/tmp"
Free disk space check passed for "hract22:/u01/app/oracle/product/121/rac121,hract22:/tmp"
Free disk space check passed for "hract21:/u01/app/oracle/product/121/rac121,hract21:/tmp"
Check for multiple users with UID value 501 passed 
User existence check passed for "grid"
Group existence check passed for "oinstall"
Group existence check passed for "dba"
Group existence check passed for "asmdba"
Membership check for user "grid" in group "oinstall" [as Primary] passed
Membership check for user "grid" in group "dba" passed
Membership check for user "grid" in group "asmdba" passed
Run level check passed
Hard limits check passed for "maximum open file descriptors"
Soft limits check passed for "maximum open file descriptors"
Hard limits check passed for "maximum user processes"
Soft limits check passed for "maximum user processes"
System architecture check passed
Kernel version check passed
..
Package existence check passed for "libaio-devel(x86_64)"
Check for multiple users with UID value 0 passed 
Current group ID check passed
Starting check for consistency of primary group of root user
Check for consistency of root user's primary group passed
Default user file creation mask check passed
Checking CRS integrity...
Clusterware version consistency passed.
CRS integrity check passed
Checking Cluster manager integrity... 
Checking CSS daemon...
Oracle Cluster Synchronization Services appear to be online.
Cluster manager integrity check passed
Checking node application existence...
Checking existence of VIP node application (required)
VIP node application check passed
Checking existence of NETWORK node application (required)
NETWORK node application check passed
Checking existence of ONS node application (optional)
ONS node application check passed
Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed
Checking if CTSS Resource is running on all nodes...
CTSS resource check passed
Querying CTSS for time offset on all nodes...
Query of CTSS for time offset passed
Check CTSS state started...
CTSS is in Observer state. Switching over to clock synchronization checks using NTP
Starting Clock synchronization checks using Network Time Protocol(NTP)...
NTP configuration file "/etc/ntp.conf" existence check passed
Liveness check passed for "ntpd"
Check for NTP daemon or service alive passed on all nodes
Check of common NTP Time Server passed
Clock time offset check passed
Clock synchronization check using Network Time Protocol(NTP) passed
Oracle Cluster Time Synchronization Services check passed
Checking integrity of file "/etc/resolv.conf" across nodes
"domain" and "search" entries do not coexist in any  "/etc/resolv.conf" file
All nodes have same "search" order defined in file "/etc/resolv.conf"
The DNS response time for an unreachable node is within acceptable limit on all nodes
Check for integrity of file "/etc/resolv.conf" passed
Time zone consistency check passed
Checking Single Client Access Name (SCAN)...
Checking TCP connectivity to SCAN Listeners...
TCP connectivity to SCAN Listeners exists on all cluster nodes
Checking name resolution setup for "ract2-scan.grid12c.example.com"...
Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf"
Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed
Checking SCAN IP addresses...
Check of SCAN IP addresses passed
Verification of SCAN VIP and Listener setup passed
Checking GNS integrity...
The GNS subdomain name "grid12c.example.com" is a valid domain name
Checking if the GNS VIP belongs to same subnet as the public network...
Public network subnets "192.168.5.0, 192.168.5.0, 192.168.5.0" match with the GNS VIP "192.168.5.0, 192.168.5.0, 192.168.5.0"
GNS VIP "192.168.5.58" resolves to a valid IP address
GNS resolved IP addresses are reachable
GNS resource configuration check passed
GNS VIP resource configuration check passed.
GNS integrity check passed
Checking VIP configuration.
Checking VIP Subnet configuration.
Check for VIP Subnet configuration passed.
Checking VIP reachability
Check for VIP reachability passed.

ASM and CRS versions are compatible
Database Clusterware version compatibility passed.
Starting check for /dev/shm mounted as temporary file system ...
Check for /dev/shm mounted as temporary file system passe

NOTE: 
No fixable verification failures to fix
Pre-check for database installation was successful


Install the database software 

[oracle@hract21 database]$ id
uid=502(oracle) gid=54321(oinstall) groups=54321(oinstall),493(vboxsf),506(asmdba),54322(dba)
[oracle@hract21 database]$ cd /media/sf_kits/oracle/12.1.0.2/database
[oracle@hract21 database]$  ./runInstaller
--> Create and Configure a Database 
 --> Server Class
  --> Oracle Real Application Cluster database installation
   --> Policy Managed  
    --> Server Pool:  Top_Priority Cardinality :2
     --> Select all 3 RAC membemers
      --> Test/Create SSH connectivity
       --> Advanced Install 
        --> Select Generell Purpose / Transaction database type
         --> Target Database Memory : 800 MByte 
          --> Select ASM and for OSDDBA use group:  dba ( default )
 
Run root.sh : hract21, hract22, hract23

Start database banka on all nodes 
[oracle@hract21 database]$  srvctl status srvpool -a
Server pool name: Free
Active servers count: 2
Active server names: hract21,hract22
NAME=hract21 STATE=ONLINE
NAME=hract22 STATE=ONLINE
Server pool name: Generic
Active servers count: 0
Active server names: 
Server pool name: Top_Priority
Active servers count: 1
Active server names: hract23
NAME=hract23 STATE=ONLINE
[oracle@hract21 database]$ srvctl modify srvpool -g Top_Priority -l 3 -u 3

*****  Cluster Resources: *****
Resource NAME               INST   TARGET       STATE        SERVER          STATE_DETAILS
--------------------------- ----   ------------ ------------ --------------- -----------------------------------------
ora.banka.db                   1   ONLINE       ONLINE       hract23         Open,STABLE  
ora.banka.db                   2   ONLINE       ONLINE       hract21         Open,STABLE  
ora.banka.db                   3   ONLINE       ONLINE       hract22         Open,STABLE 

Stop one instance 
[oracle@hract21 database]$ srvctl modify srvpool -g Top_Priority -l 2 -u 2 -f
ora.banka.db                   1   ONLINE       ONLINE       hract23         Open,STABLE  
ora.banka.db                   2   ONLINE       OFFLINE      -               Instance Shutdown,ST ABLE
ora.banka.db                   3   ONLINE       ONLINE       hract22         Open,STABLE 

Invoke dbca and create database bankb

[oracle@hract21 database]$  ./dbca
   --> Policy Managed  
    --> Server Pool:  Low_Priority Cardinality :1
     --> Target Database Memory : 800 MByte 

Check server pools :
[oracle@hract21 database]$ srvctl status srvpool -a
Server pool name: Free
Active servers count: 0
Active server names: 
Server pool name: Generic
Active servers count: 0
Active server names: 
Server pool name: Low_Priority
Active servers count: 1
Active server names: hract21
NAME=hract21 STATE=ONLINE
Server pool name: Top_Priority
Active servers count: 2
Active server names: hract22,hract23
NAME=hract22 STATE=ONLINE
NAME=hract23 STATE=ONLINE
    

For details about serverpools read follwing article : http://www.hhutzler.de/blog/managing-server-pools/
*****  Cluster Resources: *****
Resource NAME               INST   TARGET       STATE        SERVER          STATE_DETAILS
--------------------------- ----   ------------ ------------ --------------- -----------------------------------------
ora.banka.db                   1   ONLINE       ONLINE       hract23         Open,STABLE  
ora.banka.db                   2   ONLINE       OFFLINE      -               Instance Shutdown,ST ABLE
ora.banka.db                   3   ONLINE       ONLINE       hract22         Open,STABLE  
ora.bankb.db                   1   ONLINE       ONLINE       hract21         Open,STABLE  

Testing the current configuration 
Database bankB:
[oracle@hract21 ~]$ sqlplus system/sys@ract2-scan.grid12c.example.com:1521/bankb  @v
HOST_NAME               INSTANCE_NAME
------------------------------ ----------------
hract21.example.com           bankb_1
--> As database bankB runs only on one instance 

Verify load balancing for Database bankA:
[oracle@hract21 ~]$  sqlplus system/sys@ract2-scan.grid12c.example.com:1521/banka @v
HOST_NAME               INSTANCE_NAME
------------------------------ ----------------
hract23.example.com           bankA_1

[oracle@hract21 ~]$ sqlplus system/sys@ract2-scan.grid12c.example.com:1521/banka @v
HOST_NAME               INSTANCE_NAME
------------------------------ ----------------
hract22.example.com           bankA_3

--> As database bankA runs on 2 instances load balancing takes place .

 

Reference

Minimize Memory Footprint of running RAC instances to reduce OS Paging/Swapping

  • Tested version : RAC 12.1.0.2 / RAC 12.2.0.1
  • In test environments you very often my have memory resource problems and you need to reduce paging 
  • Paging/Swapping is the worse case scenario – so we need to reduce PGA/SGA size first

Using AMM

Reduce memory parameters on system 1
SQL> ALTER SYSTEM SET SGA_TARGET = 600m scope=spfile;
SQL> ALTER SYSTEM SET  pga_aggregate_target=350m scope=spfile;
SQL> startup force

After db reboot verify memory consumption  with top  
[root@ractw21 ~]# top
top - 12:06:43 up  1:50,  5 users,  load average: 2.19, 4.18, 3.50
Tasks: 471 total,   2 running, 469 sleeping,   0 stopped,   0 zombie
%Cpu(s):  3.5 us,  1.5 sy,  0.0 ni, 93.8 id,  1.0 wa,  0.1 hi,  0.3 si,  0.0 st
KiB Mem :  3781616 total,    82784 free,  2190716 used,  1508116 buff/cache
KiB Swap:  8257532 total,  6943376 free,  1314156 used.   677496 avail Mem
--> After reboot the system1 uses about 2.2 GByte memory
    paging/swappinig is low :  1.0 wa           

Now change these parameters on the complete cluster
SQL> ALTER SYSTEM SET SGA_TARGET = 600m scope=spfile sid='*';
SQL> ALTER SYSTEM SET  pga_aggregate_target=350m scope=spfile  sid='*';

Restart database
[oracle@ractw21 ~]$ svrctl stop database -db ractw
[oracle@ractw21 ~]$ srvctl stop database -db ractw

[root@ractw22 ~]# top
top - 12:17:17 up  2:01,  2 users,  load average: 17.49, 14.26, 9.98
Tasks: 499 total,   1 running, 497 sleeping,   0 stopped,   1 zombie
%Cpu(s):  2.8 us,  2.1 sy,  0.0 ni, 94.5 id,  0.3 wa,  0.0 hi,  0.3 si,  0.0 st
KiB Mem :  3781616 total,    83492 free,  1877612 used,  1820512 buff/cache
KiB Swap:  8257532 total,  5851312 free,  2406220 used.   902384 avail Mem
--> After reboot the system2 uses about 1.87 GByte memory
    paging/swappinig is low :  0.3 wa

Using memory_target

  •  Note:  avoid using memory_target  on prodcution systems 
Monitor OS resources :
top - 10:06:57 up  1:54, 10 users,  load average: 2.89, 5.43, 5.05
Tasks: 490 total,   3 running, 487 sleeping,   0 stopped,   0 zombie
Cpu(s): 10.5%us,  7.9%sy,  0.0%ni, 17.7%id,  63.1%wa,  0.0%hi,  0.9%si,  0.0%st
Mem:   3785860k total,  2866768k used,   919092k free,    30704k buffers
Swap:  4063228k total,   965472k used,  3097756k free,  1315228k cached

--> if wa values are very high for many top samples you may reduce your SGA/PGA footprint 
    to reduce OS paging/swapping.  

Verify that at least 2 instances are up and running 
Note changing values on a single instance first will allow us to easily change back these values in 
case the instance doesn't startup after reboot !

[oracle@hract21 ~]$ srvctl  status database -d bankA
Instance bankA_1 is running on node hract21
Instance bankA_3 is running on node hract22

Test your new memory settings on your local instance  
SQL> ALTER SYSTEM SET MEMORY_TARGET = 400M  scope=spfile ;
SQL> ALTER SYSTEM SET SGA_TARGET = 0  scope=spfile;
SQL> ALTER SYSTEM SET PGA_AGGREGATE_TARGET = 0  scope=spfile;
SQL> startup force
ORACLE instance started.
Total System Global Area  419430400 bytes
Fixed Size            2925120 bytes
Variable Size          335547840 bytes
Database Buffers       75497472 bytes

After single instance reboot works change parameter globally and restart database 
SQL> ALTER SYSTEM SET MEMORY_TARGET = 400M  scope=spfile sid='*' ;
SQL> ALTER SYSTEM SET SGA_TARGET = 0  scope=spfile sid='*';
SQL> ALTER SYSTEM SET PGA_AGGREGATE_TARGET = 0  scope=spfile sid='*';

Restart database :
$ srvctl stop database -d bankA
$ srvctl start database -d bankA
$ srvctl  status database -d bankA
Instance bankA_1 is running on node hract22
Instance bankA_3 is running on node hract21

Reference

 

JAVA: Automatic Resource Management (ARM)

Overview

  • The try-with-resources statement is a try statement that declares one or more resources.
  • A resource is an object that must be closed after the program is finished with it.
  • The try-with-resources statement ensures that each resource is closed at the end of the statement.
  • Any object that implements java.lang.AutoCloseable, which includes all objects which implement java.io.Closeable,  can be used as a resource.
  • try-with-resources, can have catch and finally. They work as usual and no change in it.
  • ARM returns the  Suppressed Exception details  thrown by close() statement without any add. coding

Sample Code

 
DirtyResource.java :
public class DirtyResource implements AutoCloseable
{
 /*
 *    Need to call this method if you want to access this resource
 *    @throws RuntimeException no matter how you call this method
 */
    public void accessResource()
    {
        throw new RuntimeException("I wanted to access this resource. Bad luck. Its dirty resource !!!");
    }
 
 /*
 *    The overridden closure method from AutoCloseable interface
 *    @throws Exception which is thrown during closure of this dirty resource
 */
    @Override
    public void close() throws Exception
    {
        throw new NullPointerException("Remember me. I am your worst nightmare !! I am Null pointer exception !!");
    }
}

SuppressedExceptionDemoWithTryWithResource.java
public class SuppressedExceptionDemoWithTryWithResource
{
 /*
 *     Demonstrating suppressed exceptions using try-with-resources
 */
   public static void main(String[] arguments) throws Exception
   {
      try (DirtyResource resource= new DirtyResource())
      {
          resource.accessResource();
      } catch ( Exception e1 ) 
    {
       throw e1;
    }
   }
}

Exception printout

[oracle@wls1 ARM]$ java SuppressedExceptionDemoWithTryWithResource
Exception in thread "main" java.lang.RuntimeException: I wanted to access this resource. Bad luck. Its dirty resource !!!
    at DirtyResource.accessResource(DirtyResource.java:9)
    at SuppressedExceptionDemoWithTryWithResource.main(SuppressedExceptionDemoWithTryWithResource.java:10)
    Suppressed: java.lang.NullPointerException: Remember me. I am your worst nightmare !! I am Null pointer exception !!
        at DirtyResource.close(DirtyResource.java:19)
        at SuppressedExceptionDemoWithTryWithResource.main(SuppressedExceptionDemoWithTryWithResource.java:11
  • Here we can easily indentify tha we are failing in ARM close() in DirtyResource.close

Reference

Setup DNS, NTP and DHCP for a mixed RAC/Internet usage

Note : 

You need to install your RAC Nameserver on a separate Virtualbox image/system as a NON-functional Nameserver can lead to a RAC hang scenario !!

Install BIND / DHCP RPMs and learn the needed configuration commands

Install and verify BIND installation [ bind_libs and bind_utils should be arlready installed ] 
[root@hract21 Desktop]#  yum install bind bind-utils bind-libs
[root@hract21 Desktop]# rpm -qa |grep '^bind'
bind-utils-9.8.2-0.30.rc1.el6_6.1.x86_64
bind-libs-9.8.2-0.30.rc1.el6_6.1.x86_64
bind-9.8.2-0.30.rc1.el6_6.1.x86_64

Install and verify DHCP setup 
Download and install dcping utility;
Download location:  http://pkgs.repoforge.org/dhcping  following package :
    dhcping-1.2-2.2.el6.rf.x86_64.rpm  11-Nov-2010 07:31   16K  RHEL6 and CentOS-6 x86 64bit
[root@ns1 ~]# rpm -i Downloads/dhcping-1.2-2.2.el6.rf.x86_64.rpm
 
[root@hract21 Desktop]# yum install dhcp.x86_64 
Total download size: 1.2 M
Is this ok [y/N]: y
Downloading Packages:
(1/3): dhclient-4.1.1-43.P1.0.1.el6_6.1.x86_64.rpm                                                           | 318 kB     00:00     
(2/3): dhcp-4.1.1-43.P1.0.1.el6_6.1.x86_64.rpm                                                               | 819 kB     00:00     
(3/3): dhcp-common-4.1.1-43.P1.0.1.el6_6.1.x86_64.rpm                                                        | 142 kB     00:00  

[root@hract21 Desktop]#  rpm -qa | grep -i DHCP
dhcp-common-4.1.1-43.P1.0.1.el6_6.1.x86_64
dhcp-4.1.1-43.P1.0.1.el6_6.1.x86_64

Setup Files needed: 
: /etc/named.conf
: /var/named/example.com.db
: /var/named/192.168.2.db
: /var/named/192.168.5.db
: /etc/dhcp/dhcpd.conf
: /etc/sysconfig/dhcpd  
: /etc/dhcp/dhcpd.conf
--> For details how to configure DNS/DHCP please read the details the chapters below. 

Setup,test and configure BIND service 
# service named restart 
# nslookup google.de
Server:        192.168.5.50
Address:    192.168.5.50#53

Non-authoritative answer:
Name:    google.de
Address: 173.194.112.152
Name:    google.de
Address: 173.194.112.159
Name:    google.de
Address: 173.194.112.143
Name:    google.de
Address: 173.194.112.151
#  chkconfig named on
#  chkconfig named --list
named              0:off    1:off    2:on    3:on    4:on    5:on    6:off

Setup,test and configure DHCP service 
# service dhcpd start
Starting dhcpd:                                            [  OK  ]
# chkconfig  dhcpd on
# chkconfig  dhcpd --list
dhcpd              0:off    1:off    2:on    3:on    4:on    5:on    6:off
Verify DHCP setup with  dhcping
[root@hract21 Desktop]#  dhcping -s 192.168.5.50 -c 192.168.5.197 
Got answer from: 192.168.5.50

DNS Server Setup

Our DNS server should have configured the Virtualbox Network Devices 
eth0  -> Bridged Network  : inet addr:192.168.1.XXX  Bcast:192.168.1.255  [ Internet Access ]
eth1  -> Internal Network : inet addr:192.168.5.50   Bcast:192.168.5.255  [ Public RAC Interface ]

eth0      Link encap:Ethernet  HWaddr 08:00:27:E6:71:54  
          inet addr:192.168.1.X  Bcast:192.168.1.255  Mask:255.255.255.0

eth1      Link encap:Ethernet  HWaddr 08:00:27:8D:8A:93  
          inet addr:192.168.5.50  Bcast:192.168.5.255  Mask:255.255.255.0   

Setup files used by  DNS : 
  /etc/named.conf  
  /var/named/example.com.db 
  /var/named/192.168.2.db
  /var/named/192.168.5.db


/etc/named.conf :
options {
    listen-on port 53 {  192.168.5.50; 127.0.0.1; };
    directory     "/var/named";
    dump-file     "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
    allow-query     {  any; };
    allow-recursion     {  any; };
    recursion yes;
    dnssec-enable no;
    dnssec-validation no;
};

logging {
        channel default_debug {
                file "data/named.run";
                severity dynamic;
        };
};

zone "." IN {
    type hint;
    file "named.ca";
};
zone    "5.168.192.in-addr.arpa" IN { // Reverse zone
    type master;
    file "192.168.5.db";
        allow-transfer { any; };
    allow-update { none; };
};
zone    "2.168.192.in-addr.arpa" IN { // Reverse zone
    type master;
    file "192.168.2.db";
        allow-transfer { any; };
    allow-update { none; };
};
zone  "example.com" IN {
      type master;
       notify no;
       file "example.com.db";
};

/var/named/example.com.db: 
$TTL 1H         ; Time to live
$ORIGIN example.com.
@       IN      SOA     ns1.example.com.  hostmaster.example.com.  (
                        2009011202      ; serial (todays date + todays serial #)
                        3H              ; refresh 3 hours
                        1H              ; retry 1 hour
                        1W              ; expire 1 week
                        1D )            ; minimum 24 hour
;
             IN     NS        ns1  ; name server for example.com
ns1          IN     A        192.168.5.50
grac41       IN     A        192.168.5.101  
grac42       IN     A        192.168.5.102  
grac43       IN     A        192.168.5.103  
grac41int    IN     A        192.168.2.101  
grac42int    IN     A        192.168.2.102  
grac43int    IN     A        192.168.2.103 
;
$ORIGIN grid4.example.com.
@       IN          NS        gns4.grid4.example.com. ; NS  grid4.example.com
        IN          NS        ns1.example.com.      ; NS example.com
gns4    IN          A         192.168.5.54 ; glue record



/var/named/192.168.5.db :
$TTL 1H
@       IN      SOA     ns1.example.com.  root.domin.com.  (
                        2009011201      ; serial (todays date + todays serial #)
                        3H              ; refresh 3 hours
                        1H              ; retry 1 hour
                        1W              ; expire 1 week
                        1D )            ; minimum 24 hour
      IN    NS    ns1
ns1     IN       A      192.168.5.50
;
50            PTR       ns1.example.com.
54            PTR       gns4.grid4.example.com. ; reverse mapping for GNS
101           PTR       grac41.example.com. 
102           PTR       grac42.example.com. 
103           PTR       grac43.example.com. 
201           PTR       wls1.example.com. 

/var/named/192.168.2.db :
$TTL 1H
@       IN      SOA     ns1.example.com. hostmaster.example.com.  (
                        2009011201      ; serial (todays date + todays serial #)
                        3H              ; refresh 3 hours
                        1H              ; retry 1 hour
                        1W              ; expire 1 week
                        1D )            ; minimum 24 hour
        IN      NS      ns1
ns1     IN       A         192.168.5.50
; 
101          PTR       grac41int.example.com. 
102          PTR       grac42int.example.com. 
103          PTR       grac43int.example.com.


Verify zone files and restart named deamon
[root@ns1 named]#  named-checkconf /etc/named.conf
[root@ns1 named]#  named-checkzone example.com example.com.db
zone example.com/IN: grid.example.com/NS 'gns.grid.example.com' (out of zone) has no addresses records (A or AAAA)
zone example.com/IN: grid12c.example.com/NS 'gns12c.grid12c.example.com' (out of zone) has no addresses records (A or AAAA)
zone example.com/IN: grid2.example.com/NS 'gns2.grid2.example.com' (out of zone) has no addresses records (A or AAAA)
zone example.com/IN: grid3.example.com/NS 'gns3.grid3.example.com' (out of zone) has no addresses records (A or AAAA)
zone example.com/IN: grid4.example.com/NS 'gns4.grid4.example.com' (out of zone) has no addresses records (A or AAAA)
zone example.com/IN: loaded serial 2009011202
OK
[root@ns1 named]# named-checkzone example.com  192.168.5.db
zone example.com/IN: loaded serial 2009011201
OK
[root@ns1 named]# named-checkzone example.com  192.168.2.db
zone example.com/IN: loaded serial 2009011201
OK

Verify DNS Setup

[root@ns1 ~]# nslookup google.de
Server:        192.168.5.50
Address:    192.168.5.50#53

Non-authoritative answer:
Name:    google.de
Address: 173.194.67.94

[root@ns1 ~]# nslookup grac41 
Server:        192.168.5.50
Address:    192.168.5.50#53

Name:    grac41.example.com
Address: 192.168.5.101

[root@ns1 ~]# ping -c 2  google.de
PING google.de (173.194.67.94) 56(84) bytes of data.
64 bytes from wi-in-f94.1e100.net (173.194.67.94): icmp_seq=1 ttl=38 time=66.3 ms
64 bytes from wi-in-f94.1e100.net (173.194.67.94): icmp_seq=2 ttl=38 time=62.3 ms
--- google.de ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1064ms
rtt min/avg/max/mdev = 62.373/64.344/66.316/1.987 ms

[root@ns1 ~]# ping -c 2  grac41 
PING grac41.example.com (192.168.5.101) 56(84) bytes of data.
64 bytes from grac41.example.com (192.168.5.101): icmp_seq=1 ttl=64 time=0.200 ms
 64 bytes from grac41.example.com (192.168.5.101): icmp_seq=2 ttl=64 time=0.293 ms
--- grac41.example.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.200/0.246/0.293/0.049 ms

[root@ns1 ~]#  cat /etc/resolv.conf
# Generated by NetworkManager
search example.com grid4.example.com
nameserver 192.168.5.50
[root@ns1 ~]#  netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG        0 0          0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 eth0
192.168.3.0     0.0.0.0         255.255.255.0   U         0 0          0 eth2
192.168.5.0     0.0.0.0         255.255.255.0   U         0 0          0 eth1

If the GNS server is running the following commands should work too !
[root@ns1 ~]# nslookup grac4-scan
Server:        192.168.5.50
Address:    192.168.5.50#53

Non-authoritative answer:
Name:    grac4-scan.grid4.example.com
Address: 192.168.5.167
Name:    grac4-scan.grid4.example.com
Address: 192.168.5.156
Name:    grac4-scan.grid4.example.com
Address: 192.168.5.153

[root@ns1 ~]# ping -c 2  grac4-scan
PING grac4-scan.grid4.example.com (192.168.5.167) 56(84) bytes of data.
64 bytes from 192.168.5.167: icmp_seq=1 ttl=64 time=0.176 ms
64 bytes from 192.168.5.167: icmp_seq=2 ttl=64 time=0.203 ms
--- grac4-scan.grid4.example.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.176/0.189/0.203/0.019 ms
[root@ns1 ~]# dig @192.168.5.50 grac4-scan.grid4.example.com
; <<>> DiG 9.9.3-RedHat-9.9.3-P1.el6 <<>> @192.168.5.50 grac4-scan.grid4.example.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18529
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 2, ADDITIONAL: 2

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;grac4-scan.grid4.example.com.    IN    A

;; ANSWER SECTION:
grac4-scan.grid4.example.com. 94 IN    A    192.168.5.167
grac4-scan.grid4.example.com. 94 IN    A    192.168.5.156
grac4-scan.grid4.example.com. 94 IN    A    192.168.5.153

;; AUTHORITY SECTION:
grid4.example.com.    3600    IN    NS    gns4.grid4.example.com.
grid4.example.com.    3600    IN    NS    ns1.example.com.

;; ADDITIONAL SECTION:
ns1.example.com.    3600    IN    A    192.168.5.50

;; Query time: 1 msec
;; SERVER: 192.168.5.50#53(192.168.5.50)
;; WHEN: Sun Jan 11 17:17:51 CET 2015
;; MSG SIZE  rcvd: 158

[root@ns1 ~]#  dig @192.168.5.54 grac4-scan.grid4.example.com
; <<>> DiG 9.9.3-RedHat-9.9.3-P1.el6 <<>> @192.168.5.54 grac4-scan.grid4.example.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5071
;; flags: qr aa; QUERY: 1, ANSWER: 3, AUTHORITY: 1, ADDITIONAL: 1

;; QUESTION SECTION:
;grac4-scan.grid4.example.com.    IN    A

;; ANSWER SECTION:
grac4-scan.grid4.example.com. 120 IN    A    192.168.5.153
grac4-scan.grid4.example.com. 120 IN    A    192.168.5.156
grac4-scan.grid4.example.com. 120 IN    A    192.168.5.167

;; AUTHORITY SECTION:
grid4.example.com.    10800    IN    SOA    grac4-gns-vip.grid4.example.com. grac4-gns-vip.grid4.example.com. 264601876 10800 10800 30 120

;; ADDITIONAL SECTION:
grac4-gns-vip.grid4.example.com. 10800 IN A    192.168.5.54

;; Query time: 2 msec
;; SERVER: 192.168.5.54#53(192.168.5.54)
;; WHEN: Sun Jan 11 17:17:59 CET 2015
;; MSG SIZE  rcvd: 160

If GNS is not configured or running you will get error:  can't find grac4-scan: NXDOMAIN
[grid@grac41 ~]$  srvctl stop gns
[root@ns1 ~]# ping 192.168.5.54
PING 192.168.5.54 (192.168.5.54) 56(84) bytes of data.
From 192.168.5.50 icmp_seq=2 Destination Host Unreachable
From 192.168.5.50 icmp_seq=3 Destination Host Unreachable
From 192.168.5.50 icmp_seq=4 Destination Host Unreachable
^C
--- 192.168.5.54 ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3944ms
pipe 3
[root@ns1 ~]#  nslookup grac4-scan
Server:        192.168.5.50
Address:    192.168.5.50#53

** server can't find grac4-scan: NXDOMAIN

Verify subdomain delegation with cluvfy

Starting with Oracle Database 11g release 2 (11.2.0.2), use the cluvfy comp dns component verification 
command to verify that the Grid Naming Service (GNS) subdomain delegation has been properly set up in 
the Domain Name Service (DNS) server.

Run cluvfy comp dns -server on one node of the cluster. On each node of the cluster run 
cluvfy comp dns -client to verify the DNS server setup for the cluster.

Oh grac41: 
[root@grac41 ~]# cluvfy comp dns -server -domain  grid4.example.com -vipaddress 192.168.5.54/255.255.255.0/eth1 -verbose
Verifying DNS Check 
Starting the test DNS server on IP "192.168.5.54/255.255.255.0/eth1" listening on port 53
Started the IP address "192.168.5.54/255.255.255.0/eth1" on node "grac41"

On grac42: 
[root@grac42 ~]#  cluvfy comp dns -client -domain  grid4.example.com -vip 192.168.5.54
Verifying DNS Check 
Checking if the IP address "192.168.5.54" is reachable
The IP address "192.168.5.54" is reachable from local node
Successfully connected to test DNS server
Checking if the test DNS server started on address "192.168.5.54", listening on port 53 can be queried
Check output of command "cluvfy comp dns -server" to see if it received IP address lookup for name "grac42.grid4.example.com"
Successfully connected to the test DNS server started on address "192.168.5.54", listening on port 53
Checking DNS delegation for the GNS subdomain "grid4.example.com"...
Check output of command "cluvfy comp dns -server" to see if it received IP address lookup for name "grac42.grid4.example.com"
Successfully verified DNS delegation of the GNS subdomain "grid4.example.com"

Verification of DNS Check was successful. 

--> Server should report 
Received IP address lookup query for name "grac42.grid4.example.com"
Received IP address lookup query for name "grac42.grid4.example.com"

On grac43:
[root@grac43 ~]# cluvfy comp dns -client -domain  grid4.example.com -vip 192.168.5.54
..
Verification of DNS Check was successful. 
--> Server should report 
Received IP address lookup query for name "grac43.grid4.example.com"
Received IP address lookup query for name "grac43.grid4.example.com"

On grac41 
[root@grac41 Desktop]#  cluvfy comp dns -client -domain  grid4.example.com -vip 192.168.5.54 
..
Verification of DNS Check was successful. 
--> Server should report 
Received IP address lookup query for name "grac41.grid4.example.com"
Received IP address lookup query for name "grac41.grid4.example.com"

 

Setup DHCP server

DHCP configuration file 
/etc/dhcp/dhcpd.conf :
ddns-update-style interim;
ignore client-updates;

subnet 192.168.5.0 netmask 255.255.255.0 {
        option routers                  192.168.5.1;                    # Default gateway to be used by DHCP clients
        option subnet-mask              255.255.255.0;                  # Default subnet mask to be used by DHCP clients.
        option ip-forwarding            off;                            # Do not forward DHCP requests.
        option broadcast-address        192.168.5.255;                  # Default broadcast address to be used by DHCP client.
        option domain-name-servers      192.168.5.50;                   # IP address of the DNS server. 
        option time-offset              -19000;                           # Central Standard Time
        option ntp-servers              192.168.5.50;                   # Default NTP server to be used by DHCP clients
        range                           192.168.5.150 192.168.5.254;    # Range of IP addresses that can be issued to DHCP client
        default-lease-time              21600;                            # Amount of time in seconds that a client may keep the IP address
        max-lease-time                  43200;
} 

/etc/sysconfig/dhcpd
# Command line options here
DHCPDARGS="eth1"

Restart the DHCP server :
[root@ns1 network-scripts]# service dhcpd restart

 

Verify  DHCP setup with cluvfy

[root@grac41 ~]#  $GRID_HOME/bin/cluvfy comp dhcp -clustername grac4 
Verifying DHCP Check 
Checking if any DHCP server exists on the network...
PRVG-5723 : Network CRS resource is configured to use DHCP provided IP addresses

Verification of DHCP Check was unsuccessful on all the specified nodes. 

From Oracle docu :
- You must run this command as root.
- Do not run this check while the default network Oracle Clusterware resource, configured to use a 
   DHCP-provided IP address, is online (because the VIPs get released and, since the cluster is online, 
   DHCP has provided IP, so there is no need to double the load on the DHCP server).
- Before running this command, ensure that the network resource is offline. Use the srvctl stop nodeapps command 
   to bring the network resource offline, if necessary.

As we are on a test cluster go ahead and stop the Nodeapps 
[root@grac41 Desktop]#  srvctl stop nodeapps -f

[root@grac41 ~]# $GRID_HOME/bin/cluvfy comp dhcp -clustername grac4 -verbose
Verifying DHCP Check 
Checking if any DHCP server exists on the network...
Checking if network CRS resource is configured and online
Network CRS resource is offline or not configured. Proceeding with DHCP checks.
CRS-10009: DHCP server returned server: 192.168.5.50, loan address : 192.168.5.165/255.255.255.0, lease time: 21600

At least one DHCP server exists on the network and is listening on port 67
Checking if DHCP server has sufficient free IP addresses for all VIPs...
Sending DHCP "DISCOVER" packets for client ID "grac4-scan1-vip"
CRS-10009: DHCP server returned server: 192.168.5.50, loan address : 192.168.5.165/255.255.255.0, lease time: 21600
...
CRS-10012: released DHCP server lease for client ID grac4-scan3-vip on port 67
CRS-10012: released DHCP server lease for client ID grac4-grac41-vip on port 67

DHCP server was able to provide sufficient number of IP addresses
The DHCP server response time is within acceptable limits
Verification of DHCP Check was successful. 

Note you can track  the lease operation with following OS command 
[root@ns1 ~]# tail -f  /var/lib/dhcpd/dhcpd.leases
}
lease 192.168.5.164 {
  starts 0 2015/01/11 17:29:10;
  ends 0 2015/01/11 17:29:10;
  tstp 0 2015/01/11 17:29:10;
  cltt 0 2015/01/11 17:29:10;
  binding state free;
  hardware ethernet 00:00:00:00:00:00;
  uid "\000grac4-grac41-vip";
}

 

Configure NTP


Configuration script :
/etc/ntp.conf
restrict default nomodify notrap noquery
restrict 127.0.0.1 
# -- CLIENT NETWORK -------
restrict 192.168.5.0 mask 255.255.255.0 nomodify notrap
# --- OUR TIMESERVERS -----  can't reach NTP servers - build my own server 
server 0.pool.ntp.org iburst
server 1.pool.ntp.org iburst
server 127.127.1.0
# --- NTP MULTICASTCLIENT ---
# --- GENERAL CONFIGURATION ---
# Undisciplined Local Clock.
fudge   127.127.1.0 stratum 9
# Drift file.
driftfile /var/lib/ntp/drift
broadcastdelay  0.008
# Keys file.
keys /etc/ntp/keys

Restart NTP daemon
[root@ns1 network-scripts]# service ntpd restart
Shutting down ntpd:                                        [  OK  ]
Starting ntpd:                                             [  OK  ]

Verify setup
[root@ns1 network-scripts]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 foxtrot.zq1.de  122.227.206.195  3 u    2   64    1   68.504  4608.38   1.115
 der.beste.tiger 159.173.11.127   3 u    1   64    1   38.195  4603.43  11.063
 LOCAL(0)        .LOCL.           9 l    2   64    1    0.000    0.000   0.000

 

Verify NTP setup with cluvfy

Verify NTP setup with cluvfy 
[grid@grac41 ~]$   cluvfy comp clocksync
Verifying Clock Synchronization across the cluster nodes 
Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed
Checking if CTSS Resource is running on all nodes...
CTSS resource check passed
Querying CTSS for time offset on all nodes...
Query of CTSS for time offset passed
Check CTSS state started...
CTSS is in Observer state. Switching over to clock synchronization checks using NTP
Starting Clock synchronization checks using Network Time Protocol(NTP)...
NTP Configuration file check started...
NTP Configuration file check passed
Checking daemon liveness...
Liveness check passed for "ntpd"
Check for NTP daemon or service alive passed on all nodes
NTP daemon slewing option check passed
NTP daemon's boot time configuration check for slewing option passed
NTP common Time Server Check started...
Check of common NTP Time Server passed
Clock time offset check from NTP Time Server started...
Clock time offset check passed
Clock synchronization check using Network Time Protocol(NTP) passed
Oracle Cluster Time Synchronization Services check passed
Verification of Clock Synchronization across the cluster nodes was successful.

Reference

Troubleshooting Clusterware startup problems with DTRACE

First Steps which may avoid setting up DTRACE at all

Cleanup your special sockets file in /var/tmp/.oracle

Either reboot your OS or Cleanup sockets file and reboot CRS stack :
[root@hract21 Desktop]# crsctl stop crs -f
[root@hract21 Desktop]# rm -rf /var/tmp/.oracle/*
[root@hract21 Desktop]# crsctl start crs
 CRS-4123: Oracle High Availability Services has been started.

Note: A complete OS reboot may be needed to fix hanging processes waiting on DISKWAIT 
      If possible always try to do an OS reboot. 
      An OS reboot will always cleanup  /var/tmp/.oracle/*

Quickly verify your OS with a simple sh script : chk_os.sh

#!/bin/bash 
NS=ns1.example.com
HOSTNAME1=hract21.example.com
HOSTNAME2=hract22.example.com
PRIV_IP1=192.168.2.121
PRIV_IP2=192.168.2.122
PUBLIC_IF=eth1
PRIVATE_IF=eth2

echo ""
echo "Disk Space : "
df

echo ""
echo "Major Clusterware Executable Protections : "
ls -l $GRID_HOME/bin/ohasd*
ls -l $GRID_HOME/bin/orarootagent*
ls -l $GRID_HOME/bin/oraagent*
ls -l $GRID_HOME/bin/mdnsd*
ls -l $GRID_HOME/bin/evmd*
ls -l $GRID_HOME/bin/gpnpd*
ls -l $GRID_HOME/bin/evmlogger*
ls -l $GRID_HOME/bin/osysmond.*
ls -l $GRID_HOME/bin/gipcd*
ls -l $GRID_HOME/bin/cssdmonitor*
ls -l $GRID_HOME/bin/cssdagent*
ls -l $GRID_HOME/bin/ocssd*
ls -l $GRID_HOME/bin/octssd*
ls -l $GRID_HOME/bin/crsd
ls -l $GRID_HOME/bin/crsd.bin
ls -l $GRID_HOME/bin/tnslsnr


echo ""
echo "Ping Nameserver: "
ping -c 2  $NS 

echo ""
echo "Test your PUBLIC interface and your nameserver setup"
nslookup $HOSTNAME
ping -I $PUBLIC_IF -c 2   $HOSTNAME1
ping -I $PUBLIC_IF -c 2   $HOSTNAME2
 
ping -I $PRIVATE_IF -c 2   $PRIV_IP1 
ping -I $PRIVATE_IF -c 2   $PRIV_IP2

echo ""
echo "Verify protections for HOSTNAME.pid files should be : 644"
find $GRID_HOME -name hract21.pid  -exec ls -l {} \; 

echo ""
echo "Service iptables and avahi-daemon should not run - avahi-daemon uses CW port 5353 "
service iptables status
ps -elf |grep avahi | grep -v avahi

echo ""
echo "Ports :53 :5353 :42422 :8888 should not be used by NON-Clusterware processes "
echo "  - OC4J reports : tcp   0 0 ::ffff:127.0.0.1:8888  :::*  LISTEN   501 67433979  2580/java"           
netstat -taupen | egrep ":53 |:5353 |:42424 |:8888 "

echo ""
echo "Compare profile.xml the IP Address of PUBLIC and PRIVATE Interfaces "
echo " - Devices should report UP BROADCAST RUNNING MULTICAST "
echo " - Double check NETWORK addresses matches profile.xml settings   "
echo ""
$GRID_HOME/bin/gpnptool get 2>/dev/null  |  xmllint --format - | egrep 'CSS-Profile|ASM-Profile|Network id'
echo ""
ifconfig $PUBLIC_IF | egrep 'eth|inet addr|MTU'
echo ""
ifconfig $PRIVATE_IF | egrep 'eth|inet addr|MTU'

echo "Checking ASM disk status for disk named /dev/asm ...  - you may need to changes this "
ls -l  /dev/asm*

echo ""
echo "Verify ASM disk "
su - grid -c "ssh $HOSTNAME2 ocrcheck"
su - grid -c "ssh $HOSTNAME2  asmcmd lsdsk -k"
echo ""
su - grid -c "kfed read /dev/asmdisk1_10G | grep name"
echo ""
su - grid -c "kfed read /dev/asmdisk2_10G | grep name"
echo ""
su - grid -c "kfed read /dev/asmdisk3_10G | grep name"
echo ""
su - grid -c "kfed read /dev/asmdisk4_10G | grep name"
echo ""


Output:
..
Ports :53 :5353 :42422 :8888 should not be used by NON-Clusterware processes 
  - OC4J reports : tcp   0 0 ::ffff:127.0.0.1:8888  :::*  LISTEN   501 67433979  2580/java
udp        0      0 0.0.0.0:5353                0.0.0.0:*    501        54383580   28618/mdnsd.bin     
udp        0      0 0.0.0.0:5353                0.0.0.0:*    501        54383565   28618/mdnsd.bin     
udp        0      0 0.0.0.0:5353                0.0.0.0:*    501        54383564   28618/mdnsd.bin     
udp        0      0 0.0.0.0:5353                0.0.0.0:*    501        54383563   28618/mdnsd.bin     
udp        0      0 192.168.2.255:42424         0.0.0.0:*    0          54429417   28502/ohasd.bin     
udp        0      0 230.0.1.0:42424             0.0.0.0:*    0          54429416   28502/ohasd.bin     
udp        0      0 224.0.0.251:42424           0.0.0.0:*    0          54429415   28502/ohasd.bin     
udp        0      0 192.168.2.255:42424         0.0.0.0:*    501        54412444   28827/ocssd.bin     
udp        0      0 230.0.1.0:42424             0.0.0.0:*    501        54412443   28827/ocssd.bin     
udp        0      0 224.0.0.251:42424           0.0.0.0:*    501        54412442   28827/ocssd.bin     
udp        0      0 192.168.2.255:42424         0.0.0.0:*    501        54406273   28742/gipcd.bin     
udp        0      0 230.0.1.0:42424             0.0.0.0:*    501        54406272   28742/gipcd.bin     
udp        0      0 224.0.0.251:42424           0.0.0.0:*    501        54406271   28742/gipcd.bin     
udp        0      0 192.168.5.58:53             0.0.0.0:*    0          67400781   2472/gnsd.bin 
tcp        0      0 ::ffff:127.0.0.1:8888        LISTEN      501        67433979   2580/java  
--> mdnsd.bin is using port 5353
    ohasd.bin, ohasd.bin, gipcd.bin are using port 42424
    oc4j is using port 8888           
    GNS is using port 53 

Compare profile.xml the IP Address of PUBLIC and PRIVATE Intefaces 
 - Devices should report UP BROADCAST RUNNING MULTICAST 
 - Double check NETWORK addresses matches profile.xml settings   
    <gpnp:HostNetwork id="gen" HostName="*">
      <gpnp:Network id="net1" IP="192.168.5.0" Adapter="eth1" Use="public"/>
      <gpnp:Network id="net2" IP="192.168.2.0" Adapter="eth2" Use="asm,cluster_interconnect"/>
  <orcl:CSS-Profile id="css" DiscoveryString="+asm" LeaseDuration="400"/>
  <orcl:ASM-Profile id="asm" DiscoveryString="/dev/asm*" SPFile="+DATA/ract2/ASMPARAMETERFILE/registry.253.870352347" Mode="remote"/>

eth1      Link encap:Ethernet  HWaddr 08:00:27:7D:8E:49  
          inet addr:192.168.5.121  Bcast:192.168.5.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

eth2      Link encap:Ethernet  HWaddr 08:00:27:4E:C9:BF  
          inet addr:192.168.2.121  Bcast:192.168.2.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

  --> IP="192.168.5.0" Adapter="eth1" should match --> eth1 : inet addr:192.168.5.121  Bcast:192.168.5.255  Mask:255.255.255.0 
      IP="192.168.2.0" Adapter="eth2" should match --> eth2 : inet addr:192.168.2.121  Bcast:192.168.2.255  Mask:255.255.255.0
Pages: Page 1, Page 2, Page 3, Page 4, Page 5, Page 6, Page 7, Page 8, Page 9, Page 10, Page 11, Page 12, Page 13, Page 14, Page 15, Page 16

Java Tutorial: Annotation and Reflection

Annotation and Reflection

  • Annotations have a number of uses, among them:
    • Information for the compiler — Annotations can be used by the compiler to detect errors or suppress warnings.
    •   Compile-time and deployment-time processing — Software tools can process annotation information to generate code, XML files, and so forth.
    •   Runtime processing — Some annotations are available to be examined at runtime.
  • Annotation type declarations are similar to normal interface declarations.
  • An at-sign (@) precedes the interface keyword – Sample : public @interface ClassInfo
  • The annotations are not method calls and will not, by themselves, do anything.  Rather any tool like JPA need to extract the annotations at runtime and need to do execute the designed action like: Generating an object-relational mapping.
  • Following JAVA language constructs can be annotated: Class, Constructor, Field, Method, and Package
  • The Java compiler conditionally stores annotation metadata in the class files if the annotation has a RetentionPolicy of CLASS or RUNTIME. Later, the JVM or other programs can look for the metadata to determine how to interact with the program elements or change their behavior [ via Reflection API ]
  • Annotation can  also  be used to provide some info about who change a component [ Java Class , Java Methode ]

Custom Annotations Details for Runtime processing with Reflection

ClassInfo - @Target(value = ElementType.TYPE) 
  - Provide some Info who has changed a certain specific JAVA source
  - Use reflection code below to display the Anotation Info
     if(annotation instanceof ClassInfo)
     {
         ClassInfo myAnnotation = (ClassInfo) annotation;
         System.out.println(" -> autor           : " + myAnnotation.author());
         System.out.println(" -> date            : " + myAnnotation.date());
         System.out.println(" -> comment         : " + myAnnotation.comments());
     }

CanRun - @Target(value = ElementType.METHOD) 
   - Indicate that we can/should run a certain JAVA methode via reflection
   - Run that specific methode by using reflection code:  
         method.invoke(runner);

CanChange - @Target(value = ElementType.FIELD)  
   - Indicate that we can/should modify a certain JAVA methode via reflection 
   - Change a int field  by using reflection code:  
        f.setInt(runner,k);
 
CanConstruct - @Target(value = ElementType.CONSTRUCTOR) 
   - Indicates that we can/should run a certain JAVA Constructor via reflection
   - Construct a new AnnotationRunner instance and printout the int field id1  by using reflection code: 
       ctor.setAccessible(true);
       AnnotationRunner  r = (AnnotationRunner)ctor.newInstance();
       Field f = r.getClass().getDeclaredField("id1");
       f.setAccessible(true); 

Note all of the above Anotations are used during Runtime and thus  
@Retention(value = RetentionPolicy.RUNTIME) is mandatory

Java source: ClassInfo.java

package utils;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;
@Target(value = ElementType.TYPE)
@Retention(value = RetentionPolicy.RUNTIME)
public @interface ClassInfo 
{
    String author() default "Helmut";
    String date();
    String comments();
}

Java source: CanChange.java

package utils;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;
@Target(value = ElementType.FIELD)
@Retention(value = RetentionPolicy.RUNTIME)
public @interface CanChange 
    {
    }

Java source: CanRun.java

package utils;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;
@Target(value = ElementType.METHOD)
@Retention(value = RetentionPolicy.RUNTIME)
public @interface CanRun {
}

Java source: CanConstruct.java

package utils;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;
@Target(value = ElementType.CONSTRUCTOR)
@Retention(value = RetentionPolicy.RUNTIME)
public @interface CanConstruct
    {
    }

Java source – The helper class: ReflectionUtils.java

package utils;
import java.lang.annotation.Annotation;
import java.lang.reflect.Constructor;
import java.lang.reflect.Field;
import java.lang.reflect.Method;
import java.lang.reflect.Modifier;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
/**
 * Die Klasse <code>ReflectionUtils</code> ist eine Utility-Klasse, die
 * verschiedene Hilfsmethoden zur vereinfachten Handhabung von Reflection
 * bereitstellt.
 * 
* @author Michael Inden
 * 
* Copyright 2011 by Michael Inden
 */
public final class ReflectionUtils
    {

    public static String modifierToString(final int modifier)
      {
        String modifiers = "";
        if (Modifier.isPublic(modifier))
          {
            modifiers += "public ";
          }
        if (Modifier.isProtected(modifier))
          {
            modifiers += "protected ";
          }
        if (Modifier.isPrivate(modifier))
          {
            modifiers += "private ";
          }
        if (Modifier.isStatic(modifier))
          {
            modifiers += "static ";
          }
        if (Modifier.isAbstract(modifier))
          {
            modifiers += "abstract ";
          }
        if (Modifier.isFinal(modifier))
          {
            modifiers += "final ";
          }
        if (Modifier.isVolatile(modifier))
          {
            modifiers += "volatile ";
          }
        if (Modifier.isSynchronized(modifier))
          {
            modifiers += "synchronized ";
          }
        return modifiers;
      }

    public static Field findField(final Class<?> clazz, final String fieldName)
      {
// Abbruch der Rekursion
        if (clazz == null)
          {
            return null;
          }
        try
          {
            return clazz.getDeclaredField(fieldName);
          } catch (final NoSuchFieldException ex)
          {
// rekursive Suche in Superklasse
            return findField(clazz.getSuperclass(), fieldName);
          }
      }

    public static Method findMethod(final Class<?> clazz, final String methodName, final Class<?>... parameterTypes)
      {
// Abbruch der Rekursion
        if (clazz == null)
          {
            return null;
          }
        try
          {
            return clazz.getDeclaredMethod(methodName, parameterTypes);
          } catch (final NoSuchMethodException ex)
          {
// rekursive Suche in Superklasse
            return findMethod(clazz.getSuperclass(), methodName, parameterTypes);
          }
      }

    
   public static Method[] getAllMethods(final Class<?> clazz)
      {
        final List<Method> methods = new ArrayList<Method>();
        methods.addAll(Arrays.asList(clazz.getDeclaredMethods()));
        /*
        if (clazz.getSuperclass() != null)
          {
              // rekursive Suche in Superklasse
            methods.addAll(Arrays.asList(getAllMethods(clazz.getSuperclass())));
          }
                */
        return methods.toArray(new Method[0]);
      }
       
    public static Field[] getAllFields(final Class<?> clazz)
      {
        final List<Field> fields = new ArrayList<Field>();
        //   Field[] fields = cls.getDeclaredFields();
        fields.addAll(Arrays.asList(clazz.getDeclaredFields()));
       
        return fields.toArray(new Field[0]);
      }
    
     public static Constructor[] getAllConstructors(final Class<?> clazz)
      {
        final List<Constructor> constructors = new ArrayList<Constructor>();
        //   Field[] fields = cls.getDeclaredFields();
        constructors.addAll(Arrays.asList(clazz.getConstructors()));
        return constructors.toArray(new Constructor[0]);
      }
    
    public static void printCtorInfos(final Constructor<?> ctor)
      {
        System.out.println(modifierToString(ctor.getModifiers()) + " " + ctor.getName()
                + buildParameterTypeString(ctor.getParameterTypes()));
        printAnnotations(ctor.getAnnotations());
      }

    public static void printMethodInfos(final Method method)
      {
        System.out.println(modifierToString(method.getModifiers()) + method.getReturnType() + " " + method.getName()
                + buildParameterTypeString(method.getParameterTypes()));
        printAnnotations(method.getAnnotations());
      }

    public static void printFieldInfos(final Field field)
      {
        System.out.println(ReflectionUtils.modifierToString(field.getModifiers()) + field.getType() + " "
                + field.getName());
        printAnnotations(field.getAnnotations());
      }

    public static String buildParameterTypeString(final Class<?>[] parameterTypes)
      {
        if (parameterTypes.length > 0)
          {
            return "(" + Arrays.toString(parameterTypes) + ")";
          }
        return "()";
      }

    private static void printAnnotations(final Annotation[] annotations)
      {
        if (annotations.length > 0)
          {
            System.out.println("Annotations: " + Arrays.toString(annotations));
          }
      }
    
    public static void printClassInfo(final Class<?> clazz )
      {
        // System.out.println("Canonical Class Name: " + clazz.getCanonicalName() );
        System.out.println("Class Name          : "  + clazz.getName() ); 
        System.out.println("Superclass Name     : "  + clazz.getSuperclass() ); 
        System.out.println("Interfaces          : "  + Arrays.toString(clazz.getInterfaces())); 
      //  Class c  = runner.getClass();
        Annotation[] annotations = clazz.getAnnotations();
         for(Annotation annotation : annotations)
           {
           if(annotation instanceof ClassInfo)
              {
              ClassInfo myAnnotation = (ClassInfo) annotation;
              System.out.println(" -> autor           : " + myAnnotation.author());
              System.out.println(" -> date            : " + myAnnotation.date());
              System.out.println(" -> comment         : " + myAnnotation.comments());
              }
           }
      }
            
    private ReflectionUtils()
      {
      }
    }

Java source: AnnotationRunner.java

package AnnotationTest;
import utils.CanChange;
import utils.CanRun;
import utils.CanConstruct;
import utils.ClassInfo;

@ClassInfo( author="Helmut Hutzer", 
            date="8-Feb-2014",
            comments="Intial Class Creation for Testing annotations" )
public class AnnotationRunner {
    @CanChange
    public int id1 = 1;
    public int id2= 2;
    
    @CanConstruct
    public AnnotationRunner()
        {
        }
    
    public AnnotationRunner(int v_id1, int v_id2)
    {
        id1 = v_id1;
        id2 = v_id2;
    }
    
    public void method1() 
    {
        System.out.println("Hello from method1 : " + id1);
    }

    @CanRun
    public void method2() 
    {
        System.out.println("Hello from method2 !");
    }
    
    public void set_id1(int v_id) 
    {
        id1 = v_id;
    }
    
    public void set_id2(int v_id) 
    {
        id2 = v_id;
    }
}

Java source: MyTest.java

package AnnotationTest;

import java.lang.annotation.Annotation;
import utils.CanChange;
import utils.CanRun;
import utils.CanConstruct;
import java.lang.reflect.Method;
import java.lang.reflect.Field;
import java.lang.reflect.Constructor;
import java.lang.reflect.InvocationTargetException;
public class MyTest 
{
    public static void main(String[] args) 
    {
        AnnotationRunner runner = new AnnotationRunner();     
        
        System.out.println("\n-->Inspect Class:");
        utils.ReflectionUtils.printClassInfo( runner.getClass());
        
          System.out.println("\n-->Inspect Class:");
        utils.ReflectionUtils.printClassInfo(CanChange.class);
        
        System.out.println("\n--> Exploring Constructors  ");
        Constructor[] constructors =  utils.ReflectionUtils.getAllConstructors(runner.getClass());
        Constructor ctor =null;
        for (  Constructor  constructor : constructors) 
        {
            utils.ReflectionUtils.printCtorInfos(constructor);
                // Find the constructor without any paramter
            if (constructor.getGenericParameterTypes().length == 0)
                ctor = constructor;
        } 
        
          // See ; http://docs.oracle.com/javase/tutorial/reflect/member/ctorInstance.html 
        
        if ( ctor != null )
        {
            System.out.println("--> Found a constructor without parameters ");
            Annotation annos = ctor.getAnnotation(CanConstruct.class);
            if (annos != null) 
            {
                try
                {
                    System.out.println("--> Found Annotation CanConstruct   ");
                    utils.ReflectionUtils.printCtorInfos(ctor);
                    ctor.setAccessible(true);
                    AnnotationRunner  r = (AnnotationRunner)ctor.newInstance();
                    Field f = r.getClass().getDeclaredField("id1");
                    f.setAccessible(true);
                    System.out.println("--> Created new instance of class  AnnotationRunner");
                    System.out.println("--> print Field id1 via reflecion : "+   f.get(r));
                } catch ( InstantiationException | IllegalAccessException 
                            | NoSuchFieldException | InvocationTargetException ex ) 
                     { ex.printStackTrace(); }
            }
        }
        
        System.out.println("\n --> Exploring method annotations ");
         // utils.ReflectionUtils.getAllMethods scans your superclass too !
        Method[] methods =  utils.ReflectionUtils.getAllMethods(runner.getClass());
        
        for (Method method : methods) 
          {
            utils.ReflectionUtils.printMethodInfos(method);
            Annotation annos = method.getAnnotation(CanRun.class);
            if (annos != null) 
            {
                System.out.println("--> Found CanRun.class Annotation for method: " + method.getName() );
                try {
                    System.out.println("--> Invoking this method via reflection ");
                    method.invoke(runner);
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }
            else
              {
              System.out.println("--> Found NO Annotation for method: " + method.getName() );  
              }
          }
//        Field[] fields   = runner.getClass().getFields();
        
        System.out.println("\n --> Exploring Attributes:");
        Field[] fields =  utils.ReflectionUtils.getAllFields(runner.getClass());
        for (Field f : fields) 
        {
            utils.ReflectionUtils.printFieldInfos(f);
            Annotation annos = f.getAnnotation(CanChange.class);
            if (annos != null) 
            {
                System.out.println("--> Found Annotation for field: " + f.getName() + " " + f.getType()  );  
                try 
                {                  
                    if (  f.getType().equals(int.class) )
                    {
                        int k = 99 ;
                        System.out.println("--> Current value for id1 : " + runner.id1 );
                        f.setInt(runner,k);
                        System.out.println("--> Changing int value for attr " +f.getName() + 
                                 " via reflection - New value: " + runner.id1 );
                    }
                } 
                catch (IllegalAccessException ex) 
                {
                    ex.printStackTrace();
                }
            }
        }
    }
}

Program Output running MyTest.java

-->Inspect Class:
Canonical Class Name: AnnotationTest.AnnotationRunner
Class Name          : AnnotationTest.AnnotationRunner
Superclass Name     : class java.lang.Object
Interfaces          : []
 -> autor           : Helmut Hutzer
 -> date            : 8-Feb-2014
 -> comment         : Intial Class Creation for Testing annotations

-->Inspect Class:
Canonical Class Name: utils.CanChange
Class Name          : utils.CanChange
Superclass Name     : null
Interfaces          : [interface java.lang.annotation.Annotation]

--> Exploring Constructors  
public  AnnotationTest.AnnotationRunner([int, int])
public  AnnotationTest.AnnotationRunner()
Annotations: [@utils.CanConstruct()]
--> Found a constructor without parameters 
--> Found Annotation CanConstruct   
public  AnnotationTest.AnnotationRunner()
Annotations: [@utils.CanConstruct()]
--> Created new instance of class  AnnotationRunner
--> print Field id1 via reflecion : 1

--> Exploring method annotations 
public void method1()
--> Found NO Annotation for method: method1
public void method2()
Annotations: [@utils.CanRun()]
--> Found CanRun.class Annotation for method: method2
--> Invoking this method via reflection 
Hello from method2 !
public void set_id1([int])
--> Found NO Annotation for method: set_id1
public void set_id2([int])
--> Found NO Annotation for method: set_id2

 --> Exploring Attributes:
public int id1
Annotations: [@utils.CanChange()]
--> Found Annotation for field: id1 int
--> Current value for id1 : 1
--> Changing int value for attr id1 via reflection - New value: 99
public int id2

Reference