Table of Contents
Overview JPA and 2-Phase-Commit
Mike Keith Architect at Oracle and Author
Pro JPA 2: Mastering the Java Persistence API (Second edition)
summarizes the usage of JPA in a distributed evironment the following :
- A JPA application will get the 2PC benefits the same as any other application
- The peristence unit data source is using JTA and is configured to use an XA data source
- The XA resources and transaction manager 2PC interactions happen on their own without the JPA EMF knowing or having to be involved.
- If a 2PC XA tx fails then an exception will be thrown just the same as if the tx was optimized to not have 2PC.
This was enough motivation for me working on Oracle RAC and JDBC projects to have a closer look on JPA and 2PC.
Versions used / Configuration File persistence.xml
Wildfly: 8.2 Hibernate Version: 4.3.7.Final --> Collecting Data for RAC database1 Driver Name : Oracle JDBC driver Driver Version : 12.1.0.2.0 Database Product Version: Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production DB Name: BANKA 1. Instance Name: bankA_2 - Host: hract21.example.com - Pooled XA Connections: 61 --> Collecting Data for RAC database2 Driver Name : Oracle JDBC driver Driver Version : 12.1.0.2.0 Database Product Version: Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production DB Name: BANKB 1. Instance Name: bankb_3 - Host: hract21.example.com - Pooled XA Connections: 62 persistence.xml <?xml version="1.0"?> <persistence xmlns="http://java.sun.com/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd" version="2.0"> <persistence-unit name="RacBankAHibPU" transaction-type="JTA"> <provider>org.hibernate.ejb.HibernatePersistence</provider> <jta-data-source>java:/jboss/datasources/xa_rac12g_banka</jta-data-source> <class>com.hhu.wfjpa2pc.Accounts</class> <properties> <property name="hibernate.transaction.jta.platform" value="org.hibernate.service.jta.platform.internal.JBossAppServerJtaPlatform" /> <property name="hibernate.show_sql" value="true" /> <property name="hibernate.dialect" value="org.hibernate.dialect.Oracle10gDialect"/> </properties> </persistence-unit> <persistence-unit name="RacBankBHibPU" transaction-type="JTA"> <provider>org.hibernate.ejb.HibernatePersistence</provider> <jta-data-source>java:/jboss/datasources/xa_rac12g_bankb</jta-data-source> <class>com.hhu.wfjpa2pc.Accounts</class> <properties> <property name="hibernate.transaction.jta.platform" value="org.hibernate.service.jta.platform.internal.JBossAppServerJtaPlatform" /> <property name="hibernate.show_sql" value="true" /> <property name="hibernate.dialect" value="org.hibernate.dialect.Oracle10gDialect"/> </properties> </persistence-unit> </persistence>
Running a successful 2PC operation with JPA
Call Flow - Get EntityManager for RAC Database1 [ em1=getEntityManager1(); ] - Get EntityManager for RAC Database2 [ em2=getEntityManager2(); ] - Start as Usertransacation [ ut.begin(); ] - Join transaction from EntityManager 1 [ em1.joinTransaction(); ] - Join transaction from EntityManager 2 [ em2.joinTransaction(); ] - Chance Balance on both databases bankA_acct.setBalance( bankA_acct.getBalance().add(b) ); em1.merge(bankA_acct); if (isEnableFlush() ) em1.flush(); bankB_acct.setBalance( bankB_acct.getBalance().subtract(b) ); em2.merge(bankB_acct); if (isEnableFlush() ) em2.flush(); - Finally commit the Transaction [ ut.commit(); ] Application log : 14:51:58.071 transferMoneyImpl():: Found both Entity Managers for PUs : RacBankAHibPU and RacBankBHibPU 14:51:58.074 transferMoneyImpl():: Account at bank A: User99_at_BANKA - Balance: 10000 14:51:58.075 transferMoneyImpl():: Account at bank B: User98_at_BANKB - Balance: 10000 14:51:58.076 transferMoneyImpl():: Both EMs joined our XA Transaction... 14:51:58.092 transferMoneyImpl():: Before Commit ... 14:51:58.160 transferMoneyImpl():: Tx Commit worked ! 14:51:58.165 Database Name:BANKA -- Account: User99_at_BANKA -- Balance: 11000.0 14:51:58.168 Database Name:BANKB -- Account: User98_at_BANKB -- Balance: 9000.0 14:51:58.169 transferMoneyImpl():: Leaving with TX Status:: [UT status: 6 - STATUS_NO_TRANSACTION] -> We successfully managed to transfer some money from bankA to bankB !
Testing Rollback operation with EM flush enabled [ transaction status : STATUS_MARKED_ROLLBACK ]
Account Balance transferMoneyImpl():: Account at bank A: User99_at_BANKA - Balance: 20000 transferMoneyImpl():: Account at bank B: User98_at_BANKB - Balance: 0 Note the next money transfer/transaction should trigger a constraint violation ! Call Flow - Get EntityManager for RAC Database1 [ em1=getEntityManager1(); ] - Get EntityManager for RAC Database2 [ em2=getEntityManager2(); ] - Start a User transaction [ ut.begin(); ] - Join transaction from EntityManager 1 [ em1.joinTransaction(); ] - Join transaction from EntityManager 2 [ em2.joinTransaction(); ] - Chance Balance on both databases bankA_acct.setBalance( bankA_acct.getBalance().add(b) ); em1.merge(bankA_acct); if (isEnableFlush() ) em1.flush(); bankB_acct.setBalance( bankB_acct.getBalance().subtract(b) ); em2.merge(bankB_acct); if (isEnableFlush() ) em2.flush(); - em2.flush is failing due to a constraint violation and set the TX status to : STATUS_MARKED_ROLLBACK Error : org.hibernate.exception.ConstraintViolationException: could not execute statement - Exception handler checks transaction status : STATUS_MARKED_ROLLBACK and is rolling back the TX if ( status != javax.transaction.Status.STATUS_NO_TRANSACTION ) { ut.rollback(); ... - After rollback() transaction status changed to STATUS_NO_TRANSACTION Application log : 15:11:03.920 transferMoneyImpl():: Found both Entity Managers for PUs : RacBankAHibPU and RacBankBHibPU 15:11:03.929 transferMoneyImpl():: Account at bank A: User99_at_BANKA - Balance: 20000 15:11:03.931 transferMoneyImpl():: Account at bank B: User98_at_BANKB - Balance: 0 15:11:03.931 transferMoneyImpl():: Both EMs joined our XA Transaction... 15:11:03.960 transferMoneyImpl():: FATAL ERROR - Tx Status : [UT status: 1 - STATUS_MARKED_ROLLBACK] 15:11:03.962 transferMoneyImpl():: Before TX rollback ... 15:11:03.974 transferMoneyImpl():: TX rollback worked ! 15:11:03.974 transferMoneyImpl():: Leaving with TX Status:: [UT status: 6 - STATUS_NO_TRANSACTION] Exception stack : 15:11:03.960 Error in top level function: transferMoneyImpl():: 15:11:03.960 org.hibernate.exception.ConstraintViolationException: could not execute statement 15:11:03.961 javax.persistence.PersistenceException: org.hibernate.exception.ConstraintViolationException: could not execute statement at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1763) at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1677) at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1683) at org.hibernate.jpa.spi.AbstractEntityManagerImpl.flush(AbstractEntityManagerImpl.java:1338) at com.hhu.wfjpa2pc.Jpa2pcTest.transferMoneyImpl(Jpa2pcTest.java:235) at com.hhu.wfjpa2pc.Jpa2pcTest.transferMoney(Jpa2pcTest.java:166) .. Caused by: org.hibernate.exception.ConstraintViolationException: could not execute statement at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:72) at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java ... Caused by: java.sql.SQLIntegrityConstraintViolationException: ORA-02290: check constraint (SCOTT.S_LOWER_CHK) violated
Testing Rollback operation without EM flush enabled [ transaction status : STATUS_NO_TRANSACTION ]
Account Balance transferMoneyImpl():: Account at bank A: User99_at_BANKA - Balance: 20000 transferMoneyImpl():: Account at bank B: User98_at_BANKB - Balance: 0 Note the next money transfer/transaction should trigger a constraint violation ! Call Flow - Get EntityManager for RAC Database1 [ em1=getEntityManager1(); ] - Get EntityManager for RAC Database2 [ em2=getEntityManager2(); ] - Start a User transaction [ ut.begin(); ] - Join transaction from EntityManager 1 [ em1.joinTransaction(); ] - Join transaction from EntityManager 2 [ em2.joinTransaction(); ] - Chance Balance on both databases bankA_acct.setBalance( bankA_acct.getBalance().add(b) ); em1.merge(bankA_acct); if (isEnableFlush() ) em1.flush(); bankB_acct.setBalance( bankB_acct.getBalance().subtract(b) ); em2.merge(bankB_acct); if (isEnableFlush() ) em2.flush(); - Commit the Transaction [ ut.commit(); ] fails with : ARJUNA016053: Could not commit transaction. - As the Commit itself fails Wildfly rollback the transaction - Tx Status after COMMIT error : STATUS_NO_TRANSACTION - Exception handler checks transaction status : STATUS_MARKED_ROLLBACK and is not rolling back the TX if ( status != javax.transaction.Status.STATUS_NO_TRANSACTION ) { ut.rollback(); ... - Here we don't run any rollback operation -> the TX status remains at STATUS_NO_TRANSACTION Application log : 15:27:53.818 transferMoneyImpl():: Found both Entity Managers for PUs : RacBankAHibPU and RacBankBHibPU 15:27:53.827 transferMoneyImpl():: Account at bank A: User99_at_BANKA - Balance: 20000 15:27:53.829 transferMoneyImpl():: Account at bank B: User98_at_BANKB - Balance: 0 15:27:53.829 transferMoneyImpl():: Both EMs joined our XA Transaction... 15:27:53.829 transferMoneyImpl():: Before Commit ... 15:27:53.857 transferMoneyImpl():: FATAL ERROR - Tx Status : [UT status: 6 - STATUS_NO_TRANSACTION] 15:27:53.859 transferMoneyImpl():: TX not active / TX already rolled back 15:27:53.859 transferMoneyImpl():: Leaving with TX Status:: [UT status: 6 - STATUS_NO_TRANSACTION]
Testing transaction Recovery with JPA
What we are expecting and what we are testing - Transaction Timeout is set to 600 seconds - We set a breakpoint at OracleXAResource.commit ==> This means Wildfly has written a COMMIT record to the Wildlfly LOG-STORE - After stop at the first OracleXAResource.commit breakpoint we kill the Wildfly server - Both RMs [ Oracle RAC databases ] are now counting down the Transaction Timeout - If Timeout is reached the failed transaction becomes visible in dba_2pc_pending table - Trying to get a lock on these records should lead to a ORA-1591 error - After Wildfly restart the Periodic Recovery should run OracleXAResource.commit and release all locks Preparing and running the test scenario Start Wildfly in Debug Mode : Set breakpoint on OracleXAResource.commit and run the application Stack Trace "default task-3" oracle.jdbc.xa.client.OracleXAResource.commit(OracleXAResource.java:553) org.jboss.jca.adapters.jdbc.xa.XAManagedConnection.commit(XAManagedConnection.java:338) org.jboss.jca.core.tx.jbossts.XAResourceWrapperImpl.commit(XAResourceWrapperImpl.java:107) com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelCommit(XAResourceRecord.java:461) com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(BasicAction.java:2810) com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(BasicAction.java:2726) com.arjuna.ats.arjuna.coordinator.BasicAction.phase2Commit(BasicAction.java:1820) com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1504) com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96) com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162) com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1166) com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126) com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75) org.jboss.tm.usertx.client.ServerVMClientUserTransaction.commit(ServerVMClientUserTransaction.java:173) com.hhu.wfjpa2pc.Jpa2pcTest.transferMoneyImpl(Jpa2pcTest.java:242) com.hhu.wfjpa2pc.Jpa2pcTest.transferMoney(Jpa2pcTest.java:166) Wildfly Check for prepared transaction $ $WILDFLY_HOME/bin/jboss-cli.sh --connect --file=list_prepared_xa_tx.cli {"outcome" => "success"} 0:ffffc0a805c9:f5a10ef:56039e68:d Locate and kill JBOSS server process 0 S oracle 5875 5821 7 80 0 - 413473 futex_ 08:55 ? 00:00:30 /usr/java/latest/bin/java .... -Djboss.server.base.dir=/usr/local/wildfly-8.2.0.Final/standalone -c standalone.xml 0 S oracle 6174 5680 0 80 0 - 25827 pipe_w 09:02 pts/1 00:00:00 grep java [oracle@wls1 WILDFLY]$ kill -9 5875 Now wait [ at lest 600 seconds ] until the Transaction becomes visible in dba_2pc_pending SQL> SELECT * FROM GLOBAL_NAME; GLOBAL_NAME ---------------- BANKA SQL> select * from dba_2pc_pending; LOCAL_TRAN_ID GLOBAL_TRAN_ID STATE MIX A TRAN_COMMENT ---------------------- ---------------------------------------------------------------- ---------------- --- - ---------------- FAIL_TIM FORCE_TI RETRY_TI OS_USER OS_TERMINAL HOST DB_USER COMMIT# -------- -------- -------- ------------ ------------ ---------------- ------------ ---------------- 9.21.7139 131077.00000000000000000000FFFFC0A805C90F5A10EF56039E680000000D3 prepared no 1 09:07:22 09:15:34 oracle unknown wls1.example.com 43619336 SQL> SELECT * FROM GLOBAL_NAME; GLOBAL_NAME ---------------- BANKB SQL> select * from dba_2pc_pending; LOCAL_TRAN_ID GLOBAL_TRAN_ID STATE MIX A TRAN_COMMENT ---------------------- ---------------------------------------------------------------- ---------------- --- - ---------------- FAIL_TIM FORCE_TI RETRY_TI OS_USER OS_TERMINAL HOST DB_USER COMMIT# -------- -------- -------- ------------ ------------ ---------------- ------------ ---------------- 4.15.3293 131077.00000000000000000000FFFFC0A805C90F5A10EF56039E680000000D3 prepared no 1 09:07:22 09:15:34 oracle unknown wls1.example.com 20931538 Check for locks -> Connected to scott/tiger@ract2-scan.grid12c.example.com:1521/banka select * from accounts for update * ERROR at line 1: ORA-01591: lock held by in-doubt distributed transaction 9.21.7139 -> Connected to scott/tiger@ract2-scan.grid12c.example.com:1521/bankb select * from accounts for update * ERROR at line 1: ORA-01591: lock held by in-doubt distributed transaction 4.15.3293 Restart Wildfly in Debug Mode and let the Periodic Recovery Thread commit the transaction "Periodic Recovery" oracle.jdbc.xa.client.OracleXAResource.commit(OracleXAResource.java:553) org.jboss.jca.adapters.jdbc.xa.XAManagedConnection.commit(XAManagedConnection.java:338) org.jboss.jca.core.tx.jbossts.XAResourceWrapperImpl.commit(XAResourceWrapperImpl.java:107) com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelCommit(XAResourceRecord.java:461) com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(BasicAction.java:2810) com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(BasicAction.java:2726) com.arjuna.ats.arjuna.coordinator.BasicAction.phase2Commit(BasicAction.java:1820) com.arjuna.ats.arjuna.recovery.RecoverAtomicAction.replayPhase2(RecoverAtomicAction.java:71) com.arjuna.ats.internal.arjuna.recovery.AtomicActionRecoveryModule.doRecoverTransaction(AtomicActionRecoveryModule.java:152) com.arjuna.ats.internal.arjuna.recovery.AtomicActionRecoveryModule.processTransactionsStatus(AtomicActionRecoveryModule.java:253) com.arjuna.ats.internal.arjuna.recovery.AtomicActionRecoveryModule.periodicWorkSecondPass(AtomicActionRecoveryModule.java:109) com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789) com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) -> WildFly Thread Periodic Recovery stops at OracleXAResource.commit -> Press Debugger Command : Continue -> WildFly Thread Periodic Recovery has committed Transaction Branch 1 -> WildFly Thread Periodic Recovery stops again at .OracleXAResource.commit -> Press Debugger Command : Continue -> WildFly Thread Periodic Recovery has committed Transaction Branch 2 -> Complete Transaction is now committed Verify Access to the Database records and Wildfly Prepared Transaction Cleanup -> Connected to scott/tiger@ract2-scan.grid12c.example.com:1521/banka ACCOUNT BALANCE -------------------------------- ---------- User99_at_BANKA 14000 -> Connected to scott/tiger@ract2-scan.grid12c.example.com:1521/bankb ACCOUNT BALANCE -------------------------------- ---------- User98_at_BANKB 6000 List prepared Transaction $ $WILDFLY_HOME/bin/jboss-cli.sh --connect --file=list_prepared_xa_tx.cli {"outcome" => "success"} -> After a successful transaction recovery the locks are gone
Java Code
public void transferMoneyImpl()
{
String methodName = "transferMoneyImpl():: ";
EntityManager em1;
EntityManager em2;
UserTransaction ut =null;
try
{
setRunTimeInfo(methodName + "Entering ... ");
HttpSession session = (HttpSession) FacesContext.getCurrentInstance().getExternalContext().getSession(true);
if ( session == null)
{
throw new IllegalArgumentException(methodName+ ": Could not get HTTP session : ");
}
final Object lock = session.getId().intern();
synchronized(lock)
{
em1=getEntityManager1();
em2=getEntityManager2();
//
// Note even we get an EntityManager Object we still not sure that the
// EntityManager Could open connection the underlying JDBC connection !
//
if ( em1 == null )
setRunTimeInfo(methodName + "Faild to get EM for PU: " + EMF.getPU1() );
else if ( em2 == null )
setRunTimeInfo(methodName + "Faild to get EM for PU: " + EMF.getPU2() );
else
setRunTimeInfo(methodName + "Found both Entity Managers for PUs : " +
EMF.getPU1() + " and " + EMF.getPU2() );
String bankA_acct_name = "User99_at_BANKA";
Accounts bankA_acct = em1.find(Accounts.class, bankA_acct_name);
if ( bankA_acct == null)
{
setRunTimeInfo(methodName + "Could not locate Account at bankA : " + bankA_acct_name );
return;
}
setRunTimeInfo(methodName +"Account at bank A: " + bankA_acct.getAccount() + " - Balance: " + bankA_acct.getBalance() );
String bankB_acct_name = "User98_at_BANKB";
Accounts bankB_acct = em2.find(Accounts.class, bankB_acct_name);
if ( bankB_acct == null)
{
setRunTimeInfo(methodName + "Could not locate Account at bankB : " + bankB_acct_name );
return;
}
setRunTimeInfo(methodName +"Account at bank B: " + bankB_acct.getAccount() + " - Balance: " + bankB_acct.getBalance() );
ut = (javax.transaction.UserTransaction)new InitialContext().lookup("java:comp/UserTransaction");
// Set tranaction time to 120 seconds to avoid any timeouts during testing -
// especially when testing transaction recovery by restarting Wildfly server
// Note as we kill the JAVA process both RMs will wait 120 s before Tx becomes visible in dba_2pc_pending
int tx_timeout = 120;
ut.setTransactionTimeout(tx_timeout);
ut.begin();
em1.joinTransaction();
em2.joinTransaction();
setRunTimeInfo(methodName + "Both EMs joined our XA Transaction... - TX Timeout: " + tx_timeout );
BigDecimal b = new BigDecimal(1000);
bankA_acct.setBalance( bankA_acct.getBalance().add(b) );
em1.merge(bankA_acct);
if (isEnableFlush() )
em1.flush();
bankB_acct.setBalance( bankB_acct.getBalance().subtract(b) );
em2.merge(bankB_acct);
if (isEnableFlush() )
em2.flush();
setRunTimeInfo(methodName + "Before Commit ... ");
ut.commit();
setRunTimeInfo(methodName + "Tx Commit worked !");
checkBalanceImpl();
}
} catch ( Throwable t1)
{
try
{
String tx_status = returnTXStatus(ut);
setRunTimeInfo( methodName + "FATAL ERROR - Tx Status : " + tx_status );
// Use Throwable as we don't want to loose any important imformation
// Note: Throwable is super class of Exception class
genericException("Error in top level function: " + methodName , (Exception)t1);
if ( ut != null )
{
int status = ut.getStatus();
// rollback transaction if still active - if not do nothing
if ( status != javax.transaction.Status.STATUS_NO_TRANSACTION ) {
setRunTimeInfo(methodName + "Before TX rollback ... ");
ut.rollback();
setRunTimeInfo(methodName + "TX rollback worked !");
} else
setRunTimeInfo(methodName + "TX not active / TX already rolled back");
}
} catch ( Throwable t2)
{
genericException(methodName + "FATAL ERROR during ut.rollback() ", (Exception)t2);
}
}
closeEntityManagers();
String tx_status_exit = "";
try
{
tx_status_exit = returnTXStatus(ut);
} catch ( Throwable t3)
{
genericException(methodName + " Error during returning TX status ", (Exception)t3);
}
setRunTimeInfo(methodName + "Leaving with TX Status:: " + tx_status_exit );
}
Reference
- Mike Keith / Architect at Oracle / Author Pro JPA 2: Mastering the Java Persistence API (Second edition)
- A deep dive into 2-Phase-Commit with Wildfly and Oracle RAC
One thought on “A deeper dive into JPA, 2-Phase-Commit [ 2PC ] and RAC”