Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

XA全局事务超时后回滚失败，一直报XAER_RMFAIL #4710

Closed
1 task done
WangXiaoJin opened this issue Jun 16, 2022 · 5 comments · Fixed by #5131
Closed
1 task done

XA全局事务超时后回滚失败，一直报XAER_RMFAIL #4710

WangXiaoJin opened this issue Jun 16, 2022 · 5 comments · Fixed by #5131

Comments

@WangXiaoJin
Copy link

WangXiaoJin commented Jun 16, 2022 •
edited
Loading

  • I have searched the issues of this repository and believe that this is not a duplicate.

Ⅰ. Issue Description

XA全局事务超时后回滚失败，一直报XAER_RMFAIL

Ⅱ. Describe what happened

流程：
全局事务超时时间配置为10秒。在A服务上配置了全局事务注解，A服务更新数据库然后调用B服务。B服务更新数据库后休眠15秒，15秒后提交B服务的事务。A、B服务使用同一个数据库实例的多张表。

2022-06-16 21:21:55.612 INFO  [rpcDispatch_RMROLE_1_12_16] i.s.c.r.p.c.RmBranchRollbackProcessor - rm handle branch rollback process:xid=10.90.13.27:8091:2549311505400693112,branchId=234461296923190606,branchType=XA,resourceId=jdbc:mysql://192.168.201.120:3306/spring_clond_demo,applicationData=null
2022-06-16 21:21:55.612 INFO  [rpcDispatch_RMROLE_1_12_16] io.seata.rm.AbstractRMHandler - Branch Rollbacking: 10.90.13.27:8091:2549311505400693112 234461296923190606 jdbc:mysql://192.168.201.120:3306/spring_clond_demo
2022-06-16 21:21:55.621 INFO  [rpcDispatch_RMROLE_1_12_16] i.s.r.d.xa.ResourceManagerXA - 10.90.13.27:8091:2549311505400693112-234461296923190606 rollback failed since XAER_RMFAIL: The command cannot be executed when global transaction is in the  ACTIVE state
com.mysql.cj.jdbc.MysqlXAException: XAER_RMFAIL: The command cannot be executed when global transaction is in the  ACTIVE state
	at com.mysql.cj.jdbc.MysqlXAConnection.mapXAExceptionFromSQLException(MysqlXAConnection.java:344)
	at com.mysql.cj.jdbc.MysqlXAConnection.dispatchCommand(MysqlXAConnection.java:329)
	at com.mysql.cj.jdbc.MysqlXAConnection.rollback(MysqlXAConnection.java:243)
	at io.seata.rm.datasource.xa.ConnectionProxyXA.xaRollback(ConnectionProxyXA.java:141)
	at io.seata.rm.datasource.xa.ConnectionProxyXA.xaRollback(ConnectionProxyXA.java:132)
	at io.seata.rm.datasource.xa.ResourceManagerXA.finishBranch(ResourceManagerXA.java:115)
	at io.seata.rm.datasource.xa.ResourceManagerXA.branchRollback(ResourceManagerXA.java:100)
	at io.seata.rm.AbstractRMHandler.doBranchRollback(AbstractRMHandler.java:125)
	at io.seata.rm.AbstractRMHandler$2.execute(AbstractRMHandler.java:67)
	at io.seata.rm.AbstractRMHandler$2.execute(AbstractRMHandler.java:63)
	at io.seata.core.exception.AbstractExceptionHandler.exceptionHandleTemplate(AbstractExceptionHandler.java:131)
	at io.seata.rm.AbstractRMHandler.handle(AbstractRMHandler.java:63)
	at io.seata.rm.DefaultRMHandler.handle(DefaultRMHandler.java:68)
	at io.seata.core.protocol.transaction.BranchRollbackRequest.handle(BranchRollbackRequest.java:35)
	at io.seata.rm.AbstractRMHandler.onRequest(AbstractRMHandler.java:150)
	at io.seata.core.rpc.processor.client.RmBranchRollbackProcessor.handleBranchRollback(RmBranchRollbackProcessor.java:63)
	at io.seata.core.rpc.processor.client.RmBranchRollbackProcessor.process(RmBranchRollbackProcessor.java:58)
	at io.seata.core.rpc.netty.AbstractNettyRemoting.lambda$processMessage$2(AbstractNettyRemoting.java:281)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: XAER_RMFAIL: The command cannot be executed when global transaction is in the  ACTIVE state
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
	at com.mysql.cj.jdbc.StatementImpl.executeInternal(StatementImpl.java:762)
	at com.mysql.cj.jdbc.StatementImpl.execute(StatementImpl.java:646)
	at com.mysql.cj.jdbc.MysqlXAConnection.dispatchCommand(MysqlXAConnection.java:323)
	... 20 common frames omitted
2022-06-16 21:21:55.621 INFO  [rpcDispatch_RMROLE_1_12_16] io.seata.rm.AbstractRMHandler - Branch Rollbacked result: PhaseTwo_RollbackFailed_Retryable
2022-06-16 21:21:56.058 INFO  [rpcDispatch_RMROLE_1_13_16] i.s.c.r.p.c.RmBranchRollbackProcessor - rm handle branch rollback process:xid=10.90.13.27:8091:2549311505400693112,branchId=234461296923190606,branchType=XA,resourceId=jdbc:mysql://192.168.201.120:3306/spring_clond_demo,applicationData=null
2022-06-16 21:21:56.058 INFO  [rpcDispatch_RMROLE_1_13_16] io.seata.rm.AbstractRMHandler - Branch Rollbacking: 10.90.13.27:8091:2549311505400693112 234461296923190606 jdbc:mysql://192.168.201.120:3306/spring_clond_demo
2022-06-16 21:21:56.065 INFO  [rpcDispatch_RMROLE_1_13_16] i.s.r.d.xa.ResourceManagerXA - 10.90.13.27:8091:2549311505400693112-234461296923190606 rollback failed since XAER_RMFAIL: The command cannot be executed when global transaction is in the  ACTIVE state
com.mysql.cj.jdbc.MysqlXAException: XAER_RMFAIL: The command cannot be executed when global transaction is in the  ACTIVE state
	at com.mysql.cj.jdbc.MysqlXAConnection.mapXAExceptionFromSQLException(MysqlXAConnection.java:344)
	at com.mysql.cj.jdbc.MysqlXAConnection.dispatchCommand(MysqlXAConnection.java:329)
	at com.mysql.cj.jdbc.MysqlXAConnection.rollback(MysqlXAConnection.java:243)
	at io.seata.rm.datasource.xa.ConnectionProxyXA.xaRollback(ConnectionProxyXA.java:141)
	at io.seata.rm.datasource.xa.ConnectionProxyXA.xaRollback(ConnectionProxyXA.java:132)
	at io.seata.rm.datasource.xa.ResourceManagerXA.finishBranch(ResourceManagerXA.java:115)
	at io.seata.rm.datasource.xa.ResourceManagerXA.branchRollback(ResourceManagerXA.java:100)
	at io.seata.rm.AbstractRMHandler.doBranchRollback(AbstractRMHandler.java:125)
	at io.seata.rm.AbstractRMHandler$2.execute(AbstractRMHandler.java:67)
	at io.seata.rm.AbstractRMHandler$2.execute(AbstractRMHandler.java:63)
	at io.seata.core.exception.AbstractExceptionHandler.exceptionHandleTemplate(AbstractExceptionHandler.java:131)
	at io.seata.rm.AbstractRMHandler.handle(AbstractRMHandler.java:63)
	at io.seata.rm.DefaultRMHandler.handle(DefaultRMHandler.java:68)
	at io.seata.core.protocol.transaction.BranchRollbackRequest.handle(BranchRollbackRequest.java:35)
	at io.seata.rm.AbstractRMHandler.onRequest(AbstractRMHandler.java:150)
	at io.seata.core.rpc.processor.client.RmBranchRollbackProcessor.handleBranchRollback(RmBranchRollbackProcessor.java:63)
	at io.seata.core.rpc.processor.client.RmBranchRollbackProcessor.process(RmBranchRollbackProcessor.java:58)
	at io.seata.core.rpc.netty.AbstractNettyRemoting.lambda$processMessage$2(AbstractNettyRemoting.java:281)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: XAER_RMFAIL: The command cannot be executed when global transaction is in the  ACTIVE state
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
	at com.mysql.cj.jdbc.StatementImpl.executeInternal(StatementImpl.java:762)
	at com.mysql.cj.jdbc.StatementImpl.execute(StatementImpl.java:646)
	at com.mysql.cj.jdbc.MysqlXAConnection.dispatchCommand(MysqlXAConnection.java:323)
	... 20 common frames omitted
2022-06-16 21:21:56.065 INFO  [rpcDispatch_RMROLE_1_13_16] io.seata.rm.AbstractRMHandler - Branch Rollbacked result: PhaseTwo_RollbackFailed_Retryable

有时会在70秒后出现Branch Rollbacked result: PhaseTwo_RollbackFailed_XAER_NOTA_Retryable，然后回滚成功，多数情况下会一直报错下去。

Ⅵ. Environment:

  • JDK version : 1.8
  • Seata version: 1.5.1
  • OS : Windows
  • MySQL: 5.7.30
@funky-eyes
Copy link
Contributor

多tc下由于mysql必须使用registry的rm节点进行二阶段处理,所以默认的client.rm.connectionTwoPhaseHoldTimeoutXA配置为10s,当超过10s,会断开registry的rm对应的分支事务使用的connection,此时其他的rm便可以帮助回滚,所以在多tc+多rm节点下会增加该情况出现的频率,你可以暂时将client.rm.connectionTwoPhaseHoldTimeoutXA 配置10s改为合理的时间,假设你一次调用完整连接timeout为1s,那么client.rm.connectionTwoPhaseHoldTimeoutXA应该配置在1s内,将这个配置放到你所使用的配置中心中即可

@WangXiaoJin
Copy link
Author

@a364176773 您好，配置成1s后现象还是一样的，与之前相比分支事务并没有提前成功回滚

@funky-eyes
Copy link
Contributor

@a364176773 您好，配置成1s后现象还是一样的，与之前相比分支事务并没有提前成功回滚

你的业务rt是多少？就是这个接口一个完整调用如果是100ms，那1s肯定不会提前回滚成功，你业务都结束了，连接没释放，你可以改成1个tc或那个回滚要很久的rm就1个节点先试下，肯定是没这个问题的，这个问题存在与rm注册的时候到每个tc都是不同的remoteaddress，没有唯一标识，导致begin的时候在tc1，registry在tc3，commit或rollback在tc2，tc2就找不到rm注册的那个channel，就会出现这种情况，只能靠无限重试和rm缩短释放连接的时间来优化，后续我们会做根据xid的负载均衡，这种适合xa这种模式，begin在tc1，registry也会在tc1，commit也会在tc1

@WangXiaoJin
Copy link
Author

好的，了解了，感谢回复。只能暂时考虑其他模式了。

@funky-eyes
Copy link
Contributor

下周二我们会发布1.5.2默认为xid的负载均衡算法来优化这个现象，欢迎届时使用新版本进行测试

@funky-eyes funky-eyes mentioned this issue Dec 6, 2022
Merged
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

bugfix: rollback active xa connection fail funky-eyes/seata
2 participants
@WangXiaoJin @funky-eyes