Issue124

[AionJayT]

fix issue
#124

Due to the synchronized method cause the deadlock between the pendingTxImpl and the txpool and then the event manager callback function get stuck.

Note. each handler of the event manager is a single thread. the callback execute function don't need the synchronized term unless there is another thread calling the same method.

[iamyulong]

Could you please elaborate which methods cause the deadock, for which threads?

I'm assuming a deadlock model like this:

T1: calls pendingTxImpl's synchronized method, which calls a synchronized method of txpool
T2: calls txpool's synchronized method, which calls a synchronized method of pendingTxImpl

[AionJayT]

New block Sealed -> OnBest event -> (BlockHandler thread, T1) callback-> createNewBlockTemplate -> getPendingTransactions -> txpool.snapshot.

pow thread (AionPow, T2) -> createNewBlockTemplate -> getPendingTransactions -> txpool.snapshot.

[AionJayT]

EquiHash miner (multi tread here!) -> OnSolution event -> (Consensus thread) callback -> event been process by the order of the event queue in the Consensus handler -> processSolution -> import block -> if the best block -> clearPandingTx -> txpool.remove

[AionJayT]

Found one Java-level deadlock:
=============================
"pool-5-thread-11":
  waiting to lock monitor 0x00007f7a8c02b900 (object 0x00000006c77ca150, a org.aion.zero.impl.blockchain.AionPendingStateImpl),
  which is held by "BlkHdr"
"BlkHdr":
  waiting to lock monitor 0x00007f7b0000d480 (object 0x00000006c77ca0f0, a org.aion.zero.impl.blockchain.NonceMgr),
  which is held by "pool-5-thread-11"

Java stack information for the threads listed above:
===================================================
"pool-5-thread-11":
	at org.aion.zero.impl.blockchain.AionPendingStateImpl.bestNonceSet(AionPendingStateImpl.java:583)
	- waiting to lock <0x00000006c77ca150> (a org.aion.zero.impl.blockchain.AionPendingStateImpl)
	at org.aion.zero.impl.blockchain.NonceMgr.getNonceAndAdd(NonceMgr.java:84)
	- locked <0x00000006c77ca0f0> (a org.aion.zero.impl.blockchain.NonceMgr)
	at org.aion.api.server.ApiAion.getTxNonce(ApiAion.java:498)
	- locked <0x00000006c7400948> (a org.aion.api.server.pb.ApiAion0)
	at org.aion.api.server.ApiAion.getTxNonce(ApiAion.java:494)
	- locked <0x00000006c7400948> (a org.aion.api.server.pb.ApiAion0)
	at org.aion.api.server.ApiAion.sendTransaction(ApiAion.java:454)
	at org.aion.api.server.pb.ApiAion0.process(ApiAion0.java:441)
	at org.aion.api.server.zmq.HdlrZmq.process(HdlrZmq.java:61)
	at org.aion.api.server.zmq.ProtocolProcessor.workerRun(ProtocolProcessor.java:265)
	at org.aion.api.server.zmq.ProtocolProcessor.lambda$run$3(ProtocolProcessor.java:110)
	at org.aion.api.server.zmq.ProtocolProcessor$$Lambda$58/1836303785.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9.0.4/ThreadPoolExecutor.java:1167)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9.0.4/ThreadPoolExecutor.java:641)
	at java.lang.Thread.run(java.base@9.0.4/Thread.java:844)
"BlkHdr":
	at org.aion.zero.impl.blockchain.NonceMgr.flush(NonceMgr.java:132)
	- waiting to lock <0x00000006c77ca0f0> (a org.aion.zero.impl.blockchain.NonceMgr)
	at org.aion.zero.impl.blockchain.AionPendingStateImpl.processBest(AionPendingStateImpl.java:412)
	- locked <0x00000006c77ca150> (a org.aion.zero.impl.blockchain.AionPendingStateImpl)
	at org.aion.zero.impl.blockchain.AionPendingStateImpl$1.onBest(AionPendingStateImpl.java:170)
	at org.aion.zero.impl.blockchain.AionPendingStateImpl$1.onBest(AionPendingStateImpl.java:168)
	at org.aion.evtmgr.impl.handler.BlockHandler.dispatch(BlockHandler.java:63)
	at org.aion.evtmgr.impl.abs.AbstractHandler$1.run(AbstractHandler.java:66)

Found 1 deadlock.

[iamyulong]

Awesome!

Let's focus on fixing the deadlock between NonceMgr and AionPendingStateImpl.

[iamyulong]

This PR removes synchronized keywords, because the methods are called by single thread. Needs to be careful when adding another invoke in the future