Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Two bugs since Dencun (low peers + threads not finishing) #7094

Closed
RogerHKW opened this issue May 13, 2024 · 8 comments
Closed

Two bugs since Dencun (low peers + threads not finishing) #7094

RogerHKW opened this issue May 13, 2024 · 8 comments
Assignees
@ahamlat

Comments

@RogerHKW
Copy link

RogerHKW commented May 13, 2024 •
edited
Loading

Dear Besu team,

Since Dencun, Besu suffers from 2 bugs on my setup.

The first problem is a quite low number of peers for the first ~24 hours after restarting (that is the time it takes to slowly reach my objective of 75 peers). It is even below 10 peers for 2/3 hours, which is not healthy.

I saw in another report that you are aware and are investigating.

The second problem is very different and I didn't see it reported. It also started when Dencun launched. Therefore, it might have the same root cause as the first one, or maybe one of them is the cause of the other? I hope this will help you.

So, the second problem is that every 36 hours or so, Besu throws errors in the log because "a thread took too long to run". These errors accumulate so the node starts to miss some attestations and the CPU usage grows exponentially, which ends up in a total freeze of the computer in less than 2 hours.
I restart the computer and the bug comes again about 36 hours later.

IMG_20240513_093541.jpg

If it helps:

User=besu
Group=besu
Type=simple
Restart=always
RestartSec=5
Environment="JAVA_OPTS=-Xmx11G"
Environment="BESU_OPTS=-Xmx11G"
Environment="JAVA_OPTS=-XX:-HeapDumpOnOutOfMemoryError"
Environment="LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so"
ExecStart=/usr/local/bin/besu/bin/besu \
  --network=mainnet \
  --sync-mode=SNAP \
  --data-path=/dev/data2TB/besu-mainnet \
  --data-storage-format=BONSAI \
  --bonsai-limit-trie-logs-enabled \
  --engine-jwt-secret=/var/lib/jwtsecret/jwt.hex \
  --rpc-http-enabled \
  --rpc-http-api=ETH,NET,WEB3,TXPOOL \
  --tx-pool=layered \
  --tx-pool-enable-save-restore=true \
  --tx-pool-price-bump=2 \
  --tx-pool-max-prioritized=20000 \
  --tx-pool-layer-max-capacity=80000000 \
  --tx-pool-max-future-by-sender=512 \
  --max-peers=75
[Install]
WantedBy=multi-user.target
@macfarla
Copy link
Contributor

correct - low peers issue is being tracked under issue #6805 .

The second issue - a few questions

  • what CL are you running?
  • do you see a similar spike in GC time or memory?
  • can you share logs from besu if you have them

@ahamlat ahamlat self-assigned this May 14, 2024
@ahamlat
Copy link
Contributor

ahamlat commented May 14, 2024

@RogerHKW We were actually tracking the second issue internally as well.
The issue happened on one of our nodes and we were able to investigate and suggest a fix here .
We need to double check that it is the same issue on your setup, that's why @macfarla was asking for GC time and memory metrics. Could you please share the graphs of GC time and memory on the same period as the first graph you shared.

@RogerHKW
Copy link
Author

Thanks to both of you.

Consensus and validator: Teku.

Memory usage: the amount of memory that Besu uses does not change, it is the same after reboot as during the exponential phase preceding the complete freeze.

Garbage collector: is there a simple way to track it? I tried Environment="JAVA_OPTS=-Xlog:gc*:file=/home/xxx/gc.log in the service file but Besu does not take it ("Invalid -Xlog option '-Xlog:gc*:file=/home/xxx/gc.log', see error log for details.")

Besu log: I will share the part of the file that you need in several hours (tonight in my time zone).

@ahamlat
Copy link
Contributor

ahamlat commented May 14, 2024

@RogerHKW We can get GC metrics from the GC logs, but we already collect GC metrics with Prometheus. Do you have Besu Full Grafana dashboard ?
If yes, you can share "GC time" graph in "Java Memory Management" section
image
If no, I highly recommend to install it, you can find the steps here.
If it is not related to memory usage, we need to check RPC and P2P traffic. Besu full dashboard has those metrics, could you share the metrics you have related to RPC and P2P ?

@RogerHKW
Copy link
Author

RogerHKW commented May 15, 2024 •
edited
Loading

I apologize for the delay. I got pulled out of my beloved nerdy activities yesterday by some human being.

So, I read your installation manual for Grafana. It seems that I need a screen on the validator to watch the charts. It's a machine without a screen, I access it with SSH only. A solution could consist in opening the port of Grafana but I am afraid that I would lower the security of the machine.

By the way, I installed more than 36 hours ago the new version of Besu that you published 2 days ago and it is running smoothly so far.
I keep you posted, you can count on me.

Regarding the log of Besu, here it is. I start this extract at 5:29, before the first error is thrown at 5:31:21.
I looked before 5:29 and I did not see any error. However, the chart that I posted yesterday shows that the exponential rise of CPU usage starts before the first error. It was already too high 3 hours before. The normal usage is what you see on the right, after the spike (I have a script to restart the computer automatically to kill the bug).
My uneducated hypothesis is that the threads take longer and longer to execute, they already slow down way before the first error, and the errors start to sparkle only when the durations of the threads to exceed 1 minute.
You can see that several errors per second are thrown, that is why the log that I show you here stops at 5:31:27. Posting everything would be infeasible I guess, it is already so long over these 6 seconds.


Mai 13 05:29:52 B550M besu[1094]: 2024-05-13 05:29:52.651+02:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #19,858,419 / 90 tx / 16 ws / 0 blobs / base fee 3.97 gwei / 29,994,509 (100.0%) gas / (0xfa9769f900fb1609176b81931e2b0b285fa465141ed52cf76e9ad8e9a160c1fe) in 14.537s. Peers: 68
Mai 13 05:30:02 B550M besu[1094]: 2024-05-13 05:30:02.860+02:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #19,858,420 / 131 tx / 16 ws / 0 blobs / base fee 4.46 gwei / 11,656,105 (38.9%) gas / (0x6c4f06959e963753a016a8d9016906570bae94e75ede0690c2126ec2000ebef4) in 4.066s. Peers: 69
Mai 13 05:30:17 B550M besu[1094]: 2024-05-13 05:30:17.063+02:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #19,858,421 / 169 tx / 16 ws / 0 blobs / base fee 4.34 gwei / 17,189,492 (57.3%) gas / (0x9b320a89d8d78afe1dcb862731cf4a2a4a6671755b5e779f0b2f341edeb4c0ad) in 14.048s. Peers: 75
Mai 13 05:31:15 B550M besu[1094]: 2024-05-13 05:31:13.388+02:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 3793 ms, time limit is 2000 ms
Mai 13 05:31:21 B550M besu[1094]: 2024-05-13 05:31:21.361+02:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[vert.x-worker-thread-15,5,main] has been blocked for 60135 ms, time limit is 60000 ms
Mai 13 05:31:21 B550M besu[1094]: io.vertx.core.VertxException: Thread blocked
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/jdk.internal.misc.Unsafe.park(Native Method)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1864)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3465)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3436)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1898)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2072)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.response(ExecutionEngineJsonRpcMethod.java:115)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:44)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.TracedJsonRpcProcessor.process(TracedJsonRpcProcessor.java:41)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.TimedJsonRpcProcessor.process(TimedJsonRpcProcessor.java:45)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.AuthenticatedJsonRpcProcessor.process(AuthenticatedJsonRpcProcessor.java:51)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.handlers.AbstractJsonRpcExecutor.executeRequest(AbstractJsonRpcExecutor.java:87)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcObjectExecutor.execute(JsonRpcObjectExecutor.java:57)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$0(JsonRpcExecutorHandler.java:57)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler$$Lambda$1571/0x000077514365c8e8.accept(Unknown Source)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.util.Optional.ifPresentOrElse(Optional.java:196)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$2(JsonRpcExecutorHandler.java:54)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler$$Lambda$970/0x0000775143502390.handle(Unknown Source)
Mai 13 05:31:21 B550M besu[1094]:         at app//io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)
Mai 13 05:31:21 B550M besu[1094]:         at app//io.vertx.ext.web.impl.BlockingHandlerDecorator$$Lambda$1569/0x0000775143657410.handle(Unknown Source)
Mai 13 05:31:21 B550M besu[1094]:         at app//io.vertx.core.impl.ContextBase.lambda$null$0(ContextBase.java:137)
Mai 13 05:31:21 B550M besu[1094]:         at app//io.vertx.core.impl.ContextBase$$Lambda$1105/0x00007751435675a8.handle(Unknown Source)
Mai 13 05:31:21 B550M besu[1094]:         at app//io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264)
Mai 13 05:31:21 B550M besu[1094]:         at app//io.vertx.core.impl.ContextBase.lambda$executeBlocking$1(ContextBase.java:135)
Mai 13 05:31:21 B550M besu[1094]:         at app//io.vertx.core.impl.ContextBase$$Lambda$1104/0x0000775143567380.run(Unknown Source)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
Mai 13 05:31:21 B550M besu[1094]:         at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.lang.Thread.run(Thread.java:840)
Mai 13 05:31:21 B550M besu[1094]: 2024-05-13 05:31:21.364+02:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[vert.x-worker-thread-0,5,main] has been blocked for 62004 ms, time limit is 60000 ms
Mai 13 05:31:21 B550M besu[1094]: io.vertx.core.VertxException: Thread blocked
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/jdk.internal.misc.Unsafe.park(Native Method)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.util.concurrent.ForkJoinTask.awaitDone(ForkJoinTask.java:468)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:687)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:765)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.trie.bonsai.worldview.BonsaiWorldStateUpdateAccumulator.commit(BonsaiWorldStateUpdateAccumulator.java:294)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.processBlock(AbstractBlockProcessor.java:165)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.mainnet.BlockProcessor.processBlock(BlockProcessor.java:79)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.MainnetBlockValidator.processBlock(MainnetBlockValidator.java:207)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:135)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:83)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.validateBlock(MergeCoordinator.java:551)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.rememberBlock(MergeCoordinator.java:581)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.consensus.merge.blockcreation.TransitionCoordinator.rememberBlock(TransitionCoordinator.java:164)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.engine.AbstractEngineNewPayload.syncResponse(AbstractEngineNewPayload.java:319)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.lambda$response$0(ExecutionEngineJsonRpcMethod.java:90)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod$$Lambda$1578/0x0000775143660cf0.handle(Unknown Source)
Mai 13 05:31:21 B550M besu[1094]:         at app//io.vertx.core.impl.ContextBase.lambda$null$0(ContextBase.java:137)
Mai 13 05:31:21 B550M besu[1094]:         at app//io.vertx.core.impl.ContextBase$$Lambda$1105/0x00007751435675a8.handle(Unknown Source)
Mai 13 05:31:21 B550M besu[1094]:         at app//io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264)
Mai 13 05:31:21 B550M besu[1094]:         at app//io.vertx.core.impl.ContextBase.lambda$executeBlocking$1(ContextBase.java:135)
Mai 13 05:31:21 B550M besu[1094]:         at app//io.vertx.core.impl.ContextBase$$Lambda$1104/0x0000775143567380.run(Unknown Source)
Mai 13 05:31:21 B550M besu[1094]:         at app//io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
Mai 13 05:31:21 B550M besu[1094]:         at app//io.vertx.core.impl.TaskQueue$$Lambda$886/0x00007751434a50a8.run(Unknown Source)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
Mai 13 05:31:21 B550M besu[1094]:         at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.lang.Thread.run(Thread.java:840)
Mai 13 05:31:21 B550M besu[1094]: 2024-05-13 05:31:21.366+02:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[vert.x-worker-thread-15,5,main] has been blocked for 64283 ms, time limit is 60000 ms
Mai 13 05:31:21 B550M besu[1094]: io.vertx.core.VertxException: Thread blocked
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/jdk.internal.misc.Unsafe.park(Native Method)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1864)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3465)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3436)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1898)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2072)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.response(ExecutionEngineJsonRpcMethod.java:115)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:44)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.TracedJsonRpcProcessor.process(TracedJsonRpcProcessor.java:41)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.TimedJsonRpcProcessor.process(TimedJsonRpcProcessor.java:45)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.AuthenticatedJsonRpcProcessor.process(AuthenticatedJsonRpcProcessor.java:51)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.handlers.AbstractJsonRpcExecutor.executeRequest(AbstractJsonRpcExecutor.java:87)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcObjectExecutor.execute(JsonRpcObjectExecutor.java:57)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$0(JsonRpcExecutorHandler.java:57)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler$$Lambda$1571/0x000077514365c8e8.accept(Unknown Source)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.util.Optional.ifPresentOrElse(Optional.java:196)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$2(JsonRpcExecutorHandler.java:54)
Mai 13 05:31:21 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler$$Lambda$970/0x0000775143502390.handle(Unknown Source)
Mai 13 05:31:21 B550M besu[1094]:         at app//io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)
Mai 13 05:31:21 B550M besu[1094]:         at app//io.vertx.ext.web.impl.BlockingHandlerDecorator$$Lambda$1569/0x0000775143657410.handle(Unknown Source)
Mai 13 05:31:21 B550M besu[1094]:         at app//io.vertx.core.impl.ContextBase.lambda$null$0(ContextBase.java:137)
Mai 13 05:31:21 B550M besu[1094]:         at app//io.vertx.core.impl.ContextBase$$Lambda$1105/0x00007751435675a8.handle(Unknown Source)
Mai 13 05:31:21 B550M besu[1094]:         at app//io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264)
Mai 13 05:31:21 B550M besu[1094]:         at app//io.vertx.core.impl.ContextBase.lambda$executeBlocking$1(ContextBase.java:135)
Mai 13 05:31:21 B550M besu[1094]:         at app//io.vertx.core.impl.ContextBase$$Lambda$1104/0x0000775143567380.run(Unknown Source)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
Mai 13 05:31:21 B550M besu[1094]:         at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Mai 13 05:31:21 B550M besu[1094]:         at java.base@17.0.10/java.lang.Thread.run(Thread.java:840)
Mai 13 05:31:27 B550M besu[1094]: 2024-05-13 05:31:27.066+02:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[vert.x-worker-thread-15,5,main] has been blocked for 66186 ms, time limit is 60000 ms
Mai 13 05:31:27 B550M besu[1094]: io.vertx.core.VertxException: Thread blocked
Mai 13 05:31:27 B550M besu[1094]:         at java.base@17.0.10/jdk.internal.misc.Unsafe.park(Native Method)
Mai 13 05:31:27 B550M besu[1094]:         at java.base@17.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)
Mai 13 05:31:27 B550M besu[1094]:         at java.base@17.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1864)
Mai 13 05:31:27 B550M besu[1094]:         at java.base@17.0.10/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3465)
Mai 13 05:31:27 B550M besu[1094]:         at java.base@17.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3436)
Mai 13 05:31:27 B550M besu[1094]:         at java.base@17.0.10/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1898)
Mai 13 05:31:27 B550M besu[1094]:         at java.base@17.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2072)
Mai 13 05:31:27 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.response(ExecutionEngineJsonRpcMethod.java:115)
Mai 13 05:31:27 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:44)
Mai 13 05:31:27 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.TracedJsonRpcProcessor.process(TracedJsonRpcProcessor.java:41)
Mai 13 05:31:27 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.TimedJsonRpcProcessor.process(TimedJsonRpcProcessor.java:45)
Mai 13 05:31:27 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.AuthenticatedJsonRpcProcessor.process(AuthenticatedJsonRpcProcessor.java:51)
Mai 13 05:31:27 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)
Mai 13 05:31:27 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.handlers.AbstractJsonRpcExecutor.executeRequest(AbstractJsonRpcExecutor.java:87)
Mai 13 05:31:27 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcObjectExecutor.execute(JsonRpcObjectExecutor.java:57)
Mai 13 05:31:27 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$0(JsonRpcExecutorHandler.java:57)
Mai 13 05:31:27 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler$$Lambda$1571/0x000077514365c8e8.accept(Unknown Source)
Mai 13 05:31:27 B550M besu[1094]:         at java.base@17.0.10/java.util.Optional.ifPresentOrElse(Optional.java:196)
Mai 13 05:31:27 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$2(JsonRpcExecutorHandler.java:54)
Mai 13 05:31:27 B550M besu[1094]:         at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler$$Lambda$970/0x0000775143502390.handle(Unknown Source)
Mai 13 05:31:27 B550M besu[1094]:         at app//io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)
Mai 13 05:31:27 B550M besu[1094]:         at app//io.vertx.ext.web.impl.BlockingHandlerDecorator$$Lambda$1569/0x0000775143657410.handle(Unknown Source)
Mai 13 05:31:27 B550M besu[1094]:         at app//io.vertx.core.impl.ContextBase.lambda$null$0(ContextBase.java:137)
Mai 13 05:31:27 B550M besu[1094]:         at app//io.vertx.core.impl.ContextBase$$Lambda$1105/0x00007751435675a8.handle(Unknown Source)
Mai 13 05:31:27 B550M besu[1094]:         at app//io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264)
Mai 13 05:31:27 B550M besu[1094]:         at app//io.vertx.core.impl.ContextBase.lambda$executeBlocking$1(ContextBase.java:135)
Mai 13 05:31:27 B550M besu[1094]:         at app//io.vertx.core.impl.ContextBase$$Lambda$1104/0x0000775143567380.run(Unknown Source)
Mai 13 05:31:27 B550M besu[1094]:         at java.

@RogerHKW
Copy link
Author

RogerHKW commented May 16, 2024 •
edited
Loading

Screenshot_20240516-230535~2.png

As you can see, the daily death of Besu stopped since I installed your last version.

So this PR that you mentioned might have fixed the bug!

The only problem left might be the number of peers.

@non-fungible-nelson
Copy link
Contributor

Hi there - thanks for the feedback! Closing for now as of the latest release 24.5.1 fixes. We will continue to work on peering over the next few releases as well.

@ahamlat
Copy link
Contributor

ahamlat commented Jun 4, 2024

Related fix : #7076

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ahamlat ahamlat

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@macfarla @ahamlat @non-fungible-nelson @RogerHKW