Besu v23.7.1 does not synchronize with goerli

[chichi13]

Description

I wanted to sync Goerli network with Besu v23.7.1, unfortunately the sync blocked 3 times on the block 0x928201 (9601537).

Steps to Reproduce (Bug)

1. Use v23.7.1
2. Configuration used:

ExecStart=/home/ethereum/bin/besu/bin/besu \
  --network=goerli \
  --logging=INFO \
  --host-allowlist=* \
  --metrics-enabled=true \
  --metrics-port=6061 \
  --metrics-host=0.0.0.0 \
  --rpc-http-enabled=true \
  --rpc-http-port=8545 \
  --rpc-ws-enabled=True \
  --rpc-ws-host=localhost \
  --rpc-ws-port=8554 \
  --max-peers=10 \
  --sync-mode=X_CHECKPOINT \
  --data-storage-format=BONSAI \
  --data-path=/home/ethereum/var/lib/besu \
  --Xplugin-rocksdb-high-spec-enabled \
  --engine-jwt-secret=/home/ethereum/secrets/jwtsecret \
  --api-gas-price-blocks=100 \
  --api-gas-price-percentile=99 \
  --min-gas-price=1

3. Sync the node

Expected behavior: Node syncing well

Actual behavior: Sync blocked at height 9601537 without any error.

Frequency: Everytime with v23.7.1, tried with, X_CHECKPOINT and X_SNAP.

Logs (if a bug)

• At the begining, no error:

We can see, Besu will download the pivot block 9601537 (Downloading world state from peers for pivot block 9601537).

• After a few minutes, we have a lot of DNS Errors:

• After ~6h of syncing, we have no more info in the logs:

And the RPC is not synced, the RPC return the same block height:

❯ curl -sX POST -H "Content-Type: application/json" https://***** --data '{"jsonrpc": "2.0", "id": 42, "method": "eth_blockNumber", "params": []}'
{"jsonrpc":"2.0","id":42,"result":"0x928201"}%

Versions (Add all that apply)

• Software version: v23.7.1
• Java version: 17.0.7
• OS Name & Version:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.6 LTS"
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

• Kernel Version:

Linux devhub-rpc-server-02 5.4.0-153-generic #170-Ubuntu SMP Fri Jun 16 13:43:31 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

• Cloud VM, type, size: Hetzner Bare Metal
• Consensus Client & Version if using Proof of Stake: Teku v23.6.1

Additional Information (Add any of the following or anything else that may be relevant)

• System info - 6 CPU / 64 Go RAM / 2 To SSD NVMe

---

I'd also like to point out that version 23.7.1 is marked as a release on Github. However in the Besu version 23.7.1-dev-2bced4eb. What is the truth?

[non-fungible-nelson]

Hi there - thanks for the report. Can you provide logs from Teku as well?

[jframe]

@matkt Any thoughts on this issue? Do you think your PR #5838 will help with this?

[chichi13]

Hi!

Here are some logs from Teku:

(same logs at an other time)

[matkt]

did you do a checkpoint sync with teku ? I see 8137 remaining blocks. if your CL is not close to the head the besu sync will not work because we need to have a recent pivot block

[ahamlat]

Trying to reproduce this issue, I was able to sync Besu on Goerli with no issues with version 23.7.1.
@chichi13 What is the status from you side and had you a chance to test Karim's suggestion ?

[non-fungible-nelson]

close due to CL problem --> try a fresh checkpoint sync with Teku and see if this is able to resolve the issue.

[chichi13]

Hey,

Sorry I didn't see your messages.

I think the problem came from the CL not being able to resync, but I haven't repeated the error since.

When I upgraded Besu to 23.4.4 everything worked normally again.