Keeps resyncing after restart

[kiss81]

I have a synced besu instance and started the nimbus beacon node with a trusted node sync (back fill later). It starts validating withing 10 minutes but then starts the back fill for like 8 hours.
After like 8 hours it finished the sync. (nimbus data dir was 84gb).
Then I restarted the machine and now it looks like it's starting the back fill all over again?
The back fill is faster now (like half an hour), but this wasn't the behavior before.

un 28 10:12:11 ethVal2 nimbus_beacon_node[182]: INF 2023-06-28 10:12:11.000+02:00 Slot start topics="beacnde" slot=6759659 epoch=211239 sync=synced peers=27 head=89496b0e:6759643 finalized=211236:6b1f10c3 delay=668us839ns Jun 28 10:12:11 ethVal2 nimbus_beacon_node[182]: INF 2023-06-28 10:12:11.051+02:00 Eth1 sync progress topics="elmon" blockNumber=11630899 depositsProcessed=73849 ....

besu 23.4.1
nimbus eth2 23.6.1

running this in a proxmox debian lxc container (20gb ram / 6gb swap)
I checked the startup options, but nothing related. (so it's using the default history prune option now I guess)

Is this normal behavior of the history prune now?

[kiss81]

Update:
I started the process all over:

sudo -u nimbus rm -rf /var/lib/nimbus/db
sudo -u nimbus /usr/bin/nimbus_beacon_node trustedNodeSync --network=mainnet  --trusted-node-url=https://checkpointz.pietjepuk.net --data-dir=/var/lib/nimbus --network=mainnet --backfill=false

then start the node with systemd

After it does the backfill it seems to be ok for some minutes and then it seems to start syncing again:

Jun 29 11:52:46 ethVal2 nimbus_beacon_node[182]: INF 2023-06-29 11:52:46.110+02:00 Eth1 sync progress                         topics="elmon" blockNumber=17443979 depositsProcessed=794867
Jun 29 11:52:46 ethVal2 nimbus_beacon_node[182]: INF 2023-06-29 11:52:46.419+02:00 Eth1 sync progress                         topics="elmon" blockNumber=17444977 depositsProcessed=794978
Jun 29 11:52:46 ethVal2 nimbus_beacon_node[182]: INF 2023-06-29 11:52:46.620+02:00 Eth1 sync progress    

It get's in sync and a few minutes later all starts over again...

[arnetheduck]

Backfilling is the process of downloading historical consensus blocks - this is part of the checkpoint sync - you can see that it completed by looking at the sync=synced part of Slot start.

Eth1 sync is the process of downloading deposits from your execution client - this is a separate sync needed for block production and it depends on your execution client being in sync - it's normal that it happens a bit later - as you can see from the blockNumber, it keeps making progress meaning everything here is working as expected.

[kiss81]

@arnetheduck thank you for clarifying the difference between the backfill and downloading the deposits. Something is wrong with the latter. Please have a look at #5145