No buffer space available

[estensen]

Mainnet
v22.10.3
Checkpoint sync
Bonsai
Paired with Lighthouse

Re-syncing from scratch after experiencing Internal error, backward sync completed but failed to import finalized block
Also getting a lot of Best peer has chain height below pivotBlock height hangs

trace: io.netty.channel.unix.Errors$NativeIoException: sendToAddress(..) failed: No buffer space available
2023-02-07 22:25:59.488+01:00 | vert.x-eventloop-thread-1 | WARN  | VertxPeerDiscoveryAgent | Sending to peer DiscoveryPeer{status=bonding, enode=enode://ea6b729b46180aba4b968327f3f4cc25c8780381a6169560f9e4108a7e4e76d3592c71eb65390fdf3dc1b7e09ef7bba781d9a47fbe0ddc7f5cd4fe168e9aa7c1@62.210.122.19:59998, firstDiscovered=1675805159488, lastContacted=0, lastSeen=1675805159488} failed, native error code -55, packet: 0x553c7a063aa9a90dd5d0243034c7eabf338c4f977a167e15f1f81c1557bedfcf36ad3022c1409d39009d14a24aa407c0a29c05e2c40480882009c27670c88c0956a56f7aeb9a58038ff28704baf4a8b532d47bbbf3dcad299ded384829e837540001df05cb847f00000182765f82765fcb843ed27a1382ea5e82ea5e8463e2c22303, stacktrace: io.netty.channel.unix.Errors$NativeIoException: sendToAddress(..) failed: No buffer space available

[macfarla]

This is a pretty low-level exception, I've never seen this with Besu before.
This indicates it might be related to resource limitation or exhaustion? https://stackoverflow.com/questions/6068423/java-net-socketexception-no-buffer-space-available-maximum-connections-reached
Can you give more info about your OS etc? besu config?

[estensen]

I do run a pretty minimalist node. A M1 Mac Mini with 8GB RAM which is shared between Besu and Lighthouse. I accept that that might cause issues.

JAVA_OPTS=-Xmx3g besu-22.7.4/bin/besu
  --network=mainnet --data-storage-format=bonsai
  --sync-mode=X_CHECKPOINT
  --data-path=/Volumes/WD_BLACK\ SN750\ 2TB/besu
  --engine-jwt-secret=/var/lib/jwtsecret/jwt.hex
  --nat-method=auto
  --p2p-host 
  --p2p-interface
  --metrics-enabled

CPU is 90% idle and I'm using 1-2GB swap now while re-syncing, which is pretty similar to when operational.

Are you running any setup similar to this testing new releases?

[shemnon]

I run a similar setup just fine except on a 16 GB M1 mac mini. I expect the issue is with the available RAM.

[ahamlat]

I did some tests on 8 GiB VMs with 4 cores, I didn't notice such error, but in your case, it seems to be definitely related to available RAM (used by the OS on system cache and IO buffers).
With 8 GiB machine/VM, Besu should work fine on non RPC node but performances will be affected, because Besu is IO Bound especially during sync. With almost no system cache and probably some swapping activity, Block processing time will be affected. Furthermore, Besu can consume more than 3 GiB defined in -Xmx as its embedded database use a native memory.
I used to recommend for solo stakers 16 GiB RAM if they want to run the CL and the EL on the same machine and avoid performance degradation on block processing.