🐞 [Bug]:Farmerbot fails to start fully due to an RMB communication error.

[mahendravarmayadala93]

What happened?

Farm ID : 195

The client reported that his Nodes managed by Farmerbot did not shut down.

Upon reviewing the log file, we found that the farmerbot was not starting up due to the following error:

8:20AM DBG failed to read message error="websocket: close 1006 (abnormal closure): unexpected EOF"
8:20AM DBG connecting url=wss://relay.grid.tf

Some additional notes by @scottyeager:

I checked the log file. The core thing here is that the bot never fully starts up. That is indeed due to the failure of RMB communication associated with this error

We see on each attempt of the bot to start that it adds one node successfully using the same RMB relay before failing repeatedly on the second node. So there is some successful RMB communication happening

I also checked on the rate limiting implementation for RMB. It looks like it only drops messages with an error, it isn't supposed to drop connections entirely if the user tries to send too many messages

Log File :

farmerbot_16enuun.log

which network/s did you face the problem on?

Main

Twin ID/s

No response

Version

No response

Node ID/s

626, 548, 547(Offline currently) - 3038(Online)

Farm ID/s

195

Contract ID/s

No response

Relevant log output

Config File

farm_id: 195
never_shutdown_nodes:
  - 626
power:
  periodic_wake_up_start: 09:00AM
  periodic_wake_up_limit: 3

[rawdaGastan]

• You should make sure the nodes are healthy and working before adding them to the farmerbot.
• You can try to use --continue-power-on-error flag

[TullysInc]

@rawdaGastan : There is a more recent report from a second farmer (farmID_250), about the same error lines in the logs he obtained.

farmer@bot:~/farmerbot$ tail -n 50 farmerbot.log
2024/11/18 14:08:47 Connecting to wss://tfchain.grid.tf:443...
2:08PM INF starting peer session=farmerbot-rpc-250 twin=826
2:08PM DBG connecting url=wss://tfchain.grid.tf/ws
2024/11/18 14:08:49 Connecting to wss://tfchain.grid.tf/ws...
2:08PM DBG connecting url=wss://relay.grid.tf
2:08PM DBG Add node nodeID=3736
2:08PM DBG failed to read message error="websocket: close 1006 (abnormal closure ): unexpected EOF"
2:08PM DBG connecting url=wss://relay.grid.tf
2:08PM DBG Add node nodeID=4746
2:08PM DBG failed to read message error="websocket: close 1006 (abnormal closure ): unexpected EOF"

All nodes included in this config are currently up in the dashboard, so we can possibly rule out the suspicion of the nodes being unhealthy before being added to the farmerbot. Also, the --continue-power-on-error is already included in the script that was used to set up.

farm_id: 250
included_nodes:

• 565
• 3736
• 6026
• 4746
• 4961
• 5262
• 4458
• 5985
• 3763
  never_shutdown_nodes:
• 565
  power:
  periodic_wake_up_start: 09:00AM