Intermittent connections to bitcoind

[RequestPrivacy]

Note (seemingly) similar issues here #160 (imho not related) or #116 (questionable).

Description

My fulcrum seems to loose its connection to bitcoind quite often:

Jul 02 21:04:57 debian Fulcrum[780991]: [2023-07-02 21:04:57.054] <Controller> 53961 mempool txs involving 236829 addresses
Jul 02 21:04:57 debian Fulcrum[780991]: [2023-07-02 21:04:57.200] <SynchMempool> (Debug) getrawmempool: got reply with 53968 items, 0 ignored, 0 dropped, 7 new
Jul 02 21:04:57 debian Fulcrum[780991]: [2023-07-02 21:04:57.220] <SynchMempool> (Debug) downloaded 7 txs (failed: 0, ignored: 0), elapsed so far: 0.609 secs
Jul 02 21:04:57 debian Fulcrum[780991]: [2023-07-02 21:04:57.224] <SynchMempool> (Debug) Mempool: pre-cache of 4 confirmed spends from db took 3.641 msec
Jul 02 21:04:57 debian Fulcrum[780991]: [2023-07-02 21:04:57.224] <SynchMempool> (Debug) 53968 mempool txs involving 236844 addresses
Jul 02 21:04:59 debian Fulcrum[780991]: [2023-07-02 21:04:59.678] <SynchMempool> (Debug) getrawmempool: got reply with 53976 items, 0 ignored, 0 dropped, 8 new
Jul 02 21:04:59 debian Fulcrum[780991]: [2023-07-02 21:04:59.700] <SynchMempool> (Debug) downloaded 8 txs (failed: 0, ignored: 0), elapsed so far: 0.573 secs
Jul 02 21:04:59 debian Fulcrum[780991]: [2023-07-02 21:04:59.706] <SynchMempool> (Debug) Mempool: pre-cache of 12 confirmed spends from db took 5.248 msec
Jul 02 21:04:59 debian Fulcrum[780991]: [2023-07-02 21:04:59.706] <SynchMempool> (Debug) 53976 mempool txs involving 236863 addresses

## First indication of problem
Jul 02 21:05:30 debian Fulcrum[780991]: [2023-07-02 21:05:30.081] <BitcoinD.3> (Debug) Stale connection, reconnecting.
Jul 02 21:05:30 debian Fulcrum[780991]: [2023-07-02 21:05:30.086] <BitcoinD.3> (Debug) TCP BitcoinD.3 (id: 4) 127.0.0.1:8332 socket state: 0
Jul 02 21:05:30 debian Fulcrum[780991]: [2023-07-02 21:05:30.087] <BitcoinD.3> (Debug) TCP BitcoinD.3 (id: 4) 127.0.0.1:8332 socket disconnected
Jul 02 21:05:30 debian Fulcrum[780991]: [2023-07-02 21:05:30.087] <BitcoinD.3> Lost connection to bitcoind, will retry every 5 seconds ...
Jul 02 21:05:30 debian Fulcrum[780991]: [2023-07-02 21:05:30.087] <Task.GetChainInfo> 72787: FAIL: bitcoind connection lost
Jul 02 21:05:30 debian Fulcrum[780991]: [2023-07-02 21:05:30.088] <Controller> Failed to synch blocks and/or mempool
Jul 02 21:05:30 debian Fulcrum[780991]: [2023-07-02 21:05:30.103] <BitcoinD.3> (Debug) TCP BitcoinD.3 (id: 4)  socket state: 1
Jul 02 21:05:30 debian Fulcrum[780991]: [2023-07-02 21:05:30.103] <BitcoinD.3> (Debug) TCP BitcoinD.3 (id: 4)  socket state: 2
Jul 02 21:05:30 debian Fulcrum[780991]: [2023-07-02 21:05:30.104] <BitcoinD.3> (Debug) TCP BitcoinD.3 (id: 4) 127.0.0.1:8332 socket state: 3
Jul 02 21:05:30 debian Fulcrum[780991]: [2023-07-02 21:05:30.104] <BitcoinD.3> (Debug) on_connected 4
Jul 02 21:05:34 debian Fulcrum[780991]: [2023-07-02 21:05:34.379] <Controller> Block height 796897, downloading new blocks ...
Jul 02 21:05:34 debian Fulcrum[780991]: [2023-07-02 21:05:34.382] <Controller> (Debug) Task.DL 796897 -> 796897 starting thread
Jul 02 21:05:34 debian Fulcrum[780991]: [2023-07-02 21:05:34.384] <BitcoinD.3> (Debug) on_inBlockDownload: 0 -> 1
Jul 02 21:05:34 debian Fulcrum[780991]: [2023-07-02 21:05:34.384] <BitcoinD.1> (Debug) on_inBlockDownload: 0 -> 1
Jul 02 21:05:34 debian Fulcrum[780991]: [2023-07-02 21:05:34.384] <BitcoinD.2> (Debug) on_inBlockDownload: 0 -> 1
Jul 02 21:05:34 debian Fulcrum[780991]: [2023-07-02 21:05:34.424] <ZMQ Notifier (hashblock)> (Debug) topic: "hashblock", parts: 3, bytes: 45
Jul 02 21:05:34 debian Fulcrum[780991]: [2023-07-02 21:05:34.424] <Controller> (Debug) got zmq hashblock notification: hashblock, 000000000000000000021d8d298880319efdb5098f7b71a66d99fb30049fe000, 79
Jul 02 21:05:34 debian Fulcrum[780991]: [2023-07-02 21:05:34.770] <Controller> (Debug) rmTxsInHashXTxs: removed 4568 entries in 27.856 msec sorted 62 entries (sort time: 0.034 msec)
Jul 02 21:05:34 debian Fulcrum[780991]: [2023-07-02 21:05:34.772] <Controller> (Debug) addBlock: removed 928 txs from mempool involving 4341 addresses in 77.275 msec
Jul 02 21:05:38 debian Fulcrum[780991]: [2023-07-02 21:05:38.461] <Controller> (Debug) Saved V2 undo for block 796897, 943 transactions involving 9934 scripthashes, in 32.73 msec.
Jul 02 21:05:38 debian Fulcrum[780991]: [2023-07-02 21:05:38.492] <Controller> (Debug) Got all blocks from: Task.DL 796897 -> 796897 blockCt: 1 nTx,nInp,nOutp: 943,9666,2661
Jul 02 21:05:38 debian Fulcrum[780991]: [2023-07-02 21:05:38.492] <Controller> (Debug) Task.DL 796897 -> 796897 thread is running, joining thread
Jul 02 21:05:38 debian Fulcrum[780991]: [2023-07-02 21:05:38.492] <Controller> (Debug) Task.DL 796897 -> 796897 cleaned up 7 signal/slot connections
Jul 02 21:05:38 debian Fulcrum[780991]: [2023-07-02 21:05:38.492] <Controller> Processed 1 new block with 943 txs (9666 inputs, 2661 outputs, 9934 addresses), verified ok.
Jul 02 21:05:38 debian Fulcrum[780991]: [2023-07-02 21:05:38.492] <Controller> (Debug) ~CtlTask (Task.DL 796897 -> 796897)
Jul 02 21:05:38 debian Fulcrum[780991]: [2023-07-02 21:05:38.493] <BitcoinD.2> (Debug) on_inBlockDownload: 1 -> 0
Jul 02 21:05:38 debian Fulcrum[780991]: [2023-07-02 21:05:38.493] <BitcoinD.1> (Debug) on_inBlockDownload: 1 -> 0
Jul 02 21:05:38 debian Fulcrum[780991]: [2023-07-02 21:05:38.493] <BitcoinD.3> (Debug) on_inBlockDownload: 1 -> 0
Jul 02 21:05:38 debian Fulcrum[780991]: [2023-07-02 21:05:38.495] <Controller> Block height 796897, up-to-date
Jul 02 21:05:39 debian Fulcrum[780991]: [2023-07-02 21:05:39.107] <SynchMempool> (Debug) getrawmempool: got reply with 53211 items, 0 ignored, 0 dropped, 163 new
Jul 02 21:05:39 debian Fulcrum[780991]: [2023-07-02 21:05:39.410] <SynchMempool> (Debug) downloaded 163 txs (failed: 0, ignored: 0), elapsed so far: 0.914 secs
Jul 02 21:05:39 debian Fulcrum[780991]: [2023-07-02 21:05:39.479] <SynchMempool> (Debug) Mempool: pre-cache of 318 confirmed spends from db took 68.393 msec
Jul 02 21:05:39 debian Fulcrum[780991]: [2023-07-02 21:05:39.483] <SynchMempool> (Debug) 53211 mempool txs involving 233307 addresses (exclusive lock held for 4.346 msec)
Jul 02 21:05:42 debian Fulcrum[780991]: [2023-07-02 21:05:42.076] <SynchMempool> (Debug) getrawmempool: got reply with 53220 items, 0 ignored, 0 dropped, 9 new
Jul 02 21:05:42 debian Fulcrum[780991]: [2023-07-02 21:05:42.100] <SynchMempool> (Debug) downloaded 9 txs (failed: 0, ignored: 0), elapsed so far: 0.543 secs
Jul 02 21:05:42 debian Fulcrum[780991]: [2023-07-02 21:05:42.109] <SynchMempool> (Debug) Mempool: pre-cache of 18 confirmed spends from db took 8.319 msec
Jul 02 21:05:42 debian Fulcrum[780991]: [2023-07-02 21:05:42.109] <SynchMempool> (Debug) 53220 mempool txs involving 233341 addresses
Jul 02 21:05:44 debian Fulcrum[780991]: [2023-07-02 21:05:44.623] <SynchMempool> (Debug) getrawmempool: got reply with 53229 items, 0 ignored, 0 dropped, 9 new

although fulcrum always seems to re-connect and continues to work thereafter, my wallet (Sparrow) looses its connection immediately and has problems to reliably re-connect to fulcrum. According to its dev, Sparrow tries to re-connect in increasing time periods but not sure why it fails to establish a connection sometimes.

I filed an issue over there see especially this comment. I'm not sure if this is the same problem though.

Either way, the question is if it's normal that fulcrum looses the connection so often?

System

RPi 4GB
bitcoind v25.0
fulcrum 1.7.0

conf-files with settings that might affect this issue (all else is set to default values)

## bitcoin.conf

dbcache=450
maxmempool=300
txindex=1
rpcworkqueue=16
rpcthreads=4
zmqpubhashblock=tcp://0.0.0.0:8433

## fulcrum.conf

bitcoind_timeout = 600.0
db_max_open_files = 200

[TinF0ilHat1]

Could this help? #96

So far, the RPi has not failed again, so I am hoping the autosuspend change was the trick. For the record, the change I made was to disable autosuspend in the RPI's internal USB hub that the SSD is connected to using echo 'on' > '/sys/bus/usb/devices/usb2/power/control'. It was previously set to auto.

[cculianu]

LOL. This is ridiculous that anybody would auto-configure a linux distro to do that and consider that "useful". ha ha