-
Notifications
You must be signed in to change notification settings - Fork 82
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bitcoind connection lost during syncing #116
Comments
Hmm. No this is not normal nor is it ideal. What is the setup? I presume bitcoind is local on the same box as Fulcrum? That is, this is via localhost connection? What bitcoind version? Pasting (a possibly redacted) version of the fulcrum.conf here would help ... |
Very strange. It's as if bitcoind is dropping conn itself. You can ignore this since Fulcrum reconnects of course and proceeds anyway -- however I am curious WHY it's happening. If you can reliably reproduce it or if it happens often can you try running Fulcrum with Also is this on HDD or SSD? What hardware? RPi? x86 desktop PC? What OS? Etc, etc, etc... |
Im using a HDD / Ubuntu server 22.04 / VM. Would it be helpful if I spun up a dedicated server just for Fulcrum and point it to bitcoind? bitcoind is also running on Ubuntu server 22.04. |
I was told that when syncing was completed fulcrum_db should be at least 105GB. Im currently at 55.2% and fulcrum_db is only at 20GB. @SOVEREIGN1: /fulcrum_db$ du -h 2.3G ./txhash2txnum |
Hmm. I wonder if bitcoind is taking so long to read and respond to a backlog of block requests that Fulcrum times out (because HDD)? Please run with
To be clear Fulcrum and bitcoind ideally should run on the same machine. As to whether or not you want to spin up a dedicated server -- that's up to you. I run my Fulcrum/bitcoind combo on a server I use for other things without any problems. In fact, I run bitcoind + Fulcrum for 3 coins: BTC, BCH, and LTC, and for each coin I run mainnet & testnet. That's 6 instances of bitcoind and 6 instances of Fulcrum on the same box, without issue.
That "55.2%" is by block height. You are 55.2% into the blocks by height. Later blocks on the chain are full, so data consumption will increase as you get to the later blocks. You should probably reach the 100GB mark or beyond... |
First i just want to say thank you for taking the time to help troubleshoot this with me...your work is much appreciated.
2022-05-31T02:30:19Z UpdateTip: new best=000000000000000000073ef06543252dce1cc24c9e31a6d6848d892ffc824f71 height=738656 version=0x20000000 log2_work=93.549238 tx=737714334 date='2022-05-31T02:30:02Z' progress=1.000000 cache=51.2MiB(325658txo)
2022-05-31T05:32:41Z WARNING: request rejected because http work queue depth exceeded, it can be increased with the -rpcworkqueue= setting I think your explanation regarding the fact that I am using HDD is causing the problem. I will try using the (--bd-timeout 300) command, but Im not sure how to use it being that I learned to start fulcrum by using the systemctl command via the tutorial. If the timeout command does not work I will have to purchase a SSD and rebuild everything, which is no problem. Again thanks for all of your help. |
Hi, can confirm this bug as well, just wanted to report it. Fulcrum loses connection sometimes, wonder why. It sometimes takes up to 10+ minutes to reconnect, which slows down the sync quite significantly when it happens several times a day. Can it be caused by settings in fulcrum/bitcoin core? However in btccore I use default settings. These are my changes in fulcrum file - should optimize syncing for raspberry: bitcoind_timeout = 300 Error: |
@Techeavy33 Ok so a few things you asked about, I will try to get to them all:
@DanielPX1 . This program was designed actually for higher performance hardware... . the fact that it works on Rpi as well as it does is fortunate but not a guarantee always. Perhaps you can try some of the above things too? |
Hmm. An option I have I guess is to "guesstimate" reasonable values for systems on HDD and Rpi (at least for Fulcrum). But if bitcoind rpcworkqueue can't keep up that is troublesome. (Although one thing would be to throttle it better to not ask for so many blocks in a row if HDD.. hmm). |
Hmm I did find some deficiencies in Fulcrum's code/logic when bitcoind is slow to respond. I'm working on a fix. I do think this is a fixable problem in Fulcum.. working on it. In the meantime yeah |
Hi @cculianu. The changes in the config file didn't work for me. I see you added a bug and are working on this. I will just wait to try and install fulcrum again after the bug fixes after your investigation reveals anything. Thank you again. Hey if you need me to run any tests or provide you with any feedback whatsoever I am more then happy to help. Just FYI i am still learning a lot with everything and Im am not an expert, but I would love you help you in anyway that I can to further your project. Thanks again man. |
Hey yeah since you are having the problem -- when I have the issue resolved I will let you know and some testing would be appreciaed! |
Yeah no problem man. Just let me know, and Im willing to help out in anyway I can. Thanks! |
…r on HDD STILL A WIP AAAHHH!! --- This commit should fix #116. This commit also addresses the poor performance seen in the recent Bitcoin Verde BCHN Scaling report. It also makes running Fulcrum for BCH ScaleNet more performant with default settings. What was fixed and how: - There was a slight bug with how the "ping timer" mechanism to BitcoinD worked in that it was too likely to conclude the connection was stale in some cases when doing initial synch, if the timer fired at the wrong times. This could lead to spurious disconnect/reconnect cycles sometimes during synch. - This has been addressed.by making the pingTimer use a "CoarseTimer" (as opposed to a "VeryCoarseTimer") and also adding some +/- 100 msec fuzz to when the pings should fire and when we conclude bitcoind is "stale". We made the stale threshold also be 3 ping cycles instead of 2. - Additionally, there was a bug when downloading very large blocks (hundreds of MB) or when bitcoind was otherwise slow to return blocks for whatever reason (such as if using HDD). In some cases requests for blocks would "time out" after 30 seconds during initial synch, causing a disconnect and reconnect cycle to occur, slowing down synch tremendously. - This has been addressed by anticiapting that requests for later blocks in a prefetch burst of 100 or 1000 blocks should have timeouts longer than 30 seconds. In addition this commit has some additional debug prints (off at compile time but can be re-enabled during dev testing) to be able to troubleshoot the above in the future, if need be.
… or with huge blocks + misc fixes (#120) This commit should fix issue #116. It also addresses the Bitcoin Verde Scaling Report dated May 2022 which revealed that Fulcrum has some issues downloading very large blocks (256MB+) in a series. The two issues are related, and are basically because the following could happen on a slow HDD system or with a bitcoind that is serving up very large blocks:
Hi @Techeavy33 and/or @DanielPX1 -- are either of you able to test latest master and tell me if the situation has improved? Let me know if you want me to build a binary for you for Linux64 or Arm64 and I can put it up somewhere if you can't build latest master. I think the latest commit I just pushed should fix this. |
Update: The new release 1.7.0 has the fixes Let me know if it performs better... |
ok will do. I'll have to work on it this weekend work is crazy this week. Thanks for the update will be in touch this weekend. |
Yeah no rush. I tested it extensively -- raping a slow HDD system (compiling in another terminal tab while synching). It seemed to hold up well and never a dropped conn. Without the fix it definitely would drop conn and restart the synch. I am hoping my fix is enough for all systems. Technically if bitcoind takes longer than 10 minutes to return a |
Sorry for not answering. I plan into playing with fulcrum more so I might test it later, however now I am fully synced and trying to use it with different wallets - so far successfully:) |
Ok no worries. We are all volunteers here! It's good to hear you are happy with it and that it's working like a charm. I'm going to go ahead and close this in the interests of keeping the open issues list lighter. Feel free to write in here if you do have further trouble guys... I am pretty sure the issue is solved in my testing I was both able to reproduce it by simulating a slowdown problem on an HDD (I had the system overloaded with lots of I/O) while doing a synch, and it definitely dropped conn when bitcoind took "long-ish" to respond. The new fixes definitely address this behavior/bug. Closing.. |
Hi, sorry if I'm bringing this back from the dead / closed list. I hope it's just user error but I'm seeing this issue. Again I'm running on a RPi4, raspiblitz (v1.8.0), bitcoin v23.0.0, 8GB RAM, 1Tb SSD (powered from raspi)
● fulcrum.service - Fulcrum after 5 days I'm at 48.3%, here's the end of the journal, showing the most recent occurrence of bitcoind request time out. checking on memory I found that someone has a successful sync on much the same RPi set up as mine, except they powered the SSD externally to the RPi. See following link for details; raspiblitz/raspiblitz#2924 (comment) I'll see if I can find a USB data / power splitting cable to do the same & will report back what I find out (if/when I find what I'm looking for). |
I think you're running out of RAM and the RPi is swapping like mad, thus freezing the system.
|
Thanks for your quick support, I'll give that a go (have now read up how to set ENV variables). Might be a bit of delay, currently making negative progress, can't even get the RPi to reboot (giving it 10mins - now I've used cli to attempt to stop bitcoind - and will then pull power). |
Yeah not sure about the power thing since I'm no RPi power user but I can imagine if the SSD draws too much power during periods of heavy activity it might fail. No idea to be honest .. but sounds plausible at least from 10,000ft view. Good luck! |
At the risk of tempting fate... |
Great. Ok yeah I think results will be better now. |
Below is a comment I received.
"I keep getting bitcoind connection lost during syncing. Is this normal, and should I continue to let this run? It will now only process a height once every hour or so. Has anyone else had this issue, and if so what do you do in this situation? This is my 5th attempt at installing Fulcrum. Any help would be greatly appreciated."
The text was updated successfully, but these errors were encountered: