Fulcrum: the most performant Electrum Server implementation

[openoms]

See the udpatedd article from Craig Raw: https://www.sparrowwallet.com/docs/server-performance.html

https://twitter.com/craigraw/status/1488470446211321856?s=19

Fulcrum: https://github.com/cculianu/Fulcrum

arm64 binaries are published for the RPi.

Proceeding to test.

[openoms]

First sync took ~48h on an RPi4: https://twitter.com/openoms/status/1487099572694437892
Seems to be a viable option which could be offered next to / instead of Electrs.

A quick manual guide to install: https://github.com/openoms/bitcoin-tutorials/blob/master/fulcrum.md

[openoms]

Test this install script https://github.com/openoms/raspiblitz/blob/fulcrum/home.admin/config.scripts/bonus.fulcrum.sh:

# download
wget -O bonus.fulcrum.sh https://raw.githubusercontent.com/openoms/raspiblitz/fulcrum/home.admin/config.scripts/bonus.fulcrum.sh

# check
cat bonus.fulcrum.sh

# run with debug
bash -x bonus.fulcrum.sh

Looking to test:

• sync while txindex is being built
• sync without ZRAM
• 2GB RAM

TODO:

• menu
• status
• how to connect
• integrate with mempool
• integrate with btc-rpc-explorer
• ask to uninstall database

[openoms]

More experience: #2966 (comment)

• try to use the same SSL certificate as for electrs

[Sparrowhawk1337]

Hardware

• RPi4B
• 8GB RAM
• 1Tb SSD (powered from an expansion board not from RPi)
• SanDisk Extreme 32GB microSD Card (A2, C10, U3, V30)

Software

• Fresh install of RaspiBlitz v1.7.2
• Fulcrum from script https://openoms.github.io/bitcoin-tutorials/fulcrum.html

Run 1

• no ZRAM set
• fulcrum.conf using settings for 8Gb RAM
• fulcrum sync failed at 90.9%
• log file fulcrum_log_extract_1.txt

Run 2

• no ZRAM set
• fulcrum.conf using settings for 4Gb RAM
• fulcrum sync failed at 38.7%
• log file fulcrum_log_extract_2.txt

Run 3

• ZRAM set as described here [Research] ZRAM to use compressed swap in memory (increase performance when memory usage is high) #2905 (comment)
• fulcrum.conf using settings for 4Gb RAM
• currently still syncing successfully at 65.2% with this 'top' readout
  

[Sparrowhawk1337]

Run 3 above completed Fulcrum sync successfully after 4 days.
It responds much quicker than Electrs when using it from Sparrow Wallet.

[Sparrowhawk1337]

Re: sync while txindex is being built

I have tested this on hardware as above, and settings as in "Run 3".

No issues found.

[openoms]

v1.7.0 is out https://github.com/cculianu/Fulcrum/releases/tag/v1.7.0

[openoms]

The new, version is looking good on my 4GB blitz. No errors so far and synced halfway.

Unfortunately Fulcrum v1.7.0 corrupted it's database on an ungraceful shutdown so needed to reset. This was after 3 days getting to 54%.

I feel Fulcrum is still not usable on a 4GB RAM RPi so leaving the script CLI only.

[InfiniteQE]

I flashed to the latest raspiblitz-v1.8.0-2022-07-28.img.gz and restarted my The Bitcoin Machines Pi4B 8GB which is connected to a 1TB NVME in a USB 3.1 external enclosure.
Edit : TBM case is spacious with a heatsink and cooling slots. I used a USB powered 8cm PC fan for cooling during the fulcrum sync, which maintained the temp <40 deg C.

There is an incompatibility with The Bitcoin Machines LCD and the Raspiblitz HDMI display settings, requiring me to comment out the # Raspiblitz HDMI settings from /boot/config.txt (there is also a TBM/TBMLCD-v0.4/RaspiBlitzLCDV1_0_7 typo error in the filename Raspiblitz_Logo.png)

I then ran the /config.scripts/bonus.fulcrum.sh included in the 1.8.0 img

Monitoring with htop and sudo journalctl -fu fulcrum, fulcrum took about 63 hours to fully sync.

The install script may have partially broken Electrs inside blitz menus, as the option to show Electrs connection info doesn't show the Electrs tor address.

To start directly from laptop terminal use:
electrum --oneserver --server :50002:s

The Tor Hidden Service address for electrs is (see LCD for QR code):

To connect through TOR open the Tor Browser and start with the options:
electrum --oneserver --server :50002:s --proxy socks5:127.0.0.1:9150
error='missing second parameter - see help'

From my SSH shell, I was able to obtain the tor addresses with
$ su
# cd /mnt/hdd/tor
# cd electrs
# cat hostname
# cd ..
# cd fulcrum
# cat hostname
# ufw status
50001 (v6) ALLOW Anywhere (v6) # electrs TCP
50002 (v6) ALLOW Anywhere (v6) # electrs SSL
50021 (v6) ALLOW Anywhere (v6) # Fulcrum TCP
50022 (v6) ALLOW Anywhere (v6) # Fulcrum SSL

This then allowed my to successfully connect SparrowWallet to my fulcrum_tor.onion on port 50021 (SSL on 50022 also works)

I can't say I've yet noticed any exponential increase in performance compared to Sparrow + Electrs for 1-2 wallets, but certainly Sparrow + Electrs is as slow as watching paint dry when syncing multiple (6-10) open wallets.

Thank you for the hard work, no issues at all with the install script or sync with Pi4b 8GB, I would perhaps recommend adding fulcrum to the blitz menus, as well as ufw, and if the fulcrum install script breaks the electrs menu connecting info, that may need fixing too.

Thanks

[cricktor]

On RaspiBlitz 1.7.2 I deactivated electrs and installed Fulcrum 1.6 with the provided bonus.fulcrum.sh script.

When I update with fresh microSD card and RaspiBlitz 1.8.0 image, I wouldn't want electrs to start again (I assume it won't) and I hope the database files of my current Fulcrum stay untouched. I'm OK to setup Fulcrum on RaspiBlitz by hand if needed.

Will RaspiBlitz 1.8.0 recognize during first start that a previous Fulcrum was active?
This is what partly makes me still hesitate to upgrade to current RaspiBlitz 1.8.0. Any hints how to deal with this, please?

(I dropped electrs mainly because it's a pain with large address histories and it's considerably slower than Fulcrum, too.)

[nyxnor]

Just a report.

Summary: The Fulcrum upstream build is insecure and leads to untrustworthy binaries.

Build instructions are all around the readme, if you only read the build a static binary, you will miss the dependencies that was on the top of the readme
https://github.com/cculianu/Fulcrum#building-a-static-executable-for-linux

The script to be called is: https://github.com/cculianu/Fulcrum/blob/master/contrib/build/build.sh with the options linux BRANCH.

The problem is the build script.

It clones Fulcrum itself without signature verification:
https://github.com/cculianu/Fulcrum/blob/master/contrib/build/build.sh#L73, but the git tags are signed.

Then JeMalloc without signature verification:
https://github.com/cculianu/Fulcrum/blob/master/contrib/build/build.sh#L90, adding to the fact that release is not signed https://github.com/jemalloc/jemalloc/releases

Then RocksDB without any verification:
https://github.com/cculianu/Fulcrum/blob/master/contrib/build/build.sh#L112, adding to the fact that the release is not signed https://github.com/facebook/rocksdb/releases

Then it will build a docker container https://github.com/cculianu/Fulcrum/blob/master/contrib/build/build.sh#L134 from the provided dockerfile https://github.com/cculianu/Fulcrum/blob/master/contrib/build/linux/Dockerfile

The dockerfile instead of installing from debian repos libqt5core5a v5.15.2 which is compatible with Fulcrum, installs from the QT webside https://github.com/cculianu/Fulcrum/blob/master/contrib/build/linux/Dockerfile#L26 the version v.5.15.6 without any sort of verification, in this case the only verification it could do is sha256 https://download.qt.io/official_releases/qt/5.15/5.15.6/single/qt-everywhere-opensource-src-5.15.6.tar.xz.sha256 because also the releases are not signed.

Then it downloads the qt-patch without any verification at https://github.com/cculianu/Fulcrum/blob/master/contrib/build/linux/Dockerfile#L32

All of this means, if the build process upstream is not trustworthy, by only checking the fulcrum tag is signed with authenticity verification, this only means that the upstream maintainer did sign a tag or binary, it just means he signed an insecure build, not that the build is secure because it was signed.

Fulcrum may be themost performant electrum server implementation, but it is also one of the most insecure build process of all electrum server implementations.

• ElectRS (rust) fixes checksum for every crate https://github.com/romanz/electrs/blob/master/Cargo.lock and different than Fulcrum, has proper build instructions https://github.com/romanz/electrs/blob/master/doc/install.md
• Electrum Personal Server (Python) does not seem to have external dependencies and the releases are signed https://github.com/chris-belcher/electrum-personal-server/releases
• ElectrumX (python) does not do any sort of verification of requirements https://github.com/kyuupichan/electrumx/blob/master/requirements.txt and the build process recommended in their website also does not do any sort of verification https://github.com/bauerj/electrumx-installer and not updated either

---

Proposed solution:

• do a proper build script and provide the script upstream to Fulcrum. It is to the benefit of everybody that the upstream build is secure so the signed binaries becomes more trustworthy.

[cculianu]

Fulcrum author here. I can see about checking hashes on some of the stuff the build process downloads.

A comment though:

• The reason why it doesn't install ubuntu Qt is because the build aims to create a static binary, so it needs to download the Qt sources from the Qt website, in order to compile Qt with the -static flag. This is no different than what Bitcoin Core or Bitcoin Cash Node do in their static build scripts.
  • Admittedly, Bitcoin Core and/or BCHN do check the hashes of the downloaded tarballs

That being said, if the other stuff is an issue for you guys, I can tighten up the build script a fair bit.

[nyxnor]

That is exactly what this build process is for cculianu/Fulcrum#136 (comment)
Raspiblitz doesn't use docker and cloning a lot of repos when there is already suitable versions on Debian is unecessary.

The reason why it doesn't install ubuntu Qt is because the build aims to create a static binary, so it needs to download the Qt sources from the Qt website, in order to compile Qt with the -static flag.

If only the Qt has to be downloaded from external sources (not debian packages that already have signature verification), then it would be much better.

That being said, if the other stuff is an issue for you guys, I can tighten up the build script a fair bit.

Would be much welcomed.

[cculianu]

Again, it's doing a static build. C++ building is just like that. You need to compile libs specially to make them static. Bitcoin Core does similar in that it downloads tarballs, effectively. The static build for Fulcrum is designed to run on any (modern) Linux distro...

Anyway yeah I will tighten it up and check hashes -- although you know when you clone a repo and do a checkout of a specific commit hash, that is secure. You cannot forge commit hashes on git -- doing so is as expensive/impossible as mining a sha256 coin with an insane difficulty.

[Transisto]

I followed the "Automated setup" at https://openoms.github.io/bitcoin-tutorials/fulcrum.html#automated-setup

I ended up with a tor url and port at the end of the logs but It doesn't want to connect.

Is there something more to be done?

From the instructions there no delimitation for "Manual setup" so it's not clear where the Automated Setup section end.

[ExchangeREC]

I got a RPi 4 with 4GB memory and RaspiBlitz on it.

Do you think that will work or is it more likely not enough RAM for installing Fulcrum?

[openoms]

I got a RPi 4 with 4GB memory and RaspiBlitz on it.

Do you think that will work or is it more likely not enough RAM for installing Fulcrum?

answered on Tg https://t.me/raspiblitz/120553

[BitromNO]

Hi!
i dit a raspiblitz install in proxmox, started fulcrum script, it synced really nice and fast.. but this morning it stopped with a error.

Generating debug logs. Be patient, this should take maximum 2 minutes ..
Redacting ..
# redacting file: /var/cache/raspiblitz/debug.log

***************************************************************
* RASPIBLITZ DEBUG LOGS 
***************************************************************
blitzversion: 1.9.0rc2
chainnetwork: bitcoin / main
 08:43:03 up 10 min,  2 users,  load average: 1,81, 1,44, 0,81

*** SETUPPHASE / BOOTSTRAP ***
see logs: cat /home/admin/raspiblitz.log
setupPhase--> done
state--> ready

*** BACKGROUNDSERVICE ***
to monitor Background service call: sudo journalctl -f -u background

*** BLOCKCHAIN (MAINNET) SYSTEMD STATUS ***
● bitcoind.service - Bitcoin daemon
     Loaded: loaded (/etc/systemd/system/bitcoind.service; enabled; vendor preset: enabled)
     Active: activating (start-pre) since Sun 2023-04-09 08:43:03 CEST; 7ms ago
Cntrl PID: 163249 (blitz.systemd.s)
      Tasks: 3 (limit: 9505)
     Memory: 1.3M
        CPU: 6ms
     CGroup: /system.slice/bitcoind.service
             ├─163249 /bin/bash /home/admin/config.scripts/blitz.systemd.sh log blockchain STARTED
             ├─163251 bash /home/admin/_cache.sh increment system_count_start_blockchain
             └─163256 redis-cli set system_count_start_blockchain:ts 1681022583

april 09 08:43:03 raspiblitz systemd[1]: Starting Bitcoin daemon...

*** LAST BLOCKCHAIN (MAINNET) ERROR LOGS ***
sudo journalctl -u bitcoind -b --no-pager -n8
-- Journal begins at Sat 2023-04-08 21:52:54 CEST, ends at Sun 2023-04-09 08:43:03 CEST. --
april 09 08:43:02 raspiblitz systemd[1]: Started Bitcoin daemon.
april 09 08:43:02 raspiblitz systemd[1]: bitcoind.service: Main process exited, code=killed, status=6/ABRT
april 09 08:43:02 raspiblitz systemd[1]: bitcoind.service: Failed with result 'signal'.
april 09 08:43:02 raspiblitz systemd[1]: bitcoind.service: Consumed 5.200s CPU time.
april 09 08:43:03 raspiblitz systemd[1]: bitcoind.service: Scheduled restart job, restart counter is at 114.
april 09 08:43:03 raspiblitz systemd[1]: Stopped Bitcoin daemon.
april 09 08:43:03 raspiblitz systemd[1]: bitcoind.service: Consumed 5.200s CPU time.
april 09 08:43:03 raspiblitz systemd[1]: Starting Bitcoin daemon...

*** LAST BLOCKCHAIN (MAINNET) 20 INFO LOGS ***
sudo tail -n 20 /mnt/hdd/bitcoin/debug.log
2023-04-09T06:43:02Z init message: Starting network threads…
2023-04-09T06:43:02Z net thread start
2023-04-09T06:43:02Z dnsseed thread start
2023-04-09T06:43:02Z Waiting 300 seconds before querying DNS seeds.
2023-04-09T06:43:02Z msghand thread start
2023-04-09T06:43:02Z Leaving InitialBlockDownload (latching to false)
2023-04-09T06:43:02Z addcon thread start
2023-04-09T06:43:02Z init message: Done loading
2023-04-09T06:43:02Z [tor] Got service ID bcc557b332by3jxuimytndnkas2s6ddwca5vz3gw7wytaqjfk764rsqd, advertising service ***.onion:8333
2023-04-09T06:43:02Z AddLocal(***.onion:8333,4)
2023-04-09T06:43:02Z opencon thread start
2023-04-09T06:43:02Z Fatal LevelDB error: Corruption: block checksum mismatch: /mnt/hdd/bitcoin/indexes/txindex/059788.ldb
2023-04-09T06:43:02Z You can use -debug=leveldb to get more complete diagnostic messages
2023-04-09T06:43:02Z 

************************
EXCEPTION: 15dbwrapper_error       
Fatal LevelDB error: Corruption: block checksum mismatch: /mnt/hdd/bitcoin/indexes/txindex/059788.ldb       
bitcoin in txindex       

Now my node wont start up: blockchain progress: 181 restarts

edit: markdown format

[cculianu]

Looks like your txindex on the bitcoind side got corrupted. You may have to remove all files under /mnt/hdd/bitcoin/indexes/txindex/* and restart bitcoind. Txindex should just auto-rebuild itself (I think) if it's empty.

[openoms]

Just documenting the infamous lines I got again trying to sync a slightly overloaded RPi4 4GB:

Aug 30 17:51:16 raspberrypi Fulcrum[2354747]: [2023-08-30 17:51:16.867] FATAL: Caught exception: It appears that Fulcrum was forcefully killed in the middle of committing a block to the db. We cannot figure out where exactly in the update process Fulcrum was killed, so we cannot undo the inconsistent state caused by the unexpected shutdown. Sorry!
Aug 30 17:51:16 raspberrypi Fulcrum[2354747]: The database has been corrupted. Please delete the datadir and resynch to bitcoind.