Bug: Upgraded electrs 0.8.11 --> 0.9.0 and when start the core-dump

[mariusb]

Describe the bug
Have upgraded upgraded electrs from 0.8.11 to 0.9.0 and when starting via systecrl I get:
sudo systemctl status electrs.service
● electrs.service - Electrs
Loaded: loaded (/etc/systemd/system/electrs.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: core-dump) since Wed 2021-10-06 13:08:49 SAST; 5s ago
Process: 169369 ExecStart=/home/marius/electrs/target/release/electrs --db-dir ./db --electrum-rpc-addr=0.0.0.0:50001 (code=dumped, signal=ILL)
Main PID: 169369 (code=dumped, signal=ILL)

Oct 06 13:08:49 mbbtcnode systemd[1]: electrs.service: Failed with result 'core-dump'.

Electrs version
Latest: 0.9.0

To Reproduce
Upgraded using steps here: https://github.com/romanz/electrs/blob/master/doc/usage.md

Expected behavior
Electrs to be running

Configuration

electrs.toml

type error message here

Environment variables: ELECTRS_X=Y;...
Arguments: --foo

System running electrs
Linux running Ubuntu 20.04
bitcoin core v22
Rust version: stable-x86_64-unknown-linux-gnu unchanged - rustc 1.55.0 (c8dfcfe04 2021-09-06)

Electrum client
Client name (if not upstream desktop Electrum) and version:

Additional context
Add any other context about the problem here.

[romanz]

Many thanks for reporting this issue!
Could you please share the machine/device hardware spec?
Also, could you please run electrs from command-line and share the logs?

[romanz]

Not sure, but #193 or #226 may be relevant.

[Kixunil]

Maybe also share core dump? Unless you're using auth there should be no private information.

[mariusb]

Machine:
processor Intel(R) Core(TM)2 Duo CPU E7500 @ 2.93GHz
memory 64KiB L1 cache
memory 3MiB L2 cache
memory 4GiB System Memory
disk 1TB WDC WD10EZEX-60W (200GB free)
Have restarted machine and still core-dump
When running electrs from command line:
Config { network: Bitcoin, db_path: "./db/bitcoin", daemon_dir: "/home/marius/.bitcoin", daemon_auth: UserPass("marius", ""), daemon_rpc_addr: V4(127.0.0.1:8332), daemon_p2p_addr: V4(127.0.0.1:8333), electrum_rpc_addr: V4(127.0.0.1:50001), monitoring_addr: V4(127.0.0.1:4224), wait_duration: 10s, jsonrpc_timeout: 15s, index_batch_size: 10, index_lookup_limit: Some(200), auto_reindex: true, ignore_mempool: false, sync_once: false, server_banner: "Welcome to electrs 0.9.0 (Electrum Rust Server)!", args: [] }
Illegal instruction (core dumped)
Where do I find the logs?

[Kixunil]

Since you used systemd journalctl --no-pager -u electrs should be enough to get logs.

[mariusb]

From the above command:
Oct 06 13:45:16 mbbtcnode systemd[1]: Started Electrs.
Oct 06 13:45:16 mbbtcnode electrs[1616]: Config { network: Bitcoin, db_path: "./db/bitcoin", daemon_dir: "/home/marius/.bitcoin", daemon_auth: UserPass("marius", ""), daemon_rpc_addr: V4(127.0.0.1:8332), daemon_p2p_addr: V4(127.0.0.1:8333), electrum_rpc_addr: V4(0.0.0.0:50001), monitoring_addr: V4(127.0.0.1:4224), wait_duration: 10s, jsonrpc_timeout: 15s, index_batch_size: 10, index_lookup_limit: Some(200), auto_reindex: true, ignore_mempool: false, sync_once: false, server_banner: "Welcome to electrs 0.9.0 (Electrum Rust Server)!", args: [] }
Oct 06 13:45:16 mbbtcnode systemd[1]: electrs.service: Main process exited, code=dumped, status=4/ILL
Oct 06 13:45:16 mbbtcnode systemd[1]: electrs.service: Failed with result 'core-dump'.

[mariusb]

Had a look at #193 , do you think downgrading rocksdb to 0.12.2 will help?

[mariusb]

I have run electrs also with -vvv:
/home/marius/electrs/target/release/electrs -vvv --db-dir ./db --electrum-rpc-addr=0.0.0.0:50001
Config { network: Bitcoin, db_path: "./db/bitcoin", daemon_dir: "/home/marius/.bitcoin", daemon_auth: UserPass("marius", ""), daemon_rpc_addr: V4(127.0.0.1:8332), daemon_p2p_addr: V4(127.0.0.1:8333), electrum_rpc_addr: V4(0.0.0.0:50001), monitoring_addr: V4(127.0.0.1:4224), wait_duration: 10s, jsonrpc_timeout: 15s, index_batch_size: 10, index_lookup_limit: Some(200), auto_reindex: true, ignore_mempool: false, sync_once: false, server_banner: "Welcome to electrs 0.9.0 (Electrum Rust Server)!", args: [] }
[2021-10-06T12:17:26.315Z INFO electrs::metrics::metrics_impl] serving Prometheus metrics on 127.0.0.1:4224
[2021-10-06T12:17:26.315Z DEBUG tiny_http] Server listening on 127.0.0.1:4224
[2021-10-06T12:17:26.316Z DEBUG tiny_http] Running accept thread
Illegal instruction (core dumped)

[Kixunil]

Did you compile with static or dynamic linking? You could try the other.

[mariusb]

Compiled with static: cargo build --locked --release

[mariusb]

Also tried with dynamic: ROCKSDB_INCLUDE_DIR=/usr/include ROCKSDB_LIB_DIR=/usr/lib cargo build --locked --release
and the same result:
● electrs.service - Electrs
Loaded: loaded (/etc/systemd/system/electrs.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: core-dump) since Wed 2021-10-06 16:49:32 SAST; 15s ago
Process: 2429 ExecStart=/home/marius/electrs/target/release/electrs --db-dir ./db --electrum-rpc-addr=0.0.0.0:50001 (code=dumped, signal=ILL)
Main PID: 2429 (code=dumped, signal=ILL)

Oct 06 16:49:32 mbbtcnode systemd[1]: electrs.service: Failed with result 'core-dump'.

[romanz]

Could you please try running using Docker on Debian 11, from p2p-bullseye branch?

$ docker run --rm -it debian:bullseye-slim /bin/bash
# apt git install cargo clang cmake build-essential librocksdb-dev
# git clone -b p2p-bullseye https://github.com/romanz/electrs && cd electrs/
# ROCKSDB_INCLUDE_DIR=/usr/include ROCKSDB_LIB_DIR=/usr/lib cargo build --locked --no-default-features --release
# ./target/release/electrs -vvv

If possible, please use https://asciinema.org/ to record and share the build log/errors.

[romanz]

Maybe we'll need to cherry-pick rust-rocksdb/rust-rocksdb@81a9ede into our RocksDB fork...
See rust-rocksdb/rust-rocksdb#327 for more details.

[romanz]

@mariusb Could you please run the following on your machine:

$ git clone -b 2021-09/electrs-build https://github.com/romanz/rust-rocksdb/
$ cd rust-rocksdb && cargo test --release --all

[romanz]

If the above fails, could you please also run:

$ git clone https://github.com/rust-rocksdb/rust-rocksdb/
$ cd rust-rocksdb && cargo test --release --all

[mariusb]

cd rust-rocksdb && cargo test --release --all

Good morning -- output from the above:
Compiling electrs-librocksdb-sys v6.11.4-e1 (/home/marius/rust-rocksdb/librocksdb-sys)
error: failed to run custom build command for electrs-librocksdb-sys v6.11.4-e1 (/home/marius/rust-rocksdb/librocksdb-sys)

Caused by:
process didn't exit successfully: /home/marius/rust-rocksdb/target/release/build/electrs-librocksdb-sys-dfc1b0db21df6a6a/build-script-build (exit status: 101)
--- stderr
error: header 'rocksdb/include/rocksdb/c.h' does not exist.
thread 'main' panicked at 'unable to generate rocksdb bindings: ()', librocksdb-sys/build.rs:44:10
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: build failed

[mariusb]

If the above fails, could you please also run:

$ git clone https://github.com/rust-rocksdb/rust-rocksdb/
$ cd rust-rocksdb && cargo test --release --all

And from the above:
Compiling librocksdb-sys v6.20.3 (/home/marius/rust-rocksdb/librocksdb-sys)
error: failed to run custom build command for librocksdb-sys v6.20.3 (/home/marius/rust-rocksdb/librocksdb-sys)

Caused by:
process didn't exit successfully: /home/marius/rust-rocksdb/target/release/build/librocksdb-sys-52742987e5dc8597/build-script-build (exit status: 101)
--- stderr
error: header 'rocksdb/include/rocksdb/c.h' does not exist.
thread 'main' panicked at 'unable to generate rocksdb bindings: ()', librocksdb-sys/build.rs:44:10
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: build failed

[mariusb]

Could you please try running using Docker on Debian 11, from p2p-bullseye branch?

$ docker run --rm -it debian:bullseye-slim /bin/bash
# apt git install cargo clang cmake build-essential librocksdb-dev
# git clone -b p2p-bullseye https://github.com/romanz/electrs && cd electrs/
# ROCKSDB_INCLUDE_DIR=/usr/include ROCKSDB_LIB_DIR=/usr/lib cargo build --locked --no-default-features --release
# ./target/release/electrs -vvv

If possible, please use https://asciinema.org/ to record and share the build log/errors.

Also tried this, am not a docker expert but got an error on the second line.
Here is recording: https://asciinema.org/a/WelZpFcFRrM4uFjlPCyTuogNP

[Kixunil]

I believe @romanz meant apt-get instead of apt git :)

[mariusb]

I believe @romanz meant apt-get instead of apt git :)

That make sense will try apt-get instead of apt git

[mariusb]

Gave the docker another try with apt-get install and get Unable to locate package

See recording: https://asciinema.org/a/Ji4vncM2iXdRB7DSXk05cgY7P

[romanz]

I forgot apt-get update, please run it before installing.

[mariusb]

Seems to be working but electrs is failing since it cannot find bitcoin core (which is correct since there is no bitcoin core in the docker image) in the docker but no core dump

Here is the recording: https://asciinema.org/a/WkhPRSsL726KiiXS7oEfXrb2A

[romanz]

Thanks!

You can try adding --network host --volume $HOME/.bitcoin:/root/.bitcoin:ro to the docker run invocation - to allow it to access the bitcoind cookie and connect to JSON RPC port.
Please see https://github.com/romanz/electrs/blob/master/doc/usage.md#docker-based-installation-from-source for details.

[mariusb]

You can try adding --network host --volume $HOME/.bitcoin:/root/.bitcoin:ro to the docker run invocation - to allow it to access the bitcoind cookie and connect to JSON RPC port.

Even with this suggestion it did not find bitcoin core - here is the command I used to start docker:
sudo docker run --network host --volume /home/marius/.bitcoin:/root/.bitcoin:ro --rm -it debian:bullseye-slim /bin/bash

Here is the final part of the docker output:

root@mbbtcnode:/electrs# ./target/release/electrs -vvv
Config { network: Bitcoin, db_path: "./db/bitcoin", daemon_dir: "/root/.bitcoin", daemon_auth: CookieFile("/root/.bitcoin/.cookie"), daemon_rpc_addr: 127.0.0.1:8332, daemon_p2p_addr: 127.0.0.1:8333, electrum_rpc_addr: 127.0.0.1:50001, monitoring_addr: 127.0.0.1:4224, wait_duration: 10s, jsonrpc_timeout: 15s, index_batch_size: 10, index_lookup_limit: Some(200), auto_reindex: true, ignore_mempool: false, sync_once: false, server_banner: "Welcome to electrs 0.9.0 (Electrum Rust Server)!", args: [] }
[2021-10-08T08:02:02.784Z DEBUG electrs::metrics::metrics_fake] metrics collection is disabled
[2021-10-08T08:02:04.624Z INFO  electrs::db] "./db/bitcoin": 0 SST files, 0 GB, 0 Grows
[2021-10-08T08:02:04.625Z DEBUG electrs::db] DB None
[2021-10-08T08:02:04.915Z INFO  electrs::db] closing DB at ./db/bitcoin
Error: electrs failed

Caused by:
    0: failed to open bitcoind cookie file: /root/.bitcoin/.cookie
    1: No such file or directory (os error 2)

[romanz]

It seems that your bitcoind cookie is not stored under /home/marius/.bitcoin/.
Please update this path to the directory where bitcoind writes its RPC authentication cookie - bitcoind should log it on startup:

2021-10-09T06:39:49Z Generated RPC authentication cookie /home/user/.bitcoin/.cookie

[mariusb]

The reason why I do not have a cookie file is that I have rpcuser & rpcpassword in my bitcoin.conf file which I need to connect Specter wallet to my bitcoin core node.

[romanz]

I can suggest using the following Dockerfile for building an electrs image from p2p-bullseye branch:

FROM debian:bullseye-slim as base
RUN apt-get update -qqy
RUN apt-get install -qqy librocksdb-dev=6.11.4-3

### Electrum Rust Server ###
FROM base as electrs-build
RUN apt-get install -qqy cargo clang cmake build-essential

# Install electrs
WORKDIR /build/electrs
COPY . .
ENV ROCKSDB_INCLUDE_DIR=/usr/include
ENV ROCKSDB_LIB_DIR=/usr/lib
RUN cargo install --locked --path .

FROM base as result
# Copy the binaries
COPY --from=electrs-build /root/.cargo/bin/electrs /usr/bin/electrs

WORKDIR /

You can build an image and tag it using:

$ docker build . --rm -t electrs:latest

Them you can run it:

$ docker run --network host --volume $HOME/.bitcoin:/root/.bitcoin:ro --volume $PWD/db:/db \
  --rm -t electrs:latest electrs -vv

[romanz]

The reason why I do not have a cookie file is that I have rpcuser & rpcpassword in my bitcoin.conf file which I need to connect Specter wallet to my bitcoin core node.

Got it, thanks!
I can suggest using the auth configuration option to specify user & password - note that it work only from a configuration file (not commandline):
https://github.com/romanz/electrs/blob/master/doc/usage.md#electrs-configuration (see "Authentication" section).

[mariusb]

docker build . --rm -t electrs:latest

[mariusb]

I can suggest using the following Dockerfile for building an electrs image from p2p-bullseye branch:

FROM debian:bullseye-slim as base
RUN apt-get update -qqy
RUN apt-get install -qqy librocksdb-dev=6.11.4-3

### Electrum Rust Server ###
FROM base as electrs-build
RUN apt-get install -qqy cargo clang cmake build-essential

# Install electrs
WORKDIR /build/electrs
COPY . .
ENV ROCKSDB_INCLUDE_DIR=/usr/include
ENV ROCKSDB_LIB_DIR=/usr/lib
RUN cargo install --locked --path .

FROM base as result
# Copy the binaries
COPY --from=electrs-build /root/.cargo/bin/electrs /usr/bin/electrs

WORKDIR /

You can build an image and tag it using:

$ docker build . --rm -t electrs:latest

Them you can run it:

$ docker run --network host --volume $HOME/.bitcoin:/root/.bitcoin:ro --volume $PWD/db:/db \
  --rm -t electrs:latest electrs -vv

Sorry by mistake closed the issue

Hmm tried to build the docker from this and get the following error:

Removing intermediate container 4d7f0201eca8
 ---> 94bc5b867b5c
Step 6/13 : WORKDIR /build/electrs
 ---> Running in 7f32a5e2a6af
Removing intermediate container 7f32a5e2a6af
 ---> 8cb680e33808
Step 7/13 : COPY . .
 ---> 5cabeabbb425
Step 8/13 : ENV ROCKSDB_INCLUDE_DIR=/usr/include
 ---> Running in f69a33e45b65
Removing intermediate container f69a33e45b65
 ---> ea03f77d1ccd
Step 9/13 : ENV ROCKSDB_LIB_DIR=/usr/lib
 ---> Running in 5c2262eef57c
Removing intermediate container 5c2262eef57c
 ---> 4581f3159713
Step 10/13 : RUN cargo install --locked --path .
 ---> Running in b0ae756934ca
error: `/build/electrs` does not contain a Cargo.toml file. --path must point to a directory containing a Cargo.toml file.
The command '/bin/sh -c cargo install --locked --path .' returned a non-zero code: 101

[romanz]

Please try running from the latest p2p-bullseye branch -> a45ee9c:

$ git clone -b p2p-bullseye https://github.com/romanz/electrs
$ cd electrs 
$ docker build . --rm -t electrs:latest

[mariusb]

Please try running from the latest p2p-bullseye branch -> a45ee9c:

$ git clone -b p2p-bullseye https://github.com/romanz/electrs
$ cd electrs 
$ docker build . --rm -t electrs:latest

Thanks, it now builds successfully. Will spend some time late today to see if this build run.

[romanz]

Closing for now, since the original core dump issue seems to be "worked-around" :)

[romanz]

Seems to happen also in #592.

[romanz]

@mariusb Could you please try to build and run the following branch on your platform:
https://github.com/romanz/electrs/commits/sse-cond
It uses romanz/rust-rocksdb@4935c28 (following #536 (comment)) to hopefully fix the SIGILL isse.

[mariusb]

Hi @romanz, should I build it in the docker image or in the standard electrs install?

[romanz]

Please try the standard electrs install (without the Docker).
If possible, please attach the verbose build log:

$ cargo build -vvv

[mariusb]

Ok, will do that, am just busy with other things and should get to it in a couple of hours.

I am not a git expert so can you share the correct git command to check out the relevant branch.

[romanz]

No worries, many thanks for the help!
Please try the following:

$ cd electrs
$ git fetch --all
$ git checkout sse-cond

[mariusb]

have build with -vvv from the sse-cond branch.
See output attached
Cargo build output - 20211025.txt

[mariusb]

i have manually run electrs with the following command:
/home/marius/electrs/target/release/electrs -vvv --db-dir ./db --electrum-rpc-addr="0.0.0.0:50001"
and attached the output before I manually stopped it. No core dump and it seemed to be working
Electrs running - 20211025.txt

[romanz]

Cool, many thanks!

[electrs-librocksdb-sys 6.11.4-e1] CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
[electrs-librocksdb-sys 6.11.4-e1] running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "rocksdb/include/" "-I" "rocksdb/" "-I" "rocksdb/third-party/gtest-1.8.1/fused-src/" "-I" "snappy/" "-I" "zstd/lib/" "-I" "zstd/lib/dictBuilder/" "-I" "." "-Wall" "-Wextra" "-std=c++11" "-Wno-unused-parameter" "-msse2" "-DSNAPPY=1" "-DZSTD=1" "-DNDEBUG=1" "-DOS_LINUX=1" "-DROCKSDB_PLATFORM_POSIX=1" "-DROCKSDB_LIB_IO_POSIX=1" "-o" "/home/marius/electrs/target/release/build/electrs-librocksdb-sys-64b8574fbaff0474/out/rocksdb/cache/cache.o" "-c" "rocksdb/cache/cache.cc"```

[romanz]

Should be resolved by #595.

[mariusb]

Thanks @romanz, updated today to v0.9.2 and started electrs as a service.

Took about 9 hours to rebuild the db and it is running perfectly.

Thanks for resolving this issue.

[romanz]

Cool, happy it worked :)