segmentation fault on "diesel setup"

[alexkamp]

hey,

When I run "diesel setup" in the docker container https://hub.docker.com/r/liuchong/rustup/, I get a segmentation fault. A (zipped) core dump is attached.

core.zip

Best,
Alex

[sgrif]

Can you provide additional information, such as the version of Diesel CLI you were using, the version of Rust you were using, and whether DATABASE_URL was pointing at a Postgres, SQLite, or MySQL database?

[alexkamp]

I called diesel in a directory which contains a .env file with

DATABASE_URL=postgres://postgres:***@postgres/reminder

(of course the pwd is not ***).

cargo -V
cargo 0.16.0-nightly (6e0c18c 2017-01-27)

rustc -V
rustc 1.15.1 (021bd294c 2017-02-08)

diesel_cli v0.10.1

I just noticed that in fact no diesel-command works. Even diesel -v segfaults. That made me wonder whether there is something fishy about my setup, however, compiling a hello-world with rustc works as expected. Anything more I can try?

[sgrif]

If you can give a list of steps I could follow to reproduce that would be helpful. I agree that it's probably something fishy with your setup. Maybe try compiling a program that links diesel without using the CLI?

It could also be the loading of one of the c libraries. Do you still see a segfault if you do cargo install diesel_cli --no-default-features --features postges? Do you see it if you run an application linked against pq-sys?

[alexkamp]

First of all, thanks for your help!

The behaviour does not change for diesel_cli --no-default-features --features postgres. I can not execute a program which links against pq-sys.

I wrote a docker file and a shell script to reproduce the problem.

The script starts postgreSQL in a Docker container, then it builds a container with rustup and diesel in it, links it to the postgreSQL instance and opens a root shell into the container. On my machine, diesel setup fails within this container.

environment.tar.gz

Please mind:
(1) The script does not tidy up in the end. That is, both docker container and the image stay around.
(2) The script links the project files (an empty project created with cargo) into the container. On my machine, this works only if se linux is disabled.

[dbrgn]

I'm also having segfaults for docker_cli in a musl based docker image.

Steps to reproduce:

$ docker run --rm -t -i clux/muslrust:stable /bin/bash
root@5caf11686839:/# apt-get update && apt-get install -y postgresql libpq-dev
root@5caf11686839:/# cargo install diesel_cli --no-default-features --features postgres
root@5caf11686839:/# /root/.cargo/bin/diesel 
Segmentation fault (core dumped)

When debugging the coredump:

# gdb /root/.cargo/bin/diesel core.357
Reading symbols from /root/.cargo/bin/diesel...done.
[New LWP 357]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./diesel'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000547607 in __vdsosym ()
(gdb) bt
#0  0x0000000000547607 in __vdsosym ()
#1  0x00000000005463d3 in cgt_init ()
#2  0x00007ffc194bdce0 in ?? ()
#3  0x0000000000000000 in ?? ()

Does that help?

[megametres]

Hello,
I am also having Segmentation fault within docker.
Using the following Dockerfile
`FROM rustlang/rust:nightly-alpine

RUN apk add postgresql-dev mariadb-dev sqlite-dev musl-dev gdb
RUN cargo install diesel_cli --no-default-features --features postgres

ARG USERNAME=rust
ARG USER_UID=1000
ARG USER_GID=$USER_UID
RUN addgroup --gid $USER_GID $USERNAME && adduser --uid $USER_UID --ingroup "$USERNAME" --no-create-home --disabled-password $USERNAME
RUN mkdir /app

WORKDIR /app/
RUN cargo new --lib diesel_demo
RUN echo 'diesel = { version = "1.4.4", features = ["postgres"] }
dotenv = "0.15.0"' >> diesel_demo/Cargo.toml

WORKDIR /app/diesel_demo
`

Then when I build and sh into it I got these:

`metres@machine:~/developpement/bidon$ docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -it bidon /bin/sh
/app/diesel_demo # diesel setup --database-url postgres://test:test@192.168.0.211/test
Creating migrations directory at: /app/diesel_demo/migrations
Segmentation fault (core dumped)
/app/diesel_demo # rust-gdb --args diesel setup --database-url postgres://test:test@192.168.0.211/test
GNU gdb (GDB) 9.2
Reading symbols from diesel...
(gdb) run
Starting program: /usr/local/cargo/bin/diesel setup --database-url postgres://test:test@192.168.0.211/test

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0 0x0000000000000000 in ?? ()
#1 0x00007ffff7e1b9be in diesel::pg::connection::raw::RawConnection::establish ()
#2 0x00007ffff7e1c59f in <diesel::pg::connection::PgConnection as diesel::connection::Connection>::establish ()
#3 0x00007ffff7d5999d in diesel::database::setup_database ()
#4 0x00007ffff7d2d3d9 in diesel::main ()
#5 0x00007ffff7d57743 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#6 0x00007ffff7d57ab9 in std::rt::lang_start::{{closure}} ()
#7 0x00007ffff7ea4528 in core::ops::function::impls::<impl core::ops::function::FnOnce for &F>::call_once () at /rustc/d006f5734f49625c34d6fc33bf6b9967243abca8/library/core/src/ops/function.rs:259
#8 std::panicking::try::do_call () at library/std/src/panicking.rs:381
#9 std::panicking::try () at library/std/src/panicking.rs:345
#10 std::panic::catch_unwind () at library/std/src/panic.rs:382
#11 std::rt::lang_start_internal () at library/std/src/rt.rs:51
#12 0x00007ffff7d31742 in main ()
(gdb) `

[megametres]

So I tried using the rustlang/rust:nightly-buster image and I have no more segfault. So it seems to be related to alpine...

[weiznich]

@megametres Can you try if that reproduces with the latest 1.4.5 release, as this release fixes a potential issue with in the postgres connection setup?

[megametres]

I figured out that the Cargo.toml provided in my example is never used apart to let the diesel_cli start as it required a Cargo.toml file to exists. The error is generated by diesel_cli which last version is 1.41

But on my personal project, I use the 1.4.5 release and still had the segmentation fault when using Alpine based docker images.

[c0dearm]

I can also reproduce in a Docker Alpine image. The Dockerfile itself is quite simple:

FROM rust:alpine

RUN apk add --no-cache libpq openssl-dev musl-dev postgresql-dev \
         && cargo install diesel_cli --no-default-features --features postgres

I then run the diesel setup command in a container of that image connected to a PostgreSQL database and correct DATABASE_URL environment variable:

/opt/app # diesel setup
Segmentation fault (core dumped)

Trying to establish a connection using PgConnection also ends in a Segmentation fault, my guess is that there is something wrong when using the x86_64-unknown-linux-musl target, which is the default used in Alpine and doesn't seem to be tested, at least not in the project CI.

Versions:
rust (stable) = "1.47.0"
diesel = "1.4.5"

Stack trace:

/opt/app # rust-gdb --args diesel setup
GNU gdb (GDB) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-alpine-linux-musl".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from diesel...
(gdb) run
Starting program: /usr/local/cargo/bin/diesel setup
warning: Error disabling address space randomization: Operation not permitted

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00007fefb070ba8e in diesel::pg::connection::raw::RawConnection::establish ()
#2  0x00007fefb070c66f in <diesel::pg::connection::PgConnection as diesel::connection::Connection>::establish ()
#3  0x00007fefb0649aed in diesel::database::setup_database ()
#4  0x00007fefb061d429 in diesel::main ()
#5  0x00007fefb06478b3 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#6  0x00007fefb0647c29 in std::rt::lang_start::{{closure}} ()
#7  0x00007fefb07949e8 in core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once ()
    at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/core/src/ops/function.rs:259
#8  std::panicking::try::do_call () at library/std/src/panicking.rs:373
#9  std::panicking::try () at library/std/src/panicking.rs:337
#10 std::panic::catch_unwind () at library/std/src/panic.rs:379
#11 std::rt::lang_start_internal () at library/std/src/rt.rs:51
#12 0x00007fefb0621792 in main ()

[c0dearm]

So I've been tracking down the issue to some other discussions and it seems the culprit is libpq, it seems Alpine's version is outdated, and actually it is currently flagged: https://pkgs.alpinelinux.org/package/edge/main/x86/libpq

According to #2357 and sfackler/rust-openssl#1133, upgrading libpq should solve the issue, but I wonder when this will be available in the Alpine repositories.

[weiznich]

@c0dearm Thanks for debugging this 👍. This sounds like #813 then.
As this is then clearly not a diesel issue anymore I will close this issue as not our bug.
For anyone hitting this, I think the best workaround is to build a current libpq + openssl version in your docker container by yourself.

[c0dearm]

@weiznich Also, pq-sys hasn't been updated for 2 years now, so maybe it is a good time to ditch it and replace it by something native like rust-postgres. I think it would solve a lot of issues, especially when building static binaries.

[weiznich]

Dropping pq-sys (and therefore libpq) in favor of rust-postgres brings up it's own list of issues:

• Is the performance of rust-postgres comparable to libpq? Has anyone already done reliable reproducible benchmarks on that?
• Is the API design of rust-postgres compatible with diesel's assumptions about how a database connections works. Especially in terms of design of the serializing/deserializing and Statement API? The last time I've tried to work on that (see Feature/pure postgres #2375) I found out that it is possible to get something working, but that was not necessarily nice or maintainable.
• rust-postgres pulls in quite a large number of dependencies (some of them quite heavy, like tokio). I'm not sure if that's really the way to go.

Additionally: That a crate does not receive updates in the last 2 years does not mean that it is out of date or something like that. It's just that pq-sys provides all functions that wee need in diesel and as it is just a bindgen generated wrapper crate around libpq there is not much about the API that need to change, as the API of libpq is really stable.

That all written: If somebody really want to have a pure rust connection, nobody is going to stop anyone from working on this. We are happy to accept a PR working on that, as long as it addresses those points mentioned above. Additionally diesel is designed in such a way that any thirdparty crate can provide a custom implementation of the Connection trait for whatever type it wants (as long as the impl is allowed by the orphan rule).

[Razican]

So I've been tracking down the issue to some other discussions and it seems the culprit is libpq, it seems Alpine's version is outdated, and actually it is currently flagged: pkgs.alpinelinux.org/package/edge/main/x86/libpq

According to #2357 and sfackler/rust-openssl#1133, upgrading libpq should solve the issue, but I wonder when this will be available in the Alpine repositories.

It seems this is not yet fixed. With Alpine 3.14, that uses libpq 13.4 as can be seen here I'm stil getting this segmentation fault. I get it both when doing diesel setup and also when trying to establish a database connection in my program.

Using the following code:

    if net::TcpStream::connect(format!("{}:{}", db_host, db_port)).is_ok() {
        println!("Connected to the database!");
    } else {
        panic!("Couldn't connect to the database...");
    }

    let conn = diesel::pg::PgConnection::establish(&database_url);
    println!("Connection OK? {}", conn.is_ok());

I get a "Connected to the database!" output, which means that the TCP connection is well working, but then the PgConnection::establish() gives a segmentation fault, so it never reaches the last println!().

I can also see that, for example in 13r0ck/gci-website@a4ed8d0, some projects are having to switch out of using Alpine for their builds and binaries due to missing Diesel support, which is not the ideal solution, since Alpine is a very reasonable distribution for cloud-native applications.

Is there any workaround that can make diesel work in Alpine linux? Should this issue stay closed even if diesel_cli doesn't properly run in Alpine Linux? And, if this is an issue in another library or the Rust toolchain (not sure if rust-lang/compiler-team#422 is related), should we have this as a tracking issue for its solution, pointing to the culprit?

[weiznich]

@Razican It seems very plausible that this can be caused by the incompatibility described in rust-lang/compiler-team#422. In that case there is nothing we can do on diesels side to fix the problem here. As we generally prefer to have open issues only for things that are actionable for contributors (= things you can pick up to work on) that means this issue is closed. Anyway a closed issue does not mean that you cannot discussion things here or report possible solutions or something like that. Additionally we can always reopen a issue as soon as it become actionable because of updated information or because the upstream bug is fixed.

I can also see that, for example in 13r0ck/gci-website@a4ed8d0, some projects are having to switch out of using Alpine for their builds and binaries due to missing Diesel support, which is not the ideal solution, since Alpine is a very reasonable distribution for cloud-native applications.

Just to restate the hopefully obvious here: Distributions that dynamically link musl libc (like alpine) are not really supported by upstream rustc in that way that there is not even a target to build for this libraries. You can make it somehow work by using the musl target provided by rustc as long as you statically link everything. Additionally at least for diesel is the musl target a best effort target, which means we do not test there and at least I do not really use that target. This means I do not actively maintain it, as I simply do not have the capacity to test and maintain support all possible compatible rustc targets. I do accept PR's fixing this target so if you are interested in that please submit a PR.

Is there any workaround that can make diesel work in Alpine linux?

The workaround for any kind of linker issues is always the same: Provide a compatible version of any depended library. In this case this likely requires you to manually build a libpq version that is compatible with rusts static linking of the musl libc. Likely that requires statically linking libpq + openssl as well.

Should this issue stay closed even if diesel_cli doesn't properly run in Alpine Linux? And, if this is an issue in another library or the Rust toolchain (not sure if rust-lang/compiler-team#422 is related), should we have this as a tracking issue for its solution, pointing to the culprit?

As already elaborated above: Yes this issue should stay closed, as we don't keep open tracking issues to "just" track some upstream bug.

[Razican]

Thank you for that explanation! Indeed, it was a linking issue. The solution was to link musl dynamically in Alpine by adding the following RUSTFLAGS:

RUSTFLAGS="-C target-feature=-crt-static"

[abdelmonem-mohamed17]

Thank you for that explanation! Indeed, it was a linking issue. The solution was to link musl dynamically in Alpine by adding the following RUSTFLAGS:

RUSTFLAGS="-C target-feature=-crt-static"

@Razican this solution make the image becames very big because it requires libpq openssl-dev musl-dev postgresql-dev to be installed so if the image becames bigger than ubuntu based images so why I will use alpine as a base image?

[Razican]

Thank you for that explanation! Indeed, it was a linking issue. The solution was to link musl dynamically in Alpine by adding the following RUSTFLAGS:

RUSTFLAGS="-C target-feature=-crt-static"

@Razican this solution make the image becames very big because it requires libpq openssl-dev musl-dev postgresql-dev to be installed so if the image becames bigger than ubuntu based images so why I will use alpine as a base image?

Size might not be the main reason for using Alpine. In my case its security and minimalism is much more relevant.

[abdelmonem17]

Thank you for that explanation! Indeed, it was a linking issue. The solution was to link musl dynamically in Alpine by adding the following RUSTFLAGS:

RUSTFLAGS="-C target-feature=-crt-static"

@Razican this solution make the image becames very big because it requires libpq openssl-dev musl-dev postgresql-dev to be installed so if the image becames bigger than ubuntu based images so why I will use alpine as a base image?

Size might not be the main reason for using Alpine. In my case its security and minimalism is much more relevant.

@Razican yes I agree with you but the size is became the double so you loss the advantage of the small size for the alpine

[abdelmonem17]

@Razican I can see that many pepole menthion this problem and the had to go back and prefer to use ubuntu

[weiznich]

@abdelmonem-mohamed17 Our issue tracker is no discussion forum, so please stop to add unrelated comments about your docker issues here.

[abdelmonem-mohamed17]

@weiznich It is not a docker problem by the way but anyway thank you for your help.