How to compile rust in a musl libc based linux system

[uuhan]

Hi, recently I need to compile rust in my musl libc based linux system. I download the rust's source code, configure it and make.

But it failed due to the rust/x86_64-unknown-linux-gnu/stage0/bin/rustc run failed.
It's interpreter is
/lib64/ld-linux-x86-64.so.2
but my system has just the
/lib/ld-musl-x86_64.so.1

I have already the llvm-3.5 installed, It seems to be it just lack of a runable rust stage0 binary for my system.

So, is there a STATICALLY LINKED RUST STAGE0 binary to feed my needs?
Or, how can I compile the proper stage0 rustc binary from stretch ?

[MagaTailor]

Is the musl ld compatible enough with gnu ld (and the two libc's themselves)? If so, symlinking should be sufficient.

[alexcrichton]

Yes unfortunately we only produce glibc snapshots that are dynamically linked. There are currently no statically linked snapshots of the compiler.

[nodakai]

Can you try /lib/ld-musl-x86_64.so.1 /path/to/stage0/rustc ? If successful, the rest is just shell scripting.

My understanding is it's similar to overriding a Shebang line in a shell script by explicitly calling a shell, like /bin/dash /path/to/script.sh

• http://man7.org/linux/man-pages/man8/ld.so.8.html

The .interp section in an ELF exe should serve as a "Shebang line."

[uuhan]

@petevine @nodakai Did not work. Some Symbol not found
__rawmemchr mallinfo __register_atfork

@alexcrichton Get it. : )

[nodakai]

How about referencing a pre-built glibc (eg. https://www.archlinux.org/packages/core/x86_64/glibc/ ) ? It's shipped with a dynamic loader as well.

$ strace -e file ./usr/lib/ld-linux-x86-64.so.2 --inhibit-cache --library-path ./usr/lib /bin/cat /etc/hostname 2>&1 | grep -v ENOENT
execve("./usr/lib/ld-linux-x86-64.so.2", ["./usr/lib/ld-linux-x86-64.so.2", "--inhibit-cache", "--library-path", "./usr/lib", "/bin/cat", "/etc/hostname"], [/* 52 vars */]) = 0
open("/bin/cat", O_RDONLY|O_CLOEXEC)    = 3
open("./usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
getcwd("/tmp/myglibc", 128)             = 13
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
open("/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 3
open("/etc/hostname", O_RDONLY)         = 3
d9+++ exited with 0 +++

(Not sure if strace(8) is reliable under such a hack.)

[jirutka]

@alexcrichton in #34687:
I know I'd personally love to see rustc running on Alpine, I've tried it once or twice in docker and have been sad when it doesn't work!

Well, obviously, because Alpine doesn’t use glibc, but musl libc, that is only partially binary compatible with glibc (musl strictly follows standards, but glibc does not). Therefore you can’t run rustc snapshot compiled against glibc on Alpine and it’s not possible to build rustc without rustc.

@alexcrichton in #34687:
Do you know if there are cross compilers from something like Ubuntu to Alpine Linux?

This is not about distributions, but different libc. So you need something to cross-compile between glibc and musl. It’s very similar to cross-compiling between different architectures.
Yes, there are such a tools, for example musl-cross.

@alexcrichton in #34687:
If we have that then the rest of the implementation should be relatively easy I believe.

I believe that it may be easy for someone skilled in cross compiling, but unfortunately me and other guys from Alpine community, who would like Rust on Alpine, are not. And those few skilled C programmers doesn’t like Rust, because… you know… C programmers…

/cc @japaric

[japaric]

I got a bash script that uses docker to cross compile a rustc for the host x86_64-unknown-linux-musl from a Ubuntu image. The rustc is a fully statically linked binary and doesn't depend on glibc because it was built against musl. The issue is that the resulting rustc segfaults when I try to compile anything ... rustc -V works though. The root of the problem appears to be LLVM but I haven't had to look into it / fix it. I have made my work so far public in the hope that someone might be able to push it to the finish line. See this i.r-l.o thread for the details.

[jirutka]

@japaric Thanks for your effort. I’ve discussed your gist in our IRC channel and skarnet recommends to use richfelker/musl-cross-make instead of GregorR/musl-cross. He said that GregorR’s musl-cross is pretty much obsoleted by richfelker’s (dalias’) musl-cross-make that is more up-to-date.

[japaric]

@jirutka Thanks for tip. I've updated the gist to use musl-cross-make instead of musl-cross. Unfortunately, it didn't magically fix the llvm issue.

[jirutka]

@japaric I’m trying to compile rustc using your script and it always fail in the last phase with:

build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-musl/lib/libunwind-40c0f24bb84d756d.rlib(UnwindLevel1.c.o): unrecognized relocation (0x2a) in section `.text.assert_rtn'
toolchain/bin/../lib/gcc/x86_64-linux-musl/5.2.0/../../../../x86_64-linux-musl/bin/ld: final link failed: Bad value

I’ve tried it with gcc 5.3.0 and 5.2.0, with and without --enable-llvm-static-stdcpp, but with no difference.

Do you have some idea what’s wrong?

[japaric]

@jirutka

I’m trying to compile rustc using your script and it always fail in the last phase with:

While building libunwind-40c0f24bb84d756d.rlib? That looks like a new error to me.

Are you following the gist instructions "to the letter"? In particular, are you using a Ubuntu 15.10 docker image? Because, IIRC, there some was some bug in the Ubuntu 16.04 binutils package that also caused a linker error (don't remember the details). That's why the instructions say: use Ubuntu 15.10.

Or it could be a recent regression in the unwind crate (doesn't seem likely to me though). Perhaps try checking out an older version of rust-lang/rust? My last test of the gist was around 6 days ago

[jupp0r]

Wouldn't it make sense for the stage0 rustc to be statically linked in general?

[japaric]

@jupp0r The stage0 snapshots are getting deprecated though. The new plan is to bootstrap from a previous stable release (cf. #32942). Unless you mean that we should always bootstrap from a stable rustc with host=x86_64-unknown-linux-musl (which doesn't exist right now)?

[japaric]

(Heh, I'm not totally sure if I should this here or in one of the other closely related open issues but here it goes)

Update: I got a rustc that works an Alpine. I have thrown a tarball of said compiler in my Dropbox in case anyone is interested in trying it out. I tested the compiler with the following smoke test:

$ docker run --rm -it alpine /bin/sh
# apk update
# URL="https://www.dropbox.com/sh/t3vs8sjc3u8rzzr/AACjn6w1Dq75THcV62YGCQQ5a/2016-07-25.tar.gz?dl=1"
# apk add curl # for the command below
# curl -sL $URL | tar -xz
# apk add gcc # need libgcc_s for rustc to work
# rustc -V
# ldd $(which rustc)
        /lib/ld-musl-x86_64.so.1 (0x560e95330000)
        librustc_driver-3db96603e7de68a8.so => /bin/../lib/librustc_driver-3db96603e7de68a8.so (0x7fb4c1f44000)
        libstd-b357b9eff40a747f.so => /bin/../lib/libstd-b357b9eff40a747f.so (0x7fb4c19d9000)
        libc.so => /lib/ld-musl-x86_64.so.1 (0x560e95330000)
        (..)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x7fb4ba14f000)
        (..)
# apk add musl-dev # need a bunch of libraries (rt, dl, c, m, etc) when linking a Rust binary
# echo 'fn main() { println!("Hello, world!") }' > hello.rs
# rustc hello.rs
# ./hello
Hello, world!
# echo 'fn main() { panic!("Hello, world!") }' > panic.rs
# rustc panic.rs
# RUST_BACKTRACE=1 ./panic # backtraces work
thread 'main' panicked at 'Hello, world!', panic.rs:1
stack backtrace:
   1:     0x558cae97656f - std::sys::backtrace::tracing::imp::write::h4ec66725a4c8d28d
   2:     0x558cae978c6b - std::panicking::default_hook::_{{closure}}::hcb8d8a999157470b
   3:     0x558cae978125 - std::panicking::default_hook::hc2e39a27ba5d2d45
   4:     0x558cae9786de - std::panicking::rust_panic_with_hook::h59941143cbee269d
   5:     0x558cae971c53 - std::panicking::begin_panic::hc74c3c2af73ca1a0
   6:     0x558cae971d99 - panic::main::h27a6e2b4553ca3bd
   7:     0x558cae9783a8 - std::panicking::try::call::h494c5dbd41ce3146
   8:     0x558cae97e936 - __rust_maybe_catch_panic
   9:     0x558cae977981 - std::rt::lang_start::hfb76de782d5db06c
  10:     0x558cae971dc9 - main
  11:     0x7fc9ee95772e - <unknown>

I cheated to make this work though. I hacked the x86_64-unknown-linux-musl target to make it dynamically link to libc (and libgcc_s) so it no longer produces statically linked binaries. Also when I tested a rustc compiled with jemalloc enabled it crashed with a SIGBUS. I had to build rustc with jemalloc disabled to get the working compiler I linked above.

The way forward: I think we should add a new musl target that's just like the existing x86_64-unknown-linux-musl but that produces dynamically linked binaries. Then we can start producing binary releases of rustc for that target (via cross compilation) and it a few cycles the Alpine folks should be able to bootstrap a Rust compiler themselves (they need a stable compiler to bootstrap a new).

Implementations details: The implementation seems straightforward, but the unresolved question is: what should we call this new target? @alexcrichton suggested calling the new target x86_64-alpine-linux-musl but as I stated in another comment, I don't think we should conflate the "vendor" (Alpine) with dynamic linking. I think the new target should be called x86_64-unknown-linux-musl:dynamic (the stuff after the : is open to bikeshedding, also this naming convention was originally proposed by @alexcrichton IIRC). The rationale is that everything that the original target triple conveys (ARCH, VENDOR, OS and ABI/ENV) remains the same; we are changing something (how we link) that can't be expressed by the "conventional" target triple so it makes sense (to me at least) to not modify the original target triple and just tack the extra information after original target triple.

[jirutka]

This is really great news! Excellent work, @japaric! 🎉

…add a new musl target that's just like the existing x86_64-unknown-linux-musl but that produces dynamically linked binaries.

Absolutely. Although I’m more a fan of static linking, I don’t understand Rust’s implication musl → static linking. musl can be statically linked without any problems (unlike ***** glibc…), but that doesn’t mean that it can’t be dynamically linked as well. Most of the Alpine Linux is dynamically linked.

…suggested calling the new target x86_64-alpine-linux-musl … I don't think we should conflate the "vendor" (Alpine) with dynamic linking.

Me neither, musl libc and Alpine are two different projects and Alpine is probably not the only distro that uses (dynamically linked) musl.

I think the new target should be called x86_64-unknown-linux-musl:dynamic…

I’m afraid that it’s already too late to change it, but to be consistent with others, x86_64-unknown-linux-musl should not implicate static linking. Thus it would make more sense to add :static suffix for static linking.

…the stuff after the : is open to bikeshedding…

I’m not sure about colon. It may be problematic e.g. on Windows when used in a file name. Triplet is already not a triplet, but quaternion, so maybe it would not hurt to use a quintuplet? ;)

That said, I’m probably okay with any name, I can’t wait for rustc officially supported for musl. 😸

I’m gonna try your tarball yesterday. Thanks a lot!

Could you please also update your gist with the build script?

/cc @LeoUnglaub

[japaric]

I’m afraid that it’s already too late to change it, but to be consistent with others, x86_64-unknown-linux-musl should not implicate static linking.

I agree, but it's too late. Changing it will probably break a bunch of build scripts.

Could you please also update your gist with the build script?

Sure, I'll ping you when it's updated.

[jupp0r]

I've been working on producing a static rustc the last few days, to no avail unfortunately.

[japaric]

Sure, I'll ping you when it's updated.

@jirutka The gist has been updated.

[jirutka]

@japaric I can confirm that rustc you’ve compiled works well on Alpine Linux. Could you please create a complete snapshot like those hosted on http://static.rust-lang.org? I’ve spent half day just trying to built cargo, using cargo-bootstrap, and it’s like a nightmare. :(

[jirutka]

Uff, I finally managed to build cargo on musl (with a lot of ad-hoc fixes during the process)!

cargo-0.11.0-nightly-x86_64-alpine-linux-musl.tar.gz (3.9 MiB)

[MagaTailor]

Not bad @jirutka! I wonder if panic=abort and lto works too? (I'm thinking about a smallest x86_64 fully optimized cargo binary to date)

[japaric]

Could you please create a complete snapshot like those hosted on http://static.rust-lang.org?

I think you only need to call make rustc-stage3 instead of make to get one of those. Do note that the bootstrap process is now using previous stable releases instead of the old snapshots. But, I guess you can use the snapshot with --local-rust=/path/to/rustc.

I’ve spent half day just trying to built cargo, using cargo-bootstrap, and it’s like a nightmare.

We are not building Cargo for x86_64-unknown-linux-musl yet? I thought that most of the necessary pieces were there already (cc @alexcrichton). Last time I tried, it was straightforward to cross compile Cargo to musl with these steps. Do note that you need to use this Docker image.

[jirutka]

I think you only need to call make rustc-stage3 instead of make to get one of those.

Hm, I’m still getting errors like below when compiling rustc on Alpine:

/usr/lib/gcc/x86_64-alpine-linux-musl/6.1.0/../../../../x86_64-alpine-linux-musl/bin/ld:
  /tmp/rustc.DH17TbBpNCoJ/liblibc-5aeeadd0596ef5a0.rlib(posix_spawn.lo):
  relocation R_X86_64_PC32 against protected symbol `execve' can not be used when making a shared object

I guess you can use the snapshot with --local-rust=/path/to/rustc.

Yes, that’s what I’m doing.

We are not building Cargo for x86_64-unknown-linux-musl yet?

No, https://static.rust-lang.org/cargo-dist/2016-03-21/cargo-nightly-x86_64-unknown-linux-gnu.tar.gz returns 404. BTW, it’s kinda hard to find what snapshots are available when static.rust-lang.org doesn’t provide indexes…

Last time I tried, it was straightforward to cross compile Cargo to musl with these steps.

Uh, I’m stupid… I found this issue, but wanted to try it with cargo-bootstrap and then forgot about this easier way.

[japaric]

Hm, I’m still getting errors like below when compiling rustc on Alpine:

Hmm, could that be related to #34978? If that's the case, I think a "snapshot" would still have the same problems. What version of binutils do you have installed in your Alpine box? It seems the binutils version is the root of the issue. A possible solution would be to cross compile the initial musl rustc from an older Ubuntu release (15.10 maybe?).

BTW, it’s kinda hard to find what snapshots are available when static.rust-lang.org doesn’t provide indexes…

I've been annoyed by that before. Could you open an issue in rust-lang/cargo?

[pickfire]

Yes, I have tried to build it with https://s3.amazonaws.com/rust-lang-ci/cargo-builds/fbeea902d2c9a5be6d99cc35681565d8f7832592/cargo-nightly-x86_64-unknown-linux-gnu.tar.gz as well but no luck.

Traceback (most recent call last):
  File "/home/ivan/act/pkg/aports/testing/rust/src/rustc-1.15.1-src/src/bootstrap/bootstrap.py", line 491, in <module>
    main()
  File "/home/ivan/act/pkg/aports/testing/rust/src/rustc-1.15.1-src/src/bootstrap/bootstrap.py", line 474, in main
    rb.build_bootstrap()
  File "/home/ivan/act/pkg/aports/testing/rust/src/rustc-1.15.1-src/src/bootstrap/bootstrap.py", line 297, in build_bootstrap
    self.run(args, env)
  File "/home/ivan/act/pkg/aports/testing/rust/src/rustc-1.15.1-src/src/bootstrap/bootstrap.py", line 300, in run
    proc = subprocess.Popen(args, env=env)
  File "/usr/lib/python2.7/subprocess.py", line 390, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1024, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
make: *** [Makefile:23: all] Error 1

Might as well be python 2's fault, does it supports python 3?

[ncopa]

We are adding support for ppc64le and s390x to Alpine (with musl libc). Are there any progress in a sane way to bootstrap/port rustc?

[japaric]

@ncopa A few people are working on it right now. See #40113

[corbinu]

Now that #40113 has closed does anybody know how I actually build master for Alpine. I have tried both on Alpine and Ubuntu and keep hitting walls. If anybody would be willing to help me through it I would be happy to document it for others.

[rgdoliveira]

I tried to bootstrap rust 1.20.0 on Alpine ppc64le (./x.py build) but it fails when trying to run 'cargo' ELF (that is downloaded at build time) because it was compiled using gblic.

[stefson]

It is complicated. The guy who wrote #40113 explained (tried to explain) to me how it works, see here: gentoo/musl#44 (comment)

You need for sure a fully working rust and cargo plus llvm and gcc (for linking) installed on the system you will start the build, so before that isn't done you don't have to worry about further details :)

[mati865]

@rustbot modify labels: +O-musl

Now with beta musl host toolchain available (x86_64 only) and updated bootstrap it should work when you run x.py with --build x86_64-unknown-linux-musl.

I'll test and try to address issues on next week.

[corbinu]

Tried this today on an Alpine container. I got much further this time

downloading https://static.rust-lang.org/dist/2019-04-11/cargo-beta-x86_64-unknown-linux-musl.tar.gz
################################################################################################################################################################################################# 100.0%extracting /rust/build/cache/2019-04-11/cargo-beta-x86_64-unknown-linux-musl.tar.gz
    Updating crates.io index
  Downloaded cc v1.0.35
  Downloaded getopts v0.2.17
  Downloaded cmake v0.1.38
  Downloaded petgraph v0.4.13
  Downloaded lazy_static v0.2.11
  Downloaded filetime v0.2.4
  Downloaded libc v0.2.51
  Downloaded serde_json v1.0.33
  Downloaded serde v1.0.82
  Downloaded num_cpus v1.8.0
  Downloaded serde_derive v1.0.81
  Downloaded time v0.1.40
  Downloaded ordermap v0.3.5
  Downloaded fixedbitset v0.1.9
  Downloaded itoa v0.4.3
  Downloaded toml v0.4.10
  Downloaded cfg-if v0.1.6
  Downloaded ryu v0.2.7
  Downloaded quote v0.6.10
  Downloaded proc-macro2 v0.4.24
  Downloaded syn v0.15.22
  Downloaded unicode-xid v0.1.0
error: cannot produce proc-macro for `serde_derive v1.0.81` as the target `x86_64-unknown-linux-musl` does not support these crate types
failed to run: /rust/build/x86_64-unknown-linux-musl/stage0/bin/cargo build --manifest-path /rust/src/bootstrap/Cargo.toml

[mati865]

@corbinu you have to build it dynamically. There are 2 ways to do it:

1. copy config.toml.example to config.toml (or use configure), uncomment and set

   rust/config.toml.example

   Line 475 in 72bc620

   #crt-static = false

   to true
2. use environment variable RUSTFLAGS="-C target-feature=-crt-static".

[corbinu]

@mati865 Thanks so much for trying to help. I do have the flags set

./x.py --build  x86_64-unknown-linux-musl
Updating only changed submodules
Submodules updated in 0.09 seconds
error: cannot produce proc-macro for `serde_derive v1.0.81` as the target `x86_64-unknown-linux-musl` does not support these crate types
failed to run: /rust/build/x86_64-unknown-linux-musl/stage0/bin/cargo build --manifest-path /rust/src/bootstrap/Cargo.tomlBuild completed unsuccessfully in 0:00:00
/rust # echo RUSTFLAGS
RUSTFLAGS
/rust # echo $RUSTFLAGS
-C target-feature=-crt-static
/rust #

[mati865]

That's weird. I'll check it soon and report here.

[sanmai-NL]

Is RUSTFLAGS an environment variable, or just a shell variable here? Try printenv.

[mati865]

Bootstrap doesn't respect RUSTFLAGS and defaults to --build <target> passed to x.py.

To bootstrap with musl you have to uncomment

rust/config.toml.example

Line 475 in bfb443e

#crt-static = false

uncomment and set

rust/config.toml.example

Line 115 in bfb443e

#build = "x86_64-unknown-linux-gnu" # defaults to your host platform

to x86_64-unknown-linux-musl.

---

I tested it only in quite more complex environment (see #60240) but this should work on musl distros:

./configure --build x86_64-unknown-linux-musl --set target.x86_64-unknown-linux-musl.crt-static=false
./x.py build --build x86_64-unknown-linux-musl

[petrochenkov]

Is this issue obsolete now?
It's been 3 years since #40113 was merged.

[mati865]

Yes, it is.

[petrochenkov]

Closing then.