Rust Cross Compilation (1.32.0) #56540

illegalprime · 2019-02-28T19:40:22Z

Motivation for this change

This is the same as #50866 but re-based and updated for Rust 1.32.0. It also does some other extra stuff like moving all output dirs to target/{release, debug} and being able to specify the build mode (release or debug).

the following derivations build:

I will attempt to updated this as soon as I figure out how to build more derivations. There are a lot more rust packages in nixpkgs, but these are likely a good starting point.

Any edits to or help with these derivations is appreciated.

Things done

illegalprime · 2019-03-11T16:51:37Z

not sure why, maybe @matthewbauer might know, but I get these errors on a lot of rust packages: ripgrep, zola, etc.

/nix/store/hxwlbcm8bbhdacayc90jljcd76w434r0-aarch64-unknown-linux-gnu-binutils-2.31.1/bin/aarch64-unknown-linux-gnu-ld: /nix/store/f6lrlfbg9ddvvw20j6av4ym7hxy6kxx1-rustc-1.32.0/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-e27907af0047d202.rlib(std-e27907af0047d202.std.3x3nkc5s-cgu.0.rcgu.o): Relocations in generic ELF (EM: 0)
/nix/store/hxwlbcm8bbhdacayc90jljcd76w434r0-aarch64-unknown-linux-gnu-binutils-2.31.1/bin/aarch64-unknown-linux-gnu-ld: /nix/store/f6lrlfbg9ddvvw20j6av4ym7hxy6kxx1-rustc-1.32.0/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-e27907af0047d202.rlib: error adding symbols: file in wrong format
          collect2: error: ld returned 1 exit status

it seems like libstd is in the wrong format, even though rust seems to think it's right, (because it placed it in the aarch64-unknown-linux-gnu folder). so maybe rustc is being build incorrectly.

but the fd package seems to build fine, it's the exception however. i believe it's not using the same libstd somehow, but i know it's using the same rustc

UPDATE: fixed with a weird hack bd45c09 dunno what 2 do about that

Mic92 · 2019-03-12T12:28:08Z

I have not looked yet at the rlib format, but I read it is a elf file with some extras, so using readelf should provide you some information.

illegalprime · 2019-03-12T13:06:21Z

@Mic92 yes, (see list above) cross-compiled cargo, rustc, and other packages I've tested are working! The only thing that isn't "working" is compiling rustc to be a cross-compiler that runs on the build platform and has cross-compiled host libs. I get around it with the hack I mentioned earlier.

the right architecture lib seems to get compiled, but there's something wrong with them. :/

EDIT: Specificity
The error I described above is a linking error with the first object inside of the rlib (since the offset is 0 bytes), using readelf -h we get:

File: /nix/store/qzpw6c9i8jk3lq3wbaspawc7mcwslryz-my-rustc/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-e27907af0047d202.rlib(std-e
27907af0047d202.std.3x3nkc5s-cgu.0.rcgu.o)
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              REL (Relocatable file)
  Machine:                           None
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          0 (bytes into file)
  Start of section headers:          153720 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           0 (bytes)
  Number of program headers:         0
  Size of section headers:           64 (bytes)
  Number of section headers:         558
  Section header string table index: 557

The same from the cross-compiled rustc:

File: /nix/store/7a4xsgvjzk1j49l7vp17szg3yfqqqhcb-rustc-1.32.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libs
td-3367fd61bbaa28f9.rlib(std-3367fd61bbaa28f9.std.d5c46zzr-cgu.0.rcgu.o)
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              REL (Relocatable file)
  Machine:                           AArch64
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          0 (bytes into file)
  Start of section headers:          154192 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           0 (bytes)
  Number of program headers:         0
  Size of section headers:           64 (bytes)
  Number of section headers:         560
  Section header string table index: 559

The only thing I can guess is that the Machine field is None in the bad rlib, but as you can see both of them look like they're the right architecture (and are placed in the correct location for rustc to pick them up as that architecture)

Mic92 · 2019-03-12T17:06:59Z

Can you upload? /nix/store/qzpw6c9i8jk3lq3wbaspawc7mcwslryz-my-rustc/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-e27907af0047d202.rlib the one with the broken Machine field.

Mic92 · 2019-03-12T17:08:44Z

pkgs/applications/networking/p2p/synapse-bt/default.nix


  cargoBuildFlags = [ "--all" ];

+  PKG_CONFIG_ALLOW_CROSS = 1;


Is this something we should set by default in buildRustPackage?

possibly, it's for pkg-config so ostensibly it would need to be set anywhere pkg-config is used during cross-compilation. Don't know how to handle all those cases though.

Actually it is a pkgconfig-rs thing i think

Maybe some hook in buildRustPackage that checks if pkg-config is available and sets PKG_CONFIG_ALLOW_CROSS in that case?

illegalprime · 2019-03-12T17:18:57Z

@Mic92 so this happens when you build rustc with buildPlatform == hostPlatform != targetPlatform, here you go:
libstd-e27907af0047d202.zip

Mic92 · 2019-03-12T17:29:59Z

Looks actually sane on the first glance:

$ 7z x libstd-e27907af0047d202.zip
$ ar x libstd-e27907af0047d202.rlib
$ file *.o
alloc.o:                                        ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
backtrace.o:                                    ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
dwarf.o:                                        ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
elf.o:                                          ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
fileline.o:                                     ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
posix.o:                                        ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
read.o:                                         ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
sort.o:                                         ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
state.o:                                        ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
std-e27907af0047d202.std.3x3nkc5s-cgu.0.rcgu.o: ELF 64-bit LSB relocatable, no machine, version 1 (SYSV), not stripped
std-e27907af0047d202.std.3x3nkc5s-cgu.1.rcgu.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
std-e27907af0047d202.std.3x3nkc5s-cgu.2.rcgu.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
std-e27907af0047d202.std.3x3nkc5s-cgu.3.rcgu.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
std-e27907af0047d202.std.3x3nkc5s-cgu.4.rcgu.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
std-e27907af0047d202.std.3x3nkc5s-cgu.5.rcgu.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
std-e27907af0047d202.std.3x3nkc5s-cgu.6.rcgu.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
std-e27907af0047d202.std.3x3nkc5s-cgu.7.rcgu.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
std-e27907af0047d202.std.3x3nkc5s-cgu.8.rcgu.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped
std-e27907af0047d202.std.3x3nkc5s-cgu.9.rcgu.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), with debug_info, not stripped

illegalprime · 2019-03-12T17:54:32Z

@Mic92 yea, could setting the Machine correctly really be all that's needed?

Mic92 · 2019-03-12T18:00:28Z

@illegalprime could be. I have not figured out yet, how I can run gdb on the build compiler otherwise I would set a breakpoint in the right location. The hack is ok right now, also annoying because it forces cross-compilers to build a rustc they probably never need. Maybe this can be reported upstream though. I wonder how they build those stdlibs.

matthewbauer · 2019-03-12T19:53:06Z

pkgs/build-support/rust/default.nix

+
+    rm -rf $out/lib/rustlib/${stdenv.hostPlatform.config}
+    cp -a \
+      ${pkgsCross."${host}".rustc}/lib/rustlib/${stdenv.hostPlatform.config}/. \


This should be targetpackages - pkgsCross should not be referenced internally.

if i try to use targetPackages.rustc I get:
error: attribute 'rustc' missing,

Shouldn't it be buildPackages? Or maybe just pkgs? I'm a bit confused on what the intention is.

this rustc is supposed to be the rustc whose host & target is aarch64

I think pkgs.rustc is where just the target is aarch64 and build & host are x86

matthewbauer · 2019-03-12T20:44:33Z

pkgs/applications/networking/p2p/synapse-bt/default.nix


  cargoBuildFlags = [ "--all" ];

+  PKG_CONFIG_ALLOW_CROSS = 1;


Actually it is a pkgconfig-rs thing i think

matthewbauer · 2019-03-12T20:47:57Z

pkgs/development/compilers/rust/rustc.nix

+    ccForBuild  = "${buildPackages.stdenv.cc}/bin/${buildPackages.stdenv.cc.targetPrefix}cc";
+    cxxForBuild = "${buildPackages.stdenv.cc}/bin/${buildPackages.stdenv.cc.targetPrefix}c++";
+    ccForTarget  = "${targetPackages.stdenv.cc}/bin/${targetPackages.stdenv.cc.targetPrefix}cc";
+    cxxForTarget = "${targetPackages.stdenv.cc}/bin/${targetPackages.stdenv.cc.targetPrefix}c++";


This looks wrong. /cc @Ericson2314

illegalprime · 2019-03-13T16:53:25Z

turns out it was the stripping in the fixupPhase that was creating the bad rlibs, not rust's bootstrapping process, :(

Mic92 · 2019-03-13T16:58:07Z

Good news. Anything left to do then?

illegalprime · 2019-03-13T17:07:23Z

@Mic92 i have an interest in cross compiling for 32-bit arm (armv7), doing:

nix build -f . pkgsCross.armv7l-hf-multiplatform.ripgrep

breaks because Rust doesn't recognize the triple armv7l-unknown-linux-gnueabihf, but is does recognize armv7-unknown-linux-gnueabihf (no l). my whole wanting rust-cross-compilation is for this target so I'd like to be able to get that working before this merges, should I:

somewhere within buildRustPackage take that triple and remove the l when giving it to rust
add a new entry in pkgsCross
???

if you think no. 2 is preferable, how do I do that? the entire layout of pkgsCross is a mystery to me.

EDIT: less of a mystery, after knowing it's in pkgs/top-level/stage.nix. I added my own target but had to add this as a cpu type:

diff --git a/lib/systems/parse.nix b/lib/systems/parse.nix
index 6947d41419e..c029d6a1914 100644
--- a/lib/systems/parse.nix
+++ b/lib/systems/parse.nix
@@ -76,6 +76,7 @@ rec {
     armv7r   = { bits = 32; significantByte = littleEndian; family = "arm"; version = "7"; };
     armv7m   = { bits = 32; significantByte = littleEndian; family = "arm"; version = "7"; };
     armv7l   = { bits = 32; significantByte = littleEndian; family = "arm"; version = "7"; };
+    armv7    = { bits = 32; significantByte = littleEndian; family = "arm"; version = "7"; };
     armv8a   = { bits = 32; significantByte = littleEndian; family = "arm"; version = "8"; };

illegalprime · 2019-03-13T17:13:27Z

another thing since we're here, how do people feel about https://www.hadean.com/blog/managing-rust-dependencies-with-nix-part-ii (i'm not the author, or know the author)? It seems more nix-ish than the current setup, but it didn't seem very popular in #11144.

my other main goal is to be able to support rust builds where some crates are in directories that are siblings to the project directory, without setting the entire parent folder as my src. this is because there's a lot of other stuff in that folder as well (that are dependencies for different project altogether)

Ericson2314 · 2019-03-13T21:05:31Z

I rebased @Mic92's old PR and changed some things in the past. I'll work to get that rebased so I can combine with this.

Ericson2314 · 2019-03-13T23:29:33Z

https://github.com/NixOS/nixpkgs/pull/57611/files is something that would make this easier.

Ericson2314 · 2019-03-14T04:37:21Z

https://github.com/Ericson2314/nixpkgs/branche/rust-cross here my old modification, rebased to have the same branch as this, but not yet tested). However, if #57611 is merged, we should rebase both onto that before combining.

illegalprime · 2019-03-14T15:54:53Z

After some hacking, ignore my previous 2 comments, I'm all good on this PR I think it's ready to merge! 🎉

illegalprime · 2019-03-16T01:39:22Z

pkgs/development/compilers/rust/rustc.nix

+  ] ++ optional (!withBundledLLVM) [
+    "--enable-llvm-link-shared"
+    "${setBuild}.llvm-config=${buildLlvmShared}/bin/llvm-config"
+    "${setHost}.llvm-config=${buildLlvmShared}/bin/llvm-config"


this might be wrong, need to test

so when building under ARMv7 rust tries to call llvm-config to get the libs to link against. buildLlvmShared has an llvm-config that can run on the build system but the wrong shared libraries, while llvm-config is the opposite (wrong binary, right libs). Not sure why this problem doesn't show up on AArch64.

Maybe a shim llvm-config script should be introduced, this is similar to meta-amd's: https://git.yoctoproject.org/cgit/cgit.cgi/meta-amd/tree/common/recipes-core/llvm/files/0001-llvm-config-allow-overriding-libdir-through-cmdline.patch?h=sumo

Since there should be no arch-specific one could wrap around the llvm-config
and then just do sed -e "s!${buildPackages.llvm}!${llvm}!" on the output?

illegalprime · 2019-03-25T23:22:51Z

@Mic92 NP, do you have documented somewhere the philosophy and plan for nixpkgs cross-comp? things like depsTargetTargetPropagated and that PR make me feel like there's a larger architecture goal at play that I don't quite understand.

Ericson2314 · 2019-03-26T04:25:33Z

I assume you meant me? :). So https://nixos.org/nixpkgs/manual/#ssec-stdenv-dependencies is the main section describing the dependencies and how they fit together, but in that PR I extended the corresponding section the cross compilation section. Hopefully https://hydra.nixos.org/build/91228503# has it in a moment.

edit master post PR built:

Ericson2314 · 2019-03-26T04:35:33Z

OK now that that #57611 is merged:

https://github.com/Ericson2314/nixpkgs/commits/rust-cross is the rebased version of mine
https://github.com/Ericson2314/nixpkgs/commits/rust-cross-2 is the rebased version of this, with one squash.

The first thing to note is @Mic92 did take my changes (all-packages.nix changes were moved to an I guess orphaned default.nix) I missed that before, sorry. This means there will be little-to-nothing to double check against my branch :).

The second things is @illegalprime your PR is harder to review because some commits "undo" other commits (revert or moral revert). Could you git rebase -i so the commits tell a nicer, if artificial, "story"? (Feel free to start with the rebase I did, which squashed the "remove jemalloc" change since the original commit added it back only after an prior rebase's conflict resolution.)

After that's done I'll be able to better wrap my head around everything, and also set about incorporating the new pkgsFooBar ewnly-added in #57611, which will help with cross compiling rustc (build != host == target).

Thanks in advance!

illegalprime · 2019-03-27T02:54:01Z

thanks for the links to the docs @Mic92 , i rebased it, we'll see how the tests go

Ericson2314

Thanks again @illegalprime and @Mic92! I pruned the remaining silly states from the history, and incorporated the new pkgsBuild*, so this is ready to go!

Ericson2314 · 2019-03-29T17:25:07Z

https://github.com/Ericson2314/nixpkgs/commits/rust-cross-2 has the beginnings of a buildRustCrate, but that can be merged later.

Ericson2314 · 2019-04-01T23:18:35Z

NixOS/ofborg looked good enough to me, haha.

illegalprime requested review from LnL7 and Mic92 as code owners February 28, 2019 19:40

GrahamcOfBorg added the 6.topic: rust label Feb 28, 2019

GrahamcOfBorg added 8.has: clean-up 10.rebuild-darwin: 501+ 10.rebuild-darwin: 501-1000 10.rebuild-linux: 501+ 10.rebuild-linux: 2501-5000 labels Mar 9, 2019

Mic92 reviewed Mar 12, 2019

View reviewed changes

matthewbauer reviewed Mar 12, 2019

View reviewed changes

Mic92 mentioned this pull request Mar 13, 2019

[WIP] Rust cross compilation #50866

Closed

13 tasks

illegalprime changed the title ~~[WIP] Rust cross compilation (1.32.0)~~ Rust Cross Compilation (1.32.0) Mar 13, 2019

illegalprime commented Mar 16, 2019

View reviewed changes

illegalprime mentioned this pull request Mar 17, 2019

[discussion] systems: add armv7 target #57823

Closed

10 tasks

illegalprime force-pushed the rust-cross branch from 2d6d180 to 6d06924 Compare March 27, 2019 02:52

illegalprime added 2 commits March 29, 2019 10:28

rust: Fix grammar in comment

7f6de09

cargo: No need to skip space around = at Nix level

3c93018

Ericson2314 force-pushed the rust-cross branch from 6d06924 to 2a67071 Compare March 29, 2019 14:46

Mic92 and others added 8 commits March 29, 2019 11:11

rust: fix cross-compilation

912dca1

rust: move releaseDir to target/release in the buildPhase

60761e6

rust: allow building in debug or release modes

bdd3c3f

rust: set PKG_CONFIG_ALLOW_CROSS conditionally in buildRustPackage

f93470d

rust: Don't force gcc6

54e9082

rust: disable stripping to prevent bad cross rlibs

8391528

synapse-bt: openssl should be build input

a891ae5

librsvg: make-fonts-conf: update with correct native deps

3bcede8

Ericson2314 force-pushed the rust-cross branch from 2a67071 to 3bcede8 Compare March 29, 2019 15:13

Ericson2314 approved these changes Mar 29, 2019

View reviewed changes

Ericson2314 changed the base branch from master to staging April 1, 2019 04:40

GrahamcOfBorg added the 2.status: merge conflict This PR has merge conflicts with the target branch label Apr 1, 2019

Merge branch 'staging' into rust-cross

716503d

GrahamcOfBorg removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Apr 1, 2019

Ericson2314 merged commit b621207 into NixOS:staging Apr 1, 2019

illegalprime mentioned this pull request Apr 23, 2019

[WIP] gcc: put target-specific libs in lib output #58606

Closed

14 tasks

c0bw3b mentioned this pull request May 22, 2019

buildRustPackage: fix cargo flag for release/debug build #61521

Merged

10 tasks

arcnmx mentioned this pull request Jun 22, 2019

rust: fix building in debug mode #63666

Closed

10 tasks

AtkinsChang mentioned this pull request Jun 26, 2019

rust: fix rust build-support debug build type #63762

Closed

10 tasks

Mic92 mentioned this pull request Feb 5, 2021

modules-closure.sh: rewrite to use modinfo + small rust program #109094

Open

10 tasks

Rust Cross Compilation (1.32.0) #56540

Rust Cross Compilation (1.32.0) #56540

Conversation

illegalprime commented Feb 28, 2019 • edited Loading

Motivation for this change

Things done

illegalprime commented Mar 11, 2019 • edited Loading

Mic92 commented Mar 12, 2019

illegalprime commented Mar 12, 2019 • edited Loading

Mic92 commented Mar 12, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

illegalprime commented Mar 12, 2019

Mic92 commented Mar 12, 2019 • edited Loading

illegalprime commented Mar 12, 2019

Mic92 commented Mar 12, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ericson2314 Mar 13, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

illegalprime commented Mar 13, 2019

Mic92 commented Mar 13, 2019

illegalprime commented Mar 13, 2019 • edited Loading

illegalprime commented Mar 13, 2019 • edited Loading

Ericson2314 commented Mar 13, 2019 • edited Loading

Ericson2314 commented Mar 13, 2019

Ericson2314 commented Mar 14, 2019

illegalprime commented Mar 14, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mic92 Mar 20, 2019 • edited Loading

Choose a reason for hiding this comment

illegalprime commented Mar 25, 2019

Ericson2314 commented Mar 26, 2019 • edited Loading

Ericson2314 commented Mar 26, 2019 • edited Loading

illegalprime commented Mar 27, 2019

Ericson2314 left a comment

Choose a reason for hiding this comment

Ericson2314 commented Mar 29, 2019

Ericson2314 commented Apr 1, 2019 • edited Loading

illegalprime commented Feb 28, 2019 •

edited

Loading

illegalprime commented Mar 11, 2019 •

edited

Loading

illegalprime commented Mar 12, 2019 •

edited

Loading

Mic92 commented Mar 12, 2019 •

edited

Loading

Mic92 commented Mar 12, 2019 •

edited

Loading

Ericson2314 Mar 13, 2019 •

edited

Loading

illegalprime commented Mar 13, 2019 •

edited

Loading

illegalprime commented Mar 13, 2019 •

edited

Loading

Ericson2314 commented Mar 13, 2019 •

edited

Loading

Mic92 Mar 20, 2019 •

edited

Loading

Ericson2314 commented Mar 26, 2019 •

edited

Loading

Ericson2314 commented Mar 26, 2019 •

edited

Loading

Ericson2314 commented Apr 1, 2019 •

edited

Loading