Closure size of musl C++ programs bloated due to location of libstdc++ #75476
Comments
|
cc @Ericson2314 and perhaps @dtzWill |
|
I think that #58606 would fix this. |
|
Thank you for your contributions. This has been automatically marked as stale because it has had no activity for 180 days. If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity. Here are suggestions that might help resolve this more quickly:
|
stale
bot
added
the
2.status: stale
|
The situation is better with #81844, but still pulls in glibc: |
stale
bot
removed
the
2.status: stale
|
Also: |
|
So we need a |
|
no i think we just need to strip / nuke-refs on libgcc_s.so.1 and libquadmath.so.0.0.0 |
|
Oh, sweet. |
|
I marked this as stale due to inactivity. → More info |
stale
bot
added
the
2.status: stale
|
I ran into this today, this isn't stale for me |
stale
bot
removed
the
2.status: stale
|
So the current problem is the wrapper scripts depend on bash, which depends on a glibc built version of bash Removing the wrappers from the -dev package of course solves the issue... now musl no longer depends on glibc in my local derivation, but the wrappers might be useful to someone, somewhere. Any other of these chicken/egg type problems solved already in nix? |
|
Looking deeper, my issue seems to in reality be caused by the crossSystem based gcc-lib being the -debug variant, which is done because gcc doesn't strip cross compilers for some reason which is unclear to me. The debug variant needs musl-dev, which needs bash, which needs the host libc from which that bash is built on (could be musl if pkgsMusl is used potentially). In all cases this is, I think, unwanted direct dependencies which in all likelihood don't need to exist. Just because the debug symbols are there doesn't mean the resulting closure should directly require the package with the derivation with the headers in there? Is there some sane way to exclude forcefully some dependencies from carrying forward? |
|
Should be fixed with #108060. |
|
@matthewbauer thank you so so much for fixing this issue, I'll test it out tomorrow! |
|
@matthewbauer as mentioned in the PR, this certainly fixed the issue for when the target and host arch is the same, but not when they differ as you'd expect given the change you mentioned, I spent a bit of time trying to dig into builder.sh on how I might strip the resulting libraries, but when enabling strip wholesale the crossSystem = "aarch64-unknown-linux-musl"; fails to compile a simple hellocpp project at the moment. I didn't see why that was, so I'm trying a different route in removing -g based on a new nix parameter to the gcc derivation, and seeing where that gets me |
|
I've really dug into the issue of strip here and its clear that the problem is, I believe, that the host toolchains strip is being used for the cross compiled gcc, this is fine for the binaries but for the libraries which are for the target architecture we likely want to use to the cross-compiler toolchains binutils variant of strip to do the job. strip.sh and mkDerivation don't directly provide any method for doing this that I could find. A custom fixup hook could be written for gcc that does this exactly thing, given a list of host strip dirs and target strip dirs, along with the appropriate binaries. Any thoughts on the above thinking? if so I'll write up an issue for it and try my hand at solving the issue. |
|
This particular issue is fixed from what I've seen, with a #109295 noting that things still aren't perfect when cross compiling to a different arch and why |
When cross compiling to the same kernel / arch combination, it is safe to use strip of libraries. This happens when cross-compiling musl programs. dontStrip is now set in each gcc compiler instead of in gcc/builder.sh. Fixes NixOS#75476
Describe the bug
When cross-compiling C++ programs for musl targets, the resulting closure is significantly larger (about 750-800MB extra) than programs compiled for my host machine (generic x86_64 linux). This closure includes both musl-gcc and my host gcc toolchain thanks to the cross-compilation.
I've done some digging and I believe that the root cause is due to the location of
libstdc++.soandlibstdc++.ain the musl stdenv.To Reproduce
Steps to reproduce the behavior:
nix-build -A hostnix-build -A musland compare closure sizes.Expected behavior
The closure for a C++-compiled binary should only rely on the gcc-lib output, not the full gcc itself.
Additional context
Here's my sleuthing on the problem. When building a C++ "hello world" with my host toolchain, I get the following closure:
However, if I build the same program in cross-compiling mode for musl gcc, everything gets carried along:
From the output, it's pretty obvious that the entire closure of GCC has been carried along.
What I eventually noticed was that
lddsays that libstdc++ is in gcc-lib for the host toolchain:However, the same is not true for cross-compiling against musl:
Ultimately, what I see happening is that C++-compiled programs via the host toolchain have a
RUNPATHthat points topkgs.stdenv.cc.cc.lib:The second path in the list is this:
On cross-compiled programs, however, the
RUNPATHpoints to$outinstead:The REPL bears this out:
I was able to dig up two potentially-related issues. I didn't bump them because I wasn't sure how much like mine they were, but it's possible they've been bitten by this same bug:
Metadata
Please run
nix run nixpkgs.nix-info -c nix-info -mand paste the result.Maintainer information:
The text was updated successfully, but these errors were encountered: