-
-
Notifications
You must be signed in to change notification settings - Fork 13.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
build-support/cc-wrapper: pass in non-existent --sysroot= to untangle… #210004
Conversation
… from libc I would like to add an extra `gcc` build step during linux bootstrap (NixOS#208412). This makes it early bootstrap compiler linked and targeted against `bootstrapTools` `glibc` including it's headers. Without this change `gcc`'s spec files always prefer `bootstrapTools` `glibc` for header search path (passed in as --with-native-system-header-dir=). We'can't override it with: - `-I` option as it gets stacked before gcc-specific headers, we need to keep glibc headers after gcc as gcc cleans namespace up for C standard by using #include_next and by undefining system macros. - `-idirafter` option as it gets appended after existing `glibc`-includes This `--sysroot=/nix/store/does/not/exist` hack allows us to remove existing `glibc` headers and add new ones with `-idirafter`. We use `cc-cflags-before` instead of `libc-cflags` to allow user to define their own `--sysroot=` (like `firefox` does). To keep it working prerequisite cross-symlink in gcc.libs is required: NixOS#209153
This commit broke uboot packages on staging-next. See https://hydra.nixos.org/eval/1789657?filter=uboot&compare=1789649.
|
Sounds plausible. The failure is not outright obvious:
Looking. |
yeah, include order did change and broke On FHS systems include order is:
(libc includes come before On
(libc includes come after Should be easy to workaround just for u-boot. I'll propose u-boot specific fix tomorrow morning. |
`ubootTools` build broke after NixOS#210004 where we started dropping default libc include path and switched to `-idirafter` way of specifying libc headers. Unfortunately the way it's implemented it injects -idirafter after user's flags, not before. That allows users to inject their paths before libc include paths, not after (as it would notmally happen). The change works it around for u-boot by pulling -idirafter libc flags before user's flags.
`ubootTools` build broke after NixOS#210004 where we started dropping default libc include path and switched to `-idirafter` way of specifying libc headers. Unfortunately the way it's implemented it injects -idirafter after user's flags, not before. That allows users to inject their paths before libc include paths, not after (as it would notmally happen). The change works it around for u-boot by pulling -idirafter libc flags before user's flags.
Proposed possible |
`ubootTools` build broke after NixOS#210004 where we started dropping default libc include path and switched to `-idirafter` way of specifying libc headers. Unfortunately the way it's implemented it injects -idirafter after user's flags, not before. That allows users to inject their paths before libc include paths, not after (as it would notmally happen). The change works it around for u-boot by pulling -idirafter libc flags before user's flags.
Also EDIT: also confirmed for packages: |
Will have a more detailed look in 1-2 hours to see how I broke |
False alarm. |
Proposed
Looking at |
Note: It was forgotten in non-lib output:
I'll think of a local |
After NixOS#210004 `usbmuxd2` started failing to build as: usbmuxd2-unstable> .../ld: cannot find -lstdc++fs: No such file or directory usbmuxd2-unstable> clang-11: error: linker command failed with exit code 1 (use -v to see invocation) This started happening because NixOS#210004 exposed a long-standing bug in `gcc` derivation: `cc.lib` is missing `libstdc++fs` library: $ find $(nix-build --no-link -A stdenv.cc.cc.lib) | fgrep libstdc | unnix /<<NIX>>/gcc-11.3.0-lib/lib/libstdc++fs.la /<<NIX>>/gcc-11.3.0-lib/lib/libstdc++.la /<<NIX>>/gcc-11.3.0-lib/lib/libstdc++.so.6.0.29 /<<NIX>>/gcc-11.3.0-lib/lib/libstdc++.so /<<NIX>>/gcc-11.3.0-lib/lib/libstdc++.so.6 It was not moved from `cc.out` output: $ find $(nix-build --no-link -A stdenv.cc.cc) | fgrep libstdc | unnix /<<NIX>>/gcc-11.3.0/lib/libstdc++.a /<<NIX>>/gcc-11.3.0/lib/libstdc++fs.a This change adds `cc` library lookup path back to `staging-next` until `gcc` is fixed.`
Proposed Looking at |
Proposed Looking at |
More fallout:
|
…or clang After NixOS#210004 `usbmuxd2` started failing to build as: usbmuxd2-unstable> .../ld: cannot find -lstdc++fs: No such file or directory usbmuxd2-unstable> clang-11: error: linker command failed with exit code 1 (use -v to see invocation) This started happening because NixOS#210004 exposed a long-standing bug in `gcc` derivation: `cc.lib` is missing `libstdc++fs` library: $ find $(nix-build --no-link -A stdenv.cc.cc.lib) | fgrep libstdc | unnix /<<NIX>>/gcc-11.3.0-lib/lib/libstdc++fs.la /<<NIX>>/gcc-11.3.0-lib/lib/libstdc++.la /<<NIX>>/gcc-11.3.0-lib/lib/libstdc++.so.6.0.29 /<<NIX>>/gcc-11.3.0-lib/lib/libstdc++.so /<<NIX>>/gcc-11.3.0-lib/lib/libstdc++.so.6 It was not moved from `cc.out` output: $ find $(nix-build --no-link -A stdenv.cc.cc) | fgrep libstdc | unnix /<<NIX>>/gcc-11.3.0/lib/libstdc++.a /<<NIX>>/gcc-11.3.0/lib/libstdc++fs.a This change adds `cc` library lookup path back to `staging-next` until `gcc` is fixed.`
`ipxe` build broke after NixOS#210004 where we started dropping default libc include path and switched to `-idirafter` way of specifying libc headers. Unfortunately the way it's implemented it injects `-idirafter` after user's flags, not before. That allows users to inject their paths before libc include paths, not after (as it would notmally happen). The change works it around for `ipxe` by pulling `-idirafter` libc flags before user's flags.
`wimboot` build broke after NixOS#210004 where we started dropping default libc include path and switched to `-idirafter` way of specifying libc headers. Unfortunately the way it's implemented it injects `-idirafter` after user's flags, not before. That allows users to inject their paths before libc include paths, not after (as it would notmally happen). The change works it around for `wimboot` by pulling `-idirafter` libc flags before user's flags.
After NixOS#210004 `dmd` started failing build as: ld: warning: libm.so.6, needed by ./generated/linux/release/64/lib.so, not found (try using -rpath or -rpath-link) ld: /build/druntime/generated/linux/release/64/libdruntime.so: undefined reference to `log10@GLIBC_2.2.5' This happens because --sysroot=/nix/store/does/not/exist removes not just include headers by prefixing wrong path, but also removes RUNPATH dependencies of linked libraries. It's an unintended effect. Restore the build by reversing the effect with --sysroot=/.
…untangle from libc" This reverts commit 8c80bd0 ("build-support/cc-wrapper: pass in non-existent --sysroot= to untangle from libc"). This change was good in spirit: we caught a few genuine problems with `scons` based packages (`godot`, `fluxus`) and unexpected `-idirafter` includes in various boot loadres (`ipxe`, wimboot`): NixOS#210004 (comment) Unfortunately `--sysroot=` also has a negative impact on libary search order for DT_NEEDED libraries and RUNPATHs of linked libraries. This unexpectedly broke `dmd`, `d-seams`, `llvmPackages_rocm.compiler-rt`). An interesting case of unexpected breakage is `usbmuxd2` where the bug exposed incomplete library move on `libstdc++fs` in `gcc`. The library breakage is very non-intuitive (on top of already unusual layout of `cc-wrapper` driver). Let's revert this change for now. Once it lands we can undo `--sysroot=/` workarounds merged for `staging-next`.
On top of the above fixes for There are too many subtle changes that need to be handled one by one: library search paths, include search path order, bootstrap overrides and sequencing. |
…untangle from libc" This reverts commit 8c80bd0 ("build-support/cc-wrapper: pass in non-existent --sysroot= to untangle from libc"). This change was good in spirit: we caught a few genuine problems with `scons` based packages (`godot`, `fluxus`) and unexpected `-idirafter` includes in various boot loadres (`ipxe`, wimboot`): NixOS#210004 (comment) Unfortunately `--sysroot=` also has a negative impact on libary search order for DT_NEEDED libraries and RUNPATHs of linked libraries. This unexpectedly broke `dmd`, `d-seams`, `llvmPackages_rocm.compiler-rt`). An interesting case of unexpected breakage is `usbmuxd2` where the bug exposed incomplete library move on `libstdc++fs` in `gcc`. The library breakage is very non-intuitive (on top of already unusual layout of `cc-wrapper` driver). Let's revert this change for now. Once it lands we can undo `--sysroot=/` workarounds merged for `staging-next`.
…untangle from libc" This reverts commit 8c80bd0 ("build-support/cc-wrapper: pass in non-existent --sysroot= to untangle from libc"). This change was good in spirit: we caught a few genuine problems with `scons` based packages (`godot`, `fluxus`) and unexpected `-idirafter` includes in various boot loadres (`ipxe`, wimboot`): NixOS#210004 (comment) Unfortunately `--sysroot=` also has a negative impact on libary search order for DT_NEEDED libraries and RUNPATHs of linked libraries. This unexpectedly broke `dmd`, `d-seams`, `llvmPackages_rocm.compiler-rt`). An interesting case of unexpected breakage is `usbmuxd2` where the bug exposed incomplete library move on `libstdc++fs` in `gcc`. The library breakage is very non-intuitive (on top of already unusual layout of `cc-wrapper` driver). Let's revert this change for now. Once it lands we can undo `--sysroot=/` workarounds merged for `staging-next`.
#### Immediate Benefits - Allow `gcc11` on `aarch64` - No more copying `libgcc_s` out of the bootstrap-files or other derivations - No more [static `lib{mpfr,mpc,gmp,isl}.a` hack](https://github.com/NixOS/nixpkgs/blob/2f1948af9c984ebb82dfd618e67dc949755823e2/pkgs/stdenv/linux/default.nix#L380) - *Zero* additional `gcc` builds (stage1+stage2+stageCompare) - The `gcc` derivation builds `gcc` once instead of three times. - The libraries that are linked into the final `pkgs.gcc` (`mpfr`, `mpc`, `gmp`, `isl`, `glibc`) are built by `stdenv.__bootPkgs.gcc` rather than by the `bootstrapFiles`. No more Frankenstein compiler! - stageCompare runs **concurrently** with (not in series with) with `stdenv`'s dependees. - Many other `stdenv` hacks eliminated. - `gcc` and `clang` share the same codepath for more of `cc-wrapper`. - Makes the cross and native codepaths much more similar -- another step towards "cross by default". Note that *all* the changes in this PR are controlled by flags; no old codepaths need to be removed until/if we're completely certain that this is the right way to go. #### Future Benefits - This should allow using a [foreign] `bootstrap-files` so long as `hostPlatform.canExecute bootstrapFiles`. - There will be an "avalanche of simplification" when we set `enableGccExternalBootstrap=true` and run dead code elimination. It's really quite a huge amount of code that goes away. Native-gcc has its own special codepath in so many places, while cross-gcc and clang work the same way (and are much simpler). - This should allow each of the libraries that ship with `gcc` (`lib{backtrace,atomic,cc1,decnumber,ffi,gomp,iberty,offloadatomic,quadmath,sanitizer,ssp,stdc++-v3,vtv}`) to be built in separate (one-liner) derivations which `inherit src;` from `gcc`. - Building `libstdc++-v3` in a separate derivation will eliminate a lot of accidental-reference-to-the-`bootstrapFiles` landmines. #### Incorporates - NixOS#209054 - NixOS#210004 - NixOS#36948 (unreverted) - NixOS#210325 - NixOS#210118 - NixOS#210132 - NixOS#210109 #### Closes - Closes NixOS#208412 - Closes NixOS#108111 - Closes NixOS#108305 - Closes NixOS#201254 #### Build history - First successful builds (stage1/stage2): - powerpc64le-linux at 9c7e9ef - x86_64-linux at 9c7e9ef - aarch64-linux at 4d5bc7d - First successful comparisons (stageCompare): - at 81949cf - [aarch64-linux][aarch64-compare-ofborg] - [x86\_64-linux][amd64-compare-ofborg] #### Credits This project was made possible by three important insights, none of which were mine: 1. @Ericson2314 was the first to advocate for this change, and probably the first to appreciate its advantages. External bootstrap is "cross by default". 2. @trofi has figured out a lot about how to get gcc to not mix up the copy of `libstdc++` that it depends on with the copy that it builds. Now that gcc is written in C++, it depends on `libstdc++`, builds a copy of `libstdc++`, and builds auxiliary products (like `libplugin`) which depend on `libstdc++`. @trofi developed two important techniques for keeping this straight: the use of a [nonexistent sysroot] and moving the `bootstrapFiles`' `libstdc++` into a [versioned directory]. Without these two discoveries, external bootstrap would be impossible, because the final gcc would still have references to the `bootstrapFiles`. 3. Using the undocumented variable [`user-defined-trusted-dirs`] when building glibc. When glibc `dlopen()`s `libgcc_s.so`, it uses a completely different and totally special set of rules for finding `libgcc_s.so`. This trick is the only way we can put `libgcc_s.so` in its own separate outpath without creating circular dependencies or dependencies on the bootstrapFiles. I would never have guessed to use this (or that it existed!) if it were not for a [comment in guix] which @Mic92 [mentioned]. My own role in this PR was basically: being available to go on a coding binge at an opportune moment, so we wouldn't waste a [crisis]. [aarch64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662822938 [amd64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662825857 [nonexistent sysroot]: NixOS#210004 [versioned directory]: NixOS#209054 [`user-defined-trusted-dirs`]: https://sourceware.org/legacy-ml/libc-help/2013-11/msg00026.html [comment in guix]: https://github.com/guix-mirror/guix/blob/5e4ec8218142eee8e6e148e787381a5ef891c5b1/gnu/packages/gcc.scm#L253 [mentioned]: NixOS#210112 (comment) [crisis]: NixOS#108305 [foreign]: NixOS#170857 (comment)
After NixOS/nixpkgs#210004 `dmd` started failing build as: ld: warning: libm.so.6, needed by ./generated/linux/release/64/lib.so, not found (try using -rpath or -rpath-link) ld: /build/druntime/generated/linux/release/64/libdruntime.so: undefined reference to `log10@GLIBC_2.2.5' This happens because --sysroot=/nix/store/does/not/exist removes not just include headers by prefixing wrong path, but also removes RUNPATH dependencies of linked libraries. It's an unintended effect. Restore the build by reversing the effect with --sysroot=/.
After NixOS/nixpkgs#210004 `dmd` started failing build as: ld: warning: libm.so.6, needed by ./generated/linux/release/64/lib.so, not found (try using -rpath or -rpath-link) ld: /build/druntime/generated/linux/release/64/libdruntime.so: undefined reference to `log10@GLIBC_2.2.5' This happens because --sysroot=/nix/store/does/not/exist removes not just include headers by prefixing wrong path, but also removes RUNPATH dependencies of linked libraries. It's an unintended effect. Restore the build by reversing the effect with --sysroot=/.
#### Summary By default, when you type `make`, GCC will compile itself three times. This PR inhibits that behavior by configuring GCC with `--disable-bootstrap`, and reimplements the triple-rebuild using Nix rather than `make`/`sh`. #### Immediate Benefits - Allow `gcc11` and `gcc12` on `aarch64` (without needing new `bootstrapFiles`) - Faster stdenv rebuilds: the third compilation of gcc (i.e. stageCompare) is no longer a `drvInput` of the final stdenv. This allows Nix to build stageCompare in parallel with the rest of nixpkgs instead of in series. - No more copying `libgcc_s` out of the bootstrap-files or other derivations - No more Frankenstein compiler: the final gcc and the libraries it links against (mpfr, mpc, isl, glibc) are all built by the same compiler (xgcc) instead of a mixture of the bootstrapFiles' compiler and xgcc. - No more [static lib{mpfr,mpc,gmp,isl}.a hack] - Many other small `stdenv` hacks eliminated - `gcc` and `clang` share the same codepath for more of `cc-wrapper`. #### Future Benefits - This should allow using a [foreign] `bootstrap-files` so long as `hostPlatform.canExecute bootstrapFiles`. - This should allow each of the libraries that ship with `gcc` (lib{backtrace, atomic, cc1, decnumber, ffi, gomp, iberty, offloadatomic, quadmath, sanitizer, ssp, stdc++-v3, vtv}) to be built in separate (one-liner) derivations which `inherit src;` from `gcc`, much like NixOS#132343 #### Incorporates - NixOS#210004 - NixOS#36948 (unreverted) - NixOS#210325 - NixOS#210118 - NixOS#210132 - NixOS#210109 - NixOS#213909 - NixOS#216136 - NixOS#216237 - NixOS#210019 - NixOS#216232 - NixOS#216016 - NixOS#217977 - NixOS#217995 #### Closes - Closes NixOS#108305 - Closes NixOS#108111 - Closes NixOS#201254 - Closes NixOS#208412 #### Credits This project was made possible by three important insights, none of which were mine: 1. @Ericson2314 was the first to advocate for this change, and probably the first to appreciate its advantages. Nix-driven (external) bootstrap is "cross by default". 2. @trofi has figured out a lot about how to get gcc to not mix up the copy of `libstdc++` that it depends on with the copy that it builds, by moving the `bootstrapFiles`' `libstdc++` into a [versioned directory]. This allows a Nix-driven bootstrap of gcc without the final gcc would still having references to the `bootstrapFiles`. 3. Using the undocumented variable [`user-defined-trusted-dirs`] when building glibc. When glibc `dlopen()`s `libgcc_s.so`, it uses a completely different and totally special set of rules for finding `libgcc_s.so`. This trick is the only way we can put `libgcc_s.so` in its own separate outpath without creating circular dependencies or dependencies on the bootstrapFiles. I would never have guessed to use this (or that it existed!) if it were not for a [comment in guix] which @Mic92 [mentioned]. My own role in this PR was basically: being available to go on a coding binge at an opportune moment, so we wouldn't waste a [crisis]. [aarch64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662822938 [amd64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662825857 [nonexistent sysroot]: NixOS#210004 [versioned directory]: NixOS#209054 [`user-defined-trusted-dirs`]: https://sourceware.org/legacy-ml/libc-help/2013-11/msg00026.html [comment in guix]: https://github.com/guix-mirror/guix/blob/5e4ec8218142eee8e6e148e787381a5ef891c5b1/gnu/packages/gcc.scm#L253 [mentioned]: NixOS#210112 (comment) [crisis]: NixOS#108305 [foreign]: NixOS#170857 (comment) [static lib{mpfr,mpc,gmp,isl}.a hack]: https://github.com/NixOS/nixpkgs/blob/2f1948af9c984ebb82dfd618e67dc949755823e2/pkgs/stdenv/linux/default.nix#L380
… from libc
I would like to add an extra
gcc
build step during linux bootstrap (#208412). This makes it early bootstrap compiler linked and targeted againstbootstrapTools
glibc
including it's headers.Without this change
gcc
's spec files always preferbootstrapTools
glibc
for header search path (passed in as --with-native-system-header-dir=). We'can't override it with:-I
option as it gets stacked before gcc-specific headers, we need to keep glibc headers after gcc as gcc cleans namespace up for C standard by using #include_next and by undefining system macros.-idirafter
option as it gets appended after existingglibc
-includesThis
--sysroot=/nix/store/does/not/exist
hack allows us to remove existingglibc
headers and add new ones with-idirafter
.We use
cc-cflags-before
instead oflibc-cflags
to allow user to define their own--sysroot=
(likefirefox
does).To keep it working prerequisite cross-symlink in gcc.libs is required: #209153
Description of changes
Things done
sandbox = true
set innix.conf
? (See Nix manual)nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD"
. Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/
)nixos/doc/manual/md-to-db.sh
to update generated release notes