Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build-support/cc-wrapper: pass in non-existent --sysroot= to untangle… #210004

Merged
merged 1 commit into from
Jan 14, 2023

Conversation

trofi
Copy link
Contributor

@trofi trofi commented Jan 10, 2023

… from libc

I would like to add an extra gcc build step during linux bootstrap (#208412). This makes it early bootstrap compiler linked and targeted against bootstrapTools glibc including it's headers.

Without this change gcc's spec files always prefer bootstrapTools glibc for header search path (passed in as --with-native-system-header-dir=). We'can't override it with:

  • -I option as it gets stacked before gcc-specific headers, we need to keep glibc headers after gcc as gcc cleans namespace up for C standard by using #include_next and by undefining system macros.
  • -idirafter option as it gets appended after existing glibc-includes

This --sysroot=/nix/store/does/not/exist hack allows us to remove existing glibc headers and add new ones with -idirafter.

We use cc-cflags-before instead of libc-cflags to allow user to define their own --sysroot= (like firefox does).

To keep it working prerequisite cross-symlink in gcc.libs is required: #209153

Description of changes
Things done
  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 23.05 Release Notes (or backporting 22.11 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
    • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
  • Fits CONTRIBUTING.md.

… from libc

I would like to add an extra `gcc` build step during linux bootstrap
(NixOS#208412). This makes it early
bootstrap compiler linked and targeted against `bootstrapTools` `glibc`
including it's headers.

Without this change `gcc`'s spec files always prefer `bootstrapTools` `glibc`
for header search path (passed in as --with-native-system-header-dir=). We'can't
override it with:

- `-I` option as it gets stacked before gcc-specific headers, we need to keep
  glibc headers after gcc as gcc cleans namespace up for C standard by using
  #include_next and by undefining system macros.
- `-idirafter` option as it gets appended after existing `glibc`-includes

This `--sysroot=/nix/store/does/not/exist` hack allows us to remove existing
`glibc` headers and add new ones with `-idirafter`.

We use `cc-cflags-before` instead of `libc-cflags` to allow user to define
their own `--sysroot=` (like `firefox` does).

To keep it working prerequisite cross-symlink in gcc.libs is required:
NixOS#209153
@trofi trofi merged commit 67a3a14 into NixOS:staging Jan 14, 2023
@trofi trofi deleted the cc-fake-sysroot branch January 14, 2023 08:18
@mweinelt
Copy link
Member

This commit broke uboot packages on staging-next. See https://hydra.nixos.org/eval/1789657?filter=uboot&compare=1789649.

8c80bd08b7e39229947d55104d1871f5066437d9 is the first bad commit
commit 8c80bd08b7e39229947d55104d1871f5066437d9
Author: Sergei Trofimovich <slyich@gmail.com>
Date:   Tue Jan 10 07:43:51 2023 +0000

    build-support/cc-wrapper: pass in non-existent --sysroot= to untangle from libc
    
    I would like to add an extra `gcc` build step during linux bootstrap
    (https://github.com/NixOS/nixpkgs/issues/208412). This makes it early
    bootstrap compiler linked and targeted against `bootstrapTools` `glibc`
    including it's headers.
    
    Without this change `gcc`'s spec files always prefer `bootstrapTools` `glibc`
    for header search path (passed in as --with-native-system-header-dir=). We'can't
    override it with:
    
    - `-I` option as it gets stacked before gcc-specific headers, we need to keep
      glibc headers after gcc as gcc cleans namespace up for C standard by using
      #include_next and by undefining system macros.
    - `-idirafter` option as it gets appended after existing `glibc`-includes
    
    This `--sysroot=/nix/store/does/not/exist` hack allows us to remove existing
    `glibc` headers and add new ones with `-idirafter`.
    
    We use `cc-cflags-before` instead of `libc-cflags` to allow user to define
    their own `--sysroot=` (like `firefox` does).
    
    To keep it working prerequisite cross-symlink in gcc.libs is required:
    https://github.com/NixOS/nixpkgs/pull/209153

 pkgs/build-support/cc-wrapper/default.nix | 7 +++++++
 1 file changed, 7 insertions(+)

@trofi
Copy link
Contributor Author

trofi commented Jan 26, 2023

Sounds plausible. The failure is not outright obvious:

  HOSTCC  tools/bmp_logo
  HOSTCC  tools/gdb/gdbsend.o
In file included from ././include/compiler.h:23,
                 from <command-line>:
include/errno.h:28:22: error: missing binary operator before token "("
   28 | #if CONFIG_IS_ENABLED(ERRNO_STR)
      |                      ^
In file included from include/stdlib.h:9,
                 from ././include/compiler.h:24,
                 from <command-line>:
include/malloc.h:886:22: error: missing binary operator before token "("
  886 | #if CONFIG_IS_ENABLED(SYS_MALLOC_SIMPLE)
      |                      ^
include/malloc.h:984:8: error: unknown type name 'ulong'
  984 | extern ulong mem_malloc_start;
      |        ^~~~~

Looking.

@trofi
Copy link
Contributor Author

trofi commented Jan 27, 2023

yeah, include order did change and broke #include_next.

On FHS systems include order is:

 /usr/lib/gcc/x86_64-pc-linux-gnu/12/include
 /usr/lib/gcc/x86_64-pc-linux-gnu/12/include-fixed
 /usr/include
 include
 ./arch/sandbox/include

(libc includes come before -idirafter ones)

On staging-next:

 /run/current-system/sw/include
 /nix/store/g7z994p16v096ryq0ic1jmgywcylhq40-ncurses-6.4-dev/include
 /nix/store/7ksr2nxylzg7ni895vhyw7ia2qbg2c9d-dtc-1.6.1/include
 /nix/store/76c2420qdy7gg4pdvspvsafxnpwp1qm1-flex-2.6.4/include
 /nix/store/7pg5j6k1m9y2bn2hr05j0v4nasq23xmr-openssl-3.0.7-dev/include
 /nix/store/m0kcrs788zwc6mp3qalfdyzlkw999dhm-python3-3.10.9-env/include
 /nix/store/l68yk5n1nz6qxck1f0b4zbcmbcvi8px8-util-linux-minimal-2.38.1-dev/include
 /nix/store/bycbddw4j1nkf3nipwi0qlkpfh0h2z5w-gnutls-3.7.8-dev/include
 /nix/store/ain09ii77w4c006y2y1852g36mva7haa-nettle-3.8.1-dev/include
 /nix/store/4j2zssk8nq7alxhqpna9ps6hxz0fmi8m-gmp-with-cxx-6.2.1-dev/include
 /nix/store/v1nxrwb4ls09sbbrwlkgxs7dlw8paksh-gcc-11.3.0/lib/gcc/x86_64-unknown-linux-gnu/11.3.0/include
 /nix/store/v1nxrwb4ls09sbbrwlkgxs7dlw8paksh-gcc-11.3.0/include
 /nix/store/v1nxrwb4ls09sbbrwlkgxs7dlw8paksh-gcc-11.3.0/lib/gcc/x86_64-unknown-linux-gnu/11.3.0/include-fixed
 include
 ./arch/sandbox/include
 /nix/store/nasgvbblxi25r6y1s82il4m0m4ghib68-glibc-2.35-224-dev/include

(libc includes come after -idirafter ones).

Should be easy to workaround just for u-boot. I'll propose u-boot specific fix tomorrow morning.

trofi added a commit to trofi/nixpkgs that referenced this pull request Jan 27, 2023
`ubootTools` build broke after NixOS#210004
where we started dropping default libc include path and switched to
`-idirafter` way of specifying libc headers.

Unfortunately the way it's implemented it injects -idirafter after
user's flags, not before. That allows users to inject their paths before
libc include paths, not after (as it would notmally happen).

The change works it around for u-boot by pulling -idirafter libc flags
before user's flags.
trofi added a commit to trofi/nixpkgs that referenced this pull request Jan 27, 2023
`ubootTools` build broke after NixOS#210004
where we started dropping default libc include path and switched to
`-idirafter` way of specifying libc headers.

Unfortunately the way it's implemented it injects -idirafter after
user's flags, not before. That allows users to inject their paths before
libc include paths, not after (as it would notmally happen).

The change works it around for u-boot by pulling -idirafter libc flags
before user's flags.
@trofi
Copy link
Contributor Author

trofi commented Jan 27, 2023

Proposed possible -idirafter workaround for staging-next as #212919

trofi added a commit to trofi/nixpkgs that referenced this pull request Jan 27, 2023
`ubootTools` build broke after NixOS#210004
where we started dropping default libc include path and switched to
`-idirafter` way of specifying libc headers.

Unfortunately the way it's implemented it injects -idirafter after
user's flags, not before. That allows users to inject their paths before
libc include paths, not after (as it would notmally happen).

The change works it around for u-boot by pulling -idirafter libc flags
before user's flags.
@vcunat
Copy link
Member

vcunat commented Jan 27, 2023

Also rmlint build got broken by this: https://hydra.nixos.org/build/206664410
(I wasn't sure so I built it on current staging-next with just this commit reverted.)
And there are several other regressed builds with similarly looking linker errors.

EDIT: also confirmed for packages: usbmuxd2, dmd, godot.

@trofi
Copy link
Contributor Author

trofi commented Jan 27, 2023

rmlint exposes something that look like a glib underlinking bug (and the fix will likely be in glib). If glib rebuild is already big we can revert this change and take the full hit of rebuilding everything again.

Will have a more detailed look in 1-2 hours to see how I broke glib / rmlint.

@trofi
Copy link
Contributor Author

trofi commented Jan 27, 2023

False alarm. glib looks fine at least from underlinking standpoint. It's something related to rmlint's handling of CFLAGS/LDFLAGS. Still looking.

@trofi
Copy link
Contributor Author

trofi commented Jan 27, 2023

Proposed rmlint fix as:

Looking at usbmuxd2.

@trofi
Copy link
Contributor Author

trofi commented Jan 27, 2023

usbmuxd2 exposed the fact that our gcc's lib lacks libstdc++fs.la:

$ find $(nix-build --no-link -A stdenv.cc.cc.lib) | fgrep libstdc | unnix

/<<NIX>>/gcc-11.3.0-lib/lib/libstdc++fs.la

/<<NIX>>/gcc-11.3.0-lib/lib/libstdc++.la
/<<NIX>>/gcc-11.3.0-lib/lib/libstdc++.so.6.0.29
/<<NIX>>/gcc-11.3.0-lib/lib/libstdc++.so
/<<NIX>>/gcc-11.3.0-lib/lib/libstdc++.so.6

Note: libstdc++fs.la is alone here.

It was forgotten in non-lib output:

$ find $(nix-build --no-link -A stdenv.cc.cc) | fgrep libstdc | unnix
/<<NIX>>/gcc-11.3.0/lib/libstdc++.a
/<<NIX>>/gcc-11.3.0/lib/libstdc++fs.a

I'll think of a local clangStdenv workaround (or a fix). Only 20 packages seem to use it, might not be too large to rebuild.

trofi added a commit to trofi/nixpkgs that referenced this pull request Jan 27, 2023
After NixOS#210004 `usbmuxd2` started
failing to build as:

    usbmuxd2-unstable> .../ld: cannot find -lstdc++fs: No such file or directory
    usbmuxd2-unstable> clang-11: error: linker command failed with exit code 1 (use -v to see invocation)

This started happening because NixOS#210004 exposed a long-standing bug in
`gcc` derivation: `cc.lib` is missing `libstdc++fs` library:

    $ find $(nix-build --no-link -A stdenv.cc.cc.lib) | fgrep libstdc | unnix

    /<<NIX>>/gcc-11.3.0-lib/lib/libstdc++fs.la

    /<<NIX>>/gcc-11.3.0-lib/lib/libstdc++.la
    /<<NIX>>/gcc-11.3.0-lib/lib/libstdc++.so.6.0.29
    /<<NIX>>/gcc-11.3.0-lib/lib/libstdc++.so
    /<<NIX>>/gcc-11.3.0-lib/lib/libstdc++.so.6

It was not moved from `cc.out` output:

    $ find $(nix-build --no-link -A stdenv.cc.cc) | fgrep libstdc | unnix
    /<<NIX>>/gcc-11.3.0/lib/libstdc++.a
    /<<NIX>>/gcc-11.3.0/lib/libstdc++fs.a

This change adds `cc` library lookup path back to `staging-next` until
`gcc` is fixed.`
@trofi
Copy link
Contributor Author

trofi commented Jan 27, 2023

Proposed usbmuxd2 fix as:

Looking at godot.

@trofi
Copy link
Contributor Author

trofi commented Jan 27, 2023

Proposed godot change as:

Looking at dmd.

trofi added a commit to trofi/nixpkgs that referenced this pull request Jan 28, 2023
…or clang

After NixOS#210004 `usbmuxd2` started
failing to build as:

    usbmuxd2-unstable> .../ld: cannot find -lstdc++fs: No such file or directory
    usbmuxd2-unstable> clang-11: error: linker command failed with exit code 1 (use -v to see invocation)

This started happening because NixOS#210004 exposed a long-standing bug in
`gcc` derivation: `cc.lib` is missing `libstdc++fs` library:

    $ find $(nix-build --no-link -A stdenv.cc.cc.lib) | fgrep libstdc | unnix

    /<<NIX>>/gcc-11.3.0-lib/lib/libstdc++fs.la

    /<<NIX>>/gcc-11.3.0-lib/lib/libstdc++.la
    /<<NIX>>/gcc-11.3.0-lib/lib/libstdc++.so.6.0.29
    /<<NIX>>/gcc-11.3.0-lib/lib/libstdc++.so
    /<<NIX>>/gcc-11.3.0-lib/lib/libstdc++.so.6

It was not moved from `cc.out` output:

    $ find $(nix-build --no-link -A stdenv.cc.cc) | fgrep libstdc | unnix
    /<<NIX>>/gcc-11.3.0/lib/libstdc++.a
    /<<NIX>>/gcc-11.3.0/lib/libstdc++fs.a

This change adds `cc` library lookup path back to `staging-next` until
`gcc` is fixed.`
trofi added a commit to trofi/nixpkgs that referenced this pull request Jan 28, 2023
`ipxe` build broke after NixOS#210004
where we started dropping default libc include path and switched to
`-idirafter` way of specifying libc headers.

Unfortunately the way it's implemented it injects `-idirafter` after
user's flags, not before. That allows users to inject their paths before
libc include paths, not after (as it would notmally happen).

The change works it around for `ipxe` by pulling `-idirafter` libc flags
before user's flags.
trofi added a commit to trofi/nixpkgs that referenced this pull request Jan 28, 2023
`wimboot` build broke after NixOS#210004
where we started dropping default libc include path and switched to
`-idirafter` way of specifying libc headers.

Unfortunately the way it's implemented it injects `-idirafter` after
user's flags, not before. That allows users to inject their paths before
libc include paths, not after (as it would notmally happen).

The change works it around for `wimboot` by pulling `-idirafter` libc
flags before user's flags.
trofi added a commit to trofi/nixpkgs that referenced this pull request Jan 28, 2023
After NixOS#210004 `dmd` started failing
build as:

    ld: warning: libm.so.6, needed by ./generated/linux/release/64/lib.so, not found (try using -rpath or -rpath-link)
    ld: /build/druntime/generated/linux/release/64/libdruntime.so: undefined reference to `log10@GLIBC_2.2.5'

This happens because --sysroot=/nix/store/does/not/exist removes not
just include headers by prefixing wrong path, but also removes RUNPATH
dependencies of linked libraries. It's an unintended effect.

Restore the build by reversing the effect with --sysroot=/.
trofi added a commit to trofi/nixpkgs that referenced this pull request Jan 28, 2023
…untangle from libc"

This reverts commit 8c80bd0
("build-support/cc-wrapper: pass in non-existent --sysroot= to untangle
from libc").

This change was good in spirit: we caught a few genuine problems with
`scons` based packages (`godot`, `fluxus`) and unexpected `-idirafter`
includes in various boot loadres (`ipxe`, wimboot`):

    NixOS#210004 (comment)

Unfortunately `--sysroot=` also has a negative impact on libary search
order for DT_NEEDED libraries and RUNPATHs of linked libraries. This
unexpectedly broke `dmd`, `d-seams`, `llvmPackages_rocm.compiler-rt`).

An interesting case of unexpected breakage is `usbmuxd2` where the bug
exposed incomplete library move on `libstdc++fs` in `gcc`.

The library breakage is very non-intuitive (on top of already unusual
layout of `cc-wrapper` driver). Let's revert this change for now.

Once it lands we can undo `--sysroot=/` workarounds merged for
`staging-next`.
@trofi
Copy link
Contributor Author

trofi commented Jan 28, 2023

On top of the above fixes for staging-next proposed full revert against staging: #213185 .

There are too many subtle changes that need to be handled one by one: library search paths, include search path order, bootstrap overrides and sequencing. --sysroot= is too blunt for it.

trofi added a commit to trofi/nixpkgs that referenced this pull request Jan 29, 2023
…untangle from libc"

This reverts commit 8c80bd0
("build-support/cc-wrapper: pass in non-existent --sysroot= to untangle
from libc").

This change was good in spirit: we caught a few genuine problems with
`scons` based packages (`godot`, `fluxus`) and unexpected `-idirafter`
includes in various boot loadres (`ipxe`, wimboot`):

    NixOS#210004 (comment)

Unfortunately `--sysroot=` also has a negative impact on libary search
order for DT_NEEDED libraries and RUNPATHs of linked libraries. This
unexpectedly broke `dmd`, `d-seams`, `llvmPackages_rocm.compiler-rt`).

An interesting case of unexpected breakage is `usbmuxd2` where the bug
exposed incomplete library move on `libstdc++fs` in `gcc`.

The library breakage is very non-intuitive (on top of already unusual
layout of `cc-wrapper` driver). Let's revert this change for now.

Once it lands we can undo `--sysroot=/` workarounds merged for
`staging-next`.
trofi added a commit to trofi/nixpkgs that referenced this pull request Jan 29, 2023
…untangle from libc"

This reverts commit 8c80bd0
("build-support/cc-wrapper: pass in non-existent --sysroot= to untangle
from libc").

This change was good in spirit: we caught a few genuine problems with
`scons` based packages (`godot`, `fluxus`) and unexpected `-idirafter`
includes in various boot loadres (`ipxe`, wimboot`):

    NixOS#210004 (comment)

Unfortunately `--sysroot=` also has a negative impact on libary search
order for DT_NEEDED libraries and RUNPATHs of linked libraries. This
unexpectedly broke `dmd`, `d-seams`, `llvmPackages_rocm.compiler-rt`).

An interesting case of unexpected breakage is `usbmuxd2` where the bug
exposed incomplete library move on `libstdc++fs` in `gcc`.

The library breakage is very non-intuitive (on top of already unusual
layout of `cc-wrapper` driver). Let's revert this change for now.

Once it lands we can undo `--sysroot=/` workarounds merged for
`staging-next`.
@ghost
Copy link

ghost commented Jan 31, 2023

wegank pushed a commit to wegank/nixpkgs that referenced this pull request Feb 1, 2023
#### Immediate Benefits

- Allow `gcc11` on `aarch64`
- No more copying `libgcc_s` out of the bootstrap-files or other
  derivations
- No more [static `lib{mpfr,mpc,gmp,isl}.a`
  hack](https://github.com/NixOS/nixpkgs/blob/2f1948af9c984ebb82dfd618e67dc949755823e2/pkgs/stdenv/linux/default.nix#L380)
- *Zero* additional `gcc` builds (stage1+stage2+stageCompare)
  - The `gcc` derivation builds `gcc` once instead of three times.
  - The libraries that are linked into the final `pkgs.gcc` (`mpfr`,
    `mpc`, `gmp`, `isl`, `glibc`) are built by
    `stdenv.__bootPkgs.gcc` rather than by the `bootstrapFiles`.  No
    more Frankenstein compiler!
  - stageCompare runs **concurrently** with (not in series with)
    with `stdenv`'s dependees.
- Many other `stdenv` hacks eliminated.
  - `gcc` and `clang` share the same codepath for more of
    `cc-wrapper`.
  - Makes the cross and native codepaths much more similar --
    another step towards "cross by default".

Note that *all* the changes in this PR are controlled by flags; no
old codepaths need to be removed until/if we're completely certain
that this is the right way to go.

#### Future Benefits

- This should allow using a [foreign] `bootstrap-files` so long as
  `hostPlatform.canExecute bootstrapFiles`.
- There will be an "avalanche of simplification" when we set
  `enableGccExternalBootstrap=true` and run dead code elimination.
  It's really quite a huge amount of code that goes away.
  Native-gcc has its own special codepath in so many places, while
  cross-gcc and clang work the same way (and are much simpler).
- This should allow each of the libraries that ship with `gcc`
  (`lib{backtrace,atomic,cc1,decnumber,ffi,gomp,iberty,offloadatomic,quadmath,sanitizer,ssp,stdc++-v3,vtv}`)
  to be built in separate (one-liner) derivations which `inherit
  src;` from `gcc`.
  - Building `libstdc++-v3` in a separate derivation will eliminate
    a lot of accidental-reference-to-the-`bootstrapFiles` landmines.

#### Incorporates

- NixOS#209054
- NixOS#210004
- NixOS#36948 (unreverted)
- NixOS#210325
- NixOS#210118
- NixOS#210132
- NixOS#210109

#### Closes

- Closes NixOS#208412
- Closes NixOS#108111
- Closes NixOS#108305
- Closes NixOS#201254

#### Build history

- First successful builds (stage1/stage2):
  - powerpc64le-linux at 9c7e9ef
  - x86_64-linux at 9c7e9ef
  - aarch64-linux at 4d5bc7d

- First successful comparisons (stageCompare):
  - at 81949cf
  - [aarch64-linux][aarch64-compare-ofborg]
  - [x86\_64-linux][amd64-compare-ofborg]

#### Credits

This project was made possible by three important insights, none of
which were mine:

1. @Ericson2314 was the first to advocate for this change, and
   probably the first to appreciate its advantages.  External
   bootstrap is "cross by default".

2. @trofi has figured out a lot about how to get gcc to not mix up
   the copy of `libstdc++` that it depends on with the copy that it
   builds.  Now that gcc is written in C++, it depends on
   `libstdc++`, builds a copy of `libstdc++`, and builds auxiliary
   products (like `libplugin`) which depend on `libstdc++`.  @trofi
   developed two important techniques for keeping this straight: the
   use of a [nonexistent sysroot] and moving the `bootstrapFiles`'
   `libstdc++` into a [versioned directory].  Without these two
   discoveries, external bootstrap would be impossible, because the
   final gcc would still have references to the `bootstrapFiles`.

3. Using the undocumented variable [`user-defined-trusted-dirs`]
   when building glibc.  When glibc `dlopen()`s `libgcc_s.so`, it
   uses a completely different and totally special set of rules for
   finding `libgcc_s.so`.  This trick is the only way we can put
   `libgcc_s.so` in its own separate outpath without creating
   circular dependencies or dependencies on the bootstrapFiles.  I
   would never have guessed to use this (or that it existed!) if it
   were not for a [comment in guix] which @Mic92 [mentioned].

My own role in this PR was basically: being available to go on a
coding binge at an opportune moment, so we wouldn't waste a
[crisis].

[aarch64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662822938
[amd64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662825857
[nonexistent sysroot]: NixOS#210004
[versioned directory]: NixOS#209054
[`user-defined-trusted-dirs`]: https://sourceware.org/legacy-ml/libc-help/2013-11/msg00026.html
[comment in guix]: https://github.com/guix-mirror/guix/blob/5e4ec8218142eee8e6e148e787381a5ef891c5b1/gnu/packages/gcc.scm#L253
[mentioned]: NixOS#210112 (comment)
[crisis]: NixOS#108305
[foreign]: NixOS#170857 (comment)
PetarKirov pushed a commit to PetarKirov/dlang.nix that referenced this pull request Apr 21, 2023
After NixOS/nixpkgs#210004 `dmd` started failing
build as:

    ld: warning: libm.so.6, needed by ./generated/linux/release/64/lib.so, not found (try using -rpath or -rpath-link)
    ld: /build/druntime/generated/linux/release/64/libdruntime.so: undefined reference to `log10@GLIBC_2.2.5'

This happens because --sysroot=/nix/store/does/not/exist removes not
just include headers by prefixing wrong path, but also removes RUNPATH
dependencies of linked libraries. It's an unintended effect.

Restore the build by reversing the effect with --sysroot=/.
PetarKirov pushed a commit to PetarKirov/dlang.nix that referenced this pull request Apr 21, 2023
After NixOS/nixpkgs#210004 `dmd` started failing
build as:

    ld: warning: libm.so.6, needed by ./generated/linux/release/64/lib.so, not found (try using -rpath or -rpath-link)
    ld: /build/druntime/generated/linux/release/64/libdruntime.so: undefined reference to `log10@GLIBC_2.2.5'

This happens because --sysroot=/nix/store/does/not/exist removes not
just include headers by prefixing wrong path, but also removes RUNPATH
dependencies of linked libraries. It's an unintended effect.

Restore the build by reversing the effect with --sysroot=/.
tm-drtina pushed a commit to awakesecurity/nixpkgs that referenced this pull request Apr 27, 2024
 #### Summary

By default, when you type `make`, GCC will compile itself three
times.  This PR inhibits that behavior by configuring GCC with
`--disable-bootstrap`, and reimplements the triple-rebuild using
Nix rather than `make`/`sh`.

 #### Immediate Benefits

- Allow `gcc11` and `gcc12` on `aarch64` (without needing new
  `bootstrapFiles`)
- Faster stdenv rebuilds: the third compilation of gcc
  (i.e. stageCompare) is no longer a `drvInput` of the final stdenv.
  This allows Nix to build stageCompare in parallel with the rest of
  nixpkgs instead of in series.
- No more copying `libgcc_s` out of the bootstrap-files or other
  derivations
- No more Frankenstein compiler: the final gcc and the libraries it
  links against (mpfr, mpc, isl, glibc) are all built by the same
  compiler (xgcc) instead of a mixture of the bootstrapFiles'
  compiler and xgcc.
- No more [static lib{mpfr,mpc,gmp,isl}.a hack]
- Many other small `stdenv` hacks eliminated
- `gcc` and `clang` share the same codepath for more of `cc-wrapper`.

 #### Future Benefits

- This should allow using a [foreign] `bootstrap-files` so long as
  `hostPlatform.canExecute bootstrapFiles`.
- This should allow each of the libraries that ship with `gcc`
  (lib{backtrace, atomic, cc1, decnumber, ffi, gomp, iberty,
  offloadatomic, quadmath, sanitizer, ssp, stdc++-v3, vtv}) to be
  built in separate (one-liner) derivations which `inherit src;`
  from `gcc`, much like NixOS#132343

 #### Incorporates

- NixOS#210004
- NixOS#36948 (unreverted)
- NixOS#210325
- NixOS#210118
- NixOS#210132
- NixOS#210109
- NixOS#213909
- NixOS#216136
- NixOS#216237
- NixOS#210019
- NixOS#216232
- NixOS#216016
- NixOS#217977
- NixOS#217995

 #### Closes

- Closes NixOS#108305
- Closes NixOS#108111
- Closes NixOS#201254
- Closes NixOS#208412

 #### Credits

This project was made possible by three important insights, none of
which were mine:

1. @Ericson2314 was the first to advocate for this change, and
   probably the first to appreciate its advantages.  Nix-driven
   (external) bootstrap is "cross by default".

2. @trofi has figured out a lot about how to get gcc to not mix up
   the copy of `libstdc++` that it depends on with the copy that it
   builds, by moving the `bootstrapFiles`' `libstdc++` into a
   [versioned directory].  This allows a Nix-driven bootstrap of gcc
   without the final gcc would still having references to the
   `bootstrapFiles`.

3. Using the undocumented variable [`user-defined-trusted-dirs`]
   when building glibc.  When glibc `dlopen()`s `libgcc_s.so`, it
   uses a completely different and totally special set of rules for
   finding `libgcc_s.so`.  This trick is the only way we can put
   `libgcc_s.so` in its own separate outpath without creating
   circular dependencies or dependencies on the bootstrapFiles.  I
   would never have guessed to use this (or that it existed!) if it
   were not for a [comment in guix] which @Mic92 [mentioned].

My own role in this PR was basically: being available to go on a
coding binge at an opportune moment, so we wouldn't waste a
[crisis].

[aarch64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662822938
[amd64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662825857
[nonexistent sysroot]: NixOS#210004
[versioned directory]: NixOS#209054
[`user-defined-trusted-dirs`]: https://sourceware.org/legacy-ml/libc-help/2013-11/msg00026.html
[comment in guix]: https://github.com/guix-mirror/guix/blob/5e4ec8218142eee8e6e148e787381a5ef891c5b1/gnu/packages/gcc.scm#L253
[mentioned]: NixOS#210112 (comment)
[crisis]: NixOS#108305
[foreign]: NixOS#170857 (comment)
[static lib{mpfr,mpc,gmp,isl}.a hack]: https://github.com/NixOS/nixpkgs/blob/2f1948af9c984ebb82dfd618e67dc949755823e2/pkgs/stdenv/linux/default.nix#L380
@rrbutani rrbutani added the 6.topic: stdenv Standard environment label May 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants