Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Native support for Apple Silicon #105026

Merged
merged 71 commits into from
May 16, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
0a5dcc9
gnutar: fix cross compilation on darwin
thefloweringash Nov 19, 2020
5293467
openssl: Apple Silicon support
thefloweringash Nov 19, 2020
4e09c26
openssh: disable kerberos on Apple Silicon
thefloweringash Nov 19, 2020
9ad03b1
flex: fix cross compilation
thefloweringash Nov 19, 2020
2931a2e
pbzx: fix cross compilation
thefloweringash Nov 19, 2020
1fc0499
xar: fix cross compilation
thefloweringash Nov 19, 2020
9088e44
glib: fix build
thefloweringash Nov 25, 2020
24a378c
zlib: fix cross compilation on darwin
thefloweringash Nov 19, 2020
aa33613
darwin.adv_cmds: fix cross compilation
thefloweringash Nov 19, 2020
cbe172f
darwin.ICU: cross compilation
thefloweringash Nov 19, 2020
8e33b34
pcre: disable jit on Apple Silicon
thefloweringash Nov 19, 2020
48a1e1c
pcre2: disable jit on Apple Silicon
thefloweringash Nov 28, 2020
c9af80e
fontconfig: backport atomics fixes for aarch64-darwin
thefloweringash Nov 27, 2020
5ed7e55
groff: fix build with clang >= 9
thefloweringash Nov 19, 2020
e44dd84
libtool2: macOS 11 support
thefloweringash Feb 18, 2021
bd4a45f
cctools: drop suspicious dependency
thefloweringash Nov 19, 2020
314c92c
curl: disable gssSupport for Apple Silicon
thefloweringash Nov 19, 2020
1182552
gmp: disable assembly on Apple Silicon
thefloweringash Nov 19, 2020
257cb1c
print-reexports: rehome, tbd4 support, cross compilation
thefloweringash Nov 19, 2020
b7abec3
rewrite-tbd: init
thefloweringash Nov 19, 2020
eb8f8af
libtapi: cross compilation
thefloweringash Nov 19, 2020
15637fe
bintools-wrapper: support post linker hooks
thefloweringash Nov 19, 2020
0ba7a04
darwin/binutils: include codesign_allocate
thefloweringash Nov 19, 2020
fe0d5a5
darwin/apple-sdk-11.0: inject an sdk for macOS 11
thefloweringash Nov 19, 2020
9fb8924
darwin: integrate macOS 11 sdk
thefloweringash Nov 19, 2020
c696fcd
fixDarwinDylibNames: set name of install_name_tool
thefloweringash Nov 19, 2020
9e05276
bash: provide a working binutils
thefloweringash Nov 19, 2020
a7bcb6b
darwin cross: include CoreFoundation in stdenv
thefloweringash Nov 19, 2020
4c890ce
darwin.sigtool: init
thefloweringash Nov 19, 2020
bfd9a77
darwin.postLinkSignHook: init
thefloweringash Nov 19, 2020
7eb1e36
darwin.signingUtils: init
thefloweringash Feb 15, 2021
772b665
darwin: wrap strip and install_name_tool to codesign modified files
thefloweringash Jan 21, 2021
3eacdfe
stdenv/darwin: bootstrap tools for Apple Silicon
thefloweringash Nov 19, 2020
768aae6
stdenv/darwin: Apple Silicon support
thefloweringash Nov 19, 2020
33b59b8
flake.nix: add aarch64-darwin
matthewbauer Dec 8, 2020
a6753d0
lib/systems: add aarch64-darwin to mesaPlatforms
matthewbauer Dec 8, 2020
3e7034a
darwin/xcode: provide libc++ from XCode toolchain
matthewbauer Dec 9, 2020
1a2cb94
hydra jobset support for aarch64-darwin
vcunat Jan 8, 2021
3a3df31
Add crossPkgs.aarch64-darwin
angerman Jan 9, 2021
d2c9670
darwin: add CoreFoundation to the build when explicitly required
thefloweringash Jan 29, 2021
1a45df1
ctags: fix build with newer darwin libraries
thefloweringash Dec 5, 2020
c99904e
clang_11: revert -fno-common default
thefloweringash Feb 4, 2021
23cae56
lib/systems/platforms: add Apple M1
thefloweringash Feb 9, 2021
bac1418
xcbuild: allow building arm64 binaries
bobrik Feb 5, 2021
2eb97f2
darwin.developer_cmds: wrap clang -E from clang-wrapped as clang-cpp
bobrik Feb 6, 2021
da3ae02
apple-source-releases: make it compile on aarch64
bobrik Feb 6, 2021
6815776
python3Packages.cryptography: ignore test_openssl_memleak.py on aarch…
bobrik Feb 7, 2021
f067102
removeReferencesTo: fix code signatures on aarch64-darwin
thefloweringash Feb 15, 2021
79e3429
nukeReferences: fix code signatures on aarch64-darwin
thefloweringash Feb 15, 2021
755d980
darwin: use "11.0" as sdk and minimum version on aarch64-darwin
thefloweringash Feb 15, 2021
aadc07a
mc: fix code signature on aarch64-darwin
thefloweringash Feb 12, 2021
3e47747
libxml2: cap MACOSX_DEPLOYMENT_TARGET at 10.16
thefloweringash Feb 18, 2021
577bcf3
darwin.shell_cmds: add missing dependency on launchd
bobrik Feb 6, 2021
d0eff87
darwin.file_cmds: add missing Libinfo dependency
bobrik Feb 11, 2021
68733c1
darwin.apple_sdk.frameworks: add missing dependencies
bobrik Feb 9, 2021
f9e6eb0
llvmPackages: default to llvmPackages_11 on aarch64-darwin
bobrik Feb 10, 2021
ea2e64c
libevent: cap MACOSX_DEPLOYMENT_TARGET at 10.16
thefloweringash Feb 25, 2021
c96c858
openbsm: cap MACOSX_DEPLOYMENT_TARGET at 10.16
thefloweringash Feb 25, 2021
e00c4cd
openldap: cap MACOSX_DEPLOYMENT_TARGET at 10.16
thefloweringash Feb 25, 2021
0ec1e50
vde2: cap MACOSX_DEPLOYMENT_TARGET at 10.16
thefloweringash Feb 25, 2021
1f0ae93
libtool2: add bootstrapping warning
thefloweringash Feb 26, 2021
417a96c
sbcl: update to 2.1.2 and add arm64-darwin
kloenk Mar 14, 2021
c022642
imagemagick: support aarch64-darwin
jdelStrother Mar 19, 2021
3706fe5
darwin.DarwinTools: remove reference to CFPriv.h
thefloweringash Apr 19, 2021
92d5324
apr: update libtool, fix downstream aprutil on aarch64-darwin
thefloweringash Apr 19, 2021
7f216a4
catch: backport support for Apple Silicon
thefloweringash Apr 19, 2021
e592d34
darwin.ICU: avoid platform.darwinMinVersion references on non-darwin
thefloweringash Apr 27, 2021
4851956
darwin-packages: avoid exporting attributes that don't exist
thefloweringash Apr 27, 2021
4b569c1
release.nix: add cross-compiled aarch64-darwin bootstrap tools
thefloweringash Apr 28, 2021
c31591c
all-packages.nix: use darwin.LibsystemCross if defined
thefloweringash May 16, 2021
56fcbcd
llvmPackages_11: darwin cross compilation and bootstrapping
thefloweringash May 16, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions flake.nix
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
"aarch64-linux"
"armv6l-linux"
"armv7l-linux"
"aarch64-darwin"
];

forAllSystems = f: lib.genAttrs systems (system: f system);
Expand Down
2 changes: 1 addition & 1 deletion lib/systems/default.nix
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ rec {
else null;
# The canonical name for this attribute is darwinSdkVersion, but some
# platforms define the old name "sdkVer".
darwinSdkVersion = final.sdkVer or "10.12";
darwinSdkVersion = final.sdkVer or (if final.isAarch64 then "11.0" else "10.12");
darwinMinVersion = final.darwinSdkVersion;
darwinMinVersionVariable =
if final.isMacOS then "MACOSX_DEPLOYMENT_TARGET"
Expand Down
2 changes: 1 addition & 1 deletion lib/systems/doubles.nix
Original file line number Diff line number Diff line change
Expand Up @@ -96,5 +96,5 @@ in {

embedded = filterDoubles predicates.isNone;

mesaPlatforms = ["i686-linux" "x86_64-linux" "x86_64-darwin" "armv5tel-linux" "armv6l-linux" "armv7l-linux" "armv7a-linux" "aarch64-linux" "powerpc64-linux" "powerpc64le-linux"];
mesaPlatforms = ["i686-linux" "x86_64-linux" "x86_64-darwin" "armv5tel-linux" "armv6l-linux" "armv7l-linux" "armv7a-linux" "aarch64-linux" "powerpc64-linux" "powerpc64le-linux" "aarch64-darwin"];
}
6 changes: 6 additions & 0 deletions lib/systems/examples.nix
Original file line number Diff line number Diff line change
Expand Up @@ -240,6 +240,12 @@ rec {
useiOSPrebuilt = true;
};

aarch64-darwin = {
config = "aarch64-apple-darwin";
xcodePlatform = "MacOSX";
platform = {};
};

#
# Windows
#
Expand Down
12 changes: 11 additions & 1 deletion lib/systems/platforms.nix
Original file line number Diff line number Diff line change
Expand Up @@ -375,6 +375,13 @@ rec {
};
};

apple-m1 = {
gcc = {
arch = "armv8.3-a+crypto+sha2+aes+crc+fp16+lse+simd+ras+rdm+rcpc";
cpu = "apple-a13";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GCC doesn't like either of these things:

But we force them in wrappers. Is there a better way?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh, that is weird. @thefloweringash does this help somewhere else?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

};
};

##
## MIPS
##
Expand Down Expand Up @@ -495,7 +502,10 @@ rec {
else if lib.versionOlder version "6" then sheevaplug
else if lib.versionOlder version "7" then raspberrypi
else armv7l-hf-multiplatform
else if platform.isAarch64 then aarch64-multiplatform

else if platform.isAarch64 then
if platform.isDarwin then apple-m1
else aarch64-multiplatform

else if platform.isRiscV then riscv-multiplatform

Expand Down
2 changes: 1 addition & 1 deletion pkgs/applications/graphics/ImageMagick/6.x.nix
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ let
if stdenv.hostPlatform.system == "i686-linux" then "i686"
else if stdenv.hostPlatform.system == "x86_64-linux" || stdenv.hostPlatform.system == "x86_64-darwin" then "x86-64"
else if stdenv.hostPlatform.system == "armv7l-linux" then "armv7l"
else if stdenv.hostPlatform.system == "aarch64-linux" then "aarch64"
else if stdenv.hostPlatform.system == "aarch64-linux" || stdenv.hostPlatform.system == "aarch64-darwin" then "aarch64"
else if stdenv.hostPlatform.system == "powerpc64le-linux" then "ppc64le"
else throw "ImageMagick is not supported on this platform.";
in
Expand Down
2 changes: 1 addition & 1 deletion pkgs/applications/graphics/ImageMagick/7.0.nix
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ let
if stdenv.hostPlatform.system == "i686-linux" then "i686"
else if stdenv.hostPlatform.system == "x86_64-linux" || stdenv.hostPlatform.system == "x86_64-darwin" then "x86-64"
else if stdenv.hostPlatform.system == "armv7l-linux" then "armv7l"
else if stdenv.hostPlatform.system == "aarch64-linux" then "aarch64"
else if stdenv.hostPlatform.system == "aarch64-linux" || stdenv.hostPlatform.system == "aarch64-darwin" then "aarch64"
else if stdenv.hostPlatform.system == "powerpc64le-linux" then "ppc64le"
else throw "ImageMagick is not supported on this platform.";
in
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
#! @shell@
# shellcheck shell=bash

set -eu -o pipefail +o posix
shopt -s nullglob

if (( "${NIX_DEBUG:-0}" >= 7 )); then
set -x
fi

source @signingUtils@

extraAfter=()
extraBefore=()
params=("$@")

input=

pprev=
prev=
for p in \
${extraBefore+"${extraBefore[@]}"} \
${params+"${params[@]}"} \
${extraAfter+"${extraAfter[@]}"}
do
if [ "$pprev" != "-change" ] && [[ "$prev" != -* ]] && [[ "$p" != -* ]]; then
input="$p"
fi
pprev="$prev"
prev="$p"
done

# Optionally print debug info.
if (( "${NIX_DEBUG:-0}" >= 1 )); then
# Old bash workaround, see above.
echo "extra flags before to @prog@:" >&2
printf " %q\n" ${extraBefore+"${extraBefore[@]}"} >&2
echo "original flags to @prog@:" >&2
printf " %q\n" ${params+"${params[@]}"} >&2
echo "extra flags after to @prog@:" >&2
printf " %q\n" ${extraAfter+"${extraAfter[@]}"} >&2
fi

@prog@ \
${extraBefore+"${extraBefore[@]}"} \
${params+"${params[@]}"} \
${extraAfter+"${extraAfter[@]}"}

sign "$input"
78 changes: 78 additions & 0 deletions pkgs/build-support/bintools-wrapper/darwin-strip-wrapper.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
#! @shell@
# shellcheck shell=bash

set -eu -o pipefail +o posix
shopt -s nullglob

if (( "${NIX_DEBUG:-0}" >= 7 )); then
set -x
fi

source @signingUtils@

extraAfter=()
extraBefore=()
params=("$@")

output=
inputs=()

restAreFiles=
prev=
for p in \
${extraBefore+"${extraBefore[@]}"} \
${params+"${params[@]}"} \
${extraAfter+"${extraAfter[@]}"}
do
if [ "$restAreFiles" ]; then
inputs+=("$p")
else
case "$prev" in
-s|-R|-d|-arch)
# Unrelated arguments with values
;;
-o)
# Explicit output
output="$p"
;;
*)
# Any other orgument either takes no value, or is a file.
if [[ "$p" != -* ]]; then
inputs+=("$p")
fi
;;
esac

if [ "$p" == - ]; then
restAreFiles=1
fi
fi

prev="$p"
done

# Optionally print debug info.
if (( "${NIX_DEBUG:-0}" >= 1 )); then
# Old bash workaround, see above.
echo "extra flags before to @prog@:" >&2
printf " %q\n" ${extraBefore+"${extraBefore[@]}"} >&2
echo "original flags to @prog@:" >&2
printf " %q\n" ${params+"${params[@]}"} >&2
echo "extra flags after to @prog@:" >&2
printf " %q\n" ${extraAfter+"${extraAfter[@]}"} >&2
fi

@prog@ \
${extraBefore+"${extraBefore[@]}"} \
${params+"${params[@]}"} \
${extraAfter+"${extraAfter[@]}"}

if [ "$output" ]; then
# Single explicit output
signIfRequired "$output"
else
# Multiple inputs, rewritten in place
for input in "${inputs[@]}"; do
signIfRequired "$input"
done
fi
21 changes: 21 additions & 0 deletions pkgs/build-support/bintools-wrapper/default.nix
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@
, extraPackages ? [], extraBuildCommands ? ""
, buildPackages ? {}
, useMacosReexportHack ? false

# Darwin code signing support utilities
, postLinkSignHook ? null, signingUtils ? null
}:

with lib;
Expand Down Expand Up @@ -339,6 +342,24 @@ stdenv.mkDerivation {
''
)

##
## Code signing on Apple Silicon
##
+ optionalString (targetPlatform.isDarwin && targetPlatform.isAarch64) ''
echo 'source ${postLinkSignHook}' >> $out/nix-support/post-link-hook

export signingUtils=${signingUtils}

wrap \
${targetPrefix}install_name_tool \
${./darwin-install_name_tool-wrapper.sh} \
"${bintools_bin}/bin/${targetPrefix}install_name_tool"

wrap \
${targetPrefix}strip ${./darwin-strip-wrapper.sh} \
"${bintools_bin}/bin/${targetPrefix}strip"
''

##
## Extra custom steps
##
Expand Down
26 changes: 25 additions & 1 deletion pkgs/build-support/bintools-wrapper/ld-wrapper.sh
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,8 @@ declare -a libDirs
declare -A libs
declare -i relocatable=0 link32=0

linkerOutput="a.out"

if
[ "$NIX_DONT_SET_RPATH_@suffixSalt@" != 1 ] \
|| [ "$NIX_SET_BUILD_ID_@suffixSalt@" = 1 ] \
Expand Down Expand Up @@ -153,6 +155,24 @@ then
done
fi

# Determine linkerOutput
prev=
for p in \
${extraBefore+"${extraBefore[@]}"} \
${params+"${params[@]}"} \
${extraAfter+"${extraAfter[@]}"}
do
case "$prev" in
-o)
# Informational for post-link-hook
linkerOutput="$p"
;;
*)
;;
esac
prev="$p"
done

if [[ "$link32" = "1" && "$setDynamicLinker" = 1 && -e "@out@/nix-support/dynamic-linker-m32" ]]; then
# We have an alternate 32-bit linker and we're producing a 32-bit ELF, let's
# use it.
Expand Down Expand Up @@ -223,7 +243,11 @@ fi

PATH="$path_backup"
# Old bash workaround, see above.
exec @prog@ \
@prog@ \
${extraBefore+"${extraBefore[@]}"} \
${params+"${params[@]}"} \
${extraAfter+"${extraAfter[@]}"}

if [ -e "@out@/nix-support/post-link-hook" ]; then
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a hook to sign everything the linker produces, without putting explicit signing references in the general linker wrapper.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this interact with stripping binaries? Don't signatures cover the whole binaries? (I am just driving by and reading through the comments)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any modification of the binary, including strip and install_name_tool will cause the signature to be invalid. However, upstream's versions of these tools recreate a valid signature. To be like upstream, I've also wrapped strip and install_name_tool to recreate valid signatures. One of the todo list: add a similar wrapper for remove-references-to, which is valid to call on a binary. This is probably the correct solution to #105026 (comment). The current version will have broken this again (sorry @bobrik!).

Initially I used a fixup phase hook, which generally worked, but ran into a few cases that needed extra work. meson, for example, calls install_name_tool internally on binaries that are then executed to generate caches in some package. I initially patched meson to call codesign again, but I think wrapping is more correct. I also had to be very carefully to the signing hook run last in the fixup phase, which seems to run before postFixup.

source @out@/nix-support/post-link-hook
fi
29 changes: 0 additions & 29 deletions pkgs/build-support/nuke-references/builder.sh

This file was deleted.

5 changes: 5 additions & 0 deletions pkgs/build-support/nuke-references/darwin-sign-fixup.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Fixup hook for nukeReferences, not stdenv

source @signingUtils@

fixupHooks+=(signIfRequired)
28 changes: 26 additions & 2 deletions pkgs/build-support/nuke-references/default.nix
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,35 @@
# path (/nix/store/eeee...). This is useful for getting rid of
# dependencies that you know are not actually needed at runtime.

{ stdenvNoCC, perl }:
{ lib, stdenvNoCC, perl, signingUtils, shell ? stdenvNoCC.shell }:

let
stdenv = stdenvNoCC;

darwinCodeSign = stdenv.targetPlatform.isDarwin && stdenv.targetPlatform.isAarch64;
in

stdenvNoCC.mkDerivation {
name = "nuke-references";
builder = ./builder.sh;

dontUnpack = true;
dontConfigure = true;
dontBuild = true;

installPhase = ''
mkdir -p $out/bin
substituteAll ${./nuke-refs.sh} $out/bin/nuke-refs
chmod a+x $out/bin/nuke-refs
'';

postFixup = lib.optionalString darwinCodeSign ''
mkdir -p $out/nix-support
substituteAll ${./darwin-sign-fixup.sh} $out/nix-support/setup-hooks.sh
'';

# FIXME: get rid of perl dependency.
inherit perl;
inherit (builtins) storeDir;
shell = lib.getBin shell + (shell.shellPath or "");
signingUtils = if darwinCodeSign then signingUtils else null;
}
33 changes: 33 additions & 0 deletions pkgs/build-support/nuke-references/nuke-refs.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
#! @shell@

fixupHooks=()

if [ -e @out@/nix-support/setup-hooks.sh ]; then
source @out@/nix-support/setup-hooks.sh
fi

excludes=""
while getopts e: o; do
case "$o" in
e) storeId=$(echo "$OPTARG" | @perl@/bin/perl -ne "print \"\$1\" if m|^\Q@storeDir@\E/([a-z0-9]{32})-.*|")
if [ -z "$storeId" ]; then
echo "-e argument must be a Nix store path"
exit 1
fi
excludes="$excludes(?!$storeId)"
;;
esac
done
shift $(($OPTIND-1))

for i in "$@"; do
if test ! -L "$i" -a -f "$i"; then
cat "$i" | @perl@/bin/perl -pe "s|\Q@storeDir@\E/$excludes[a-z0-9]{32}-|@storeDir@/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-|g" > "$i.tmp"
if test -x "$i"; then chmod +x "$i.tmp"; fi
mv "$i.tmp" "$i"

for hook in "${fixupHooks[@]}"; do
eval "$hook" "$i"
done
fi
done
Loading