apple_sdk_11_0: make available for use on x86_64-darwin #176661

reckenrode · 2022-06-07T04:55:59Z

Description of changes

I discussed with @toonn on Matrix yesterday and this morning about the possibility of using the aarch64-darwin SDK on x86_64-darwin as an interim solution for software that needs a newer SDK. This was prompted by my wanting to build MoltenVK without needing Xcode.

I opened the PR against master because the changes required were quite small, and there are no rebuilds. The basic functionality tested was MoltenVK built with the SDK from nixpkgs (no Xcode).

See below for a screenshot of a functional demo app built almost entirely with nixpkgs (vkcube.app in vulkan-tools unfortunately requires ibtool from Xcode, but the actual code is compiled against the 10.12 SDK in nixpkg and linked against a MoltenVK built with the 11.0 SDK). A PR for MoltenVK will be forthcoming once this lands.

Sandbox: set to relaxed.

Things done

reckenrode · 2022-06-07T05:00:57Z

@thefloweringash Also pinging you since you did the initial work on the 11.0 SDK.

nixos-discourse · 2022-06-07T05:05:30Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/902

toonn

LGTM

Only part I wonder about is why libobjc and Libsystem were gotten from the source releases before, I think that's the biggest change?

doc/stdenv/platform-notes.chapter.md

reckenrode · 2022-06-07T11:37:50Z

LGTM

Only part I wonder about is why libobjc and Libsystem were gotten from the source releases before, I think that's the biggest change?

It’s conditional based on the platform. On aarch64-darwin, they come from the SDK. What this change does is always get them from the 11.0 SDK regardless (while keeping the defaults for each platform as-is).

# Pick the source of libraries: either Apple's open source releases, or the
# SDK.
useAppleSDKLibs = stdenv.hostPlatform.isAarch64;

selectAttrs = attrs: names:
  lib.listToAttrs (lib.concatMap (n: if attrs ? "${n}" then [(lib.nameValuePair n attrs."${n}")] else []) names);

chooseLibs = (
  # There are differences in which libraries are exported. Avoid evaluation
  # errors when a package is not provided.
  selectAttrs (
    if useAppleSDKLibs
      then apple_sdk
      else appleSourcePackages
  ) ["Libsystem" "LibsystemCross" "libcharset" "libunwind" "objc4" "configd" "IOKit"]
) // {
  inherit (
    if useAppleSDKLibs
      then apple_sdk.frameworks
      else appleSourcePackages
  ) Security;
};

SuperSandro2000 · 2022-06-07T13:13:06Z

doc/stdenv/platform-notes.chapter.md

+  { stdenv, overrideCC, xcbuild, AppKit, Foundation, Libsystem }:
+  let
+    clang_11_0 = stdenv.cc.override {
+      bintools = stdenv.cc.bintools.override { libc = Libsystem; };
+      libc = Libsystem;
+    };
+    stdenv_11_0 =
+      if stdenv.isx86_64 then
+        (overrideCC stdenv clang_11_0).override
+          {
+            targetPlatform = stdenv.targetPlatform // {
+              darwinMinVersion = "10.12";
+              darwinSdkVersion = "11.0";
+            };
+          }
+      else
+        stdenv;
+      xcbuild_11_0 = xcbuild.override { stdenv = stdenv_11_0; };
+    in
+    stdenv_11_0.mkDerivation {
+      buildInputs = [ AppKit Foundation ];
+      nativeBuildInputs = [ xcbuild_11_0 ];
+      # ...
+    }


If we want to use this in nixpkgs I would suggest the following points.

be very careful. I think mixing SDKs could easily backfire and we need to carefully test this to make sure we don't break everything.

Can we add this to top level to make sure all packages use the same overrides? It would also make updating easier in case we discover bugs and need to change things later. This would mean we move this to top-level or darwin packages and remove the nix code here.

After discussing it with @toonn on Matrix, I pushed an update that adds a stdenv attribute to the 11.0 SDK attribute.

willcohen · 2022-06-09T14:55:42Z

Just thinking about EOL dates, planning for this seems like a very reasonable stopgap between now and the next stable release. Since 10.15 is going to be EOL this fall, macOS 11 will be the oldest supported release at that time. I know that the tentative plan for #101229 is to try to get SDKs bumped by the fall nixos release, and if it's feasibly to get that up to macOS 11 by then, this workaround may have a max shelf life of something like 6 months and will be a chance to get all the stuff that's been lagging on darwin up to speed in advance of that bump (getting Qt 5.15 and 6 stuff settled, anything using go, etc)

toonn · 2022-06-09T18:28:56Z

We're not bumping that far. And I'm not entirely sure we even should keep up with Apple's support schedule. That leaves a lot of hardware out there without access to Nixpkgs and no tangible advantage for users on more recent systems.

The current idea is to keep the default SDK recent-ish (emphasis on the ish) and provide ways to build with newer SDKs only for packages that really need it.

toonn

I think this is the most elegant way to expose this so it looks good to me.

willcohen · 2022-06-09T19:12:44Z

Makes sense — sorry for the confusion!

SuperSandro2000

LGTM

toonn · 2022-06-10T10:39:25Z

One sensible test that someone on aarch64-darwin should run is whether this has any effect on bootstrapping stdenv.

reckenrode · 2022-06-10T12:22:47Z

One sensible test that someone on aarch64-darwin should run is whether this has any effect on bootstrapping stdenv.

What would be the best way to do that? When I try to build stdenv on aarch64-darwin after applying the change, nix still gets it from the cache.

I’m running nix store gc right now to clear things out, so I can disable substituters (--option substituters '' and have it rebuild as much as possible, but that’s going to take some time.

reckenrode · 2022-06-11T00:53:47Z

After running nix store gc, and trying to rebuild stdenv, I get an error.

$ nix build --rebuild .#stdenv --option substituters ''
error: derivation '/nix/store/f6d7lwrknzcwbsjds36sgfl6cqpp82qy-bootstrap-stage0-clang-wrapper-11.1.0.drv' requires non-existent output 'out' from input derivation '/nix/store/ns4bvhwbwjjwizgvm4h0yf5gz2p7x7mf-bash-5.1-p16.drv'

Running this on master, I also get a (different) error.

$ nix build --rebuild github:NixOS/nixpkgs#stdenv --option substituters ''
error: derivation '/nix/store/4xfgcw1ajb12hjgvqa2rxz2m88w6ab6x-clang-11-12-LLVMgold-path.patch.drv' requires non-existent output 'lib' from input derivation '/nix/store/rd66ahrsdnghs9yjibscm7rrxlw09afq-llvm-11.1.0.drv'

toonn · 2022-06-13T09:41:11Z

@reckenrode, does the new build command have a similar option to --check? That should normally rebuild rather than substitute even if it's already in the store.

Those errors don't look good though, it seems that stdenv doesn't even build on master?

reckenrode · 2022-06-13T12:43:48Z

@reckenrode, does the new build command have a similar option to --check? That should normally rebuild rather than substitute even if it's already in the store.

I think --rebuild does the same thing. It rebuilds the derivation then compares it to the original. That succeeds; but, as far as I know, it’s not doing a bootstrap of stdenv (because none of the inputs changed).

Those errors don't look good though, it seems that stdenv doesn't even build on master?

That’s the way I read it, assuming disabling substituters is a supported scenario. Is there a better way of forcing a bootstrap of stdenv? I tried changing the name of bootstrapTools in its derivation this morning, and stdenv is rebuilding now, but that seems kind of hacky.

reckenrode · 2022-06-13T15:08:28Z

Following up: the hacky approach worked (changed name = "bootstrap-tools"; to name = "bootstrap-tools-foo";. The aarch64-darwin stdenv rebuilt without issue.

nixos-discourse · 2022-07-27T12:24:06Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-macos-monthly/12330/31

SuperSandro2000 · 2022-08-03T13:53:19Z

Can you use sdk 11 in a nix-shell that only has stdenv.mkDerivation?

https://discourse.nixos.org/t/remove-entries-from-nix-cflags-compile/20728

PlumpMath · 2023-05-23T01:46:14Z

ipetkov/crane#325 (comment)

problem x86_64-darwin with rust

apple_sdk_11_0: make available for use on x86_64-darwin

ofborg bot added the 6.topic: darwin Running or building packages on Darwin label Jun 7, 2022

github-actions bot added the 8.has: documentation label Jun 7, 2022

ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux labels Jun 7, 2022

This comment was marked as duplicate.

Sign in to view

reckenrode force-pushed the apple-sdk-11-x86_64 branch from 201416d to f10fb80 Compare June 7, 2022 05:44

toonn reviewed Jun 7, 2022

View reviewed changes

doc/stdenv/platform-notes.chapter.md Outdated Show resolved Hide resolved

reckenrode force-pushed the apple-sdk-11-x86_64 branch from f10fb80 to 374c0ad Compare June 7, 2022 11:39

SuperSandro2000 reviewed Jun 7, 2022

View reviewed changes

reckenrode force-pushed the apple-sdk-11-x86_64 branch 3 times, most recently from 9d5db4a to 2d92e04 Compare June 7, 2022 20:13

reckenrode mentioned this pull request Jun 8, 2022

python311: 3.11.0a7 -> 3.11.0b3 #173998

Merged

13 tasks

toonn approved these changes Jun 9, 2022

View reviewed changes

SuperSandro2000 approved these changes Jun 9, 2022

View reviewed changes

bobby285271 added the 12.approvals: 2 This PR was reviewed and approved by two reputable people label Jun 10, 2022

toonn mentioned this pull request Jun 13, 2022

Update macOS SDK #101229

Closed

reckenrode mentioned this pull request Jul 10, 2022

Using MacOS SDK 11.0 on x86 #146385

Closed

06kellyjac mentioned this pull request Jul 13, 2022

go,goBuildModules: bump default go version to 1.18 #179622

Merged

13 tasks

stephank mentioned this pull request Jul 14, 2022

cmake: detect libc location at runtime #181431

Merged

13 tasks

aiotter mentioned this pull request Jul 14, 2022

zig: build on darwin #181432

Merged

13 tasks

arximboldi mentioned this pull request Jul 28, 2022

qt5.qtwebengine is marked broken on x86_64-darwin, however maybe it can be made work? #183383

Closed

This was referenced Aug 12, 2022

v8 fails to build on darwin #158076

Closed

randomx: 1.1.9 -> 1.1.10 #186623

Merged

yihuang mentioned this pull request Aug 25, 2022

Problem: archive node takes a long time to startup crypto-org-chain/cronos#667

Merged

13 tasks

reckenrode mentioned this pull request Oct 3, 2022

[backport release-22.05] apple_sdk_11_0: make available for use on x86_64-darwin #194145

Merged

13 tasks

viraptor mentioned this pull request Oct 31, 2022

nncp: 8.8.0 -> 8.8.1 #198701

Merged

13 tasks

veprbl mentioned this pull request Nov 5, 2022

apple-sdk-13 #199730

Closed

jonathanlking mentioned this pull request Jan 5, 2023

lima: 0.13.0 -> 0.14.2 #206285

Merged

13 tasks

dominikb1888 referenced this pull request Jan 12, 2023

kitty: use apple_sdk_11 and mark as broken on x86_64-darwin

3126ef6

rkaippully pushed a commit to awakesecurity/nixpkgs that referenced this pull request Jul 3, 2023

Merge pull request NixOS#176661 from reckenrode/apple-sdk-11-x86_64

ba4ab5d

apple_sdk_11_0: make available for use on x86_64-darwin

rkaippully mentioned this pull request Jul 3, 2023

Backport go-1.19.1 awakesecurity/nixpkgs#123

Merged

jsoo1 pushed a commit to awakesecurity/nixpkgs that referenced this pull request Jul 3, 2023

Merge pull request NixOS#176661 from reckenrode/apple-sdk-11-x86_64

47ba1f7

apple_sdk_11_0: make available for use on x86_64-darwin

jsoo1 pushed a commit to awakesecurity/nixpkgs that referenced this pull request Jul 14, 2023

Merge pull request NixOS#176661 from reckenrode/apple-sdk-11-x86_64

bb819f0

apple_sdk_11_0: make available for use on x86_64-darwin

jsoo1 pushed a commit to awakesecurity/nixpkgs that referenced this pull request Jul 15, 2023

Merge pull request NixOS#176661 from reckenrode/apple-sdk-11-x86_64

3086152

apple_sdk_11_0: make available for use on x86_64-darwin

jsoo1 pushed a commit to awakesecurity/nixpkgs that referenced this pull request Jul 21, 2023

Merge pull request NixOS#176661 from reckenrode/apple-sdk-11-x86_64

8ab1883

apple_sdk_11_0: make available for use on x86_64-darwin

jsoo1 pushed a commit to awakesecurity/nixpkgs that referenced this pull request Aug 7, 2023

Merge pull request NixOS#176661 from reckenrode/apple-sdk-11-x86_64

e968f47

apple_sdk_11_0: make available for use on x86_64-darwin

jsoo1 pushed a commit to awakesecurity/nixpkgs that referenced this pull request Sep 7, 2023

Merge pull request NixOS#176661 from reckenrode/apple-sdk-11-x86_64

ae0016d

apple_sdk_11_0: make available for use on x86_64-darwin

awake-bot pushed a commit to awakesecurity/nixpkgs that referenced this pull request Oct 20, 2023

Merge pull request NixOS#176661 from reckenrode/apple-sdk-11-x86_64

de5a5e5

apple_sdk_11_0: make available for use on x86_64-darwin

cpick mentioned this pull request Jan 24, 2024

Build failure: Wezterm (x86-64 darwin) #239384

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

apple_sdk_11_0: make available for use on x86_64-darwin #176661

apple_sdk_11_0: make available for use on x86_64-darwin #176661

reckenrode commented Jun 7, 2022

reckenrode commented Jun 7, 2022

nixos-discourse commented Jun 7, 2022

This comment was marked as duplicate.

This comment was marked as duplicate.

toonn left a comment

reckenrode commented Jun 7, 2022 •

edited by SuperSandro2000

Loading

SuperSandro2000 Jun 7, 2022 •

edited

Loading

reckenrode Jun 7, 2022

willcohen commented Jun 9, 2022 •

edited

Loading

toonn commented Jun 9, 2022

toonn left a comment

willcohen commented Jun 9, 2022

SuperSandro2000 left a comment

toonn commented Jun 10, 2022

reckenrode commented Jun 10, 2022

reckenrode commented Jun 11, 2022

toonn commented Jun 13, 2022

reckenrode commented Jun 13, 2022

reckenrode commented Jun 13, 2022

nixos-discourse commented Jul 27, 2022

SuperSandro2000 commented Aug 3, 2022

PlumpMath commented May 23, 2023

apple_sdk_11_0: make available for use on x86_64-darwin #176661

apple_sdk_11_0: make available for use on x86_64-darwin #176661

Conversation

reckenrode commented Jun 7, 2022

Description of changes

Things done

reckenrode commented Jun 7, 2022

nixos-discourse commented Jun 7, 2022

This comment was marked as duplicate.

This comment was marked as duplicate.

toonn left a comment

Choose a reason for hiding this comment

reckenrode commented Jun 7, 2022 • edited by SuperSandro2000 Loading

SuperSandro2000 Jun 7, 2022 • edited Loading

Choose a reason for hiding this comment

reckenrode Jun 7, 2022

Choose a reason for hiding this comment

willcohen commented Jun 9, 2022 • edited Loading

toonn commented Jun 9, 2022

toonn left a comment

Choose a reason for hiding this comment

willcohen commented Jun 9, 2022

SuperSandro2000 left a comment

Choose a reason for hiding this comment

toonn commented Jun 10, 2022

reckenrode commented Jun 10, 2022

reckenrode commented Jun 11, 2022

toonn commented Jun 13, 2022

reckenrode commented Jun 13, 2022

reckenrode commented Jun 13, 2022

nixos-discourse commented Jul 27, 2022

SuperSandro2000 commented Aug 3, 2022

PlumpMath commented May 23, 2023

reckenrode commented Jun 7, 2022 •

edited by SuperSandro2000

Loading

SuperSandro2000 Jun 7, 2022 •

edited

Loading

willcohen commented Jun 9, 2022 •

edited

Loading