codesign on ARM requires CLT or Xcode to sign binaries

[fxcoudert]

1. On Apple Silicon, we use codesign to provide an ad hoc signature to the MachO files we pour from bottles, after rewriting them (binaries and shared libraries)
2. codesign is present at /usr/bin/codesign on base macOS (without Xcode or CLT)
3. But that version can only verify signatures, not sign MachO files. Signing requires codesign_allocate to be installed, which is called by codesign, and is part of Xcode or CLT.
4. Therefore, we cannot reliably pour bottles for Apple Silicon unless we have CLT or Xcode installed.

This leads to very bad user experience. Moreover, it also happens with brew-downloaded git bottles for ARM (now that we have them): leading to horrible user experience. #7857 (comment)

As a long-term solution, we could imagine having ruby-macho providing a pure-ruby signing mechanism. That would be ideal, but the first implementation had problems. poking @woodruffw on that

As a short-term solution, we need to:

• bail out early when neither CLT nor Xcode is installed on ARM
• in that case, tell users to install the CLT

[fxcoudert]

As for the reason why this happened: the user reports that installing the 11.1.0 RC uninstalled their previously installed CLT 😢

[SMillerDev]

As far as I remember, enrolling in betas that don't have a CLT version available will remove the CLT. Just like installing a normal update will usually have an update for the CLT as well. All brew installs need a CLT or XCode though, so I think aborting with a failure is the friendliest thing to do.

[mbirth]

Funny thing: The CLT is downloadable from the Developer Website, it's just not in their software repository.

[carlocab]

Not too surprising: it's still only RC, and not yet fully supported.

[johnalanwoods]

@fxcoudert, so as we know, HB installs CLT on initial install.

So, this will only affect users who are savvy enough to beta test macOS/RCs, (where the CLT might be removed).

I don't see this as a huge issue, don't you agree? In all circumstances home-brew would have installed CLT on any stable macOS release prior to use.

[mbirth]

Just to remind you guys: "RC"s are what were "GM"s before - basically the release version, only a few days earlier. So it's not so weird for people to upgrade to an RC version if they don't want to wait hours for the download to complete on release day.

[carlocab]

basically the release version, only a few days earlier

I think "basically" and "earlier" are keywords here: there are still some features that distinguish it from the released version, and that makes it not fully supported.

it's not so weird for people to upgrade to an RC version if they don't want to wait hours for the download to complete on release day

I agree. But the price you pay for avoiding (among other things) the download wait on release day is that you get a piece of software that doesn't have the same guarantees as to the released version. One of those is the availability of CLT via softwareupdate. The workaround isn't difficult; though perhaps brew could generate more informative error messages here (which is actually part of the suggested short-term solution in the OP).

[SMillerDev]

Just to remind you guys: "RC"s are what were "GM"s before - basically the release version, only a few days earlier.

And they aren't supported by brew as brew doctor will/should tell you.

The developer previews of macOS 12 will also be like the release version, just a few months earlier and with bugs that might still be fixed later.

[gibfahn]

As the installer installs the CLT anyway, and it's already listed as a requirement in the installation instructions, it seems reasonable to assume everyone will always have it and thus always do the MacOS::CLT.installed? check. Should only affect folks who have accidentally lost their CLT/Xcode due to issues like this one.

(I have also been hit by this when upgrading to RCs, feels like an issue to raise with Apple via https://feedbackassistant.apple.com/ rather than brew though)

[fxcoudert]

For @woodruffw or whoever is willing to implement this in ruby: Go has made its own implementation, see https://github.com/golang/go/blob/master/src/cmd/internal/codesign/codesign.go

The small details of the implementation are discussed at length in the two linked issues: golang/go#42684 and golang/go#38485

[carlocab]

Upgrading to 11.1 gets rid of CLT. xcode-select --install now works, however.

[johnalanwoods]

It doesn't remove the CLT, you just need to select it again in Xcode prefs (or execute a path switch). ![Uploading 102193907...]

[carlocab]

Xcode prefs? I’m afraid I can’t do that since I don’t have Xcode installed...

But I’ll have a look to see if Homebrew just misplaced the CLT.

[johnalanwoods]

@carlocab, still don't need to reinstall.
You just need to select the path for the CLT again.

sudo xcode-select -switch /Library/Developer/CommandLineTools

My point is, installed 11.1 (or other system updates), doesn't delete the CLT.

[fxcoudert]

@johnalanwoods it may depend on the circumstances, because on our CI machines, the update to 11.1 definitely removed the CLT entirely.

[carlocab]

Yep, it definitely removed CLT in my case. When CLT is installed, this happens:

❯ xcode-select --install
xcode-select: error: command line tools are already installed, use "Software Update" to install updates

Predictably, that is not the error message I saw when I ran it after upgrading to 11.1.