Discussion: longterm Homebrew prefix on Apple Silicon Macs

[mistydemeo]

Feature suggestion

We need to determine which prefix we'll use as the default prefix for Apple Silicon-based Macs on a permanent basis.

A detailed description of the proposed feature

In #9117, we switched to a new prefix of /opt/homebrew for installations on Apple Silicon. This was written and shipped with heroic speed to help prevent strange issues with bleeding edge users on the first consumer Apple Silicon Macs. I'd like to make sure we have a conversation about our longterm plans here before we commit to something. This is an aspect with potential consequences for users, so it's important we've fully weighed the pros and cons before we come to a final decision.

Today, Homebrew's prefix is /usr/local, with the repository in /usr/local/Homebrew. In the initial phase of Apple Silicon migration, in order to avoid using the same prefix for both Intel and ARM binaries, we have two prefixes instead: /usr/local for Intel, and /opt/homebrew for Apple Silicon. Eventually, the need for /usr/local as a Rosetta installation of Homebrew will go away and Apple Silicon users will have only one installation.

Way back in the mists of history, /usr/local was chosen for a few reasons:

1. /usr/local/bin is in the default PATH, so most things a user will use will end up seeing Homebrew-installed tools without configuration.
2. Buildsystems look in /usr/local/include, /usr/local/lib and /usr/local/lib/pkgconfig by default, so non-Homebrew software will find Homebrew-installed libraries without configuration.

These reasons were all in contrast to MacPorts, which installed its software in /opt/local. /usr/local was chosen in response to the issues users were experiencing at the time. We have also received some feedback from users who would prefer a different default prefix, namely:

1. Some non-Homebrew tools also install libraries and binaries in /usr/local, for the same reason Homebrew does, and so there are sometimes conflicts between them.
2. Some users don't want Homebrew-installed software to be found by default / without configuration, making an alternate prefix preferable.

Consequently, the question here is: once Apple Silicon Homebrew becomes the primary installation on Apple Silicon Macs, which should be prefix be?

• /usr/local
• /opt/homebrew
• Something else

We have a bit of time before we start creating our first bottles, so let's make sure we've discussed this in detail before then.

The motivation for the feature

Ensuring the best user experience.

How the feature would be relevant to at least 90% of Homebrew users

Eventually, close to 100% of Homebrew users will be using Apple Silicon.

What alternatives to the feature have been considered

See the discussion above.

[sjackman]

My preference is /opt/homebrew for Homebrew on Apple Silicon. (I'm not on the TSC.)
Thanks for starting this discussion, Misty!

[reitermarkus]

Some non-Homebrew tools also install libraries and binaries in /usr/local, for the same reason Homebrew does, and so there are sometimes conflicts between them.

I'd say /opt/homebrew (or really anything other than /usr/local) is preferable in this case.

/usr/local/bin is in the default PATH

I think we can still accomplish this by adding /etc/paths.d/homebrew.

[claui]

My preference is /opt/homebrew (I’m not a TSC member).

Early feedback from people outside of Homebrew has been largely positive. I’m looking forward to getting one of the most often-cited pain points solved.

[claui]

Sorry fat fingers

[claui]

I think we can still accomplish this by adding /etc/paths.d/homebrew.

Any chance how to get launchd to pick this up, too, so GUI apps can inherit it?

That would keep brewed commands (and scripts that use them) working when launched from Xcode or a text editor.

[iMichka]

Eventually, the need for /usr/local as a Rosetta installation of Homebrew will go away and Apple Silicon users will have only one installation.

I am unsure about this. Some software might never compile under ARM. So we will always need 2 prefixes, one for ARM, one for Rosetta.

[mistydemeo]

My preference is /opt/homebrew for Homebrew on Apple Silicon. (I'm not on the TSC.)

Can you tell me more, from a Linuxbrew perspective, how well this works for users? Do users get confused by libraries and headers not being found by compilers by default?

I am unsure about this. Some software might never compile under ARM. So we will always need 2 prefixes, one for ARM, one for Rosetta.

Rosetta isn't staying around forever. If it's like PowerPC support, we'll only have support for running Intel binaries on ARM for another few years.

[MikeMcQuaid]

Consequently, the question here is: once Apple Silicon Homebrew becomes the primary installation on Apple Silicon Macs, which should be prefix be?

/opt/homebrew.

Reasoning: we've talked a bunch in the past about whether we should move the default prefix away from /usr/local because:

• people endlessly complain about it
• more and more software now dumps crap in there
• superenv has (for a pretty long time now) meant that far less stuff breaks outside of /usr/local
• CMake and friends have a habit of opportunistically linking into /usr/local

The main reason I can see against is that e.g. RubyGems won't automatically find things in /usr/local but, in my experience, this has been becoming more and more of a thing since we use keg-only and versioned formulae more liberally.

You could use a subdirectory under /opt/homebrewindicating the architecture to separate the installation trees, e.g. something like

@cmarquardt Thanks! That's something we considered but it feels inelegant to me given the existing locations. I've locked this thread not because of your (insightful) comment but just generally to keep this discussion limited to Homebrew maintainers.

I am unsure about this. Some software might never compile under ARM. So we will always need 2 prefixes, one for ARM, one for Rosetta.

Agreed @iMichka.

/usr/local/bin is in the default PATH

I think we can still accomplish this by adding /etc/paths.d/homebrew.

In my experience it's less of an issue with PATH and more of an issue with "default compiler lookup paths" for things like RubyGems that don't want to configure this.

[mistydemeo]

In my experience it's less of an issue with PATH and more of an issue with "default compiler lookup paths" for things like RubyGems that don't want to configure this.

Yep - it's the library lookup path and the header lookup path from the compiler, not the executable PATH.

Agreed @\iMichka.

As I mentioned earlier, Rosetta is only staying around for awhile. We should plan for it for the next 2-3 years but we shouldn't have plans that include its existence past that. 😄

[iMichka]

As I mentioned earlier, Rosetta is only staying around for awhile. We should plan for it for the next 2-3 years but we shouldn't have plans that include its existence past that

Remember what happened with windows XP or Internet Explorer 6 ? 😄 I think it will stay for a long time as a lot of software will never get ported.

[mistydemeo]

I think it will stay for a long time as a lot of software will never get ported.

A lot of PowerPC software never got ported and Rosetta 1 went away anyway. Same deal with 32-bit software support. Apple are pretty willing to yank away the rug after having given a few years' advance warning. I don't see Rosetta 2 as being any different from previous cases!

[vitorgalvao]

Making a comment because locking a conversation also blocks reactions (why?).

I fully agree with @mistydemeo. Apple will say bye-bye to Intel as soon as they can and won’t shed a single tear for software that no longer compiles. Tim might’ve said they would be supporting Intel Macs “for years to come”, but apparently that applied to OS support on older Macs, not making new machines (not finding the source, but Greg Joswiak mentioned it in a recent interview).

As for HB’s primary installation, no opposition to /opt/homebrew.

[iMichka]

I'm fine with what we came up then:

• /opt/homebrew for the ARM install location on ARM Macs.
• /opt/homebrew is completely forbidden on Linux due to that (as a side-effect)
• /usr/local for the Intel install location on both Intel and ARM Macs
• /home/linuxbrew/.linuxbrew for Intel or ARM install locations on Linux based on Intel or ARM hardware

[reitermarkus]

I think there is a general agreement on /opt/homebrew.

@Homebrew/tsc, please vote by ticking your name below and change the prefix to your preferred option.

• @fxcoudert (/opt/homebrew)
• @iMichka (/opt/homebrew)
• @MikeMcQuaid (/opt/homebrew)
• @mistydemeo (/usr/local)
• @reitermarkus (/opt/homebrew)

[fxcoudert]

I think /opt/homebrew is the least painful of choices, so let's go with that. We'll need to document it clearly, and it will break a ton of stuff (in particular, a lot of Python packages look for libraries in /usr/local/lib).

[mistydemeo]

Looks like we have a consensus on /opt/homebrew! I'm glad we had this discussion, and I think we've come to a good consensus.

Based on this: Mike's already done the initial work towards the /opt/homebrew prefix on Apple Silicon, so we just need to build on that existing framework.

I'll leave this issue open for the time being so we can document the followup steps. We can close it once we have PRs for action items. Here's a proposed set of items:

• Update the installer to install in /opt/homebrew when running under Apple Silicon. (Currently we bail out on Apple Silicon due to the experimental nature of support.)
• (Optional) Allow the installer to create a Rosetta x86_64 installation even when the installer is run from an ARM64-mode shell.
• Ensure brew.sh or parent process always execs itself in x86_64 mode in a /usr/local prefix on Apple Silicon.
• Either ensure that /opt/homebrew/bin is in the user's default PATH (via /etc/paths.d/homebrew?), or ensure we've documented that users must do this manually.
• Document and publicized the transition, including providing help on how to adapt to the change.

[scpeters]

Users would need to add /opt/homebrew to more environment variables than PATH in order to build against these packages, can we consider providing /opt/homebrew/setup.*sh scripts that would configure environment variables (PATH, LIBRARY_PATH, DYLD_LIBRARY_PATH, CMAKE_PREFIX_PATH, PKG_CONFIG_PATH, etc) for the current shell? I work with a robotics software framework that installs to /opt and provides setup.*sh scripts like this, and I find them very convenient. It's perhaps more essential in that case, as they have a separate installation prefix for each release (/opt/ros/melodic, /opt/ros/noetic, etc), but I think it could still be useful for homebrew as well.

[jonchang]

Hmm, would brew sh in this case add the appropriate paths?

[mistydemeo]

Running commands via brew bundle exec should also help make sure they're found.

[scpeters]

Hmm, would brew sh in this case add the appropriate paths?

Running commands via brew bundle exec should also help make sure they're found.

Those are both good suggestions for building packages in the homebrew environment, which is what my initial comment mentioned, but upon thinking further, there's also the matter of runtime when you would also want DYLD_LIBRARY_PATH to be set if there are any plugins in /opt/homebrew. If it's only two environment variables to enable runtime (PATH and DYLD_LIBRARY_PATH) then I don't think we need an entire script for that.

one quibble I have with brew sh that would keep me from using it more regularly is that it wipes out any environment variables I might have set in my profile, but that's by design I suppose

[fxcoudert]

configure environment variables (PATH, LIBRARY_PATH, DYLD_LIBRARY_PATH, CMAKE_PREFIX_PATH, PKG_CONFIG_PATH, etc)

That's the main downside of leaving /usr/local: many build scripts and compiler tools look there by default. It was cited as an upside by some, though, so maybe we don't want to mess with variables other than PATH. Also, the number of possible variables to add is endless, as each build system has its own.

(DYLD_LIBRARY_PATH should not be necessary, that's for libraries copied outside of their install path, which brew does fix for us automatically.)

[MikeMcQuaid]

Update the installer to install in /opt/homebrew when running under Apple Silicon. (Currently we bail out on Apple Silicon due to the experimental nature of support.)
Document and publicized the transition, including providing help on how to adapt to the change.

We should hold off on this until we are planning on providing official "support". This means having the majority of packages bottled and building.

Ensure brew.sh or parent process always execs itself in x86_64 mode in a /usr/local prefix on Apple Silicon.

I've discussed that in #9287 (review) but it's worth noting also my comments in #9126.

TL;DR: what we care about is installing things in the "wrong" prefix and not running e.g. brew info or brew config as the "wrong" architecture.

Either ensure that /opt/homebrew/bin is in the user's default PATH (via /etc/paths.d/homebrew?), or ensure we've documented that users must do this manually.

brew doctor already enforces this.

[MikeMcQuaid]

Users would need to add /opt/homebrew to more environment variables than PATH in order to build against these packages, can we consider providing /opt/homebrew/setup.*sh scripts that would configure environment variables (PATH, LIBRARY_PATH, DYLD_LIBRARY_PATH, CMAKE_PREFIX_PATH, PKG_CONFIG_PATH, etc) for the current shell?

brew shellenv is already used on Homebrew on Linux for this purpose.

DYLD_LIBRARY_PATH doesn't work how you'd expect on modern macOS so isn't worth setting. CMAKE_PREFIX_PATH and PKG_CONFIG_PATH are already (at least partially) handled by our installations of both and the lack of system installations.

That's the main downside of leaving /usr/local: many build scripts and compiler tools look there by default. It was cited as an upside by some, though, so maybe we don't want to mess with variables other than PATH. Also, the number of possible variables to add is endless, as each build system has its own.

This is already a common problem with keg-only software (e.g. OpenSSL most notably) with pretty simple solutions at this point. As long as Homebrew handles our own usage and provides the necessary paths with helpers like brew shellenv, brew --prefix wget, etc. then the rest is up to other software to consume.

[scpeters]

(DYLD_LIBRARY_PATH should not be necessary, that's for libraries copied outside of their install path, which brew does fix for us automatically.)

I have needed to set DYLD_LIBRARY_PATH when loading a plugin that links to other libraries in a non-standard path, but it is a bit of an edge case.

brew shellenv is already used on Homebrew on Linux for this purpose.

Great! That's just what I was looking for.

DYLD_LIBRARY_PATH doesn't work how you'd expect on modern macOS so isn't worth setting.

If you're referring to system integrity protection, yeah that makes it really confusing. My most recent experience involved wondering why env | grep DYLD would not show any values for DYLD_LIBRARY_PATH when echo $DYLD_LIBRARY_PATH does (it's because I was using /usr/bin/env which is protected by SIP).

[fxcoudert]

I have needed to set DYLD_LIBRARY_PATH when loading a plugin that links to other libraries in a non-standard path, but it is a bit of an edge case

In Catalina and later, I think it's almost unusable (disabled as a security risk). My point was: if something needs to set DYLD_LIBRARY_PATH to work, it's basically a bug and we should treat it as such.

[iMichka]

Heads up that the bottle code might need a small tweak if you introduce a new prefix: see #9310 where I stumbled over an issue with bottles having different prefixes.