[BUG] 3005 onedir package broken on M1 Macs

[sheagcraig]

Description
/opt/salt/bin/run and any commands which call it (e.g. /opt/salt/bin/salt-call.sh` on the Salt 3005 onedir installer (https://repo.saltproject.io/salt/py3/macos/latest/salt-3005-1-macos-x86_64.pkg) result in a lengthy traceback due to openssl architecture issues.

Traceback (most recent call last):
  File "PyInstaller/loader/pyimod03_ctypes.py", line 53, in __init__
  File "ctypes/__init__.py", line 374, in __init__
OSError: dlopen(/opt/homebrew/opt/openssl@1.1/lib/libcrypto.dylib, 0x0006): tried: '/opt/homebrew/opt/openssl@1.1/lib/libcrypto.dylib' (mach-o file, but is an incompatible architecture (have (arm64), need (x86_64))), '/opt/homebrew/Cellar/openssl@1.1/1.1.1q/lib/libcrypto.1.1.dylib' (mach-o file, but is an incompatible architecture (have (arm64), need (x86_64)))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "salt/utils/pyinstaller/rthooks/pyi_rth_subprocess.py", line 7, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod02_importers.py", line 493, in exec_module
  File "salt/utils/pyinstaller/rthooks/_overrides.py", line 10, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod02_importers.py", line 493, in exec_module
  File "salt/utils/vt.py", line 32, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod02_importers.py", line 493, in exec_module
  File "salt/utils/crypt.py", line 8, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod02_importers.py", line 493, in exec_module
  File "salt/loader/__init__.py", line 23, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod02_importers.py", line 493, in exec_module
  File "salt/utils/event.py", line 63, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod02_importers.py", line 493, in exec_module
  File "salt/channel/client.py", line 13, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod02_importers.py", line 493, in exec_module
  File "salt/crypt.py", line 31, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod02_importers.py", line 493, in exec_module
  File "salt/utils/rsax931.py", line 142, in <module>
  File "salt/utils/rsax931.py", line 95, in _init_libcrypto
  File "salt/utils/rsax931.py", line 88, in _load_libcrypto
  File "ctypes/__init__.py", line 452, in LoadLibrary
  File "PyInstaller/loader/pyimod03_ctypes.py", line 55, in __init__
pyimod03_ctypes.install.<locals>.PyInstallerImportError: Failed to load dynlib/dll '/opt/homebrew/opt/openssl@1.1/lib/libcrypto.dylib'. Most likely this dynlib/dll was not found when the application was frozen.
[97316] Failed to execute script 'pyi_rth_subprocess' due to unhandled exception!
[ERROR   ] An un-handled exception was caught by Salt's global exception handler:
PyInstallerImportError: Failed to load dynlib/dll '/opt/homebrew/opt/openssl@1.1/lib/libcrypto.dylib'. Most likely this dynlib/dll was not found when the application was frozen.
Traceback (most recent call last):
  File "PyInstaller/loader/pyimod03_ctypes.py", line 53, in __init__
  File "ctypes/__init__.py", line 374, in __init__
OSError: dlopen(/opt/homebrew/opt/openssl@1.1/lib/libcrypto.dylib, 0x0006): tried: '/opt/homebrew/opt/openssl@1.1/lib/libcrypto.dylib' (mach-o file, but is an incompatible architecture (have (arm64), need (x86_64))), '/opt/homebrew/Cellar/openssl@1.1/1.1.1q/lib/libcrypto.1.1.dylib' (mach-o file, but is an incompatible architecture (have (arm64), need (x86_64)))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "salt/utils/pyinstaller/rthooks/pyi_rth_subprocess.py", line 7, in <module>
    from salt.utils.pyinstaller.rthooks._overrides import PyinstallerPopen
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod02_importers.py", line 493, in exec_module
  File "salt/utils/pyinstaller/rthooks/_overrides.py", line 10, in <module>
    import salt.utils.vt
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod02_importers.py", line 493, in exec_module
  File "salt/utils/vt.py", line 32, in <module>
    import salt.utils.crypt
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod02_importers.py", line 493, in exec_module
  File "salt/utils/crypt.py", line 8, in <module>
    import salt.loader
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod02_importers.py", line 493, in exec_module
  File "salt/loader/__init__.py", line 23, in <module>
    import salt.utils.event
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod02_importers.py", line 493, in exec_module
  File "salt/utils/event.py", line 63, in <module>
    import salt.channel.client
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod02_importers.py", line 493, in exec_module
  File "salt/channel/client.py", line 13, in <module>
    import salt.crypt
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod02_importers.py", line 493, in exec_module
  File "salt/crypt.py", line 31, in <module>
    import salt.utils.rsax931
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "PyInstaller/loader/pyimod02_importers.py", line 493, in exec_module
  File "salt/utils/rsax931.py", line 142, in <module>
    libcrypto = _init_libcrypto()
  File "salt/utils/rsax931.py", line 95, in _init_libcrypto
    libcrypto = _load_libcrypto()
  File "salt/utils/rsax931.py", line 88, in _load_libcrypto
    return cdll.LoadLibrary(_find_libcrypto())
  File "ctypes/__init__.py", line 452, in LoadLibrary
  File "PyInstaller/loader/pyimod03_ctypes.py", line 55, in __init__
pyimod03_ctypes.install.<locals>.PyInstallerImportError: Failed to load dynlib/dll '/opt/homebrew/opt/openssl@1.1/lib/libcrypto.dylib'. Most likely this dynlib/dll was not found when the application was frozen.

Setup
M1 Mac, download and install https://repo.saltproject.io/salt/py3/macos/latest/salt-3005-1-macos-x86_64.pkg

Steps to Reproduce the behavior

1. Run /opt/salt/bin/salt-call.sh

Expected behavior
Salt operates as expected.

Versions Report
Unable to run due to total salt command failure.

Additional context
This has worked (running the x86_64 pkg on M1/arm Macs) up through the last 3004 release.

[twangboy]

I don't think we support M1 macs yet, neither in classic packaging nor onedir packaging.

[rrrix]

@twangboy would the SaltStack project be open to supporting a Configuration Option or Environment Variable to allow a user to explicitly specify the path to their preferred *.dylib library location? Right now, on macOS/Darwin, the logic is a bit wonky, for example just for libcrypto.dylib:

It will look for libcrypto.dylib in the following paths in the following order:

1. /opt/salt/lib/libcrypto.dylib
2. $HOMEBREW_PREFIX/opt/openssl/lib/libcrypto.dylib
3. /usr/local/opt/openssl/lib/libcrypto.dylib
4. /opt/local/lib/libcrypto.dylib
5. only on macOS 10.15 Catalina:
   • /usr/lib/libcrypto.*.dylib - takes the last entry (if found)
6. only before macOS 11 Big Sur:
   • /usr/lib/libcrypto.dylib
7. die() :(

Another interesting problem, is I like to have multiple installations of Salt on my machine at a given time - I usually monkeypatch my Salt Python files just for me because I like to live dangerously while I'm hacking, but others can't or don't want to do that.

Right now Salt is distributed on macOS with amd64/x86_64 architecture binaries, including all the binaries in /opt/salt/lib/*.dylib:

$ lipo -info /opt/salt/lib/libcrypto.1.1.dylib
Non-fat file: /opt/salt/lib/libcrypto.1.1.dylib is architecture: x86_64

If I then try to pip install salt with a native arm64e Python distribution, (like the bundled Python 3.9 in Xcode 14) and then import salt.client, I receive the following exception:

$ /usr/bin/python3 -c import salt.client
/Users/rrrix/git/corpeng/salt-bootstrap/.venv/lib/python3.9/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/rrrix/git/corpeng/salt-bootstrap/.venv/lib/python3.9/site-packages/salt/client/__init__.py", line 28, in <module>
    import salt.cache
  File "/Users/rrrix/git/corpeng/salt-bootstrap/.venv/lib/python3.9/site-packages/salt/cache/__init__.py", line 12, in <module>
    import salt.loader
  File "/Users/rrrix/git/corpeng/salt-bootstrap/.venv/lib/python3.9/site-packages/salt/loader/__init__.py", line 23, in <module>
    import salt.utils.event
  File "/Users/rrrix/git/corpeng/salt-bootstrap/.venv/lib/python3.9/site-packages/salt/utils/event.py", line 63, in <module>
    import salt.channel.client
  File "/Users/rrrix/git/corpeng/salt-bootstrap/.venv/lib/python3.9/site-packages/salt/channel/client.py", line 13, in <module>
    import salt.crypt
  File "/Users/rrrix/git/corpeng/salt-bootstrap/.venv/lib/python3.9/site-packages/salt/crypt.py", line 31, in <module>
    import salt.utils.rsax931
  File "/Users/rrrix/git/corpeng/salt-bootstrap/.venv/lib/python3.9/site-packages/salt/utils/rsax931.py", line 142, in <module>
    libcrypto = _init_libcrypto()
  File "/Users/rrrix/git/corpeng/salt-bootstrap/.venv/lib/python3.9/site-packages/salt/utils/rsax931.py", line 95, in _init_libcrypto
    libcrypto = _load_libcrypto()
  File "/Users/rrrix/git/corpeng/salt-bootstrap/.venv/lib/python3.9/site-packages/salt/utils/rsax931.py", line 88, in _load_libcrypto
    return cdll.LoadLibrary(_find_libcrypto())
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/ctypes/__init__.py", line 444, in LoadLibrary
    return self._dlltype(name)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/ctypes/__init__.py", line 366, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(/opt/salt/lib/libcrypto.dylib, 0x0006): tried: '/opt/salt/lib/libcrypto.dylib' (mach-o file, but is an incompatible architecture (have (x86_64), need (arm64e))), '/opt/salt/.pyenv/versions/3.9.12/openssl/lib/libcrypto.1.1.dylib' (mach-o file, but is an incompatible architecture (have (x86_64), need (arm64e)))
[ERROR   ] An un-handled exception was caught by Salt's global exception handler:
OSError: dlopen(/opt/salt/lib/libcrypto.dylib, 0x0006): tried: '/opt/salt/lib/libcrypto.dylib' (mach-o file, but is an incompatible architecture (have (x86_64), need (arm64e))), '/opt/salt/.pyenv/versions/3.9.12/openssl/lib/libcrypto.1.1.dylib' (mach-o file, but is an incompatible architecture (have (x86_64), need (arm64e)))
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/rrrix/git/corpeng/salt-bootstrap/.venv/lib/python3.9/site-packages/salt/client/__init__.py", line 28, in <module>
    import salt.cache
  File "/Users/rrrix/git/corpeng/salt-bootstrap/.venv/lib/python3.9/site-packages/salt/cache/__init__.py", line 12, in <module>
    import salt.loader
  File "/Users/rrrix/git/corpeng/salt-bootstrap/.venv/lib/python3.9/site-packages/salt/loader/__init__.py", line 23, in <module>
    import salt.utils.event
  File "/Users/rrrix/git/corpeng/salt-bootstrap/.venv/lib/python3.9/site-packages/salt/utils/event.py", line 63, in <module>
    import salt.channel.client
  File "/Users/rrrix/git/corpeng/salt-bootstrap/.venv/lib/python3.9/site-packages/salt/channel/client.py", line 13, in <module>
    import salt.crypt
  File "/Users/rrrix/git/corpeng/salt-bootstrap/.venv/lib/python3.9/site-packages/salt/crypt.py", line 31, in <module>
    import salt.utils.rsax931
  File "/Users/rrrix/git/corpeng/salt-bootstrap/.venv/lib/python3.9/site-packages/salt/utils/rsax931.py", line 142, in <module>
    libcrypto = _init_libcrypto()
  File "/Users/rrrix/git/corpeng/salt-bootstrap/.venv/lib/python3.9/site-packages/salt/utils/rsax931.py", line 95, in _init_libcrypto
    libcrypto = _load_libcrypto()
  File "/Users/rrrix/git/corpeng/salt-bootstrap/.venv/lib/python3.9/site-packages/salt/utils/rsax931.py", line 88, in _load_libcrypto
    return cdll.LoadLibrary(_find_libcrypto())
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/ctypes/__init__.py", line 444, in LoadLibrary
    return self._dlltype(name)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/ctypes/__init__.py", line 366, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(/opt/salt/lib/libcrypto.dylib, 0x0006): tried: '/opt/salt/lib/libcrypto.dylib' (mach-o file, but is an incompatible architecture (have (x86_64), need (arm64e))), '/opt/salt/.pyenv/versions/3.9.12/openssl/lib/libcrypto.1.1.dylib' (mach-o file, but is an incompatible architecture (have (x86_64), need (arm64e)))

Incidentially, here are all of the libcrypto*.dylib files currently on my system:

/opt/homebrew/Cellar/openssl@3/3.0.5/lib/libcrypto.3.dylib
/opt/homebrew/Cellar/openssl@1.1/1.1.1q/lib/libcrypto.1.1.dylib                     * native, very common
/opt/salt/.pyenv/versions/3.9.12/openssl/lib/libcrypto.1.1.dylib                    * x86_64 👎
/Library/Application Support/com.canonical.multipass/lib/libcrypto.1.1.dylib
/Library/AutoPkg/Python3/Python.framework/Versions/3.10/lib/libcrypto.1.1.dylib
/Library/AutoPkg/Python3/Python.framework/Versions/3.7/lib/libcrypto.1.1.dylib
/Library/Frameworks/Python.framework/Versions/3.10/lib/libcrypto.1.1.dylib          * native, very common
/Library/Frameworks/Python.framework/Versions/3.9/lib/libcrypto.1.1.dylib           * native, very common
/usr/local/munki/Python.framework/Versions/3.9/lib/libcrypto.1.1.dylib
/Users/rrrix/Library/Developer/Xcode/macOS DeviceSupport/10.16/usr/lib/libcrypto.dylib

Currently Salt is basically broken for anyone using Apple Silicon that isn't comfortable hacking in the internals of it.

Perhaps something along the lines of:

1. A "common" environment variable such as SALT_LD_LIBRARY_PATH, where one can symlink their desired .dylib files into
2. An environment variable just for SALT_LIBCRYPTO_PATH
3. A Minion configuration option libcrypto_paths: []

Also related: #61340

Seems like any of the above options would be fairly easy and let people have at least some kind of configuration-based workaround instead of forcing their filesystem to the structure that Salt currently expects.

I would love to make this my first PR :) (I've been working on a few patches for other issues, but haven't submitted PR's yet)

[sheagcraig]

@rrrix nice to have someone else in on the Mac party here. I'm just getting started on trying to build 3005 on my M1 to see where things break and start chipping away at this. Looks like you have a way to fix the dylib issue above? I'm not above hacking if you want to give me a nudge in the right direction.

[jpmckinney]

In case it assists other people searching for this issue, my error ends with:

  File "/envs/deploy310/src/salt/salt/cli/ssh.py", line 3, in <module>
    import salt.client.ssh
  File "/envs/deploy310/src/salt/salt/client/__init__.py", line 28, in <module>
    import salt.cache
  File "/envs/deploy310/src/salt/salt/cache/__init__.py", line 12, in <module>
    import salt.loader
  File "/envs/deploy310/src/salt/salt/loader/__init__.py", line 23, in <module>
    import salt.utils.event
  File "/envs/deploy310/src/salt/salt/utils/event.py", line 63, in <module>
    import salt.channel.client
  File "/envs/deploy310/src/salt/salt/channel/client.py", line 13, in <module>
    import salt.crypt
  File "/envs/deploy310/src/salt/salt/crypt.py", line 31, in <module>
    import salt.utils.rsax931
  File "/envs/deploy310/src/salt/salt/utils/rsax931.py", line 142, in <module>
    libcrypto = _init_libcrypto()
  File "/envs/deploy310/src/salt/salt/utils/rsax931.py", line 95, in _init_libcrypto
    libcrypto = _load_libcrypto()
  File "/envs/deploy310/src/salt/salt/utils/rsax931.py", line 88, in _load_libcrypto
    return cdll.LoadLibrary(_find_libcrypto())
  File "/lib/python3.10/ctypes/__init__.py", line 452, in LoadLibrary
    return self._dlltype(name)
  File "/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(/usr/local/opt/openssl@1.1/lib/libcrypto.dylib, 0x0006): tried: '/usr/local/opt/openssl@1.1/lib/libcrypto.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/opt/openssl@1.1/lib/libcrypto.dylib' (no such file), '/usr/local/opt/openssl@1.1/lib/libcrypto.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/usr/local/Cellar/openssl@1.1/1.1.1l_1/lib/libcrypto.1.1.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/Cellar/openssl@1.1/1.1.1l_1/lib/libcrypto.1.1.dylib' (no such file), '/usr/local/Cellar/openssl@1.1/1.1.1l_1/lib/libcrypto.1.1.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))

Edit: If I put env HOMEBREW_PREFIX=/opt/homebrew before the command like in #61340 (comment), I get a new, different error.

  File "/envs/deploy310/src/salt/salt/client/ssh/__init__.py", line 312, in __init__
    self.thin = salt.utils.thin.gen_thin(
  File "/envs/deploy310/src/salt/salt/utils/thin.py", line 658, in gen_thin
    tops = get_tops(extra_mods=extra_mods, so_mods=so_mods)
  File "/envs/deploy310/src/salt/salt/utils/thin.py", line 442, in get_tops
    _add_dependency(tops, mod)
  File "/envs/deploy310/src/salt/salt/utils/thin.py", line 233, in _add_dependency
    if os.path.basename(obj.__file__).split(".")[0] == "__init__":
  File "/lib/python3.10/posixpath.py", line 142, in basename
    p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not NoneType

obj is equal to <module 'salt' (<_frozen_importlib_external._NamespaceLoader object at 0x1027c3d30>)>.

That said, maybe someone else wouldn't get this second error.

Edit: Alright, well I had been importing salt.clients.ssh, which in newer versions seems to do more work (on thin notably). I instead just copied the logic I needed out of that file.

[jpmckinney]

Noting that 3006 fixes the issue for me (on M1 Mac).

[techdragon]

@jpmckinney Do you mean that the x86 package works correctly or something else?

[jpmckinney]

Yes, the pkg at https://docs.saltproject.io/salt/install-guide/en/latest/topics/install-by-operating-system/macos.html#install-macos