wheel 0.36.0 fails on macOS 11 (Big Sur) when MACOSX_DEPLOYMENT_TARGET=11

[fxcoudert]

On previously macOS versions, the major version were 10.15, 10.14, etc. But on macOS Big Sur (11.x.y) the major version is now 11, and the next major version will be 12, etc.

Therefore, because MACOSX_DEPLOYMENT_TARGET typically specifies a major macOS version, it is now natural to have MACOSX_DEPLOYMENT_TARGET=11. This is accepted by all system build tools and compilers.

This triggered several build failures in cpython 3.9.0, which are now fixed (see python/cpython#23556). This is also triggering problems now in wheel. We are seeing this error as part of Homebrew build and testing (Homebrew/homebrew-core#66063 (comment)).

---

I install Python 3.9.0, with an upstream patch (python/cpython#23556), and wheel 0.36.0. Then I run pip3 install markupsafe, which gives the following error:

    File "/usr/local/Cellar/python@3.9/3.9.0_5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/local/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 337, in run
      impl_tag, abi_tag, plat_tag = self.get_tag()
    File "/usr/local/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 260, in get_tag
      plat_name = get_platform(self.bdist_dir)
    File "/usr/local/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 52, in get_platform
      result = calculate_macosx_platform_tag(archive_root, result)
    File "/usr/local/lib/python3.9/site-packages/wheel/macosx_libfile.py", line 356, in calculate_macosx_platform_tag
      assert len(base_version) == 2
  AssertionError
  ----------------------------------------
  ERROR: Failed building wheel for markupsafe

[fxcoudert]

What's happening? calculate_macosx_platform_tag is called with platform_tag being macosx-11-x86_64. Therefore base_version becomes (11, ), and this triggers the assert on line 356. I'm not actually sure why that assert is there, it seems contradictory with the check on len(base_version) above.

Even if we remove that assert, or the platform_tag were different: if MACOSX_DEPLOYMENT_TARGET=11 is set in the environment, then the code below will set base_version to deploy_target, which will be (11, ). That's not wrong, but it will trigger another assert on line 371.

If I simply remove the two asserts, then I can run pip3 install markupsafe and I get this behaviour:

Collecting markupsafe
  Downloading MarkupSafe-1.1.1.tar.gz (19 kB)
Building wheels for collected packages: markupsafe
  Building wheel for markupsafe (setup.py) ... done
  Created wheel for markupsafe: filename=MarkupSafe-1.1.1-cp39-cp39-macosx_11_0_x86_64.whl size=16482 sha256=da3dc29e1dbf01ee611984e85a58e822211c599221d5e2cafe8ce5e15694ba53
  Stored in directory: /Users/fx/Library/Caches/pip/wheels/e0/19/6f/6ba857621f50dc08e084312746ed3ebc14211ba30037d5e44e
Successfully built markupsafe
Installing collected packages: markupsafe
Successfully installed markupsafe-1.1.1

I am not sure whether the platform tag macosx_11_0_x86_64 is what is expected here, or if it should be macosx_11_x86_64 (i.e., only recording the major macOS version).

[fxcoudert]

On further thought, I think since distutils.util.get_platform() returns macosx-11-x86_64, the platform tag should be macosx_11_x86_64 according to PEP 425.

[agronholm]

@Czaki comments? It's your code.

[agronholm]

Also, how come the existing tests don't fail when run on macOS 11? Does this only happen when the env variable is set?

[fxcoudert]

@agronholm I don't know how you run your tests. This happens when MACOSX_DEPLOYMENT_TARGET=11 at runtime, or when Python was built with MACOSX_DEPLOYMENT_TARGET=11.

[Czaki]

@fxcoudert could You check MACOSX_DEPLOYMENT_TARGET=11.0? Every time when I work with MACOSX_DEPLOYMENT_TARGET I set <mayor>.<minor>.

I also check pypi warehouse source code: https://github.com/pypa/warehouse/blob/0b6c95b1318a1572507d33f8abf220eef2984b19/warehouse/forklift/legacy.py#L127 and its requirements to have mayor and minor provided. So tag should be macosx_11_0_x86_64

I do not upgrade to BigSur yet.

When I working on this code there is no option to build a wheel for the system below 10.6. So I do not expect to provide only major version.

[fxcoudert]

@Czaki the numbering of macOS has changed from 10.major.minor to major.minor.minor2. Like you said, it is usual (but not mandatory) to set MACOSX_DEPLOYMENT_TARGET to the major macOS version: before, that was 10.15, 10.14, etc. Now it is 11, and the next major version will be 12.

In any case, MACOSX_DEPLOYMENT_TARGET=11 is valid and should not lead to failure.

[Czaki]

@Czaki the numbering of macOS has changed from 10.major.minor to major.minor.minor2. Like you said, it is usual (but not mandatory) to set MACOSX_DEPLOYMENT_TARGET to the major macOS version: before, that was 10.15, 10.14, etc. Now it is 11, and the next major version will be 12.

In any case, MACOSX_DEPLOYMENT_TARGET=11 is valid and should not lead to failure.

Ok. I understand. At this moment I do not have access to a machine with BigSur. I stuck on the 10.x series. I could create PR with a patch but I cannot test it locally.
If MACOSX_DEPLOYMENT_TARGET=11.0 cause build to pass the fix will be short. I only request your help.

During writing this code it is impossible to predict that Apple will change the numbering scheme.

But to solve this problem there should be a decision done in the warehouse. Currently, the warehouse will accept macosx_11_0_x86_64 and decline macosx_11_x86_64. As you pointed, it should accept macosx_11_x86_64, because 0 is the patch number. And this decision should be made by a proper person from pypa (@agronholm - did you have an idea who should be called)

To prepare the patch I also need some .so or .dylib file to check the results of binary file inspection.

@fxcoudert If I prepare a patch did you have time to test it?

[fxcoudert]

According to pypa/packaging#319 (comment):

In order to actually remove the minor version we'd have to coordinate with wheel (and maybe create a PEP?) Instead we just pin the minor version to 0.

It seems there is a need for a coordinated discussion about this.

[Czaki]

Ok. So I will prepare bugfix patch. But I still have a problem with testing it.

[ronaldoussoren]

Please pin the minor version to 0 and treat MACOSX_DEPLOYMENT_TARGET=N as MACOSX_DEPLOYMENT_TARGET=N.0.

There are already wheels for macosx_11_0_universal2 on PyPI, and PyPI requires that there is a minor number.

[fxcoudert]

@Czaki I can help test, for sure

[emeryberger]

The fix in e6102e5 only works for Mac x86-64; it still fails on Apple Silicon.

[agronholm]

Would vendoring the latest packaging help?

[alphavector]

The fix in e6102e5 only works for Mac x86-64; it still fails on Apple Silicon.

@emeryberger, it actually works for Apple Silicon. Some packages will build their own separate dedicated environment with an old wheel version when installed (e.g. numpy). You need to write an issue for each package to make the wheel version >=0.36.1.

[agronholm]

I do wonder what reason they have for doing that.

[gzz2000]

I do wonder what reason they have for doing that.

I am building numpy on M1. I succeeded by checking out the newest commit, instead of staying at 1.20.1 release. They seems to have done exactly the same thing: to upgrade wheel, in this commit: numpy/numpy@0a8dca0

[mattip]

The problem is not wheel, it is packaging. If you do not specify any packaging version, you get an older one. I am not sure where it comes from. Maybe wheel should specify a dependency on a particular minimum version of packaging?

[gzz2000]

The problem is not wheel, it is packaging. If you do not specify any packaging version, you get an older one. I am not sure where it comes from. Maybe wheel should specify a dependency on a particular minimum version of packaging?

Yes it is packaging. I guess upgrading packaging would also make wheel upgraded. Before upgrading it, I was encountering errors like this, which indicated that it was a wheel version issue:

  writing manifest file 'numpy.egg-info/SOURCES.txt'
  Copying numpy.egg-info to build/bdist.macosx-11-arm64/wheel/numpy-1.20.1-py3.9.egg-info
  running install_scripts
  Traceback (most recent call last):
    File "/opt/homebrew/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
      main()
    File "/opt/homebrew/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/opt/homebrew/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py", line 204, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
    File "/private/var/folders/70/cgczdqgs207c849t2v498q580000gp/T/pip-build-env-e_cshiz1/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 211, in build_wheel
      return self._build_with_temp_dir(['bdist_wheel'], '.whl',
    File "/private/var/folders/70/cgczdqgs207c849t2v498q580000gp/T/pip-build-env-e_cshiz1/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 197, in _build_with_temp_dir
      self.run_setup()
    File "/private/var/folders/70/cgczdqgs207c849t2v498q580000gp/T/pip-build-env-e_cshiz1/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 248, in run_setup
      super(_BuildMetaLegacyBackend,
    File "/private/var/folders/70/cgczdqgs207c849t2v498q580000gp/T/pip-build-env-e_cshiz1/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 142, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 513, in <module>
      setup_package()
    File "setup.py", line 505, in setup_package
      setup(**metadata)
    File "/private/var/folders/70/cgczdqgs207c849t2v498q580000gp/T/pip-install-3gtrssd_/numpy_d4eaa76fbd3a463d9624512327629d0d/numpy/distutils/core.py", line 169, in setup
      return old_setup(**new_attr)
    File "/private/var/folders/70/cgczdqgs207c849t2v498q580000gp/T/pip-build-env-e_cshiz1/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 165, in setup
      return distutils.core.setup(**attrs)
    File "/opt/homebrew/Cellar/python@3.9/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/opt/homebrew/Cellar/python@3.9/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/opt/homebrew/Cellar/python@3.9/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/private/var/folders/70/cgczdqgs207c849t2v498q580000gp/T/pip-build-env-e_cshiz1/overlay/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 328, in run
      impl_tag, abi_tag, plat_tag = self.get_tag()
    File "/private/var/folders/70/cgczdqgs207c849t2v498q580000gp/T/pip-build-env-e_cshiz1/overlay/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 252, in get_tag
      plat_name = get_platform(self.bdist_dir)
    File "/private/var/folders/70/cgczdqgs207c849t2v498q580000gp/T/pip-build-env-e_cshiz1/overlay/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 48, in get_platform
      result = calculate_macosx_platform_tag(archive_root, result)
    File "/private/var/folders/70/cgczdqgs207c849t2v498q580000gp/T/pip-build-env-e_cshiz1/overlay/lib/python3.9/site-packages/wheel/macosx_libfile.py", line 356, in calculate_macosx_platform_tag
      assert len(base_version) == 2
  AssertionError

[agronholm]

The problem is not wheel, it is packaging. If you do not specify any packaging version, you get an older one. I am not sure where it comes from. Maybe wheel should specify a dependency on a particular minimum version of packaging?

Wheel vendors packaging. But the traceback here indicates a problem that has already been fixed in the latest wheel release.

[arogozhnikov]

I have M1 and upgraded to newest versions: wheel==0.36.2, packaging==21.0, but still get the same exception when installing scipy (numpy installed from wheels just in case).

I've also tried running this one-time command without success.

[agronholm]

Are you sure it's actually using the latest wheel? Do other libraries build binary wheels properly? If so, then the culprit is most likely the scientific Python stack with its custom build processes.

[keleftheriou]

Possibly related: python/cpython#102362