Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

In a fresh miniconda environment, pip detects the macOS version number as 10.16, causing the new 1.17.0 onnxruntime wheels to not be found #19371

Closed
joshuacwnewton opened this issue Feb 1, 2024 · 5 comments
Assignees
Labels
platform:windows issues related to the Windows platform

Comments

@joshuacwnewton
Copy link

joshuacwnewton commented Feb 1, 2024

Describe the issue

We have a GitHub Actions workflow that performs the following steps:

  • Installs a set of Python packages on an Ubuntu runner.
  • Freezes all installed packages using pip freeze.
  • Installs the frozen packages across multiple OSs (macOS, Windows, etc.)

The macos-11 runner began failing after the January 31st release of 1.17.0 with the following error:

ERROR: Could not find a version that satisfies the requirement onnxruntime==1.17.0 (from versions: 1.7.0, 1.8.0, 1.8.1, 1.9.0, 1.10.0, 1.11.0, 1.11.1, 1.12.0, 1.12.1, 1.13.1, 1.14.1, 1.15.0, 1.15.1, 1.16.0, 1.16.1, 1.16.2, 1.16.3)
ERROR: No matching distribution found for onnxruntime==1.17.0

The Ubuntu/Windows runners seem to be able to install the frozen requirement onnxruntime==1.17.0 just fine (alongside all of the other requirements we freeze). So, I don't think that this is a dependency conflict with another package.

If I had to guess, this might be because of the change in platform signifiers for macOS wheels:

  • 1.16.3: onnxruntime-1.16.3-cp38-cp38-macosx_10_15_x86_64.whl
  • 1.17.0: onnxruntime-1.17.0-cp38-cp38-macosx_11_0_universal2.whl

But, the docs on universal2 seem to suggest we meet the pip/packaging requirements. (We install pip==23.3.2 and packaging==23.2.)

To reproduce

pip install onnxruntime on a macos-11 runner will install 1.16.3.

pip install onnxruntime==1.17.0 on a macos-11 runner causes a crash.

Note: We pip install into a fresh Python 3.9 conda environment (pip+conda is a bit weird, so I wouldn't want to discount that as the potential cause). The packages that are installed when initializing the conda environment are:

 The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    pip-23.3.1                 |   py39hecd8cb5_0         2.6 MB
    python-3.9.18              |       h5ee71fb_0        12.7 MB
    setuptools-68.2.2          |   py39hecd8cb5_0         950 KB
    tzdata-2023d               |       h04d1e81_0         117 KB
    wheel-0.41.2               |   py39hecd8cb5_0         109 KB
    ------------------------------------------------------------
                                           Total:        16.4 MB

The following NEW packages will be INSTALLED:

  ca-certificates    pkgs/main/osx-64::ca-certificates-2023.12.12-hecd8cb5_0 
  libcxx             pkgs/main/osx-64::libcxx-14.0.6-h9765a3e_0 
  libffi             pkgs/main/osx-64::libffi-3.4.4-hecd8cb5_0 
  ncurses            pkgs/main/osx-64::ncurses-6.4-hcec6c5f_0 
  openssl            pkgs/main/osx-64::openssl-3.0.12-hca72f7f_0 
  pip                pkgs/main/osx-64::pip-23.3.1-py39hecd8cb5_0 
  python             pkgs/main/osx-64::python-3.9.18-h5ee71fb_0 
  readline           pkgs/main/osx-64::readline-8.2-hca72f7f_0 
  setuptools         pkgs/main/osx-64::setuptools-68.2.2-py39hecd8cb5_0 
  sqlite             pkgs/main/osx-64::sqlite-3.41.2-h6c40b1e_0 
  tk                 pkgs/main/osx-64::tk-8.6.12-h5d9f67b_0 
  tzdata             pkgs/main/noarch::tzdata-2023d-h04d1e81_0 
  wheel              pkgs/main/osx-64::wheel-0.41.2-py39hecd8cb5_0 
  xz                 pkgs/main/osx-64::xz-5.4.5-h6c40b1e_0 
  zlib               pkgs/main/osx-64::zlib-1.2.13-h4dc903c_0 

Then, we explicitly upgrade pip in a separate step:

Requirement already satisfied: pip!=21.2.* in ./python/envs/venv_sct/lib/python3.9/site-packages (23.3.1)
Collecting pip!=21.2.*
  Using cached pip-23.3.2-py3-none-any.whl.metadata (3.5 kB)
Using cached pip-23.3.2-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 23.3.1
    Uninstalling pip-23.3.1:
      Successfully uninstalled pip-23.3.1
Successfully installed pip-23.3.2

Urgency

Not terribly urgent for us, but if this affects macOS more generally, then it is probably quite urgent!

Platform

Mac

EDIT: Silly bot! I chose Mac as my OS, but you've added the platform:windows tag. Oops.

OS Version

11

We also tested 12 and the same behavior persists. Presumably, this is also the case for 13/14?

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.17.0

ONNX Runtime API

Python

Architecture

X64

Execution Provider

Default CPU

Execution Provider Library Version

No response

@github-actions github-actions bot added the platform:windows issues related to the Windows platform label Feb 1, 2024
@joshuacwnewton joshuacwnewton changed the title Pip installation of 1.17.0 fails on macos-11 GitHub Actions runner ("ERROR: No matching distribution") Pip installation of 1.17.0 fails on macos GitHub Actions runners ("ERROR: No matching distribution") Feb 1, 2024
@snnn snnn self-assigned this Feb 1, 2024
@snnn
Copy link
Member

snnn commented Feb 2, 2024

I have difficulty on reproducing the issue.

I created a Github Action with macOS-11, then tried to install the latest ONNX Runtime version. It worked well.

https://github.com/snnn/ghtest/actions/runs/7749332259/job/21133675909

@joshuacwnewton
Copy link
Author

joshuacwnewton commented Feb 2, 2024

Well, that's good to hear, at least. It probably means that it isn't urgent or widespread. :)

I tried to reproduce using a fresh miniconda environment, and there were no issues: https://github.com/joshuacwnewton/test-github-actions/actions/runs/7750152011/job/21135944774 EDIT: The miniconda environment was never activated, causing the wheel to get installed for the base environment, which is Python 3.12. This was not a proper reproduction, see below.

I can only assume that the problem lies the specific environment of the project I am working on, then. I will continue debugging on my end and report back when I find out what conditions this happens under.

@thorrester
Copy link

I'm noticing a similar issue. I can install in a macos env, but it leads to segmentation faults in macos github actions.
link

@joshuacwnewton
Copy link
Author

joshuacwnewton commented Feb 2, 2024

I can install in a macos env, but it leads to segmentation faults in macos github actions. link

I think that may be a different issue? (One that would be better served by opening a separate issue.)


My issue is specifically related to pip not being able to see the newly released 1.17.0 under certain circumstances (which I am actively debugging at the moment to try and find reproducible steps for). This is similar to reports for other packages such as...

From the above issue, there is the following remark:

One possibility is the Conda Python is running on Rosetta and is therefore fetching x86_64 images instead. There are a lof of possibilities.

This seems like it might be on the right track, given the change from x64 wheels to universal2 wheels? Regardless, I'm getting the sense that this isn't an onnxruntime issue, and is actually more of a packaging/toolchain issue.

@joshuacwnewton
Copy link
Author

joshuacwnewton commented Feb 2, 2024

Alright, I was able to reproduce the issue in an isolated workflow.

Running python -c "import platform; print(platform.platform())":

10.16 and 11.0 seem to have once been synonyms for the same OS (beta vs. stable), so this looks like an upstream bug of some sort, whether in conda or in pip. In other words, the failure has more to do with the macOS version bump of the onnxruntime wheels (10_15 -> 11_0) than it does the universal2 wheel specifier. And, for some reason, activating a fresh miniconda environment causes the macOS version to be misdetected. Weird!!

Marking this issue as resolved for now. I will try to get to the bottom of how to fix this and update this comment later on, but I'll try not to bother the onnxruntime devs further with any more pings. :P

EDIT: With the problem now well-understood, I've found these related issues:

EDIT2: In the future, this line of debugging could be done much faster using the command python3 -m pip debug --verbose, which I found out about just now.

EDIT3: Looks like this is a dupe of:

That issue suggests the fix of SYSTEM_VERSION_COMPAT=0. And in my testing, I can confirm that this fixes the issue. 🎉

@joshuacwnewton joshuacwnewton changed the title Pip installation of 1.17.0 fails on macos GitHub Actions runners ("ERROR: No matching distribution") In a fresh miniconda environment, pip detects the macOS version number as 10.16, causing the new 1.17.0 onnxruntime wheels to not be found Feb 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
platform:windows issues related to the Windows platform
Projects
None yet
Development

No branches or pull requests

3 participants