pypi-mirror failing for macos and Linux packages when run on Windows

[ethanwhite]

Replicate (on a Windows machine; replicated with Python 3.11 and 3.12; works fine on Ubuntu 22.04):

conda create -n odstest python=3
conda activate odstest
pip install offlinedatasci
offlinedatasci install python_libraries /ods/path

Or directly using pypi-mirror:

import pypi_mirror

pypi_mirror.download(platform = ['macosx_10_2_x86_64'],
                     pip = 'pip3',
                     dest = '~/odstest/pypi-download/'
                     pkgs = ["matplotlib", "notebook", "numpy", "pandas"],
                     python_version = '3.12',
                     allow_binary = True)

pip struggles repeatedly to rectify dependencies in Jupyter land and then fails with:

ERROR: Cannot install notebook==6.5.5 and notebook==6.5.6 because these package versions have conflicting dependencies.

ERROR: Cannot install notebook==6.5.5 and notebook==6.5.6 because these package versions have conflicting dependencies.

The conflict is caused by:
    notebook 6.5.6 depends on pyzmq<25 and >=17
    notebook 6.5.5 depends on pyzmq<25 and >=17

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
Error in function: download_python_libraries. Error: Command '['pip3', 'download', '-d', WindowsPath('pythonlibraries'), '--only-binary', ':all:', '--platform', 'manylinux_2_17_x86_64', '--python-version', '3.11', 'matplotlib', 'notebook', 'numpy', 'pandas']' returned non-zero exit status 1.

My best guess is that even though we're asking it to download for macOS something about the system being Windows is still being seen by pypi_mirror causing problems with finding dependencies that fit. Hopefully some additional optional argument can fix this.

Note that pypi_mirror.download() is a thin wrapper around pip download, so one route is to replicate with the core pip call which should open up more useful avenues for searching/asking for help.

[ethanwhite]

I've been trying to track this down and my current guess is that it relates to this line in the jupyter-core pyproject.toml:

dependencies = [
  "platformdirs>=2.5",
  "traitlets>=5.3",
  "pywin32>=300 ; sys_platform == 'win32' and platform_python_implementation != 'PyPy'"
]

Running the following line on Windows:

pip download -d test-dl --platform macosx_10_12_X86_64 --python-version 311 --implementation cp --abi cp311 --only-binary=:all: notebook

Yields a string of errors of the form:

The conflict is caused by:
    jupyter-core 5.7.2 depends on pywin32>=300; sys_platform == "win32" and platform_python_implementation != "PyPy"

This suggests that somehow sys_platform (which is the result of sys.platform; see https://hatch.pypa.io/1.9/config/dependency/) is still getting passed through as "win32", thus triggering a Windows dependency that by definition can't be satisfied.

[ethanwhite]

I can confirm that this ( ⬆️ ) is the source of the error. Running with --debug shows that the failure is due to:

Criterion((SpecifierRequirement('pywin32>=300; sys_platform == "win32" and platform_python_implementation != "PyPy"')

Looks like this is a known issue pypa/pip#11664 (and many smaller issues including pypa/pip#12466)

[ethanwhite]

Also fails to download Linux packages

[ethanwhite]

OK, so this is a widely known issue with pip and therefore something we aren't going to fix. I'm going to patch this by only running creating the Windows mirror if the OS is Windows and once pypa/pip#11664 is fixed we can add full functionality for Windows teaching servers.