poetry selects wrong version for nightly PyTorch on OSX #6150

PaperclipBadger · 2022-08-11T10:32:19Z

I am on the latest Poetry version.
I have searched the issues of this repo and believe that this is not a duplicate.
If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

OS version and name: macOS Monterey 12.4
Poetry version: 1.2.0b3
Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/PaperclipBadger/59a1a7df41ce0af8725a694abf84ec0b

Issue

I'm trying to install the nightly build of PyTorch, so that I can use the new MPS acceleration.

Adding the PyTorch index as a source works (in 1.2.0b3, but not 1.1.14):

[[tool.poetry.source]]
name = "pytorch-nightly-cpu"
url = "https://download.pytorch.org/whl/nightly/cpu"
indexed = true
secondary = true

But poetry selects the 1.13.0.dev20220811+cpu version, which does not have an build for OSX:

% poetry add --allow-prereleases --source pytorch-nightly-cpu torch -vvv 
Loading configuration file /Users/PaperclipBadger/Library/Preferences/pypoetry/config.toml
Loading configuration file /Users/PaperclipBadger/Code/mypackage/poetry.toml
Adding repository pytorch-nightly-cpu (https://download.pytorch.org/whl/nightly/cpu) and setting it as secondary
Using virtualenv: /Users/PaperclipBadger/.pyenv/versions/3.10.5/envs/mypackage
Project environment contains an empty path in sys_path, ignoring.
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS
Creating new session for download.pytorch.org
[urllib3.connectionpool] Starting new HTTPS connection (1): download.pytorch.org:443
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch/ HTTP/1.1" 200 146418
Source (pytorch-nightly-cpu): 120 packages found for torch *
Using version ^1.13.0.dev20220811+cpu for torch

Updating dependencies
Resolving dependencies...
   1: fact: mypackage is 0.1.0
   1: derived: mypackage
   1: fact: mypackage depends on torch (^1.13.0.dev20220811+cpu)
   1: selecting mypackage (0.1.0)
   1: derived: torch (>=1.13.0.dev20220811+cpu,<2.0.0)
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch/ HTTP/1.1" 200 146418
Source (pytorch-nightly-cpu): 1 packages found for torch >=1.13.0.dev20220811+cpu,<2.0.0
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch/ HTTP/1.1" 200 146418
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch-1.13.0.dev20220811%2Bcpu-cp310-cp310-linux_x86_64.whl HTTP/1.1" 200 192738976
... (if left running, this will download every nightly cpu build without finding a match)

adding the dependency directly to pyproject.toml also doesn't work but gives a different error:

torch = {version = "==1.13.0.dev20220811", allow-prereleases = true, source = "pytorch-nightly-cpu"}

% poetry install -vvv
Loading configuration file /Users/PaperclipBadger/Library/Preferences/pypoetry/config.toml
Loading configuration file /Users/PaperclipBadger/Code/mypackage/poetry.toml
Adding repository pytorch-nightly-cpu (https://download.pytorch.org/whl/nightly/cpu) and setting it as secondary
Using virtualenv: /Users/PaperclipBadger/.pyenv/versions/3.10.5/envs/mypackage
Project environment contains an empty path in sys_path, ignoring.
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 1 install, 0 updates, 0 removals, 1 skipped

  • Installing torch (1.13.0.dev20220810+cpu)
  • Installing typing-extensions (4.3.0): Skipped for the following reason: Already installed
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS
Creating new session for download.pytorch.org
[urllib3.connectionpool] Starting new HTTPS connection (1): download.pytorch.org:443
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch/ HTTP/1.1" 200 146418
Skipping wheel torch-1.13.0.dev20220810+cpu-cp310-cp310-linux_x86_64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220810+cpu-cp310-cp310-win_amd64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220810+cpu-cp311-cp311-linux_x86_64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220810+cpu-cp37-cp37m-linux_x86_64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220810+cpu-cp37-cp37m-win_amd64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220810+cpu-cp38-cp38-linux_x86_64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220810+cpu-cp38-cp38-win_amd64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220810+cpu-cp39-cp39-linux_x86_64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220810+cpu-cp39-cp39-win_amd64.whl as this is not supported by the current environment

  Stack trace:

  5  ~/.pyenv/versions/3.10.5/envs/mypackage/lib/python3.10/site-packages/poetry/installation/executor.py:245 in _execute_operation
      243│ 
      244│             try:
    → 245│                 result = self._do_execute_operation(operation)
      246│             except EnvCommandError as e:
      247│                 if e.e.returncode == -2:

  4  ~/.pyenv/versions/3.10.5/envs/mypackage/lib/python3.10/site-packages/poetry/installation/executor.py:317 in _do_execute_operation
      315│             return 0
      316│ 
    → 317│         result: int = getattr(self, f"_execute_{method}")(operation)
      318│ 
      319│         if result != 0:

  3  ~/.pyenv/versions/3.10.5/envs/mypackage/lib/python3.10/site-packages/poetry/installation/executor.py:437 in _execute_install
      435│ 
      436│     def _execute_install(self, operation: Install | Update) -> int:
    → 437│         status_code = self._install(operation)
      438│ 
      439│         self._save_url_reference(operation)

  2  ~/.pyenv/versions/3.10.5/envs/mypackage/lib/python3.10/site-packages/poetry/installation/executor.py:472 in _install
      470│             archive = self._download_link(operation, Link(package.source_url))
      471│         else:
    → 472│             archive = self._download(operation)
      473│ 
      474│         operation_message = self.get_operation_message(operation)

  1  ~/.pyenv/versions/3.10.5/envs/mypackage/lib/python3.10/site-packages/poetry/installation/executor.py:609 in _download
      607│ 
      608│     def _download(self, operation: Install | Update) -> Path:
    → 609│         link = self._chooser.choose_for(operation.package)
      610│ 
      611│         return self._download_link(operation, link)

  RuntimeError

  Unable to find installation candidates for torch (1.13.0.dev20220810+cpu)

  at ~/.pyenv/versions/3.10.5/envs/mypackage/lib/python3.10/site-packages/poetry/installation/chooser.py:103 in choose_for
       99│ 
      100│             links.append(link)
      101│ 
      102│         if not links:
    → 103│             raise RuntimeError(f"Unable to find installation candidates for {package}")
      104│ 
      105│         # Get the best link
      106│         chosen = max(links, key=lambda link: self._sort_key(package, link))
      107│

Looking at the PyTorch index, we see that indeed there are no builds tagged +cpu for OSX - the OSX builds do not have a local version specifier: https://download.pytorch.org/whl/nightly/torch/

torch-1.13.0.dev20220811+cpu-cp310-cp310-linux_x86_64.whl
torch-1.13.0.dev20220811+cpu-cp310-cp310-win_amd64.whl
torch-1.13.0.dev20220811+cpu-cp311-cp311-linux_x86_64.whl
torch-1.13.0.dev20220811+cpu-cp37-cp37m-linux_x86_64.whl
torch-1.13.0.dev20220811+cpu-cp37-cp37m-win_amd64.whl
torch-1.13.0.dev20220811+cpu-cp38-cp38-linux_x86_64.whl
torch-1.13.0.dev20220811+cpu-cp38-cp38-win_amd64.whl
torch-1.13.0.dev20220811+cpu-cp39-cp39-linux_x86_64.whl
torch-1.13.0.dev20220811+cpu-cp39-cp39-win_amd64.whl
torch-1.13.0.dev20220811-cp310-none-macosx_10_9_x86_64.whl
torch-1.13.0.dev20220811-cp310-none-macosx_11_0_arm64.whl
torch-1.13.0.dev20220811-cp37-none-macosx_10_9_x86_64.whl
torch-1.13.0.dev20220811-cp37-none-macosx_11_0_arm64.whl
torch-1.13.0.dev20220811-cp38-none-macosx_10_9_x86_64.whl
torch-1.13.0.dev20220811-cp38-none-macosx_11_0_arm64.whl
torch-1.13.0.dev20220811-cp39-none-macosx_10_9_x86_64.whl
torch-1.13.0.dev20220811-cp39-none-macosx_11_0_arm64.whl

poetry should select 1.13.0.dev20220811, not 1.13.0.dev20220811+cpu. This is what pip does:

% pip install --pre torch --extra-index-url https://download.pytorch.org/whl/nightly/cpu
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/nightly/cpu
Collecting torch
  Using cached https://download.pytorch.org/whl/nightly/cpu/torch-1.13.0.dev20220811-cp310-none-macosx_11_0_arm64.whl (50.6 MB)
Requirement already satisfied: typing-extensions in /Users/PaperclipBadger/.pyenv/versions/3.10.5/envs/mypackage/lib/python3.10/site-packages (from torch) (4.3.0)
Installing collected packages: torch
Successfully installed torch-1.13.0.dev20220811

The text was updated successfully, but these errors were encountered:

dimbleby · 2022-08-13T16:22:13Z

I think poetry behaviour is unlikely to change here.

You'll perhaps do better asking the pytorch project to use a consistent versioning scheme - it looks a lot like an accident that macosx is treated differently from the others and if so they'll surely welcome a bug report / fix.

neersighted · 2022-09-07T15:17:33Z

Closing this as it can be solved with multi-constraint versions, and it's external to Poetry.

PaperclipBadger · 2022-09-08T08:31:07Z

@neersighted can you elaborate on how to solve this with multi-constraint versions? Is this something I can do now as a user?

neersighted · 2022-09-08T08:49:43Z

You can add an additional constraint for macOS that depends on the macOS version number (and keep the +cpu tagged version for platforms that are not macOS).

PaperclipBadger · 2022-09-09T01:45:52Z

I tried that, but it doesn't work (assuming the constraint you meant was platform = "darwin" and not something more specific). Even though poetry correctly determines that mypackage depends on 1.13.0dev20220811 (no +cpu), it only downloads the +cpu binaries for win32 and linux.

I'm pretty sure darwin is the correct platform, even thought the torch binaries are have macosx_11_0 where I would expect the platform to be:

>>> sys.platform
'darwin'

torch = [
  { version = "==1.13.0.dev20220811,!=1.13.0.dev20220811+cpu", platform = "darwin", allow-prereleases = true, source = "pytorch-nightly-cpu" },
  { version = "==1.13.0.dev20220811", platform = "linux", allow-prereleases = true, source = "pytorch-nightly-cpu" }
]

$ poetry install -vvv
Loading configuration file /Users/PaperclipBadger/Library/Preferences/pypoetry/config.toml
Loading configuration file /Users/PaperclipBadger/Code/mypackage/poetry.toml
Adding repository pytorch-nightly-cpu (https://download.pytorch.org/whl/nightly/cpu) and setting it as secondary
Using virtualenv: /Users/PaperclipBadger/.pyenv/versions/3.10.5/envs/mypackage
Project environment contains an empty path in sys_path, ignoring.
Updating dependencies
Resolving dependencies...
   1: fact: mypackage is 0.1.0
   1: derived: mypackage
   0: Duplicate dependencies for torch
   0: Merging requirements for torch (==1.13.0.dev20220811)
   1: fact: mypackage depends on torch (==1.13.0.dev20220811,!=1.13.0.dev20220811+cpu)
   1: selecting mypackage (0.1.0)
   1: derived: torch (==1.13.0.dev20220811)
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS
Creating new session for download.pytorch.org
[urllib3.connectionpool] Starting new HTTPS connection (1): download.pytorch.org:443
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch/ HTTP/1.1" 200 151772
Source (pytorch-nightly-cpu): 2 packages found for torch 1.13.0.dev20220811
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch/ HTTP/1.1" 200 151772
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch-1.13.0.dev20220811%2Bcpu-cp310-cp310-linux_x86_64.whl HTTP/1.1" 304 0
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch-1.13.0.dev20220811%2Bcpu-cp310-cp310-win_amd64.whl HTTP/1.1" 304 0
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch-1.13.0.dev20220811%2Bcpu-cp311-cp311-linux_x86_64.whl HTTP/1.1" 304 0
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch-1.13.0.dev20220811%2Bcpu-cp37-cp37m-linux_x86_64.whl HTTP/1.1" 200 192787858
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch-1.13.0.dev20220811%2Bcpu-cp37-cp37m-win_amd64.whl HTTP/1.1" 200 163106888
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch-1.13.0.dev20220811%2Bcpu-cp38-cp38-linux_x86_64.whl HTTP/1.1" 200 192735786
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch-1.13.0.dev20220811%2Bcpu-cp38-cp38-win_amd64.whl HTTP/1.1" 200 163130031
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch-1.13.0.dev20220811%2Bcpu-cp39-cp39-linux_x86_64.whl HTTP/1.1" 200 192735643
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch-1.13.0.dev20220811%2Bcpu-cp39-cp39-win_amd64.whl HTTP/1.1" 200 163039062
Source (pytorch-nightly-cpu): Downloading wheel: torch-1.13.0.dev20220811%2Bcpu-cp310-cp310-linux_x86_64.whl
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch-1.13.0.dev20220811%2Bcpu-cp310-cp310-linux_x86_64.whl HTTP/1.1" 304 0
   1: fact: torch (1.13.0.dev20220811+cpu) depends on typing-extensions (*)
   1: selecting torch (1.13.0.dev20220811+cpu)
   1: derived: typing-extensions
Creating new session for pypi.org
[urllib3.connectionpool] Starting new HTTPS connection (1): pypi.org:443
[urllib3.connectionpool] https://pypi.org:443 "GET /pypi/typing-extensions/json HTTP/1.1" 304 0
Source (PyPI): 18 packages found for typing-extensions *
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/typing-extensions/ HTTP/1.1" 403 None
Source (pytorch-nightly-cpu): Authorization error accessing https://download.pytorch.org/whl/nightly/cpu/typing-extensions/
   1: selecting typing-extensions (4.3.0)
   1: Version solving took 72.352 seconds.
   1: Tried 1 solutions.

Writing lock file

Finding the necessary packages for the current system

Package operations: 0 installs, 1 update, 0 removals, 1 skipped

  • Updating torch (1.13.0.dev20220811 -> 1.13.0.dev20220811+cpu)
  • Installing typing-extensions (4.3.0): Skipped for the following reason: Already installed
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch/ HTTP/1.1" 200 151772
Skipping wheel torch-1.13.0.dev20220811+cpu-cp310-cp310-linux_x86_64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220811+cpu-cp310-cp310-win_amd64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220811+cpu-cp311-cp311-linux_x86_64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220811+cpu-cp37-cp37m-linux_x86_64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220811+cpu-cp37-cp37m-win_amd64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220811+cpu-cp38-cp38-linux_x86_64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220811+cpu-cp38-cp38-win_amd64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220811+cpu-cp39-cp39-linux_x86_64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220811+cpu-cp39-cp39-win_amd64.whl as this is not supported by the current environment

  Stack trace:

  6  ~/.pyenv/versions/3.10.5/envs/mypackage/lib/python3.10/site-packages/poetry/installation/executor.py:245 in _execute_operation
      243│ 
      244│             try:
    → 245│                 result = self._do_execute_operation(operation)
      246│             except EnvCommandError as e:
      247│                 if e.e.returncode == -2:

  5  ~/.pyenv/versions/3.10.5/envs/mypackage/lib/python3.10/site-packages/poetry/installation/executor.py:317 in _do_execute_operation
      315│             return 0
      316│ 
    → 317│         result: int = getattr(self, f"_execute_{method}")(operation)
      318│ 
      319│         if result != 0:

  4  ~/.pyenv/versions/3.10.5/envs/mypackage/lib/python3.10/site-packages/poetry/installation/executor.py:444 in _execute_update
      442│ 
      443│     def _execute_update(self, operation: Install | Update) -> int:
    → 444│         status_code = self._update(operation)
      445│ 
      446│         self._save_url_reference(operation)

  3  ~/.pyenv/versions/3.10.5/envs/mypackage/lib/python3.10/site-packages/poetry/installation/executor.py:483 in _update
      481│ 
      482│     def _update(self, operation: Install | Update) -> int:
    → 483│         return self._install(operation)
      484│ 
      485│     def _remove(self, operation: Uninstall) -> int:

  2  ~/.pyenv/versions/3.10.5/envs/mypackage/lib/python3.10/site-packages/poetry/installation/executor.py:472 in _install
      470│             archive = self._download_link(operation, Link(package.source_url))
      471│         else:
    → 472│             archive = self._download(operation)
      473│ 
      474│         operation_message = self.get_operation_message(operation)

  1  ~/.pyenv/versions/3.10.5/envs/mypackage/lib/python3.10/site-packages/poetry/installation/executor.py:609 in _download
      607│ 
      608│     def _download(self, operation: Install | Update) -> Path:
    → 609│         link = self._chooser.choose_for(operation.package)
      610│ 
      611│         return self._download_link(operation, link)

  RuntimeError

  Unable to find installation candidates for torch (1.13.0.dev20220811+cpu)

  at ~/.pyenv/versions/3.10.5/envs/mypackage/lib/python3.10/site-packages/poetry/installation/chooser.py:103 in choose_for
       99│ 
      100│             links.append(link)
      101│ 
      102│         if not links:
    → 103│             raise RuntimeError(f"Unable to find installation candidates for {package}")
      104│ 
      105│         # Get the best link
      106│         chosen = max(links, key=lambda link: self._sort_key(package, link))
      107│

neersighted · 2022-09-10T10:40:15Z

Sorry, I realize now that Poetry doesn't consider the two version numbers to be different in a multiple constraint entry. I would suggest solving this by using Poetry's normal resolution on non-macOS platforms and hardcoding direct URL dependencies to the macOS wheels using the correct platform markers. The long-term fix to allow proper resolution would be consistent versioning.

github-actions · 2024-03-01T11:04:03Z

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

PaperclipBadger added kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels Aug 11, 2022

neersighted added kind/question User questions (candidates for conversion to discussion) status/external-issue Issue is caused by external project (platform, dep, etc) labels Sep 7, 2022

neersighted closed this as completed Sep 7, 2022

dimbleby mentioned this issue Sep 10, 2022

"poetry install" (1.2.0) installs the wrong torch version in the first time on OSX #6468

Closed

3 tasks

mkniewallner removed the status/triage This issue needs to be triaged label Sep 18, 2022

dimbleby mentioned this issue Mar 3, 2023

Install CPU version of torch on Mac #7597

Closed

4 tasks

dimbleby mentioned this issue May 16, 2023

Poetry fails resolving PyTorch on MacOS #7933

Closed

4 tasks

dimbleby mentioned this issue May 30, 2023

Instructions for installing PyTorch #6409

Open

1 task

github-actions bot locked as resolved and limited conversation to collaborators Mar 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

poetry selects wrong version for nightly PyTorch on OSX #6150

poetry selects wrong version for nightly PyTorch on OSX #6150

PaperclipBadger commented Aug 11, 2022

dimbleby commented Aug 13, 2022

neersighted commented Sep 7, 2022

PaperclipBadger commented Sep 8, 2022

neersighted commented Sep 8, 2022

PaperclipBadger commented Sep 9, 2022

neersighted commented Sep 10, 2022 •

edited

Loading

github-actions bot commented Mar 1, 2024

poetry selects wrong version for nightly PyTorch on OSX #6150

poetry selects wrong version for nightly PyTorch on OSX #6150

Comments

PaperclipBadger commented Aug 11, 2022

Issue

dimbleby commented Aug 13, 2022

neersighted commented Sep 7, 2022

PaperclipBadger commented Sep 8, 2022

neersighted commented Sep 8, 2022

PaperclipBadger commented Sep 9, 2022

neersighted commented Sep 10, 2022 • edited Loading

github-actions bot commented Mar 1, 2024

neersighted commented Sep 10, 2022 •

edited

Loading