package in rye workspace cannot depend on another workspace package for build-system requirements

[mmerickel]

Steps to Reproduce

pyproject.toml

[project]
name = "my-workspace"
version = "0.0.0"

[tool.rye]
virtual = true

[tool.rye.workspace]
members = [
    "src/my-setuptools-extension",
    "src/my-app",
]

src/my-setuptools-extension/pyproject.toml

[build-system]
requires = [
    "setuptools",
    "wheel",
]
build-backend = "setuptools.build_meta"

[project]
name = "my-setuptools-extension"
version = "0.0.0"

dependencies = [
    "setuptools",
]

[project.entry-points."distutils.commands"]
build_webassets = "my_setuptools_ext:BuildWebAssetsCommand"

src/my-app/pyproject.toml

[build-system]
requires = [
    "setuptools",
    "wheel",
    "my-setuptools-extension",
]
build-backend = "setuptools.build_meta"

[project]
name = "my-app"
version = "0.0.0"

src/my-app/setup.py

This step is optional and here for completeness.

import setuptools  # noqa: F401 isort:skip must import before distutils
from distutils.command.build import build as _BuildCommand
from setuptools import setup
from setuptools.command.develop import develop as _DevelopCommand
from setuptools.command.sdist import sdist as _SDistCommand


class SDistCommand(_SDistCommand):
    sub_commands = [('build_webassets', None)] + _SDistCommand.sub_commands


class BuildCommand(_BuildCommand):
    sub_commands = [('build_webassets', None)] + _DevelopCommand.sub_commands


class DevelopCommand(_DevelopCommand):
    def run(self):
        self.run_command('build_webassets')
        _DevelopCommand.run(self)


setup(
    cmdclass={
        'sdist': SDistCommand,
        'develop': DevelopCommand,
        'build': BuildCommand,
    }
)

Expected Result

The workspace should support build-system.requires such that it is built and made available for other packages in the workspace to use.

Actual Result

❯ rye sync
Initializing new virtualenv in /Users/luser/scratch/rye-test/.venv
Python version: cpython@3.12.1
Generating production lockfile: /Users/luser/scratch/rye-test/requirements.lock
error: Failed to build editables
  Caused by: Failed to build editable: file:///Users/luser/scratch/rye-test/src/my-app
  Caused by: Failed to install requirements from build-system.requires (resolve)
  Caused by: No solution found when resolving: setuptools, wheel, my-setuptools-extension
  Caused by: Because my-setuptools-extension was not found in the package registry and you require my-setuptools-extension, we can conclude that the requirements are unsatisfiable.
error: could not write production lockfile for workspace

Caused by:
    failed to generate lockfile

Rye isn't solving this problem and assuming the requirements are in a remote registry.

Version Info

rye 0.27.0
commit: 0.27.0 (2024-02-26)
platform: macos (aarch64)
self-python: cpython@3.12
symlink support: true
uv enabled: true

Additional info

In our current repo, we do this by running a specific build step plus find-links:

$ env/bin/pip wheel -w wheels src/my-setuptools-extension
$ env/bin/pip install --find-links wheels -e src/my-app

[ischaojie]

This may be a problem with uv astral-sh/uv#1661. Have you tried it in the latest version of rye?

[mmerickel]

@ischaojie rye 0.31.0 exhibits the same issue - I think this is very specific to build-system.requires - I don't have problems with just normal (dev-)dependencies between projects in the workspace. The updated error outputs the following:

❯ rye sync
Reusing already existing virtualenv
Generating production lockfile: /Users/luser/scratch/rye-test/requirements.lock
error: Failed to build editables
  Caused by: Failed to build editable: file:///Users/luser/scratch/rye-test/src/my-app
  Caused by: Failed to install requirements from build-system.requires (resolve)
  Caused by: No solution found when resolving: setuptools, wheel, my-setuptools-extension
  Caused by: Because my-setuptools-extension was not found in the package registry and you require my-setuptools-extension, we can conclude that the requirements are unsatisfiable.
error: could not write production lockfile for workspace

Caused by:
    Failed to run uv compile /var/folders/0b/gyxh16fx56d7hd_nqbxj1mxc0000gp/T/.tmp5aChR2/requirements.txt. uv exited with status: exit status: 2

[zanieb]

This should be addressed upstream in the latest release of uv (0.1.27).

[mmerickel]

This should be addressed upstream in the latest release of uv (0.1.27).

@zanieb This issue is still there in the latest rye 0.33 which is using uv 0.1.37.

[charliermarsh]

Yeah I don't think we support this. I don't know how we would know to find my-setuptools-extension locally.

[mmerickel]

Just to say it (ignoring the implementation details and philosophies of rye/uv) - it's in the configured workspace so it could be found. Since it's a build-system dependency instead of a runtime dependency I think conceptually it'd require an initial pass of the workspace to find package names + build dependencies prior to actually building each package, then properly sorting packages so they are built in the right order with respect to build dependencies.

Something similar would be required if build dependencies were supported in lockfiles which I don't think they are right now - correct me if I'm wrong.

[mmerickel]

Anecdotally here is how I do this without rye:

$ uv pip install -U pip
$ .venv/bin/pip wheel \
  -w .venv/wheels \
  -r requirements/01-setup.in -c requirements/01-setup.txt
$ uv pip install \
  --color always \
  --find-links .venv/wheels \
  --refresh \
  --reinstall \
  -r requirements/02-src.in -c requirements/02-src.txt \
  -r requirements/03-dev.in -c requirements/03-dev.txt

We use pip-tools to lock a 01-setup, then build wheels of those into a wheelhouse. Then that wheelhouse is used via --find-links when building the packages that depend on those special build-time dependencies. This strategy also allows us to lock the versions of packages used at build time like setuptools.

edit I had to update this to use --refresh as well because build dependencies are cached by default so updates to the setup dependencies that pip wheel generated were not being used later via uv pip install if the version number on the wheel didn't change (which it doesn't when those are editable dependencies in the workspace).

A workspace would help a lot here to avoid needing the --refresh and all of its gotchas/implications.