Allow users to provide pre-defined metadata for resolution #7442

charliermarsh · 2024-09-16T21:14:09Z

Summary

This PR enables users to provide pre-defined static metadata for dependencies. It's intended for situations in which the user depends on a package that does not declare static metadata (e.g., a setup.py-only sdist), and that is expensive to build or even cannot be built on some architectures. For example, you might have a Linux-only dependency that can't be built on ARM -- but we need to build that package in order to generate the lockfile. By providing static metadata, the user can instruct uv to avoid building that package at all.

For example, to override all anyio versions:

[project]
name = "project"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = ["anyio"]

[[tool.uv.dependency-metadata]]
name = "anyio"
requires-dist = ["iniconfig"]

Or, to override a specific version:

[project]
name = "project"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = ["anyio"]

[[tool.uv.dependency-metadata]]
name = "anyio"
version = "3.7.0"
requires-dist = ["iniconfig"]

The current implementation uses Metadata23 directly, so we adhere to the exact schema expected internally and defined by the standards. Any entries are treated similarly to overrides, in that we won't even look for anyio@3.7.0 metadata in the above example. (In a way, this also enables #4422, since you could remove a dependency for a specific package, though it's probably too unwieldy to use in practice, since you'd need to redefine the rest of the metadata, and do that for every package that requires the package you want to omit.)

This is under-documented, since I want to get feedback on the core ideas and names involved.

Closes #7393.

charliermarsh · 2024-09-16T21:20:19Z

It would be nice if we had some sort of uv show here to make it easy for users to add these... uv show anyio==3.7.0, then copy-paste into pyproject.toml.

charliermarsh · 2024-09-16T21:25:28Z

I'm not a fan of the name [[tool.uv.static-metadata]]. I want [[tool.uv.metadata]] but I know that's very dicey to use here.

zanieb · 2024-09-16T21:39:49Z

What do you think of metadata-overrides? Could version be optional for a blanket override?

(Very cool feature!)

charliermarsh · 2024-09-16T23:02:40Z

Yes and yes.

charliermarsh · 2024-09-16T23:26:16Z

Done and done. Should I go ahead and add docs here @zanieb? Any other concerns?

zanieb · 2024-09-16T23:49:10Z

I don't have any other concerns!

Do we want to introduce new things like this in preview? Or just go for it? (I am leaning towards not using preview until later in uv's life)

charliermarsh · 2024-09-17T01:23:29Z

I was thinking we'd just go for it hah.

charliermarsh · 2024-09-17T02:58:36Z

Added docs.

charliermarsh · 2024-09-17T02:58:59Z

docs/concepts/resolution.md

@@ -258,6 +265,46 @@ If multiple overrides are provided for the same package, they must be differenti
 [markers](#platform-markers). If a package has a dependency with a marker, it is replaced
 unconditionally when using overrides — it does not matter if the marker evaluates to true or false.

+## Metadata overrides


I would appreciate a read here.

charliermarsh · 2024-09-17T02:59:43Z

docs/concepts/resolution.md

@@ -258,6 +265,46 @@ If multiple overrides are provided for the same package, they must be differenti
 [markers](#platform-markers). If a package has a dependency with a marker, it is replaced
 unconditionally when using overrides — it does not matter if the marker evaluates to true or false.

+## Metadata overrides
+
+Metadata overrides allow overriding the metadata of a specific package. For example, to provide


My only complaint with the name ("Metadata overrides") is that the primary goal isn't to override... It's to allow the user to provide static metadata so that we can skip builds. I think override implies changing the metadata in some way, but the goal here is just to enable users to pre-provide the true metadata.

I think override implies changing the metadata in some way, but the goal here is just to enable users to pre-provide the true metadata.

Fair, but I think if we want to frame it that way we need to provide a way to (1) populate the metadata automatically and/or (2) validate that it matches the distribution at install time.

I wouldn't be upset about "Dependency metadata" instead if we plan to add features like (1) and (2). What do you think?

I do want to do (1) and (2), but I probably won't do them ASAP. I don't know what the right API looks like for (1) 🤔

I'm now leaning towards "dependency metadata".

BurntSushi

This is quite the hammer. I like it.

BurntSushi · 2024-09-17T13:42:45Z

crates/uv-resolver/src/lock/mod.rs

+                    if !metadata.requires_dist.is_empty() {
+                        table.insert(
+                            "requires-dist",
+                            value(serde::Serialize::serialize(


Is this just to avoid a use serde::Serialize;?

BurntSushi · 2024-09-17T13:46:02Z

crates/uv/src/lib.rs

@@ -856,7 +859,7 @@ async fn run(cli: Cli) -> Result<ExitStatus> {
                )
                .collect::<Vec<_>>();

-            commands::tool_install(
+            Box::pin(commands::tool_install(


What motivated this?

I got a Clippy warning that the future was too large, I'm not sure why it appeared here, maybe because the size of the settings struct increased. Is Box::pin bad?

No, not at all. That Clippy warning seems worth heeding without evidence to the contrary. Just curious.

I've made a similar change a few times.

BurntSushi · 2024-09-17T13:49:18Z

docs/concepts/projects.md

+Alternatively, you can provide the `flash-attn` metadata upfront via the
+[`metadata-override`](../reference/settings.md#metadata-override) setting, thereby forgoing the need
+to build the package during the dependency resolution phase. For example, to provide the
+`flash-attn` metadata upfront, include the following in your `pyproject.toml`:


I think the question that comes to mind for me when reading this is, "why would I want to skip building a package?" The answer that comes immediately to mind is "for performance," but I'm not sure that's right. I think this is most useful when building a package in some circumstances is difficult or impossible, and this provides an escape hatch. Maybe that context can be added here? Basically, say more about why users might want to use this feature.

charliermarsh · 2024-09-17T17:54:36Z

I rebranded as dependency-metadata. From my perspective it's good to go.

zanieb · 2024-09-17T18:58:46Z

docs/concepts/projects.md

+```toml title="pyproject.toml"
+[[tool.uv.dependency-metadata]]
+name = "flash-attn"
+version = "2.6.3"
+requires-dist = ["torch", "einops"]
+```


Where does this come from? Can we link to the PyPI inspector or something?

This MR contains the following updates: | Package | Update | Change | |---|---|---| | [astral-sh/uv](https://github.com/astral-sh/uv) | patch | `0.4.9` -> `0.4.13` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>astral-sh/uv (astral-sh/uv)</summary> ### [`v0.4.13`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0413) [Compare Source](astral-sh/uv@0.4.12...0.4.13) ##### Enhancements - Add `socks` support ([#7503](astral-sh/uv#7503)) - Avoid warning about bad Python interpreter links for empty project environment directories ([#7527](astral-sh/uv#7527)) - Improve invalid environment warning messages ([#7544](astral-sh/uv#7544)) - Use more verbose spelling of "virtualenv" during creation ([#7523](astral-sh/uv#7523)) - Do not use a user-facing warning for "Waiting to acquire lock..." message ([#7502](astral-sh/uv#7502)) ##### Performance - Use a single buffer for hints on resolver errors ([#7497](astral-sh/uv#7497)) ##### Bug fixes - Allow Python pre-releases to be used if they are first on the `PATH` ([#7470](astral-sh/uv#7470)) - Avoid deleting the project environment directory if it is not a virtual environment ([#7522](astral-sh/uv#7522)) - Do not error if the `CACHEDIR.TAG` file exists but cannot be written to ([#7550](astral-sh/uv#7550)) - Treat invalid platform as more compatible than invalid Python ([#7556](astral-sh/uv#7556)) - Use portable paths when serializing sources ([#7504](astral-sh/uv#7504)) - Compute resolver hints using the final reduced derivation tree ([#7546](astral-sh/uv#7546)) - Bump the wheel and sdist cache versions ([#7560](astral-sh/uv#7560)) - Heal cache entries with missing source distributions ([#7559](astral-sh/uv#7559)) ##### Rust libraries - Bump minimum supported Rust version from 1.80 -> 1.81 ##### Documentation - Add `UV_LINK_MODE` to Docker caching example ([#7510](astral-sh/uv#7510)) - Clarify behavior of of overrides in CLI reference ([#7537](astral-sh/uv#7537)) ### [`v0.4.12`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0412) [Compare Source](astral-sh/uv@0.4.11...0.4.12) ##### Enhancements - Allow users to provide pre-defined metadata for resolution ([#7442](astral-sh/uv#7442)) - Invalidate existing tool environments on Python interpreter mismatch ([#7451](astral-sh/uv#7451)) ##### Bug fixes - Avoid fatal error when searching for egg-info with missing directory ([#7498](astral-sh/uv#7498)) ##### Documentation - Add note on cache growth for self-hosted GitHub runners ([#5757](astral-sh/uv#5757)) ### [`v0.4.11`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0411) [Compare Source](astral-sh/uv@0.4.10...0.4.11) ##### Enhancements - Add `--no-editable` support to `uv sync` and `uv export` ([#7371](astral-sh/uv#7371)) - Add support for `--only-dev` to `uv sync` and `uv export` ([#7367](astral-sh/uv#7367)) - Add support for remaining pip-supported file extensions ([#7387](astral-sh/uv#7387)) - Generate shell completion for `uvx` ([#7388](astral-sh/uv#7388)) - Include `uv export` command in `requirements.txt` output ([#7374](astral-sh/uv#7374)) - Prune unzipped source distributions in `uv cache prune --ci` ([#7446](astral-sh/uv#7446)) - Warn when trying to `uv sync` a package without build configuration ([#7420](astral-sh/uv#7420)) - Support requests for pre-releases in the `--python` option ([#7335](astral-sh/uv#7335)) ##### Bug fixes - Avoid erroneous version warning for `.dist-info` directories ([#7444](astral-sh/uv#7444)) - Avoid removing seed packages for `uv venv --seed` environments ([#7410](astral-sh/uv#7410)) - Avoid unnecessary progress bar initializations ([#7412](astral-sh/uv#7412)) - Error when `tool.uv.sources` contains duplicate package names ([#7383](astral-sh/uv#7383)) - Include `--branch` et al when resolving unnamed URLs in `uv add` ([#7447](astral-sh/uv#7447)) - Include `dev-dependencies` in `--no-sources` invocations ([#7408](astral-sh/uv#7408)) - Include the parent interpreter in Python discovery when `--system` is used ([#7440](astral-sh/uv#7440)) - Respect `--no-sources` in PEP 723 scripts ([#7409](astral-sh/uv#7409)) - Respect `pyproject.toml` credentials from user-provided requirements ([#7474](astral-sh/uv#7474)) - Use consistent PyPI cache bucket ([#7443](astral-sh/uv#7443)) - Use unambiguous relative paths in `uv export` ([#7378](astral-sh/uv#7378)) ##### Documentation - Add documentation on platform-specific dependencies ([#7411](astral-sh/uv#7411)) - Add documentation for passing installer options on Linux ([#6839](astral-sh/uv#6839)) - Separate project data from configuration settings ([#7053](astral-sh/uv#7053)) ##### Error messages - Hint at missing `project.name` ([#6803](astral-sh/uv#6803)) - Surface dedicated `project.name` error for workspaces ([#7399](astral-sh/uv#7399)) - Remove duplicate warning for settings discovery errors ([#7384](astral-sh/uv#7384)) ### [`v0.4.10`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0410) [Compare Source](astral-sh/uv@0.4.9...0.4.10) ##### Enhancements - Allow `uv tool upgrade --all` to continue on individual upgrade failure ([#7333](astral-sh/uv#7333)) - Support globs as cache keys in `tool.uv.cache-keys` ([#7268](astral-sh/uv#7268)) - Add Python package (`__main__.py`) support to `uv run` ([#7281](astral-sh/uv#7281)) - Add zip application support to `uv run` ([#7289](astral-sh/uv#7289)) - Add `--token` option to `self update` command ([#7279](astral-sh/uv#7279)) ##### Performance - Use `globwalk` for `cache-keys` matching ([#7337](astral-sh/uv#7337)) ##### Bug fixes - Always treat archive-like requirements as local files ([#7364](astral-sh/uv#7364)) - Apply `--no-install` options when constructing resolution ([#7277](astral-sh/uv#7277)) - Avoid clobbering existing `py.typed` files contents in `uv init` ([#7338](astral-sh/uv#7338)) - Avoid enforcing platform compatibility when validating lockfile ([#7305](astral-sh/uv#7305)) - Avoid installing transitive dev dependencies ([#7318](astral-sh/uv#7318)) - Avoid selecting prerelease Python installations without opt-in ([#7300](astral-sh/uv#7300)) - Fix PPC64 page size in binary builds. ([#7298](astral-sh/uv#7298)) - Include pre-release Python versions in `uv python list` ([#7290](astral-sh/uv#7290)) - Make version ID optional for source builds ([#7362](astral-sh/uv#7362)) - Support relative paths in `uv add --script` ([#7301](astral-sh/uv#7301)) ##### Documentation - Fix documentation typos for `uv build --build-constraint` flag ([#7330](astral-sh/uv#7330)) - Fix grammatical error in CLI docs ([#7353](astral-sh/uv#7353)) ##### Error messages - Add dedicated lock errors for wheel-only distributions ([#7307](astral-sh/uv#7307)) - Avoid treating `.whl` sources as source distributions ([#7303](astral-sh/uv#7303)) - Clarify Python requirement source for script incompatibilities ([#7339](astral-sh/uv#7339)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).

charliermarsh added the enhancement New feature or improvement to existing functionality label Sep 16, 2024

charliermarsh requested review from BurntSushi, zanieb and konstin September 16, 2024 21:14

charliermarsh force-pushed the charlie/predefined-deps branch 2 times, most recently from 130d476 to da3d55b Compare September 16, 2024 21:16

charliermarsh force-pushed the charlie/predefined-deps branch 3 times, most recently from a9ef2ad to 65c9841 Compare September 16, 2024 23:25

charliermarsh force-pushed the charlie/predefined-deps branch 2 times, most recently from b52a7c9 to 66ac8fe Compare September 17, 2024 02:55

charliermarsh commented Sep 17, 2024

View reviewed changes

konstin approved these changes Sep 17, 2024

View reviewed changes

BurntSushi approved these changes Sep 17, 2024

View reviewed changes

charliermarsh force-pushed the charlie/predefined-deps branch from 66ac8fe to 70a034d Compare September 17, 2024 17:52

charliermarsh force-pushed the charlie/predefined-deps branch from 70a034d to 14df756 Compare September 17, 2024 18:00

zanieb reviewed Sep 17, 2024

View reviewed changes

zanieb approved these changes Sep 17, 2024

View reviewed changes

charliermarsh added 2 commits September 17, 2024 23:04

Allow users to provide pre-defined metadata for resolution

3cd899a

We might need to make this a generic setting rather than project data

e9e349c

charliermarsh added 4 commits September 17, 2024 23:04

Treat as settings

aba68fa

Treat as settings

76aa2af

Rename to metadata-overrides

19beb4a

Rebrand as dependency metadata

42d8b5f

charliermarsh force-pushed the charlie/predefined-deps branch from 14df756 to 42d8b5f Compare September 18, 2024 03:11

charliermarsh enabled auto-merge (squash) September 18, 2024 03:12

charliermarsh merged commit fda2276 into main Sep 18, 2024
58 checks passed

charliermarsh deleted the charlie/predefined-deps branch September 18, 2024 03:18

BrewTestBot mentioned this pull request Sep 18, 2024

uv 0.4.12 Homebrew/homebrew-core#191112

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow users to provide pre-defined metadata for resolution #7442

Allow users to provide pre-defined metadata for resolution #7442

charliermarsh commented Sep 16, 2024 •

edited

Loading

charliermarsh commented Sep 16, 2024

charliermarsh commented Sep 16, 2024

zanieb commented Sep 16, 2024 •

edited

Loading

charliermarsh commented Sep 16, 2024

charliermarsh commented Sep 16, 2024

zanieb commented Sep 16, 2024

charliermarsh commented Sep 17, 2024

charliermarsh commented Sep 17, 2024

charliermarsh Sep 17, 2024

charliermarsh Sep 17, 2024

zanieb Sep 17, 2024

charliermarsh Sep 17, 2024

charliermarsh Sep 17, 2024

BurntSushi left a comment

BurntSushi Sep 17, 2024

BurntSushi Sep 17, 2024

charliermarsh Sep 17, 2024

BurntSushi Sep 17, 2024 •

edited

Loading

zanieb Sep 17, 2024

BurntSushi Sep 17, 2024

charliermarsh commented Sep 17, 2024

zanieb Sep 17, 2024

Allow users to provide pre-defined metadata for resolution #7442

Allow users to provide pre-defined metadata for resolution #7442

Conversation

charliermarsh commented Sep 16, 2024 • edited Loading

Summary

charliermarsh commented Sep 16, 2024

charliermarsh commented Sep 16, 2024

zanieb commented Sep 16, 2024 • edited Loading

charliermarsh commented Sep 16, 2024

charliermarsh commented Sep 16, 2024

zanieb commented Sep 16, 2024

charliermarsh commented Sep 17, 2024

charliermarsh commented Sep 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BurntSushi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BurntSushi Sep 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

charliermarsh commented Sep 17, 2024

Choose a reason for hiding this comment

charliermarsh commented Sep 16, 2024 •

edited

Loading

zanieb commented Sep 16, 2024 •

edited

Loading

BurntSushi Sep 17, 2024 •

edited

Loading