Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optional Dependencies for conda v2 #55

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

travishathaway
Copy link
Contributor

This is a second attempt to draft a CEP for optional dependencies for use by conda compatible package managers (e.g. conda and mamba).

Please see the CEP itself for more information.

Copy link

@h-vetinari h-vetinari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is amazing to see, thanks so much for picking this up!

Just one question/comment about the syntax.

I haven't yet thought through the implications of the virtual packages in detail, but it looks like a very elegant approach!

Comment on lines +59 to +60
# For multiple optional dependencies
conda install package_name[extras=optional_a,optional_b]

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does the parsing here work? It looks like it would break the *args,**kwargs pattern proposed in conda/conda#11053 (which you're referencing already).

I'm also wondering if we really need two ways to do this? package_name[optional_a,optional_b] looks perfectly alright to me, and matches with what pip is doing.

While it's out of scope for this CEP, I'd really like to keep the door open to eventually extend the syntax to be able to use (exposition only, all kwargs need a proper design)

conda install package_name[optional_a,optional_b,cuda=True,license=no_gpl,cpu=avx512,...]

It seems to me that if we add the extras= way to specify optional dependencies, we don't gain much and risk making it much harder to eventually solve that problem.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for following the pip syntax.

@wolfv
Copy link
Contributor

wolfv commented Jun 9, 2023

Can packages in the repodata also depend on other packages + optional dependencies?


### Packager point of view

**TBD**
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Points to take into account here:

  • How will this be specified in the conda-build recipes
  • How will this be exposed in repodata.json
  • How will this be reflected in the info/ metadata folder inside the package

these "virtual metapackages":

```bash
conda remove package_name@optional_a
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know how to feel about the new syntax for an arguably odd case. Why do we need it to begin with? If one wants to remove the extras, they could use the same spec à la package_name[optional_a]? And if you just want to remove the extras, they can simply spell them out.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, but I think there are good use cases for something like the “meta-packages” to exist:

  1. Install package_name[optional_deps]
  2. package_name gets an update and the extra “optional_deps” adds another dependency
  3. You upgrade the (automatically, with the other packages) package_name@optional_deps package and also get the new dependency that was added with the upgrade.

Otherwise, if there are no “virtual meta-packages” (or anything that keeps track of the extras you installed) the user wouldn't get the new dependency.

Also, another thing that I would love to have, but which would differ from the way pip does it (at least to my knowledge), is to not stop at optional dependencies, but go further with optional modules/files.
Let's say my project has the following structure:

src/package/...
src/package/core/...
src/package/feature_a/...
src/package/feature_b/...

And in the meta.yaml I could configure it like this (just an example):

# if files get specifically included in an extra,
# they would be excluded from the "core" package
extras:
    feature_a:
        include_files: ["feature_a/"]
        dependencies: ["optional1",..]
    feature_b:
        include_files: ["feature_b/"]
        dependencies: ["optional2",...]
# maybe also something like feature_c
    feature_c:
        include_extras: ["feature_a","feature_b"]
# optionally also include further files/dependencies etc.

Then conda install package, would only include /core.
And conda install package[feature_a] would include /core and /feature_a and feature_a's dependencies.
And conda install package[feature_c] would include everything.

@mbargull
Copy link
Member

This CEP was mentioned by @chenghlee in a discussion at https://matrix.to/#/!ugIkjExjZULQyVDDNz:gitter.im/$vMqrBnqcqeOfqNq27A4epLJ_xDuIjXiu-59XN_KeBbk?via=gitter.im&via=matrix.org

I didn't yet read the text or discussion much, but just wanted to copy my points from the chat:

There are multiple steps; main points (list not necessarily exhaustive):

  1. Define the actual semantics on what "optional"/"extra" would mean (ask different people and you'd get different answers...).
    I.e., figure out how this fits into the general dependency resolution and what defaults should be (and look out for typical corner cases, e.g., differences between env create/install/update.)
  2. Define the data storage/communication layer, i.e., "repodata CEP update" and stuff.
  3. Define the user interface for this; e.g., (amongst other things) how would a user say "install foo with bar".
    • This cannot be install foo[bar] since we have [...] already reserved for "match spec".
    • Would have to be foo[extra=bar]/foo[extras=[bar]] or the like if this would use he match spec syntax; could be something entirely different, too, though.

@xhochy
Copy link

xhochy commented Mar 1, 2024

I don't understand why we need this feature. We already use in several cases multi-output recipes for optional dependencies where a package[feature] on the pip side is translated to a package-feature package. For the sake of simplicity, I would prefer to avoid to define an additional way of doing it and keep the current way of doing it.

@h-vetinari
Copy link

We already use in several cases multi-output recipes for optional dependencies where a package[feature] on the pip side is translated to a package-feature package.

Yes, but it's a crutch, and a persistent paper cut that people run into and contributes substantially to friction and user-frustration for requirement-incompatibilities between PyPI and conda. Even a bunch of conda-forge recipes get this wrong (and don't even get a warning that this doesn't do what's expected!). It should IMO absolutely be fixed just based on those two points alone.

If we zoom out a little bit further, this "string-based work arounds instead of sufficiently expressive syntax/metadata" is a constant source of pain (the other huge pain point on this being the completely overloaded build strings), which I mention here because IMO both should be solved with the same syntax.

conda install package[feature,build_number=1,cuda=True]

would be orders of magnitude more user-friendly than the following jumble of symbols

conda install package-feature=*=cuda*_1

and that's even before getting into conda/conda#11442, conda/conda-build#4498, conda/conda-libmamba-solver#446, etc.

@xhochy
Copy link

xhochy commented Mar 17, 2024

My main concern is that I want to avoid is adding an attribute to repodata.json as this will take longer to propagate to all implementations.

It would be nice if the problem could be solved by mapping the square bracket syntax to plain packages, i.e. make this a client-side convention.

One example could be to map jaxlib[build_number=1,cuda,docs] to jaxlib=*=*_1 jaxlib-cuda jaxlib-docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants