Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python: parse conda.env in addition to requirements.txt #2227

Open
RXminuS opened this issue Oct 23, 2018 · 24 comments
Open

Python: parse conda.env in addition to requirements.txt #2227

RXminuS opened this issue Oct 23, 2018 · 24 comments
Labels
Keep Exempt this from being marked by stalebot T: new-ecosystem Requests for new ecosystems/languages

Comments

@RXminuS
Copy link

RXminuS commented Oct 23, 2018

Right now Depandabot can only deal with requirement.txt files from PyPy but a lot of our repositories use Anaconda yaml files because of the performance increase with the Tensorflow installation. Thus, adding Conda yaml support would be essential for us!

@greysteil
Copy link
Contributor

Interesting - are there any open source repos you can point me to that use that setup (that I can use as fixtures when I write this)?

@RXminuS
Copy link
Author

RXminuS commented Oct 23, 2018

https://github.com/fastai/fastai the environment.yml (or environment.yaml) It's fairly popular in ML/AI communities to do it like this

@gr33ndata
Copy link

I second that, it would be useful to update both requirements.txt and environment.yml

@palfrey
Copy link

palfrey commented May 21, 2019

I'm using Conda requirement files (https://github.com/palfrey/waveform-necklace/blob/master/conda-requirements.txt) and hitting issues (see palfrey/waveform-necklace#19)

@oji
Copy link

oji commented Sep 16, 2019

We use conda as well, with the environment.yml variant. Is there any plan to add support for this? Thanks!

@rebelagentm
Copy link
Contributor

@palfrey @oji Thank you! We think this will be useful and agree on supporting it, but the team is stretched thin right now as we work to integrate and scale Dependabot into GitHub. :octocat:

If you're keen to help us add support, we would be happy to review any pull requests for it on dependabot-core.

@infin8x infin8x transferred this issue from dependabot/feedback Jun 29, 2020
@infin8x infin8x added the T: new-ecosystem Requests for new ecosystems/languages label Jul 2, 2020
@alexvicegrab
Copy link

Would be useful to have.

@epruesse
Copy link

This would be very interesting and useful to have.

Some notes:

  • conda is a general purpose package manager not limited to Python
  • anaconda cloud is the default service for hosting packages
  • miniconda is the minimal distribution including the package manager
  • channels are collections/distributions of packages. Anyone can create one, by default hosted at anaconda cloud, but not necessarily so.
  • Common channels besides the basic defaults it comes with conda-forge for general purpose software and bioconda for life science specific packages.
  • There is (currently) no standard for placing the environment.yaml. The file format really just describes an arbitrary selection of packages one might have in a conda "virtual environment".
  • A more specific file would be recipe/meta.yaml, which contains be the package build instructions (different format), including dependencies split into build, host and run (where build vs host separates the build system from development libraries in the case of cross compilation).

@brl0
Copy link

brl0 commented Jun 11, 2021

I too think this would be great.

The community around this package manager (and some related ones) really seems to be growing and blossoming. I think it is also worth noting that while many conda related packages and resources are python related, it is not strictly a python package/environment manager, as it also supports Python, R, Node.js, Java, among others.

Not sure if this is useful, but here are some possibly relevant resources.

I would think the conda-forge community, probably the core team in particular, would be the most authoritative resource (besides the original developers Anaconda/ContinuumIO).

I think it is worth noting the derivative or related package managers:

  • miniconda: Miniconda is a free minimal installer for conda
  • miniforge: minimal installer for Conda specific to conda-forge
  • mamba: reimplementation of the conda package manager in C++

Another related project of interest is conda-lock, which is hosted by the conda-incubator organization.

Libraries.io bibliothecary (also written in Ruby) supports conda (see here) using a simple python web service conda-parser.

Also recently saw a couple of interesting and possibly loosely relevant packages in the Regro GitHub organization, which is a sister organization to conda-forge. In particular, these repositories do a lot of work mapping conda dependencies:

Binderhub is an open source project and service that allows users to share reproducible interactive computing environments from code repositories, and it can make use of conda environment files via repo2docker. I think that adding conda support to dependabot would help make it easier to maintain and support these environments.

Hopefully some of this is helpful or interesting and helps motivate adding conda support.

@mrietberg
Copy link

@rebelagentm Any update on the support for Conda?

@rebelagentm
Copy link
Contributor

@mrietberg 👋🏻 I'm no longer working on Dependabot. Tagging @asciimike here, as he may be able to provide an update.

@asciimike
Copy link
Contributor

We don't currently have plans to add support for Conda (tracking 👍🏻 on new-ecosystem requests is a rough way to gauge our prioritization), though we'd be happy to review PRs that the community creates.

@1kastner
Copy link

Some notes:

  • conda is a general purpose package manager not limited to Python

  • [...]

  • There is (currently) no standard for placing the environment.yaml. The file format really just describes an arbitrary selection of packages one might have in a conda "virtual environment".

  • A more specific file would be recipe/meta.yaml, which contains be the package build instructions (different format), including dependencies split into build, host and run (where build vs host separates the build system from development libraries in the case of cross compilation).

Here we might need to differentiate between conda packages that are meant to be installed and work material that is meant to be handed out for some event or as supplementary material for a publication. When I prepare seminar notebooks, I do hand out an environment.yml for a quick setup but I would never craft a recipe/meta.yaml. When we look at the repositories that are part of literature (such as introduction to ML etc.), we also rarely see this. In my opinion, these recipes are reserved for standalone packages that other people might import (such as a Python module) or use from the CLI (actually any kind of software). Now, should these people be forced to use recipes anyway?

@thomas-bc
Copy link

Any progress on this? 🙏

@pavelzw
Copy link

pavelzw commented Sep 6, 2023

Maybe it could make sense looking into pixi. They use the conda universe but in a somewhat structured way. Pixi projects always use a pixi.toml or pyproject.toml and have a pixi.lock which contain the lockfile. Somewhat similar to poetry but for conda.
Probably it's easier to implement this into dependabot since this has a clearly defined project structure.

@jtroe
Copy link

jtroe commented Jan 10, 2024

Is this on the roadmap at all? This would be a welcome enhancement for my work at @Esri.

@Zeitsperre
Copy link

Another comment here that this would be immensely helpful to have. Thanks!

popenc added a commit to quanted/cts_celery that referenced this issue Apr 9, 2024
Dependabot still doesn't recognize conda environment. Using requirements.txt to keep track of version updates (dependabot/dependabot-core#2227)
@jonjanego jonjanego added the Keep Exempt this from being marked by stalebot label May 2, 2024
bblakely-anl added a commit to aica-iwg/aica-agent that referenced this issue May 20, 2024
Remove pip from dependabot (no conda support yet - boo! dependabot/dependabot-core#2227)
bblakely-anl added a commit to aica-iwg/aica-agent that referenced this issue May 20, 2024
Remove pip from dependabot (no conda support yet - boo! dependabot/dependabot-core#2227)
@isaacsanders
Copy link
Contributor

https://github.com/conda/conda-lock is the main locking mechanism for people who use lockfiles in conda, I think.

@pavelzw
Copy link

pavelzw commented Aug 9, 2024

nowadays pixi is: conda/conda-lock#615

@jezdez
Copy link

jezdez commented Oct 8, 2024

nowadays pixi is: conda/conda-lock#615

Hi all, conda steering council member here.

I think that's a bit of a misrepresentation of the facts. The linked issue discusses how conda-lock could position itself to pixi, which implements an own locking mechanism incompatible to conda-lock.

I believe the conda-lock maintainer has decided that it would become a kind of backend for conda-lock, in addition to the existing formats. Pixi has not supplanted conda-lock or become the default. See the upcoming conda-lock 3.0 release.

As context, conda-lock is, for better or worse, a conda community project with approval from the conda steering council, while pixi is a commercial project by Prefix.dev. I think that it matters to state this in a consequential feature request like this here.

@beckermr
Copy link

Thanks for the comment @jezdez. I am another conda steering council member and 100% agree.

@ytausch
Copy link

ytausch commented Oct 12, 2024

pixi is a commercial project by Prefix.dev.

Since pixi is licensed under BSD-3-Clause, I think "commercial" is a bit of an overstatement here.

@xhochy
Copy link

xhochy commented Oct 14, 2024

pixi is a commercial project by Prefix.dev.

Since pixi is licensed under BSD-3-Clause, I think "commercial" is a bit of an overstatement here.

The difference here is the controlling entity. conda-lock is part of the conda organisation and pixi is controlled by a single entity. The two tools also cater to a different workflow/developer UX. Thus it would make sense to support both of them.

@maxkapur
Copy link

Would it be possible to change the title of this issue to something like Ecosystem request: conda?

I'm worried the current title makes it sound like conda is Python-specific, which it isn't (I use it to install the Ruby/Jekyll stack for my static site, for example).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Keep Exempt this from being marked by stalebot T: new-ecosystem Requests for new ecosystems/languages
Projects
Status: Planned
Development

No branches or pull requests