Add conda dev environment

[tobiasdiez]

We add a conda enviroment that is meant for development, i.e. includes tools that one probably needs while working on sage development such as linters, ssh (for contributing to trac) etc.

Moreover, we extract the conda-related things from docs/bootstrap to a new bootstrap-conda script in order to simplify the conda workflow (see updated documentation).

Github workflow run: https://github.com/sagemath/sagetrac-mirror/actions/runs/2268985510

CC: @isuruf @dimpase @saraedum @mkoeppe @slel

Component: build

Author: Tobias Diez

Branch: 87f360c

Reviewer: Matthias Koeppe

Issue created by migration from https://trac.sagemath.org/ticket/33740

[dimpase]

comment:2

This looks quite odd to me.
Why can't the yml file in the branch be generated from what's already known about Conda packages?

[mkoeppe]

comment:3

I agree, not a good idea to treat this file different from all other of our files that are generated at bootstrap time.

Also jupyter_sphinx should really be added by adding a distros/ file, not ad hoc like this.

[tobiasdiez]

comment:4

Replying to @dimpase:

This looks quite odd to me.
Why can't the yml file in the branch be generated from what's already known about Conda packages?

The issue is not so much to generate the file but how it is used. Running bootstrap requires a few dependencies which however are only specified in the environment file. So its a classical chicken-egg problem.

Currently this is resolved by requiring the developers to first create a conda env with the necessary tools for bootstrap, and then later update the env with all dependencies from the env file. With my proposal the overhead occurs only when dependencies are changed (which then requires to commit the changed env file). Overall this should be less work and more convenient.

I agree, not a good idea to treat this file different from all other of our files that are generated at bootstrap time.

The dev environment is special in this respect, since its the only file that you would possibly like to use before the bootstrap.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

aa144e6

Add distros/conda for jupyter_sphinx

[sagetrac-git]

Changed commit from 413b081 to aa144e6

[tobiasdiez]

comment:6

Replying to @mkoeppe:

Also jupyter_sphinx should really be added by adding a distros/ file, not ad hoc like this.

Done.

[dimpase]

comment:7

ok, but please make the initial egg (say, ./bootstrap-conda-dev) minimal - apart from m4 and other autotools, not much else is needed for ./bootstrap.

[tobiasdiez]

comment:8

Mhh, having a conda environment just for the bootstrap would also work. But then one still has to run conda create sage-dev -f bootstrap-conda-dev.yml, bootstrap and conda sage-dev update -f environment.yml.

Having one conda environment config containing all necessary tools for bootstrap, compiling and running sage as well as a few additional dev tools is the most convenient, don't you think so?

[mkoeppe]

comment:9

Replying to @dimpase:

ok, but please make the initial egg (say, ./bootstrap-conda-dev) minimal - apart from m4 and other autotools, not much else is needed for ./bootstrap.

+1 on just splitting the generation of the yml files from src/doc/bootstrap (it doesn't really belong there anyway) into a new script. It won't need any of the dependencies that the full bootstrap has.

[dimpase]

comment:10

perhaps this should rather become a conda-forge package?

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

2811559

Introduce new bootstrap-conda command

[sagetrac-git]

Changed commit from aa144e6 to 2811559

[tobiasdiez]

comment:12

Replying to @mkoeppe:

Replying to @dimpase:

ok, but please make the initial egg (say, ./bootstrap-conda-dev) minimal - apart from m4 and other autotools, not much else is needed for ./bootstrap.

+1 on just splitting the generation of the yml files from src/doc/bootstrap (it doesn't really belong there anyway) into a new script. It won't need any of the dependencies that the full bootstrap has.

Like this?

[dimpase]

comment:13

that's much nicer, but where is the top-level invocation of bootstrap-conda?
Do you mean it's an alternative to ./bootstrap ?

[tobiasdiez]

comment:14

I wasn't sure about this one. The procedure now looks as follows:

• bootstrap-conda
• conda create
• bootstrap
• configure
• ...

So there is no point in invoking bootstrap-conda during bootstrap.

[mkoeppe]

comment:15

Still need to invoke this so that the generated files end up in the sdist.

[mkoeppe]

comment:16

(See the shell function save in bootstrap)

[dimpase]

comment:17

Replying to @tobiasdiez:

I wasn't sure about this one. The procedure now looks as follows:

• bootstrap-conda
• conda create
• bootstrap
• configure
• ...

So there is no point in invoking bootstrap-conda during bootstrap.

I thought that ./bootstrap recognises whether we're on conda, or not.
If so, it can invoke bootstrap-conda+conda create.

Otherwise, I'd like to see ./bootstrap getting an optional argument, conda,
to do the calling of bootstrap-conda+conda create and then proceeding.

[mkoeppe]

comment:18

Replying to @dimpase:

I thought that ./bootstrap recognises whether we're on conda, or not.

Not a good idea. The output of bootstrap must not depend on the system configuration. It is packed into the sdist.

[dimpase]

comment:19

shouldn't ./bootstrap-conda run on #!/bin/sh, just like ./bootstrap ?

[tobiasdiez]

comment:21

Replying to @dimpase:

shouldn't ./bootstrap-conda run on #!/bin/sh, just like ./bootstrap ?

With sh the script is not working, so I took bash as in doc/bootstrap. If that needs to be changed, I need help as its beyond my basic shell script knowledge.

[tobiasdiez]

comment:22

Replying to @dimpase:

Otherwise, I'd like to see ./bootstrap getting an optional argument, conda,
to do the calling of bootstrap-conda+conda create and then proceeding.

I like the idea to further simplify the workflow, but lets keep this ticket small and do further streamlining efforts in #33765.

[tobiasdiez]

comment:23

Replying to @mkoeppe:

Still need to invoke this so that the generated files end up in the sdist.

Okay, added.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

b42f5f5

Readd how to use specific python version

[tobiasdiez]

comment:56

Replying to @mkoeppe:

Also, in the edits of the documentation, the information on how to select a Python version have been lost

Added again.

[mkoeppe]

comment:57

Replying to @tobiasdiez:

Replying to @mkoeppe:

In this change:

-      $ conda create -n sage-build
+      $ conda env create --file environment.yml
       $ conda activate sage-build

I would suggest to keep the -n so that it's easy to see what needs to be changed if the user wants to use a different name

Normally the user doesn't need to change the name

It's as normal as having a second installation in a separate directory

[tobiasdiez]

comment:58

Replying to @mkoeppe:

Replying to @tobiasdiez:

Replying to @mkoeppe:

In this change:

-      $ conda create -n sage-build
+      $ conda env create --file environment.yml
       $ conda activate sage-build

I would suggest to keep the -n so that it's easy to see what needs to be changed if the user wants to use a different name

Normally the user doesn't need to change the name

It's as normal as having a second installation in a separate directory

If you want to install two sages in parallel, you should also be able to find the --name arg of conda. But if you insist I can readd the name.

[mkoeppe]

comment:59

Yes please. I added it on purpose in the original version.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

87f360c

Readd name arg

[sagetrac-git]

Changed commit from b42f5f5 to 87f360c

[tobiasdiez]

comment:61

Replying to @mkoeppe:

Yes please. I added it on purpose in the original version.

okay done

[mkoeppe]

comment:62

Broken: https://github.com/sagemath/sagetrac-mirror/runs/6356314150

[mkoeppe]

comment:63

conda's pari is not being accepted.

[tobiasdiez]

comment:64

This is not specific to this branch and also occurs for develop https://github.com/sagemath/sage/runs/6236167749?check_suite_focus=true
I've opened #33828 for this.

[mkoeppe]

Reviewer: Matthias Koeppe

[mkoeppe]

comment:65

Thanks for checking, I agree. Then it's good to go

[tobiasdiez]

comment:66

Thanks!

[vbraun]

Changed branch from public/build/conda-dev-env to 87f360c

[slel]

comment:69

The changes in this ticket made ./bootstrap -q less quiet.

Before:

$ ./bootstrap -q

After:

$ ./bootstrap -q
./bootstrap-conda:46: generate conda enviroment files

Follow-up at #34097.

[slel]

Changed commit from 87f360c to none