Migrate gitpod to conda

[tobiasdiez]

The current solution to prebuild the gitpod docker image using github actions does not work nicely, since prebuilds triggered shortly after a new release still use the old gitpod image (and it may take up to a few days until the new image is available). This is especially notable for the develop branch, which is the default entry point if you want to start a new branch on gitpod.

In this ticket, we migrate the gitpod config to use the conda environment, which is now generally stable enough for serious development (e.g. almost all tests pass). As a positive side effect, the gitpod config simplifies and startup times are considerably reduced.

Depends on #33740
Depends on #33855

CC: @mkoeppe @dimpase @isuruf

Component: build

Author: Tobias Diez

Branch/Commit: 5748c21

Reviewer: Matthias Koeppe

Issue created by migration from https://trac.sagemath.org/ticket/33739

[sagetrac-git]

Changed commit from a1602c9 to 9361a33

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

9361a33

Cleanup

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

bd2f3d3

Remove gitpod docker image build

[sagetrac-git]

Changed commit from 9361a33 to bd2f3d3

[tobiasdiez]

Dependencies: #33740

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

10338a7	Change ci as well
0cf7334	Add sage-setup as install-requires dep
6d2b149	Use relative paths to not install src package
1dba3ba	Maybe thats the correct shebang?
c1b2fbf	Revert install_requires change
ad56ba3	Go back to two pip installs for now
b42f5f5	Readd how to use specific python version
87f360c	Readd name arg
82fe6a6	Merge remote-tracking branch 'origin/public/build/conda-dev-env' into public/build/conda_gitpod
67c74f2	Update based on recent changes in deps

[sagetrac-git]

Changed commit from bd2f3d3 to 67c74f2

[tobiasdiez]

comment:7

This is now using the latest version of #33740, so ready for review again.

[mkoeppe]

comment:8

I don't know if it's a good idea to make this change because it means that gitpod can no longer be used for trying out package updates in the Sage distribution.

Too bad that Gitpod has decided that a project can only have one Gitpod configuration - an obvious design limitation

[tobiasdiez]

comment:9

Can you not install the updated package manually using make xyz? But yes, there are a few things that might be harder to test in a conda environment. On the other hand, you also got a few new tools at your hand. For example, you can easily install specific versions of dependencies using conda.

I think its also not a good idea to optimize gitpod for package update tickets, they should only be a small subset of all tickets. And for those the conda-based gitpod works more reliable and has shorter startup times.

[mkoeppe]

comment:10

Replying to @tobiasdiez:

Can you not install the updated package manually using make xyz?

As the documentation says -
"Note that make is not used at all. All dependencies
(including all Python packages) are provided by conda. [...] this will invalidate the
use of any Sage-the-distribution commands such as sage -i. Do not use them."

[mkoeppe]

comment:11

I think the way forward could be to work on the devcontainer ticket (#33671). There can be multiple devcontainers (we could have them for all the platforms that we generate Docker images for). This would be a good replacement for what gitpod right now offers in terms of testing package upgrades.

When done, I agree that switching gitpod to conda is a good idea.

Also consider that both the conda stuff and the gitpod stuff are new -- for onboarding current developers, right now I think it's better if gitpod shows them something that is more familiar.

[tobiasdiez]

comment:12

Replying to @mkoeppe:

I think the way forward could be to work on the devcontainer ticket (#33671). There can be multiple devcontainers (we could have them for all the platforms that we generate Docker images for). This would be a good replacement for what gitpod right now offers in terms of testing package upgrades.

How would devcontainer help here? Gitpod doesn't support them yet and its not clear yet when they will and in which form; the linked github issue reads more like that the are considering this as a alternative syntax for their gitpod.yml file and not that they would offer prebuilds for all defined devcontainers.

[tobiasdiez]

comment:13

Replying to @mkoeppe:

Replying to @tobiasdiez:

Can you not install the updated package manually using make xyz?

As the documentation says -
"Note that make is not used at all. All dependencies
(including all Python packages) are provided by conda. [...] this will invalidate the
use of any Sage-the-distribution commands such as sage -i. Do not use them."

I just run make pari in the conda-based gitpod and it worked just fine.

[mkoeppe]

comment:14

Replying to @tobiasdiez:

Replying to @mkoeppe:

I think the way forward could be to work on the devcontainer ticket (#33671). There can be multiple devcontainers (we could have them for all the platforms that we generate Docker images for). This would be a good replacement for what gitpod right now offers in terms of testing package upgrades.

How would devcontainer help here?

If done right, it would make working on package upgrades almost as convenient as using gitpod. Sure, using local compilation (unless we orchestrate devcontainer prebuild), but convenient

[tobiasdiez]

comment:15

Replying to @mkoeppe:

Replying to @tobiasdiez:

Replying to @mkoeppe:

I think the way forward could be to work on the devcontainer ticket (#33671). There can be multiple devcontainers (we could have them for all the platforms that we generate Docker images for). This would be a good replacement for what gitpod right now offers in terms of testing package upgrades.

How would devcontainer help here?

If done right, it would make working on package upgrades almost as convenient as using gitpod. Sure, using local compilation (unless we orchestrate devcontainer prebuild), but convenient

Even with a conda-based gitpod, you can simply deactivate the conda environment and run configure && make. This is pretty much a disposable devcontainer without prebuild.

[mkoeppe]

comment:16

Replying to @tobiasdiez:

Even with a conda-based gitpod, you can simply deactivate the conda environment and run configure && make.

That's true, of course

This is pretty much a disposable devcontainer without prebuild.

Well no, because it wouldn't even have a SAGE_LOCAL. So it offers nothing over just building the whole thing locally

[tobiasdiez]

comment:17

The advantage would be that you can run whatever commands you want without worrying how it affects the rest of your system / the main installation of sage.

Anyway, I still think that package updates are only a small subset of all tickets and making them temporarily slightly less convenient to test with gitpod is a price worth paying if gitpod gets more useful for the rest of the tickets. Moreover, as you mention, gitpod support is new so developers can easily go back to whatever they were using before to test package updates until further improvements like #33671 are implemented.

It would be nice if this ticket could be merged before the sagedays so that I don't have to wait 5mins during my talk until the workspace started.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

f56a535

Install esbonio using conda

[sagetrac-git]

Changed commit from 67c74f2 to f56a535

[mkoeppe]

comment:19

Is it really starting up faster?

[mkoeppe]

comment:20

Regression: The SageMath jupyter kernel cannot be selected (Command Palette - Create: New Jupyter Notebook, then "Select Kernel" in the upper right corner)

[mkoeppe]

Changed dependencies from #33740 to #33740, #33855

[mkoeppe]

comment:29

Let's add it in #33855.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

6c5903d	src/sage_setup/command/sage_install.py: Split sage_install_and_clean into several classes
40b44a1	build/pkgs/sagelib/spkg-install: Do installation of sagemath kernelspec here, not in sage_setup.command.sage_install
1c1dbc5	src/setup.py: No need to use custom sage_install
6862287	build/pkgs/sagelib/spkg-install: Un-poison SAGE_DOC for installing the kernelspec
88d769b	Merge #33855

[sagetrac-git]

Changed commit from f56a535 to 88d769b

[mkoeppe]

comment:32

Of course by itself #33855 doesn't fix it -- the same line in spkg-install needs to be run manually...

[sagetrac-git]

Changed commit from 88d769b to 2b9ef5a

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

6c5903d	src/sage_setup/command/sage_install.py: Split sage_install_and_clean into several classes
4473375	src/sage_setup/command/sage_install.py: Add sage_develop, use it in src/setup.py, pkgs/sagemath-standard/setup.py
2b9ef5a	Merge #33855

[mkoeppe]

comment:34

Rolled back the merge and merged the new version of #33855.

[mkoeppe]

comment:35

Now the kernel is available

[mkoeppe]

Reviewer: Matthias Koeppe

[mkoeppe]

comment:37

I haven't tested it thoroughly, but it looks like a nice improvement. It does start faster than the previous solution using our huge prebuilt Docker images.

[tobiasdiez]

comment:38

Thanks!

[vbraun]

comment:39

Merge failure on top of:

054e8ed8d9 Trac #33316: Drop support for GCC < 6.3 in Sage 9.7

fa3b529 Trac #25833: Upgrade to MathJax 3 and configure Sage

e2ebf44 Trac #33825: Use pytest-xdist to run pytest in parallel

b400962 Trac #33824: make gens and orbits of PermutationGroup immutable

af23627 Trac #33809: some pathlib in combinat and groups

955b5d9 Trac #33803: Fixes for the distributions sagemath-objects, sagemath-categories

3e6b41f Trac #33799: Replace SAGE_TMP in doctests of sage.misc.{persist,ostools}, sage.doctest, sage.repl

a3fd718 Trac #33797: sage.misc.temporary_file: Remove use of SAGE_TMP

2ca0530 Trac #33794: modules/fp_graded/morphism.py test failure

7037fba Trac #33793: sage.misc.cython: Replace use of SPYX_TMP by a new cached function in sage.misc.temporary_file

d115270 Trac #33787: Installation manual: Update section "system-wide install"

0ae5565 Trac #33782: ci-cygwin: Update python version

833f53d Trac #33779: Remove use of SAGE_TMP in sage.interfaces.latte

b376a8d Trac #33771: SSLContext needs an argument

df168c8 Trac #33763: Refactor src/sage/docs

9597eaf Trac #33748: make accessing coefficients of finite-field elements easier

f02236f Trac #33744: Compute bases/circuits in MatroidUnion

8943dc0 Trac #33743: Faster random tree generator

773ec37 Trac #33740: Add conda dev environment

5e65c16 Trac #33734: variety() for polynomial systems over ℚ using msolve

8e7dcca Trac #33733: allow to use flint for Stirling numbers

6f4efb0 Updated SageMath version to 9.7.beta0

merge was not clean: conflicts in .github/workflows/tox.yml

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

5748c21

Merge tag '9.7.beta1' into t/33739/public/build/conda_gitpod

[sagetrac-git]

Changed commit from 2b9ef5a to 5748c21

[vbraun]

Changed branch from public/build/conda_gitpod to 5748c21