Enable gitpod integration

[tobiasdiez]

Gitpod allows one to setup a complete dev environment in the cloud. It is free to use for up to 50 hours per month. In this ticket the config necessary for making this work with sagemath is added.

You can try this out by going to https://gitpod.io/#https://github.com/sagemath/sagetrac-mirror/tree/public/build%2Fgitpod.

It currently takes a bit more than 1 hour until everything is setup. After this ticket is merged, we can enable the automatic prebuild using a github app (https://www.gitpod.io/docs/prebuilds#on-github). With this enabled, every push to the develop branch would trigger a prebuild of the complete environment (including the build of all dependencies and cythonizion) so that one has a up-to-date code env in a matter of a few seconds.

---

While setting things up, I noticed a few issues. Not sure if they are known problems or even by design. Please let me know if I should open a ticket for them to improve things.

• gitpod has brew installed on linux by default. Sagemath thinks this is the primary package manager and, e.g., configure suggests to run brew xyz to install new packages. Is brew indeed preferred over apt-get? This is now Prefer Linux package manager over linuxbrew; add nix detection #32753.
• gitpod comes also with pyenv preinstalled. This breaks the build unless one sets pyenv global system. Otherwise all python packages will be installed in the python env set by pyenv. This is now Unbreak the build when pyenv is present #29285.
• Currently gitpod prebuild has a time limit of 1 hour (with the plan to increase it in the future). To improve our build time I propose to add a new make option that uses prebuilt wheels from pypi instead of compiling everything locally. This is now Add make option to use prebuilt wheels from PyPI #32754.
• Maxima doesn't work with the homebrew version of ecl? From Support system installations of ECL and fix the kenzo SPKG build #29617 I got the impression it should work, but it fails with

;;; Loading "/workspace/sagetrac-mirror/local/var/tmp/sage/build/maxima-5.45.0.p0/src/src/../lisp-utils/defsystem.lisp"
;;; Loading #P"/home/linuxbrew/.linuxbrew/Cellar/ecl/21.2.1_1/lib/ecl-21.2.1/cmp.fas"
#P"/workspace/sagetrac-mirror/local/var/tmp/sage/build/maxima-5.45.0.p0/src/lisp-utils/defsystem.lisp"
> 
("../src/" "./" (MAKE::HOME-SUBDIRECTORY "lisp/systems/")
 "/usr/local/lisp/Registry/")
> 
;;; Loading "/workspace/sagetrac-mirror/local/var/tmp/sage/build/maxima-5.45.0.p0/src/src/maxima.system"

;  - Compiling defsystem "maxima" 
;    - Compiling module "package" 
;      - Compiling source file
;        "/workspace/sagetrac-mirror/local/var/tmp/sage/build/maxima-5.45.0.p0/src/src/maxima-package.lisp"
;;;
;;; Compiling /workspace/sagetrac-mirror/local/var/tmp/sage/build/maxima-5.45.0.p0/src/src/maxima-package.lisp.
;;; OPTIMIZE levels: Safety=2, Space=0, Speed=3, Debug=2
;;;
;;; End of Pass 1.
;;; Internal error:
;;;   ** Error code 1 when executing
;;; (EXT:RUN-PROGRAM "gcc-5" ("-I." "-I/home/linuxbrew/.linuxbrew/Cellar/ecl/21.2.1_1/include/" "-I/home/linuxbrew/.linuxbrew/opt/gmp/include" "-I/home/linuxbrew/.linuxbrew/opt/libffi/include" "-I/home/linuxbrew/.linuxbrew/opt/bdw-gc/include" "-D_GNU_SOURCE" "-D_FILE_OFFSET_BITS=64" "-g" "-O2" "-fPIC" "-D_THREAD_SAFE" "-Dlinux" "-O2" "-c" "binary-ecl/maxima-package.c" "-o" "binary-ecl/maxima-package.o")):
;;; exec: No such file or directory
;      - Binary file binary-ecl/maxima-package.fas is old or does not exist. 
;        Compile (and load) source file /workspace/sagetrac-mirror/local/var/tmp/sage/build/maxima-5.45.0.p0/src/src/maxima-package.lisp instead? y
;      - Should I bother you if this happens again? y
;      - Compiling source file
;        "/workspace/sagetrac-mirror/local/var/tmp/sage/build/maxima-5.45.0.p0/src/src/maxima-package.lisp"
;;;
;;; Compiling /workspace/sagetrac-mirror/local/var/tmp/sage/build/maxima-5.45.0.p0/src/src/maxima-package.lisp.
;;; OPTIMIZE levels: Safety=2, Space=0, Speed=3, Debug=2
;;;
;;; End of Pass 1.
;;; Internal error:
;;;   ** Error code 1 when executing
;;; (EXT:RUN-PROGRAM "gcc-5" ("-I." "-I/home/linuxbrew/.linuxbrew/Cellar/ecl/21.2.1_1/include/" "-I/home/linuxbrew/.linuxbrew/opt/gmp/include" "-I/home/linuxbrew/.linuxbrew/opt/libffi/include" "-I/home/linuxbrew/.linuxbrew/opt/bdw-gc/include" "-D_GNU_SOURCE" "-D_FILE_OFFSET_BITS=64" "-g" "-O2" "-fPIC" "-D_THREAD_SAFE" "-Dlinux" "-O2" "-c" "binary-ecl/maxima-package.c" "-o" "binary-ecl/maxima-package.o")):
;;; exec: No such file or directory
;      - Loading binary file "binary-ecl/maxima-package.fas" 
Condition of type: FILE-ERROR
Filesystem error with pathname #P"/workspace/sagetrac-mirror/local/var/tmp/sage/build/maxima-5.45.0.p0/src/src/binary-ecl/maxima-package.fas".

This should be investigated as part of #29159.

Depends on #33068
Depends on #30933

CC: @mkoeppe @saraedum @koffie

Component: build

Branch/Commit: public/build/gitpod @ 974eb4c

Reviewer: Tobias Diez

Issue created by migration from https://trac.sagemath.org/ticket/32749

[mkoeppe]

comment:3

Use of linuxbrew is entirely untested; see #29159

[mkoeppe]

comment:4

The problem with ecl is clearly due to a broken ecl package in linuxbrew - it tries to use a nonexisting compiler.

[mkoeppe]

comment:5

Probably build/bin/sage-guess-package-system should be changed so that brew is only tested after the Linux package managers.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

6bb485f	Do not install ecl from brew
1ea950d	Cleanup gitpod config
d792c6b	Add docs

[sagetrac-git]

Changed commit from bd1fe07 to d792c6b

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

1ae507d

Remove uunnecessary changes to sage-venv

[sagetrac-git]

Changed commit from d792c6b to 1ae507d

[tobiasdiez]

Changed work issues from Cleanup config, add vs extensions and add docs to none

[tobiasdiez]

comment:8

Thanks for the feedback. I've now removed the linuxbrew install of ecl and opened tickets for the other things. So this is now ready for review.

[mkoeppe]

comment:9

Does it work?

[mkoeppe]

comment:10

Consider opening a bug report against the ecl packaging on linuxbrew

[mkoeppe]

comment:11

Let's resolve the pyenv issues in #29285

[tobiasdiez]

comment:12

Replying to @mkoeppe:

Does it work?

Yes! You can also test this out using the above link. Currently one has to wait for the build to finish. This waiting time can be removed by activating the github integration after this ticket is merged.

[tobiasdiez]

comment:13

Replying to @mkoeppe:

Consider opening a bug report against the ecl packaging on linuxbrew

Will do after this ticket is merged (since I can then reference the gitpod workspace as a reproducible example).

[mkoeppe]

comment:15

It is asking: "We noticed a new virtual environment has been created. Do you want to select it for the workspace folder?"

Add the correct answer to the documentation?

[mkoeppe]

comment:16

Maybe you want to use make V=0 so that there's less output during build?

[mkoeppe]

comment:17

Does this need Sage's Jupyter notebook or does it bring its own?

configure --disable-notebook can be used to avoid building notebook and its many Python deps

[mkoeppe]

comment:18

Given that the preinstalled linux is old and the linuxbrew has some broken packages, one could wonder whether it would be better to use conda to set up a richer environment with much shorter time for building.
The scipy people seem to be doing that - https://scipy.github.io/devdocs/dev/contributor/quickstart_gitpod.html - but I haven't looked at any details.

[mkoeppe]

comment:19

The build on Gitpod finished for me, but then the workspace timed out; on opening it again, starting ./sage fails with
ImportError: libzmq.so.5: cannot open shared object file: No such file or directory.
Am I doing something wrong?

[tobiasdiez]

comment:20

Replying to @mkoeppe:

It is asking: "We noticed a new virtual environment has been created. Do you want to select it for the workspace folder?"

Add the correct answer to the documentation?

This will be no longer necessary once we have added the VS config.

[tobiasdiez]

comment:21

Replying to @mkoeppe:

Maybe you want to use make V=0 so that there's less output during build?

Normally the user wouldn't see this, and everything that is printed during the init stage would become the prebuilt log that can be viewed in the admin interface. So I think its fine to have it more detailed, in case something goes wrong.

[sagetrac-git]

Changed commit from 4a9edfc to bde9f74

[tobiasdiez]

comment:124

Thanks for thinking along. The problem is more this strange 502 error that I now get during the prebuild. I've now opened gitpod-io/gitpod#7413 to investigate where the error comes from. We also might merging this ticket and activate the github app, because then you should also have access to the prebuild logs (which is not possible for the manual prebuild).

However, the docker build already works and has not much room for improvement (the only non-Python packages that are currently not finished is singular and maxima). For this I would just wait until gitpod increases its time limit. Using github actions to build docker images, that are then reused by gitpod sounds pretty complicated to me. Gitpod also handles caching and invalidating of the images for you (one needs to rebuild the base docker images for branches that change non-Python packages or if the docker file changes).

What still has room for improvement (but also almost fits into the 1h limit) is the prebuild of the Python packages. For this it would be good to reuse wheels to decrease the build time, or maybe use conda. But maybe we should first try it withput this optimization and also wait for gitpod to increase the time limit.

May I ask you to push changes to the gitpod docker and yml files to a different branch. Everytime one changes the docker file, one has to sit through 1h. Thanks.

[mkoeppe]

Changed dependencies from #33068 to #33068, #30933

[mkoeppe]

comment:125

Sure, I'll work in #33103.

[mkoeppe]

comment:126

Feel free to revert the branch on the ticket here to what works for you.

[mkoeppe]

comment:127

Replying to @tobiasdiez:

We also might merging this ticket and activate the github app, because then you should also have access to the prebuild logs (which is not possible for the manual prebuild).

I think you may want to try this first in your personal GitHub account. Getting this to work shouldn't require to make org-level changes to the sagemath organization on GitHub.

[mkoeppe]

comment:128

Replying to @tobiasdiez:

or maybe use conda

Looking at https://github.com/gitpod-io/workspace-images/blob/master/full/Dockerfile, nix seems to be available, so this may be another option

[tobiasdiez]

comment:129

Replying to @mkoeppe:

Feel free to revert the branch on the ticket here to what works for you.

Could you maybe revert to 3adf51 for me pls? I don't have a dev pc available and used gitpod to make the necessary changes...
I was wondering if one should keep your changes to the symlink (commit ​c7cc5cd). What was the reason for this change?

[tobiasdiez]

comment:130

Replying to @mkoeppe:

Replying to @tobiasdiez:

or maybe use conda

Looking at https://github.com/gitpod-io/workspace-images/blob/master/full/Dockerfile, nix seems to be available, so this may be another option

Is this usually quicker or are the more system packages available?

[sagetrac-git]

Changed commit from bde9f74 to 974eb4c

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

974eb4c

.gitpod.yml, docker/.gitpod.Dockerfile: Reverse SAGE_LOCAL symlink

[mkoeppe]

comment:132

Replying to @tobiasdiez:

Could you maybe revert to 3adf51 for me pls? I don't have a dev pc available and used gitpod to make the necessary changes...
I was wondering if one should keep your changes to the symlink (commit ​c7cc5cd).

Done, with that change cherry-picked.

What was the reason for this change?

Just an easier solution that works without using sudo, and integrates better with #33103

[mkoeppe]

comment:133

Replying to @tobiasdiez:

Replying to @mkoeppe:

nix seems to be available, so this may be another option

Is this usually quicker or are the more system packages available?

More packages, because someone made an effort to package all of sage not so long ago. Not sure in what shape it is though

[slel]

comment:134

The nix package for Sage seems maintained up to date.

• https://github.com/NixOS/nixpkgs/tree/master/pkgs/applications/science/math/sage
• https://github.com/NixOS/nixpkgs/commits/master/pkgs/applications/science/math/sage

[tobiasdiez]

comment:135

Thanks for the pointer. Nix seems promising indeed, but I think it would make sense to first create a workable nix environment (e.g. shell.nix file) and test it for 'normal' local building before using for gitpod.

The approach with prefix + symlink was not working for me. Every prebuild that I triggered timed-out without any error message; also when running it from my own fork using the gitpod app integration. I guess something from the gitpod build context slips into the image/prebuild setup. Thus, I've reverted to use of the prebuild of system packages in home/gitpod/sage-prebuild and setting env variables so that the prebuild packages are found. That works. Interestingly, this is somewhat similar to how nix works. Current state can be found at https://github.com/tobiasdiez/sagetrac-mirror.

During the docker build it suggests to install the following packages

sudo apt-get install  libflint-arb-dev libcdd-dev libcdd-tools ecl libec-dev eclib-tools libflint-dev flintqs libgiac-dev xcas lcalc liblfunction-dev pari-gp2c libpari-dev pari-doc pari-elldata pari-galdata pari-galpol pari-seadata libprimesieve-dev singular libsingular4-dev tox

Some are excluded (like pari and flint) but other would make sense to install I guess. Should I try manually adding them, or should the sage-get-system-packages command perhaps be changed?

We now have three possible options for the gitpod integration and before we invest more time working on them separately it is probably worthwhile to choose on one of them to proceed further:

• 1. Only use the gitpod infrastructure to build docker images (non-python packages) and prebuilds (python packages). This is the current version of this ticket.
• 2. Use github action to build a docker image containing all sage packages. This is gitpod integration using Docker images from portability testing workflow #33103.
• 3. Use version 1, but install non-python packages using nix.

[mkoeppe]

comment:136

I would suggest that we go with 2) (#33103) because its maintenance basically comes for free from the portability test infrastructure.

[tobiasdiez]

comment:137

Is fine with me. However, that approach has a few disadvantages as far as I can see:

• The relevant github action might only be triggered after quite some time upon a new (beta) relase as all other github workflows are then also triggered. During this time one would still use the old docker image.
• The docker image is quite big and thus it takes longer to open a new gitpod workspace.
• It is harder to add gitpod-specific customization to the docker image (e.g. installing tools that are only needed in this context).

These are certainly not deal breakers but should be kept in mind.

The first approach has also a few disadvantages:

• Currently less packages are built due to the gitpod timeout of 1h
• Not automatically tested (although this would probably be easy to do)

[mkoeppe]

comment:138

Great points. I think we can address the current disadvantages of approach 2 in a follow-up ticket (#33113). In particular, as discussed in #33103 comment:19, could use docker/.gitpod.Dockerfile again with COPY --from to make the final image smaller (e.g., currently /home/gitpod/sage has the old source tree)

[tobiasdiez]

comment:139

Replying to @mkoeppe:

Great points. I think we can address the current disadvantages of approach 2 in a follow-up ticket (#33113). In particular, as discussed in #33103 comment:19, could use docker/.gitpod.Dockerfile again with COPY --from to make the final image smaller (e.g., currently /home/gitpod/sage has the old source tree)

But you also need to transfer the installation of the system packages to the final image, so this may be quite complicated. Probably you end up with something quite close to the dockerfile of this ticket. Maybe that's actually the best of the two worlds: essentially use the docker file from this ticket, but instead of the prebuild layer just copy the sage-local directory from the github docker image?

[mkoeppe]

comment:140

No, it could just use FROM with the build stage https://github.com/mkoeppe/sage/pkgs/container/sage%2Fsage-docker-gitpod-standard-configured (or ...-with-system-packages - which is currently not pushed)
and then COPY --from the build stage https://github.com/mkoeppe/sage/pkgs/container/sage%2Fsage-docker-gitpod-standard-with-targets.

[mkoeppe]

comment:141

Replying to @mkoeppe:

No, it could just use FROM with the build stage https://github.com/mkoeppe/sage/pkgs/container/sage%2Fsage-docker-gitpod-standard-configured (or ...-with-system-packages - which is currently not pushed)
and then COPY --from the build stage https://github.com/mkoeppe/sage/pkgs/container/sage%2Fsage-docker-gitpod-standard-with-targets.

Now implemented in #33103 comment:95

[tobiasdiez]

comment:142

Let's close this in favor of #33103.

[mkoeppe]

Changed author from Tobias Diez to none

[mkoeppe]

Reviewer: Tobias Diez