GH Actions workflow that builds documentation on each push

[mkoeppe]

On each push to a branch in sagemath/sagetrac-mirror, it builds the HTML documentation and, if this succeeds, deploys it to netlify for easy inspection.
Example: https://62018055188ce050610e98a6--sagemath-tobias.netlify.app
Workflow runs: https://github.com/sagemath/sagetrac-mirror/actions/workflows/doc-build.yml

Once this ticket is merged, I will add a badge pointing to the github workflow (similar to the linting badge), where one then can get the url for this particular deployment. Due to various open issues at netlify, it is sadly not possible right now to set the deployment url to the branch name so we use the commit hash for this.

For pushes to the develop branch it deploys the docs to the 'production' website
https://sagemath-tobias.netlify.app/

I've chosen netlify (say over github pages) mainly since it easily allows to push multiple versions of the same page under different urls (and does an intelligent diff so that the upload is smaller).

Related: sagemath/documentation#24 #33263 #33294

Follow-up:

• Change the deployment url to something more official. For this reason, I've created the site sagemath-tobias.netlify.org, so if you want to create an official sage account, sagemath.netlify.org is still available. Then one just needs to change the secrets in the github repo.

• Incorporate the post-fixes of https://github.com/sagemath/documentation into the main repo here, so that the preview looks the same as the published page.

• For pushes with tags (or to master), publish it on docs.sagemath.org website (for example using https://github.com/marketplace/actions/deploy-to-github-pages or by completely migrating to netlify).

CC: @slel @jhpalmieri @haraldschilly @williamstein @tobiasdiez @kwankyu @vbraun

Component: website/wiki

Author: Tobias Diez

Branch: 31665de

Reviewer: Matthias Koeppe

Issue created by migration from https://trac.sagemath.org/ticket/31415

[haraldschilly]

comment:3

I'm the one creating the documentation pages in the last few years. It's more than just building the files, because the output requires post-processing to make the pages fit for the CDN and I also have a script to create index pages. I'm also maintaining patches on top of Sage. So, all I want to add is that this task can't be automated right now.

[mkoeppe]

comment:4

Are these patches available somewhere, and should we look into whether they should be integrated into Sage?

[slel]

comment:6

Related:

• Sage documentation search page broken #29576: Missing js objects on doc search page

[mkoeppe]

comment:7

Moving this ticket to 9.4, as it seems unlikely that it will be merged in 9.3, which is in the release candidate stage

[mkoeppe]

comment:11

Looks like you are working on this, Tobias?

[tobiasdiez]

Branch: public/build/doc_build_ci

[tobiasdiez]

Commit: a94d1e8

[tobiasdiez]

comment:12

Yes, here is what I have for now.

On each push, it builds the documentation and deploys it to netlify. Example: https://61ffda629c2d6f02653112ff--sagemath-tobias.netlify.app/

However, what Harald says above is still true. Currently, there are symlinks in the documentation tree (to import static files), which yields to display errors. I have the same display problems locally (because I'm using WSL and Windows is not resolving the Linux symlinks). So that's something that still needs fixing. Any ideas how to remove the symlinks?

---

New commits:

4123116	Add continuous deployment of docs
2b989ce	Why are you not showing up?
361c95b	Back to full workflow
0847118	Fix path to build docs
915da27	Trigger build again
f5c91d8	Absolute path not relative...
a8db267	Don't comment on commit
4421342	Use netlify deploy cli over api
a94d1e8	Copy docs before deployment

[kwankyu]

comment:14

Replying to @tobiasdiez:

Yes, here is what I have for now.

On each push, it builds the documentation and deploys it to netlify. Example: https://61ffda629c2d6f02653112ff--sagemath-tobias.netlify.app/

However, what Harald says above is still true. Currently, there are symlinks in the documentation tree (to import static files), which yields to display errors.

Where and how can I see the display errors?

[tobiasdiez]

comment:15

Attachment: Screenshot 2022-02-07 134004.png

Sorry, should have provided an example. I think all subpages that are not index.html are affected, for example https://61ffda629c2d6f02653112ff--sagemath-tobias.netlify.app/reference/structure/sage/structure/sage_object.html. The problem is the symlink that normally resolves _static to the correct path. This doesn't work on netlify (nor github pages) and leads to the following error:

[kwankyu]

comment:16

Replying to @tobiasdiez:

Sorry, should have provided an example. I think all subpages that are not index.html are affected, for example https://61ffda629c2d6f02653112ff--sagemath-tobias.netlify.app/reference/structure/sage/structure/sage_object.html. The problem is the symlink that normally resolves _static to the correct path. This doesn't work on netlify (nor github pages) and leads to the following error:

I see. Thanks. The symlinks are there to save disk space. But for CDN, it seems that the best option is just to copy files to resolve symlinks. See this:

https://superuser.com/questions/216919/how-to-copy-symlinks-to-target-as-normal-folders

[tobiasdiez]

comment:17

That would work indeed, thanks for the idea! Will try it this evening.

Why are symlinks actually needed in the first place? Is there no easy way to specify the correct path to the static files from these pages? (For example, in webpack you have ~ which resolves to the root of the project regardless from where it is accessed.)

[kwankyu]

comment:18

Replying to @tobiasdiez:

That would work indeed, thanks for the idea! Will try it this evening.

Why are symlinks actually needed in the first place? Is there no easy way to specify the correct path to the static files from these pages? (For example, in webpack you have ~ which resolves to the root of the project regardless from where it is accessed.)

Sphinx might have dictated hard-coded location of static files directory. But I don't know if this location can be changed.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

e0bac5f	Replace symlinks
e7c8e46	Don't use alias for now due to various netlify bugs

[sagetrac-git]

Changed commit from a94d1e8 to e7c8e46

[mkoeppe]

comment:20

merge #33309?

[sagetrac-git]

Changed commit from e7c8e46 to 870745e

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

5f7086f	Print url in action overview
870745e	Copy only contents of folder not the folder itself

[sagetrac-git]

Changed commit from 2c97b7c to 31665de

[tobiasdiez]

comment:59

Okay, it is now using the commit sha. Thanks for the suggestion.

[mkoeppe]

comment:60

Can you update the ticket description please?

[mkoeppe]

comment:61

I don't think the ticket and the comments in the yml file need to apologize for not using the branch name. Branch names are not very important in Sage - all branches are short-lived ticket branches.

[mkoeppe]

Replying to @mkoeppe:

• For pushes with tags (or to master), publish it on docs.sagemath.org website (for example using https://github.com/marketplace/actions/deploy-to-github-pages or by completely migrating to netlify).

How about adding another alias for the tag name?

[tobiasdiez]

comment:64

Replying to @mkoeppe:

Can you update the ticket description please?

Done.

[tobiasdiez]

comment:65

Replying to @mkoeppe:

I don't think the ticket and the comments in the yml file need to apologize for not using the branch name. Branch names are not very important in Sage - all branches are short-lived ticket branches.

It's the same spirit in netlify. It would be just nicer to have branch-based urls, since then you can easily refer to the latest version of the documentation of a ticket (e.g. if you share it with others). Netlify in addition would create a unique url for each deployment, so if you want to refer to a particular deploy this would still possible.

[mkoeppe]

comment:66

Even if fixed, it would make more sense for us to have URLs keyed to ticket numbers, not branch names

[tobiasdiez]

comment:67

Replying to @mkoeppe:

Replying to @mkoeppe:

• For pushes with tags (or to master), publish it on docs.sagemath.org website (for example using https://github.com/marketplace/actions/deploy-to-github-pages or by completely migrating to netlify).

How about adding another alias for the tag name?

I would prefer to keep this ticket for documentation previews for tickets; lets leave the CD of the main docs for another ticket.

[tobiasdiez]

comment:68

Replying to @mkoeppe:

Even if fixed, it would make more sense for us to have URLs keyed to ticket numbers, not branch names

Agreed.

[mkoeppe]

Reviewer: Matthias Koeppe

[mkoeppe]

comment:69

OK

[tobiasdiez]

comment:70

Thanks!

[vbraun]

Changed branch from public/build/doc_build_ci to 31665de

[mkoeppe]

Changed commit from 31665de to none

[mkoeppe]

Replying to @mkoeppe:

• Change the deployment url to something more official. For this reason, I've created the site sagemath-tobias.netlify.org, so if you want to create an official sage account, sagemath.netlify.org is still available. Then one just needs to change the secrets in the github repo.

Ticket is merged now - should provide the secrets now

[tobiasdiez]

comment:74

Replying to @mkoeppe:

Replying to @mkoeppe:

• Change the deployment url to something more official. For this reason, I've created the site sagemath-tobias.netlify.org, so if you want to create an official sage account, sagemath.netlify.org is still available. Then one just needs to change the secrets in the github repo.

Ticket is merged now - should provide the secrets now

Secrets are in place already, it just uses branchxyz--sagemath-tobias.netlify.org. This can stay for now (from my side). Depending on where and how you want to host the documentation in the future, it might make sense to completely migrate to netlify and register the docs.sagemath.org domain there, probably with a new netlify account. Netlify also has a free open source plan that should suffice for sage. https://www.netlify.com/legal/open-source-policy

[mkoeppe]

comment:75

Ah, thanks, I guess that's good enough for now.

[mkoeppe]

comment:76

I see that you have added the badge now to the ticket box. When the docbuild fails and nothing is deployed, would it be possible to link to the Actions run instead of the failed deployment - https://2ed27f1014ad0db190c370210e54b99b9ea8a3e1--sagemath-tobias.netlify.app/ ?

[tobiasdiez]

comment:77

Not easily. One could use the github api to query the workflow status and then decide about which link to use...

[mkoeppe]

comment:78

then maybe just add a second link for the docbuild logs?