Resurrect "sage -sws2rst" via a pip-installable package sage_sws2rst #28838

sagetrac-tmonteil · 2019-12-03T19:43:45Z

An important feature is to still be able to handle old .sws worksheets because a lot of pedagogical material was written on that format.

With the removal of sagenb, also sws2rst disappeared, see #30076 and related tickets.

We extract and translate the required material from sagenb to a new optional package sage_sws2rst that makes sage -sws2rst work.

As of this ticket, the source code of sage_sws2rst is in build/pkgs/sage_sws2rst/src, which is a pip-installable Python package; and one can build an sdist from there, for example using the spkg-src script. The source tree could as well be split out to a separate repository.

See also:

https://groups.google.com/d/msg/sage-devel/N61HlftxcLU/iVlQaacUAQAJ

CC: @vbraun @dimpase @kcrisman @jhpalmieri

Component: python3

Author: Frédéric Chapoton, Matthias Koeppe, Sébastien Labbé

Branch/Commit: c026892

Reviewer: Dima Pasechnik, Sébastien Labbé, Matthias Koeppe

Issue created by migration from https://trac.sagemath.org/ticket/28838

The text was updated successfully, but these errors were encountered:

kiwifb · 2019-12-03T21:36:35Z

comment:1

Is converting .sws file covered by sagenb_export?

sagetrac-tmonteil · 2019-12-04T12:05:07Z

comment:2

It seems not directly: after a quick look, the script seems to parse the html page, but maybe it could try to parse the html that belongs to the .sws archive (not sure if it has the same structure). Let me CC Volker for a better understanding of what is doable.

fchapoton · 2019-12-04T15:49:47Z

comment:3

Here is a first sketch. The moved files are not at the level of sage requirements, as many doctests are missing. If this matters for you, please take the job.

New commits:

`8b5de13`	`first sketch of sws2rst tools (moved from sagenb)`

fchapoton · 2019-12-04T15:49:47Z

Branch: u/chapoton/28838

fchapoton · 2019-12-04T15:49:47Z

Commit: 8b5de13

sagetrac-tmonteil · 2019-12-08T22:17:46Z

comment:4

Shouldn't this go to a subdirectory of src/ext/ (e.g. src/ext/sws2rst/) ?

fchapoton · 2019-12-09T08:02:05Z

comment:5

Salut Thierry.

Feel free to put it anywhere you like. I am not going to do anything more here.

embray · 2020-01-06T14:10:03Z

comment:6

Ticket retargeted after milestone closed

mkoeppe · 2020-08-13T16:56:07Z

comment:8

outdated, sws stuff was removed in #30076

sagetrac-tmonteil · 2020-08-13T18:05:28Z

comment:9

It is even more important now that there is no sagenb anymore, while there are still sws files that people might want to convert (e.g. from a former course). I will try to work on this.

dimpase · 2020-08-13T18:39:15Z

comment:10

it needs sagenb, a non-starter.

dimpase · 2020-08-13T18:39:15Z

Reviewer: Dima Pasechnik

sagetrac-tmonteil · 2020-08-13T19:02:24Z

comment:11

Replying to @dimpase:

it needs sagenb, a non-starter.

It only needs a small part of it (see Frédéric's patch), and the feature is still needed.

seblabbe · 2020-09-06T20:04:02Z

comment:73

Great!

Let me add a commit...

sagetrac-git · 2020-09-06T20:08:50Z

Branch pushed to git repo; I updated commit sha1. New commits:

`add4cc7`	`28838: more robust handling of strong and em, added doctests`

sagetrac-git · 2020-09-06T20:08:50Z

Changed commit from fe3d62d to add4cc7

seblabbe · 2020-09-06T20:14:10Z

comment:75

I am trying to run the tests with SAGE_CHECK, but before that I now get

$ make sage_sws2rst
...
[sage_sws2rst-none] /home/slabbe/GitBox/sage/build/pkgs/sage_sws2rst/spkg-install: 6: source: not found
[sage_sws2rst-none] Error: failed to source /home/slabbe/GitBox/sage/build/bin/sage-dist-helpers
[sage_sws2rst-none] Is /home/slabbe/GitBox/sage the correct SAGE_ROOT?
...
***************************************************************
Error building Sage.

The following package(s) may have failed to build (not necessarily
during this run of 'make sage_sws2rst'):

* package:         sage_sws2rst-none
  last build time: sept. 6 22:09
  log file:        /home/slabbe/GitBox/sage/logs/pkgs/sage_sws2rst-none.log
...

Same thing if I run SAGE_CHECK=yes make sage_sws2rst. Why does it failed to source?

mkoeppe · 2020-09-06T20:24:10Z

comment:76

Ah, sorry, I'll fix that

seblabbe · 2020-09-06T20:24:39Z

comment:77

So if I undo the change

-#! /usr/bin/env bash
+#! /bin/sh

then I am able to make sage_sws2rst. But then SAGE_CHECK=yes make sage_sws2rst fails with

[sage_sws2rst-none] Successfully installed sage-sws2rst-9.2
[sage_sws2rst-none] Removed build tracker: '/tmp/pip-req-tracker-jp5og0mz'
[sage_sws2rst-none] /home/slabbe/GitBox/sage/build/pkgs/sage_sws2rst/spkg-install: 
ligne 17: ./spkg-check: Aucun fichier ou dossier de ce type

Indeed, where is the spkg-check file?

sagetrac-git · 2020-09-06T20:24:51Z

Branch pushed to git repo; I updated commit sha1. New commits:

`2e63095`	`build/pkgs/sage_sws2rst/spkg-install: Remove bashism`

sagetrac-git · 2020-09-06T20:24:51Z

Changed commit from add4cc7 to 2e63095

sagetrac-git · 2020-09-06T20:25:53Z

Changed commit from 2e63095 to a363073

sagetrac-git · 2020-09-06T20:25:53Z

Branch pushed to git repo; I updated commit sha1. New commits:

`a363073`	`build/pkgs/sage_sws2rst/spkg-check: New`

seblabbe · 2020-09-06T20:32:02Z

comment:80

make sage_sws2rst now gives:

[sage_sws2rst-none] /home/slabbe/GitBox/sage/build/pkgs/sage_sws2rst/spkg-install: 
239: /home/slabbe/GitBox/sage/build/bin/sage-dist-helpers: 
Syntax error: "(" unexpected (expecting "}")

sagetrac-git · 2020-09-06T20:33:31Z

Branch pushed to git repo; I updated commit sha1. New commits:

`8e27734`	`build/pkgs/sage_sws2rst/spkg-install: Back to using bash`

sagetrac-git · 2020-09-06T20:33:31Z

Changed commit from a363073 to 8e27734

seblabbe · 2020-09-06T20:38:57Z

comment:82

Good. make sage_sws2rst works and SAGE_CHECK=yes make sage_sws2rst now gives:

[sage_sws2rst-none] Successfully built sage-sws2rst
[sage_sws2rst-none] Installing collected packages: sage-sws2rst
[sage_sws2rst-none]   Created temporary directory: /tmp/pip-unpacked-wheel-jwcrc2ua
[sage_sws2rst-none] 
[sage_sws2rst-none] Successfully installed sage-sws2rst-9.2
[sage_sws2rst-none] Removed build tracker: '/tmp/pip-req-tracker-w5ksuaxe'
[sage_sws2rst-none] + set -e
[sage_sws2rst-none] + sage-sws2rst -h
[sage_sws2rst-none] + python3 sage_sws2rst/comments2rst.py
[sage_sws2rst-none] + python3 sage_sws2rst/results2rst.py
[sage_sws2rst-none] + cd test
[sage_sws2rst-none] + sage-sws2rst Adding_Pictures_and_screenshots.sws
[sage_sws2rst-none] Processing Adding_Pictures_and_screenshots.sws
[sage_sws2rst-none] File at Adding_Pictures_and_screenshots.rst
[sage_sws2rst-none] Image directory at Adding_Pictures_and_screenshots_media
[sage_sws2rst-none] + sage-sws2rst MAT_141_day_25.sws
[sage_sws2rst-none] Processing MAT_141_day_25.sws
[sage_sws2rst-none] File at MAT_141_day_25.rst
[sage_sws2rst-none] Image directory at MAT_141_day_25_media
touch "/home/slabbe/GitBox/sage/local/var/lib/sage/installed/sage_sws2rst-none"
make[1] : on quitte le répertoire « /home/slabbe/GitBox/sage/build/make »

real	0m3,936s
user	0m3,554s
sys	0m0,403s
Sage build/upgrade complete!

Seems good!

seblabbe · 2020-09-06T20:49:17Z

comment:83

In a folder with the following makefile:

download:
	wget http://slabbe.org/Sage/2010-perpignan/CIRM_Tutoriel_3_francais.sws
	wget http://slabbe.org/Sage/2010-perpignan/Sage_tour_horizon.sws
	wget http://slabbe.org/Sage/2010-perpignan/CIRM_Tutorial_3.sws
	wget "http://slabbe.org/Sage/2010-perpignan/5, 6, 100.sws"
	wget http://slabbe.org/Sage/2010-perpignan/La_Suite_Un.sws
	wget "http://slabbe.org/Sage/2010-perpignan/Outils de dessins.sws"
	wget http://slabbe.org/Sage/2010-perpignan/CIRM_Tutorial_1.sws
	wget http://slabbe.org/Sage/2010-perpignan/CIRM_Tutoriel_1_francais.sws
	wget http://slabbe.org/Sage/2010-days25.5-tutoriel1.sws
	wget http://slabbe.org/Sage/2010-11-python-montreal-sage-demo.sws
	wget http://slabbe.org/Sage/2009-canadam-sagewords.sws

sws2rst:
	sage -sws2rst CIRM_Tutoriel_3_francais.sws
	sage -sws2rst Sage_tour_horizon.sws
	sage -sws2rst CIRM_Tutorial_3.sws
	sage -sws2rst "5, 6, 100.sws"
	sage -sws2rst La_Suite_Un.sws
	sage -sws2rst "Outils de dessins.sws"
	sage -sws2rst CIRM_Tutorial_1.sws
	sage -sws2rst CIRM_Tutoriel_1_francais.sws
	sage -sws2rst 2010-days25.5-tutoriel1.sws
	sage -sws2rst 2010-11-python-montreal-sage-demo.sws
	#sage -sws2rst 2009-canadam-sagewords.sws

rst2html:
	rst2html.py CIRM_Tutoriel_3_francais.rst a.out
	rst2html.py Sage_tour_horizon.rst a.out
	rst2html.py CIRM_Tutorial_3.rst a.out
	rst2html.py 5,_6,_100.rst a.out
	rst2html.py La_Suite_Un.rst a.out
	rst2html.py Outils_de_dessins.rst a.out
	rst2html.py CIRM_Tutorial_1.rst a.out
	rst2html.py CIRM_Tutoriel_1_francais.rst a.out
	rst2html.py 2010-days25.5-tutoriel1.rst a.out
	rst2html.py 2010-11-python-montreal-sage-demo.rst a.out
	#rst2html.py 2009-canadam-sagewords.rst a.out

then make download && make sws2rst works and

make rst2html

gives only very few warnings:

$ make rst2html 
rst2html.py CIRM_Tutoriel_3_francais.rst a.out
CIRM_Tutoriel_3_francais.rst:781: (WARNING/2) Inline emphasis start-string without end-string.
rst2html.py Sage_tour_horizon.rst a.out
rst2html.py CIRM_Tutorial_3.rst a.out
CIRM_Tutorial_3.rst:754: (WARNING/2) Inline emphasis start-string without end-string.
rst2html.py 5,_6,_100.rst a.out
rst2html.py La_Suite_Un.rst a.out
rst2html.py Outils_de_dessins.rst a.out
rst2html.py CIRM_Tutorial_1.rst a.out
CIRM_Tutorial_1.rst:176: (WARNING/2) Explicit markup ends without a blank line; unexpected unindent.
CIRM_Tutorial_1.rst:334: (WARNING/2) Inline strong start-string without end-string.
rst2html.py CIRM_Tutoriel_1_francais.rst a.out
CIRM_Tutoriel_1_francais.rst:178: (WARNING/2) Explicit markup ends without a blank line; unexpected unindent.
CIRM_Tutoriel_1_francais.rst:338: (WARNING/2) Inline strong start-string without end-string.
rst2html.py 2010-days25.5-tutoriel1.rst a.out
2010-days25.5-tutoriel1.rst:149: (WARNING/2) Block quote ends without a blank line; unexpected unindent.
2010-days25.5-tutoriel1.rst:182: (WARNING/2) Explicit markup ends without a blank line; unexpected unindent.
2010-days25.5-tutoriel1.rst:312: (WARNING/2) Inline strong start-string without end-string.
* Unknown equation environment pmatrix
rst2html.py 2010-11-python-montreal-sage-demo.rst a.out
2010-11-python-montreal-sage-demo.rst:495: (WARNING/2) Explicit markup ends without a blank line; unexpected unindent.
2010-11-python-montreal-sage-demo.rst:534: (WARNING/2) Explicit markup ends without a blank line; unexpected unindent.
#rst2html.py 2009-canadam-sagewords.rst a.out

It is true, that sometimes it does not translates the html properly. For instance, the following paragraph appears in the file CIRM_Tutorial_3.rst:

`Project Euler Problem 6 <http://projecteuler.net/index.php?section=problems&id=6>`_
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

<blockquote><p>The sum of the squares of the first ten natural numbers is,</p><div style="text-align: center;">1<img alt="^(" src="http://projecteuler.net/index.php?section=problems&id=6" style="display: none;"/><sup>2</sup><img alt=")" src="http://projecteuler.net/index.php?section=problems&id=6" style="display: none;"/> \+ 2<img alt="^(" src="http://projecteuler.net/index.php?section=problems&id=6" style="display: none;"/><sup>2</sup><img alt=")" src="http://projecteuler.net/index.php?section=problems&id=6" style="display: none;"/> \+ ... \+ 10<img alt="^(" src="http://projecteuler.net/index.php?section=problems&id=6" style="display: none;"/><sup>2</sup><img alt=")" src="http://projecteuler.net/index.php?section=problems&id=6" style="display: none;"/> = 385</div><p>The square of the sum of the first ten natural numbers is,</p><div style="text-align: center;">(1 \+ 2 \+ ... \+ 10)<img alt="^(" src="http://projecteuler.net/index.php?section=problems&id=6" style="display: none;"/><sup>2</sup><img alt=")" src="http://projecteuler.net/index.php?section=problems&id=6" style="display: none;"/> = 55<img alt="^(" src="http://projecteuler.net/index.php?section=problems&id=6" style="display: none;"/><sup>2</sup><img alt=")" src="http://projecteuler.net/index.php?section=problems&id=6" style="display: none;"/> = 3025</div><p>Hence the difference between the sum of the squares of the first ten natural numbers and the square of the sum is 3025 <img alt="−" border="0" height="3" src="http://projecteuler.net/images/symbol_minus.gif" style="vertical-align: middle;" width="9"/> 385 = 2640.</p><p>Find the difference between the sum of the squares of the first one hundred natural numbers and the square of the sum.</p></blockquote>

but I am quite sure it was the same problem before. From me it is a positive review.

Let's deal with blockquote and other missing styles in another ticket if somebody finds it desirable.

vbraun · 2020-09-10T20:48:55Z

comment:85

[dochtml] [thematic_] The HTML pages are in local/share/doc/sage/html/en/thematic_tutorials.
[dochtml] Error building the documentation.
[dochtml] Traceback (most recent call last):
[dochtml]   File "/home/release/Sage/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
[dochtml]     return _run_code(code, main_globals, None,
[dochtml]   File "/home/release/Sage/local/lib/python3.8/runpy.py", line 87, in _run_code
[dochtml]     exec(code, run_globals)
[dochtml]   File "/home/release/Sage/local/lib/python3.8/site-packages/sage_setup/docbuild/__main__.py", line 2, in <module>
[dochtml]     main()
[dochtml]   File "/home/release/Sage/local/lib/python3.8/site-packages/sage_setup/docbuild/__init__.py", line 1730, in main
[dochtml]     builder()
[dochtml]   File "/home/release/Sage/local/lib/python3.8/site-packages/sage_setup/docbuild/__init__.py", line 353, in _wrapper
[dochtml]     build_many(build_other_doc, L)
[dochtml]   File "/home/release/Sage/local/lib/python3.8/site-packages/sage_setup/docbuild/__init__.py", line 296, in build_many
[dochtml]     _build_many(target, args, processes=NUM_THREADS)
[dochtml]   File "/home/release/Sage/local/lib/python3.8/site-packages/sage_setup/docbuild/utils.py", line 289, in build_many
[dochtml]     raise worker_exc.original_exception
[dochtml] OSError: /home/release/Sage/src/doc/en/thematic_tutorials/sws2rst.rst: WARNING: document isn't included in any toctree

sagetrac-git · 2020-09-11T16:10:05Z

Changed commit from 8e27734 to c026892

sagetrac-git · 2020-09-11T16:10:05Z

Branch pushed to git repo; I updated commit sha1. New commits:

`0161466`	`Merge tag '9.2.beta12' into t/28838/28838`
`c026892`	`src/doc/en/thematic_tutorials/toctree.rst: Add sws2rst`

mkoeppe · 2020-09-12T23:04:41Z

comment:89

Docbuild is fixed, back to positive review

vbraun · 2020-09-18T20:23:13Z

Changed branch from public/28838 to c026892

sagetrac-tmonteil mannequin added this to the sage-9.0 milestone Dec 3, 2019

sagetrac-tmonteil mannequin added c: python3 labels Dec 3, 2019

embray modified the milestones: sage-9.0, sage-9.1 Jan 6, 2020

mkoeppe modified the milestones: sage-9.1, sage-9.2 May 5, 2020

mkoeppe removed this from the sage-9.2 milestone Aug 13, 2020

mkoeppe added the s: needs review label Aug 13, 2020

sagetrac-tmonteil mannequin added this to the sage-9.3 milestone Aug 13, 2020

sagetrac-tmonteil mannequin added s: needs work and removed s: needs review labels Aug 13, 2020

dimpase removed this from the sage-9.3 milestone Aug 13, 2020

dimpase added s: positive review and removed s: needs work labels Aug 13, 2020

sagetrac-tmonteil mannequin added this to the sage-9.2 milestone Aug 13, 2020

mkoeppe added s: positive review and removed s: needs review labels Sep 6, 2020

vbraun added s: needs work and removed s: positive review labels Sep 10, 2020

mkoeppe added s: needs review and removed s: needs work labels Sep 11, 2020

mkoeppe added s: positive review and removed s: needs review labels Sep 12, 2020

vbraun removed the s: positive review label Sep 18, 2020

vbraun closed this as completed in fda96c4 Sep 18, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Resurrect "sage -sws2rst" via a pip-installable package sage_sws2rst #28838

Resurrect "sage -sws2rst" via a pip-installable package sage_sws2rst #28838

sagetrac-tmonteil mannequin commented Dec 3, 2019

kiwifb commented Dec 3, 2019

sagetrac-tmonteil mannequin commented Dec 4, 2019

fchapoton commented Dec 4, 2019

fchapoton commented Dec 4, 2019

fchapoton commented Dec 4, 2019

sagetrac-tmonteil mannequin commented Dec 8, 2019

fchapoton commented Dec 9, 2019

embray commented Jan 6, 2020

mkoeppe commented Aug 13, 2020

sagetrac-tmonteil mannequin commented Aug 13, 2020

dimpase commented Aug 13, 2020

dimpase commented Aug 13, 2020

sagetrac-tmonteil mannequin commented Aug 13, 2020

seblabbe commented Sep 6, 2020

sagetrac-git mannequin commented Sep 6, 2020

sagetrac-git mannequin commented Sep 6, 2020

seblabbe commented Sep 6, 2020

mkoeppe commented Sep 6, 2020

seblabbe commented Sep 6, 2020

sagetrac-git mannequin commented Sep 6, 2020

sagetrac-git mannequin commented Sep 6, 2020

sagetrac-git mannequin commented Sep 6, 2020

sagetrac-git mannequin commented Sep 6, 2020

seblabbe commented Sep 6, 2020

sagetrac-git mannequin commented Sep 6, 2020

sagetrac-git mannequin commented Sep 6, 2020

seblabbe commented Sep 6, 2020

seblabbe commented Sep 6, 2020

vbraun commented Sep 10, 2020

sagetrac-git mannequin commented Sep 11, 2020

sagetrac-git mannequin commented Sep 11, 2020

mkoeppe commented Sep 12, 2020

vbraun commented Sep 18, 2020

Resurrect "sage -sws2rst" via a pip-installable package sage_sws2rst #28838

Resurrect "sage -sws2rst" via a pip-installable package sage_sws2rst #28838

Comments

sagetrac-tmonteil mannequin commented Dec 3, 2019

kiwifb commented Dec 3, 2019

sagetrac-tmonteil mannequin commented Dec 4, 2019

fchapoton commented Dec 4, 2019

fchapoton commented Dec 4, 2019

fchapoton commented Dec 4, 2019

sagetrac-tmonteil mannequin commented Dec 8, 2019

fchapoton commented Dec 9, 2019

embray commented Jan 6, 2020

mkoeppe commented Aug 13, 2020

sagetrac-tmonteil mannequin commented Aug 13, 2020

dimpase commented Aug 13, 2020

dimpase commented Aug 13, 2020

sagetrac-tmonteil mannequin commented Aug 13, 2020

seblabbe commented Sep 6, 2020

sagetrac-git mannequin commented Sep 6, 2020

sagetrac-git mannequin commented Sep 6, 2020

seblabbe commented Sep 6, 2020

mkoeppe commented Sep 6, 2020

seblabbe commented Sep 6, 2020

sagetrac-git mannequin commented Sep 6, 2020

sagetrac-git mannequin commented Sep 6, 2020

sagetrac-git mannequin commented Sep 6, 2020

sagetrac-git mannequin commented Sep 6, 2020

seblabbe commented Sep 6, 2020

sagetrac-git mannequin commented Sep 6, 2020

sagetrac-git mannequin commented Sep 6, 2020

seblabbe commented Sep 6, 2020

seblabbe commented Sep 6, 2020

vbraun commented Sep 10, 2020

sagetrac-git mannequin commented Sep 11, 2020

sagetrac-git mannequin commented Sep 11, 2020

mkoeppe commented Sep 12, 2020

vbraun commented Sep 18, 2020