Enhancing the switchers setup

[JulienPalard]

Currently the contributions to python-docs-theme are made hard because of the language and version switchers.

I made review a bit easier by adding a github action to build the doc and provide it as a built artifact, so reviewers can just download and test locally.

But still the enhancement of the doc is made hard, for example python/python-docs-theme#46 has been slowed down because of this (sry @obulat).

Solution 1

I once had an idea to enhance the situation: we could provide an "API" on the form of a simple .js file at the root of docs.python.org listing the available versions and languages.

pros:

• It removes the switchers ugly hack in docsbuild-scripts.
• It make the theme easy to test locally: just drop a versions.js at the root with some sample data.
• A project using our theme with no need for switchers will not use a version.js file and have no switchers.
• A project using our theme with the need for switchers could set them up easily (add a version.js file).

cons:

• This is already the case, but we should be aware of the SEO penality that we could have if we redrow the page after load to render the switchers.
• The impementation will probably be tied to our specific hiearchy: /{LANG}/{VERSION}/ with the language being optional, defaulting to english.
• It may not follow the current state of the art of doing this, as I did not reviewd how other themes do this, how readthedocs does it, how for example https://docs.djangoproject.com/en/3.1/ does it.

Other ideas, and feedback welcome.

cc @pradyunsg @obulat

[pradyunsg]

Django hosts their documentation on ReadTheDocs, which provides the whole multiple version management stuff.

https://readthedocs.org/projects/django/

There's a custom Sphinx theme for Django's docs, maintained in the same repository:

https://github.com/django/django/tree/main/docs/_theme

[pradyunsg]

@humitos pointed out to me today that Django's RTD-built hosted site is different: https://django.readthedocs.io/en/stable/

[humitos]

Hi! The Django documentation is a tricky example to follow because they have some custom configuration that is not easy to get access to. If you follow the instructions from their README you will get an output similar to the one hosted on RTD: dark green background, yellow links, and small font --which is not the one deployed at https://docs.djangoproject.com/

Honestly, I don't know how they build the docs deployed as their official documentation.

It may not follow the current state of the art of doing this, as I did not reviewd how other themes do this, how readthedocs does it, how for example docs.djangoproject.com/en/3.1 does it.

Read the Docs uses /{lang}/{version}/ style and both are required. That is, going to just /en/ or just /3.8/ both fail. These can be configured to be redirected to some page instead, tho. For example: /en/ -> /en/3.9/ or /3.8/ -> /en/3.8/. Besides, the root / always redirects to the default version defined in the project's settings.

[JulienPalard]

Solution 2

I understand now, how RTD generates their version switchers, they inject all known versions in sphinx's conf.py then use it in the theme.

I like it because there's no javascript involved at runtime to display the page, but we can't easily do it as we can't cleanly modify conf.py (without using another ugly hack in docsbuild-script, replacing a hack with a hack is not my cup of tea).

The only way we can pass things to the HTML templates is by using sphinx-build -A which (I bet) won't allow structured data, only a string. Maybe we could use something like:

-A "versions=3.11:dev (3.11),3.10:pre (3.10),3.9:3.9,3.8:3.8,3.7:3.7,3.6:3.6,3.5:3.5,2.7:2.7"
-A "languages=en:English,es:Spanish,fr:French,ja:Japanese,ko:Korean,pt-br:Brazilian Portuguese,zh-cn:Simplified Chinese,zh-tw:Traditional Chinese"

and in the theme something like:

<select id="version_select">
{% for version in versions.split(',') %}
    <option value="{{ version.split(":")[0] }}">{{ version.split(":")[1] }}</option>
{% endfor %}
</select>

(yes there's a missing piece: the "currently selected version"),

pros:

It remove the switchers specific bits from cpython.
It removes the switchers ugly hack in docsbuild-scripts.
It make the theme easy to test locally: just call sphinx with `-A versions=...`.
A project using our theme with no need for switchers can just not use `-A versions=...` and have no switchers.
A project using our theme with the need for switchers could set them up easily by using `-A versions=...` too.
Switchers are rendered without the need of javascript which is nice (avoid re-rendering after page load).

cons:

The impementation will probably be tied to our specific hiearchy: /{LANG}/{VERSION}/ with the language being optional, defaulting to english.
It may not follow the current state of the art of doing this, too...
Updating the list of versions means rebuilding all HTML.

[JulienPalard]

pinging @obulat that may be interested in this topic.

[pradyunsg]

There's more interesting stuff too!

If you're using a theme that RTD doesn't know about, then it still will embed JS to inject a version selector.

See the discussion at pradyunsg/sphinx-basic-ng#12 and compare https://pradyunsg.me/furo to https://pip.pypa.io/ to see what's different on RTD hosted sites (ads and version selector).

[JulienPalard]

it still will embed JS to inject a version selector

It looks like an hybrid between the two first solutions, let's call it solution 3:

• Setup the switchers using JS at runtime.
• Hardcode the list of versions in the embedded JS using sphinx templating.

As we do have control on our theme, I don't think doing it in JS is better than doing it in HTML, so I feel solution 2 is cleaner (no runtime redraw).

For the moment I feel that solution 2 is the best.

[pradyunsg]

Yea, I agree. The RTD embed JS is smart-ish, so if we do the right things in the template, we can prevent a redraw if we're building on RTD.

The "big" question that I don't know the answer to is what the effort looks like for porting our existing docs to RTD, and what we'd need the RTD setup to look like.

[JulienPalard]

This is a thread about enhancing the switchers setup, I do understand that the way we build our switchers may ease or may make harder a possible switch to RTD, but there's no decision about porting to RTD yet. I'm opening a dedicated issue for this.

=> #5

[humitos]

I understand now, how RTD generates their version switchers, they inject all known versions in sphinx's conf.py then use it in the theme.

I like it because there's no javascript involved at runtime to display the page

Yes and no 😄

The flyout is built statically but it's then modified via JS when the page loads. This is because the active versions, translations, and others may have changed since the docs were built. JS hit the endpoint /_/api/v2/footer_html/ to get this data updated and injects the response replacing the flyout generated statically at built time. Using JS removes the need of rebuilding all the existing docs every time the data displayed in the flyout changes.

Theme authors will have more control over how the flyout is generated once we make some decisions in readthedocs/readthedocs.org#8052 and implement that solution.

[JulienPalard]

Yes and no smile

Haha now I'm lost again... So RTD solution looks like Solution 1 but with providing HTML, not a list of versions and it does not need to touch conf.py?

Using JS removes the need of rebuilding all the existing docs every time the data displayed in the flyout changes.

Make sense! I'm adding it to pros and cons of proposed solutions.

[obulat]

I am sorry I can't add anything to this discussion at the moment, I feel kind of lost here :)

[humitos]

So RTD solution looks like Solution 1 but with providing HTML, not a list of versions and it does not need to touch conf.py?

Maybe the confusion here is that we are using RTD to refer to different things. We have Read the Docs the building/hosting platform (https://readthedocs.org) and sphinx-rtd-theme (https://github.com/readthedocs/sphinx_rtd_theme).

• RTD the platform does not require modifying the conf.py (*) to generate the flyout with the version selector. It's all JS-based hitting the /footer_html/ endpoint to get the data (now an HTML blob, in the near future, hopefully, a JSON). In fact, it works exactly the same for MkDocs where there is no conf.py
• sphinx-rtd-theme uses the Sphinx context to statically render the flyout

When using sphinx-rtd-theme to host a project on https://readthedocs.org both things happen. So, if block the /footer_html/ call/URL in your browser and access that project, you will see the flyout generated statically by sphinx-rtd-theme. After unblocking that URL, you will see the dynamic JS-based flyout. It may or may not be different, depending on the data shown has changed since the last time it was built.

I hope this clarifies.

(*) it does modify it, tho --as you found it in sphinx's conf.py.tmpl, but that's just to show something the most accurate as possible in case the JS doesn't work for some reason.