✨ Using sphinx toc functions #219

choldgraf · 2020-06-27T18:10:29Z

This PR refactors our navigation bar, sidebar, and table of contents logic to use Sphinx toc functions instead of our own custom parsers. However, the output shouldn't change because it uses beautifulsoup to parse the HTML that Sphinx gives us in order to add the right classes and structure.

In implementing this, I also noticed some other bugs that our old parsing code had, that we didn't notice. (e.g., in our demo page, clicking the api documentation sidebar link doesn't expand that page's children). This should fix those bugs too since we're just using the basic Sphinx sidebar functionality, nothing custom.

Pros

This code is (IMO) much, much simpler and more maintainable than using a ton of custom jinja templates + python code to parse docutils
We will be more resistant to bugs over time, because we depend on Sphinx for handling the logic of toc / sidebar / etc
We'll get more predictable behavior for "toctree" features (such as :numbered:)
This theme should now behave more consistently with other sphinx themes

Cons

We add a dependency on beautifulsoup (which I hope isn't too bad, since that is such a standard in the Python world)

@jorisvandenbossche would love to know what you think here

resolves #220 #110
also can resolve #218 #135 (but needs tests)

choldgraf · 2020-06-29T15:42:27Z

src/scss/index.scss

@@ -283,35 +288,26 @@ table.field-list {
    }
  }

+  // For page headers with toctrees under them
+  li.page-header {


This just simplifies the CSS a little bit - we were using a bunch of level-specific selectors for our rules, but I think we can get the same effect with the generic rules here

Do you remember where/when this "page-header" class is used? I don't directly find it in the output HTML somewhere

@choldgraf I don't see this "page-header" class use anywhere, so for now going to remove it again. If it actually had a purpose, please explain and I will add it back :)

go for it - I have no idea what I was thinking here because this PR is from 7 months ago :-)

choldgraf · 2020-06-29T15:42:43Z

pydata_sphinx_theme/docs-sidebar.html

    </ul>
-
+  </div>


I guess we weren't closing out this div before 🤷

choldgraf · 2020-06-30T23:51:53Z

Friendly ping on this - this PR is also fixing a pretty big bug that we're experiencing in the Jupyter docs (pages that are > 2 levels deep don't show up in the sidebar).

I could try to implement the fix with our current docutils approach but it seemed like this PR would actually be a simpler way to solve the bug and solve ourselves some headache in the future as well...

jorisvandenbossche · 2020-07-02T21:39:10Z

(I am only just back, will try to take a look at this tomorrow, but from the description I am not fully fond of the idea ... but maybe we should have a call about it one of the coming days to easier discuss it?)

choldgraf · 2020-07-02T21:40:27Z

I'm happy to take a call to chat it through. This PR was mostly borne out of me being frustrated with debugging issues over and over again in our custom docutils parsing code :-)

My personal opinion is that this approach is much, much simpler than what we're currently doing. Rather than replicating 80% of Sphinx's toctree functionality with our own custom docutils parsing, we instead use the Sphinx toctree function and then just add a few classes here and there.

Basically, here's what we do now:

- Use sphinx TocTree class that outputs docutils objects
- Manually parse docutils to pull out the things we want
  - Replicate edge-cases like "toctrees that are underneath headers", and "multiple toctrees per page"
  - make lots of assumptions about the docutils tree that we have discovered are often not true
- Manually write HTML converters for the resulting docutils objects
- Write a bunch of custom jinja templates to build the HTML we want

and this PR makes it:

- Use sphinx's toctree function that outputs HTML
- Manually parse the HTML to add classes and remove the pieces we don't want

(FWIW, I'm also happy to chat on a quick call now if you wanna chat it through)

choldgraf · 2020-07-08T01:48:19Z

@jorisvandenbossche I just pushed an update to this PR that adds the ability for the get_nav_object function to return a raw BeautifulSoup object with minor modifications, in case theme editors would like to use it.

I also tested out how this could be used by a theme by trying it out in the sphinx-book-theme. You can see that in action in this PR: executablebooks/sphinx-book-theme#130

In particular, see how I'm calling get_nav_object here:

https://github.com/executablebooks/sphinx-book-theme/pull/130/files#diff-95c3c14f442154a7a77462f1e36e81e7R56

I'm then just doing some minor editing to that BeautifulSoup object and then returning my own HTML. What do you think? I can develop in this direction a bit further if you think it's a nice compromise.

choldgraf · 2020-07-13T03:47:53Z

Just a note that I'm just gonna implement this for the SBT theme. I'd really prefer to upstream this but I can't wait super long to get it in. Here's the PR that we can use for reference in the future:

executablebooks/sphinx-book-theme#130

Note that as I was implementing that PR, I also realized that sphinx-style page / section numbering also doesn't work in this theme (unless we manually add it in), so we should document that (or get this PR merged)

jorisvandenbossche · 2020-07-15T19:54:28Z

Just a note that I'm just gonna implement this for the SBT theme. I'd really prefer to upstream this but I can't wait super long to get it in.

OK, no problem, that way it can be test drived in jupyter-book-theme before upstreaming it ;-)

Sorry for the delay, and leaving now for holidays. But as I said on our call: I am certainly on board with the idea! (but I haven't yet been able to test it with some other projects)

choldgraf · 2020-08-28T18:19:57Z

@jorisvandenbossche what are your thoughts on this? FWIW we've had this in jupyter book for a few months now and it has worked fine.

mattip · 2020-09-04T07:48:57Z

FWIW, this PR changed numpy document build time (without multiprocessing via -j) from 300 secs to 352 secs. Most of the time is spent in page writing. Is that expected?

choldgraf · 2020-09-04T18:07:43Z

I wouldn't expect much performance change unless beautifulsoup is being extremely inefficient somehow...that's weird

that said, I'm on paternity leave now and don't have the time to keep rebasing this PR as the pydata theme changes, and it doesn't seem like folks are super interested in this functionality, so I am not planning to put a lot more work into it (though will note again that this has simplified things a lot in the sphinx-book-theme).

jorisvandenbossche · 2020-12-30T21:24:20Z

pydata_sphinx_theme/__init__.py

+        for i in range(1, context["theme_show_toc_level"] + 1):
+            for li in soup.select("li.toc-h{}".format(i + 1)):
+                ul = li.find_parent("ul")
+                classes = ul.get("class", [])
+                if "visible" not in classes:
+                    ul["class"] = classes + ["visible"]


@choldgraf this is "a" way that I found to get show_toc_level working, but I find it kind of hacky ..

The problem with the basic

for ul in soup("ul"): ul["class"] = ...

loop above this loop is that this is a flat iterator, and so you don't know how "deep" you are for each ul.

This "workaround" uses the toc-hN class names we added above, to find the appropriate levels of ul for which to add the class. I didn't directly find a way with beautifulsoup to only search a certain number of levels down.

this seems reasonable to me :-), but maybe we should add a comment to explain your reasoning here?

Hmm, not sure why I missed that earlier. But of course the code that adds those toc-hN classes (the classes that I was then using here to navigate the soup as a workaround), is already recursively iterating through the lists keeping track of the level ... :)
So simplified this

choldgraf · 2020-12-31T15:09:06Z

thanks for the pushes! can you update your "to do" list to reflect the things that you've tackled here?

choldgraf · 2021-01-09T16:44:12Z

@jorisvandenbossche do you think that we can leave:

sidebar html without skipping first level

to another PR? It feels to me like a feature request whereas this PR is meant to be a more "under the hood" change that doesn't affect end functionality but makes the theme less buggy. Then we can just merge this one now so it's easier to iterate with smaller updates. What do you think?

choldgraf · 2021-01-17T21:54:21Z

ping @jorisvandenbossche - I wanna get this PR merged before it goes stale again but I don't have the bandwidth to implement the extra logic for the sidebar right now...

jorisvandenbossche · 2021-01-19T15:22:58Z

@choldgraf reviving this. This PR certainly doesn't need to do everything that is possible with the new functions.
Some things I noticed while now trying out further:

The TOC for a page with multiple top-level headers doesn't work anymore: https://pydata-sphinx-theme--219.org.readthedocs.build/en/219/demo/mult_headers.html. That's probably something we should fix here.
Currently toctree "captions" are not yet included in the output (so this doesn't yet resolve adding captions to left sidebar #135), and also if I include them, they are not styled appropriately at all. This can easily go in a follow-up PR.

jorisvandenbossche · 2021-01-22T16:02:37Z

The TOC for a page with multiple top-level headers doesn't work anymore

This is actually something you fixed in executablebooks/sphinx-book-theme#215, so copied that part of the PR here.

choldgraf · 2021-01-22T20:20:26Z

So it sounds like we are good to merge this, and then we'll need to re-implement #135 using the new setup, right?

…or back compat

jorisvandenbossche · 2021-01-23T15:10:20Z

OK, this is merged ;)

I further removed the header-page CSS, renamed the new functions that returns the HTML to make the name more correct (generate_nav_html, like in sphinx book theme), and added a quick version of the old functions based on beautifulsoup for some backwards compatibility (we can see if we then want to deprecate them).

So it sounds like we are good to merge this, and then we'll need to re-implement #135 using the new setup, right?

Indeed, next steps are adding support for captions, and I also want to add support for collapsing subsections in the sidebar.
I have a messy branch experimenting with those on top of this PR, will try to extract some PRs from it.

choldgraf · 2021-01-23T15:35:09Z

hooray! what a journey :-)

are there issues for collapsible sidebar items etc? We should make sure not to lose those threads (and many may be upstreamable from the book theme)

cheekyshibe · 2021-02-07T10:05:28Z

OK, this is merged ;)

I further removed the header-page CSS, renamed the new functions that returns the HTML to make the name more correct (generate_nav_html, like in sphinx book theme), and added a quick version of the old functions based on beautifulsoup for some backwards compatibility (we can see if we then want to deprecate them).

So it sounds like we are good to merge this, and then we'll need to re-implement #135 using the new setup, right?

Indeed, next steps are adding support for captions, and I also want to add support for collapsing subsections in the sidebar.
I have a messy branch experimenting with those on top of this PR, will try to extract some PRs from it.

Support for captions and collapsing subsections~ Wohh sounds wonderful. 👍

choldgraf mentioned this pull request Jun 27, 2020

Adding headers before a toctree causes it to disappear in the sidebar #218

Closed

choldgraf force-pushed the missing_sidebar branch 3 times, most recently from 647eff3 to b381823 Compare June 29, 2020 15:22

choldgraf changed the title ~~fixing single-page header bug~~ Using sphinx toc functions Jun 29, 2020

choldgraf force-pushed the missing_sidebar branch 8 times, most recently from 6a10768 to 7027c90 Compare June 29, 2020 15:37

choldgraf commented Jun 29, 2020

View reviewed changes

pydata_sphinx_theme/docs-sidebar.html Outdated

</ul>

</div>

Copy link

Collaborator Author

choldgraf Jun 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we weren't closing out this div before 🤷

fixing single-page header bug

7b56b8b

choldgraf force-pushed the missing_sidebar branch from 7027c90 to 7b56b8b Compare June 29, 2020 15:43

choldgraf mentioned this pull request Jun 30, 2020

adding captions to left sidebar #135

Closed

choldgraf mentioned this pull request Jul 2, 2020

Show TOC from root index.html on left sidebar #221

Closed

adding raw mode to sidebar code

73876b7

choldgraf mentioned this pull request Jul 15, 2020

Math characters aren't rendered in sidebars #226

Closed

update expected test output

b2086d9

jorisvandenbossche reviewed Dec 30, 2020

View reviewed changes

temp

7080fc0

jorisvandenbossche mentioned this pull request Jan 19, 2021

Add section with subpages to demo docs + fix hover styling #300

Merged

jorisvandenbossche added 3 commits January 19, 2021 13:11

Merge remote-tracking branch 'upstream/master' into missing_sidebar

69b12a6

undo temp

f908b0b

much simpler way to support show_toc_level config

7ad0584

fix multiple headers in toc

f3a6c41

jorisvandenbossche mentioned this pull request Jan 22, 2021

uneven spacing between navbar items #302

Closed

jorisvandenbossche added 2 commits January 23, 2021 15:34

remove page-header

c14ea0e

rename to generate_nav/toc_html + keep bs-based version of old ones f…

7e1cd20

…or back compat

jorisvandenbossche changed the title ~~Using sphinx toc functions~~ ✨ Using sphinx toc functions Jan 23, 2021

jorisvandenbossche merged commit d2211c3 into pydata:master Jan 23, 2021

jorisvandenbossche mentioned this pull request Mar 28, 2021

Add branch coverage, improve overall coverage #356

Closed

jorisvandenbossche mentioned this pull request Apr 9, 2021

prototyping function for sidebar #110

Closed

12rambau mentioned this pull request Oct 18, 2022

Consider using a python function for our sidebar templating #109

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ Using sphinx toc functions #219

✨ Using sphinx toc functions #219

choldgraf commented Jun 27, 2020 •

edited by jorisvandenbossche

Loading

choldgraf Jun 29, 2020

jorisvandenbossche Jan 19, 2021

jorisvandenbossche Jan 22, 2021

choldgraf Jan 22, 2021

choldgraf Jun 29, 2020

choldgraf commented Jun 30, 2020 •

edited

Loading

jorisvandenbossche commented Jul 2, 2020

choldgraf commented Jul 2, 2020 •

edited

Loading

choldgraf commented Jul 8, 2020 •

edited

Loading

choldgraf commented Jul 13, 2020

jorisvandenbossche commented Jul 15, 2020

choldgraf commented Aug 28, 2020

mattip commented Sep 4, 2020

choldgraf commented Sep 4, 2020 •

edited

Loading

jorisvandenbossche Dec 30, 2020 •

edited

Loading

choldgraf Dec 31, 2020 •

edited

Loading

jorisvandenbossche Jan 19, 2021

choldgraf commented Dec 31, 2020

choldgraf commented Jan 9, 2021

choldgraf commented Jan 17, 2021

jorisvandenbossche commented Jan 19, 2021

jorisvandenbossche commented Jan 22, 2021

choldgraf commented Jan 22, 2021

jorisvandenbossche commented Jan 23, 2021

choldgraf commented Jan 23, 2021

cheekyshibe commented Feb 7, 2021

✨ Using sphinx toc functions #219

✨ Using sphinx toc functions #219

Conversation

choldgraf commented Jun 27, 2020 • edited by jorisvandenbossche Loading

Pros

Cons

choldgraf Jun 29, 2020

Choose a reason for hiding this comment

jorisvandenbossche Jan 19, 2021

Choose a reason for hiding this comment

jorisvandenbossche Jan 22, 2021

Choose a reason for hiding this comment

choldgraf Jan 22, 2021

Choose a reason for hiding this comment

choldgraf Jun 29, 2020

Choose a reason for hiding this comment

choldgraf commented Jun 30, 2020 • edited Loading

jorisvandenbossche commented Jul 2, 2020

choldgraf commented Jul 2, 2020 • edited Loading

choldgraf commented Jul 8, 2020 • edited Loading

choldgraf commented Jul 13, 2020

jorisvandenbossche commented Jul 15, 2020

choldgraf commented Aug 28, 2020

mattip commented Sep 4, 2020

choldgraf commented Sep 4, 2020 • edited Loading

jorisvandenbossche Dec 30, 2020 • edited Loading

Choose a reason for hiding this comment

choldgraf Dec 31, 2020 • edited Loading

Choose a reason for hiding this comment

jorisvandenbossche Jan 19, 2021

Choose a reason for hiding this comment

choldgraf commented Dec 31, 2020

choldgraf commented Jan 9, 2021

choldgraf commented Jan 17, 2021

jorisvandenbossche commented Jan 19, 2021

jorisvandenbossche commented Jan 22, 2021

choldgraf commented Jan 22, 2021

jorisvandenbossche commented Jan 23, 2021

choldgraf commented Jan 23, 2021

cheekyshibe commented Feb 7, 2021

choldgraf commented Jun 27, 2020 •

edited by jorisvandenbossche

Loading

choldgraf commented Jun 30, 2020 •

edited

Loading

choldgraf commented Jul 2, 2020 •

edited

Loading

choldgraf commented Jul 8, 2020 •

edited

Loading

choldgraf commented Sep 4, 2020 •

edited

Loading

jorisvandenbossche Dec 30, 2020 •

edited

Loading

choldgraf Dec 31, 2020 •

edited

Loading