Slow page builds in 6.0 when generating navigation bar #381
Comments
|
I am having the same issue with pandas, which is related to the API docs -> #364 |
|
Perhaps we need to consider adding some explicit benchmarks... in the meantime, i'll see if i can dig up some numbers AP for the docs site build here so we can bisect a little. |
|
Here are some rough findings...
Looking at the data, my takeaway is b822548 is the place to start looking, when it jumps up and doesn't come back down....
|
|
I think our demo docs are a bit too small to really find the culprit. |
|
So I build a subset of the pandas API docs (removing the narrative user guide, as the slowdown comes from the writing phase, and it's the API docs that has many pages): https://gist.githubusercontent.com/jorisvandenbossche/f5ff72ee2eea52c30193abc2e9b5cd05/raw/bcc68040a7b3691e58828bf2dddaaed7d9866f57/profile-pandas-docs.svg Most of the time is spent in |
|
And the version with using the lxml parser through bs4: https://gist.githubusercontent.com/jorisvandenbossche/8aab410b0231a74d755ed54e656e5b7c/raw/8720e2df4426088483b6b775bac982cdb35bdc19/profile-pandas-docs-lxml.svg (for a big site like pandas, this doesn't seem to make much difference) |
|
We probably want to include a configuration option like readthedocs theme has: |
|
Well, if the lxml thing is a red herring (or maybe i don't have a handle on how to interpret the profiling): can more things be cached along the way? I don't know enough about the toctree data structure, but seems like it would be possible to generate The Tree, and then slice off the pieces needed per page? |
|
Yeah, I think sphinx is definitely doing a lot of duplicated effort .. However, to do that on our side might require some deeper plumbing into sphinx. For example, a large part of the time is spent in this The toctree it it is resolving is the same in many cases, but each time (for each page), the |
|
Same problem. The average building time increases from less than 10min to about 15min. writing output... [ 97%]
writing output... [ 97%]
waiting for workers... <- that takes a long time :( Does anybody know it is waiting for what exactly?
generating indices... genindex py-modindex done
copying notebooks ... [100%]
highlighting module code... [100%]
writing additional pages... search doneSeems that generating the collapsible sidebar spends a lot of time? Updated: 15min used in my 8core 16g machine. But for GitHub Actions (Ubuntu 2core 8g), now is over 1 hour.., https://github.com/MegEngine/Documentation/runs/2619670415
It's hard to accept. :( Updated again, the build artifact now is over 1.2G, most from API HTML files, and each file has over 10000 lines. |
|
In case anyone lands here and is looking for a workaround, the instructions in https://pydata-sphinx-theme.readthedocs.io/en/latest/user_guide/configuring.html#selectively-remove-pages-from-your-sidebar helped a lot for our project that uses sphinx-book-theme (which in turn uses pydata-sphinx-theme) and has many autogenerated API docs. (I found this issue by profiling my sphinx-build command using cprofile and identifying that |
|
Ah thanks for linking that @hawkinsp - that section was added to address this issue, so I think that we can close this one and I'll update the top comment with a link to that section |
choldgraf
changed the title
See numpy/numpy#18756. Build times went from around 10 minutes to 30+ on CircleCI.
Things to try
toctreeone time per page, and split it into two items for the navbar/sidebarTo resolve this
We decided that the slowdown here is somewhat unavoidable, as long as you want to keep multiple levels within your navigation bar. For some tips about speeding up site builds, see: https://pydata-sphinx-theme.readthedocs.io/en/latest/user_guide/configuring.html#selectively-remove-pages-from-your-sidebar
The text was updated successfully, but these errors were encountered: