Exceptions raised in a Directive cause parallel builds to hang

[Cadair]

Describe the bug

While it seems that raising exceptions in directives is at least frowned upon, the behaviour of when this happens during a parallel build seems to not be what I would expect.

If a (custom) directive raises an exception (regular Python exception, or sphinx ExtensionError) the parallel build will hang indefinitely after printing the error to screen.

To Reproduce

Modify an extension to raise an error inside the run() method of a Directive class, then build the project with -j auto.

Expected behaviour
The build process exits with a failing code after the error is displayed.

Environment info

• OS: Linux
• Python version: 3.9
• Sphinx version: 3.3.2
• Sphinx extensions: https://github.com/OpenAstronomy/sphinx-changelog but also reproduceable with other Directives

[tk0miya]

Is it really? I suppose Sphinx handles the error on the parallel build. As a trial, I added the following directive into conf.py:

from sphinx.util.docutils import SphinxDirective

class MyDirective(SphinxDirective):
    def run(self):
        raise

def setup(app):
    app.add_directive('my', MyDirective)

And sphinx-build exits successfully with displaying an error.

$ make clean html SPHINXOPTS=-jauto
Removing everything under '_build'...
Running Sphinx v3.5.1
making output directory... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 14 source files that are out of date
updating environment: [new config] 14 added, 0 changed, 0 removed
reading sources... [100%] subdir/foo .. subdir/qux
waiting for workers...

Sphinx parallel build error:
RuntimeError: No active exception to reraise
make: *** [Makefile:20: html] Error 2

Is there any example project?

[Cadair]

I just made this example branch: https://github.com/Cadair/sunpy/tree/test_sphinx_error

I too couldn't recreate it with a very small project, maybe because there isn't enough parallel work to do to spin up enough workers or something? I have also noticed the original bug with the changelog extension on multiple projects.

[tk0miya]

Thanks. I reproduce the error now!

FROM python:3.8-slim

RUN apt update;  apt install -y build-essential curl git make unzip vim
RUN git clone https://github.com/Cadair/sunpy
WORKDIR /sunpy
RUN git checkout test_sphinx_error
RUN pip install extension-helpers
RUN pip install numpy
RUN pip install .[docs]
WORKDIR /sunpy/docs
RUN pip install matplotlib
RUN pip install sqlalchemy
RUN pip install scikit-image
RUN pip install lxml
RUN pip install zeep
RUN pip install pandas
RUN pip install h5netcdf
RUN pip install drms
RUN make html SPHINXOPTS=-jauto

And I believe #8957 will fix this.

[Cadair]

Thanks a lot!