bpo-45292: [PEP 654] add the ExceptionGroup and BaseExceptionGroup classes

[iritkatriel]

This adds the ExceptionGroup and BaseExceptionGroup classes, but does not yet update the traceback display code to work with them correctly.

https://bugs.python.org/issue45292

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @iritkatriel for commit c26ff6b38333ab6c990791628f827c5ab97998c2 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @pablogsal for commit c26ff6b38333ab6c990791628f827c5ab97998c2 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

[iritkatriel]

This is part 1 - it rebased very easily so not much has changed in the area since April.

[gvanrossum]

Make sure you run the buildbot for leak checks.

I'd recommend asking for a review from a core dev who is better than me at catching subtle mistakes in C code (Serhiy?).

[iritkatriel]

Make sure you run the buildbot for leak checks.

Done.

I'd recommend asking for a review from a core dev who is better than me at catching subtle mistakes in C code (Serhiy?).

Would be great to have a review from @serhiy-storchaka if he's got the time and interest, and also @vstinner and @ncoghlan .

[Zac-HD]

This also seems weird to me: I'd expect that if I want MyEG to hold a BaseException, I'd better create a MyBaseEG to do so, ala:

class MyBEG(BaseExceptionGroup):
    @staticmethod
    def create(msg, excs):
        # per creation suggestion above
        if all(isinstance(e, Exception) for e in excs):
            return MyEG(msg, excs)
        return MyBEG(msg, excs)

class MyEG(MyBEG, ExceptionGroup):
    pass

[gvanrossum]

For user-defined subclasses, we have to punt on the special behavior. And yes, user classes should probably do what you show, but we don't want to enforce it. (That would be making assumptions.)

[Zac-HD]

Fair enough!

IMO defining custom ExceptionGroup subclasses should be very rare anyway (maybe just Hypothesis and Trio?), and we can certainly deal with whatever complexity that requires.

[gvanrossum]

Eh, why would Hypothesis need to subclass EG? While we were called on arbitrarily forbidding subclassing (as we did in an earlier PEP draft) I don't see why it would be useful. What is your use case? Maybe there's something we can add to the design to avoid the need? (For Trio I could imagine they'd want to provide some kind of backwards compatibility with Trio's current solution.)

[iritkatriel]

Maybe a subclass to annotate the nested exceptions? See https://discuss.python.org/t/accepting-pep-654-exception-groups-and-except/10813/9

[Zac-HD]

I'd prefer not to change the exception type if we can avoid it, but the more important objection is that it complicates the reporting solely for implementation reasons:

class Explanation(Exception):
    __module__ = "builtins"
    def __str__(self) -> str:
        return f"\n{self.args[0]}"

try:
    why = "Failed!"
    raise AssertionError(why)
except Exception as e:
    msg = "    You can reproduce this error by ...\n    ..."
    raise Explanation(msg) from e

    # Ideally something more like:
    e.__note__ = msg
    raise

$ python example.py
Traceback (most recent call last):
  File "example.py", line 8, in <module>
    raise AssertionError(why)
AssertionError: Failed!
                                                                        # These lines are
The above exception was the direct cause of the following exception:    # confusing for new 
                                                                        # users, and they
Traceback (most recent call last):                                      # only exist due 
  File "example.py", line 10, in <module>                               # to implementation
    raise Explanation(msg) from e                                       # constraints :-(
Explanation:                                                            # 
    You can reproduce this error by ...
    ...

[iritkatriel]

Hm, that does not look very principled. Why make it possible to print the __note__ when an exception is printed as the child of an EG but not when it's printed at the toplevel? Maybe Hypotheses could wrap exceptions in some kind of proxy exception?

I'm assuming that the note describes the leaf's membership in the EG rather than the leaf itself. If that's not the case I agree it doesn't make sense.

[Zac-HD]

The note is a description of the leaf, which could equally well be used for a single bare exception (though print() suffices).

[iritkatriel]

Sounds like the requirement is to add a feature to BaseException to enrich exceptions in a way that shows up in their __str__.

[iritkatriel]

I've created bpo45607 to track this request from Zac.

[gvanrossum]

(Buffer flush.)

[gvanrossum]

Eh, why would Hypothesis need to subclass EG? While we were called on arbitrarily forbidding subclassing (as we did in an earlier PEP draft) I don't see why it would be useful. What is your use case? Maybe there's something we can add to the design to avoid the need? (For Trio I could imagine they'd want to provide some kind of backwards compatibility with Trio's current solution.)

[gvanrossum]

@iritkatriel, have you managed to convince another core dev to review exceptions.c yet?

[gvanrossum]

Okay, I'll bite. What would you like to see supported upstream? A separate metadata item for each sub-exception? I'd say if you don't want to store that on the sub-exception in some dunder-ish attribute, you could indeed subclass EG to add that. There's an API so you can replicate this whenever an EG is being filtered. But I don't think we're going to support metadata that automatically gets replicated based on just this example -- in most cases I suspect you can just store an extra attribute on the individual exceptions.

[iritkatriel]

@iritkatriel, have you managed to convince another core dev to review exceptions.c yet?

Not that I know of.

[iritkatriel]

A PR for the traceback display code is here: iritkatriel#31

Once this PR is merged I will make a PR from that branch into main. But in the meantime it can be reviewed there, diffed against this PR.

[gvanrossum]

Anyway, if you want to override the way the traceback gets printed, subclassing EG isn't enough -- traceback-printing is (a) built into the C code, and (b) reimplemented in pure Python in traceback.py. To get something extra printed (e.g. err.__note__ if it exists) you will have to write your own traceback-printing code -- though there are various reusable bits and pieces in traceback.py.

[gvanrossum]

Anyway, if you want to override the way the traceback gets printed, subclassing EG isn't enough -- traceback-printing is (a) built into the C code, and (b) reimplemented in pure Python in traceback.py. To get something extra printed (e.g. err.__note__ if it exists) you will have to write your own traceback-printing code and invoke it at the right point -- though there are various reusable bits and pieces in traceback.py.

[iritkatriel]

(rebased)

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @iritkatriel for commit dbd72d1 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

[gvanrossum]

“And so it begins…”

[1st1]

@iritkatriel Congrats Irit. This is very impressive work.