Use dataclass(frozen=True) for compatibility with Python 3.13 (incomplete, needs help)

[simonw]

• Closes Python 3.13 error: cannot inherit frozen dataclass from a non-frozen one #1969
• Executed pre-commit run --all-files with no errors
• The change is fully covered by automated unit tests
• Documented in docs/ as appropriate
• Added an entry to the CHANGES file

[simonw]

Interesting test failure from docbuild here:

Document: getting/tutorial
--------------------------
**********************************************************************
File "getting/tutorial.rst", line 78, in default
Failed example:
    speed = 23 * ureg.snail_speed
Expected:
    Traceback (most recent call last):
    ...
    UndefinedUnitError: 'snail_speed' is not defined in the unit registry
Got:
    Traceback (most recent call last):
      File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/doctest.py", line 1350, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest default[0]>", line 1, in <module>
        speed = 23 * ureg.snail_speed
      File "<string>", line 4, in __setattr__
    dataclasses.FrozenInstanceError: cannot assign to field 'name'
**********************************************************************

[simonw]

I'm a bit stuck on what to do next with this one.

[codspeed-hq]

CodSpeed Performance Report

Merging #2037 will not alter performance

_{Comparing simonw:python-3.13 (01ea994) with master (1e46b2e)}

Summary

✅ 437 untouched benchmarks

[hgrecco]

While I have been a long proponent of making exception immutable (i.e. frozen when using a dataclasses) this has bit me many times. Do you know what is the change in Python 3.13 that makes this required?

[LecrisUT]

See: https://github.com/python/cpython/blame/3.13/Lib/dataclasses.py#L1035-L1044
It seems that was always meant to be like that, but it was not correctly checked before for multiple inheritance

[martinhoyer]

I've tried to play with it, but haven't found any good solution with keeping the dataclasses frozen in errors.py. Even if you use functools.cached_property, or other way to avoid mutation within, the errors will eventually get mutated, setting __cause__, etc. How does freezing a class that inherits non-frozen one supposed to work?

I'm not experienced enough to be able to help here, sorry.
Wouldn't using __slots__ make sense?

[LecrisUT]

@hgrecco would it be ok to move them all to frozen=False in order to unblock this issue? The doctest failure is fairly weird, because if you simply raise pint.errors.UndefinedUnitError("anything") you do not get the same traceback issue. I couldn't find the cause of this issue either. What I was able to find is that ureg.Quantity("10 * snail_speed") does output the correct exception 🤷

[LecrisUT]

I have finally figured out the issue. The issue is with double inheriting with AttributeError which is trying to set name = property

pint/pint/errors.py

Lines 133 to 134 in 2839f6e

	@dataclass(frozen=False)
	class UndefinedUnitError(AttributeError, PintError):

https://github.com/python/cpython/blob/3024b16d51bb7f74177c5a5038cc9a56fd2b26bd/Objects/exceptions.c#L2400-L2404

Removing that inheritance made the sphinx tests pass, but most likely that's not what we want here

[LecrisUT]

Suggested change

	class UndefinedUnitError(AttributeError, PintError):
	class UndefinedUnitError(PintError):

This technically would make the tests pass and as far as I was able to dig, this is the only problematic constructor? All of the others do not seem to have any constructor arguments

[khaeru]

I tried to poke around to see if there is anything here I could help with.

Interesting test failure from docbuild here:

Document: getting/tutorial
--------------------------
**********************************************************************
…
    dataclasses.FrozenInstanceError: cannot assign to field 'name'
**********************************************************************

• The logs for this GHA job have been garbage-collected, so I can't inspect directly.
• I checked-out the branch and tried to run this line from the docs.yml workflow file:

  pint/.github/workflows/docs.yml

  Line 50 in 2839f6e

  run: sphinx-build -a -j auto -b doctest -d build/doctrees docs build/doctest

• I do not see the above error.
• I notice that:
  • On my system, I am using Python 3.12.
  • docs.yml specifies Python 3.10:

    pint/.github/workflows/docs.yml

    Line 20 in 2839f6e

    python-version: "3.10"

    (and a very old version of actions/setup-python)
  • .readthedocs.yaml specifies Python 3.11:

    pint/.readthedocs.yaml

    Line 5 in 2839f6e

    python: "3.11"

@simonw, I wonder whether, if you were to rebase the branch and update those latter two files to Python 3.12, the CI error would disappear. My guess is:

• It is not a goal that the Pint docs build on every supported version of Python, but rather only the latest, and only that the Pint test suite works on every supported version.
• Since 3.10/3.11 were during the time frame when the underlying behaviour of CPython w.r.t. frozen dataclasses was being changed, it is probably easiest to skip to a recent version and proceed from there.

If you lack time to do this, please let me know and I can open a superseding PR with those suggested changes.

[LecrisUT]

@khaeru Run the sphinx-build command in a Python 3.13 environment. That is where you will encounter the error

[tacaswell]

Running this locally, I see the failures when un-pickling the exceptions via tblib with restores some extra __dunder__ values.

It looks like this is a dependency of dask and installs (via copyreg) its handlers to pickle/unpickle exceptions. If I uninstall tblib the tests pass locally for me with py313, however I have not tried to build the docs so maybe there are more problems.

I think we either need to make tblib smarter (presumably something can be done with __reduce_ex__...I assume that frozen dataclasses are pickable so there must be a way), do some funny stuff with __setattr__ to exempt a fixed list of keys from read-only constraint, or drop making the exceptions being frozen.

[LecrisUT]

@tacaswell I think you are experiencing a different issue. Either it is an issue with the other dataclass objects, in which case you should be able to reproduce with 3.12, or if it only occurs in 3.13 and it has a similar traceback as in the second message, then you are having an exception regardless. This PR is only patching the error dataclasses. Anyways we really need this PR merged. I don't think my proposed fix removing AttributeError would have disastrous consequences and it would at least allow pint to work.

@hgrecco gentle ping

[tacaswell]

My judgement is that removing AttributeError from the inheritance is a major API break as if you have

try:
    some_code_that_raises_UndefinedUnitError()
except AttributeError:
    ...

would be broken.

[LecrisUT]

My judgement is that removing AttributeError from the inheritance is a major API break as if you have

try:
    some_code_that_raises_UndefinedUnitError()
except AttributeError:
    ...

would be broken.

Yes, but realistically, why would you except AttributeError instead of PintError or UndefinedUnitError? I guess because you are accessing an attribute like ureg.snail_speed you could assume you want to check if the attribute exists, but other interfaces like ureg["snail_speed"] you wouldn't expect to have an AttributeError. Would using raise from also be an option, i.e. try catch in the __getattribute__ section and forward it as a chain instead of having them inherit one another.

[tacaswell]

Yes, but realistically, why would you except AttributeError instead of PintError or UndefinedUnitError?

Why users do things does not really come into what is or is not an API break.

However, a reason I would want to use the base exception rather than the pint-specific exception is if you are developing a packaging that can use multiple unit libraries where pint is optional and you can use AttributeError in a module that does not import pint that calls code that may.

[dopplershift]

However, a reason I would want to use the base exception rather than the pint-specific exception is if you are developing a packaging that can use multiple unit libraries where pint is optional and you can use AttributeError in a module that does not import pint that calls code that may.

Sure, but to my knowledge, there is no such multi-library spec/interface that actually exists to specify this. So IMO this goes squarely in the box of "nice to have", which is firmly below "currently broken on Python 3.13".

Also, pint is versioned 0.x, so major breaks, while considered carefully, are seemingly allowed.

If fixing tblib is an option, then great. Right now, pint and anything that depends on it is currently unimportable on the latest version of Python and we really need to get this fixed, one way or another.

[tacaswell]

diff --git a/pint/errors.py b/pint/errors.py
index f080f52..a1eeffc 100644
--- a/pint/errors.py
+++ b/pint/errors.py
@@ -103,7 +103,7 @@ class DefinitionError(ValueError, PintError):
 
 
 @dataclass(frozen=True)
-class DefinitionSyntaxError(ValueError, PintError):
+class DefinitionSyntaxError(ValueError):
     """Raised when a textual definition has a syntax error."""
 
     msg: str
@@ -116,7 +116,7 @@ class DefinitionSyntaxError(ValueError, PintError):
 
 
 @dataclass(frozen=True)
-class RedefinitionError(ValueError, PintError):
+class RedefinitionError(ValueError):
     """Raised when a unit or prefix is redefined."""
 
     name: str
@@ -131,7 +131,7 @@ class RedefinitionError(ValueError, PintError):
 
 
 @dataclass(frozen=True)
-class UndefinedUnitError(AttributeError, PintError):
+class UndefinedUnitError(AttributeError):
     """Raised when the units are not defined in the unit registry."""
 
     unit_names: str | tuple[str, ...]
@@ -151,7 +151,7 @@ class UndefinedUnitError(AttributeError, PintError):
 
 
 @dataclass(frozen=True)
-class PintTypeError(TypeError, PintError):
+class PintTypeError(TypeError):
     def __reduce__(self):
         return self.__class__, tuple(getattr(self, f.name) for f in fields(self))
 
@@ -234,7 +234,7 @@ class LogarithmicUnitCalculusError(PintTypeError):
 
 
 @dataclass(frozen=True)
-class UnitStrippedWarning(UserWarning, PintError):
+class UnitStrippedWarning(PintError):
     msg: str
 
     def __reduce__(self):
@@ -248,7 +248,7 @@ class UnexpectedScaleInContainer(Exception):
 
 
 @dataclass(frozen=True)
-class UndefinedBehavior(UserWarning, PintError):
+class UndefinedBehavior(PintError):
     msg: str
 
     def __reduce__(self):

Even with this patch I am still seeing the pickle failures with tblib (which is expected because it registers on Exception).

Dropping all of the frozen=True does not work because it looks like errors from flexparser use frozen dataclasses and are mixed into the pint classes.

[andrewgsavage]

You can make changes to flexparser. It used to be part of pint but was split out a few versions ago.

…

On Wed, 16 Oct 2024, 23:54 Thomas A Caswell, ***@***.***> wrote: diff --git a/pint/errors.py b/pint/errors.py index f080f52..a1eeffc 100644--- a/pint/errors.py+++ b/pint/errors.py@@ -103,7 +103,7 @@ class DefinitionError(ValueError, PintError): @DataClass(frozen=True)-class DefinitionSyntaxError(ValueError, PintError):+class DefinitionSyntaxError(ValueError): """Raised when a textual definition has a syntax error.""" msg: str@@ -116,7 +116,7 @@ class DefinitionSyntaxError(ValueError, PintError): @DataClass(frozen=True)-class RedefinitionError(ValueError, PintError):+class RedefinitionError(ValueError): """Raised when a unit or prefix is redefined.""" name: str@@ -131,7 +131,7 @@ class RedefinitionError(ValueError, PintError): @DataClass(frozen=True)-class UndefinedUnitError(AttributeError, PintError):+class UndefinedUnitError(AttributeError): """Raised when the units are not defined in the unit registry.""" unit_names: str | tuple[str, ...]@@ -151,7 +151,7 @@ class UndefinedUnitError(AttributeError, PintError): @DataClass(frozen=True)-class PintTypeError(TypeError, PintError):+class PintTypeError(TypeError): def __reduce__(self): return self.__class__, tuple(getattr(self, f.name) for f in fields(self)) @@ -234,7 +234,7 @@ class LogarithmicUnitCalculusError(PintTypeError): @DataClass(frozen=True)-class UnitStrippedWarning(UserWarning, PintError):+class UnitStrippedWarning(PintError): msg: str def __reduce__(self):@@ -248,7 +248,7 @@ class UnexpectedScaleInContainer(Exception): @DataClass(frozen=True)-class UndefinedBehavior(UserWarning, PintError):+class UndefinedBehavior(PintError): msg: str def __reduce__(self): Even with this patch I am still seeing the pickle failures with tblib (which is expected because it registers on Exception). Dropping all of the frozen=True does not work because it looks like errors from flexparser use frozen dataclasses and are mixed into the pint classes. — Reply to this email directly, view it on GitHub <#2037 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADEMLEFWHDXFU4Y6ROAJNDDZ33VDZAVCNFSM6AAAAABKZYNZ7WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJYGEYDGMRXHE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[tacaswell]

I should start by apologizing from showing up randomly and having a bunch of Strong Opinions, but hopefully Ryan will vouch for me ;)

---

Looking at FlexParser, we would have to end up making basically all of the dataclasses there non-frozon as well as the exceptions are multiply inherited from Statement. That in turn will require changing a bunch of other stuff in pint where Statement sub-classes are defined.

After a very quick skim of the code I have a couple of design questions (which maybe better as an issue on flexparser, but keeping here as this is the driver for asking these questions)

Do the error classes have to be both a Statement and an Exception ? I think that is the source of a lot of the pain here. I think a medium-scale change that would help is to drop the MI on the error classes and add a new exception class (or classes, one per error type) that only inherit from Exception and have a Statement as an attribute (if back-compat is a big concern, proxying the Statement API through does not look like it would be hard).

A second concern is that while Statement is frozen (which makes sense) the set_position method uses object.__setattr__ to escape the read-only aspect. Because this goes around the assumptions that dataclass thinks are true, it means you can change the hash of an object which breaks the Python object model in bad ways. I suspect the simplest fix is to exclude those fields from the hash.

[LecrisUT]

I should start by apologizing from showing up randomly and having a bunch of Strong Opinions, but hopefully Ryan will vouch for me ;)

No worries, and you are right about considering these API breakages. I was also considering unfreezing the FlexParser exceptions, but that would also require some coordination and version fixing of the dependencies with higher and lower bounds on the current and future version. This thing could get messy.

About your previous patch, the main issue is with UndefinedUnitError(AttributeError), not with UndefinedUnitError(PintError). Does patching that part out fix the pickling issue, if not there might be something deeper still.

If we want to properly fix the AttributeError issue, let's try the raise from approach. I'll try out to see if I can find the relevant place to patch that in.

Edit: I've hit a brick wall. It seems that exception chaining cannot be used with try except to catch either of those error. Tried with ExceptionGroup and that is even more complicated. Maybe an API breakage is inevitable?

    try:
        try:
            raise TypeError('Something awful has happened')
        except TypeError as e:
            raise ValueError('There was a bad value') from e
    # This works because it checks only the last exception thrown
    except ValueError as e:
        print(e.__context__)
    # This does not work because it does not catch other exceptions in the chain
    # except TypeError as e:
    #     print(e.__context__)

[hgrecco]

This I think is a good question:

Do the error classes have to be both a Statement and an Exception ? I think that is the source of a lot of the pain here

The answer is no. I like the idea of immutability for Statement and ParsedStatment. But maybe an Exception could include as an attribute an immutable Statement.

First, thank you all for the late reply.

This bug is due to my love of immutable objects. But I might have overstepped myself here, or at least Python thinks I have.

As far as I can tell, there are the following solutions:

1. Unfreeze Exceptions in Flexparser and in Pint.
2. Freeze Exception and in Pint, removing the AttributeError from the inheritance.
3. Undataclass exceptions in Flexparser and in Pint.

All are problematic in a sense.

1. Will require to synchronize and bump all versions and dependencies. Will break use code building their own exceptions.
2. Will break user code looking from AttributeError. Will break use code building their own exceptions.
3. Will require to make more constructors in Flexparser and Pint and synchronize and bump all versions and dependencies. Will break use code building their own exceptions.

My gut feeling is that (3) is more work for devs, but better for the user

Opinions?

[LecrisUT]

Ultimately any approach is welcome as long as it unblocks this issue and it seems like the user would have a surprise regardless of the solution. Also worth considering that pint is still not in a stable semver and many impacted users are in this thread.

---

1. Unfreeze Exceptions in Flexparser and in Pint.

3. Undataclass exceptions in Flexparser and in Pint.

For the user these would be equivalent and as long as there are no other major breaking changes that would have the user freeze or upper-bound either pint or flexparser I think it's ok doing. Should first check with sourcegraph on that though.

---

2. Freeze Exception and in Pint, removing the AttributeError from the inheritance.

I need to investigate more carefully when I'm at the computer, but it seems this case is very limited (only 1 true case in sourcegraph search) and even there the code readability would be improved by switching to except UndefinedUnitError

Indeed this would be a surprising breaking change, but I think it would be a preferred design for the long run, i.e. for the user to be more selective on the error source. Catching AttributeError is too vague and if they wanted a catch-all they would have used Exception.

---

Ultimately this is a design question of how should a library catch and raise their own, third-party and built-in exceptions. I'm still struggling to think about this one, but one design would be to always transform the exceptions to your own type with raise ... from and make sure that anything that is controlled by the library raises its own Exception type, otherwise it would be a probable bug in the library.

[hgrecco]

I just merge a PR in Flexparser to undataclass exceptions. If there are no showstoppers, tomorrow I will try to try it here.

[andrewgsavage]

fixed now

[tacaswell]

I am still seeing issues:

ipython
Python 3.13.0+ (heads/3.13:23468acac4f, Nov  8 2024, 09:41:11) [GCC 14.2.1 20240910]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.30.0.dev -- An enhanced Interactive Python. Type '?' for help.

In [1]: import pint
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[1], line 1
----> 1 import pint

File ~/.virtualenvs/cp313/lib/python3.13/site-packages/pint/__init__.py:18
     14 from __future__ import annotations
     16 from importlib.metadata import version
---> 18 from .delegates.formatter._format_helpers import formatter
     19 from .errors import (  # noqa: F401
     20     DefinitionSyntaxError,
     21     DimensionalityError,
   (...)
     27     UnitStrippedWarning,
     28 )
     29 from .formatting import register_unit_format

File ~/.virtualenvs/cp313/lib/python3.13/site-packages/pint/delegates/__init__.py:12
      1 """
      2     pint.delegates
      3     ~~~~~~~~~~~~~~
   (...)
      8     :license: BSD, see LICENSE for more details.
      9 """
     10 from __future__ import annotations
---> 12 from . import txt_defparser
     13 from .base_defparser import ParserConfig, build_disk_cache_class
     14 from .formatter import Formatter

File ~/.virtualenvs/cp313/lib/python3.13/site-packages/pint/delegates/txt_defparser/__init__.py:12
      1 """
      2     pint.delegates.txt_defparser
      3     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   (...)
      8     :license: BSD, see LICENSE for more details.
      9 """
     10 from __future__ import annotations
---> 12 from .defparser import DefParser
     14 __all__ = [
     15     "DefParser",
     16 ]

File ~/.virtualenvs/cp313/lib/python3.13/site-packages/pint/delegates/txt_defparser/defparser.py:10
      7 import flexparser as fp
      9 from ..base_defparser import ParserConfig
---> 10 from . import block, common, context, defaults, group, plain, system
     13 class PintRootBlock(
     14     fp.RootBlock[
     15         ty.Union[
   (...)
     29     ]
     30 ):
     31     pass

File ~/.virtualenvs/cp313/lib/python3.13/site-packages/pint/delegates/txt_defparser/common.py:47
     43     def set_location(self, value: str) -> None:
     44         super().__setattr__("location", value)
---> 47 @dataclass()
     48 class ImportDefinition(fp.IncludeStatement[ParserConfig]):
     49     value: str
     51     @property
     52     def target(self) -> str:

File ~/.pybuild/cp313/lib/python3.13/dataclasses.py:1295, in dataclass.<locals>.wrap(cls)
   1294 def wrap(cls):
-> 1295     return _process_class(cls, init, repr, eq, order, unsafe_hash,
   1296                           frozen, match_args, kw_only, slots,
   1297                           weakref_slot)

File ~/.pybuild/cp313/lib/python3.13/dataclasses.py:1038, in _process_class(cls, init, repr, eq, order, unsafe_hash, frozen, match_args, kw_only, slots, weakref_slot)
   1035 if has_dataclass_bases:
   1036     # Raise an exception if any of our bases are frozen, but we're not.
   1037     if any_frozen_base and not frozen:
-> 1038         raise TypeError('cannot inherit non-frozen dataclass from a '
   1039                         'frozen one')
   1041     # Raise an exception if we're frozen, but none of our bases are.
   1042     if all_frozen_bases is False and frozen:

TypeError: cannot inherit non-frozen dataclass from a frozen one

In [2]:

with the main branch of both flexparser and pint:

(cp313) ~/s/p/t/build_the_world main|+8-2…9
✔ 10:12:56 [belanna] @ pip list | grep -i pint
Pint                           0.24.2.dev14+g01ea994.d20241108
(cp313) ~/s/p/t/build_the_world main|+8-2…9
✔ 10:12:58 [belanna] @ pip list | grep -i flexparser
flexparser                     0.3.2.dev5+g7165ae4
(cp313) ~/s/p/t/build_the_world main|+8-2…9
✔ 10:13:05 [belanna] @

[LecrisUT]

Can't really replicate it:

$ python3.13 -m venv venv-3.13
$ source venv-3.13/bin/activate
(venv-3.13) lecris@Tuxedo:~/tmp/python$ pip install pint
Collecting pint
  Downloading Pint-0.24.4-py3-none-any.whl.metadata (8.5 kB)
Collecting platformdirs>=2.1.0 (from pint)
  Downloading platformdirs-4.3.6-py3-none-any.whl.metadata (11 kB)
Collecting typing-extensions>=4.0.0 (from pint)
  Using cached typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB)
Collecting flexcache>=0.3 (from pint)
  Downloading flexcache-0.3-py3-none-any.whl.metadata (7.0 kB)
Collecting flexparser>=0.4 (from pint)
  Downloading flexparser-0.4-py3-none-any.whl.metadata (18 kB)
Downloading Pint-0.24.4-py3-none-any.whl (302 kB)
Downloading flexcache-0.3-py3-none-any.whl (13 kB)
Downloading flexparser-0.4-py3-none-any.whl (27 kB)
Downloading platformdirs-4.3.6-py3-none-any.whl (18 kB)
Using cached typing_extensions-4.12.2-py3-none-any.whl (37 kB)
Installing collected packages: typing-extensions, platformdirs, flexparser, flexcache, pint
Successfully installed flexcache-0.3 flexparser-0.4 pint-0.24.4 platformdirs-4.3.6 typing-extensions-4.12.2

[notice] A new release of pip is available: 24.2 -> 24.3.1
[notice] To update, run: pip install --upgrade pip
(venv-3.13) $ python -c "import pint"

I've also tried the original issue here with ureg.snail_speed and it throws a UndefinedUnitError just fine. The only thing that strikes me is that the g01ea994 does not match the master branch short-hash g861953e (drop the leading g when you read these)

[hgrecco]

@tacaswell

(cp313) ~/s/p/t/build_the_world main|+8-2…9
✔ 10:12:56 [belanna] @ pip list | grep -i pint
Pint 0.24.2.dev14+g01ea994.d20241108

Have you tried upgrading to Pint 0.24.4?

[tacaswell]

🐑 thought I had master checked out, but I was still on the branch from this PR (+ some local changes).

Sorry for the noise and thank you for fixing the issue!