Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't pickle generic types #511

Closed
pitrou opened this issue Dec 7, 2017 · 8 comments · Fixed by python/cpython#6216
Closed

Can't pickle generic types #511

pitrou opened this issue Dec 7, 2017 · 8 comments · Fixed by python/cpython#6216
Assignees

Comments

@pitrou
Copy link
Member

pitrou commented Dec 7, 2017

Briefly:

>>> import pickle, typing
>>> tp = typing.List[int]
>>> pickle.loads(pickle.dumps(tp))
Traceback (most recent call last):
  File "<ipython-input-5-ae365d48e86d>", line 1, in <module>
    pickle.loads(pickle.dumps(tp))
PicklingError: Can't pickle typing.List[int]: it's not the same object as typing.List

This can be an issue with certain advanced serialization libraries such as dill or cloudpickle, which are often used by distributed computing frameworks.

@ilevkivskyi
Copy link
Member

This is a known issue, unfortunately it is not easy to fix, see discussion in #306. In short, X.__qualname__ == X[int].__qualname__ and pickle doesn't like this since it looks up classes by names. PEP 560 will fix this in Python 3.7, not sure however what to do with the older versions.

@pitrou
Copy link
Member Author

pitrou commented Dec 7, 2017

Stupid question, but why isn't List[int] an instance of List?

@ilevkivskyi
Copy link
Member

Currently, List[int] is a subclass of List, all subscripted generics are currently actual class objects. One of the motivation of PEP 560 was to make them just normal objects that are aliases (proxies) for the original class object.

@pitrou
Copy link
Member Author

pitrou commented Dec 7, 2017

I guess my question was: why the whole design of having types be classes rather than instances? :-)

@ilevkivskyi
Copy link
Member

This is a very old (relative to me) decision, I don't know for sure, but I think one of the main motivation was that things like this should work:

class C(List[int], Generic[T]):
    ...

@pitrou
Copy link
Member Author

pitrou commented Dec 7, 2017

Regardless of the class issue, it seems it's difficult to find a reliable way to rebuild a generic type from its attributes. The problem is, both GenericMeta.__new__ and GenericMeta.__getitem__ do destructive conversion and fixup on inputs, and you can't derive the original inputs from them, because __extra__, __bases__ and friends only give you the converted form. The internal design seems actually hostile to any kind of instance rebuilding.

@pitrou
Copy link
Member Author

pitrou commented Dec 7, 2017

This is the __reduce__-like function I came up with for generic types:
https://gist.github.com/pitrou/a14a012871e852fdfaddbd8141d59611

Unfortunately, for some reason it succeeds on Generator[int, float, str] but fails on Generator[int, None, None] (see the comment about extra being stuffed into bases).

@ilevkivskyi ilevkivskyi self-assigned this Jan 27, 2018
@ilevkivskyi
Copy link
Member

Just a note that with PEP 560 in place the problems boils down to TypeVars and therefore is closely related to #512. The solution would be probably the same: consider type variables as immutable and pickle them like classes (they are also considered immutable by copy etc., and pickled as just f{__module__}.{__qualname__}).

miss-islington pushed a commit to miss-islington/cpython that referenced this issue Mar 26, 2018
… by copy and pickle (pythonGH-6216)

This also fixes python/typingGH-512
This also fixes python/typingGH-511

As was discussed in both issues, some typing forms deserve to be treated
as immutable by copy and pickle modules, so that:
* copy(X) is X
* deepcopy(X) is X
* loads(dumps(X)) is X  GH- pickled by reference

This PR adds such behaviour to:
* Type variables
* Special forms like Union, Any, ClassVar
* Unsubscripted generic aliases to containers like List, Mapping, Iterable

This not only resolves inconsistencies mentioned in the issues, but also
improves backwards compatibility with previous versions of Python
(including 3.6).

Note that this requires some dances with __module__ for type variables
(similar to NamedTuple) because the class TypeVar itself is define in typing,
while type variables should get module where they were defined.

https://bugs.python.org/issue32873
(cherry picked from commit 8349403)

Co-authored-by: Ivan Levkivskyi <levkivskyi@gmail.com>
ilevkivskyi added a commit to python/cpython that referenced this issue Mar 26, 2018
… by copy and pickle (GH-6216)

This also fixes python/typing#512
This also fixes python/typing#511

As was discussed in both issues, some typing forms deserve to be treated
as immutable by copy and pickle modules, so that:
* copy(X) is X
* deepcopy(X) is X
* loads(dumps(X)) is X  # pickled by reference

This PR adds such behaviour to:
* Type variables
* Special forms like Union, Any, ClassVar
* Unsubscripted generic aliases to containers like List, Mapping, Iterable

This not only resolves inconsistencies mentioned in the issues, but also
improves backwards compatibility with previous versions of Python
(including 3.6).

Note that this requires some dances with __module__ for type variables
(similar to NamedTuple) because the class TypeVar itself is define in typing,
while type variables should get module where they were defined.

https://bugs.python.org/issue32873
miss-islington added a commit to python/cpython that referenced this issue Mar 26, 2018
… by copy and pickle (GH-6216)

This also fixes python/typingGH-512
This also fixes python/typingGH-511

As was discussed in both issues, some typing forms deserve to be treated
as immutable by copy and pickle modules, so that:
* copy(X) is X
* deepcopy(X) is X
* loads(dumps(X)) is X  GH- pickled by reference

This PR adds such behaviour to:
* Type variables
* Special forms like Union, Any, ClassVar
* Unsubscripted generic aliases to containers like List, Mapping, Iterable

This not only resolves inconsistencies mentioned in the issues, but also
improves backwards compatibility with previous versions of Python
(including 3.6).

Note that this requires some dances with __module__ for type variables
(similar to NamedTuple) because the class TypeVar itself is define in typing,
while type variables should get module where they were defined.

https://bugs.python.org/issue32873
(cherry picked from commit 8349403)

Co-authored-by: Ivan Levkivskyi <levkivskyi@gmail.com>
cmnrd pushed a commit to tud-ccc/mocasin that referenced this issue May 4, 2021
…zed form with cloudpickle.

In Python 3.6, typing.List[str] cannot be pickled, therefore hydra
configuration cannot be passed with pickle. PEP 560 fixes the
issue in Python 3.7. As a workaround, we use more advanced serializtin
library cloudpickle.
See also: python/typing#511
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants