Allow unpacking of TypedDict into TypedDict #13353

eflorico · 2022-08-07T19:07:06Z

Description

Closes #9408 (and duplicates, e.g. #10585, #11108). Related tracking issue: #11753

Allows unpacking of TypedDicts into other TypedDicts:

class Foo(TypedDict):
    a: int

class Bar(TypedDict):
    a: int
    b: int

foo1: Foo = {'a': 1}
foo2: Foo = {**foo1}
bar: Bar = {**foo1, 'b': 2}

Should handle NotRequired fields correctly (they are counted as "maybe" present, we don't rely on them being present, but they should never be extraneous or have an incompatible type).

Also handles overwriting of fields correctly, for example:

class Foo(TypedDict):
    a: int
    b: str

class Bar(TypedDict):
    a: int
    b: int

foo: Foo = {'a': 1, 'b': '...'}
bar: Bar = {**foo} # Error: b is str instead of int
bar: Bar = {**foo, 'b': 2} # No error: b is overwritten

Limitations

Unpacking only works if the type of the unpacked expression is correctly inferred as a TypedDict. This is the case for variables or function return values, but not for dict expressions such as in foo: Foo = {**{'a': 1}}
Unpacking into unions does not work (foo_or_bar: Union[Foo, Bar] = {**foo1}). This is because find_typeddict_context can currently only handle one TypedDict in the context
Unpacking from unions does not work (foo: Bar = {**foo_or_bar}). I'm not sure yet how that could be added, I feel like it might take a bit more work

Test Plan

I have added quite detailed tests in check-typeddict.test.

eflorico · 2022-08-07T19:08:22Z

test-data/unit/check-typeddict.test

@@ -1653,9 +1653,9 @@ a.update({'x': 1})
 a.update({'x': ''}) # E: Incompatible types (expression has type "str", TypedDict item "x" has type "int")
 a.update({'x': 1, 'y': []})
 a.update({'x': 1, 'y': [1]})
-a.update({'z': 1}) # E: Unexpected TypedDict key "z"
-a.update({'z': 1, 'zz': 1}) # E: Unexpected TypedDict keys ("z", "zz")
-a.update({'z': 1, 'x': 1}) # E: Expected TypedDict key "x" but found keys ("z", "x")


I had to change TypedDict error messages to correctly handle keys that may be there (coming from an unpacked TypedDict with NotRequired fields). I found the resulting new error messages to be clearer, but I'm open to feedback on that

eflorico · 2022-08-07T19:10:15Z

mypy/checkexpr.py

-                    context=item_value,
+            if item_name in items:
+                item_value_expr, item_actual_type = items[item_name]
+                self.chk.check_subtype(


I'm calling check_subtype instead of check_simple_assignment because that allows consistent handling of both immediate and unpacked items. Let me know if check_simple_assignment does any important additional checks that might be important

eflorico · 2022-08-07T19:44:43Z

test-data/unit/check-typeddict.test

+[builtins fixtures/dict.pyi]
+[typing fixtures/typing-typeddict.pyi]
+
+


Both of the following test cases are for things that are not working yet, but that I hope will work at some point in the future. Is this the right way to test for that?

eflorico · 2022-08-07T19:45:10Z

mypy/checkexpr.py

+                # Unpack operator (**) was used; unpack all items of the type of this expression into items list
+                value_type = self.accept(value_expr, callee)
+                proper_type = get_proper_type(value_type)
+                if isinstance(proper_type, TypedDictType):


This will not work for unions of TypedDicts (or other more complex types). I'm open to suggestions to improve this

sobolevn

Thanks, this is not a full review. Just a couple of minor comments 🙂

mypy/checkexpr.py

sobolevn · 2022-08-07T22:19:06Z

mypy/checkexpr.py

-            return callee.required_keys <= set(validated_kwargs.keys()) <= set(callee.items.keys())
+            return (
+                callee.required_keys
+                <= set(dict(validated_kwargs).keys())


I think that keys() might be enough here 🤔

>>> set() <= {'a': 1}.keys() True

You're right! Interesting to see that dict_keys behaves like a set. Changed!

eflorico · 2022-08-08T20:26:42Z

Thanks, this is not a full review. Just a couple of minor comments 🙂

Thanks for your comments! I have updated the PR accordingly

github-actions · 2022-08-08T21:16:51Z

Diff from mypy_primer, showing the effect of this PR on open source code:

steam.py (https://github.com/Gobot1234/steam.py)
- steam/profile.py:217: error: Expected TypedDict key to be string literal  [misc]
+ steam/profile.py:217: error: Any cannot be unpacked into TypedDict (must be TypedDict)  [misc]
- steam/profile.py:302: error: Expected TypedDict key to be string literal  [misc]
+ steam/profile.py:302: error: Any cannot be unpacked into TypedDict (must be TypedDict)  [misc]

HansAarneLiblik · 2022-10-13T09:59:53Z

Any progress on this PR ?

eflorico · 2022-10-18T22:51:49Z

@HansAarneLiblik I have recently tried to merge master, but my changes were clashing with these recent changes. I haven't had the chance to figure out yet how to best solve this. If you have time to merge the current master and figure out how to unite both changes, I'd appreciate your help! I will only get around to look into this MR again in about two weeks

@JukkaL

Fixes #9408 Fixes #4122 Fixes #6462 Supersedes #13353 This PR enables two similar technically unsafe behaviors for TypedDicts, as @JukkaL explained in #6462 (comment) allowing an "incomplete" TypedDict as an argument to `.update()` is technically unsafe (and a similar argument applies to `**` syntax in TypedDict literals). These are however very common patterns (judging from number of duplicates to above issues), so I think we should support them. Here is what I propose: * Always support cases that are safe (like passing the type itself to `update`) * Allow popular but technically unsafe cases _by default_ * Have a new flag (as part of `--strict`) to fall back to current behavior Note that unfortunately we can't use just a custom new error code, since we need to conditionally tweak some types in a plugin. Btw there are couple TODOs I add here: * First is for unsafe behavior for repeated TypedDict keys. This is not new, I just noticed it when working on this * Second is for tricky corner case involving multiple `**` items where we may have false-negatives in strict mode. Note that I don't test all the possible combinations here (since the phase space is huge), but I think I am testing all main ingredients (and I will be glad to add more if needed): * All syntax variants for TypedDicts creation are handled * Various shadowing/overrides scenarios * Required vs non-required keys handling * Union types (both as item and target types) * Inference for generic TypedDicts * New strictness flag More than half of the tests I took from the original PR #13353

ilevkivskyi · 2023-06-26T23:27:25Z

This was superseded by #15425

Thank you for your efforts anyway, the test cases were in particular helpful!

Eric Wolf added 9 commits August 7, 2022 15:44

Mostly working

2c3850a

Clean up and fix edge cases

0d32e60

Cleanup, comments, tests

aaa4b20

Fix key error messages

9dda0f6

Format

e755d0e

Improve tests and error messages

74c31eb

Merge tests

4f89b41

Fix flake8 issues

63e4ca8

Merge branch 'master' into typeddict-spread

c4af4fd

eflorico commented Aug 7, 2022

View reviewed changes

eflorico mentioned this pull request Aug 7, 2022

Star-star syntax not allowed with TypedDict #9408

Closed

Tests and better error message for unions

df33b86

eflorico commented Aug 7, 2022

View reviewed changes

sobolevn reviewed Aug 7, 2022

View reviewed changes

This comment has been minimized.

Sign in to view

Eric Wolf added 3 commits August 8, 2022 22:11

Merge branch 'master' into typeddict-spread

c96fd8a

Update error messages in tests

4b9a739

Update according to feedback

114e12a

TeamSpen210 mentioned this pull request Apr 4, 2023

Handle assignments to a TypedDict with {**mapping} correctly #14946

Closed

ilevkivskyi mentioned this pull request Jun 12, 2023

Support flexible TypedDict creation/update #15425

Merged

ilevkivskyi closed this Jun 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow unpacking of TypedDict into TypedDict #13353

Allow unpacking of TypedDict into TypedDict #13353

eflorico commented Aug 7, 2022 •

edited

Loading

eflorico Aug 7, 2022

eflorico Aug 7, 2022

eflorico Aug 7, 2022

eflorico Aug 7, 2022

sobolevn left a comment

sobolevn Aug 7, 2022

eflorico Aug 8, 2022

This comment has been minimized.

eflorico commented Aug 8, 2022

github-actions bot commented Aug 8, 2022

HansAarneLiblik commented Oct 13, 2022

eflorico commented Oct 18, 2022

ilevkivskyi commented Jun 26, 2023

		[builtins fixtures/dict.pyi]
		[typing fixtures/typing-typeddict.pyi]

Allow unpacking of TypedDict into TypedDict #13353

Allow unpacking of TypedDict into TypedDict #13353

Conversation

eflorico commented Aug 7, 2022 • edited Loading

Description

Limitations

Test Plan

eflorico Aug 7, 2022

Choose a reason for hiding this comment

eflorico Aug 7, 2022

Choose a reason for hiding this comment

eflorico Aug 7, 2022

Choose a reason for hiding this comment

eflorico Aug 7, 2022

Choose a reason for hiding this comment

sobolevn left a comment

Choose a reason for hiding this comment

sobolevn Aug 7, 2022

Choose a reason for hiding this comment

eflorico Aug 8, 2022

Choose a reason for hiding this comment

This comment has been minimized.

eflorico commented Aug 8, 2022

github-actions bot commented Aug 8, 2022

HansAarneLiblik commented Oct 13, 2022

eflorico commented Oct 18, 2022

ilevkivskyi commented Jun 26, 2023

eflorico commented Aug 7, 2022 •

edited

Loading