Fix crashes in synthetic types #3322

ilevkivskyi · 2017-05-04T16:13:20Z

UPDATE: see #3322 (comment) below.

This PR adds better processing of "synthetic" types (NewType, NamedTuple, TypedDict) to the third pass and moves some processing from the second to the third pass. As @JukkaL noticed, this is still not perfect, since some MROs are recomputed in the third pass. I think the right way to fix this is just use unions in this case, the point is that users will practically never see these fallbacks, and this will also allow us to have better type expansions for fallbacks of generic tuple types. @JukkaL what do you think?

Additional fourth pass seems to be an overkill for this. Although, I have a big project in my future plans (probably after we are more or less done with protocols): providing better support for forward references (there are a dozen similar issues about this) and recursive types. Even for this fourth pass seems to be a bit an overkill, and I would rather proceed as Jukka proposed in option (3) in #1701 (comment)

JukkaL · 2017-05-04T16:36:55Z

Using unions as fallbacks for typed dicts has the problem that the union could be arbitrarily complex for a typed dict with multiple nested typed dicts, etc. It's actually just the kind of situation where unions may not be a good option. My current thinking is basically like this:

If there are generally at most 2 or 3 items, a union is probably okay. Examples: inferring Optional[x]; 1 if cond() else 'x' -> infer Union[int, str] (the latter doesn't work yet though).
If there can be a large number of items, a join (which doesn't usually produce unions) would be better. For example, we shouldn't infer unions from list expressions [x, y, ...] since there could be dozens of items, each with a slightly different type.

Also, we use is_subtype checks during the third pass of semantic analysis (in typeanal.py) and these are also probably not good. A simple approach that wouldn't perhaps require a full new pass would be to build a list of deferred lambdas/closures during pass 3 that would perform the final checks/computations without having to do AST traversal. Then pass 4 would just call these closures. However, even this might not be enough -- we may have to perform multiple runs of certain passes, I think. For example, we can't necessarily calculate a join until we have all the fallbacks, and we can't calculate fallbacks until we can do joins, so finding the right order of these operations can be hard. We could perhaps detect incomplete types and defer operations on them to another pass.

ilevkivskyi · 2017-05-04T22:36:57Z

I am marking this as work-in-progress until we resolve the question of third/fourth pass, also I have discovered some spurious errors with this PR in --strict-optional mode, because of incompatibility of tuple fallbacks.

ilevkivskyi · 2017-05-09T11:20:11Z

@JukkaL
I removed the controversial part from this PR, now it only fixes crashes reported in #3308

The rest can be fixed later in a separate PR when we will decide what to do with fourth pass.

gvanrossum · 2017-05-23T17:06:01Z

Can you resolve the merge conflict?

ilevkivskyi · 2017-05-23T17:39:26Z

Can you resolve the merge conflict?

Resolved.

ilevkivskyi · 2017-05-24T15:19:52Z

@gvanrossum I have fixed the merge conflict, do you have any comments on this?

gvanrossum

Really, all looks fine, just style nits. Thanks!

gvanrossum · 2017-05-24T18:45:21Z

mypy/nodes.py

@@ -731,6 +731,7 @@ class ClassDef(Statement):
    info = None  # type: TypeInfo  # Related TypeInfo
    metaclass = ''  # type: Optional[str]
    decorators = None  # type: List[Expression]
+    analyzed = None  # type: Optional[Expression]


Does this need an update to serialize() (perhaps in the comment)?

I think we don't need to serialize this (since it is needed only for semantic analysis). I updated the comment.

gvanrossum · 2017-05-24T18:55:40Z

mypy/semanal.py

@@ -3686,6 +3691,11 @@ def visit_class_def(self, tdef: ClassDef) -> None:
        if tdef.info.mro:
            tdef.info.mro = []  # Force recomputation
            calculate_class_mro(tdef, self.fail_blocker)
+        if tdef.analyzed:


Add is not None?

gvanrossum · 2017-05-24T18:57:23Z

mypy/semanal.py

+        if tdef.analyzed:
+            if isinstance(tdef.analyzed, TypedDictExpr):
+                self.analyze(tdef.analyzed.info.typeddict_type)
+            if isinstance(tdef.analyzed, NamedTupleExpr):


Probably use elif to make it totally clear that these are separate cases. Also, nothing to do for NewType here?

We (intentionally) support only functional syntax for NewType, therefore it can't appear in ClassDef analyzed.

gvanrossum · 2017-05-24T19:04:46Z

test-data/unit/check-classes.test

+from typing import List, NamedTuple
+from mypy_extensions import TypedDict
+
+class NM(NamedTuple):


I'd order the classes and the test expressions the same way.

gvanrossum · 2017-05-24T19:06:44Z

test-data/unit/check-classes.test

+y1 = NM(x=[])
+reveal_type(x) # E: Revealed type is 'TypedDict(x=builtins.list[Any], _fallback=__main__.TD)'
+reveal_type(x1) # E: Revealed type is 'TypedDict(x=builtins.list[Any], _fallback=typing.Mapping[builtins.str, builtins.list[Any]])'
+reveal_type(y) # E: Revealed type is 'Tuple[builtins.list[Any], fallback=__main__.NM]'


No reveal for y1?

gvanrossum · 2017-05-24T19:08:08Z

test-data/unit/check-classes.test

+[out]
+
+-- The two tests below will not crash after
+-- https://github.com/python/mypy/issues/3319 is fixed


Please remember to remove the -skip then.

gvanrossum · 2017-05-24T19:09:24Z

test-data/unit/check-classes.test

+[case testCrashInvalidArgsSyntheticClassSyntax]
+from typing import List, NamedTuple
+from mypy_extensions import TypedDict
+


I vote for fewer (or no) blank lines in tests.

ilevkivskyi · 2017-05-24T19:54:43Z

@gvanrossum Thanks for review! I fixed all points. (For some reasons Travis is now much slower than AppVeyor.)

gvanrossum · 2017-05-24T20:28:30Z

Will merge once tests pass. Perhaps Travis is squeezing the python project because the sprints cause many runs for many repos in the project..
.?

Ivan Levkivskyi and others added 8 commits May 4, 2017 11:04

Fix NewType crash

e52f9c5

Fix first kind of crashes in TypedDict and NamedTuple

554c6fa

Fix second kind of crashes with TypedDict and NamedTuple

590d1da

Fix tuple fallback expansion

ff786ca

Fix tuple fallback expansion also for generic aliases

e2f4237

Fix lint

3e0ef98

Add tests

310efe4

Merge branch 'master' into fix-forward-crash

66c0c69

ilevkivskyi changed the title ~~Fix crashes in synthetic types~~ [WIP] Fix crashes in synthetic types May 4, 2017

Merge branch 'master' into fix-forward-crash

ce514d1

This was referenced May 7, 2017

join is not associative in the presence of unions #3339

Open

Two crashes on recursive named tuples #3340

Closed

Ivan Levkivskyi added 2 commits May 9, 2017 13:12

Revert some controversial changes

3a14a53

Add accidentally removed import

e30e994

ilevkivskyi changed the title ~~[WIP] Fix crashes in synthetic types~~ Fix crashes in synthetic types May 9, 2017

ilevkivskyi mentioned this pull request May 23, 2017

Crashes related to calling isinstance() on a NamedTuple #3419

Closed

Merge branch 'master' into fix-forward-crash

57ca8f1

Fix typo in a test comment

32159cc

gvanrossum reviewed May 24, 2017

View reviewed changes

Address review comments

70e3f63

gvanrossum merged commit ccd95e6 into python:master May 24, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix crashes in synthetic types #3322

Fix crashes in synthetic types #3322

ilevkivskyi commented May 4, 2017 •

edited

Loading

JukkaL commented May 4, 2017

ilevkivskyi commented May 4, 2017 •

edited

Loading

ilevkivskyi commented May 9, 2017

gvanrossum commented May 23, 2017

ilevkivskyi commented May 23, 2017

ilevkivskyi commented May 24, 2017

gvanrossum left a comment

gvanrossum May 24, 2017

ilevkivskyi May 24, 2017

gvanrossum May 24, 2017

gvanrossum May 24, 2017

ilevkivskyi May 24, 2017

gvanrossum May 24, 2017

gvanrossum May 24, 2017

ilevkivskyi May 24, 2017

gvanrossum May 24, 2017

gvanrossum May 24, 2017

ilevkivskyi commented May 24, 2017 •

edited

Loading

gvanrossum commented May 24, 2017

Fix crashes in synthetic types #3322

Fix crashes in synthetic types #3322

Conversation

ilevkivskyi commented May 4, 2017 • edited Loading

JukkaL commented May 4, 2017

ilevkivskyi commented May 4, 2017 • edited Loading

ilevkivskyi commented May 9, 2017

gvanrossum commented May 23, 2017

ilevkivskyi commented May 23, 2017

ilevkivskyi commented May 24, 2017

gvanrossum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ilevkivskyi commented May 24, 2017 • edited Loading

gvanrossum commented May 24, 2017

ilevkivskyi commented May 4, 2017 •

edited

Loading

ilevkivskyi commented May 4, 2017 •

edited

Loading

ilevkivskyi commented May 24, 2017 •

edited

Loading