Complete type analysis of variadic types (python#15991)

This PR closes the first part of support for `TypeVarTuple`: the
"static" analysis of types (of course everything is static in mypy, but
some parts are more static): `semanal`/`typeanal`, `expand_type()`,
`map_instance_to_supertype()`, `erase_type()` (things that precede
and/or form foundation for type inference and subtyping). This one was
quite tricky, supporting unpacks of forward references required some
thinking.

What is included in this PR:
* Moving argument count validation from `semanal_typeargs` to
`typeanal`. In one of previous PRs I mentioned that `get_proper_type()`
may be called during semantic analysis causing troubles if we have
invalid aliases. So we need to move validation to early stage. For
instances, this is not required, but I strongly prefer keeping instances
and aliases similar. And ideally at some point we can combine the logic,
since it gets more and more similar. At some point we may want to
prohibit using `get_proper_type()` during semantic analysis, but I don't
want to block `TypeVarTuple` support on this, since this may be a
significant refactoring.
* Fixing `map_instance_to_supertype()` and `erase_type()`. These two are
straightforward, we either use `expand_type()` logic directly (by
calling it), or following the same logic.
* Few simplifications in `expandtype` and `typeops` following previous
normalizations of representation, unless there is a flaw in my logic,
removed branches should be all dead code.
* Allow (only fixed) unpacks in argument lists for non-variadic types.
They were prohibited for no good reason.
* (Somewhat limited) support for forward references in unpacks. As I
mentioned this one is tricky because of how forward references are
represented. Usually they follow either a life cycle like: `Any` ->
`<known type>`, or `<Any>` -> `<placeholder>` -> `<known type>` (second
one is relatively rare and usually only appears for potentially
recursive things like base classes or type alias targets). It looks like
`<placeholder>` can never appear as a _valid_ unpack target, I don't
have a proof for this, but I was not able to trigger this, so I am not
handling it (possible downside is that there may be extra errors about
invalid argument count for invalid unpack targets). If I am wrong and
this can happen in some valid cases, we can add handling for unpacks of
placeholders later. Currently, the handling for `Any` stage of forward
references is following: if we detect it, we simply create a dummy valid
alias or instance. This logic should work for the same reason having
plain `Any` worked in the first place (and why all tests pass if we
delete `visit_placeholder_type()`): because (almost) each time we
analyze a type, it is either already complete, or we analyze it _from
scratch_, i.e. we call `expr_to_unanalyzed_type()`, then
`visit_unbound_type()` etc. We almost never store "partially analyzed"
types (there are guards against incomplete references and placeholders
in annotations), and when we do, it is done in a controlled way that
guarantees a type will be re-analyzed again. Since this is such a tricky
subject, I didn't add any complex logic to support more tricky use cases
(like multiple forward references to fixed unpacks in single list). I
propose that we release this, and then see what kind of bug reports we
will get.
* Additional validation for type arguments position to ensure that
`TypeVarTuple`s are never split. Total count is not enough to ban case
where we have type variables `[T, *Ts, S, U]` and arguments `[int, int,
*Us, int]`. We need to explicitly ensure that actual suffix and prefix
are longer or equal to formal ones. Such splitting would be very hard to
support, and is explicitly banned by the PEP.
* Few minor cleanups.

Some random comments:
* It is tricky to preserve valid parts of type arguments, if there is an
argument count error involving an unpack. So after such error I simply
set all arguments to `Any` (or `*tuple[Any, ...]` when needed).
* I know there is some code duplication. I tried to factor it away, but
it turned out non-trivial. I may do some de-duplication pass after
everything is done, and it is easier to see the big picture.
* Type applications (i.e. when we have `A[int, int]` in runtime context)
are wild west currently. I decided to postpone variadic support for them
to a separate PR, because there is already some support (we will just
need to handle edge cases and more error conditions) and I wanted
minimize size of this PR.
* Something I wanted to mention in one of previous PRs but forgot: Long
time ago I proposed to normalize away type aliases inside `Unpack`, but
I abandoned this idea, it doesn't really give us any benefits.

As I said, this is the last PR for the "static part", in the next PR I
will work on fixing subtyping and inference for variadic instances. And
then will continue with remaining items I mentioned in my master plan in
python#15924

Fixes python#15978

---------

Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>

mypy/erasetype.py

@@ -165,9 +165,41 @@ def visit_type_var(self, t: TypeVarType) -> Type:
             return self.replacement
         return t
 
+    # TODO: below two methods duplicate some logic with expand_type().
+    # In fact, we may want to refactor this whole visitor to use expand_type().
+    def visit_instance(self, t: Instance) -> Type:
+        result = super().visit_instance(t)
+        assert isinstance(result, ProperType) and isinstance(result, Instance)
+        if t.type.fullname == "builtins.tuple":
+            # Normalize Tuple[*Tuple[X, ...], ...] -> Tuple[X, ...]
+            arg = result.args[0]
+            if isinstance(arg, UnpackType):
+                unpacked = get_proper_type(arg.type)
+                if isinstance(unpacked, Instance):
+                    assert unpacked.type.fullname == "builtins.tuple"
+                    return unpacked
+        return result
+
+    def visit_tuple_type(self, t: TupleType) -> Type:
+        result = super().visit_tuple_type(t)
+        assert isinstance(result, ProperType) and isinstance(result, TupleType)
+        if len(result.items) == 1:
+            # Normalize Tuple[*Tuple[X, ...]] -> Tuple[X, ...]
+            item = result.items[0]
+            if isinstance(item, UnpackType):
+                unpacked = get_proper_type(item.type)
+                if isinstance(unpacked, Instance):
+                    assert unpacked.type.fullname == "builtins.tuple"
+                    if result.partial_fallback.type.fullname != "builtins.tuple":
+                        # If it is a subtype (like named tuple) we need to preserve it,
+                        # this essentially mimics the logic in tuple_fallback().
+                        return result.partial_fallback.accept(self)
+                    return unpacked
+        return result
+
     def visit_type_var_tuple(self, t: TypeVarTupleType) -> Type:
         if self.erase_id(t.id):
-            return self.replacement
+            return t.tuple_fallback.copy_modified(args=[self.replacement])
         return t
 
     def visit_param_spec(self, t: ParamSpecType) -> Type:

mypy/expandtype.py

@@ -212,10 +212,15 @@ def visit_erased_type(self, t: ErasedType) -> Type:
 
     def visit_instance(self, t: Instance) -> Type:
         args = self.expand_types_with_unpack(list(t.args))
-        if isinstance(args, list):
-            return t.copy_modified(args=args)
-        else:
-            return args
+        if t.type.fullname == "builtins.tuple":
+            # Normalize Tuple[*Tuple[X, ...], ...] -> Tuple[X, ...]
+            arg = args[0]
+            if isinstance(arg, UnpackType):
+                unpacked = get_proper_type(arg.type)
+                if isinstance(unpacked, Instance):
+                    assert unpacked.type.fullname == "builtins.tuple"
+                    args = list(unpacked.args)
+        return t.copy_modified(args=args)
 
     def visit_type_var(self, t: TypeVarType) -> Type:
         # Normally upper bounds can't contain other type variables, the only exception is
@@ -285,7 +290,7 @@ def expand_unpack(self, t: UnpackType) -> list[Type]:
         ):
             return [UnpackType(typ=repl)]
         elif isinstance(repl, (AnyType, UninhabitedType)):
-            # Replace *Ts = Any with *Ts = *tuple[Any, ...] and some for Never.
+            # Replace *Ts = Any with *Ts = *tuple[Any, ...] and same for Never.
             # These types may appear here as a result of user error or failed inference.
             return [UnpackType(t.type.tuple_fallback.copy_modified(args=[repl]))]
         else:
@@ -377,15 +382,8 @@ def visit_overloaded(self, t: Overloaded) -> Type:
             items.append(new_item)
         return Overloaded(items)
 
-    def expand_types_with_unpack(
-        self, typs: Sequence[Type]
-    ) -> list[Type] | AnyType | UninhabitedType:
-        """Expands a list of types that has an unpack.
-
-        In corner cases, this can return a type rather than a list, in which case this
-        indicates use of Any or some error occurred earlier. In this case callers should
-        simply propagate the resulting type.
-        """
+    def expand_types_with_unpack(self, typs: Sequence[Type]) -> list[Type]:
+        """Expands a list of types that has an unpack."""
         items: list[Type] = []
         for item in typs:
             if isinstance(item, UnpackType) and isinstance(item.type, TypeVarTupleType):
@@ -396,24 +394,21 @@ def expand_types_with_unpack(
 
     def visit_tuple_type(self, t: TupleType) -> Type:
         items = self.expand_types_with_unpack(t.items)
-        if isinstance(items, list):
-            if len(items) == 1:
-                # Normalize Tuple[*Tuple[X, ...]] -> Tuple[X, ...]
-                item = items[0]
-                if isinstance(item, UnpackType):
-                    unpacked = get_proper_type(item.type)
-                    if isinstance(unpacked, Instance):
-                        assert unpacked.type.fullname == "builtins.tuple"
-                        if t.partial_fallback.type.fullname != "builtins.tuple":
-                            # If it is a subtype (like named tuple) we need to preserve it,
-                            # this essentially mimics the logic in tuple_fallback().
-                            return t.partial_fallback.accept(self)
-                        return unpacked
-            fallback = t.partial_fallback.accept(self)
-            assert isinstance(fallback, ProperType) and isinstance(fallback, Instance)
-            return t.copy_modified(items=items, fallback=fallback)
-        else:
-            return items
+        if len(items) == 1:
+            # Normalize Tuple[*Tuple[X, ...]] -> Tuple[X, ...]
+            item = items[0]
+            if isinstance(item, UnpackType):
+                unpacked = get_proper_type(item.type)
+                if isinstance(unpacked, Instance):
+                    assert unpacked.type.fullname == "builtins.tuple"
+                    if t.partial_fallback.type.fullname != "builtins.tuple":
+                        # If it is a subtype (like named tuple) we need to preserve it,
+                        # this essentially mimics the logic in tuple_fallback().
+                        return t.partial_fallback.accept(self)
+                    return unpacked
+        fallback = t.partial_fallback.accept(self)
+        assert isinstance(fallback, ProperType) and isinstance(fallback, Instance)
+        return t.copy_modified(items=items, fallback=fallback)
 
     def visit_typeddict_type(self, t: TypedDictType) -> Type:
         fallback = t.fallback.accept(self)
@@ -453,11 +448,8 @@ def visit_type_alias_type(self, t: TypeAliasType) -> Type:
         # Target of the type alias cannot contain type variables (not bound by the type
         # alias itself), so we just expand the arguments.
         args = self.expand_types_with_unpack(t.args)
-        if isinstance(args, list):
-            # TODO: normalize if target is Tuple, and args are [*tuple[X, ...]]?
-            return t.copy_modified(args=args)
-        else:
-            return args
+        # TODO: normalize if target is Tuple, and args are [*tuple[X, ...]]?
+        return t.copy_modified(args=args)
 
     def expand_types(self, types: Iterable[Type]) -> list[Type]:
         a: list[Type] = []

mypy/maptype.py

@@ -1,8 +1,8 @@
 from __future__ import annotations
 
-from mypy.expandtype import expand_type
+from mypy.expandtype import expand_type_by_instance
 from mypy.nodes import TypeInfo
-from mypy.types import AnyType, Instance, TupleType, Type, TypeOfAny, TypeVarId, has_type_vars
+from mypy.types import AnyType, Instance, TupleType, TypeOfAny, has_type_vars
 
 
 def map_instance_to_supertype(instance: Instance, superclass: TypeInfo) -> Instance:
@@ -25,8 +25,7 @@ def map_instance_to_supertype(instance: Instance, superclass: TypeInfo) -> Insta
             if not alias._is_recursive:
                 # Unfortunately we can't support this for generic recursive tuples.
                 # If we skip this special casing we will fall back to tuple[Any, ...].
-                env = instance_to_type_environment(instance)
-                tuple_type = expand_type(instance.type.tuple_type, env)
+                tuple_type = expand_type_by_instance(instance.type.tuple_type, instance)
                 if isinstance(tuple_type, TupleType):
                     # Make the import here to avoid cyclic imports.
                     import mypy.typeops
@@ -91,8 +90,7 @@ def map_instance_to_direct_supertypes(instance: Instance, supertype: TypeInfo) -
 
     for b in typ.bases:
         if b.type == supertype:
-            env = instance_to_type_environment(instance)
-            t = expand_type(b, env)
+            t = expand_type_by_instance(b, instance)
             assert isinstance(t, Instance)
             result.append(t)
 
@@ -103,15 +101,3 @@ def map_instance_to_direct_supertypes(instance: Instance, supertype: TypeInfo) -
         # type arguments implicitly.
         any_type = AnyType(TypeOfAny.unannotated)
         return [Instance(supertype, [any_type] * len(supertype.type_vars))]
-
-
-def instance_to_type_environment(instance: Instance) -> dict[TypeVarId, Type]:
-    """Given an Instance, produce the resulting type environment for type
-    variables bound by the Instance's class definition.
-
-    An Instance is a type application of a class (a TypeInfo) to its
-    required number of type arguments.  So this environment consists
-    of the class's type variables mapped to the Instance's actual
-    arguments.  The type variables are mapped by their `id`.
-    """
-    return {binder.id: arg for binder, arg in zip(instance.type.defn.type_vars, instance.args)}

mypy/semanal_typeargs.py

@@ -18,7 +18,6 @@
 from mypy.options import Options
 from mypy.scope import Scope
 from mypy.subtypes import is_same_type, is_subtype
-from mypy.typeanal import fix_type_var_tuple_argument, set_any_tvars
 from mypy.types import (
     AnyType,
     CallableType,
@@ -88,36 +87,7 @@ def visit_type_alias_type(self, t: TypeAliasType) -> None:
             # types, since errors there have already been reported.
             return
         self.seen_aliases.add(t)
-        # Some recursive aliases may produce spurious args. In principle this is not very
-        # important, as we would simply ignore them when expanding, but it is better to keep
-        # correct aliases. Also, variadic aliases are better to check when fully analyzed,
-        # so we do this here.
         assert t.alias is not None, f"Unfixed type alias {t.type_ref}"
-        # TODO: consider moving this validation to typeanal.py, expanding invalid aliases
-        # during semantic analysis may cause crashes.
-        if t.alias.tvar_tuple_index is not None:
-            correct = len(t.args) >= len(t.alias.alias_tvars) - 1
-            if any(
-                isinstance(a, UnpackType) and isinstance(get_proper_type(a.type), Instance)
-                for a in t.args
-            ):
-                correct = True
-        else:
-            correct = len(t.args) == len(t.alias.alias_tvars)
-        if not correct:
-            if t.alias.tvar_tuple_index is not None:
-                exp_len = f"at least {len(t.alias.alias_tvars) - 1}"
-            else:
-                exp_len = f"{len(t.alias.alias_tvars)}"
-            self.fail(
-                "Bad number of arguments for type alias,"
-                f" expected: {exp_len}, given: {len(t.args)}",
-                t,
-                code=codes.TYPE_ARG,
-            )
-            t.args = set_any_tvars(
-                t.alias, t.line, t.column, self.options, from_error=True, fail=self.fail
-            ).args
         is_error = self.validate_args(t.alias.name, t.args, t.alias.alias_tvars, t)
         if not is_error:
             # If there was already an error for the alias itself, there is no point in checking
@@ -144,34 +114,21 @@ def visit_callable_type(self, t: CallableType) -> None:
                     t.arg_types[star_index] = p_type.args[0]
 
     def visit_instance(self, t: Instance) -> None:
+        super().visit_instance(t)
         # Type argument counts were checked in the main semantic analyzer pass. We assume
         # that the counts are correct here.
         info = t.type
         if isinstance(info, FakeInfo):
             return  # https://github.com/python/mypy/issues/11079
-        t.args = tuple(flatten_nested_tuples(t.args))
-        if t.type.has_type_var_tuple_type:
-            # Regular Instances are already validated in typeanal.py.
-            # TODO: do something with partial overlap (probably just reject).
-            # also in other places where split_with_prefix_and_suffix() is used.
-            correct = len(t.args) >= len(t.type.type_vars) - 1
-            if any(
-                isinstance(a, UnpackType) and isinstance(get_proper_type(a.type), Instance)
-                for a in t.args
-            ):
-                correct = True
-            if not correct:
-                exp_len = f"at least {len(t.type.type_vars) - 1}"
-                self.fail(
-                    f"Bad number of arguments, expected: {exp_len}, given: {len(t.args)}",
-                    t,
-                    code=codes.TYPE_ARG,
-                )
-                any_type = AnyType(TypeOfAny.from_error)
-                t.args = (any_type,) * len(t.type.type_vars)
-                fix_type_var_tuple_argument(any_type, t)
         self.validate_args(info.name, t.args, info.defn.type_vars, t)
-        super().visit_instance(t)
+        if t.type.fullname == "builtins.tuple" and len(t.args) == 1:
+            # Normalize Tuple[*Tuple[X, ...], ...] -> Tuple[X, ...]
+            arg = t.args[0]
+            if isinstance(arg, UnpackType):
+                unpacked = get_proper_type(arg.type)
+                if isinstance(unpacked, Instance):
+                    assert unpacked.type.fullname == "builtins.tuple"
+                    t.args = unpacked.args
 
     def validate_args(
         self, name: str, args: Sequence[Type], type_vars: list[TypeVarLikeType], ctx: Context

mypy/test/testtypes.py

@@ -1464,7 +1464,7 @@ def make_call(*items: tuple[str, str | None]) -> CallExpr:
 class TestExpandTypeLimitGetProperType(TestCase):
     # WARNING: do not increase this number unless absolutely necessary,
     # and you understand what you are doing.
-    ALLOWED_GET_PROPER_TYPES = 7
+    ALLOWED_GET_PROPER_TYPES = 8
 
     @skipUnless(mypy.expandtype.__file__.endswith(".py"), "Skip for compiled mypy")
     def test_count_get_proper_type(self) -> None: