Internal error appearing only with looser settings than defaults

[obi1kenobi]

Bug Report

mypy crashes with an internal error when scanning the arviz open-source project on the current master branch when using a custom (I believe purely looser) configuration than mypy's default:

[mypy]
no_implicit_optional = False
strict_optional = False
warn_redundant_casts = False
check_untyped_defs = False
disallow_untyped_calls = False
disallow_incomplete_defs = False
disallow_untyped_defs = False
disallow_untyped_decorators = False
ignore_missing_imports = True

Running with mypy's defaults does not result in a crash, and mypy I believe correctly reports many violations.

I found this while building typing_copilot, an open-source tool to make it easier to start using mypy in large codebases. The tool is able to find the strictest possible mypy configuration that produces no errors, and therefore enables a ratcheting workflow:

• Use typing_copilot to initially find the strictest passing configuration, set that as your starting mypy.ini file
• Use mypy in a CI environment to prevent regressions.
• Use typing_copilot to tighten the mypy.ini as your type coverage improves.
• Use typing_copilot in a CI environment to enforce that any time mypy.ini could be tightened, it actually gets tightened.

Sadly, this issue is preventing that initial search for the strictest passing configuration from working, since the configuration which causes the error is the loosest configuration supported by typing_copilot and therefore represents the "starting point" of the search.

Here's a link to the corresponding issue in its repo: obi1kenobi/typing_copilot#1

To Reproduce

1. Clone the arviz project repo and check out the master branch (commit hash 24ce65a44d8b48eb9377fdee38246d059ce3d201 at time of writing).
2. Make a virtualenv, install the project's dependencies from the several requirements.txt files. Ensure you have xarray==0.16.1 installed, since that dependency appears to play a part in the error.
3. Place the configuration given above into a file called custom_mypy.ini.
4. Run mypy --config-file ./custom_mypy.ini -p arviz --show-traceback and observe mypy crash with an internal error.
5. Then run mypy -p arviz --show-traceback (i.e. using the default configuration) and observe that mypy does not crash.

Expected Behavior

I did not expect mypy to crash, and I especially was surprised to see that loosening mypy configuration makes it crash.

Actual Behavior

When run with the custom config, mypy crashes with the following traceback:

/my/code/path/arviz/venv/lib/python3.8/site-packages/xarray/core/dataarray.py:2040: error: INTERNAL ERROR -- Please try using mypy master on Github:
https://mypy.rtfd.io/en/latest/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.790+dev.0b4a2c9d8edb57e2b61219c96eefe010073a5c14
Traceback (most recent call last):
  File "/my/code/path/arviz/venv/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/__main__.py", line 11, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/main.py", line 90, in main
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/build.py", line 180, in build
    result = _build(
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/build.py", line 254, in _build
    graph = dispatch(sources, manager, stdout)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/build.py", line 2630, in dispatch
    process_graph(graph, manager)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/build.py", line 2953, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/build.py", line 3051, in process_stale_scc
    graph[id].type_check_first_pass()
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/build.py", line 2111, in type_check_first_pass
    self.type_checker().check_first_pass()
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/checker.py", line 294, in check_first_pass
    self.accept(d)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/checker.py", line 401, in accept
    stmt.accept(self)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/nodes.py", line 940, in accept
    return visitor.visit_class_def(self)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/checker.py", line 1723, in visit_class_def
    self.accept(defn.defs)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/checker.py", line 401, in accept
    stmt.accept(self)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/nodes.py", line 1005, in accept
    return visitor.visit_block(self)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/checker.py", line 1972, in visit_block
    self.accept(s)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/checker.py", line 401, in accept
    stmt.accept(self)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/nodes.py", line 677, in accept
    return visitor.visit_func_def(self)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/checker.py", line 726, in visit_func_def
    self._visit_func_def(defn)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/checker.py", line 730, in _visit_func_def
    self.check_func_item(defn, name=defn.name)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/checker.py", line 792, in check_func_item
    self.check_func_def(defn, typ, name)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/checker.py", line 975, in check_func_def
    self.accept(item.body)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/checker.py", line 401, in accept
    stmt.accept(self)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/nodes.py", line 1005, in accept
    return visitor.visit_block(self)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/checker.py", line 1972, in visit_block
    self.accept(s)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/checker.py", line 401, in accept
    stmt.accept(self)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/nodes.py", line 1196, in accept
    return visitor.visit_if_stmt(self)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/checker.py", line 3255, in visit_if_stmt
    self.accept(b)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/checker.py", line 401, in accept
    stmt.accept(self)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/nodes.py", line 1005, in accept
    return visitor.visit_block(self)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/checker.py", line 1972, in visit_block
    self.accept(s)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/checker.py", line 401, in accept
    stmt.accept(self)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/nodes.py", line 1130, in accept
    return visitor.visit_for_stmt(self)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/checker.py", line 3478, in visit_for_stmt
    iterator_type, item_type = self.analyze_iterable_item_type(s.expr)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/checker.py", line 3511, in analyze_iterable_item_type
    return iterator, echk.check_method_call_by_name(nextmethod, iterator, [], [], expr)[0]
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/checkexpr.py", line 2346, in check_method_call_by_name
    method_type = analyze_member_access(method, base_type, context, False, False, True,
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/checkmember.py", line 126, in analyze_member_access
    result = _analyze_member_access(name, typ, mx, override_info)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/checkmember.py", line 143, in _analyze_member_access
    return analyze_instance_member_access(name, typ, mx, override_info)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/checkmember.py", line 225, in analyze_instance_member_access
    return analyze_member_var_access(name, typ, info, mx)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/checkmember.py", line 379, in analyze_member_var_access
    return analyze_var(name, v, itype, info, mx, implicit=implicit)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/checkmember.py", line 581, in analyze_var
    dispatched_type = meet.meet_types(mx.original_type, itype)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/meet.py", line 50, in meet_types
    return t.accept(TypeMeetVisitor(s))
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/types.py", line 833, in accept
    return visitor.visit_instance(self)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/meet.py", line 497, in visit_instance
    args.append(self.meet(ta, sia))
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/meet.py", line 647, in meet
    return meet_types(s, t)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/meet.py", line 50, in meet_types
    return t.accept(TypeMeetVisitor(s))
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/types.py", line 1386, in accept
    return visitor.visit_tuple_type(self)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/meet.py", line 584, in visit_tuple_type
    return TupleType(items, tuple_fallback(t))
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/typeops.py", line 44, in tuple_fallback
    return Instance(info, [join_type_list(typ.items)])
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/join.py", line 531, in join_type_list
    joined = join_types(joined, t)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/join.py", line 105, in join_types
    return t.accept(TypeJoinVisitor(s))
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/types.py", line 833, in accept
    return visitor.visit_instance(self)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/join.py", line 158, in visit_instance
    nominal = join_instances(t, self.s)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/join.py", line 371, in join_instances
    elif t.type.bases and is_subtype_ignoring_tvars(t, s):
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/subtypes.py", line 143, in is_subtype_ignoring_tvars
    return is_subtype(left, right, ignore_type_params=True)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/subtypes.py", line 93, in is_subtype
    return _is_subtype(left, right,
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/subtypes.py", line 135, in _is_subtype
    return left.accept(SubtypeVisitor(orig_right,
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/types.py", line 833, in accept
    return visitor.visit_instance(self)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/subtypes.py", line 264, in visit_instance
    if right.type.is_protocol and is_protocol_implementation(left, right):
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/subtypes.py", line 550, in is_protocol_implementation
    is_compat = is_subtype(subtype, supertype, ignore_pos_arg_names=ignore_names)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/subtypes.py", line 93, in is_subtype
    return _is_subtype(left, right,
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/subtypes.py", line 135, in _is_subtype
    return left.accept(SubtypeVisitor(orig_right,
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/types.py", line 1825, in accept
    return visitor.visit_partial_type(self)
  File "/my/code/path/arviz/venv/lib/python3.8/site-packages/mypy/subtypes.py", line 468, in visit_partial_type
    raise RuntimeError
RuntimeError: 
/my/code/path/arviz/venv/lib/python3.8/site-packages/xarray/core/dataarray.py:2040: : note: use --pdb to drop into pdb

Your Environment

• Mypy version used: current master -- 0.790+dev.0b4a2c9d8edb57e2b61219c96eefe010073a5c14
• Mypy command-line flags: --config-file ./custom_mypy.ini -p arviz --show-traceback
• Mypy configuration options from mypy.ini (and other config files): See top of issue post
• Python version used: 3.8.2
• Operating system and version: Ubuntu 20.04 LTS running under Windows Subsystem for Linux 2

I'm not familiar with the mypy codebase, but I'd love to help in any way I can since this is blocking my progress on typing_copilot, which I hope will be a tool useful to anyone seeking to start using mypy in a large existing Python codebase.

[gvanrossum]

This may or may not be easy to fix. The problem is probably that suppressing errors in earlier stages of the analysis sometimes causes Any types to appear in places where later stages don’t expect them. But it may not be easy to track down the source of the Any that’s causing the crash (if it is one).

[obi1kenobi]

Thanks for the reply, that makes sense and is indeed very unfortunate.

Would it be helpful if I write a script to try out different combinations of the config settings I have, and find the minimal such set where the bug still happens? Or is there any other way I can be helpful here?

[gvanrossum]

What would be most helpful is a short repro. This is not easy to script though. Start by looking at the file and line printed at the end of the crash.

[hauntsaninja]

This is a duplicate of #9437, so let's move discussion there. That issue mentions strict_optional=False as the source of the problem and provides a smaller repro.