mypy tests in test_exports.py are *extremely* slow.

[jakkdl]

With the timeout on the test run on main https://github.com/python-trio/trio/actions/runs/5148864245

I ran pytest with --durations=10 and realized just how slow the mypy tests are. On my machine they are 70% of the runtime.

running on 3.12

65.34s call     trio/_tests/test_exports.py::test_static_tool_sees_class_members[mypy-trio]
40.21s call     trio/_tests/test_exports.py::test_static_tool_sees_all_symbols[mypy-trio]
13.44s call     trio/_tests/test_exports.py::test_static_tool_sees_class_members[mypy-trio.abc]
6.77s call     trio/_tests/test_exports.py::test_static_tool_sees_class_members[mypy-trio.lowlevel]
4.67s call     trio/_tests/test_exports.py::test_static_tool_sees_class_members[mypy-trio.testing]
3.78s call     trio/_tests/test_unix_pipes.py::test_del
2.01s call     trio/_tests/test_timeouts.py::test_fail
2.01s call     trio/_tests/test_timeouts.py::test_sleep
1.92s call     trio/_tests/test_util.py::test_coroutine_or_error
1.52s call     trio/_tests/test_dtls.py::test_openssl_retransmit_doesnt_break_stuff

total time 174.26, the mypy tests sum up to 130.43, or 75%

Running on python3.11, I got 80.84/128.92=63%

Alternatives:

1. create a @superslow + --run-super-slow decorator, (or @mypy-tests + --run-mypy-tests) and only run the mypy tests on a single machine in CI.
2. optimize/rewrite the tests, so they make fewer / a single mypy call. Though even with a single call it's probably going to be somewhat slow.
3. set up a shared mypy cache across machines
4. Don't run the tests in CI at all, or like run them weekly/monthly instead of on-push. pyright + jedi export tests should catch most/all errors anyway, and are both much faster.
5. run mypy with --no-site-packages in export tests #2661 was a no-go. It's maybe possible to do some magic with skipping import-following, but it quickly gets quite messy because you still want to import all the trio submodules.

[Fuyukai]

Alternative 6: Remove all the really stupid dynamicism, and then you won't even have to run this test!

[A5rocks]

I believe we can snoop mypy's cache (which should be pretty stable version to version, but less guarantees). I set some time out for that earlier but got a bit distracted. I'll take a look tomorrow probably if nobody beats me to that.

[A5rocks]

OK I finally got around to looking at it: I believe we can just look at .mypy_cache/*/3.11/trio/<module>.data.json, which looks kinda like this (snipped to an item of the symbol table):

Details

    "SocketListener": {
      ".class": "SymbolTableNode",
      "kind": "Gdef",
      "node": {
        ".class": "TypeInfo",
        "_promote": [],
        "abstract_attributes": [],
        "alt_promote": null,
        "bases": [
          {
            ".class": "Instance",
            "args": ["trio.SocketStream"],
            "type_ref": "trio.abc.Listener"
          }
        ],
        "dataclass_transform_spec": null,
        "declared_metaclass": null,
        "defn": {
          ".class": "ClassDef",
          "fullname": "trio.SocketListener",
          "name": "SocketListener",
          "type_vars": []
        },
        "deletable_attributes": [],
        "flags": [],
        "fullname": "trio.SocketListener",
        "has_param_spec_type": false,
        "metaclass_type": "abc.ABCMeta",
        "metadata": {},
        "module_name": "trio",
        "mro": [
          "trio.SocketListener",
          "trio.abc.Listener",
          "trio.abc.AsyncResource",
          "builtins.object"
        ],
        "names": {
          ".class": "SymbolTable",
          "__init__": {
            ".class": "SymbolTableNode",
            "kind": "Mdef",
            "node": {
              ".class": "FuncDef",
              "abstract_status": 0,
              "arg_kinds": [0, 0],
              "arg_names": ["self", "socket"],
              "dataclass_transform_spec": null,
              "flags": [],
              "fullname": "trio.SocketListener.__init__",
              "name": "__init__",
              "type": {
                ".class": "CallableType",
                "arg_kinds": [0, 0],
                "arg_names": ["self", "socket"],
                "arg_types": ["trio.SocketListener", "trio.socket.SocketType"],
                "bound_args": [],
                "def_extras": { "first_arg": "self" },
                "fallback": "builtins.function",
                "from_concatenate": false,
                "implicit": false,
                "is_ellipsis_args": false,
                "name": "__init__ of SocketListener",
                "ret_type": { ".class": "NoneType" },
                "type_guard": null,
                "unpack_kwargs": false,
                "variables": []
              }
            }
          },
          "accept": {
            ".class": "SymbolTableNode",
            "kind": "Mdef",
            "node": {
              ".class": "FuncDef",
              "abstract_status": 0,
              "arg_kinds": [0],
              "arg_names": ["self"],
              "dataclass_transform_spec": null,
              "flags": ["is_coroutine"],
              "fullname": "trio.SocketListener.accept",
              "name": "accept",
              "type": {
                ".class": "CallableType",
                "arg_kinds": [0],
                "arg_names": ["self"],
                "arg_types": ["trio.SocketListener"],
                "bound_args": [],
                "def_extras": { "first_arg": "self" },
                "fallback": "builtins.function",
                "from_concatenate": false,
                "implicit": false,
                "is_ellipsis_args": false,
                "name": "accept of SocketListener",
                "ret_type": {
                  ".class": "Instance",
                  "args": [
                    {
                      ".class": "AnyType",
                      "missing_import_name": null,
                      "source_any": null,
                      "type_of_any": 6
                    },
                    {
                      ".class": "AnyType",
                      "missing_import_name": null,
                      "source_any": null,
                      "type_of_any": 6
                    },
                    "trio.SocketStream"
                  ],
                  "type_ref": "typing.Coroutine"
                },
                "type_guard": null,
                "unpack_kwargs": false,
                "variables": []
              }
            }
          },
          "aclose": {
            ".class": "SymbolTableNode",
            "kind": "Mdef",
            "node": {
              ".class": "FuncDef",
              "abstract_status": 0,
              "arg_kinds": [0],
              "arg_names": ["self"],
              "dataclass_transform_spec": null,
              "flags": ["is_coroutine"],
              "fullname": "trio.SocketListener.aclose",
              "name": "aclose",
              "type": {
                ".class": "CallableType",
                "arg_kinds": [0],
                "arg_names": ["self"],
                "arg_types": ["trio.SocketListener"],
                "bound_args": [],
                "def_extras": { "first_arg": "self" },
                "fallback": "builtins.function",
                "from_concatenate": false,
                "implicit": false,
                "is_ellipsis_args": false,
                "name": "aclose of SocketListener",
                "ret_type": {
                  ".class": "Instance",
                  "args": [
                    {
                      ".class": "AnyType",
                      "missing_import_name": null,
                      "source_any": null,
                      "type_of_any": 6
                    },
                    {
                      ".class": "AnyType",
                      "missing_import_name": null,
                      "source_any": null,
                      "type_of_any": 6
                    },
                    { ".class": "NoneType" }
                  ],
                  "type_ref": "typing.Coroutine"
                },
                "type_guard": null,
                "unpack_kwargs": false,
                "variables": []
              }
            }
          },
          "socket": {
            ".class": "SymbolTableNode",
            "kind": "Mdef",
            "node": {
              ".class": "Var",
              "flags": ["is_initialized_in_class", "is_ready"],
              "fullname": "trio.SocketListener.socket",
              "name": "socket",
              "type": "trio.socket.SocketType"
            }
          }
        },
        "self_type": null,
        "slots": null,
        "tuple_type": null,
        "type_vars": [],
        "typeddict_type": null
      }
    },

I think that works? I can make an actual test soon, hopefully.

[jakkdl]

Aah, that's what you meant. Yeah I've recently been looking at parsing the cache for other reasons and it should be quite straightforward here as we only care about the existence of symbols and don't even need to parse the type.
If necessary, this combined with a shared mypy cache would be lightning fast.

[A5rocks]

Here's roughly how long it takes if we just look at mypy cache (this is just running the mypy commands with none of the processing but should be about the same) on my Windows laptop running 3.11:

================================================ slowest 10 durations =================================================
7.46s call     trio/_tests/test_exports.py::test_static_tool_sees_all_symbols[mypy-trio.from_thread]
6.92s call     trio/_tests/test_exports.py::test_static_tool_sees_all_symbols[mypy-trio.socket]
6.65s call     trio/_tests/test_exports.py::test_static_tool_sees_all_symbols[mypy-trio.to_thread]
6.48s call     trio/_tests/test_exports.py::test_static_tool_sees_all_symbols[mypy-trio.testing]
6.41s call     trio/_tests/test_exports.py::test_static_tool_sees_all_symbols[mypy-trio.lowlevel]
6.03s call     trio/_tests/test_exports.py::test_static_tool_sees_all_symbols[mypy-trio.abc]
5.93s call     trio/_tests/test_exports.py::test_static_tool_sees_all_symbols[mypy-trio]

Edit: however, there are some discrepancies between running it on the command line and via the mypy.api.main function which I don't quite understand yet.