Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Wildly inconsistent performance for seemingly trivial changes to source code #14271

Open
huguesb opened this issue Dec 10, 2022 · 1 comment · May be fixed by #14274
Open

Wildly inconsistent performance for seemingly trivial changes to source code #14271

huguesb opened this issue Dec 10, 2022 · 1 comment · May be fixed by #14274
Labels
bug mypy got something wrong performance

Comments

@huguesb
Copy link
Contributor

huguesb commented Dec 10, 2022

My company is currently using 0.971 internally. I was looking into upgrading to 0.991 and discovered a significant performance regression. Encourage by the recent commit log I tried the master branch as well, which is definitely better but still much worse than 0.971 on our codebase: roughly 4x slower: from ~300s to ~1200s for whole-codebase run without incremental cache.

In the course of narrowing down the slowdown (via per-file and per-line timing stats), I discovered a surprisingly large performance difference for seemingly very similar code. Consider:

from typing import Any, Dict

class Foo(object):
    def __init__(self) -> None:
        self.field = {
            'XXX': {
                'XXX': 'XXX',
                'XXX': {
                    'XXX': [
                        {
                            'XXX': 'XXX',
                            'XXX': 'XXX',
                            'XXX': None,
                            'XXX': None
                        },
                        {
                            'XXX': 'XXX',
                            'XXX': 'XXX',
                            'XXX': None,
                            'XXX': None
                        },
                        {
                            'XXX': 'XXX',
                            'XXX': 'XXX',
                            'XXX': None,
                            'XXX': None
                        },
                        {
                            'XXX': 'XXX',
                            'XXX': 'XXX',
                            'XXX': None,
                            'XXX': None
                        },
                        {
                            'XXX': 'XXX',
                            'XXX': 'XXX',
                            'XXX': None,
                            'XXX': None
                        }
                    ],
                },
                'XXX': {
                    'XXX': [
                        {
                            'XXX': 'XXX',
                            'XXX': 'XXX',
                            'XXX': {
                                'XXX': [
                                    {
                                        'XXX': {
                                            'XXX': {
                                                'XXX': 'XXX',
                                                'XXX': 'XXX'
                                            }
                                        }
                                    },
                                    {
                                        'XXX': {
                                            'XXX': {
                                                'XXX': 'XXX',
                                                'XXX': 'XXX'
                                            }
                                        }
                                    },
                                ]
                            },
                            'XXX': {
                                'XXX': [
                                    {
                                        'XXX': {
                                            'XXX': {
                                                'XXX': 2,
                                                'XXX': [
                                                    {
                                                        'XXX': 'XXX',
                                                        'XXX': {
                                                            'XXX': 'XXX',
                                                            'XXX': 'XXX'
                                                        }
                                                    },
                                                    {
                                                        'XXX': 'XXX',
                                                        'XXX': {
                                                            'XXX': 'XXX',
                                                            'XXX': 'XXX'
                                                        }
                                                    }
                                                ]
                                            },
                                            'XXX': {
                                                'XXX': 'XXX',
                                                'XXX': 'XXX',
                                                'XXX': [
                                                    {
                                                        'XXX': 'XXX',
                                                        'XXX': 'XXX',
                                                    }
                                                ]
                                            }
                                        }
                                    },
                                ]
                            }
                        },
                        {
                            'XXX': 'XXX',
                            'XXX': 'XXX',
                            'XXX': None,
                            'XXX': {
                                'XXX': [
                                    {
                                        'XXX': {
                                            'XXX': {
                                                'XXX': 2,
                                                'XXX': [
                                                    {
                                                        'XXX': 'XXX',
                                                        'XXX': {
                                                            'XXX': 'XXX',
                                                            'XXX': 'XXX'
                                                        }
                                                    },
                                                    {
                                                        'XXX': 'XXX',
                                                        'XXX': {
                                                            'XXX': 'XXX',
                                                            'XXX': 'XXX'
                                                        }
                                                    }
                                                ]
                                            },
                                            'XXX': {
                                                'XXX': 'XXX',
                                                'XXX': 'XXX',
                                                'XXX': [
                                                    {
                                                        'XXX': 'XXX',
                                                        'XXX': 'XXX',
                                                    }
                                                ]
                                            }
                                        }
                                    },
                                ]
                            }
                        },
                        {
                            'XXX': 'XXX',
                            'XXX': 'XXX',
                            'XXX': None,
                            'XXX': {
                                'XXX': [
                                    {
                                        'XXX': {
                                            'XXX': {
                                                'XXX': 2,
                                                'XXX': [
                                                    {
                                                        'XXX': 'XXX',
                                                        'XXX': {
                                                            'XXX': 'XXX',
                                                            'XXX': 'XXX'
                                                        }
                                                    },
                                                    {
                                                        'XXX': 'XXX',
                                                        'XXX': {
                                                            'XXX': 'XXX',
                                                            'XXX': 'XXX'
                                                        }
                                                    }
                                                ]
                                            },
                                            'XXX': {
                                                'XXX': 'XXX',
                                                'XXX': 'XXX',
                                                'XXX': [
                                                    {
                                                        'XXX': 'XXX',
                                                        'XXX': 'XXX',
                                                    }
                                                ]
                                            }
                                        }
                                    },
                                ]
                            }
                        },
                        {
                            'XXX': 'XXX',
                            'XXX': 'XXX',
                            'XXX': None,
                            'XXX': {
                                'XXX': [
                                    {
                                        'XXX': {
                                            'XXX': {
                                                'XXX': 2,
                                                'XXX': [
                                                    {
                                                        'XXX': 'XXX',
                                                        'XXX': {
                                                            'XXX': 'XXX',
                                                            'XXX': 'XXX'
                                                        }
                                                    },
                                                    {
                                                        'XXX': 'XXX',
                                                        'XXX': {
                                                            'XXX': 'XXX',
                                                            'XXX': 'XXX'
                                                        }
                                                    }
                                                ]
                                            },
                                            'XXX': {
                                                'XXX': 'XXX',
                                                'XXX': 'XXX',
                                                'XXX': [
                                                    {
                                                        'XXX': 'XXX',
                                                        'XXX': 'XXX',
                                                    }
                                                ]
                                            }
                                        }
                                    },
                                ]
                            }
                        },
                        {
                            'XXX': 'XXX',
                            'XXX': 'XXX',
                            'XXX': None,
                            'XXX': None
                        }
                    ]
                },
                'XXX': {
                    'XXX': 'XXX',
                    'XXX': 'XXX',
                    'XXX': [
                        {
                            'XXX': 'XXX',
                            'XXX': 'XXX'
                        },
                        {
                            'XXX': 'XXX',
                            'XXX': 'XXX'
                        },
                        {
                            'XXX': 'XXX',
                            'XXX': 'XXX'
                        },
                        {
                            'XXX': 'XXX',
                            'XXX': 'XXX'
                        },
                        {
                            'XXX': 'XXX',
                            'XXX': 'XXX'
                        },
                        {
                            'XXX': 'XXX',
                            'XXX': 'XXX'
                        },
                        {
                            'XXX': 'XXX',
                            'XXX': 'XXX'
                        },
                        {
                            'XXX': 'XXX',
                            'XXX': 'XXX'
                        },
                        {
                            'XXX': 'XXX',
                            'XXX': 'XXX'
                        }
                    ]
                }
            }
        }


reveal_type(Foo.field)

Results in:

test-san.py:286: note: Revealed type is "builtins.dict[builtins.str, builtins.dict[builtins.str, typing.Collection[builtins.str]]]"
Success: no issues found in 1 source file

real	0m1.238s
user	0m1.140s
sys	0m0.089s

But changing the code to:

from typing import Any, Dict

class Foo(object):
    def __init__(self) -> None:
        self.field: Dict[Any, Any] = NotImplemented

    def reset(self) -> None:
        self.field = {
            # same contents as above...
        }

changes the runtime quite significantly:

test-san.py:289: note: Revealed type is "builtins.dict[Any, Any]"
Success: no issues found in 1 source file

real	0m10.080s
user	0m9.790s
sys	0m0.256s

and a further code change as follows:

from typing import Any, Dict

class Foo(object):
    def __init__(self) -> None:
        self.field = NotImplemented

    def reset(self) -> None:
        self.field = {
            # same contents as above...
        }

leads to another noticeable perf hit:

$ time mypy test-san.py
test-san.py:9: error: Incompatible types in assignment (expression has type "Dict[str, Dict[str, Collection[str]]]", variable has type "_NotImplementedType")  [assignment]
test-san.py:289: note: Revealed type is "builtins._NotImplementedType"
Found 1 error in 1 file (checked 1 source file)

real	0m22.067s
user	0m21.666s
sys	0m0.319s

I haven't had time to profile this yet but I suspect it is related to #9477 and #12707 and that somehow one version of the code hits the optimization and another doesn't

Your Environment

  • Mypy version used: master as of 695ea3017fee084c9d2ec17d9b28f8af905e3b63, mypyc-compiled
  • Mypy command-line flags: N/A
  • Mypy configuration options from mypy.ini (and other config files): N/A
  • Python version used: 3.7.8
  • OS: macOS 12.5.1 (x86_64)
@huguesb huguesb added the bug mypy got something wrong label Dec 10, 2022
@huguesb huguesb linked a pull request Dec 10, 2022 that will close this issue
Open
@huguesb
Copy link
Contributor Author

huguesb commented Dec 10, 2022

Confirmed via profiling that the above-linked optimizations do not currently trigger in the 2nd and 3rd version of the code as the instance variable being typed means the expression is evaluated with a type context, which currently inhibits the fast path. #14274 broadens the applicability of the fast path and resolves this particular issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug mypy got something wrong performance
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

checkexpr: broader fast path for literal container expressions huguesb/mypy
2 participants
@huguesb @AlexWaygood