Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flatten handlers #70

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Flatten handlers #70

wants to merge 1 commit into from

Conversation

jayvdb
Copy link
Member

@jayvdb jayvdb commented May 7, 2016

pyflakes has traditionally recursed with a handler for every
level of the ast. The ast depth can become very large, especially
for an expression containing many binary operators.

Python has a maximum recursion limit, defaulting to a low number
like 1000, which resulted in a RuntimeError for the ast of:

x = 1 + 2 + 3 + ... + 1001

This change avoids recursing for nodes that do not have a specific
handler.

Checker.nodeDepth and node.depth change from always being the
ast depth, which varied based on Python version due to ast differences,
to being the number of nested handlers within pyflakes.

@jayvdb
Copy link
Member Author

jayvdb commented May 7, 2016

This is just a WIP; while passes all tests, it is very likely that more tests are needed, and some of the handlers need to be switched back to being handleChildren instead of handleChildrenFlattened.

@jayvdb jayvdb force-pushed the flatten branch 4 times, most recently from e3b323c to ea032f4 Compare May 7, 2016 17:55
@jayvdb
Copy link
Member Author

jayvdb commented May 7, 2016

Not really a WIP any more. tests now exist for every case which didnt fail when they should have before. I've run it against against ~300 of the largest OSS Python codebases, and it has no differences except ordering of errors within the same line.

A slightly older version of this change had a minor perf hit, of 3-5 mins increase for what was a 55 min task of processing all of those ~300 OSS codebases . I expect that perf hit has been reduced slightly with this version, and I am rerunning that task now, but this isnt properly isolated perf testing so if this is a serious concern we'll need to do proper perf testing.

@jayvdb
Copy link
Member Author

jayvdb commented May 8, 2016

master+#61+#68:
real 52m59.855s
user 51m13.372s
sys 0m26.637s

This patch+#61:
real 54m25.811s
user 52m3.561s
sys 0m28.246s

@jayvdb
Copy link
Member Author

jayvdb commented Sep 15, 2016

@sigmavirus24, it looks like an infrastructure failure on Appveyor, due to BitBucket. I've successfully run the build twice on Appveyor. Could you restart it, to see if it is working again now on your account.

Also a review would be lovely, especially since you looked at this general problem a while back.
https://bugs.launchpad.net/pyflakes/+bug/1507827

Copy link
Member

@sigmavirus24 sigmavirus24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't tested this yet but here's a first pass at a review.

if handler is False:
return

if not handler:
handler = self.getNodeHandler(node.__class__)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You've moved this outside of the try. I suspect it was in there for a reason.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You will need to do better than suspect. :P

DELETE = PRINT = EXEC = EXPR = RAISE = handleChildrenFlattened
ASSIGN = TRYFINALLY = handleChildren
FOR = ASYNCFOR = WHILE = IF = WITH = ASYNCWITH = handleChildren
WITHITEM = ASYNCWITHITEM = handleChildrenFlattened
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not put all of the nodes using handleChildrenFlattened together?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have grouped them by type of node in a readible manner, as best I could with this subset of all nodes

I felt the "with-item" nodes should go after the "with" nodes and other branching nodes.

if handler and handler not in _may_flatten:
yield node, parent, handler
else:
nodes[:] += ((child, node)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the [:] do here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pyflakes has traditionally recursed with a handler for every
level of the ast.  The ast depth can become very large, especially
for an expression containing many binary operators.

Python has a maximum recursion limit, defaulting to a low number
like 1000, which resulted in a RuntimeError for the ast of:

    x = 1 + 2 + 3 + ... + 1001

This change avoids recursing for nodes that do not have a specific
handler.

Checker.nodeDepth and node.depth changes from always being the
ast depth, which varied between Python version due to ast differences,
to being the number of nested handlers within pyflakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants