-
-
Notifications
You must be signed in to change notification settings - Fork 30.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ast.parse()
believes valid context manager py38 syntax to be invalid when feature_version=(3, 8)
is passed
#115881
Comments
ast.parse()
believes valid py38 syntax to be invalid when feature_version=(3, 8)
is passedast.parse()
believes valid context manager py38 syntax to be invalid when feature_version=(3, 8)
is passed
I think there isn't going to be any super clean way to do this because this new construct has a non trivial syntactic overlap with the case reported here. This was precisely what made it impossible to do with the old parser. I think the best way is to revert 0daba82 but maybe I am missing some elegant way to do it so I am open to alternatives |
CC: @lysnikolaou |
Thanks for reporting! I think we can fix by adding another grammar rule. I don't know if that meets the bar for elegant, if not, I agree reverting could be fine. I'm thinking of something like: diff --git a/Grammar/python.gram b/Grammar/python.gram
index 174b4dbb6f..1526b52ec8 100644
--- a/Grammar/python.gram
+++ b/Grammar/python.gram
@@ -392,6 +392,8 @@ for_stmt[stmt_ty]:
with_stmt[stmt_ty]:
| invalid_with_stmt_indent
+ | 'with' '(' e=expression ','? ')' ':' tc=[TYPE_COMMENT] b=block {
+ _PyAST_With(CHECK(asdl_withitem_seq*, _PyPegen_singleton_seq(p, _PyAST_withitem(e, NULL, p->arena))), b, NEW_TYPE_COMMENT(p, tc), EXTRA) }
| 'with' '(' a[asdl_withitem_seq*]=','.with_item+ ','? ')' ':' tc=[TYPE_COMMENT] b=block {
CHECK_VERSION(stmt_ty, 9, "Parenthesized context managers are", _PyAST_With(a, b, NEW_TYPE_COMMENT(p, tc), EXTRA)) }
| 'with' a[asdl_withitem_seq*]=','.with_item+ ':' tc=[TYPE_COMMENT] b=block { By the way, I notice that |
I prefer to revert the commit. The grammar is already quite complicated and we are automatically generating docs from it so I would recommend you try to keep complexity controlled |
Maybe it's because I am reading this in a phone but I cannot see any difference 😅 In any case we were aware of that being a thing and other similar cases but in all of them we were ok with the change since tuples will always fail as context managers 😉 |
We'll want to backport the fix as well, so I agree that simply reverting feels safer for now (and possibly consider a more complex solution for Python 3.13+ only? But I'll defer to those more experienced with hacking on the parser on whether that's worth it or not :) |
Yup, I agree with reverting this. Like I've said a few times before, deprecating |
…ven with low `feature_version` passed (#115920)
…gers even with low `feature_version` passed (python#115920)
Yes, we do use it at mypy! That's how I came across this issue in the first place -- after we started using a new version of black at mypy, I realised mypy could no longer type check its own codebase if you were using |
Got it. Thanks for the context @AlexWaygood! |
The problem with I think at the very least we should document the limitations in the docs, to ensure that the fact that for example context managers with parentheses may not be detected is clear to end users. What do people think? |
…gers even with low `feature_version` passed (python#115920)
The fix for the original issue has been merged and backported, but I'll leave this open so we can continue discussing what to do about the broader issue, and if necessary link more PRs here |
Yup, let's document this at the very least. If we were ever to remove this, we'd have to go through a deprecation, cause it seems to be used in quite a few places. |
Thanks Alex! Yes, I believe it's used by at least mypy, black, pytype. We should document more clearly that I think the functionality is useful, even if limited, so I'd prefer not to deprecate |
Okay, we did the revert, we've updated docs and I did the backports, so I think we can close. Let me know if I missed something! |
…gers even with low `feature_version` passed (python#115920)
…gers even with low `feature_version` passed (python#115920)
…gers even with low `feature_version` passed (python#115920)
Bug report
Bug description:
The following code is completely valid on Python 3.8:
However, following 0daba82 (which was backported to Python 3.11 and Python 3.10),
ast.parse()
incorrectly throws an error if you try to parse this code withfeature_version=(3, 8)
:Cc. @hauntsaninja / @pablogsal
CPython versions tested on:
3.8, 3.9, 3.10, 3.11, 3.12
Operating systems tested on:
macOS
Linked PRs
ast.parse()
parses conditional context managers even with lowfeature_version
passed #115920ast.parse()
parses conditional context managers even with lowfeature_version
passed (#115920) #115959ast.parse()
parses conditional context managers even with lowfeature_version
passed (#115920) #115960The text was updated successfully, but these errors were encountered: