-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix incorrect linenos on fstring tokens with escaped newlines #4423
Conversation
I don't think this can affect Black itself much (maybe for formatting ranges), but I ran into this with https://github.com/JelleZijlstra/lib2toast
for more information, see https://pre-commit.ci
if __name__ == "__main__": | ||
code = sys.stdin.read() | ||
tokens = get_tokens(code) | ||
text = f"assert_tokenizes({code!r}, {tokens!r})" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
consider using the builtin tokenize
module for generating tests? that way you don't have to verify the locations before adding the test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure how compatible exactly are the tokens between Python and blib2to3 though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
They don't match exactly in all cases (e.g., builtin has an extra encoding token at the beginning). We can try to work towards exact compatibility, but not sure that's necessary.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, for now this is good
I don't think this can affect Black itself much (maybe for formatting
ranges), but I ran into this with https://github.com/JelleZijlstra/lib2toast
cc @tusharsadhwani