DRAFT: Apply pyupgrade.

[Carreau]

• Merge Add conversion to f-strings using flynt #308
• Rebase after Add conversion to f-strings using flynt #308 has been merged
• Evaluate whether the approach in Add conversion to f-strings using flynt #308 fits directly or needs to be adjusted
• Unit tests
• Documentation

---

(pyupgrade is "a tool (and pre-commit hook) to automatically upgrade syntax for newer versions of the language.")

Just seeing how it looks like, we would of course need to pass options in, so i'm thinking only once there is a config file; likely.

This is likely incorrect as the edited lines could likely be off once Black has been ran as it can rewrap stuff.

It also does use pyupgrade private internal API, and the author has explicitly stated they they were not willing to have public API beyond the CLI.

We should also likely run this before Black; maybe even before isort? Not sure if it might remove imports... But at least it does not pass the AST unchange part.

---

Late for today, just submitting in case you decide to also write the same thing.

[akaihola]

Oh wow, another splendid idea! But I agree with your notes regarding options, config file first, and the AST check challenge.

Also, I'm not sure if you meant this by "pass options in", but I think this should be off by default and activated using a command line / configuration file option.

[akaihola]

I agree with your note about pyupgrade changes being better done before Black reformatting, maybe even as the first thing before isort.

[akaihola]

edited_linenums isn't necessarily correct any more at this point. If Black has e.g. divided a long line on multiple lines, every line number after that gets shifted. This would cause wrong pyupgrade changes to be selected and desired ones skipped. This is why changes by all tools (pyupgrade, isort, Black) need to be done first, and only then the diff be calculated between the user's original version and the automatically modified one.

[Carreau]

Yes, that what I was thinking, or recomputing the diff WRT the revision after each tool.

I think the problem with (all tools) then (compute diff), is you may end up with chunks larger than you wants.

say original

a = set((1, 2))
b    =     c

user edit

- b    =     c
+ b    =     d

(black + pyupgrade) then (compute diff), will change both lines, which you don't want to apply.
Though,
(black, diff) should be applied, then (pyupgrade, diff) should not.

[akaihola]

There's also the additional potential problem of tools which may change the order of lines in code. isort is an example of such a tool, and that's why Darker just includes modifications from it as part of the starting point, and compares Black results to that. I wonder if any of the rules in pyupgrade could potentially also change the order of lines in the source code?

Here's an illustration of what code reordering would cause if such changes would be attempted to apply to only modified lines:

Let's say I have this module committed to the HEAD of my repository:

1  import  os
2  import black
3  import sys

I then manually correct the first line to import os without the extra space:

*1  import os
 2  import black
 3  import sys

Run isort:

isort myfile.py

Get this as a result:

 1  import os
*2  import sys
*3
*4  import black

Git shows the difference:

git diff -U0

git diff -U0|cat
diff --git a/imp.py b/imp.py
index 61a152d..1ed0c5e 100644
--- a/imp.py
+++ b/imp.py
@@ -1,2 +1 @@
-import  os
-import black
+import os
@@ -3,0 +3,2 @@ import sys
+
+import black

Only the first chunk falls on line 1 (the only edited line) and would be applied by a Darker-like tool. The result would be:

1  import os
2  import sys

So I would actually lose my Black import as a result of applying sorted imports partially.

[Carreau]

from my look at upgrade that does nto seem to be the case, it really appears to do AST node replacements, so I would not be too worried.

[Carreau]

Oh wow, another splendid idea! But I agree with your notes regarding options, config file first, and the AST check challenge.

Thanks !

Also, I'm not sure if you meant this by "pass options in", but I think this should be off by default and activated using a command line / configuration file option.

Yes, off by default would be my preference I think. And the you can opt-in on per-repo config files.
I was also expecting the work on linters you did in the other PR to refactor this area so did not push much on the impl.

[akaihola]

@Carreau do you still think this patch would fit Darker's purpose? If so, we probably need to try and rebase on master – I can help there if the conflicts are complicated.

[akaihola]

@Carreau, I rebased this on master and removed the actual call to the pyup() function since it needs to be thought through anyway. I only now looked more closely to what you hade already implemented, and I believe we can make this work. Let's see if this happens by version 1.5.0 or only in 1.6.0 of Darker.

[akaihola]

Support for flynt is cooking in #308. The same approach might fit well for pyupgrade. Let's rebase this after that one has been merged.

[akaihola]

Flynt support in #308 has been merged. We can now resume implementing pyupgrade support.

[akaihola]

Moving to version 1.8.0. There are a plenty of changes for 1.7.0 already, I now prefer to get it out first.