Upgrade all code to be Python3 only #5065

humitos · 2019-01-03T16:11:29Z

This PR drops Python 2 support and upgrade the code to be Python3 only:

This PR touches a lot of files. We should probably want to define what's the process we want to follow here but I wanted to have the PR done so we can see a diff and list all the things we need to do here.

pyupgrade has a way to be used as a pre-commit command so it will have effect only over the modified files in the PR. We are doing this already with yapf and we are still not covering all the code and complicating the review process. Although, we should consider this and discuss what's the path we want to follow.

IMPORTANT NOTE: this is huge PR that touches so many files. So, it's hard to review taking a look at the diff as a whole. You can click on the first commit and review one by one by clicking on "Next" excluding the ones that are script passes:

pyupgrade 1.11.0 on all code
isort all the files
Run pre-commit on all the files

Closes #4543

pyupgrade --py3-plus --py3-only (*.py)

This file is generated by our code and can be used by Python2 when building the documentation, so we need to keep the `u` before the definition of the string.

stsewd

Just one test got changed incorrectly, but the rest looks good. Not sure why the search tests are failing

stsewd · 2019-01-06T04:56:38Z

readthedocs/config/validation.py



 def validate_list(value):
    """Check if ``value`` is an iterable."""
-    if isinstance(value, (dict, string_types)):
+    if isinstance(value, (dict, (str,))):


This should be (dict, str)

Yeah, I suppose it was a mistake in the original code, and it should have been string_type (in singular).

stsewd · 2019-01-06T05:22:55Z

Maybe we have a dependency in the search code that only runs on python2?

humitos · 2019-01-07T09:39:12Z

Maybe we have a dependency in the search code that only runs on python2?

I don't think so, otherwise the search wouldn't be working: we are running python3 in our production servers.

humitos · 2019-01-08T19:20:50Z

pyupgrade has a way to be used as a pre-commit command so it will have effect only over the modified files in the PR. We are doing this already with yapf and we are still not covering all the code and complicating the review process. Although, we should consider this and discuss what's the path we want to follow.

We discussed about this and we decided that we want to do it "all at once".

stsewd · 2019-01-08T19:26:23Z

What about doing the autoformat in a separated PR? This was really large to review

humitos · 2019-01-08T19:40:57Z

Yes. I don't expect that nobody review what pyupgrade does, actually. I just checked some changes and I trusted on it.

I will create a PR with all the manual changes that I had to do, branching from this PR.

humitos · 2019-01-08T19:42:19Z

Or... at least, I will make commits that make sense separating what are automatic tools from manually touching the files.

$ pre-commit run python-import-sorter --files `find readthedocs -name '*.py'` It was ran with this configuration: readthedocs/common#28

humitos · 2019-01-09T12:31:12Z

This is ready for review: all tests and linting are passing.

  py36: commands succeeded
  lint: commands succeeded
  docs: commands succeeded
  congratulations :)

NOTE: py36 is randomly failing on travis (re-triggering without cache now)

We still have a problem with pre-commit because isort and yapf competes for the order of imports but also for the number of blank lines. As we prefer the behavior of isort we ran it after yapf when pre-commit is called without hooks ids.

In the end, running pre-commit multiple times at this point should not produce different outputs.

NOTE: I disabled add-trailling-comma because I got tired of the inconsistency that it produces 😁

I would be happy if we merge this sooner than later because otherwise we will be dealing with many of merge conflicts with the other PRs.

humitos · 2019-01-09T12:51:52Z

Another note. I just tried black instead of yapf and I don't really like some decisions that it takes about the format of the code which, to me, it produces code that's visually harder than read that the rules that we have written in yapf.

Also, I found some of these in our entire repo:

INTERNAL ERROR: Black produced code that is not equivalent to the source.  Please report a bug on https://github.com/ambv/black/issues.  This diff might be helpful: /tmp/blk_dp6tr05z.log

…/deprecate-python2

Removes `from __future__ import ...`

humitos · 2019-01-21T18:45:07Z

All checks are passing locally. Although, Travis is doing something wrong with search tests and they are failing on connection :/

ericholscher

Let's do it. gulp

humitos · 2019-01-21T19:12:55Z

🙏

humitos added 7 commits January 3, 2019 16:42

Do not run py27 on tox

23b20cd

Do not run py27 on Travis

678027c

pyupgrade 1.11.0 on all code

f7dea72

pyupgrade --py3-plus --py3-only (*.py)

Use unicode string for Sphinx config file

f707466

This file is generated by our code and can be used by Python2 when building the documentation, so we need to keep the `u` before the definition of the string.

Remove unused imports to avoid lint issues

1b58939

Run tests with python3.6 on Travis

271884a

Do not include search tests when running migrations env

849abb3

humitos added this to the 3.0 milestone Jan 3, 2019

humitos added the Needed: design decision A core team decision is required label Jan 3, 2019

humitos requested a review from a team January 3, 2019 18:14

stsewd reviewed Jan 6, 2019

View reviewed changes

humitos added 7 commits January 8, 2019 20:56

isort all the files

c78187b

$ pre-commit run python-import-sorter --files `find readthedocs -name '*.py'` It was ran with this configuration: readthedocs/common#28

Remove six and future from requirements

c56b289

Manually remove usage of future module

979f216

Remove skipif

f124be5

Remove usage of from django.utils import six

05f5309

Use ipaddress from standard library

685e547

Remove six leafovers

796d09b

humitos force-pushed the humitos/deprecate-python2 branch from f293a58 to 796d09b Compare January 8, 2019 20:18

Bug fix on types checking

c2a662c

humitos removed the Needed: design decision A core team decision is required label Jan 8, 2019

humitos added 3 commits January 8, 2019 21:22

Merge branch 'master' into humitos/deprecate-python2

bc3afc7

Import missing modules removed by mistake

686c820

Keep __future__ in conf.py because we build with Py2 also

25eb8c4

humitos added 2 commits January 9, 2019 13:08

Upgdate common submodule

efa731f

pre-commit all the files again

632e18c

humitos requested a review from a team January 9, 2019 12:32

humitos added the Accepted Accepted issue on our roadmap label Jan 14, 2019

humitos self-assigned this Jan 15, 2019

humitos mentioned this pull request Jan 20, 2019

Issues with current pre-commit readthedocs/common#14

Closed

humitos added 14 commits January 21, 2019 18:04

Merge branch 'master' of github.com:rtfd/readthedocs.org into humitos…

d8dacde

…/deprecate-python2

pyupgrade over new code with conflicts

0f7b8d7

isort new files

1ffc03e

Remove future leafovers

e363f6f

isort over migration files

5798557

Removes `from __future__ import ...`

isort on all tests files

55e1a2f

Remove skipif for six.PY2

4aded8b

Update common submodule with the proper PR config

447e3b7

pre-commit all the files

b5bc9c7

Run add-trailing-comma over all the files

7d08eed

Do not touch conf.py for testing

022594f

Last pre-commit pass

91a205e

Lint changes

9f11db5

noqa on one line

ed67c54

ericholscher approved these changes Jan 21, 2019

View reviewed changes

humitos added 2 commits January 21, 2019 20:35

Remove py36 from travis.matrix.include

6092d0e

Move ES env vars to py36 job

f68d84a

humitos merged commit 59d9e3e into master Jan 21, 2019

delete-merged-branch bot deleted the humitos/deprecate-python2 branch January 21, 2019 19:51

humitos mentioned this pull request Jan 23, 2019

Validate webhook's payload #4940

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade all code to be Python3 only #5065

Upgrade all code to be Python3 only #5065

humitos commented Jan 3, 2019 •

edited

Loading

stsewd left a comment

stsewd Jan 6, 2019

humitos Jan 8, 2019

stsewd commented Jan 6, 2019

humitos commented Jan 7, 2019

humitos commented Jan 8, 2019

stsewd commented Jan 8, 2019

humitos commented Jan 8, 2019

humitos commented Jan 8, 2019

humitos commented Jan 9, 2019

humitos commented Jan 9, 2019

humitos commented Jan 21, 2019

ericholscher left a comment

humitos commented Jan 21, 2019

Upgrade all code to be Python3 only #5065

Upgrade all code to be Python3 only #5065

Conversation

humitos commented Jan 3, 2019 • edited Loading

stsewd left a comment

Choose a reason for hiding this comment

stsewd Jan 6, 2019

Choose a reason for hiding this comment

humitos Jan 8, 2019

Choose a reason for hiding this comment

stsewd commented Jan 6, 2019

humitos commented Jan 7, 2019

humitos commented Jan 8, 2019

stsewd commented Jan 8, 2019

humitos commented Jan 8, 2019

humitos commented Jan 8, 2019

humitos commented Jan 9, 2019

humitos commented Jan 9, 2019

humitos commented Jan 21, 2019

ericholscher left a comment

Choose a reason for hiding this comment

humitos commented Jan 21, 2019

humitos commented Jan 3, 2019 •

edited

Loading