Port to Python3

[BryanQuigley]

Python(2) is only supported until 2020, maybe a good time to make a plan to port to python3. I'm not sure if it's important to support both 2 and 3 at once. I noticed because boinc-client depends on python2 in Ubuntu 18.04 (and python2 is no longer installed by default)

My initial check indicates python is used quite a bit.
py/*
samples/vm_wrapper/
sched/
test/
tools/

Any obvious places to start porting?

[SETIguy]

I typically start by trying to alter the brain wiring of people who think that backwards compatibility is unnecessary, but in this case it seems too late for that.

…

On Thu, Apr 19, 2018 at 8:29 AM, BryanQuigley ***@***.***> wrote: Python(2) is only supported until 2020, maybe a good time to make a plan to port to python3. I'm not sure if it's important to support both 2 and 3 at once. I noticed because boinc-client depends on python2 in Ubuntu 18.04 (and python2 is no longer installed by default) My initial check indicates python is used quite a bit. py/* samples/vm_wrapper/ sched/ test/ tools/ Any obvious places to start porting? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2475>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKXcsoPBtx-YiCzHGX5VOhOudSKzkjQeks5tqK1KgaJpZM4TcAy0> .

-- Eric Korpela korpela@ssl.berkeley.edu AST:7731^29u18e3

[davidpanderson]

What parts of the Python code don't work with Python 3?

…

On Thu, Apr 19, 2018 at 10:44 AM, SETIguy ***@***.***> wrote: I typically start by trying to alter the brain wiring of people who think that backwards compatibility is unnecessary, but in this case it seems too late for that. On Thu, Apr 19, 2018 at 8:29 AM, BryanQuigley ***@***.***> wrote: > Python(2) is only supported until 2020, maybe a good time to make a plan > to port to python3. I'm not sure if it's important to support both 2 and 3 > at once. I noticed because boinc-client depends on python2 in Ubuntu 18.04 > (and python2 is no longer installed by default) > > My initial check indicates python is used quite a bit. > py/* > samples/vm_wrapper/ > sched/ > test/ > tools/ > > Any obvious places to start porting? > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#2475>, or mute the thread > <https://github.com/notifications/unsubscribe-auth/AKXcsoPBtx- YiCzHGX5VOhOudSKzkjQeks5tqK1KgaJpZM4TcAy0> > . > -- Eric Korpela ***@***.*** AST:7731^29u18e3 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2475 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA8KgZK_l9JrkPJa-piIPDocEo4h3HBAks5tqM0RgaJpZM4TcAy0> .

[BryanQuigley]

For starters, print statements are now functions.. (I only see the old style print statements), so at least the code I checked in py/* and test/* would fail to run at all. (There are many other changes I haven't checked)

It appears that the python2 dependency in boinc-client may be a Debian specific modification.. for /usr/bin/update-boinc-applinks. Investigating.. as I don't see any others in boinc-client upstream..

[Ferroin]

The print statements becoming functions can be handled in a backwards compatible manner (supporting Python versions 2.6 and newer) by just converting all the print statements to functions and adding this:

from __future__ import print_function

to the top of each file (before any other python statements, but it can be after the docstring).

[BryanQuigley]

Agreed, almost all of the changes can be done in a backward compatible way. The key thing is if it's worth that extra work for specific scripts.

[adamradocz]

What can we lose if we drop the python 2? Is it worth the effort making the script?
Do you know any project which relies on python2?

[ChristianBeer]

The problem is with projects that use a Linux distribution that has no python3 or it is difficult to install python3. For example CentOS 6/7 which comes with python2 and installing python3 is a bit difficult and you can't set python3 as the system default. So we sould need to retain python2 support.

Please note that PR #2477 was opened to address this.

[SETIguy]

There are also a significant number of packages that are no longer actively developed that may be in use at projects that may not have been ported to Python3. InformixDB is one example that we used to used in SETI@home web code. Of course we've only got 2 years of python2 support left, so eventually even the RedHat derivatives will need to move to Python3. There are other things that have changed. If you depend on a global variable being changed by being used as a loop counter, that'll break. Printing explicit tuples gets broken by the print is now a function behavior. Strings are now all unicode, so anything that depended on them being ASCII breaks. The syntax of exceptions changed. Default behavior of integer division changed to floating point in python3. To get integer division you use the // operator, which returns an integer value (in a floating point quantity). There are also a lot of things that look like valid code that don't work in python3 for reasons I typically don't have the time to identify.

…

On Sat, Apr 21, 2018 at 2:42 AM, Christian Beer ***@***.***> wrote: The problem is with projects that use a Linux distribution that has no python3 or it is difficult to install python3. For example CentOS 6/7 which comes with python2 and installing python3 is a bit difficult and you can't set python3 as the system default. So we sould need to retain python2 support. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2475 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKXcstTtwNw9K7-HgmT6W6SV8F_4ZNq-ks5tqv7pgaJpZM4TcAy0> .

-- Eric Korpela korpela@ssl.berkeley.edu AST:7731^29u18e3

[marius311]

Just in case people aren't aware of it, but quite a lot of the things mentioned in this thread are automatically fixed by the "2to3" tool. I've found that running that then fixing a few remaining things by hand is usually a pretty painless path to upgrading. There's also the "six" package with aids in keeping 2/3 compatibility (which we definitely need to do).

Anyway, I very much support getting python3 working sooner rather than later, and am happy to offer whatever input I can.

[Ferroin]

The biggest thing I can see that would be lost is support for certain old versions of RHEL and CentOS. I don't remember the exact versions where they finally included Python 3, but I do know at least a couple of other software projects have delayed full Python 3 support to keep support for these older platforms (see for example https://github.com/ansible/ansible, which is just now seeing the beginnings of proper Python 3 support).

[Ferroin]

Also, do keep in mind that support past a certain point for older CentOS versions means having to deal with Python 2.6, and getting code to work properly there and on Python 3 gets rather complicated fast.

[SETIguy]

Please keep in mind that these are not old versions of RHEL, Scientific Linux, Centos, etc. They (EL6 and EL7) are currently supported versions and they will be supported for years to come. That's why they have Enterprise in the name. RHEL 7 is the latest release of Red Hat Enterprise Linux. It uses Python 2.7.5. RHEL 6 uses Python 2.6.6. Because Python does not believe in backward compatibility (even between minor versions), and because the Red Hat package interface (yum) uses python, it's not even possible to upgrade from 2.6.X to 2.7.X without reinstalling the OS. RHEL 8, when it arrives, will probably support Python 3. But RHEL 8 isn't available yet.

…

On Mon, Apr 23, 2018 at 5:13 AM, Austin S. Hemmelgarn < ***@***.***> wrote: The biggest thing I can see that would be lost is support for certain old versions of RHEL and CentOS. I don't remember the exact versions where they finally included Python 3, but I do know at least a couple of other software projects have delayed full Python 3 support to keep support for these older platforms (see for example https://github.com/ansible/ansible, which is just now seeing the beginnings of proper Python 3 support). — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2475 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKXcsnTcq77MrBDEjKVk8blWmAAJ3Tqxks5trcVcgaJpZM4TcAy0> .

-- Eric Korpela korpela@ssl.berkeley.edu AST:7731^29u18e3

[Ferroin]

RHEL 6 is approximately 7 years old. That's old, period, regardless of whether or not it's an enterprise release. It's in the ELS support phase on top of that, which is their way of saying you need to upgrade (and that you should not be deploying new systems running it). Personally, I also consider RHEL 7 old despite it still being in the first production phase, but I'm also the type who hates the standard enterprise practice of only handling API/ABI breaks all at once every half decade.

I'm absolutely serious though that it is extremely complicated to support both Python 2.6 and Python 3.x with the same codebase once you get past truly trivial scripts. There are enough differences in the core language that it pretty severely restricts what you can actually do, and that's without having to deal with the library differences.

[SETIguy]

Scientific Linux 6 (which is what we use) is still distributed and supported. Any of our SL6 production servers that haven't been replaced will still be running SL6 until that stops. At that point we'll probably go with whatever the latest release of Ubuntu LTS. Any of our SL7 servers will stay at SL7 as long as that is supported. But what we're definitely not going to do is reinstall operating systems to support changes to BOINC.

…

On Tue, Apr 24, 2018 at 4:58 AM, Austin S. Hemmelgarn < ***@***.***> wrote: RHEL 6 is approximately 7 years old. That's old, period, regardless of whether or not it's an enterprise release. It's in the ELS support phase on top of that, which is their way of saying you need to upgrade (and that you should not be deploying new systems running it). Personally, I also consider RHEL 7 old despite it still being in the first production phase, but I'm also the type who hates the standard enterprise practice of only handling API/ABI breaks all at once every half decade. I'm absolutely serious though that it is *extremely* complicated to support both Python 2.6 *and* Python 3.x with the same codebase once you get past truly trivial scripts. There are enough differences in the core language that it pretty severely restricts what you can actually do, and that's without having to deal with the library differences. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2475 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKXcstSjLLfSlXcsG1UMyimahHpxq5Sfks5trxNUgaJpZM4TcAy0> .

-- Eric Korpela korpela@ssl.berkeley.edu AST:7731^29u18e3

[BryanQuigley]

FYI, the boinc client python2 dependency for Debian/Ubuntu was an old script we don't think is necessary anymore - https://bugs.launchpad.net/ubuntu/+source/boinc/+bug/1765576. Sorry for the noise on that..

extremely complicated to support both Python 2.6 and Python 3.x
To add: Python 2.7 and Python 3.x support at the same time is much much easier.. So that really makes SL6 the biggest limitation.

@SETIguy What do the remaining SL6 servers specifically do (or what part of boinc python code is touched)? Any other reasons to upgrade them (performance, other program updates)?

[jessefowers]

I would be willing to help with this. Even with 2to3 its going to take a lot of manual labor. If/when the decision is made to drop support for 2 and port all python code to 3, what would that process entail? Would we have to port all the code in one big patch? What volume of code are we looking at here?

[davidpanderson]

The goal here is to make the code work with both 2 and 3. The code consists of some scripts (tools/make_project, tools/update_versions, tools/upgrade, and sched/start) and the modules they use, which are in py/Boinc. About 4500 lines total.

…

On Fri, Apr 27, 2018 at 7:44 PM, Jesse Fowers ***@***.***> wrote: I would be willing to help with this. Even with 2to3 its going to take a lot of manual labor. If/when the decision is made to drop support for 2 and port all python code to 3, what would that process entail? Would we have to port all the code in one big patch? What volume of code are we looking at here? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2475 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA8KgVTUmeasacNTY3uNWzFOmRYRCM7eks5ts9eQgaJpZM4TcAy0> .

[jessefowers]

The goal here is to make the code work with both 2 and 3.
Straddling 2 and 3 sounds do-able.

python-future might be a good resource. It is both a python module that builds on 2to3 and a great website detailing how to write 2/3 compatible code.

From their website:

python-future is the missing compatibility layer between Python 2 and Python 3. It allows you to use a single, clean Python 3.x-compatible codebase to support both Python 2 and Python 3 with minimal overhead.

[jessefowers]

Right now I am assuming that the existing Python code runs fine on 2.x. If we use only use 2to3 we could break the code against 2.x. What is hopefully achievable is code that runs on both 2 and 3 without any dependence on a compatibility module. If the existing code requires changes that go beyond syntax adjustments to run on 3 then we need to consider adding a dependency on a compatibility module like future, if we want to continue to support 2.x.

I suggest we exhaust all of the 'easy' syntax only adjustments and see how far that gets us.

I have read through the contributors guide and would like to help with this. I have not contributed to this project before so please tell me the preferred way to proceed. I can fork the project and begin work adjusting syntax in the scripts and modules that @davidpanderson listed so that it works with 2 and 3, but it looks like some of that is already started with #2477. Please advise.

[jessefowers]

Hi again - I just wanted to check in on this issue. I would love assist with this issue if their is a real need. Please advise on what you would need from me. Thanks!

[AenBleidd]

Could this be closed in favor of #3241?

CC: @TheAspens

[smoe]

With a bit of luck, my #3259 also closes (most of?) this.

[Delizald]

Did #3259 fixed the Py2 and Py3 compatibility? I can see that it was merged

[AenBleidd]

It would be nice to verify that there are no other incompatibilities

[davidpanderson]

The code itself works with python 2 and 3.
The issue is the MySQL interface, and distutils.
We need python3 versions of these

[AenBleidd]

There are two open issues (mentioned in the comment above): #5428 and #5429.
I see no reason to keep this one open since it's a general ticket, and two above a exact issues.