Migrate to the next version of Python requests when released

[potiuk]

Hello Maintainers,

I am a PMC member of Apache Airflow, and I wanted to give you a bit of heads-up with rather important migration to the upcoming version of requests library in your Python release.

Since you are using requests library in your project, you are affected.

As discussed at length in https://issues.apache.org/jira/browse/LEGAL-572 we found out that the chardet library used by requests library was a mandatory dependency to requests and since it has LGPL licence, we should not release any Apache Software with it.

Since then (and since in Airflow we rely on requests heavily) we have been working with the requests maintainers and "charset-normalizer" maintainer to make it possible to replace chardet with MIT-licensed charset-normalizer instead so that requests library can be used in Python releases by Apache projects.

This was a bumpy road but finally the PR by @ashb has been merged: psf/requests#5797 and we hope soon a new version of requests library will be released.

This is just a heads-up. I will let you know when it is released, but I have a kind requests as well - I might ask the maintainers to release a release candidate of requests and maybe you could help to test it before it is released, that would be some re-assurance for the maintainers of requests who are very concerned about stability of their releases.

Let me know if you need any more information and whether you would like to help in testing the candidate when it is out.

[potiuk]

Requests 2.26.0 released without the mandatory chardet dependency:

https://pypi.org/project/requests/2.26.0/

Instead of chardet, use the MIT-licensed charset_normalizer for Python3 to remove license ambiguity for projects bundling requests. If chardet is already installed on your machine it will be used instead of charset_normalizer to keep backwards compatibility. (#5797)

You can also install chardet while installing requests by specifying [use_chardet_on_py3] extra as follows:

shell pip install "requests[use_chardet_on_py3]"

Python2 still depends upon the chardet module.

[Kami]

Sorry for the delay and thank for notifying us about this issue / change.

I will look into updating setup.py metadata so we require requests>=2.26.0 and including that change in the next release (I will bump a minor version to be on the safe side).

We don't support Python 2 anymore so hopefully there won't be any problems, but I will also go over requests changelog to make sure we avoid any possible surprises.

[Kami]

Looks like latest version if requests also dropped support for Python 3.5 which means this change will be more involved.

For now, I will add a conditional to setup.py to use older version of requests under Python 3.5, but in the future we should probably also consider dropping support for Python 3.5 to simplify things.

[Kami]

Since Python 3.5 is EOL for more than a year now (https://endoflife.date/python), I think it's reasonable for us to drop support for it in the release after the next one.

Next one will be 3.4.0 and should go out this year and the one after that will be 3.5.0 and will likely go out some time next year.

What do other people think? /cc @c-w @vdloo @tonybaloney @micafer @RunOrVeith

EDIT: Per package download stats (https://pypistats.org/packages/apache-libcloud), Python 3.5, on average, represents less than 2% of daily downloads.

[tonybaloney]

Since Python 3.5 is EOL for more than a year now (https://endoflife.date/python), I think it's reasonable for us to drop support for it in the release after the next one.

Next one will be 3.4.0 and should go out this year and the one after that will be 3.5.0 and will likely go out some time next year.

What do other people think? /cc @c-w @vdloo @tonybaloney @micafer @RunOrVeith

100%

[vdloo]

sounds good, people can pin the 3.4.0 release if they still need to run on python 3.5 👍

[micafer]

I fully agree.

[RunOrVeith]

I also agree

[Kami]

#1619 should mostly address this issue (will be fully addressed when dropping support for Python 3.5).

It's also worth noting that we don't have a fixed version specified in setup.py so in most cases, latest version of requests will already be installed when installing Libcloud.

[Kami]

A lot of the libraries and tooling has already dropped support for Python 3.5 a while back so my recent PRs and changes have been quite a pain since I needed to implement a bunch of workarounds to get it to work under Python 3.5.

So the sooner we drop support for it, the better.

[Kami]

@potiuk and others - Just a heads up - I decided to remove that requests pinning change all together for now - edfb4c5.

The reasoning behind it is here - #1627 (comment).

In short, we are a library and not application and strict version pinning like that has a big chance to brake many user installations due to conflicting dependency versions (e.g. user app where they rely on Libcloud among many other libraries relies on requests <= 2.26.0, but we require >= 2.25.0)

In fact, it's already braking some places where we rely on Libcloud, but can't use newer version of requests - aka requests is pinned to an older version which is not compatible with latest version Libcloud version due to that strict version pinning.

And per my commit message, I think that should also be fine from licensing perspective since we don't bundle requests directly and in most cases when installing libcloud, latest version of requests will be correctly picked and installed (unless user already has request pinned to an older version).

[justinmclean]

It not fine from an ASF Policy point of view you can not have a dependancy that has a Category X license.

[Kami]

@justinmclean I read the ASF policy (https://www.apache.org/legal/resolved.html) and some of the related JIRA issues (https://issues.apache.org/jira/browse/LEGAL-572, https://issues.apache.org/jira/browse/LEGAL-220) and it doesn't appear that the answer is that straight and clear cut in this scenario (or maybe my interpretation is wrong or I missed some ticket). I haven't really found much prior art either.

There are a couple of things at play here:

1. We don't bundle any dependency directly with the library. Dependencies are installed when installing library from PyPi using pip on install time.
2. Library with a problematic dependency in this case is a transitive dependency of requests on which Libcloud relies on and not a direct dependency.

I'm of course all for being on the "safe side as much as possible", but sadly the issue is more complicated, because as mentioned above, Libcloud is a library and not an application. Libcloud is usually used as one of the many build blocks among many other libraries to build different application on top and similar.

This means we should use as loose requirements as possible and not pin dependencies to specific versions since this will prevent many users from using the library if some other dependency in their stack depends on an older or a conflicting version (which is very likely the case with requests since 2.26.0 was released just recently and I'm personally aware of multiple situations like that myself).

Apache Airflow is a bit different and doesn't have the same problem since they are an application and not a library. And applications are usually encouraged (and it's also a best practice for reproducibility, secure and other reasons) to pin all the dependencies to specific versions.

In addition to that, we still support Python 3.5 and requests 2.2.26 only supports Python >= 3.6.

[potiuk]

See my comment here: https://issues.apache.org/jira/browse/LEGAL-572?focusedCommentId=17442504&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17442504 - I quite agree with @Kami

[potiuk]

And just to add to that - Airlfow is both a library AND application :). Which makes us take even a bit different approach (we have as open dependencies as we can afford, but we also have constraint files to allow installation to be reproducible: https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#constraints-files

We added the "requests" limitation, but I definitely agree library such as libcloud should not do it unless they are really broken by some library versions.

[Kami]

@potiuk Thanks for your input and additional context (and of course many thanks for bringing this issue to our and other projects attention and coordinating with the requests project, etc.)

@justinmclean Also thanks for your input. With all the info above, we would like to get some additional clarify on how to proceed here.

We will of course comply with any legal requirements, but we need to know how to proceed so we can plan and execute accordingly.

If there is no other way, but to update our setup.py metadata file but to require requests >= 2.26.0 (which will likely cause issues with a lot of our users and they won't be able to upgrade), we will also need to drop support for Python 3.5 since that version is only available for Python >= 3.6 (that's something we have been planning to do in the future anyway, but not just yet).