CI: Drop Python 3.5 support #29212

datapythonista · 2019-10-24T19:05:47Z

closes Drop Python 3.5 support #29034
tests added / passed
passes black pandas
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

TomAugspurger

Can you update pyproject.toml and install.rst too?

Grepping for PY35 reveals a few options for cleanup. Happy to do those as followups

pandas/compat/__init__.py:15:PY35 = sys.version_info[:2] == (3, 5)
pandas/io/json/_json.py:1118:            if compat.PY35:
pandas/tests/io/json/test_json_table_schema.py:8:from pandas.compat import PY35
pandas/tests/io/json/test_json_table_schema.py:27:    if PY35:
pandas/tests/io/json/test_pandas.py:10:from pandas.compat import PY35, is_platform_32bit, is_platform_windows
pandas/tests/io/json/test_pandas.py:169:        if not numpy and PY35 and orient in ("index", "columns"):
pandas/tests/io/json/test_pandas.py:183:        if not numpy and PY35 and orient in ("index", "columns"):
pandas/tests/io/json/test_pandas.py:218:        if not numpy and PY35 and orient in ("index", "columns"):
pandas/tests/io/json/test_pandas.py:259:        if not numpy and (orient == "index" or (PY35 and orient == "columns")):
pandas/tests/io/json/test_pandas.py:326:        if not numpy and (orient == "index" or (PY35 and orient == "columns")):
pandas/tests/io/json/test_pandas.py:661:        if not numpy and PY35 and orient in ("index", "columns"):
pandas/tests/io/json/test_pandas.py:679:        if not numpy and PY35 and orient in ("index", "columns"):
pandas/tests/io/json/test_pandas.py:695:        if not numpy and PY35 and orient == "index":
pandas/tests/io/json/test_pandas.py:1619:        if PY35:
pandas/tests/frame/test_arithmetic.py:248:                r"unorderable types: .*complex\(\)",  # PY35
pandas/tests/groupby/aggregate/test_aggregate.py:576:        if compat.PY35:
pandas/tests/groupby/aggregate/test_aggregate.py:624:        if compat.PY35:
pandas/tests/scalar/period/test_period.py:13:from pandas.compat import PY35
pandas/tests/scalar/period/test_period.py:1554:    PY35,

WillAyd · 2019-10-24T19:22:44Z

I suggest leave it to community for follow ups as they are kind of easy to do. I can think of issues to:

Rip out any Py35 compat
Replace OrderedDict with just dict (maybe?)
Convert annotation comments to 36 inline syntax
Use f-strings instead of .format (maybe?)

TomAugspurger · 2019-10-24T19:25:43Z

SGTM.

…

On Thu, Oct 24, 2019 at 2:22 PM William Ayd ***@***.***> wrote: I suggest leave it to community for follow ups as they are kind of easy to do. I can think of issues to: 1. Rip out any Py35 compat 2. Replace OrderedDict with just dict (maybe?) 3. Convert annotation comments to 36 inline syntax — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#29212?email_source=notifications&email_token=AAKAOITYTX4Q6EL7CXGFCG3QQHYYXA5CNFSM4JEYZLM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECGEXQY#issuecomment-546065347>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKAOISFLC7MNDF5IEVCOP3QQHYYXANCNFSM4JEYZLMQ> .

gfyoung · 2019-10-24T19:39:38Z

@datapythonista : For the MacOS build (ci/deps/azure-macos-35.yaml → ci/deps/azure-macos-36.yaml), I think you need to re-specify the Python version to 3.6.*.

datapythonista · 2019-10-24T19:54:20Z

Thanks for the review and the comments @TomAugspurger and @gfyoung , addressed them.

Good point @WillAyd, I included 1 here, but makes sense to create issues for 2 and 3, and not sure if we could create some for f-strings too. We don't want to get crazy with that, but in many cases that can be a nice clean up.

WillAyd · 2019-10-25T00:38:35Z

ci/deps/azure-35-compat.yaml

@@ -1,30 +0,0 @@
-name: pandas-dev


I think we want to recreate this file on 36 to pin to the lowest support versions of everything

Thanks for the info @WillAyd, didn't know that was what we were testing with that build. I restored it, moved to Python 3.6, and renamed it from compat to minimum_versions, so it's more obvious for other people what's the build about.

Cool thanks! I think that should be pinned to python 3.6.0 as well instead of allowing the latest 3.6 release to be picked up

makes sense, updated

…36 not py35

simonjayhawkins · 2019-10-25T17:09:42Z

pandas/tests/groupby/aggregate/test_aggregate.py

@@ -573,8 +573,6 @@ def test_agg_with_one_lambda(self):

        # sort for 35 and earlier


WillAyd

lgtm

WillAyd · 2019-10-25T17:17:42Z

ci/deps/azure-36-minimum_versions.yaml

  - pytest-xdist
  - pytest-mock
  - pytest-azurepipelines
  - pip
  - pip:
-    # for python 3.5, pytest>=4.0.2, cython>=0.29.13 is not available in conda
-    - cython>=0.29.13
-    - pytest==4.5.0
    - html5lib==1.0b2


Can be done as a follow up but not sure why this minimum is pinned to a beta, and I couldn't find anything in our docs for it. Probably safe to move to 1.0.1 which is available on conda

datapythonista

Looks like we were xfailing a test not parsing small years in Python 3.5, but the problem was an old version of dateutil. Should we require a higher dateutil because of this, or just xfail as I did here?

CC: @jreback @jbrockmendel @pganssle

datapythonista · 2019-10-27T18:06:56Z

pandas/tests/scalar/period/test_period.py

-    # https://travis-ci.org/MacPython/pandas-wheels/jobs/574706922
-    PY35,
+    # Bug in old versions of python-dateutil (failing in 2.6.1)
+    StrictVersion(dateutil.__version__) <= StrictVersion("2.6.1"),


Failing because of dateutil, not Python 3.5

Looks like dateutil 2.7.0 was released in March 2018. I think we're safe to bump up to at least that.

Either way, really glad to see the reason for the failing test identified.

Here is the version breakdown of applications pinning dateutil versions. I think a disproportionate number are pinning 2.4.* because of the way fallback options for ambiguous YMD/MDY/DMY triples changed after that version.

I think it's probably fine to pin to a more recent version, but I'm not entirely sure it's warranted based on the fact that some people might experience a specific bug if they are using an old version of dateutil. I'd probably just leave the skip condition in place on the test until you have some actual new feature of dateutil that you want to use, but in the long run I'm not clear how much it will really matter.

ci/deps/azure-36-minimum_versions.yaml

…arrays' is caused by py36

jreback

lgtm. once the final build passes.

doc/source/getting_started/install.rst

WillAyd · 2019-11-08T00:38:11Z

Not sure on the 36 failure here but I think @TomAugspurger cleaned up the solvability of the environment. Merge master and try again?

datapythonista · 2019-11-08T03:33:51Z

Seems like Python 3.6 has something (I guess a bug) causing the error TypeError: only integer scalar arrays can be converted to a scalar index when converting strings to bytes in some of our cases.

Checked locally, and seems like tests pass again in Python 3.6.1. Updated this PR, so the minimum supported version of Python is 3.6.1.

WillAyd · 2019-11-08T05:13:39Z

Interesting. Not totally against it, though I wonder if we should also up our minimum numpy version as part of this.

Per NEP 29 I think they only support 1.15 with Python 3.6:

https://numpy.org/neps/nep-0029-deprecation_policy.html

So maybe worth bumping from 13.3 to that instead?

datapythonista · 2019-11-08T11:17:37Z

I don't have an opinion on bumping numpy version. But I think this PR is complex enough, and dropping old numpy versions is unrelated to this. I'd discuss and implement that separately.

NEP-29 is recent. Not sure when Python 3.5 is expected to be dropped by numpy based on it, but 1.15 supports even Python 3.4.

jreback · 2019-11-08T14:47:07Z

pandas/compat/__init__.py

@@ -12,7 +12,6 @@
 import sys
 import warnings

-PY35 = sys.version_info[:2] == (3, 5)
 PY36 = sys.version_info >= (3, 6)


can remove this entirely in a followup I think (by-definition PY36 will always be true)

jreback · 2019-11-08T14:48:41Z

thanks @datapythonista very nice!

very small comment for possible followup.

Also would take a bump in numpy version I think we can go to min 1.15, but open an issue for it first.

datapythonista added 2 commits October 24, 2019 13:56

CI: Drop Python 3.5 support

83a41e6

Updating documentation

e0dd3b3

datapythonista added CI Continuous Integration Python 3.5 labels Oct 24, 2019

datapythonista mentioned this pull request Oct 24, 2019

Drop Python 3.5 support #29034

Closed

TomAugspurger reviewed Oct 24, 2019

View reviewed changes

Removing PY35 compatibility stuff, and other updates

74f0cc4

datapythonista added 2 commits October 24, 2019 15:46

Fixing test in Mac

da1a24f

Fix typo in previous commit

fca2531

WillAyd requested changes Oct 25, 2019

View reviewed changes

datapythonista added 2 commits October 25, 2019 10:40

Restoring build with the minimum versions of dependencies, but for py…

50a18ad

…36 not py35

Pinning to Python 3.6.0

92cce08

simonjayhawkins reviewed Oct 25, 2019

View reviewed changes

WillAyd approved these changes Oct 25, 2019

View reviewed changes

datapythonista added 3 commits October 25, 2019 12:53

Fixing black error, moving dependency to conda

bb1b53d

Removing comment about py35

4bbb571

xfailing for the right reason (dateutil version)

ad93f0a

datapythonista commented Oct 27, 2019

View reviewed changes

datapythonista added 2 commits October 27, 2019 16:15

Better fix for the xfail in dateutil < 2.7.0

29e59ec

Merge remote-tracking branch 'upstream/master' into py35

0648171

jreback reviewed Oct 27, 2019

View reviewed changes

ci/deps/azure-36-minimum_versions.yaml Show resolved Hide resolved

Temporary running tests in Python 3.5 to see if 'only integer scalar …

2b1abca

…arrays' is caused by py36

jreback reviewed Oct 29, 2019

View reviewed changes

doc/source/getting_started/install.rst Show resolved Hide resolved

simonjayhawkins mentioned this pull request Nov 7, 2019

CLN: core.computation, mostly typing #29436

Closed

Merge remote-tracking branch 'upstream/master' into py35

ddbd726

datapythonista added 3 commits November 8, 2019 00:39

Merge branch 'py35' of github.com:datapythonista/pandas into py35

df468bf

Restoring minimal versions build to py36

eebaec9

Requiring 3.6.1 (3.6 seems to have a bug failing our tests)

4c0a066

Merge remote-tracking branch 'upstream/master' into py35

ad08687

jreback approved these changes Nov 8, 2019

View reviewed changes

jreback added this to the 1.0 milestone Nov 8, 2019

jreback merged commit 18efcb2 into pandas-dev:master Nov 8, 2019

WillAyd mentioned this pull request Nov 8, 2019

Bump NumPy to 1.15 #29486

Closed

This was referenced Nov 8, 2019

CLN: Removing Python 3.6 or higher references that are always true #29492

Merged

DEPS: Minimum numpy version for pandas 1.0 #29512

Closed

alimcmaster1 mentioned this pull request Nov 10, 2019

CLN: String formatting % -> f-strings #29518

Merged

2 tasks

simonjayhawkins mentioned this pull request Nov 16, 2019

CI: bump mypy 0.730 #29653

Merged

Reksbril pushed a commit to Reksbril/pandas that referenced this pull request Nov 18, 2019

CI: Drop Python 3.5 support (pandas-dev#29212)

2b0f221

alimcmaster1 mentioned this pull request Nov 29, 2019

CLN: OrderedDict -> Dict #29923

Merged

2 tasks

proost pushed a commit to proost/pandas that referenced this pull request Dec 19, 2019

CI: Drop Python 3.5 support (pandas-dev#29212)

25a2a5b

proost pushed a commit to proost/pandas that referenced this pull request Dec 19, 2019

CI: Drop Python 3.5 support (pandas-dev#29212)

1d4c6a4

alimcmaster1 mentioned this pull request Dec 25, 2019

CLN: OrderedDict -> Dict #30469

Closed

TomAugspurger mentioned this pull request Jan 7, 2020

Try supporting Python 3.6.0 #30794

Closed

snowman2 mentioned this pull request Jan 30, 2020

Drop Python 3.5 support pyproj4/pyproj#499

Closed

khaeru mentioned this pull request Feb 11, 2020

F-strings are only compatible with Python ≥3.6, not 3.5 iiasa/ixmp#168

Closed

simonjayhawkins mentioned this pull request Mar 26, 2020

Why pandas doesn't support python 3.6.1? #33038

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: Drop Python 3.5 support #29212

CI: Drop Python 3.5 support #29212

datapythonista commented Oct 24, 2019

TomAugspurger left a comment

WillAyd commented Oct 24, 2019 •

edited

Loading

TomAugspurger commented Oct 24, 2019 via email

gfyoung commented Oct 24, 2019 •

edited

Loading

datapythonista commented Oct 24, 2019

WillAyd Oct 25, 2019

datapythonista Oct 25, 2019

WillAyd Oct 25, 2019

datapythonista Oct 25, 2019

simonjayhawkins Oct 25, 2019

WillAyd left a comment

WillAyd Oct 25, 2019

datapythonista left a comment

datapythonista Oct 27, 2019

jbrockmendel Oct 27, 2019

pganssle Oct 30, 2019

jreback left a comment

WillAyd commented Nov 8, 2019

datapythonista commented Nov 8, 2019

WillAyd commented Nov 8, 2019

datapythonista commented Nov 8, 2019

jreback Nov 8, 2019

jreback commented Nov 8, 2019

		@@ -573,8 +573,6 @@ def test_agg_with_one_lambda(self):

		# sort for 35 and earlier

CI: Drop Python 3.5 support #29212

CI: Drop Python 3.5 support #29212

Conversation

datapythonista commented Oct 24, 2019

TomAugspurger left a comment

Choose a reason for hiding this comment

WillAyd commented Oct 24, 2019 • edited Loading

TomAugspurger commented Oct 24, 2019 via email

gfyoung commented Oct 24, 2019 • edited Loading

datapythonista commented Oct 24, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WillAyd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

datapythonista left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

WillAyd commented Nov 8, 2019

datapythonista commented Nov 8, 2019

WillAyd commented Nov 8, 2019

datapythonista commented Nov 8, 2019

Choose a reason for hiding this comment

jreback commented Nov 8, 2019

WillAyd commented Oct 24, 2019 •

edited

Loading

gfyoung commented Oct 24, 2019 •

edited

Loading