Remove py27 CI jobs #24942

h-vetinari · 2019-01-26T00:24:17Z

The start of a looooooooong process to get rid of the py2 compat code.

I kept most of the CI jobs that used to be 2.7 at the lower end of the various supported versions, with 2x 2.7->3.5, 1x 2.7->3.6 and only one real upgrade from 2.7->3.7 (first CI job for 3.7 & windows).

@jreback @TomAugspurger

ci/deps/azure-windows-37.yaml

jreback · 2019-01-26T00:50:50Z

keep in mind that we likely won't merge anything PY2 removal for a month or so.

h-vetinari · 2019-01-26T00:59:49Z

keep in mind that we likely won't merge anything PY2 removal for a month or so.

Which makes a lot of sense from a backports POV, but I didn't think about that. Guess I just couldn't/can't wait. ;)

codecov · 2019-02-01T23:18:29Z

Codecov Report

Merging #24942 into master will increase coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #24942      +/-   ##
==========================================
+ Coverage   92.37%   92.37%   +<.01%     
==========================================
  Files         166      166              
  Lines       52420    52388      -32     
==========================================
- Hits        48423    48396      -27     
+ Misses       3997     3992       -5

Flag	Coverage Δ
#multiple	`90.8% <ø> (ø)`	⬆️
#single	`42.9% <ø> (+0.01%)`	⬆️

Impacted Files	Coverage Δ
pandas/io/clipboard/clipboards.py	`28.23% <0%> (-2.36%)`	⬇️
pandas/core/indexes/base.py	`96.32% <0%> (-0.23%)`	⬇️
pandas/core/reshape/merge.py	`94.4% <0%> (-0.08%)`	⬇️
pandas/core/indexes/multi.py	`95.59% <0%> (-0.02%)`	⬇️
pandas/core/indexes/range.py	`97.36% <0%> (-0.01%)`	⬇️
pandas/core/indexes/datetimes.py	`96.27% <0%> (-0.01%)`	⬇️
pandas/core/resample.py	`97.23% <0%> (-0.01%)`	⬇️
pandas/core/generic.py	`96.63% <0%> (-0.01%)`	⬇️
pandas/core/tools/timedeltas.py	`98% <0%> (ø)`	⬆️
pandas/core/tools/numeric.py	`100% <0%> (ø)`	⬆️
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bb43726...b88b87d. Read the comment docs.

codecov · 2019-02-01T23:18:30Z

Codecov Report

Merging #24942 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #24942      +/-   ##
==========================================
- Coverage   91.25%   91.25%   -0.01%     
==========================================
  Files         172      172              
  Lines       52976    52976              
==========================================
- Hits        48342    48341       -1     
- Misses       4634     4635       +1

Flag	Coverage Δ
#multiple	`89.82% <ø> (ø)`	⬆️
#single	`41.74% <ø> (ø)`	⬆️

Impacted Files	Coverage Δ
pandas/util/testing.py	`88.98% <0%> (-0.1%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a61d823...fe83f86. Read the comment docs.

h-vetinari · 2019-03-14T16:04:43Z

@jreback @TomAugspurger
The last couple of builds have had weird (but consistent) travis errors in the travis_35 job, where the worker nodes crash after about ~10 sec (so I'm thinking it's not related to actual tests segfaulting, but somehow the interplay between different versions of "infrastructure packages).

jreback

@h-vetinari this is again changing too much. I want only the actual env files to be changed. you are changing all kinds of CI things and something is still breaking. I would start anew and simply remove the PY2 builds and leave the actual updates of versions to another PR.

jreback · 2019-03-14T17:20:51Z

ci/deps/travis-35.yaml

@@ -3,14 +3,13 @@ channels:
  - defaults
  - conda-forge
 dependencies:
+  - backports.lzma


jreback · 2019-03-14T17:21:11Z

ci/deps/travis-35.yaml

@@ -19,33 +18,40 @@ dependencies:
  - numexpr
  - numpy=1.13*
  - openpyxl=2.4.0
+  - pathlib


this is not needed as its builtin

jreback · 2019-03-14T17:21:51Z

ci/deps/travis-35.yaml

  - hypothesis>=3.58.0
  - isort
+  # satisfy fastparquet deps with conda


i would rather remove fastparquet from this build then, we have it installed for 1 or more of 3.6/3.7 already I think (pls check)

jreback · 2019-03-14T17:23:06Z

pandas/tests/io/test_excel.py

@@ -2182,6 +2182,8 @@ def test_write_cells_merge_styled(self, merge_cells, ext, engine):
            assert xcell_b1.font == openpyxl_sty_merged
            assert xcell_a2.font == openpyxl_sty_merged

+    @pytest.mark.xfail(PY35 and not PY36, reason='only fails on Linux?',


why is this changing?

Because I wasn't able to make this test work on PY35 regardless of the versions I tried.

jreback · 2019-03-14T17:23:28Z

pandas/tests/io/test_sql.py

@@ -28,7 +28,7 @@
 import pytest

 import pandas.compat as compat
-from pandas.compat import PY2, PY36, lrange, range, string_types
+from pandas.compat import PY2, PY35, PY36, lrange, range, string_types


really really don't change things, if you are forcing chanegs then roll back the env changes which cause this.

jreback · 2019-03-14T17:23:37Z

pandas/tests/test_algos.py

@@ -11,8 +11,9 @@

 from pandas._libs import (
    algos as libalgos, groupby as libgroupby, hashtable as ht)
-from pandas.compat import PY2, lrange, range
-from pandas.compat.numpy import np_array_datetime64_compat
+from pandas.compat import lrange, range


see my comment above

Again, if I don't fix the regex here, the tests will fail.

h-vetinari

@jreback
Pushing some changes based on your review. This PR contains some xfails that I wasn't able to solve so far, as well as two changes to regexes that would otherwise fail. If the goal is a passing CI, those are unavoidable.

h-vetinari · 2019-03-14T19:25:31Z

pandas/tests/io/test_excel.py

@@ -2182,6 +2182,8 @@ def test_write_cells_merge_styled(self, merge_cells, ext, engine):
            assert xcell_b1.font == openpyxl_sty_merged
            assert xcell_a2.font == openpyxl_sty_merged

+    @pytest.mark.xfail(PY35 and not PY36, reason='only fails on Linux?',


Because I wasn't able to make this test work on PY35 regardless of the versions I tried.

h-vetinari · 2019-03-14T19:26:08Z

pandas/tests/test_algos.py

@@ -11,8 +11,9 @@

 from pandas._libs import (
    algos as libalgos, groupby as libgroupby, hashtable as ht)
-from pandas.compat import PY2, lrange, range
-from pandas.compat.numpy import np_array_datetime64_compat
+from pandas.compat import lrange, range


Again, if I don't fix the regex here, the tests will fail.

h-vetinari · 2019-03-14T19:27:43Z

ci/deps/travis-35.yaml

  - hypothesis>=3.58.0
  - isort
+  # satisfy fastparquet deps with conda


Removed. Remaining jobs with fastparquet:

ci/deps\azure-windows-36.yaml:9: - fastparquet>=0.2.1 ci/deps\travis-36-doc.yaml:9: - fastparquet>=0.2.1 ci/deps\travis-36.yaml:10: - fastparquet>=0.2.1

jorisvandenbossche · 2019-03-14T20:25:15Z

@h-vetinari can you point to the test errors for which you needed to add the additional xfails? (or do a new build without any of them) Ideally, we should investigate each of those in more detail why they are failing instead of just xfailing them (unless you already know the reason for each of them)

h-vetinari · 2019-03-14T23:19:36Z

@jorisvandenbossche
The fails are distributed over the CI runs throughout the thread, but I've reverted them in the last commit (including the regex fixes @jreback wanted me to revert), so then there's a single place to inspect the failures. Also - who knows - maybe some of them resolved themselves...

h-vetinari · 2019-03-15T09:38:23Z

@jorisvandenbossche @TomAugspurger @jreback
Please check out https://travis-ci.org/pandas-dev/pandas/jobs/506535600, it fails already in the single stage with the tests I would have xfailed. When I first started this PR I played around with bumping the versions of xlrd etc (even when unpinning pymysql/xlsxwriter), but that didn't solve things either.

Considering the fact that removing support for PY35 is already being talked about in #25725, I think a small handful of xfailed tests is not a big burden. Certainly those are less trivial to fix than for example the skips that have been added for regex fails (of which there about 30).

jreback · 2019-03-15T11:05:13Z

as i said above pls start over and JUST remove the 2.7 jobs; since things are working now this will just work. after that you can do independent PR to add new / change things.

something is breaking and certainly needs fixing
but not here

jreback · 2019-03-15T11:05:50Z

PY35 is not being removed at this time

h-vetinari · 2019-03-15T11:44:05Z

@jreback: as i said above pls start over and JUST remove the 2.7 jobs; since things are working now this will just work. after that you can do independent PR to add new / change things.

Of course it's easier to just rip things out, but I actually went to the effort here of thinking of a good distribution for the CI, updating the jobs, and making them pass again.

That's also how the last numpy bump was made. Tbh, I feel these conditions are quite arbitrary, but well... I'll delete the former PY2 jobs later today, and post the fixes in a separate PR.

PY35 is not being removed at this time

I know that. But adding 6-7 PY35-specific xfails (for non-core functionality) aren't that severe IMO, considering the point in the lifecycle.

jreback · 2019-03-15T11:48:27Z

@h-vetinari as I have explained many many times. PRs which do lots of things take way more time to review and are much harder to merge. It is the same case here.

h-vetinari · 2019-03-15T17:03:01Z

@jreback
This is green now. If only it always were as easy as deleting CI jobs, haha.

jreback

just the 1 comment

doc/source/whatsnew/v0.25.0.rst

h-vetinari

Sorry, somehow didn't get a notification for the first comment.

doc/source/whatsnew/v0.25.0.rst

jreback · 2019-03-17T15:12:04Z

thanks, would also appreciate a PR against : https://github.com/Macpython/pandas-wheels (the daily branch, not master).

h-vetinari · 2019-03-17T15:13:10Z

thanks, would also appreciate a PR against : https://github.com/Macpython/pandas-wheels (the daily branch, not master).

This same PR or the PY35 one?

jreback · 2019-03-17T15:32:30Z

neither. one to remove the PY2 jobs only (this is where we build wheels), and is our daily testing.

h-vetinari added 2 commits January 26, 2019 01:18

Remove py27 CI jobs

7966035

Remove jinja-pin for 3.7 build

b224cd6

h-vetinari commented Jan 26, 2019

View reviewed changes

ci/deps/azure-windows-37.yaml Outdated Show resolved Hide resolved

gfyoung added Build Library building on various platforms 2/3 Compat labels Jan 26, 2019

gfyoung added this to the 0.25.0 milestone Jan 26, 2019

h-vetinari added 2 commits February 2, 2019 00:11

Merge remote-tracking branch 'upstream/master' into drop_py2_ci

aac2192

Reflect renamed file names in CI config / scripts

b88b87d

h-vetinari added 13 commits February 2, 2019 01:34

fix travis.yml

fdc0035

Fix azure builds

5de5d5f

Update minimal versions to make env resolution possible

dcccd6e

Merge remote-tracking branch 'upstream/master' into drop_py2_ci

5588fbd

More dep resolutions

a01f50e

Try if bumping pymysql/xlsxwriter solves failures

19ff3a5

fix naming of azure jobs

835679e

Unpin pymysql/xlsxwriter in affected builds to check...

7827b71

Merge remote-tracking branch 'upstream/master' into drop_py2_ci

8d390df

temp fix to ease discussion: xfail failing tests

e41689f

One last test & non-strict xfail

0c50978

Merge remote-tracking branch 'upstream/master' into drop_py2_ci

8292752

non-strict xfail for flaky test

4d5c3c8

jreback mentioned this pull request Feb 7, 2019

[WIP] remove ci testing for python 2 #25217

Closed

4 tasks

mroeschke mentioned this pull request Feb 7, 2019

Remove python 2.7 testing #25216

Closed

h-vetinari added 3 commits February 17, 2019 14:18

Merge remote-tracking branch 'upstream/master' into drop_py2_ci

e641585

Unpin moto

b32f434

Retrigger after connectivity issues

eae4308

jreback requested changes Mar 14, 2019

View reviewed changes

h-vetinari added 2 commits March 14, 2019 20:30

Merge remote-tracking branch 'upstream/master' into drop_py2_ci

d63830a

Review (jreback)

979341a

h-vetinari commented Mar 14, 2019

View reviewed changes

h-vetinari added 2 commits March 15, 2019 00:15

Tentatively remove xfails

d3a50e0

Tentatively revert regex fixes

f7492d4

h-vetinari added 3 commits March 15, 2019 16:16

Remove all former PY2 jobs (review jreback)

2632664

Merge remote-tracking branch 'upstream/master' into drop_py2_ci

c159cfa

Remove tox.ini

fe83f86

jreback requested changes Mar 15, 2019

View reviewed changes

doc/source/whatsnew/v0.25.0.rst Show resolved Hide resolved

h-vetinari commented Mar 17, 2019

View reviewed changes

doc/source/whatsnew/v0.25.0.rst Show resolved Hide resolved

jreback approved these changes Mar 17, 2019

View reviewed changes

jreback merged commit 707c720 into pandas-dev:master Mar 17, 2019

h-vetinari mentioned this pull request Mar 17, 2019

Restore CI after PY2 drop #25752

Merged

WillAyd mentioned this pull request Mar 18, 2019

Build Failures #25762

Closed

h-vetinari deleted the drop_py2_ci branch March 25, 2019 06:51

h-vetinari mentioned this pull request Mar 28, 2019

CI: build failing because of gbq testing #25910

Closed

AlexandreDecan mentioned this pull request Apr 1, 2019

BUG: GH24011 - Rich comparisons of Timestamps now return NotImplemented #24021

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove py27 CI jobs #24942

Remove py27 CI jobs #24942

h-vetinari commented Jan 26, 2019

jreback commented Jan 26, 2019

h-vetinari commented Jan 26, 2019

codecov bot commented Feb 1, 2019

codecov bot commented Feb 1, 2019 •

edited

Loading

h-vetinari commented Mar 14, 2019

jreback left a comment

jreback Mar 14, 2019

jreback Mar 14, 2019

jreback Mar 14, 2019

jreback Mar 14, 2019

h-vetinari Mar 14, 2019

jreback Mar 14, 2019

jreback Mar 14, 2019

h-vetinari Mar 14, 2019

h-vetinari left a comment

h-vetinari Mar 14, 2019

h-vetinari Mar 14, 2019

h-vetinari Mar 14, 2019

jorisvandenbossche commented Mar 14, 2019

h-vetinari commented Mar 14, 2019

h-vetinari commented Mar 15, 2019

jreback commented Mar 15, 2019

jreback commented Mar 15, 2019

h-vetinari commented Mar 15, 2019 •

edited

Loading

jreback commented Mar 15, 2019 •

edited

Loading

h-vetinari commented Mar 15, 2019

jreback left a comment •

edited

Loading

h-vetinari left a comment

jreback commented Mar 17, 2019

h-vetinari commented Mar 17, 2019

jreback commented Mar 17, 2019

Remove py27 CI jobs #24942

Remove py27 CI jobs #24942

Conversation

h-vetinari commented Jan 26, 2019

jreback commented Jan 26, 2019

h-vetinari commented Jan 26, 2019

codecov bot commented Feb 1, 2019

Codecov Report

codecov bot commented Feb 1, 2019 • edited Loading

Codecov Report

h-vetinari commented Mar 14, 2019

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

h-vetinari left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorisvandenbossche commented Mar 14, 2019

h-vetinari commented Mar 14, 2019

h-vetinari commented Mar 15, 2019

jreback commented Mar 15, 2019

jreback commented Mar 15, 2019

h-vetinari commented Mar 15, 2019 • edited Loading

jreback commented Mar 15, 2019 • edited Loading

h-vetinari commented Mar 15, 2019

jreback left a comment • edited Loading

Choose a reason for hiding this comment

h-vetinari left a comment

Choose a reason for hiding this comment

jreback commented Mar 17, 2019

h-vetinari commented Mar 17, 2019

jreback commented Mar 17, 2019

codecov bot commented Feb 1, 2019 •

edited

Loading

h-vetinari commented Mar 15, 2019 •

edited

Loading

jreback commented Mar 15, 2019 •

edited

Loading

jreback left a comment •

edited

Loading