CI: unify environment creation #24632

saurav2608 · 2019-01-05T03:58:29Z

codecov · 2019-01-05T09:48:44Z

Codecov Report

Merging #24632 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #24632      +/-   ##
==========================================
- Coverage   92.38%   92.37%   -0.01%     
==========================================
  Files         166      166              
  Lines       52395    52395              
==========================================
- Hits        48403    48402       -1     
- Misses       3992     3993       +1

Flag	Coverage Δ
#multiple	`90.8% <ø> (ø)`	⬆️
#single	`43% <ø> (ø)`	⬆️

Impacted Files	Coverage Δ
pandas/util/testing.py	`88% <0%> (-0.1%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cb31b2b...a8d22f3. Read the comment docs.

codecov · 2019-01-05T09:48:44Z

Codecov Report

Merging #24632 into master will decrease coverage by 0.04%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #24632      +/-   ##
==========================================
- Coverage   91.81%   91.77%   -0.05%     
==========================================
  Files         175      175              
  Lines       52580    52580              
==========================================
- Hits        48278    48255      -23     
- Misses       4302     4325      +23

Flag	Coverage Δ
#multiple	`90.33% <ø> (-0.04%)`	⬇️
#single	`40.72% <ø> (-1.26%)`	⬇️

Impacted Files	Coverage Δ
pandas/io/clipboard/__init__.py	`39.21% <0%> (-17.65%)`	⬇️
pandas/io/gbq.py	`75% <0%> (-12.5%)`	⬇️
pandas/io/clipboard/clipboards.py	`18.51% <0%> (-12.35%)`	⬇️
pandas/core/frame.py	`96.79% <0%> (-0.12%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update de3a85c...e400387. Read the comment docs.

saurav2608 · 2019-01-05T10:06:13Z

This sets up DB in both travis and azure. Have not tested DB jobs on Azure but maybe worth a try.

jreback

needs some edits

ci/setup_env.sh

jreback · 2019-01-05T19:32:16Z

yes remove the original files

datapythonista

besides the comments, there is an inconsistent use of the echo commands, with cases like:

echo Error
echo "done"
echo "[done]"

And some times with a bare echo before and some without (I guess those add blank lines).

I'd be consistent and use the same format everywhere. I don't think we are using the squared brackets in any other script. So, I'd simply use: echo "some message" everywhere. But feel free to check the log and pick the format that makes more sense, but use the same everywhere.

.travis.yml

ci/setup_env.sh

datapythonista · 2019-01-06T00:29:24Z

and yes, please delete the replaced files in this same PR.

datapythonista

Check azure, your script is failing.

ci/setup_env.sh

saurav2608 · 2019-01-06T14:10:20Z

@datapythonista - Azure is failing on conda update specifically - conda update -q conda. I was reading here that conda now suggests conda update -n base conda. I can change the script to reflect this. However, why are we even doing this? We are downloading and installing the latest miniconda build. Why would we need to update conda just after that.

Another question is on line 32 of setup_env.sh. Why is the rm -rf is needed in the current setup. I would think that the VM running the tests is spun fresh everytime we start the test.

datapythonista · 2019-01-07T10:24:15Z

Yes, please update the conda update statement to the new syntax. Anaconda can release a new conda version, but not create a new installable immediately, that's why it's common to update conda itself after installation. It's similar to when you install Linux and immediately run dnf update or equivalent. The iso is older than the versions of the latest packages.

I'm not sure about the rm -rf. I guess travis can reuse the containers or something. If the environment is clean for every build, it surely doesn't make sense. But please leave it for now, I'm sure it was added for a reason.

saurav2608 · 2019-01-28T09:20:21Z

@datapythonista / @jreback - any thoughts on this PR?

jreback · 2019-01-28T12:26:23Z

this needs to wait until after 0.24.1 is out at a minimum

jreback · 2019-02-16T19:51:26Z

can you merge master and i'll loo again here

saurav2608 · 2019-02-18T15:25:37Z

@jreback - green.

jreback · 2019-02-19T14:00:31Z

@saurav2608 pls confirm for each build that we are running the same number of tests both before and after. This change could easy miss testing things and we would not easily know. (take the build on master right before you merged). check on both travis and azure; link the builds here.

saurav2608 · 2019-02-19T17:27:39Z

@jreback : I compared this build to one just before. Details -

PR	platform	No of Tests	Link
#24632	Azure	336,141	https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=8496&view=ms.vss-test-web.build-test-results-tab
#25209	Azure	336,141	https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=8495&view=ms.vss-test-web.build-test-results-tab

For Travis the order was a bit different. But I compared the this build with the one just before. And compared 4 builds

PR>build>Job	platform	No of Tests	Link
#24632 > Build #52470 > Job #52470.1	Travis	43095 passed, 2814 skipped, 1008 xfailed, 20 xpassed, 58 warnings in 678.23 seconds	https://travis-ci.org/pandas-dev/pandas/jobs/494931336
#25362 > Build #52468 > Job #52468.1	Travis	43095 passed, 2814 skipped, 1008 xfailed, 20 xpassed, 58 warnings in 659.78 seconds	https://travis-ci.org/pandas-dev/pandas/jobs/494917125
#24632 > Build #52470 > Job #52470.2	Travis	45407 passed, 1325 skipped, 783 xfailed, 20 xpassed, 24 warnings in 1144.70 seconds	https://travis-ci.org/pandas-dev/pandas/jobs/494931338
#25362 > Build #52468 > Job #52468.2	Travis	45407 passed, 1325 skipped, 783 xfailed, 20 xpassed, 24 warnings in 1093.64 seconds	https://travis-ci.org/pandas-dev/pandas/jobs/494917126
#24632 > Build #52470 > Job #52470.3	Travis	45338 passed, 917 skipped, 1013 xfailed, 20 xpassed, 42 warnings in 880.59 seconds	https://travis-ci.org/pandas-dev/pandas/jobs/494931340
#25362 > Build #52468 > Job #52468.3	Travis	45338 passed, 917 skipped, 1013 xfailed, 20 xpassed, 42 warnings in 833.46 seconds	https://travis-ci.org/pandas-dev/pandas/jobs/494917127
#24632 > Build #52470 > Job #52470.4	Travis	45410 passed, 842 skipped, 1014 xfailed, 21 xpassed, 90 warnings in 1305.06 seconds	https://travis-ci.org/pandas-dev/pandas/jobs/494931341
#25362 > Build #52468 > Job #52468.4	Travis	45410 passed, 842 skipped, 1014 xfailed, 21 xpassed, 90 warnings in 1243.80 seconds	https://travis-ci.org/pandas-dev/pandas/jobs/494917128

h-vetinari

Didn't see this PR before, I'll close #25377 once this gets merged.

ci/setup_env.sh

jreback · 2019-02-20T14:06:06Z

@saurav2608 ok this looks fine. I still am going to wait till we do 0.24.2, we will have to backport this as well I think. We have had CI brokeness in the past from even minor changes in the CI code.

WillAyd · 2019-02-27T23:58:29Z

Can you merge master on this?

h-vetinari · 2019-03-11T07:00:16Z

What's the status of this PR? I constantly need to look up manually which version is in failing posix environments. If this is still taking longer, can we merge #25377 in the interim?

h-vetinari · 2019-03-24T21:01:54Z

What's the status of this PR? I constantly need to look up manually which version is in failing posix environments. If this is still taking longer, can we merge #25377 in the interim?

jreback · 2019-03-24T21:03:11Z

this needs a rebase and the PY2 and numpy bump merged first.

jreback · 2019-03-30T19:21:13Z

@saurav2608 if you wan to rebase this, i think we can do this now.

saurav2608 · 2019-03-31T12:04:32Z

@saurav2608 if you wan to rebase this, i think we can do this now.

@jreback - green

jreback · 2019-04-05T00:43:10Z

thanks @jreback and thanks @datapythonista for guiding this along.

saurav-chakravorty added 5 commits January 5, 2019 09:18

CI: unify environment creation

04b9abb

Merge remote-tracking branch 'upstream/master' into CI

4ae7b16

Change permissions

2b2b565

removed pip cmd

e9a8442

removed wrong exit 0

a8d22f3

jreback requested changes Jan 5, 2019

View reviewed changes

ci/setup_env.sh Outdated Show resolved Hide resolved

ci/setup_env.sh Outdated Show resolved Hide resolved

saurav-chakravorty added 2 commits January 5, 2019 21:08

remove hash-bang

730ab53

Merge branch 'master' into CI

c5d357f

jreback added the CI Continuous Integration label Jan 5, 2019

jreback requested changes Jan 5, 2019

View reviewed changes

datapythonista reviewed Jan 6, 2019

View reviewed changes

saurav-chakravorty added 5 commits January 6, 2019 13:02

further cleanup

abaf652

removed space aroung var

14d164a

minor typo

6d469db

Changing build files for Azure

8d34048

corrected file name

6b22414

datapythonista requested changes Jan 6, 2019

View reviewed changes

saurav-chakravorty added 6 commits January 7, 2019 19:33

upper case variables and some cleanup.

625ddce

add export path

e68b1e2

remove debugging echo

39e3365

bug in install DB

595815a

Correct spelling of Linux

ae538e5

Only Travis has DBs

784244a

datapythonista approved these changes Jan 8, 2019

View reviewed changes

datapythonista mentioned this pull request Jan 22, 2019

TST: Review if we are really testing with different locales #23923

Closed

jreback added this to the 0.25.0 milestone Feb 16, 2019

saurav-chakravorty added 3 commits February 17, 2019 16:03

Merge remote-tracking branch 'upstream/master' into CI

54cca4d

Merge remote-tracking branch 'upstream/master' into CI

e66a69c

Merge remote-tracking branch 'upstream/master' into CI

b9cedb1

saurav2608 mentioned this pull request Feb 20, 2019

CI: Add 'conda list' for azure/posix after activate 'pandas-dev' #25377

Closed

h-vetinari reviewed Feb 20, 2019

View reviewed changes

ci/setup_env.sh Show resolved Hide resolved

Merge remote-tracking branch 'upstream/master' into CI

549a297

Merge remote-tracking branch 'upstream/master' into CI

e400387

jreback approved these changes Apr 5, 2019

View reviewed changes

jreback merged commit b974633 into pandas-dev:master Apr 5, 2019

jorisvandenbossche mentioned this pull request Apr 13, 2019

DOC: clipboard library missing in doc env on travis #26075

Closed

saurav2608 deleted the CI branch April 15, 2019 18:30

datapythonista mentioned this pull request May 14, 2019

CI: Use bash for Windows scripts on Azure #26344

Closed

jorisvandenbossche mentioned this pull request May 15, 2019

CI/DOC: fix clipboard on travis #26103

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: unify environment creation #24632

CI: unify environment creation #24632

saurav2608 commented Jan 5, 2019 •

edited by jreback

Loading

codecov bot commented Jan 5, 2019

codecov bot commented Jan 5, 2019 •

edited

Loading

saurav2608 commented Jan 5, 2019

jreback left a comment

jreback commented Jan 5, 2019

datapythonista left a comment

datapythonista commented Jan 6, 2019

datapythonista left a comment

saurav2608 commented Jan 6, 2019

datapythonista commented Jan 7, 2019

saurav2608 commented Jan 28, 2019

jreback commented Jan 28, 2019

jreback commented Feb 16, 2019

saurav2608 commented Feb 18, 2019

jreback commented Feb 19, 2019 •

edited

Loading

saurav2608 commented Feb 19, 2019

h-vetinari left a comment •

edited

Loading

jreback commented Feb 20, 2019

WillAyd commented Feb 27, 2019

h-vetinari commented Mar 11, 2019

h-vetinari commented Mar 24, 2019

jreback commented Mar 24, 2019

jreback commented Mar 30, 2019

saurav2608 commented Mar 31, 2019

jreback commented Apr 5, 2019

CI: unify environment creation #24632

CI: unify environment creation #24632

Conversation

saurav2608 commented Jan 5, 2019 • edited by jreback Loading

codecov bot commented Jan 5, 2019

Codecov Report

codecov bot commented Jan 5, 2019 • edited Loading

Codecov Report

saurav2608 commented Jan 5, 2019

jreback left a comment

Choose a reason for hiding this comment

jreback commented Jan 5, 2019

datapythonista left a comment

Choose a reason for hiding this comment

datapythonista commented Jan 6, 2019

datapythonista left a comment

Choose a reason for hiding this comment

saurav2608 commented Jan 6, 2019

datapythonista commented Jan 7, 2019

saurav2608 commented Jan 28, 2019

jreback commented Jan 28, 2019

jreback commented Feb 16, 2019

saurav2608 commented Feb 18, 2019

jreback commented Feb 19, 2019 • edited Loading

saurav2608 commented Feb 19, 2019

h-vetinari left a comment • edited Loading

Choose a reason for hiding this comment

jreback commented Feb 20, 2019

WillAyd commented Feb 27, 2019

h-vetinari commented Mar 11, 2019

h-vetinari commented Mar 24, 2019

jreback commented Mar 24, 2019

jreback commented Mar 30, 2019

saurav2608 commented Mar 31, 2019

jreback commented Apr 5, 2019

saurav2608 commented Jan 5, 2019 •

edited by jreback

Loading

codecov bot commented Jan 5, 2019 •

edited

Loading

jreback commented Feb 19, 2019 •

edited

Loading

h-vetinari left a comment •

edited

Loading