Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The new pricing model for Travis CI #5028

Closed
5 tasks done
hugovk opened this issue Nov 2, 2020 · 30 comments
Closed
5 tasks done

The new pricing model for Travis CI #5028

hugovk opened this issue Nov 2, 2020 · 30 comments

Comments

@hugovk
Copy link
Member

hugovk commented Nov 2, 2020

Travis CI announced a new pricing model today:

It's not entirely clear how this affects open source, but zopefoundation/meta#38 (comment) looks like the best interpretation:

  • You get a free trial of 10k credits to begin (equates to 1,000 Linux minutes; macOS costs more)
  • When the trial runs out, they want you to pick a paid plan
  • They will offer free credits for OSS "reviewed and allocated on a case by case basis", email support to apply
  • Once these run out, you can email support again and ask for more (does this mean not being able to build during the review process?)

It looks like we should move more stuff from Travis CI to GitHub Actions.

cc @python-pillow/pillow-team

Pillow

This repo, and the most active.

image

  • Let's ditch lint, it's on GHA
  • The new CPU architectures Arm64, ppc64le, s390x are unique to Travis, let's keep them there
  • 3.6-3.9 (AMD64) builds: let's remove these, we have them on GHA. Is there anything unique to these that aren't included in the equivalent GHA jobs?

pillow-wheels

The biggest build, less active. Essential for releases.

image

With great timing, @nulano (thank you!) had already done the work and opened this PR to move all the jobs to GHA, except for Arm64 which need to stay there: python-pillow/pillow-wheels#169.

Let's get that reviewed and merged.

docker-images

A big build, less active.

image

Let's shift it all over to GHA. We'll need to set credentials as secrets to upload to Docker Hub. Or alternatively upload to GitHub Packages.

Also

Least active, might as well move these over too, should be easy enough.

Checklist

@aclark4life
Copy link
Member

Yeah I think @wiredfool likes GitHub Actions … I'm game.

@umarcor
Copy link

umarcor commented Nov 3, 2020

The new CPU architectures Arm64, ppc64le, s390x are unique to Travis, let's keep them there

See dbhi/qus. It was created and it is used precisely for migrating ARM, PPC64 and S390X jobs to GHA.

@nulano nulano mentioned this issue Nov 3, 2020
@nulano
Copy link
Contributor

nulano commented Nov 3, 2020

The new CPU architectures Arm64, ppc64le, s390x are unique to Travis, let's keep them there

See dbhi/qus. It was created and it is used precisely for migrating ARM, PPC64 and S390X jobs to GHA.

Thanks for that, it works quite well. However, the performance is unsurprisingly an order of magnitude slower; arm64 takes ~30m, ppc64le ~55m, s390x ~45m.

Since most of that time is spent installing dependencies, I think it would be reasonable to migrate these jobs to https://github.com/python-pillow/docker-images (since they run in a Docker container anyway). The build and test steps alone take ~10m, which is much more reasonable.

I'm not sure if it is feasible to migrate the arm64 wheel builds to GHA as they compile all dependencies from scratch instead of downloading from Ubuntu, so I wouldn't be surprised to see those take multiple hours.

@nulano
Copy link
Contributor

nulano commented Nov 4, 2020

You get a free trial of 10k credits to begin (equates to 1,000 Linux minutes; macOS costs more)

To quantify this, here is the usage for the past month (6 Oct to 4 Nov):
https://travis-ci.com/github/python-pillow?tab=insights: 38 builds, 1,580 minutes
https://travis-ci.org/github/python-pillow?tab=insights: 208 builds, 12,503 minutes

That is ~140,000 credits per month, assuming it is all Linux (it is more if accounting for macOS builds).

Looking at just the arm64 wheels on Travis, one build takes ~1h54m total, so 10,000 credits is enough for just 8 builds!


They do at least mention the possibility of a renewable amount:

How many credits (build minutes) you’d like to request (should your run out of credits again you can repeat the process to request more or discuss a renewable amount)

@umarcor
Copy link

umarcor commented Nov 4, 2020

Thanks for that, it works quite well. However, the performance is unsurprisingly an order of magnitude slower; arm64 takes ~30m, ppc64le ~55m, s390x ~45m.

Correct. qemu-user is 5-10 times slower than native execution or KVM. At the same time, in qemu-user some features (lower level signals, etc.) might not be complete, compared to qemu-system. Hence, combining qemu-user and containers might not fit all the use cases. However, when it does, it comes quite handy.

Since most of that time is spent installing dependencies, I think it would be reasonable to migrate these jobs to python-pillow/docker-images (since they run in a Docker container anyway). The build and test steps alone take ~10m, which is much more reasonable.

Agree. Installing dependencies should be decoupled from building and testing a project. Images with dependencies can be updated weekly or monthly, while the project is expected to be tested more frequently.

@nulano
Copy link
Contributor

nulano commented Nov 4, 2020

I've created python-pillow/docker-images#94 to migrate the docker-images job.

@hugovk
Copy link
Member Author

hugovk commented Nov 16, 2020

Two weeks after the Travis CI change I'm now out of credits, their UI both says they will be replenished and that they won't be replenished. I've emailed them asking for credits, but nothing yet. So it'd be great to move ahead with this!

We now have PRs for all five repos:

Reviews/merge especially welcome on #5029.

And @wiredfool please could you add DOCKER_USERNAME and DOCKER_PASSWORD to https://github.com/python-pillow/docker-images/settings/secrets/actions?

Thanks!

@wiredfool
Copy link
Member

I've added credentials to the docker-images repo. Any existing credentials aren't going to work anymore, so the automated build on travis is going to fail going forward, if it ever has credits to run again.

@hugovk
Copy link
Member Author

hugovk commented Nov 16, 2020

Thanks!

There's no .travis.yml at docker-images now, so there's nothing for it to build. For good measure, I've also deleted the cron at https://travis-ci.com/github/python-pillow/docker-images/settings and flipped off the "Build pushed branches" and "Build pushed pull requests" toggles.

@nulano
Copy link
Contributor

nulano commented Nov 17, 2020

I just got a sort-of notification (blue dot on my profile icon) on GitHub about "GitHub Container Registry" beta: long link to docs

Looks like it is a free alternative to Docker Hub which recently introduced new usage limits of 200 container image requests per 6 hours and is replacing GitHub Packages Docker.

It seems to be easier to set up than before (maybe even easier than Docker Hub), so while I don't think this is needed right now, it might be worth looking into at some point in the future if the limit becomes an issue.


For good measure, I've also deleted the cron at https://travis-ci.com/github/python-pillow/docker-images/settings and flipped off the "Build pushed branches" and "Build pushed pull requests" toggles.

I didn't realize there was a cron job as it was not specified in .travis.yml, but it should be easy to set up on GHA as well: https://docs.github.com/en/free-pro-team@latest/actions/reference/events-that-trigger-workflows#scheduled-events

@hugovk
Copy link
Member Author

hugovk commented Nov 17, 2020

Yeah, the GitHub registry may be useful in the future, but we don't need it right now as GitHub has been working with Docker Hub so GHA isn't affected by the rate limits: actions/runner-images#1445 (comment)

That's not the case with Travis CI, where the limits do apply: https://travis-ci.community/t/docker-hub-you-have-reached-your-pull-rate-limit/10517. So it's good we've moved our Docker stuff off of Travis!


Yup, we had cron enabled for the three main repos, you can only set it via the web UI for Travis. The settings:

  • Branch: master
  • Interval: weekly (not daily or monthly)
  • Options: Do not run if there has been a build in the last 24h (not "Always run")

It would be good to add cron to these three GHA once all merged. GHA is good that you can actually define POSIX cron syntax. We should set them to run at different times/days, would be nice to try and pick "quiet" times, but we're of course a global team so that might not exist :)


I'll merge #5029 tomorrow if there's no further feedback.

@umarcor
Copy link

umarcor commented Nov 17, 2020

It would be good to add cron to these three GHA once all merged. GHA is good that you can actually define POSIX cron syntax. We should set them to run at different times/days, would be nice to try and pick "quiet" times, but we're of course a global team so that might not exist :)

Hint: you can also set 'dispatch' event keys, in order to programmatically sequence workflows in the same repo or across repos. For example:

Note: the default GITHUB_TOKEN cannot trigger multiple subsequent workflows. That's a limitation for preventing unexperienced users creating infinite loops. However, you can achieve it if you use a regular token instead.

@hugovk
Copy link
Member Author

hugovk commented Dec 2, 2020

I see the credits have now begun counting down at https://travis-ci.com/organizations/python-pillow/plan:

Credits

3200 available credits (purchase date: December 02, 2020)
You have used 6800 of 10000 credits
3200 available

image


OSS only credits

0 available credits (next replenish date: December 02, 2020)
You have used 0 of your 0 monthly credits
0 available

image


I'll email Travis CI support requesting OSS credits for the python-pillow organisation. In the meantime, to save precious credits I've turned off the remaining crons at https://travis-ci.com/github/python-pillow/Pillow/settings and https://travis-ci.com/github/python-pillow/pillow-wheels/settings.

Let's also use [CI skip] when we don't want to test on Travis CI. ([CI skip] doesn't work on GHA.)


For reference, for my own account, after being out of credits and unable to test anything on Travis for 16 days, they've added 50k whilst they're waiting (11 days and counting) for "managerial approval":

https://twitter.com/hugovk/status/1334054612672901121

So I won't be too surprised if we're out of credits for the January release, which would prevent building aarch64 wheels (the rest are done by GHA). It might be that we can build those on a personal account, or add them to the release later on. Let's see how it goes.

@hugovk
Copy link
Member Author

hugovk commented Dec 2, 2020

Sent:

Hello,

Please may I open a new support ticket to request OSS credits for the open-source Pillow project? I'm one of the maintainers and I believe it meets your OSS requirements.

Pillow is a graphics library for Python, with some 17 million downloads per month. The project is 10 years old and in active development, forked from an earlier 25-year-old library called PIL.

We depend upon Travis CI for both CI (to test changes) and for CD (to build binary packages called "wheels" for releases).

I would be grateful if you could ensure we have credits before 2 January 2021, when our next quarterly release is due out and need Travis to build binary wheels.

Thank you!

Received:

Hello,

Thank you for reaching out to Travis-CI Support. This is an automated message to notify you that we have received your ticket (23740) and one of our support agent will get back to you to assist with this matter. In case you are seeing a bug and haven't shared this information already, please reply to this email by sharing the steps to reproduce along with expected and actual results and your system (environment) information to supplement the troubleshoot process.

Thank you for your patience while one of the support agent gets in touch with you!

Cheers,
Your Friends @Travis-CI

@hugovk
Copy link
Member Author

hugovk commented Dec 9, 2020

Seven days later, no reply beyond an autoreply, we're down 950 credits, from 3200 to 2250:

image

At this rate, we'll be out in 2.5 weeks, roughly Christmas day 🎅 🎄 🎁

I'm not getting my hopes up, someone else was told "credit allocation for OSS projects is on hold as per directives from management".

@aclark4life
Copy link
Member

@hugovk Only loosely following but I'd say we don't need Travis anymore, and by that I mean I'm not waiting around for them to give us credits. I'd rather find an alternative, even if we have to pay a Travis competitor. Can you tell me if we've completely moved to GitHub Actions or if we need to do more work and once we move there are we done? Or do we need to look for additional, possibly paid alternatives? Thanks

@hugovk
Copy link
Member Author

hugovk commented Dec 9, 2020

We're in quite a good situation now.

We've moved pretty much everything we can and everything essential over to GitHub Actions.

We've kept testing of three CPU architectures on Travis for this repo, because they're not available elsewhere. These are quite new to Travis anyway. We could probably ditch two and just keep the big-endian one, that's the most useful.

Similarly, we've moved all wheel building to GHA except some (aarch64) not available elsewhere. These are additions this summer, so probably not the biggest loss if we need to skip them.

All the other repos are migrated 👍

I'm not too worried if these things suddenly stop working and have pretty much given up on Travis.

There may be alternatives to fill the gaps, it'll need further investigation (e.g. #5028 (comment)) to find sonething feasible, and see what other projects do.

@hugovk
Copy link
Member Author

hugovk commented Dec 11, 2020

In python-pillow/pillow-wheels#172 (comment) @nulano suggested temporarily switching off Travis to conserve the last remaining credits. That's a very good idea, as they're going to run out anyway soon.

At https://travis-ci.com/github/python-pillow/Pillow/settings and https://travis-ci.com/github/python-pillow/pillow-wheels/settings I've flipped these left two toggles off:

image

Let's remember to turn the pillow-wheels one back on for the release, and when/if we ever get credits.

@nulano
Copy link
Contributor

nulano commented Dec 11, 2020

We've kept testing of three CPU architectures on Travis for this repo, because they're not available elsewhere. These are quite new to Travis anyway. We could probably ditch two and just keep the big-endian one, that's the most useful.
(...)
There may be alternatives to fill the gaps, it'll need further investigation (e.g. #5028 (comment)) to find sonething feasible, and see what other projects do.

I've created #5088 and python-pillow/docker-images#98 to move the three non-x86 builds in this repo to GHA Docker tests.

I haven't investigated whether that approach is reasonable for the Arm64 wheel builds yet, but I'm guessing not.

@hugovk
Copy link
Member Author

hugovk commented Jan 6, 2021

Update: After triggering a wheel build (#4997 (comment)), gone from 2250(?) to 1100:

image

Should be good for one more build before we're out. I sent Travis support a ping 3 days ago, nothing yet.

@hugovk
Copy link
Member Author

hugovk commented Apr 12, 2021

Amazing, after ignoring my ticket ping on 6th January, three days ago Travis sent me this!

Hello Hugo​ ,

Please accept our sincere apologies for any inconvenience caused due to delays in processing your open-source credit request while we made improvements to the platform.

Open source has always been and always will be at the core of what Travis CI stands for. After reviewing your open source qualifications, I am pleased to notify you that 25k credits were added to your account allowing you to resume running OSS builds. When your credits begin running low again, please reach back out to the Support team.

Credits Consumption Metric
OS. # CREDITS PER STARTED BUILD MINUTE
Linux. 10
Experimental FreeBSD. 10
Windows. 20
MacOS. 50

Please let us know if you have any questions!

Thanks and happy building!,

Your Friends @travis-ci

Note: OSS credits may be used solely with respect to open source projects that reside on a public repository. Please note that in no event shall OSS credits be used in support of commercial or enterprise projects or with respect to any project that resides on a private repository. Travis CI shall determine the amount and frequency of all OSS Credit allotments at its sole discretion and reserves the right to deny, reduce or suspend such allotments.

https://travis-ci.com/organizations/python-pillow/plan now shows:

image

My confidence in Travis is low after this six-month episode, and I've not received any updates for other projects I'd contacted them about. Plus the necessity to keep an eye on and request credits again.

But I'll re-enable Travis for our repos here, and let's try it for a bit and see how it pans out.

@hugovk
Copy link
Member Author

hugovk commented Apr 12, 2021

And now re-enabled for pushed branches and PRs at https://travis-ci.com/github/python-pillow/pillow-wheels/settings (our only remaining Travis dependency), and enabled a monthly cron:

image

@aclark4life
Copy link
Member

@hugovk Yeah reads more like "OOPS we got called out, and now here's some credits" … but yeah, give it a whirl, while at the same time not relying too heavily on it, I'd say.

@hugovk
Copy link
Member Author

hugovk commented Sep 9, 2021

Let's close this now, we've moved nearly everything to GitHub Actions, and Travis CI is ticking over okay for where we are using it.

https://app.travis-ci.com/organizations/python-pillow/plan currently looks like:

image

Which is the same as April #5028 (comment) 🤷

OSS-only tab:

image

@hugovk
Copy link
Member Author

hugovk commented Jan 28, 2022

Another ~5 months later, the numbers haven't changed, and the same as April 2021:

https://app.travis-ci.com/organizations/python-pillow/plan

image

image

@radarhere
Copy link
Member

Just to state it in this issue - if Travis CI ever does run out of credits, python-pillow/pillow-wheels#247 can be used to move the aarch64 jobs to GitHub Actions, albeit with a much longer execution time.

@hugovk
Copy link
Member Author

hugovk commented Feb 5, 2023

In python-pillow/pillow-wheels#367 @radarhere noticed Travis CI is playing up again.

  • Luckily, that PR is the backup plan, we can build those aarch64 wheels on GitHub Actions instead using emulation.
  • Unluckily, instead of each wheel taking about 10 minutes to build on Travis CI, they each take between 1.5 and 4 hours on GHA!

Plan A is to get Travis CI working again.

At https://app.travis-ci.com/github/python-pillow/pillow-wheels, it says:

image

Builds have been temporarily disabled for public repositories due to a negative credit balance. Please go to the Plan page to replenish your credit balance or alter your Consume paid credits for OSS setting.

"More options" > "Requests" confirms that recent builds have been cancelled: "Owner python-pillow does not have enough credits." 🤷

OK, so let's go to the Plan page.

Nope! 🤷

image

There was an error, please try again.

Trying again makes no difference. 🤷 Same thing for the link to "Consume paid credits for OSS setting". 🤷

Well, I see the same error on my own page at https://app.travis-ci.com/github/hugovk/pillow-wheels, but do have "Credits" although zero "OSS only credits". Neither are negative... 🤷

image image

Anyway, I also see this:

image

No Pillow listed. 🤷 Maybe I need to "Review and add your authorized organizations".

But https://github.com/apps/travis-ci/installations/12696639 shows it's already installed for the wheels repo: 🤷

image

At this point I give up and will contact Travis CI support...

@hugovk
Copy link
Member Author

hugovk commented Feb 6, 2023

Good news! Quickly fixed by Travis support:

Thanks for reaching out and very sorry for the issues described here. When you are ready please push a build and let us know if these issues now appear to be resolved.

Very sorry for the hassle once again and we will follow up on your reply.

Pushing a new build now works and the red banner has been removed:

However, I can't see the plan page, nor trigger or restart builds. I've replied to ask about that.

@hugovk
Copy link
Member Author

hugovk commented Sep 29, 2023

However, I can't see the plan page, nor trigger or restart builds. I've replied to ask about that.

After some back and forth, the response was that only admins can see the plan. And I could trigger and restart builds after logout/login/resync.

@hugovk
Copy link
Member Author

hugovk commented Sep 29, 2023

Cross post from #7418 (comment):

Screenshot 2023-09-28 at 7 18 02 AM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants