-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Did Dockerhub rate limit affect Github Action? #1445
Comments
Hello @BrightRan , thank you for raising it. I have found the following Docker use-cases in GA (I could miss something): jobs:
my_first_job:
steps:
- run: docker pull alpine:3.11 This use-case allow customers to do login by theirselves: steps:
- run: |
docker login --username ${{ secrets.docker_username }} --password ${{ secrets.docker_password }}
docker pull alpine:3.11 jobs:
container:
runs-on: ubuntu-latest
container: node:10.16-jessie
steps:
- run: |
echo This job does specify a container.
echo It runs in the container instead of the VM.
name: Run in container Looks like pull of such container is happening at the first step of pipeline and we can't login before somehow. jobs:
my_first_job:
runs-on: ubuntu-latest
steps:
- uses: docker://alpine:3.11 this use case will be affected too steps:
- run: docker login
- uses: docker://alpine:3.11 Container pull happens at the first step of the build before docker login jobs:
my_first_job:
runs-on: ubuntu-latest
steps:
- uses: my_custom_action_that_uses_docker_inside@v1 In the first use-case, customer can overcome limit by login to docker as a first step of pipeline: DockerHub will use IP rate limit so our customers could be affected. I don't have much experience with docker so could be wrong, may be actions/runner team can help us to understand if there is a way to login in use-cases 2,3,4. cc: @hross , @ericsciple , @TingluoHuang |
@maxim-lobanov , |
@BrightRan , for 4 - yes. steps:
- run: docker login
- uses: docker://alpine:3.11 |
@maxim-lobanov , |
The great thing about this thread is... We are actually actively working on providing more native login capabilities in actions. See this ADR. This should make it easier for customers to set up authentication for scenarios @dakale is adjusting the syntax with @chrispat (today, actually). We should have this ready to go by the end of the month at the latest, which should mean better support before the docker limits are imposed. Hey @dakale can you add each of these examples to your ADR (1-4) and explain how they can be addressed with the new syntax? |
@hross , Happy to hear that we are working on feature that will fix the issue. |
I agree with the scenarios as described by @maxim-lobanov , I think it accurately reflects all the ways we support containers. This only affects For I should note that (due to my mistake), the link to the ADR @hross posted is out of date. We have iterated on it since, and the change is significant relative to this issue. Check the PR for the most up to date document: https://github.com/github/c2c-actions-runtime/pull/813/files That ADR, as currently designed, only solves I have no way to gauge what the impact of the rate limiting will be, since I dont know the range of IPs used for the hosted runner vms or how much of the available rate they might consume, but 100 per 6 hours is very very low. Its hard to make a judgement on whether its worth revising the ADR again to account for rate limiting, but I do think its worth considering given how low the limit is. |
We're also thinking about this in Azure Pipelines. We already have a technical solution for scenarios 2 and 3 (container resources can use a service connection for authentication). Most of our current usage is anonymous, though, so we'll have to help customers understand the impact and adapt. |
Judging from the ADRs being private, will GitHub users not get to see the syntax until it's out (and can't have breaking changes)? Since the rate limit would almost definetely affect users, would it be worth adding a deprecation notice once a fix is available eventually forcing all users to pull while being logged in? Could help solve problems before they arise if people new to GitHub Actions start using it and expect it to work without knowing about the limit or that public CIs don't get some kind of whitelist |
@nihaals the syntax is very simple as we are simply adding a credentials element to the |
I made what I think is a related feature request that may make it nicer to login with Actions which sounds like will work with the new syntax What do you think about the deprecation notice on unauthorised Docker Hub images? Probably blocked by finding out if official images are exempt and the new syntax |
As I understand it one scary and unresolved problem with (1) is going to be sharing Hub credentials with forked repos. Any solution that involves a Does the solution that's being worked on address this aspect? |
At the moment we do not have a solution to that particular problem but we are investigating options. |
After the latest GitHub update, you can specify credentials for container downloading: https://github.blog/changelog/2020-09-24-github-actions-private-registry-support-for-job-and-service-containers/ |
I read the blog post. The new feature, being able to authenticate with registries, looks like it will have the nice side effect of solving the rate limiting problem for job containers and service containers by allowing people to authenticate, which would stop them from being forced to share the rate limit pool with other GitHub Actions users. However, do you know whether it will work for Docker container actions? For example, from the docs (https://docs.github.com/en/free-pro-team@latest/actions/reference/workflow-syntax-for-github-actions#example-using-a-docker-hub-action): jobs:
my_first_job:
steps:
- name: My first step
uses: docker://alpine:3.8 In this example, the image used for the Docker container action is being pulled by the GitHub Actions runner from Docker Hub. Because credentials aren't specified for the step, I'm assuming this is being done without authenticating with Docker Hub. The images used for Docker container actions must be public, which means people don't have to authenticate to use them, but if they choose not to authenticate as they use them, they'll be subject to the IP-based rate limiting, affected by other GitHub Actions users. Is this correct? |
@chrispat sorry to ask again, but it's getting worryingly close to November and, as far as I can tell, there's still no safe way for open source projects to use authenticated access to Docker Hub. Has there been any progress in finding a way for forks of repos to be able to inherit Docker Hub authentication from the original repo? |
For publicly accessible containers we are working with docker hub to make sure you will not be impacted by the new rate limits. If you need private containers we still do not have a solution for that problem for forks of public repos. |
Thank you @chrispat! As a maintainer of an open source project that uses public Docker images, that's really good news. |
@chrispat just to confirm, does this mean we will NOT need to add dockerhub auth to all of our jobs that pull public dockerhub images? |
That is correct. It is important to note that this will only apply to the runners hosted by GitHub, if you are using self-hosted runners you will need to configure your own credentials. |
Thank you for the confirmation! :) |
@smcgivern Did they share any timeline? Is this a known issue they are working to fix? We started using the ARM runners and ran into this problem. |
They did not, sorry. They said they had no ETA currently. |
Seem to be hitting this today; our runners are not special (not ARM, not GPU). Hoping it's just a temporary blip, but raising it in case others are also impacted. |
We are hitting this error today on public shared runners trying to pull a public Docker Hub image, https://hub.docker.com/_/postgres, so the rate limits are still in effect as far as we can see. Pretty annoying given that there's no easy way to cache pulled images on GitHub Actions either. We're having to resort to workarounds like authenticating with Docker Hub with a bot account, or trying to move off of Docker Hub as much as possible. |
I don't think it's unreasonable to be required to authenticate with Docker Hub. If I were hosting a free service providing Docker images, I would find it helpful to know who's using up my bandwidth! I wonder if GitHub could provide credentials for a "GitHub Actions" Docker Hub account to all GitHub Actions workflows like they do with GitHub could even coordinate with Docker so that Docker could impose a rate limit on this account if they want to treat it differently, considering it would represent multiple users instead of just one user. This would allow users to start using Docker Hub with less of a chance of running into this issue without having to set up their own Docker Hub credentials in their workflows. |
@mattwelke I would not have an issue with reasonable rate limits if GitHub Actions made it reasonably easy to cache pulled images. Fifteen minutes of searching gave me no results other than https://github.com/marketplace/actions/docker-cache, which is an unofficial action that I would rather not have to trust. I'm not assigning blame here, as both GitHub Actions and Docker Hub are largely free services, but it's a bit silly that their interaction runs into rate limits for no reason. |
All of the shared public runners are automatically logged in to docker hub with a shared account that has significantly higher rate limits on Docker Hub. I just ran a job that pulled that image and was not able to reproduce the issue. |
This issue is sporadic, it "fixes" itself after re-running the job, so it's to be expected that it can't be easily reproduced. |
We have seen this issue sporadicaly and regularly over the last few months. |
If you are seeing this issue and it's in a public repo could you link me to workflow runs please so I can see date/time/other run info so we can dig in <3 |
We see this too, unfortunately not on a public repo... this is the raw log from the run:
And a relevant part of the workflow:
On a side-note: |
@nebuk89 Was the comment above useful? (i.e. should I post the moar logs if we see this happening again) |
Got the `toomanyrequests' error on a dokku Github actions pipeline:
Hope this helps :) |
this hitting me on a private repo too. impossible to know exactly why/how. |
@nebuk89, Seen this today too when attempting to pull job link -> https://github.com/nofusscomputing/centurion_erp/actions/runs/11330393667/job/31508120435. |
Same here, getting the rate limit error on our private repo while trying to pull
|
Also just started getting this today. |
I had docker pull rate llimit issues when I was authenticated to Docker Hub. Since today: Pull issues again... i'd never seen rate limit issue from GitHub Actions with unauthenticated user :/ |
Also getting this today on a private repo! |
+1. I'm facing the same issue while fetching public images from the Docker Registry on my GHA Workflows. |
This was an issue on docker's side docker/hub-feedback#2414 |
Associated community ticket: https://gh.neting.ccmunity/t/did-dockerhub-rate-limit-affect-github-action/128158
The customer noticed that Docker hub has updated their pricing, Download rate limit and retention policy recently. He is wondering if the download rate limit affects on Github Action.
Will the steps like below are affected?
According to the introductions from the docs about "Download rate limit", the limit seems only applies for the anonymous users with an eventual limit of 100 pulls per six hours. Logged in users will not be affected at this time.
I setup a workflow with the following step that pull an image 101 times in a loop.
It can successfully download the image 101 times without any warning or error messages about the download rate limit.
Note: Consider there may be image cache, I have used the "docker image rm" command to remove the image every time before downloading it again.
Question:
When we try to pull some public images in the workflow, if we do not login with any of our accounts, whether GitHub has a default account to login the docker?
If not, in my above example, why it is not hit the download rate limit?
The text was updated successfully, but these errors were encountered: