apt-get update fails #2919
Comments
dibir-magomedsaygitov
added
OS: Ubuntu
investigate
|
Hey, @newhoggy ! |
|
We've started getting this on all our actions for the past 30 minutes or so |
|
Also seeing this on all of our actions. @LeonidLapshin do you have an update on this? I see you said you thought it was a temporary issue but it appears to be occurring again. |
|
This hit me too. https://github.com/scottgigante/SingleCellOpenProblems/runs/2113835105?check_suite_focus=true Fixed it by moving to ubuntu-18.04. |
|
Happened on both on:
workflow_dispatch:
jobs:
test:
runs-on: ubuntu-18.04
steps:
- run: sudo apt update |
|
Seems that the problem is somehow connected with Opensuse repositories, because my test builds on Ubuntu 20 are successful, but there is definitely a percent of errors and all of them are: |
|
Just to share very quickly, here's a workaround if you just need to get workflows running again: (I posted with more context in #2938, but that was closed as a duplicate of this.) |
|
Yeah, it's happening for me again. https://github.com/input-output-hk/cardano-node/runs/2113990824?check_suite_focus=true |
2489: Remove problematic apt-get repository r=newhoggy a=newhoggy `apt-get` can sometimes fail due to a problematic apt-get repository. See actions/runner-images#2919 Remove this repository since we don't use it anyway to work around the problem. Co-authored-by: John Ky <john.ky@iohk.io>
2997: Remove problematic apt-get repository r=dcoutts a=newhoggy `apt-get update` can sometimes fail due to a problematic apt-get repository. See actions/runner-images#2919 Remove this repository since we don't use it anyway to work around the problem. Co-authored-by: John Ky <john.ky@iohk.io>
|
I think GitHub should really strongly consider removing or commenting out some of the apt sources that they're adding to the images. Right now, an If someone is running Maybe I'm in the minority, but I would be 100% willing to put up with fixing any of my org's builds to add needed repos if they're removed, in exchange for less flaky The whole set could be cut down to something like That's that many fewer ways for an update to fail. Plus, since almost everyone running workflows doesn't want to update the version of mongo, or podman, or whatever else is installed, this is a significant amount of network traffic which can be removed. Either that or mirror more of these packages to a repo owned and operated by GitHub / Microsoft, so they can control things. |
|
@sirosen exactly! MS should not add anything to what Ubuntu ships with. All of these extra PPAs should be removed. Alternative: provide a ubuntu-20.04-vanilla. I'd then switch to that.
mirroring wouldn't cut it for me: it is not only about trusting operationally the upstream PPA ops, but that plus trusting the upstream developers feeding into those PPAs. eg "dl.bintray.com" gives me shivers .. the idea that the build box we use in CIs at any time could be fed crap/attacks |
|
Hi, @newhoggy ! |
|
@LeonidLapshin no, I think this is only related to MS specific infrastructure, and I think MS should not try to blame SuSE for screwing up. In fact, my suspicion is that MS is running some caching and/or TLS intercepting middleboxes. The latter would be even more worrying. The reason is that this happened twice in the last weeks, for different PPAs, at least:
I think it is unlikely the those 2 different and independent upstream PPAs would screw up TLS certificate updates within just two weeks. All of this makes me nervous enough to think about moving all our runners to self-hosted. A security-wise broken CI/CD pipeline is like a compromised supply chain: you are dead without knowing. |
|
Hey, @oberstet !
download.opensuse.org uses a wildcard cert issued by Letsencrypt and the OSCP verification URL is
Somehow this web-server returned invalid response (I guess that all software sometimes throw errors) and the connection was not succeded. That can be caused by almost everything in the middle, as you can see that the problem does not reproduces every time so it's really hard to say something definite. Just an assumption:
on March, 7 and 17 respectively, related to
I guess that Letsencrypt infra had some troubles with this scope of web services and this can be the root cause of apt-get update failure, hope these details are efficient enough. |
|
Thanks for checking and having a close look! Obviously, I don't know what really happened. As a matter of fact, I just ran the command you posted, and it works just fine (now). pls see below.
Like Windows? ;) seriously, I hope the GitHub infrastructure still runs on Linux. anyways, I don't "believe" LE is to blame: I did check the failing HTTPS link in Chromium/Ubuntu at the time when it was broken in the CI: no problems. Sure, certificate stapling and online revocation checking .. is complex. All the x509 stuff is actually deeply broken by design .. I have some strong opinions here .. sorry;) But as said: I do not know what actually happened .. maybe LE was the origin of the problem, maybe, by co-incidence both SuSE and bintray.com. I don't know. Finally: all of that actually doesn't matter, because the real solution IMO is to remove all PPAs. Just what Ubuntu ships. Done.
|
Description
apt-get updatefails withCertificate verification failed.Area for Triage:
Bug:
See https://github.com/input-output-hk/ouroboros-network/runs/2102194655?check_suite_focus=true
Virtual environments affected
Image version
Environment: ubuntu-20.04
Version: 20210302.0
Included Software: https://github.com/actions/virtual-environments/blob/ubuntu20/20210302.0/images/linux/Ubuntu2004-README.md
Image Release: https://github.com/actions/virtual-environments/releases/tag/ubuntu20%2F
Expected behavior
This should not fail.
Actual behavior
apt-get updatefails withCertificate verification failed.Repro steps
For repository and create PR.
The text was updated successfully, but these errors were encountered: