Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

apt-get update fails #2919

Closed
1 of 7 tasks
newhoggy opened this issue Mar 13, 2021 · 16 comments
Closed
1 of 7 tasks

apt-get update fails #2919

newhoggy opened this issue Mar 13, 2021 · 16 comments
Assignees
Labels
Area: Scripting and command line investigate Collect additional information, like space on disk, other tool incompatibilities etc. OS: Ubuntu

Comments

@newhoggy
Copy link

Description
apt-get update fails with Certificate verification failed.

Err:48 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04  Release
  Certificate verification failed: The certificate is NOT trusted. The received OCSP status response is invalid.  Could not handshake: Error in the certificate verification. [IP: 195.135.221.134 443]

Area for Triage:

Bug:

Run sudo apt-get update
  sudo apt-get update
  sudo apt-get -y install libsodium23 libsodium-dev
  sudo apt-get -y install libsystemd0 libsystemd-dev
  sudo apt-get -y remove --purge software-properties-common
  sudo apt-get -y autoremove
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    CABAL_VERSION: 3.4.0.0
    CABAL_BUILDDIR: dist-newstyle
    CACHE_VERSION: 9w76Z3Q
Hit:1 http://azure.archive.ubuntu.com/ubuntu focal InRelease
Get:2 http://azure.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:3 http://azure.archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB]
Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [109 kB]
Get:5 http://dl.google.com/linux/chrome/deb stable InRelease [1811 B]
Hit:6 https://packages.microsoft.com/repos/azure-cli focal InRelease
Get:7 https://packages.microsoft.com/ubuntu/20.04/prod focal InRelease [10.5 kB]
Get:8 https://cli-assets.heroku.com/apt ./ InRelease [2879 B]
Hit:10 https://storage.googleapis.com/bazel-apt stable InRelease
Get:11 https://packages.cloud.google.com/apt cloud-sdk InRelease [6739 B]
Get:12 https://download.mono-project.com/repo/ubuntu stable-focal InRelease [4416 B]
Ign:13 https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/4.4 InRelease
Get:14 https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/4.4 Release [5372 B]
Get:15 https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/4.4 Release.gpg [801 B]
Hit:16 http://ppa.launchpad.net/apt-fast/stable/ubuntu focal InRelease
Get:9 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9383 B]
Get:17 http://ppa.launchpad.net/git-core/ppa/ubuntu focal InRelease [23.8 kB]
Get:19 http://azure.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [844 kB]
Ign:20 https://dl.bintray.com/sbt/debian  InRelease
Get:21 http://azure.archive.ubuntu.com/ubuntu focal-updates/main Translation-en [202 kB]
Get:22 http://azure.archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [12.8 kB]
Get:23 http://azure.archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [156 kB]
Get:24 http://azure.archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [23.3 kB]
Get:25 http://azure.archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [750 kB]
Get:26 http://azure.archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [157 kB]
Get:27 http://azure.archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [16.3 kB]
Hit:28 https://dl.yarnpkg.com/debian stable InRelease
Hit:29 http://ppa.launchpad.net/hvr/ghc/ubuntu focal InRelease
Get:30 https://apt.postgresql.org/pub/repos/apt focal-pgdg InRelease [81.6 kB]
Get:31 http://dl.google.com/linux/chrome/deb stable/main amd64 Packages [1084 B]
Hit:32 http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu focal InRelease
Hit:18 https://packagecloud.io/github/git-lfs/ubuntu focal InRelease
Get:33 https://packages.microsoft.com/ubuntu/20.04/prod focal/main amd64 Packages [61.0 kB]
Ign:34 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04  InRelease
Get:35 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [528 kB]
Get:36 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [114 kB]
Get:37 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [7264 B]
Get:38 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [133 kB]
Get:39 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [19.5 kB]
Get:40 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [550 kB]
Get:41 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [80.7 kB]
Get:42 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [10.6 kB]
Get:43 https://cli-assets.heroku.com/apt ./ Packages [620 B]
Get:44 https://dl.bintray.com/sbt/debian  Release [815 B]
Get:45 https://packages.cloud.google.com/apt cloud-sdk/main amd64 Packages [163 kB]
Get:46 https://download.mono-project.com/repo/ubuntu stable-focal/main amd64 Packages [46.8 kB]
Get:47 https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/4.4/multiverse amd64 Packages [8916 B]
Err:48 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04  Release
  Certificate verification failed: The certificate is NOT trusted. The received OCSP status response is invalid.  Could not handshake: Error in the certificate verification. [IP: 195.135.221.134 443]
Get:49 http://ppa.launchpad.net/git-core/ppa/ubuntu focal/main amd64 Packages [3056 B]
Get:50 https://apt.postgresql.org/pub/repos/apt focal-pgdg/main amd64 Packages [191 kB]
Reading package lists...
E: The repository 'https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04  Release' no longer has a Release file.
Error: Process completed with exit code 100.

See https://github.com/input-output-hk/ouroboros-network/runs/2102194655?check_suite_focus=true

Virtual environments affected

  • Ubuntu 16.04
  • Ubuntu 18.04
  • Ubuntu 20.04
  • macOS 10.15
  • macOS 11.0
  • Windows Server 2016 R2
  • Windows Server 2019

Image version
Environment: ubuntu-20.04
Version: 20210302.0
Included Software: https://github.com/actions/virtual-environments/blob/ubuntu20/20210302.0/images/linux/Ubuntu2004-README.md
Image Release: https://github.com/actions/virtual-environments/releases/tag/ubuntu20%2F

Expected behavior
This should not fail.

Actual behavior
apt-get update fails with Certificate verification failed.

Repro steps
For repository and create PR.

@dibir-magomedsaygitov dibir-magomedsaygitov added OS: Ubuntu investigate Collect additional information, like space on disk, other tool incompatibilities etc. and removed needs triage labels Mar 15, 2021
@LeonidLapshin LeonidLapshin self-assigned this Mar 15, 2021
@LeonidLapshin
Copy link
Contributor

Hey, @newhoggy !
Seems that the problem was temporary and does not persist now. Could you please check it one more time?

@jdelStrother
Copy link

We've started getting this on all our actions for the past 30 minutes or so

@jorgensen-eric-PFG
Copy link

jorgensen-eric-PFG commented Mar 15, 2021

Also seeing this on all of our actions. @LeonidLapshin do you have an update on this? I see you said you thought it was a temporary issue but it appears to be occurring again.

@scottgigante
Copy link

This hit me too. https://github.com/scottgigante/SingleCellOpenProblems/runs/2113835105?check_suite_focus=true

Fixed it by moving to ubuntu-18.04.

@0x2b3bfa0
Copy link

Happened on both ubuntu-18.04 and ubuntu-20.04 earlier today, but seems to be fixed now.

on:
  workflow_dispatch:

jobs:
  test:
    runs-on: ubuntu-18.04
    steps:
    - run: sudo apt update

@LeonidLapshin
Copy link
Contributor

Seems that the problem is somehow connected with Opensuse repositories, because my test builds on Ubuntu 20 are successful, but there is definitely a percent of errors and all of them are: https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04 Release' no longer has a Release file.
Opensuse statuspage is green so I guess that the problem should be rare or temporary.

@sirosen
Copy link

sirosen commented Mar 15, 2021

Just to share very quickly, here's a workaround if you just need to get workflows running again:

sudo rm /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
sudo apt-get update -y

(I posted with more context in #2938, but that was closed as a duplicate of this.)

@newhoggy
Copy link
Author

iohk-bors bot added a commit to IntersectMBO/cardano-node that referenced this issue Mar 16, 2021
2489: Remove problematic apt-get repository r=newhoggy a=newhoggy

`apt-get` can sometimes fail due to a problematic apt-get repository.  See actions/runner-images#2919

Remove this repository since we don't use it anyway to work around the problem.

Co-authored-by: John Ky <john.ky@iohk.io>
iohk-bors bot added a commit to IntersectMBO/ouroboros-network that referenced this issue Mar 16, 2021
2997: Remove problematic apt-get repository r=dcoutts a=newhoggy

`apt-get update` can sometimes fail due to a problematic apt-get repository.  See actions/runner-images#2919

Remove this repository since we don't use it anyway to work around the problem.

Co-authored-by: John Ky <john.ky@iohk.io>
@scottgigante
Copy link

@sirosen
Copy link

sirosen commented Mar 16, 2021

I think GitHub should really strongly consider removing or commenting out some of the apt sources that they're adding to the images.

Right now, an apt update will hit these (if I'm grepping right):

azure.archive.ubuntu.com
dl.google.com
ppa.launchpad.net
security.ubuntu.com
apt.postgresql.org
cli-assets.heroku.com
dl.bintray.com
dl.yarnpkg.com
download.mono-project.com
download.opensuse.org
packagecloud.io
packages.cloud.google.com
packages.microsoft.com
repo.mongodb.org
storage.googleapis.com

If someone is running apt update, it's probably because they want to install something which isn't already installed. If it's an Ubuntu package it should be supported, but for anything else users can add the repos themselves. (I don't even know if the VMs have universe packages enabled, but I wouldn't even expect that as a user. Only enabling the main repo seems pretty reasonable.)

Maybe I'm in the minority, but I would be 100% willing to put up with fixing any of my org's builds to add needed repos if they're removed, in exchange for less flaky apt update.

The whole set could be cut down to something like

azure.archive.ubuntu.com
security.ubuntu.com
ppa.launchpad.net

That's that many fewer ways for an update to fail.

Plus, since almost everyone running workflows doesn't want to update the version of mongo, or podman, or whatever else is installed, this is a significant amount of network traffic which can be removed.

Either that or mirror more of these packages to a repo owned and operated by GitHub / Microsoft, so they can control things.

@maxim-lobanov
Copy link
Contributor

@sirosen , In general, I agree since it is not the first time when such big number of apt repos bring issues. But we need to investigate it carefully before. I have created the separate issue to track this investigation: #2951

@oberstet
Copy link

@sirosen exactly!

MS should not add anything to what Ubuntu ships with. All of these extra PPAs should be removed. Alternative: provide a ubuntu-20.04-vanilla. I'd then switch to that.

mirror more of these packages to a repo owned and operated by GitHub / Microsoft, so they can control thing

mirroring wouldn't cut it for me: it is not only about trusting operationally the upstream PPA ops, but that plus trusting the upstream developers feeding into those PPAs.

eg "dl.bintray.com" gives me shivers .. the idea that the build box we use in CIs at any time could be fed crap/attacks

@LeonidLapshin
Copy link
Contributor

Hi, @newhoggy !
I'm still sure that this problem is highly related to Opensuse, repository list on Ubuntu images will be discussed/changed within the Investigate reducing the count of apt sources on Hosted Ubuntu images issue, so I'll close this one, but feel free to reopen it if you have any concerns.
Thank you!.

@oberstet
Copy link

@LeonidLapshin no, I think this is only related to MS specific infrastructure, and I think MS should not try to blame SuSE for screwing up.

In fact, my suspicion is that MS is running some caching and/or TLS intercepting middleboxes. The latter would be even more worrying.

The reason is that this happened twice in the last weeks, for different PPAs, at least:

  • dl.bintray.com
  • download.opensuse.org

I think it is unlikely the those 2 different and independent upstream PPAs would screw up TLS certificate updates within just two weeks.

All of this makes me nervous enough to think about moving all our runners to self-hosted. A security-wise broken CI/CD pipeline is like a compromised supply chain: you are dead without knowing.

@LeonidLapshin
Copy link
Contributor

Hey, @oberstet !
We've taken a closer look to these issues and it seems that cases are different:

  • dl.bintray.com returns 502 error code (server-side) in this case the connection was succeeded, but unfortunately we got 502 error
  • download.opensuse.org can not establish https connection due to OSCP error

download.opensuse.org uses a wildcard cert issued by Letsencrypt and the OSCP verification URL is http://r3.i.lencr.org/


PS C:\Users\Leonid> openssl s_client -connect download.opensuse.org:443 -servername download.opensuse.org -status
CONNECTED(000001A4)
depth=1 C = US, O = Let's Encrypt, CN = R3
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = opensuse.org
verify return:1
OCSP response:
======================================
OCSP Response Data:
    OCSP Response Status: successful (0x0)
    Response Type: Basic OCSP Response
    Version: 1 (0x0)
    Responder Id: C = US, O = Let's Encrypt, CN = R3

opensuse-letsencrypt

Somehow this web-server returned invalid response (I guess that all software sometimes throw errors) and the connection was not succeded. That can be caused by almost everything in the middle, as you can see that the problem does not reproduces every time so it's really hard to say something definite.

Just an assumption:
If you look at Letsencrypt status page you will see that there was

Degraded Performance...

...some maintenance on our backend infrastructure...

on March, 7 and 17 respectively, related to

acme-v01.api.letsencrypt.org (Production), acme-v02.api.letsencrypt.org (Production), ocsp.root-x1.letsencrypt.org, {e1,e2,r3,r4}.o.lencr.org & ocsp.int-x{3..4}.letsencrypt.org".

I guess that Letsencrypt infra had some troubles with this scope of web services and this can be the root cause of apt-get update failure, hope these details are efficient enough.

@oberstet
Copy link

Thanks for checking and having a close look!

Obviously, I don't know what really happened. As a matter of fact, I just ran the command you posted, and it works just fine (now). pls see below.

That can be caused by almost everything in the middle

Like Windows? ;) seriously, I hope the GitHub infrastructure still runs on Linux. anyways, I don't "believe" LE is to blame: I did check the failing HTTPS link in Chromium/Ubuntu at the time when it was broken in the CI: no problems.

Sure, certificate stapling and online revocation checking .. is complex. All the x509 stuff is actually deeply broken by design .. I have some strong opinions here .. sorry;)

But as said: I do not know what actually happened .. maybe LE was the origin of the problem, maybe, by co-incidence both SuSE and bintray.com. I don't know.

Finally: all of that actually doesn't matter, because the real solution IMO is to remove all PPAs. Just what Ubuntu ships. Done.

Bildschirmfoto von 2021-03-18 18-37-36

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: Scripting and command line investigate Collect additional information, like space on disk, other tool incompatibilities etc. OS: Ubuntu
Projects
None yet
Development

No branches or pull requests

10 participants