Go: Transitive dependency failures block Dependabot from opening any PR's

[evanfuller]

Package ecosystem
gomod
Package manager version
1
Language version
N/A
Manifest location and content prior to update

The manifest lives in a private repo, so I will give an abbreviated example:

module github.com/someorg/somemodule

require (
	github.com/auth0-community/go-auth0 v0.0.1
	github.com/aws/aws-sdk-go v1.0.1
)

dependabot.yml content

version: 2
updates:
  - package-ecosystem: "gomod"
    directory: "/"
    schedule:
      interval: "daily"

What you expected to see, versus what you actually saw

In the example I gave, I would expect a pull request to be opened on my repository to bump the AWS dependency to the latest, since there are many newer versions than 1.0.0. However, the fact that the go-auth0 dependency has been deleted off of Github causes Dependabot to print a bunch of error logs like

proxy | 2022/01/04 19:43:37 [060] 404 https://github.com:443/auth0-community/go-auth0

before eventually giving up on the retries. No pull requests get opened, but I would expect that despite this partial failure, dependabot could still open one for the aws-sdk dependency.

This is of course a toy example: since the dependency has been deleted from GitHub, there is no reason for me to list github.com/auth0-community/go-auth0 in my go.mod file. The challenge comes from the fact that in production, I have a version-tagged gomod dependency github.com/myorg/myproject v1.1.4, which itself has a dependency on github.com/auth0-community/go-auth0: since the auth0 repo has been deleted, dependabot fails to open any PR's.

[jeffwidman]

What version of go is specified in your go.mod file?

[evanfuller]

This occurs across several repos in our organization, but most are on 1.16 I believe.

[DasSkelett]

We've hit a similar issue in the past due to deleted versions, but for a direct dependency even: freifunkMUC/wg-embed#3

go: golang.zx2c4.com/wireguard@v0.0.20201118: reading golang.zx2c4.com/wireguard/go.mod at revision v0.0.20201118: unknown revision v0.0.20201118

Ideally Dependabot could recover itself from this situation and just update the version to the latest one, if it can't fetch a differential changelog / commit log in this case so be it, better than being stuck forever.

Technically a different issue with direct<->indirect dependencies, but I think it's the same category of "Dependabot should keep (mostly) working if it encounters non-fatal errors".

[jeffwidman]

👋 sorry for the slow response.

A couple of thoughts:

• We've improved support for transitive dependencies, for example [gomod] Support updating indirect dependencies #6361 although note that you'll need to configure it as bumping transitive deps isn't enabled by default.
• The issue here is a broken transitive dep... most of what we do in Dependabot is actually glue code. This is particularly true for the go_modules ecosystem... some other ecosystems Dependabot has to do more than just glue code, but for go almost all the heavy lifting happens in the native tooling via go get / go mod tidy. So if the native tools are failing, so will Dependabot.
• For most package managing, it's not safe to ignore an unreachable dep, because it may have a constraint that prevents updating another dep. If we knew it was safe to ignore, then yes, we should skip, but if there's a significant chance of opening an incorrect PR, we'd rather error. go tooling has the same philosophy IIRC.
• It's been about a year since I worked heavily in go, but IIRC, the problem generally here was that some library somewhere in the dep graph was still actively pinning the unreachable dep, so needed to be manually updated by a human.
• When I hit this, what also helped a lot was switching to the newer go.mod format of go 1.17 which heavily pruned the dep graph to remove transitive deps that were actually unused... so many of those outdated deps simply got pruned completely away.

I don't think there's much we can / should do here, so I'm going to close as won't fix, but if for some reason I'm wrong, please comment and we can re-open.