/bin/sh: 1: go: not found error on managed Arm64 hosts

[ahrtr]

Which github workflows are flaking?

I only see this issue once. It succeeded after retrying.

https://github.com/etcd-io/bbolt/actions/runs/6881283997/job/18717321172?pr=611

BBOLT_VERIFY=all TEST_FREELIST_TYPE=hashmap go test -v -race=false -cpu=2  -timeout 30m
/bin/sh: 1: go: not found
make: *** [Makefile:45: test] Error 127
Error: Process completed with exit code 2.

cc @alexellis @jmhbnz

Which tests are flaking?

managed Arm64 hosts

Github Action link

No response

Reason for failure (if possible)

No response

Anything else we need to know?

No response

[jmhbnz]

Based on the logs it looks like the setup-go action is failing, but not reporting any kind of failure.

Here is what we see for the failed setup:

Run actions/setup-go@v4
  with:
    go-version: 1.21.4
    check-latest: false
    token: ***
    cache: true
Setup go version spec 1.21.4
Attempting to download 1.21.4...
matching 1.21.4...
Not found in manifest.  Falling back to download directly from Go

And here is what a successful setup looks like:

Run actions/setup-go@v4
Setup go version spec 1.21.4
Attempting to download 1.21.4...
matching 1.21.4...
Not found in manifest.  Falling back to download directly from Go
Install from dist
Acquiring go1.21.4 from https://storage.googleapis.com/golang/go1.21.4.linux-arm64.tar.gz
Extracting Go...
/usr/bin/tar xz --warning=no-unknown-keyword --overwrite -C /home/runner/actions-runner/_work/_temp/0b2d25d8-64aa-41bb-aa31-38721a6c7844 -f /home/runner/actions-runner/_work/_temp/231394a6-e69d-4374-9732-9521a15a713d
Successfully extracted go to /home/runner/actions-runner/_work/_temp/0b2d25d8-64aa-41bb-aa31-38721a6c7844
Adding to the cache ...
Successfully cached go to /home/runner/actions-runner/_work/_tool/go/1.21.4/arm64
Added go to the path
Successfully set up Go version 1.21.4
/home/runner/actions-runner/_work/_tool/go/1.21.4/arm64/bin/go env GOMODCACHE
/home/runner/actions-runner/_work/_tool/go/1.21.4/arm64/bin/go env GOCACHE
/home/runner/go/pkg/mod
/home/runner/.cache/go-build
Received 29360128 of 76088131 (38.6%), 28.0 MBs/sec
Received 76088131 of 76088131 (100.0%), 49.6 MBs/sec
Cache Size: ~73 MB (76088131 B)
/usr/bin/tar -xf /home/runner/actions-runner/_work/_temp/a7d91216-e232-4557-a274-b032e4796170/cache.tzst -P -C /home/runner/actions-runner/_work/bbolt/bbolt --use-compress-program unzstd
Cache restored successfully
Cache restored from key: setup-go-Linux-undefined-go-1.21.4-72bed14912b79c5b708e654fdb3ab3d8fe8a6ff4512544aded414c0a936c2375
go version go1.21.4 linux/arm64

I'm not seeing any related open issues in https://github.com/actions/setup-go/issues or in https://docs.actuated.dev/troubleshooting so I think we will need someone from the actuated team to take a look at this.

@alexellis - Can someone from the actuated team please take a look at this?

[alexellis]

We use the setup-go action extensively with actuated, openfaas and inlets (various versions of it) and haven't experienced issues with it.

Is there a minimal repro here? cc @welteki

Was this seen only once?

I only see this issue once. It succeeded after retrying.

[ahrtr]

Was this seen only once?

YES, I only saw it once.

[alexellis]

Hi @ahrtr I'd suggest closing this issue and monitoring it.

I created a repro and couldn't reproduce the problem.

First run:

https://github.com/actuated-samples/checkout-go/actions/runs/6928200904/job/18843522679

Second run:

https://github.com/actuated-samples/checkout-go/actions/runs/6928217399/job/18843574018

Workflow:

https://github.com/actuated-samples/checkout-go/blob/master/.github/workflows/ci.yaml

Alex

[ahrtr]

OK, thx.

[ahrtr]

See the error again,

https://github.com/etcd-io/etcd/actions/runs/6946758921/job/18899004213?pr=16909

[alexellis]

Can you reproduce this if you copy/fork the sample repo I created?

[ahrtr]

https://github.com/etcd-io/bbolt/actions/runs/8065572741/job/22031667659

Setup go version spec 1.21.6
Attempting to download 1.21.6...
matching 1.21.6...
Not found in manifest.  Falling back to download directly from Go

[jmhbnz]

/assign

[alexellis]

Hey @jmhbnz I pinged you several times since you were in the actuated Slack asking for higher capacity. I think you should log in and see what we did for you.

Regarding go not found, that's a bizarre error isn't it? Perhaps something to do with a bad caching key?

Alex

[jmhbnz]

Hey @jmhbnz I pinged you several times since you were in the actuated Slack asking for higher capacity. I think you should log in and see what we did for you.

Thanks for the follow-up, replied in Actuated Slack.

Regarding go not found, that's a bizarre error isn't it? Perhaps something to do with a bad caching key?

Absolutely bizarre, I agree some kind of interest cache / networking bug but very rare to see instances of it. I have not put time into trying to dig into it further for this reason.

[jmhbnz]

Discussed during sig-etcd triage. Closing as this is an extremely rare issue on infrastructure that we don't manage. Additionally we are in future moving to a new arm64 prow cluster from sig-k8s-infra so this issue will likely go away there anyway.

[alexellis]

Ack.

If you have any suggestions on repro or anything else that may be relevant, please feel free to let us know.

[jmhbnz]

Recording another instance: https://github.com/etcd-io/etcd/actions/runs/8651099263/job/23721045685

Run actions/setup-go@0c52d547c9bc32b1aa3301fd7a9cb496313a4491
  with:
    go-version: 1.22.2
    check-latest: false
    token: ***
    cache: true
Setup go version spec 1.22.2
Attempting to download 1.22.2...
Request timeout: /repos/actions/go-versions/git/trees/main
Falling back to download directly from Go
Error: Request timeout: /dl/?mode=json&include=all

Raised slack discussion on actuated: https://self-actuated.slack.com/archives/C043BB2NCUW/p1712861928174919

[jmhbnz]

Raised: actions/setup-go#470

[jmhbnz]

Closing - We have migrated off GitHub actions for arm64 tests and now run these in prow alongside all other workflows.