When building Ubuntu the build fails every time with 6 fails regarding node and go version - Expected path '/opt/hostedtoolcache/go/1.21.9/x64/bin/go' to exist, but it did not exist.

[Gerardkavanagh]

Description

When building Ubuntu the build fails every time with 6 fails regarding node and go versions, even though the version have installed ok. Expected path '/opt/hostedtoolcache/go/1.21.9/x64/bin/go' to exist, but it did not exist.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• Windows Server 2019
• Windows Server 2022

Image version and build link

runner images version = release
os image type = ubuntu2004
OSTagPrefix = ubuntu20
IncludePrerelease = False

Name Value

---

name Ubuntu 20.04 (20240324) Image Update
published_at 2024-03-25T12:50:35Z
target_commitish 26ec93f
tag_name ubuntu20/20240324.1
url https://api.github.com/repos/actions/runner-images/releases/148212268
prerelease False
Note: switching to 'ubuntu20/20240324.1'.

Is it regression?

Na

Expected behavior

Expect the build to pass if the node or go version was installed.

azure-arm.build_image: Download node-18.20.1-linux-arm64.tar.gz
azure-arm.build_image: Downloading package from https://github.com/actions/node-versions/releases/download/18.20.1-8541411805/node-18.20.1-linux-arm64.tar.gz to /tmp/node-18.20.1-linux-arm64.tar.gz...
azure-arm.build_image: Package downloaded in 1.51 seconds
azure-arm.build_image: Extract node-18.20.1-linux-arm64.tar.gz content...
azure-arm.build_image: Invoke installation script...
azure-arm.build_image: Check if Node.js hostedtoolcache folder exist...
azure-arm.build_image: Delete Node.js 18.20.1 if installed
azure-arm.build_image: Create Node.js 18.20.1 folder
azure-arm.build_image: Copy Node.js binaries to hostedtoolcache folder
azure-arm.build_image: Create complete file

azure-arm.build_image: Download go-1.21.9-linux-arm64.tar.gz
azure-arm.build_image: Downloading package from https://github.com/actions/go-versions/releases/download/1.21.9-8548849157/go-1.21.9-linux-arm64.tar.gz to /tmp/go-1.21.9-linux-arm64.tar.gz...
azure-arm.build_image: Package downloaded in 2.69 seconds
azure-arm.build_image: Extract go-1.21.9-linux-arm64.tar.gz content...
azure-arm.build_image: Invoke installation script...
azure-arm.build_image: Check if Go hostedtoolcache folder exist...
azure-arm.build_image: Delete Go 1.21.9 if installed
azure-arm.build_image: Create Go 1.21.9 folder
azure-arm.build_image: Copy Go binaries to hostedtoolcache folder

Actual behavior

azure-arm.build_image: Context node
azure-arm.build_image: [+] 16.. version folder exists 10ms (6ms|4ms)
azure-arm.build_image: [+] Validate bin/node 8ms (5ms|3ms)
azure-arm.build_image: [+] Validate bin/npm 8ms (5ms|3ms)
azure-arm.build_image: [+] 18.. version folder exists 9ms (6ms|3ms)
azure-arm.build_image: [-] Validate bin/node 590ms (586ms|4ms)
azure-arm.build_image: Expected path '/opt/hostedtoolcache/node/18.20.1/x64/bin/node' to exist, but it did not exist.
azure-arm.build_image: at $ExecutablePath | Should -Exist, /imagegeneration/tests/Toolset.Tests.ps1:58
azure-arm.build_image: at , /imagegeneration/tests/Toolset.Tests.ps1:58
azure-arm.build_image: [-] Validate bin/npm 8ms (6ms|2ms)
azure-arm.build_image: Expected path '/opt/hostedtoolcache/node/18.20.1/x64/bin/npm' to exist, but it did not exist.
azure-arm.build_image: at $ExecutablePath | Should -Exist, /imagegeneration/tests/Toolset.Tests.ps1:58
azure-arm.build_image: at , /imagegeneration/tests/Toolset.Tests.ps1:58
azure-arm.build_image: [+] 20.. version folder exists 5ms (3ms|1ms)
azure-arm.build_image: [-] Validate bin/node 7ms (5ms|1ms)
azure-arm.build_image: Expected path '/opt/hostedtoolcache/node/20.12.1/x64/bin/node' to exist, but it did not exist.
azure-arm.build_image: at $ExecutablePath | Should -Exist, /imagegeneration/tests/Toolset.Tests.ps1:58
azure-arm.build_image: at , /imagegeneration/tests/Toolset.Tests.ps1:58
azure-arm.build_image: [-] Validate bin/npm 6ms (5ms|1ms)
azure-arm.build_image: Expected path '/opt/hostedtoolcache/node/20.12.1/x64/bin/npm' to exist, but it did not exist.
azure-arm.build_image: at $ExecutablePath | Should -Exist, /imagegeneration/tests/Toolset.Tests.ps1:58
azure-arm.build_image: at , /imagegeneration/tests/Toolset.Tests.ps1:58
azure-arm.build_image: Context go
azure-arm.build_image: [+] 1.20.* version folder exists 6ms (3ms|2ms)
azure-arm.build_image: [+] Validate bin/go 4ms (3ms|1ms)
azure-arm.build_image: [+] 1.21.* version folder exists 5ms (4ms|1ms)
azure-arm.build_image: [-] Validate bin/go 7ms (5ms|1ms)
azure-arm.build_image: Expected path '/opt/hostedtoolcache/go/1.21.9/x64/bin/go' to exist, but it did not exist.
azure-arm.build_image: at $ExecutablePath | Should -Exist, /imagegeneration/tests/Toolset.Tests.ps1:58
azure-arm.build_image: at , /imagegeneration/tests/Toolset.Tests.ps1:58
azure-arm.build_image: [+] 1.22.* version folder exists 5ms (3ms|2ms)
azure-arm.build_image: [-] Validate bin/go 7ms (5ms|1ms)
azure-arm.build_image: Expected path '/opt/hostedtoolcache/go/1.22.2/x64/bin/go' to exist, but it did not exist.
azure-arm.build_image: at $ExecutablePath | Should -Exist, /imagegeneration/tests/Toolset.Tests.ps1:58
azure-arm.build_image: at , /imagegeneration/tests/Toolset.Tests.ps1:58

Repro steps

Build the pipeline and select either Ubuntu 22.04 or 20.04

Uploading log.txt…

[erik-bershel]

Hey @Gerardkavanagh!
Please answer some question before I can continue to help you:

• Do you used main branch to build image?
• Did you changed code before build?
• When did you start having these problems and when was the last unsuccessful launch?

Now Ubuntu builds should be broken by other issue which is in progress. You may check here: #9664

[Gerardkavanagh]

Yes, I used main branch and made no changes to the code.

The issues started on Friday 5th April.

[erik-bershel]

Then something wrong should be with your repository clone. Please, update it and try again to run build. We had successful builds on Monday: Ubuntu-22.04, Ubuntu-20.04. Now you should get mentioned earlier mistake right after the build start.

[mhever]

Same thing here, when using the latest stable March release for ubuntu20, ubuntu22 the code was trying to install the wrong architecture file for some packages.

Since stable latest is from April, it works again as expected.

[erik-bershel]

stable March release for ubuntu20, ubuntu22

You should not use branches. It will almost never work - stale code will not work because of cumulative changes. We do not support build from release branches.

[mhever]

stable March release for ubuntu20, ubuntu22

You should not use branches. It will almost never work - stale code will not work because of cumulative changes. We do not support build from release branches.

Let me rephrase this. While the latest stable tag was pointing to for example ubuntu20/20240324.1 we were facing the exact same issue. Since April releases, it's working again. We are using
https://api.github.com/repos/actions/runner-images/releases with "prerelease==false" filter to get the latest stable. This was failing for a while with the same error as above.

Never had the time to deep dive into the issue at that time of the issue, and since the April stable release the packer build just works again.

[Gerardkavanagh]

@erik-bershel Typical I took my last cut of the code Sunday night, tried to fix the issues locally, and then raised an issue. Tested now again with fresh code and all worked again as expected.

Thank you for looking to it.

[erik-bershel]

@mhever just to be clear.

We are using
https://api.github.com/repos/actions/runner-images/releases with "prerelease==false" filter to get the latest stable.

Why should you do this? Every time to build an image you need to clone the repo, chose main and execute build-script (doc). There is no point to use releases - in our case it's just abstraction and point in time but not the exact result of the build. You'll never get the same result like in the releases (we too). We have only one supported and maintained branch - main. All other branches are unstable. I would even say that they are almost guaranteed to not work - the code becomes outdated within a week in almost all cases. Also there is no connection between releases in progress and current state of code because of the same reason.

[mhever]

All righty. Then what exactly is the point of having the tags and releases in the runner-images repo? You got me super perplexed right now.

[mhever]

Following up in #9675 instead of hijacking this issue.