Update to go 1.23.2.

[blakerouse]

Proposed commit message

Update to the latest go 1.23.2.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @blakerouse? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[blakerouse]

buildkite test this

[blakerouse]

@rowlandgeoff Any ideas on why this is only to fail to build on ARM with the golang-crossbuild?

[rowlandgeoff]

@rowlandgeoff Any ideas on why this is only to fail to build on ARM with the golang-crossbuild?

Not a clue, that's outside my knowledge. @v1v, any ideas here?

[v1v]

Not a clue, that's outside my knowledge. @v1v, any ideas here?

I know very little about the Beats build system and the CI Buildkite pipelines, however

exec /crossbuild: exec format error

That's somehow related to the wrong architecture in the runner versus the architecture that was built for, to list one of them, see this as it contains some details about some other possible errors.

If you use 1.23.1, does it happen? Just curious if the issue is related to the docker golang-crossbuild or something else.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b go-1.23.2 upstream/go-1.23.2
git merge upstream/main
git push upstream go-1.23.2

[shmsr]

I think I know what's happening here. I've been casually playing around with BuildKite (BK) and crossbuilding for some time.

Here cross compiling is failing on a linux/arm64 host.

Related issues for Go 1.23.{0,1,2}, supposed to be fixed in Go 1.23.3+:

• runtime: build fails when run via QEMU for linux/amd64 running on linux/arm64 golang/go#69255
• runtime: multi-arch build via qemu fails to exec go binary golang/go#68976

To check, I tried with Go 1.22.8, it worked fine. See: 4d8ca5c (all checks passed) as part of this #41140

But cross compile works where target is linux/arm64 and the host is linux/amd64.

$ PLATFORMS=linux/arm64 PACKAGES=tar.gz mage -v package

...
...

docker run \
  --env EXEC_UID=1000 \
  --env EXEC_GID=1000 \
  -v /home/ubuntu/.gvm/pkgsets/go1.23.2/global/pkg/mod:/go/pkg/mod:ro \
  --env DEV=false \
  --rm \
  --env GOFLAGS="-mod=readonly -buildvcs=false" \
  --env MAGEFILE_VERBOSE=true \
  --env MAGEFILE_TIMEOUT= \
  --env SNAPSHOT=false \
  -v /home/ubuntu/beats:/go/src/github.com/elastic/beats \
  -w /go/src/github.com/elastic/beats/metricbeat \
  docker.elastic.co/beats-dev/golang-crossbuild:1.23.2-arm \
  --build-cmd "build/mage-linux-amd64 golangCrossBuild" \
  --platforms linux/arm64

For example, this will work, local as well as CI when the host is linux/amd64. When the host is linux/arm64 the buildMage function makes a binary mage-linux-arm64 here which when exec'ed is creating the issue.

But I noticed one thing in our BK pipeline:

So, if you see the BK pipeline for metricbeat and others (except agentbeat):

• Target:

  beats/.buildkite/metricbeat/pipeline.yml

  Line 329 in 2ff38df

  PLATFORMS: "+all linux/amd64 linux/arm64 windows/amd64 darwin/amd64 darwin/arm64"

Do note that we are cross-compiling for linux/arm64 here as well.

• Host:

  beats/.buildkite/metricbeat/pipeline.yml

  Line 339 in 2ff38df

  image: "${IMAGE_UBUNTU_X86_64}"

But still we have a step for Package for arm64. See:

• Target:

  beats/.buildkite/metricbeat/pipeline.yml

  Line 350 in 2ff38df

  PLATFORMS: "linux/arm64"

• Host:

  beats/.buildkite/metricbeat/pipeline.yml

  Line 361 in 2ff38df

  imagePrefix: "${AWS_IMAGE_UBUNTU_ARM_64}"

Here we face the failure when using Go 1.23.2 most probably because of the aforementioned issue with Go 1.23.x i.e., trying to cross compile on a linux/arm64 targeting linux/arm64.

But see how it is done for agentbeat:

• On host

  beats/.buildkite/x-pack/pipeline.xpack.agentbeat.yml

  Line 53 in 2ff38df

  image: "${IMAGE_UBUNTU_X86_64}"

  (linux/amd64) we cross compile for all platforms

  beats/.buildkite/x-pack/pipeline.xpack.agentbeat.yml

  Line 37 in 2ff38df

  PLATFORMS: "+all linux/amd64 linux/arm64 windows/amd64 darwin/amd64 darwin/arm64"

Single step for cross-building for the required platforms.

So, my question shouldn't we improve the pipelines to do the cross-compiling in one step like we are doing for agentbeat.

In this PR as well I made some improvements in BK pipeline for agentbeat by removing the commands that seemed necessary.

So, could you also check if we even need the additional step for metricbeat, packetbeat, etc. Is it intentional to have the "Packaging Linux arm64" step because of some reason?

[blakerouse]

Thank you so much for the detailed reasoning behind why this was failing. I was shocked that a simple update of golang would cause the error, but it being an issue with upstream golang makes a lot of sense.

So, could you also check if we even need the additional step for metricbeat, packetbeat, etc. Is it intentional to have the "Packaging Linux arm64" step because of some reason?

@rowlandgeoff Would be best to answer that information, or some others from that team. I do not know, I was just trying to update to go 1.23.

[gizas]

Do you think that this will also solve the issue raised in #40755 (comment) ? (Also tested in #41129) ?

[ycombinator]

@blakerouse Given the issues with upstream Golang and that we're probably ~3 weeks away from a 1.23.3 release, should we close this PR unmerged and wait for the 1.23.3 release?

[mauri870]

Support for QEMU as a platform in Go has always been limited and generally operates on a best-effort basis. I encountered numerous issues when implementing runtime and atomic operations for various architectures due to QEMU's limitations. If this is indeed a QEMU issue, it would be helpful to clearly identify which specific issue it is, so we can ensure it gets properly addressed.

Additionally, I wonder if it makes sense to set up daily or weekly gotip builds of golang-crossbuild. This would allow us to detect breaking changes and test cutting-edge features without waiting for a new release.