auto instrument failed: offset not found: go.opentelemetry.io/otel/internal/global.tracer:delegate (1.27.0)

[SpeCT]

Describe the bug

I'm trying to setup golang microservice container + sidecar container in a way it is implemented in e2e tests in this repo. The goal is to try to use feature implemented in PR #523 - try to combine automatic instrumentation with some manually created spans. The sidecar container is not able to start due to the following error:

{"level":"info","ts":1717094019.600399,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:107","msg":"starting instrumentation..."}
{"level":"info","ts":1717094019.6026487,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:191","msg":"Mounting bpffs","allocations_details":{"StartAddr":140693078781952,"EndAddr":140693079306240,"NumCPU":16}}
{"level":"info","ts":1717094019.6041243,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:177","msg":"loading probe","name":"go.opentelemetry.io/otel/internal/global/client"}
{"level":"error","ts":1717094019.6054547,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:180","msg":"error while loading probes, cleaning up","name":"go.opentelemetry.io/otel/internal/global/client","error":"offset not found: go.opentelemetry.io/otel/internal/global.tracer:delegate (1.27.0)","stacktrace":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).load\n\t/app/internal/pkg/instrumentation/manager.go:180\ngo.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).Run\n\t/app/internal/pkg/instrumentation/manager.go:127\ngo.opentelemetry.io/auto.(*Instrumentation).Run\n\t/app/instrumentation.go:151\nmain.main\n\t/app/cli/main.go:108\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:271"}
{"level":"info","ts":1717094019.605531,"logger":"Instrumentation.Manager.go.opentelemetry.io/otel/internal/global/client","caller":"probe/probe.go:213","msg":"Closed","Probe":"go.opentelemetry.io/otel/internal/global/client"}
{"level":"info","ts":1717094019.6055605,"logger":"Instrumentation.Manager.net/http/server","caller":"probe/probe.go:213","msg":"Closed","Probe":"net/http/server"}
{"level":"info","ts":1717094019.6055734,"logger":"Instrumentation.Manager.net/http/client","caller":"probe/probe.go:213","msg":"Closed","Probe":"net/http/client"}
{"level":"info","ts":1717094019.6055799,"logger":"Instrumentation.Manager","caller":"instrumentation/manager.go:205","msg":"Cleaning bpffs"}
{"level":"error","ts":1717094019.605618,"logger":"go.opentelemetry.io/auto","caller":"cli/main.go:109","msg":"instrumentation crashed","error":"offset not found: go.opentelemetry.io/otel/internal/global.tracer:delegate (1.27.0)","stacktrace":"main.main\n\t/app/cli/main.go:109\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:271"}

Environment

• OS: MacOSX Sonoma 14.5, Docker v26.1.1 (4cf5afa)
• Go Version: 1.22.3 (using golang:1.22.3 base image)
• Version: Official docker image autoinstrumentation-go:v0.12.0-alpha

To Reproduce

// trimmed main.go

package main

import (
  // ...

  "go.opentelemetry.io/otel"
)
var tracer = otel.Tracer("custom-gobackendapi")

func main() {
  // setup router
}

func handler(w http.ResponseWriter, r *http.Request) {
  ctx, span := tracer.Start(r.Context(), "call GetMongoRows")
  defer span.End()

  // do magic
}

module bitbucket.org/blah/blah/gobackendapi

go 1.22.0

require (
	github.com/gorilla/mux v1.8.1
	go.opentelemetry.io/otel v1.27.0
	golang.org/x/oauth2 v0.20.0
)

require (
	github.com/go-logr/logr v1.4.1 // indirect
	github.com/go-logr/stdr v1.2.2 // indirect
	go.opentelemetry.io/otel/metric v1.27.0 // indirect
	go.opentelemetry.io/otel/trace v1.27.0 // indirect
)

FROM golang:1.22.3

WORKDIR /var/gobackendapi
VOLUME /var/gobackendapi

EXPOSE 80

CMD go build -o main && ./main

services:
  gobackendapi:
    build: { context: ./packages/gobackendapi, dockerfile: Dockerfile.dev }
    volumes: 
      - ./packages/gobackendapi:/var/gobackendapi
      - gobackendapi_proc:/host/proc
    ports:
      - '8090:12000'
    depends_on:
      - backendapi
    environment:
      - PORT=12000
      - BACKENDAPI_URL=http://backendapi:12000

  gobackendapi-otel:
    image: ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go:v0.12.0-alpha
    command: ["/otel-go-instrumentation", "-global-impl"]
    privileged: true
    pid: "host"
    volumes: ["gobackendapi_proc:/host/proc"]
    depends_on:
      - otel-collector
    environment:
      - OTEL_GO_AUTO_TARGET_EXE=/var/gobackendapi/main
      - OTEL_SERVICE_NAME=gobackendapi
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
      - OTEL_RESOURCE_ATTRIBUTES=service.name=gobackendapi,deployment.environment=local-playground
      - OTEL_PROPAGATORS=tracecontext,baggage
      - OTEL_GO_AUTO_GLOBAL=true
      # - OTEL_GO_AUTO_SHOW_VERIFIER_LOG=true
  
  otel-collector:
    image: otel/opentelemetry-collector
    ...

  backendapi:
    image: some-nestjs-service
    ...
    
volumes:
  gobackendapi_proc:

Expected behavior

sidecar instrument successfully starts

Additional context

Everything worked kinda ok before I tried to add otel.Tracer() to my microservice.

[RonFed]

The latest released image is not supporting the latest SDK package and latest Go version. (We should release a newer version soon). In addition, the env var used to configure this option was added recently (#849) and will be included in the next release.
cc @MrAlias

[SpeCT]

@RonFed thank you!
Somehow I thought e2e tests are green here, and the entire #523 is included into 0.12.0-alpha release. This was the primary reason why I decided to use exactly the same versions as in e2e tests in this repo.

PS: just tried to build sidecar image from sources - works like a charm. looking forward to see new release published

[MrAlias]

This should be resolved in the latest release. I'm going to close this as resolve.

Please re-open if this was in error.