AWS Lambda OTEL Metric to Managed Prometheus: Failed to export batch code 404/403

[kbcwc]

Hi all,

Goal

I'm trying to send a metric from an AWS Lambda function to an AWS Managed Service for Prometheus workspace using the OTEL Lambda Layer.

Setup

The lambda code:

from typing import Dict, Any
from opentelemetry import metrics


def lambda_handler(event: Dict, _: Any) -> Dict:
meter = metrics.get_meter("rgt_smoketest")
    counter = meter.create_counter(
        name="rgt_smoketest_first_counter",
        description="TODO",
        unit="1",
    )

    counter.add(1, attributes={"foo": "bar"})

    print("HELLO WORLD")
    return {
        "statusCode": 200,
    }

I use two layers:
arn:aws:lambda:eu-west-1:184161586896:layer:opentelemetry-python-0_8_0:1 for the OTEL SDK
and
arn:aws:lambda:eu-west-1:184161586896:layer:opentelemetry-collector-arm64-0_10_0:3 for the Collector.

The Lambda execution role has the necessary permissions for executing remote_write to Prometheus:

"aps:RemoteWrite",
                     "aps:GetMetricMetadata",
                     "aps:GetLabels",
                     "aps:GetSeries"

The collector config looks like

receivers:
  otlp:
    protocols:
      http:
        endpoint: "localhost:4318"

exporters:
  prometheusremotewrite:
    endpoint: https://<my_amp_workspace_URL>/api/v1/remote_write
    external_labels:
      some: labels
    auth:
      authenticator: sigv4auth
  logging:
    loglevel: debug

extensions:
  sigv4auth:
    region: eu-west-1

service:
  extensions: [sigv4auth]
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheusremotewrite, logging]

and the environment variables are

AWS_LAMBDA_EXEC_WRAPPER = "/opt/otel-instrument"
OPENTELEMETRY_COLLECTOR_CONFIG_FILE="/var/task/collector.yaml"
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT="<my_amp_workspace_URL>/v1/remote_write"

Output

When I invoke the Lambda using the event

{
  "key": "value"
}

I get logs:
1 [ERROR] 2024-08-22T10:02:37.834Z de78cc58-9345-42f0-a9cf-73394d37a5a4 Failed to export batch code: 404, reason: 404 page not found

2 [ERROR] 2024-08-22T10:02:37.944Z de78cc58-9345-42f0-a9cf-73394d37a5a4 Failed to export batch code: 403, reason: { "message": "Missing Authentication Token" }

Log no. 2 I only receive when setting OTEL_EXPORTER_OTLP_METRICS_ENDPOINT environment variable.

A full test call output looks like

Response
{
  "statusCode": 200
}

Function Logs
{"level":"info","ts":1724861178.698147,"msg":"Launching OpenTelemetry Lambda extension","version":"UNSET"}
{"level":"info","ts":1724861178.7087815,"logger":"telemetryAPI.Listener","msg":"Listening for requests","address":"sandbox.localdomain:53612"}
{"level":"info","ts":1724861178.7096024,"logger":"telemetryAPI.Client","msg":"Subscribing","baseURL":"http://127.0.0.1:9001/2022-07-01/telemetry"}
TELEMETRY	Name: collector	State: Subscribed	Types: [Platform]
{"level":"info","ts":1724861178.7175033,"logger":"telemetryAPI.Client","msg":"Subscription success","response":"\"OK\""}
{"level":"info","ts":1724861178.7175808,"logger":"NewCollector","msg":"Using config URI from environment","uri":"/var/task/collector.yaml"}
{"level":"info","ts":1724861178.7787385,"caller":"service@v0.105.0/service.go:116","msg":"Setting up own telemetry..."}
{"level":"info","ts":1724861178.7789638,"caller":"service@v0.105.0/service.go:119","msg":"OpenCensus bridge is disabled for Collector telemetry and will be removed in a future version, use --feature-gates=-service.disableOpenCensusBridge to re-enable"}
{"level":"info","ts":1724861178.7793615,"caller":"service@v0.105.0/telemetry.go:96","msg":"Serving metrics","address":":8888","metrics level":"Normal"}
{"level":"info","ts":1724861178.7795944,"caller":"exporter@v0.105.0/exporter.go:280","msg":"Deprecated component. Will be removed in future releases.","kind":"exporter","data_type":"metrics","name":"logging"}
{"level":"warn","ts":1724861178.783448,"caller":"common/factory.go:68","msg":"'loglevel' option is deprecated in favor of 'verbosity'. Set 'verbosity' to equivalent value to preserve behavior.","kind":"exporter","data_type":"metrics","name":"logging","loglevel":"debug","equivalent verbosity level":"Detailed"}
{"level":"info","ts":1724861178.7987573,"caller":"service@v0.105.0/service.go:198","msg":"Starting otelcol-lambda...","Version":"UNSET","NumCPU":2}
{"level":"info","ts":1724861178.7989025,"caller":"extensions/extensions.go:34","msg":"Starting extensions..."}
{"level":"info","ts":1724861178.7989392,"caller":"extensions/extensions.go:37","msg":"Extension is starting...","kind":"extension","name":"sigv4auth"}
{"level":"info","ts":1724861178.7990153,"caller":"extensions/extensions.go:52","msg":"Extension started.","kind":"extension","name":"sigv4auth"}
{"level":"info","ts":1724861178.7992315,"caller":"otlpreceiver@v0.105.0/otlp.go:152","msg":"Starting HTTP server","kind":"receiver","name":"otlp","data_type":"metrics","endpoint":"localhost:4318"}
{"level":"info","ts":1724861178.7994325,"caller":"service@v0.105.0/service.go:224","msg":"Everything is ready. Begin running and processing data."}
{"level":"info","ts":1724861178.7995071,"caller":"localhostgate/featuregate.go:63","msg":"The default endpoints for all servers in components have changed to use localhost instead of 0.0.0.0. Disable the feature gate to temporarily revert to the previous default.","feature gate ID":"component.UseLocalHostAsDefaultHost"}
EXTENSION	Name: collector	State: Ready	Events: [INVOKE, SHUTDOWN]
START RequestId: b57e0ed5-4908-4440-b084-d58a5a339e5a Version: $LATEST
[ERROR]	2024-08-28T16:06:20.216Z	b57e0ed5-4908-4440-b084-d58a5a339e5a	Failed to export batch code: 404, reason: 404 page not found

[ERROR]	2024-08-28T16:06:20.316Z	b57e0ed5-4908-4440-b084-d58a5a339e5a	Failed to export batch code: 403, reason: {"message":"Missing Authentication Token"}
END RequestId: b57e0ed5-4908-4440-b084-d58a5a339e5a
REPORT RequestId: b57e0ed5-4908-4440-b084-d58a5a339e5a	Duration: 276.84 ms	Billed Duration: 277 ms	Memory Size: 128 MB	Max Memory Used: 120 MB	Init Duration: 1586.67 ms

Question

I verified with AWS Support that from their side this setup looks OK. Could somebody on here provide a suggestion on what I might be missing in terms of config? Happy to provide more information if that helps.

Thanks a lot!