Jaeger SPM failing with 501 Not Implemented

[a0ndr]

Hello, I've been trying to figure this out for hours.

When trying to access the Monitor tab, a javascript error is thrown with a bunch of requests failing with 501.
If I try to access one of those endpoints, I get shown this message:

{"data":null,"total":0,"limit":0,"offset":0,"errors":[{"code":501,"msg":"metrics querying is currently disabled"}]}

But I've enabled it - at least I think so.
This is my config:

# jaeger.yml
service:
  extensions: [jaeger_storage, jaeger_query]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger_storage_exporter, spanmetrics]
    metrics/spanmetrics:
      receivers: [spanmetrics]
      exporters: [prometheus]
  telemetry:
    resource:
      service.name: jaeger
    metrics:
      level: detailed
      address: 0.0.0.0:8888
    logs:
      level: DEBUG

extensions:
  jaeger_query:
    storage:
      traces: some_storage
      metrics: some_metrics_storage
  jaeger_storage:
    backends:
      some_storage:
        memory:
          max_traces: 100000
    metric_backends:
      some_metrics_storage:
        prometheus:
          endpoint: http://prometheus:9090
          normalize_calls: true
          normalize_duration: true

connectors:
  spanmetrics:

receivers:
  otlp:
    protocols:
      grpc:
      http:
        endpoint: "0.0.0.0:4318"

processors:
  batch:

exporters:
  jaeger_storage_exporter:
    trace_storage: some_storage
  prometheus:
    endpoint: "0.0.0.0:8889"

It's just ^C ^V'd from the repo.

jaeger-ui.json:

{
  "monitor": {
    "menuEnabled": true
  },
  "dependencies": {
    "menuEnabled": false
  },
  "archiveEnabled": true
}

docker-compose.yml:

services:
  db:
    image: mariadb:10.11.7
    volumes:
      - db_data:/var/lib/mysql
    environment:
      - MARIADB_ROOT_PASSWORD=root
      - MARIADB_DATABASE=api
    ports:
      - "8081:3306"
  pma:
    image: phpmyadmin
    environment:
      - PMA_HOST=db
    ports:
      - "8082:80"
    depends_on:
      - db

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "8084:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    networks:
      - ts

  jaeger:
    image: jaegertracing/all-in-one:latest
    command: ["--config-file", "/etc/jaeger/config.yml", "--query.ui-config", "/etc/jaeger/jaeger-ui.json"]
    ports:
      - "8085:16686"
      - "8086:8888"
      - "8087:8889"
      - "8088:4317"
      - "8089:4318"
    environment:
      - JAEGER_QUERY_METRICS_BACKEND=prometheus
      - JAEGER_QUERY_METRICS_PROMETHEUS_URL=http://prometheus:9090
    volumes:
      - ./jaeger.yml:/etc/jaeger/config.yml
      - ./jaeger-ui.json:/etc/jaeger/jaeger-ui.json
    networks:
      - ts

volumes:
    db_data:

networks:
  ts:
    name: ts

prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'otel_metrics'
    static_configs:
      - targets: ['host.docker.internal:8080']
  - job_name: aggregated-trace-metrics
    static_configs:
      - targets: [ 'jaeger:8889' ]

Logs - Jaeger:

2024-12-07 17:51:18 2024/12/07 16:51:18 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2024-12-07 17:51:18 2024/12/07 16:51:18 application version: git-commit=65cff3c30823ea20d3dc48bae39d5685ae307da5, git-version=v1.64.0, build-date=2024-12-06T21:17:15Z
2024-12-07 17:51:18 {"level":"info","ts":1733590278.368889,"caller":"flags/service.go:99","msg":"Mounting metrics handler on admin server","route":"/metrics"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3690712,"caller":"flags/service.go:105","msg":"Mounting expvar handler on admin server","route":"/debug/vars"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3692749,"caller":"flags/admin.go:119","msg":"Mounting health check on admin server","route":"/"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3694386,"caller":"flags/admin.go:133","msg":"Starting admin HTTP server","http-addr":":14269"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3694565,"caller":"flags/admin.go:111","msg":"Admin server started","http.host-port":"[::]:14269","health-status":"unavailable"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3695288,"caller":"grpc@v1.68.1/clientconn.go:162","msg":"[core] original dial target is: \"localhost:4317\""}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3696969,"caller":"grpc@v1.68.1/clientconn.go:440","msg":"[core] [Channel #1]Channel created"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3697248,"caller":"grpc@v1.68.1/clientconn.go:193","msg":"[core] [Channel #1]parsed dial target is: resolver.Target{URL:url.URL{Scheme:\"dns\", Opaque:\"\", User:(*url.Userinfo)(nil), Host:\"\", Path:\"/localhost:4317\", RawPath:\"\", OmitHost:false, ForceQuery:false, RawQuery:\"\", Fragment:\"\", RawFragment:\"\"}}"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3697975,"caller":"grpc@v1.68.1/clientconn.go:194","msg":"[core] [Channel #1]Channel authority set to \"localhost:4317\""}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3700867,"caller":"memory/factory.go:77","msg":"Memory storage initialized","configuration":{"MaxTraces":0}}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.370965,"caller":"static/provider.go:149","msg":"Loading sampling strategies","filename":"/etc/jaeger/sampling_strategies.json"}
2024-12-07 17:51:18 {"level":"warn","ts":1733590278.3712661,"caller":"static/provider.go:71","msg":"Default operations level strategies will not be included for Ratelimiting service strategies.This behavior will be changed in future releases. Cf. https://github.com/jaegertracing/jaeger/issues/5270"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3952115,"caller":"grpc@v1.68.1/server.go:684","msg":"[core] [Server #2]Server created"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3953838,"caller":"server/grpc.go:93","msg":"Starting jaeger-collector gRPC server","grpc.host-port":"[::]:14250"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3953955,"caller":"server/http.go:45","msg":"Starting jaeger-collector HTTP server","http host-port":":14268"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3957014,"caller":"app/collector.go:139","msg":"Not listening for Zipkin HTTP traffic, port not configured"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.396179,"caller":"handler/otlp_receiver.go:69","msg":"OTLP receiver status change","status":"StatusStarting"}
2024-12-07 17:51:18 {"level":"warn","ts":1733590278.396243,"caller":"internal@v0.115.0/warning.go:40","msg":"Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks.","documentation":"https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3964453,"caller":"grpc@v1.68.1/server.go:684","msg":"[core] [Server #4]Server created"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3966029,"caller":"otlpreceiver@v0.115.0/otlp.go:112","msg":"Starting GRPC server","endpoint":":4317"}
2024-12-07 17:51:18 {"level":"warn","ts":1733590278.3968182,"caller":"internal@v0.115.0/warning.go:40","msg":"Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks.","documentation":"https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3957107,"caller":"grpc@v1.68.1/server.go:880","msg":"[core] [Server #2 ListenSocket #3]ListenSocket created"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3969643,"caller":"otlpreceiver@v0.115.0/otlp.go:169","msg":"Starting HTTP server","endpoint":":4318"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.39699,"caller":"grpc@v1.68.1/server.go:880","msg":"[core] [Server #4 ListenSocket #5]ListenSocket created"}
2024-12-07 17:51:18 {"level":"warn","ts":1733590278.3974583,"caller":"internal@v0.115.0/warning.go:40","msg":"Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks.","documentation":"https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3975708,"caller":"grpc@v1.68.1/server.go:684","msg":"[core] [Server #6]Server created"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.4258623,"caller":"app/static_handler.go:92","msg":"Using UI configuration","path":"/etc/jaeger/jaeger-ui.json"}
2024-12-07 17:51:18 {"level":"warn","ts":1733590278.4460382,"caller":"internal@v0.115.0/warning.go:40","msg":"Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks.","documentation":"https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.446269,"caller":"app/server.go:254","msg":"Query server started","http_addr":"[::]:16686","grpc_addr":"[::]:16685"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.4463565,"caller":"healthcheck/handler.go:118","msg":"Health Check state change","status":"ready"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.4463825,"caller":"app/server.go:334","msg":"Starting GRPC server","port":16685,"addr":":16685"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.4464035,"caller":"app/server.go:320","msg":"Starting HTTP server","port":16686,"addr":":16686"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.4464421,"caller":"grpc@v1.68.1/server.go:880","msg":"[core] [Server #6 ListenSocket #7]ListenSocket created"}
2024-12-07 17:51:23 {"level":"info","ts":1733590283.37132,"caller":"grpc@v1.68.1/clientconn.go:345","msg":"[core] [Channel #1]Channel exiting idle mode"}
2024-12-07 17:51:23 {"level":"info","ts":1733590283.398096,"caller":"grpc@v1.68.1/resolver_wrapper.go:200","msg":"[core] [Channel #1]Resolver state updated: {\n  \"Addresses\": [\n    {\n      \"Addr\": \"127.0.0.1:4317\",\n      \"ServerName\": \"\",\n      \"Attributes\": null,\n      \"BalancerAttributes\": null,\n      \"Metadata\": null\n    },\n    {\n      \"Addr\": \"[::1]:4317\",\n      \"ServerName\": \"\",\n      \"Attributes\": null,\n      \"BalancerAttributes\": null,\n      \"Metadata\": null\n    }\n  ],\n  \"Endpoints\": [\n    {\n      \"Addresses\": [\n        {\n          \"Addr\": \"127.0.0.1:4317\",\n          \"ServerName\": \"\",\n          \"Attributes\": null,\n          \"BalancerAttributes\": null,\n          \"Metadata\": null\n        }\n      ],\n      \"Attributes\": null\n    },\n    {\n      \"Addresses\": [\n        {\n          \"Addr\": \"[::1]:4317\",\n          \"ServerName\": \"\",\n          \"Attributes\": null,\n          \"BalancerAttributes\": null,\n          \"Metadata\": null\n        }\n      ],\n      \"Attributes\": null\n    }\n  ],\n  \"ServiceConfig\": null,\n  \"Attributes\": null\n} (resolver returned new addresses)"}
2024-12-07 17:51:23 {"level":"info","ts":1733590283.3982933,"caller":"grpc@v1.68.1/balancer_wrapper.go:109","msg":"[core] [Channel #1]Channel switches to new LB policy \"pick_first\""}
2024-12-07 17:51:23 {"level":"info","ts":1733590283.3983426,"caller":"gracefulswitch/gracefulswitch.go:193","msg":"[pick-first-lb] [pick-first-lb 0xc00066f080] Received new config {\n  \"shuffleAddressList\": false\n}, resolver state {\n  \"Addresses\": [\n    {\n      \"Addr\": \"127.0.0.1:4317\",\n      \"ServerName\": \"\",\n      \"Attributes\": null,\n      \"BalancerAttributes\": null,\n      \"Metadata\": null\n    },\n    {\n      \"Addr\": \"[::1]:4317\",\n      \"ServerName\": \"\",\n      \"Attributes\": null,\n      \"BalancerAttributes\": null,\n      \"Metadata\": null\n    }\n  ],\n  \"Endpoints\": [\n    {\n      \"Addresses\": [\n        {\n          \"Addr\": \"127.0.0.1:4317\",\n          \"ServerName\": \"\",\n          \"Attributes\": null,\n          \"BalancerAttributes\": null,\n          \"Metadata\": null\n        }\n      ],\n      \"Attributes\": null\n    },\n    {\n      \"Addresses\": [\n        {\n          \"Addr\": \"[::1]:4317\",\n          \"ServerName\": \"\",\n          \"Attributes\": null,\n          \"BalancerAttributes\": null,\n          \"Metadata\": null\n        }\n      ],\n      \"Attributes\": null\n    }\n  ],\n  \"ServiceConfig\": null,\n  \"Attributes\": null\n}"}
2024-12-07 17:51:23 {"level":"info","ts":1733590283.3983624,"caller":"grpc@v1.68.1/balancer_wrapper.go:182","msg":"[core] [Channel #1 SubChannel #8]Subchannel created"}
2024-12-07 17:51:23 {"level":"info","ts":1733590283.3983724,"caller":"grpc@v1.68.1/clientconn.go:544","msg":"[core] [Channel #1]Channel Connectivity change to CONNECTING"}
2024-12-07 17:51:23 {"level":"info","ts":1733590283.398412,"caller":"grpc@v1.68.1/clientconn.go:1202","msg":"[core] [Channel #1 SubChannel #8]Subchannel Connectivity change to CONNECTING"}
2024-12-07 17:51:23 {"level":"info","ts":1733590283.3984275,"caller":"grpc@v1.68.1/clientconn.go:1322","msg":"[core] [Channel #1 SubChannel #8]Subchannel picks a new address \"127.0.0.1:4317\" to connect"}
2024-12-07 17:51:23 {"level":"info","ts":1733590283.3984666,"caller":"pickfirst/pickfirst.go:184","msg":"[pick-first-lb] [pick-first-lb 0xc00066f080] Received SubConn state update: 0xc00066f110, {ConnectivityState:CONNECTING ConnectionError:<nil> connectedAddress:{Addr: ServerName: Attributes:<nil> BalancerAttributes:<nil> Metadata:<nil>}}"}
2024-12-07 17:51:23 {"level":"info","ts":1733590283.3989046,"caller":"grpc@v1.68.1/clientconn.go:1202","msg":"[core] [Channel #1 SubChannel #8]Subchannel Connectivity change to READY"}
2024-12-07 17:51:23 {"level":"info","ts":1733590283.399079,"caller":"pickfirst/pickfirst.go:184","msg":"[pick-first-lb] [pick-first-lb 0xc00066f080] Received SubConn state update: 0xc00066f110, {ConnectivityState:READY ConnectionError:<nil> connectedAddress:{Addr:127.0.0.1:4317 ServerName:localhost:4317 Attributes:<nil> BalancerAttributes:<nil> Metadata:<nil>}}"}
2024-12-07 17:51:23 {"level":"info","ts":1733590283.3993762,"caller":"grpc@v1.68.1/clientconn.go:544","msg":"[core] [Channel #1]Channel Connectivity change to READY"}

I tried everything. I just cannot figure it out. Any help is very appreciated

[dosubot[bot]]

Hey @a0ndr! I'm here to help you with your technical issues. If you need assistance with bugs, questions, or contributing, I'm here to assist you.

The 501 Not Implemented error you're encountering typically indicates that a requested feature or service is not supported by the server. In the context of Jaeger, this error often occurs when metrics querying is disabled or not properly configured ^[1].

Here are a few steps to troubleshoot and resolve the issue:

1. Verify Configuration: Ensure that the metrics section in your Jaeger configuration is correctly set up. Your configuration seems to have the necessary components, but double-check that the metrics store is correctly referenced in the jaeger_query extension:

   extensions:
     jaeger_query:
       storage:
         traces: some_storage
         metrics: some_metrics_storage

2. Prometheus Configuration: Make sure that the Prometheus server is running and accessible at the specified endpoint (http://prometheus:9090). You can test this by accessing the Prometheus UI or using a tool like curl to check the endpoint.

3. Jaeger UI Configuration: Your jaeger-ui.json file correctly enables the Monitor tab with "menuEnabled": true. Ensure that this configuration file is correctly mounted and used by the Jaeger service.

4. Check Logs for Errors: Review the Jaeger logs for any errors or warnings related to metrics querying or Prometheus connectivity. The logs you provided don't show any obvious issues, but keep an eye out for any new log entries that might indicate a problem.

5. JavaScript Error: The TypeError indicating this.config is undefined suggests there might be an issue with how the configuration is being accessed in the Jaeger UI. The Jaeger UI handles such errors by issuing a warning and returning the default configuration ^[2]. Ensure that the UI configuration is correctly injected and accessible.

If these steps don't resolve the issue, consider checking the network connectivity between Jaeger and Prometheus, and ensure that all services are correctly started and configured in your Docker setup.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[yurishkuro]

image: jaegertracing/all-in-one:latest

This is Jaeger v1, but you are using v2 config. The image should be jaegertracing/jaeger

[a0ndr]

That seems to fix the SPM issue, however, now I cannot ingest any traces. The UI is just not showing my interface. I'm getting traces export: Post "http://localhost:8088/v1/traces": EOF and traces export: Post "http://localhost:8088/v1/traces": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x06\x04\x00\x00\x00\x00\x00\x00\x05\x00\x00@\x00" errors. Neither gRPC or HTTP ports work
Jaeger isn't showing any errors in it's logs.

[a0ndr]

Okay, I don't know how, but I fixed it