[CON-284] Add OpenTelemetry tracing for express and bunyan

[jonaylor89]

Description

This PR adds the OpenTelemetry sdk to content node as well as several instrumentations

• Express (and http)
• Bunyan
• Redis
• Postgres

which provide trace data to content node. This PR does not add an exporter for traces so the only noticable change with this PR is the additional of trace data in bunyan logs. I'll follow up with more PRs to add an exporter(s) (e.g. Jaegar) as well as instrumentation for bull queues to get trace data in the processors there.

{
  "name": "audius_creator_node",
  "hostname": "bfe4e3ec8309",
  "pid": 5791,
  "level": 30,
  "msg": "AHHHHHHHHHH",
  "time": "2022-08-04T03:57:56.495Z",
  "v": 0,
  "trace_id": "06d7347e5e24235afdafdf4a2bad5310",
  "span_id": "cc07f4f1f5bfb525",
  "trace_flags": "01",
  "resource.span": {
    "attributes": {
      "http.url": "http://127.0.0.1:3000/prometheus_metrics",
      "http.host": "127.0.0.1:3000",
      "net.host.name": "127.0.0.1",
      "http.method": "GET",
      "http.client_ip": "172.22.0.10",
      "http.target": "/prometheus_metrics",
      "http.user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.134 Safari/537.36",
      "http.flavor": "1.0",
      "net.transport": "ip_tcp"
    },
    "links": [],
    "events": [],
    "status": {
      "code": 0
    },
    "endTime": [
      0,
      0
    ],
    "_ended": false,
    "_duration": [
      -1,
      -1
    ],
    "name": "GET /prometheus_metrics",
    "_spanContext": {
      "traceId": "06d7347e5e24235afdafdf4a2bad5310",
      "spanId": "cc07f4f1f5bfb525",
      "traceFlags": 1
    },
    "kind": 1,
    "startTime": [
      1659585476,
      492679321
    ],
    "resource": {
      "attributes": {
        "service.name": "content-node-server",
        "telemetry.sdk.language": "nodejs",
        "telemetry.sdk.name": "opentelemetry",
        "telemetry.sdk.version": "1.5.0"
      }
    },
    "instrumentationLibrary": {
      "name": "@opentelemetry/instrumentation-http",
      "version": "0.31.0"
    },
    "_spanLimits": {
      "attributeValueLengthLimit": null,
      "attributeCountLimit": 128,
      "linkCountLimit": 128,
      "eventCountLimit": 128
    },
    "_spanProcessor": {},
    "_attributeValueLengthLimit": null
  },
  "resource.service.name": "content-node-server",
  "logLevel": "info"
}

Tests

This was tested by adding some test logs to content node routes, pinging routes like /health_check and /prometheus_metrics, and observing the log data exported.

Monitoring - How will this change be monitored? Are there sufficient logs / alerts?

These changes will add a new label on logs that can be filtered for in loggly (traceid, spanid)

[SidSethi]

i see this is in draft but you've tagged reviewers - do you want me to do a review?
also CN tests failing

[jonaylor89]

i see this is in draft but you've tagged reviewers - do you want me to do a review? also CN tests failing

Ya, it's still a draft. I'm still adding finishing touches to stuff and didn't mean to ping y'all.

Also, the tests are failing because of tracer.ts being typescript so it should just be a build/lint fix