Handle correlation IDs

[callumforrester]

Requests may come in with correlation IDs. If they do not, we should generate them.

[callumforrester]

@keithralphs or @DiamondJoseph is this issue still relevant or is there a telemetry issue to supersede it?

[keithralphs]

I think the telemetry stuff will encompass this, though there may be some nuance as to how exactly we want to do it, so I would keep this here for now until we've fully worked that out. My plan is to get Joe to apply Otel to his newly subprocessed BlueAPI so we'll prorbably address it then.

[stan-dot]

what is the further use of this correlation? the term implies some other related data structure but it is not clear what that is

[DiamondJoseph]

https://playbook.microsoft.com/code-with-engineering/observability/correlation-id/

This should be injected automatically at the gateway to the services, propagate through all involved services and be part of standard logging. The observability platform will use it for metrics, but also of use for trouble shooting.

[stan-dot]

Isn't it better if we just always add it ourselves, at the first injection - such as the first blueapi request?

why would we ever rely on the user to generate them? @callumforrester

I think we get this for free with using pydantic models

[DiamondJoseph]

"the gateway to the services" is "us" and is the entrypoint to the system for a user request.

[DiamondJoseph]

Ah see what you mean.

Requests may come in with correlation IDs. If they do not, we should generate them.

Requests will eventually come from a service gateway that will inject the correlation id, which should then be propagated through outward requests/emitted documents. For now, since the gateway is still being created, if the request has one (e.g. it's been generated in GDA to trace/debug requests from that client) we propagate it, else we create it.

[stan-dot]

"the gateway to the services" is "us"
there is no 'us' in the system diagram

there is blueapi, keycloak, runengine, idk

[callumforrester]

Now done by @keithralphs tracing changes