Add options to specify and pass metadata on vector sink to help in routing message

[rucciva]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

I would like to have a vector aggregator to aggregate my logs and metrics produced by many independent cluster and needs to route it based on several metadata related to the cluster where those logs and metrics come, e.g. logs from cluster x should be routed to loki sink with x tenant id.
It also would be better if this metadata can be modified by reverse proxy so that i can do custom authn and authz.
I can already achieve this using HTTP sink and source since both of them can use/process auth and headers, but HTTP sink can only process logs data.

Attempted Solutions

modify individual event attributes to manually add metadata. The cons are

• it can't be processed by reverse proxy
• its requires separate way to add metadata to metrics and logs since metrics schema are stricter

Proposal

Add features to specify auth and headers just like HTTP sink, and provide a way to access HTTP header in vector source just like HTTP source

References

No response

Version

vector 0.31.0 (x86_64-unknown-linux-gnu 0f13b22 2023-07-06 13:52:34.591204470)

Tasks

Preview Give feedback

No tasks being tracked yet.

[jszwedko]

Thanks for this feature request @rucciva !

A few notes:

• Supporting authentication between the vector source and sink is covered by Auth between Vectors #2017
• The http sink is capable of processing logs, metrics, and traces if the native_json codec is used

I still think this issue is useful though to record the desire to pass additional headers.

[jszwedko]

Hi @rucciva !

We discussed this a bit more and think this would be a better fit for using the http source/sink rather than adding the ability to set / read headers to the vector source/sink. Would that work for you?

[rucciva]

hi @jszwedko , thanks for the response. i think i can use the native_json as you mentioned for metrics

but i'm still not sure, whether i'll be able to access http header when receiving metrics or even when vector source/sink support the header, it still won't be able to give access to http header?

[jszwedko]

hi @jszwedko , thanks for the response. i think i can use the native_json as you mentioned for metrics

but i'm still not sure, whether i'll be able to access http header when receiving metrics or even when vector source/sink support the header, it still won't be able to give access to http header?

I think you could use https://vector.dev/docs/reference/configuration/sources/http_server/#headers. However the http_server source cannot currently receive metrics: #17056. I think, if we fixed that, it would work.

[rucciva]

i see, i should subscribe to #17056 then, thank you. cheers