Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[receiver/kafka]: Replace "topic" setting by "traces_topic", "logs_topic" and "metrics_topic" #32735

Open
wildum opened this issue Apr 29, 2024 · 6 comments
Labels
enhancement New feature or request receiver/kafka

Comments

@wildum
Copy link
Contributor

wildum commented Apr 29, 2024

Component(s)

receiver/kafka

Is your feature request related to a problem? Please describe.

When the setting "topic" is not specified, the same kafka receiver config can be used in all three pipelines if the topic names match the default values:

receivers:
  kafka:

pipelines: 
  metrics:
     receivers: [kafka] # consumes topic otlp_metrics
  logs:
     receivers: [kafka] # consumes topic otlp_logs
  traces:
     receivers: [kafka] # consumes topic otlp_spans

If the topic is set to any value, this structure will not work:

receivers:
  kafka:
    topic: custom_traces_topic

pipelines: 
  metrics:
     receivers: [kafka] # consumes topic custom_traces_topic => THIS WON'T WORK
  logs:
     receivers: [kafka] # consumes topic custom_traces_topic => THIS WON'T WORK
  traces:
     receivers: [kafka] # consumes topic custom_traces_topic

What happens in this case is that the three receivers will try to claim the same topic. This is a race condition that will succeed in 1/3 of scenarios.

To avoid this problem, the user must create three different configs for each pipeline. This is inconsistent with the default behavior that allows having one config for all three pipelines if the topic names are matching.

Describe the solution you'd like

The otlp receiver provides traces_url_path, metrics_url_path, and logs_url_path configuration to allow the URL paths that signal data needs to be sent to be modified per signal type.

The same principle can be applied to the topics of the kafka receiver:

receivers:
  kafka:
    traces_topic: custom_traces_topic # default otlp_spans
    metrics_topic: custom_metrics_topic # default otlp_metrics
    logs_topic: custom_logs_topic # default otlp_logs

pipelines: 
  metrics:
     receivers: [kafka] # consumes topic custom_metrics_topic
  logs:
     receivers: [kafka] # consumes topic custom_logs_topic
  traces:
     receivers: [kafka] # consumes topic custom_traces_topic

Describe alternatives you've considered

No response

Additional context

No response

@wildum wildum added enhancement New feature or request needs triage New item requiring triage labels Apr 29, 2024
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@ChrsMark
Copy link
Member

ChrsMark commented May 21, 2024

Wouldn't defining 3 different receivers work here?

receivers:
  kafka/metrics:
    topic: custom_metrics_topic
  kafka/logs:
    topic: custom_logs_topic
  kafka/traces:
    topic: custom_traces_topic

pipelines: 
  metrics:
     receivers: [kafka/metrics] 
  logs:
     receivers: [kafka/logs] 
  traces:
     receivers: [kafka/traces]

@wildum
Copy link
Contributor Author

wildum commented May 21, 2024

This works but this is inconsistent with the default behavior which allows using one receiver for three topics if you name your topics accordingly.

@ChrsMark
Copy link
Member

Yeap, I think it makes sense to have more granular configuration options per signal.
/cc @pavolloffay @MovieStoreGuy

@wildum
Copy link
Contributor Author

wildum commented Jul 4, 2024

@ChrsMark I just opened a PR to make the change ^

Copy link
Contributor

github-actions bot commented Sep 3, 2024

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request receiver/kafka
Projects
None yet
Development

No branches or pull requests

3 participants