Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sumologic exporter and metric pipelines into OT Collector & agent #1647

Merged
merged 15 commits into from
Dec 1, 2021
Merged

Sumologic exporter and metric pipelines into OT Collector & agent #1647

merged 15 commits into from
Dec 1, 2021

Conversation

mat-rumian
Copy link
Contributor

@mat-rumian mat-rumian commented Jun 10, 2021 •
edited
Loading

Description

Addition of the Sumologic exporter to the Tracing OT Collector pipeline. This will add possibility to send received metrics on the OTC receiver (OTLP protocol) side to Sumo.

Testing performed
  • Redeploy fluentd and fluentd-events pods
  • Confirm events, logs, and metrics are coming in

@mat-rumian mat-rumian requested a review from a team as a code owner June 10, 2021 22:09
@mat-rumian
Copy link
Contributor Author

@pmm-sumo I'm not sure from where should I take a value for source_host tag in the Sumologic exporter configuration - it can be set here

@mat-rumian
Copy link
Contributor Author

@perk-sumo maybe you will be able to help me to get a single environment variable related to the metrics endpoint url.

{{ include "kubernetes.sources.envs" (dict "Context" $ctx "Type" "metrics") }} - this returns all environment variables related to the metrics. I would like to add only one env var which is related to "default-metrics" source.

@mat-rumian mat-rumian requested a review from kasia-kujawa June 10, 2021 22:20
@sumo-drosiek
Copy link
Contributor

@mat-rumian Regarding hostname. I think we should use hostname from exporters (attribute net.peer.name), otherwise we should use local hostname, but it is not implemented in exporter righrt now

Regarding metrics:
You can add helper for single source. For now there is no possibility to extract single source (related code)

@mat-rumian mat-rumian changed the title [WIP] Sumologic exporter into OT Agent & Collector Sumologic exporter into OT Agent & Collector Jun 11, 2021
@mat-rumian mat-rumian changed the title Sumologic exporter into OT Agent & Collector Sumologic exporter and metric pipelines into OT Collector & agent Jun 11, 2021
pmalek-sumo
pmalek-sumo reviewed Jun 11, 2021
View reviewed changes
deploy/helm/sumologic/values.yaml Outdated Show resolved Hide resolved
deploy/helm/sumologic/values.yaml Outdated Show resolved Hide resolved
deploy/helm/sumologic/values.yaml Outdated Show resolved Hide resolved
@sumo-drosiek
Copy link
Contributor

@mat-rumian LGTM

@perk-sumo perk-sumo added this to the v2.2 milestone Jun 29, 2021
@pmalek-sumo
Copy link
Contributor

@sumo-drosiek @mat-rumian Do we want to merge this in? And include in 2.2?

@mat-rumian
Copy link
Contributor Author

@pmalek-sumo yes, it is needed in case of customer call for metrics from OpenTelemetry instrumentation

pmalek-sumo
pmalek-sumo reviewed Oct 1, 2021
View reviewed changes
tests/tracing/static/simple.output.yaml Outdated
queue_size: 5000
source_category:
source_host: '%{host.name}'
source_name:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be empty?

tests/tracing/static/simple.output.yaml Outdated
enabled: false
num_consumers: 10
queue_size: 5000
source_category:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be empty?

tests/tracing/static/collection-monitoring-false.output.yaml Outdated
enabled: false
num_consumers: 10
queue_size: 5000
source_category:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be empty?

tests/tracing/static/collection-monitoring-false.output.yaml Outdated
queue_size: 5000
source_category:
source_host: '%{host.name}'
source_name:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be empty?

deploy/helm/sumologic/templates/traces/otelcol/deployment.yaml
{{ include "kubernetes.sources.envs" (dict "Context" $ctx "Type" "traces") | nindent 8 }}
{{- include "kubernetes.sources.env" (dict "Context" $ctx "Type" "metrics" "Name" "default" ) | nindent 8 }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

{{ and {{, can we stick to one or the other?

@sumo-drosiek
Copy link
Contributor

Isn't that kind of duplication of replacement fluentd by otc?

@perk-sumo perk-sumo modified the milestones: v2.2, v2.3 Oct 25, 2021
@perk-sumo
Copy link
Contributor

Hey, I think we'd like to go with otelcol-sumo binary (the one we use for metadata) in this case. Let's revisit that for the v2.3.

@perk-sumo perk-sumo modified the milestones: v2.4, v2.3 Nov 24, 2021
@perk-sumo
Copy link
Contributor

Hm, now that I think of it I think this should be enabled in both tracing otelcol and the metadata one.

mat-rumian and others added 9 commits December 1, 2021 10:05
@mat-rumian @pmaciolek
Add indents
1cfe06a
@mat-rumian @pmaciolek
Add resourcedetection, setup source_host
5fb1293
@mat-rumian @pmaciolek
Fix tests
a025d00
@mat-rumian @pmaciolek
Setup detector, fix host.name
440d51d
@mat-rumian @pmaciolek
Add space
f16e990
@mat-rumian @pmaciolek
Add space, fix tests
86d3fac
@mat-rumian @pmalek @pmaciolek
Comments update
b4a1e63 
Co-authored-by: Patryk Małek <pmalek@sumologic.com>
@mat-rumian @pmalek @pmaciolek
Comment update
a3ea758 
Co-authored-by: Patryk Małek <pmalek@sumologic.com>
@mat-rumian @pmalek @pmaciolek
Comment update
5db08eb 
Co-authored-by: Patryk Małek <pmalek@sumologic.com>
@pmm-sumo pmm-sumo force-pushed the mat-rumian-sumologic-exporter-into-otc branch from 887cf6f to 5db08eb Compare December 1, 2021 09:05
perk-sumo
perk-sumo approved these changes Dec 1, 2021
View reviewed changes
Copy link
Contributor

@perk-sumo perk-sumo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@pmalek-sumo
Copy link
Contributor

@pmm-sumo Can we add a meaningful description to this PR 🙏 ? And after that let's merge it

@pmalek-sumo pmalek-sumo merged commit fb09256 into main Dec 1, 2021
@pmalek-sumo pmalek-sumo deleted the mat-rumian-sumologic-exporter-into-otc branch December 1, 2021 11:46
This was referenced Mar 14, 2022
Merged
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pmalek-sumo pmalek-sumo pmalek-sumo left review comments

@perk-sumo perk-sumo perk-sumo approved these changes

@pmm-sumo pmm-sumo pmm-sumo approved these changes

@github-actions github-actions[bot] github-actions[bot] approved these changes

@sumo-drosiek sumo-drosiek Awaiting requested review from sumo-drosiek

@kasia-kujawa kasia-kujawa Awaiting requested review from kasia-kujawa

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v2.3
Development

Successfully merging this pull request may close these issues.
5 participants
@mat-rumian @sumo-drosiek @pmalek-sumo @perk-sumo @pmm-sumo