Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Otel statefulset cloudwatch collector #2982

Merged
merged 7 commits into from
Apr 20, 2023
Merged

Otel statefulset cloudwatch collector #2982

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
2d9ce39
Adding the otel statefulset cloudwatch collector to the helm chart
rnishtala-sumo Apr 11, 2023
9dc2d88
Adding tests for otelcloudwatch
rnishtala-sumo Apr 13, 2023
50ea9db
Updating fargate readme for logs collection
rnishtala-sumo Apr 13, 2023
f205f8f
feat(fargate): Cloudwatch receiver persistence is configurable and no…
rnishtala-sumo Apr 17, 2023
f9662e3
fix(doc): documentation changes to include example cloudwatch receive…
rnishtala-sumo Apr 18, 2023
eed8faa
fix(configmap): Separated the configmap for cloudwatch collector
rnishtala-sumo Apr 19, 2023
6e146de
docs(readme): Updated the new configuration keys for otelcloudwatch i…
rnishtala-sumo Apr 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .changelog/2982.added.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Added the otel cloudwatch statefulset logs collector (fargate) with documentation updates
9 changes: 8 additions & 1 deletion deploy/helm/sumologic/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,13 @@ The following table lists the configurable parameters of the Sumo Logic chart an
| `sumologic.events.persistence.persistentVolume.pvcLabels` | Additional PersistentVolumeClaim labels for persistent storage volumes | `{}` |
| `sumologic.logs.enabled` | Set the enabled flag to false for disabling logs ingestion altogether. | `true` |
| `sumologic.logs.collector.allowSideBySide` | Allow running otel and Fluent Bit side by side. This will result in duplicated logs being ingested. Only enable this if you're **certain** it's what you want. | `false` |
| `sumologic.logs.collector.otelcol.enabled` | Enable OpenTelemtry logs collector. | `true` |
| `sumologic.logs.collector.otelcol.enabled` | Enable OpenTelemetry logs collector. | `true` |
| `sumologic.logs.collector.otelcloudwatch.enabled` | Flag to enable CloudWatch Collection | `false` |
| `sumologic.logs.collector.otelcloudwatch.logGroups` | Log Groups configuration for AWS CloudWatch receiver | `{}` |
| `sumologic.logs.collector.otelcloudwatch.persistence.enabled` | Flag to control persistence for the CloudWatch collector | `true` |
| `sumologic.logs.collector.otelcloudwatch.pollInterval` | CloudWatch poll interval | `1m` |
| `sumologic.logs.collector.otelcloudwatch.region` | EKS Fargate cluster region | `""` |
| `sumologic.logs.collector.otelcloudwatch.roleArn` | AWS role ARN, to authenticate with CloudWatch | `""` |
| `sumologic.logs.container.enabled` | Enable collecting logs from Kubernetes containers. | `true` |
| `sumologic.logs.container.format` | Format for container logs. | `fields` |
| `sumologic.logs.multiline.enabled` | Enable multiline detection for Kubernetes container logs. | `true` |
Expand Down Expand Up @@ -489,6 +495,7 @@ The following table lists the configurable parameters of the Sumo Logic chart an
| `otellogs.config.merge` | Configuration for log collector otelcol, merged with defaults. See also https://github.com/SumoLogic/sumologic-otel-collector/blob/main/docs/configuration.md. | {} |
| `otellogs.config.override` | Configuration for log collector otelcol, replaces defaults. See also https://github.com/SumoLogic/sumologic-otel-collector/blob/main/docs/configuration.md. | {} |
| `otellogs.daemonset` | OpenTelemetry Collector Daemonset customization options. See [values.yaml] for more details. | See [values.yaml] |
| `otelcloudwatch.statefulset` | OpenTelemetry Cloudwatch Collector statefulset customization options. See [values.yaml] for more details. | See [values.yaml] |
| `otellogs.additionalDaemonSets` | OpenTelemetry Collector Daemonset per node customization options. See [Best Practices](/docs/best-practices.md#setting-different-resources-on-different-nodes-for-logs-collector). | `{}` |
| `otellogs.metrics.enabled` | Enable OpenTelemetry Collector metrics | `true` |
| `otellogs.serviceLabels` | Add custom labels to OpenTelemetry Collector Service | `{}` |
Expand Down
54 changes: 54 additions & 0 deletions deploy/helm/sumologic/conf/logs/collector/otelcloudwatch/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
exporters:
## ref: https://github.com/open-telemetry/opentelemetry-collector/tree/v0.73.0/exporter/otlphttpexporter
otlphttp:
endpoint: http://${LOGS_METADATA_SVC}.${NAMESPACE}.svc.cluster.local.:4318
sending_queue:
queue_size: 10

extensions:
## ref: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.73.0/extension/storage/filestorage
file_storage:
compaction:
directory: /var/lib/storage/otc
on_rebound: true
directory: /var/lib/storage/otc
timeout: 10s

## ref: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.73.0/extension/healthcheckextension
health_check: {}

## ref: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.73.0/extension/pprofextension
pprof: {}

processors:
## ref: https://github.com/open-telemetry/opentelemetry-collector/tree/v0.73.0/processor/batchprocessor
batch:
send_batch_max_size: 2000
send_batch_size: 1000
timeout: 1s

receivers:
awscloudwatch:
region: {{ .Values.sumologic.logs.collector.otelcloudwatch.region }}
logs:
poll_interval: {{ .Values.sumologic.logs.collector.otelcloudwatch.pollInterval }}
groups:
named:
{{ toYaml .Values.sumologic.logs.collector.otelcloudwatch.logGroups | indent 10 }}

service:
extensions:
- health_check
- file_storage
- pprof
pipelines:
logs/collector/otelcloudwatch:
receivers:
- awscloudwatch
processors:
- batch
exporters:
- otlphttp
telemetry:
logs:
level: {{ .Values.otellogs.logLevel | quote }}
36 changes: 35 additions & 1 deletion deploy/helm/sumologic/conf/logs/otelcol/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,27 @@
receivers:
{{ if eq (include "logs.collector.fluentbit.enabled" .) "true" }}
fluentforward:
endpoint: 0.0.0.0:24321
{{ end }}
{{ if or .Values.sumologic.logs.collector.otelcol.enabled .Values.sumologic.logs.collector.otelcloudwatch.enabled }}
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
{{ end }}
extensions:
health_check: {}
{{ if .Values.metadata.persistence.enabled }}
## Configuration for File Storage extension
## ref: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/release/v0.37.x/extension/storage/filestorage
file_storage:
directory: /var/lib/storage/otc
timeout: 10s
compaction:
on_rebound: true
directory: /tmp
{{ end }}
pprof: {}
exporters:
{{ if .Values.sumologic.logs.container.enabled }}
## ref: https://github.com/SumoLogic/sumologic-otel-collector/tree/main/pkg/exporter/sumologicexporter
Expand Down Expand Up @@ -582,7 +606,17 @@ service:
- fluentforward
{{ end }}
{{ end }}

## Use this pipeline for cloudwatch logs
{{- if .Values.sumologic.logs.collector.otelcloudwatch.enabled }}
logs/otlp/otelcloudwatch:
receivers:
- otlp
processors:
- memory_limiter
- batch
exporters:
- sumologic/containers
{{ end }}
{{ if eq (include "logs.collector.otelcol.enabled" .) "true" }}
{{ if .Values.sumologic.logs.container.enabled }}
## This is the same pipeline like for logs/fluent/containers with the following modifications:
Expand Down
28 changes: 28 additions & 0 deletions deploy/helm/sumologic/templates/_helpers/_logs.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,10 @@ Example Usage:
{{ $enabled }}
{{- end -}}

{{- define "logs.collector.otelcloudwatch.enabled" -}}
{{- $enabled := and (eq (include "logs.enabled" .) "true") (eq .Values.sumologic.logs.collector.otelcloudwatch.enabled true) -}}
{{- end -}}

{{/*
Check if Fluent-Bit logs collector is enabled.
It's enabled if logs in general are enabled and fluent-bit.enabled is set to true.
Expand Down Expand Up @@ -131,6 +135,10 @@ Return the log format for the Sumologic exporter for container logs.
{{- template "sumologic.metadata.name.logs.collector" . }}
{{- end -}}

{{- define "sumologic.metadata.name.logs.collector.statefulset" -}}
{{- template "sumologic.metadata.name.logs.collector.cloudwatch" . }}
{{- end -}}

{{- define "sumologic.metadata.name.logs.collector.service" -}}
{{- template "sumologic.metadata.name.logs.collector" . }}
{{- end -}}
Expand Down Expand Up @@ -159,6 +167,22 @@ Return the log format for the Sumologic exporter for container logs.
{{- template "sumologic.labels.app.logs.collector" . }}
{{- end -}}

{{- define "sumologic.labels.app.logs.cloudwatch.configmap" -}}
{{- template "sumologic.fullname" . }}-otelcloudwatch-logs-collector
{{- end -}}

{{- define "sumologic.labels.app.logs.cloudwatch.service" -}}
{{- template "sumologic.metadata.name.logs.collector.cloudwatch" . }}
{{- end -}}

{{- define "sumologic.labels.app.logs.cloudwatch.service-headless" -}}
{{- template "sumologic.labels.app.logs.cloudwatch.service" . }}-headless
{{- end -}}

{{- define "sumologic.labels.app.logs.collector.statefulset" -}}
{{- template "sumologic.fullname" . }}-otelcloudwatch-logs-collector
{{- end -}}

{{- define "sumologic.labels.app.logs.pod" -}}
{{- template "sumologic.labels.app.logs" . }}
{{- end -}}
Expand Down Expand Up @@ -219,6 +243,10 @@ Return the log format for the Sumologic exporter for container logs.
{{- template "sumologic.fullname" . }}-otelcol-logs-collector
{{- end -}}

{{- define "sumologic.metadata.name.logs.collector.cloudwatch" -}}
{{- template "sumologic.fullname" . }}-otelcol-cloudwatch-collector
{{- end -}}

{{- define "sumologic.labels.logs" -}}
sumologic.com/app: fluentd-logs
sumologic.com/component: logs
Expand Down
21 changes: 21 additions & 0 deletions deploy/helm/sumologic/templates/logs/collector/otelcol-cloudwatch/configmap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{{- if .Values.sumologic.logs.collector.otelcloudwatch.enabled }}
{{ $baseConfig := (tpl (.Files.Get "conf/logs/collector/otelcloudwatch/config.yaml") .) | fromYaml }}
{{ $mergeConfig := .Values.otellogs.config.merge }}
{{ $overrideConfig := .Values.otellogs.config.override }}
{{ $finalConfig := "" }}
{{ if $overrideConfig }}
{{ $finalConfig = $overrideConfig }}
{{ else }}
{{ $finalConfig = mergeOverwrite $baseConfig $mergeConfig }}
{{ end }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "sumologic.metadata.name.logs.collector.cloudwatch" . }}
labels:
app: {{ template "sumologic.labels.app.logs.cloudwatch.configmap" . }}
{{- include "sumologic.labels.common" . | nindent 4 }}
data:
config.yaml: |
{{- $finalConfig | toYaml | nindent 4 }}
{{- end }}
Loading