-
Notifications
You must be signed in to change notification settings - Fork 183
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Expose http sources for metrics in values.yaml #672
Changes from all commits
ee0887a
4939f1a
4e8c909
567f3d6
3e61563
9cd905c
8fcbe09
0c11787
4ec3554
0e84844
b7be567
f251e02
5e6be12
23d4d18
2af9aaf
d97b315
e4b6073
df6de55
b0621b2
cd2ebac
3e52c14
6ef11c2
7c46476
ade7dd4
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -35,7 +35,7 @@ | |
@type sumologic | ||
@id sumologic.endpoint.events | ||
sumo_client {{ include "sumologic.sumo_client" . | quote }} | ||
endpoint "#{ENV['SUMO_ENDPOINT_EVENTS']}" | ||
endpoint "#{ENV['SUMO_ENDPOINT_EVENTS_SOURCE']}" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Would this be considered a breaking change for non-helm users? Helm should handle this, but what abotu non-helm. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. actually since non-helm users just run However there IS a breaking change here, and that is a result of the keys in the sumologic secret being changed (added suffix There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good catch, yeah we dont what the breaking change from the secret. |
||
data_type logs | ||
disable_cookies true | ||
verify_ssl {{ .Values.fluentd.verifySsl | quote }} | ||
|
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -353,27 +353,157 @@ Generate metrics match configuration | |||||
Example usage (as one line): | ||||||
|
||||||
{{ include "utils.metrics.match" (dict | ||||||
"Values" .Values | ||||||
"Match" "prometheus.metrics.kubelet" | ||||||
"Values" . | ||||||
"Tag" "prometheus.metrics.kubelet" | ||||||
"Endpoint" "SUMO_ENDPOINT_METRICS" | ||||||
"Storage" .Values.fluentd.buffer.filePaths.metrics.default | ||||||
"Id" sumologic.endpoint.metrics | ||||||
)}} | ||||||
*/}} | ||||||
{{- define "utils.metrics.match" -}} | ||||||
<match {{ .Match }}> | ||||||
<match {{ .Tag }}> | ||||||
{{- if .Drop }} | ||||||
@type null | ||||||
{{- else }} | ||||||
@type sumologic | ||||||
@id {{ .Id }} | ||||||
endpoint "#{ENV['{{ .Endpoint }}']}" | ||||||
{{- .Values.fluentd.metrics.outputConf | nindent 2 }} | ||||||
sumo_client {{ include "sumologic.sumo_client" .Context | quote }} | ||||||
endpoint "#{ENV['{{ include "terraform.sources.endpoint" .Endpoint}}']}" | ||||||
{{- .Context.Values.fluentd.metrics.outputConf | nindent 2 }} | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What does There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I need to pass k8s context to have access to .Values,
|
||||||
<buffer> | ||||||
{{- if or .Values.fluentd.persistence.enabled (eq .Values.fluentd.buffer.type "file") }} | ||||||
{{- if or .Context.Values.fluentd.persistence.enabled (eq .Context.Values.fluentd.buffer.type "file") }} | ||||||
@type file | ||||||
path {{ .Storage }} | ||||||
{{- else }} | ||||||
@type memory | ||||||
{{- end }} | ||||||
@include buffer.output.conf | ||||||
</buffer> | ||||||
{{- end }} | ||||||
</match> | ||||||
{{ end -}} | ||||||
|
||||||
{{/* | ||||||
Convert source name to terraform name: | ||||||
* converts all `-` to `_` | ||||||
* adds `_source` suffix | ||||||
|
||||||
Example usage: | ||||||
|
||||||
{{ include "terraform.sources.name" $source }} | ||||||
|
||||||
*/}} | ||||||
{{- define "terraform.sources.name" -}} | ||||||
{{ replace "-" "_" . }}_source | ||||||
{{- end -}} | ||||||
|
||||||
{{/* | ||||||
Convert source name to terraform metric name: | ||||||
* converts all `-` to `_` | ||||||
* adds `_metrics_source` suffix | ||||||
|
||||||
Example usage: | ||||||
|
||||||
{{ include "terraform.sources.name_metrics" $source }} | ||||||
|
||||||
*/}} | ||||||
{{- define "terraform.sources.name_metrics" -}} | ||||||
{{ replace "-" "_" . }}_metrics_source | ||||||
{{- end -}} | ||||||
|
||||||
{{/* | ||||||
Generate endpoint variable string for given string | ||||||
|
||||||
Example usage: | ||||||
|
||||||
{{ include "terraform.sources.endpoint" "logs" }} | ||||||
|
||||||
*/}} | ||||||
{{- define "terraform.sources.endpoint" -}} | ||||||
SUMO_ENDPOINT_{{ replace "-" "_" . | upper }} | ||||||
{{- end -}} | ||||||
|
||||||
{{/* | ||||||
Generate endpoint variable string for given string | ||||||
|
||||||
Example usage: | ||||||
|
||||||
{{ include "terraform.sources.endpoint" "logs" }} | ||||||
|
||||||
*/}} | ||||||
{{- define "terraform.sources.endpoint_name" -}} | ||||||
{{ printf "endpoint-%s" . }} | ||||||
{{- end -}} | ||||||
|
||||||
{{/* | ||||||
Generate line for local terraform section | ||||||
* `terraform.sources.local = value` | ||||||
|
||||||
Example usage: | ||||||
|
||||||
{{ include "terraform.sources.local" $source }} | ||||||
|
||||||
*/}} | ||||||
{{- define "terraform.sources.local" -}} | ||||||
{{ printf "%-43s = \"%s\"" .Name .Value }} | ||||||
{{- end -}} | ||||||
|
||||||
{{/* | ||||||
Generate line for data terraform section | ||||||
|
||||||
Example usage: | ||||||
|
||||||
{{ include "terraform.sources.data" (dict "Endpoint" "enpoint-default-metrics" "Name" "default") }} | ||||||
|
||||||
*/}} | ||||||
{{- define "terraform.sources.data" -}} | ||||||
{{ printf "%-41s = \"${sumologic_http_source.%s.url}\"" .Endpoint .Name }} | ||||||
{{- end -}} | ||||||
|
||||||
{{/* | ||||||
Generate resource sections | ||||||
|
||||||
Example usage: | ||||||
|
||||||
{{ include "terraform.sources.resource" (dict "Source" $source "Context" $ctx) }} | ||||||
|
||||||
*/}} | ||||||
{{- define "terraform.sources.resource" -}} | ||||||
{{- $source := .Source -}} | ||||||
{{- $ctx := .Context -}} | ||||||
resource "sumologic_http_source" "{{ .Name }}" { | ||||||
name = local.{{ .Name }} | ||||||
collector_id = "${sumologic_collector.collector.id}" | ||||||
{{- if $source.category }} | ||||||
category = {{ if $ctx.fluentd.events.sourceCategory }}{{ $ctx.fluentd.events.sourceCategory | quote }}{{- else}}{{ "\"${var.cluster_name}/${local.events_source}\"" }}{{- end}} | ||||||
{{- end }} | ||||||
{{- if $source.fields }} | ||||||
{{- range $fkey, $fvalue := $source.fields }} | ||||||
{{ $fkey }} = "{{ $fvalue }}" | ||||||
{{- end -}} | ||||||
{{ end }} | ||||||
} | ||||||
{{- end -}} | ||||||
|
||||||
{{/* | ||||||
get configuration variable name for sources confg map | ||||||
|
||||||
Example usage: | ||||||
|
||||||
{{ include "terraform.sources.config-map-variable" (dict "Context" .Values "Name" $name "Endpoint" $endpoint) }} | ||||||
|
||||||
*/}} | ||||||
{{- define "terraform.sources.config-map-variable" -}} | ||||||
{{- $name := .Name -}} | ||||||
{{- $ctx := .Context -}} | ||||||
{{- $endpoint := .Endpoint -}} | ||||||
{{- if not $endpoint -}} | ||||||
{{- $source := (index $ctx.sumologic.sources "default") -}} | ||||||
{{- if (index $ctx.sumologic.sources .Name "config-name") -}} | ||||||
{{- $endpoint = index $ctx.sumologic.sources .Name "config-name" -}} | ||||||
{{- else -}} | ||||||
{{- $endpoint = printf "endpoint-%s" (include "terraform.sources.name_metrics" $name) -}} | ||||||
{{- end -}} | ||||||
{{- end -}} | ||||||
{{ $endpoint }} | ||||||
{{- end -}} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixes prometheus issue. It has been saying that configuration for
http://$(CHART).$(NAMESPACE).svc.cluster.local:9888/prometheus.metrics.control-plane
is duplicatedThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The only thing i find misleading with this setup is that we collect from other control plane components and do not prefix them with control-plane. Any reason to just remove the control-plane part?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, the fluentd configuration will be more complex: either more complex regex for tags or output section per endpoint
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How many metrics is each of these producing? We need to be conscious of blacklisting and if we are close to the limit, we should distribute across multiiple HTTP sources to prevent that.