Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose http sources for metrics in values.yaml #672

Merged
merged 24 commits into from
Jun 1, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions deploy/helm/prometheus-overrides.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -401,13 +401,13 @@ prometheus:
sourceLabels: [__name__]
- # control plane metrics
# coredns
url: http://$(CHART).$(NAMESPACE).svc.cluster.local:9888/prometheus.metrics.control-plane
url: http://$(CHART).$(NAMESPACE).svc.cluster.local:9888/prometheus.metrics.control-plane.coredns
writeRelabelConfigs:
- action: keep
regex: coredns
sourceLabels: [job]
- # etcd server
url: http://$(CHART).$(NAMESPACE).svc.cluster.local:9888/prometheus.metrics.control-plane
url: http://$(CHART).$(NAMESPACE).svc.cluster.local:9888/prometheus.metrics.control-plane.kube-etcd
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixes prometheus issue. It has been saying that configuration for http://$(CHART).$(NAMESPACE).svc.cluster.local:9888/prometheus.metrics.control-plane is duplicated

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only thing i find misleading with this setup is that we collect from other control plane components and do not prefix them with control-plane. Any reason to just remove the control-plane part?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the fluentd configuration will be more complex: either more complex regex for tags or output section per endpoint

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How many metrics is each of these producing? We need to be conscious of blacklisting and if we are close to the limit, we should distribute across multiiple HTTP sources to prevent that.

writeRelabelConfigs:
- action: keep
regex: kube-etcd
Expand Down
2 changes: 1 addition & 1 deletion deploy/helm/sumologic/conf/events/events.conf
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
@type sumologic
@id sumologic.endpoint.events
sumo_client {{ include "sumologic.sumo_client" . | quote }}
endpoint "#{ENV['SUMO_ENDPOINT_EVENTS']}"
endpoint "#{ENV['SUMO_ENDPOINT_EVENTS_SOURCE']}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this be considered a breaking change for non-helm users? Helm should handle this, but what abotu non-helm.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually since non-helm users just run kubectl apply -f fluentd-sumologic.yaml to upgrade, the new yaml file should have the renamed environment variables so it should be fine.

However there IS a breaking change here, and that is a result of the keys in the sumologic secret being changed (added suffix -source), which means that non-helm users will have to re-run the setup job in order to get an updated secret (though the collector and sources will remain untouched).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, yeah we dont what the breaking change from the secret.

data_type logs
disable_cookies true
verify_ssl {{ .Values.fluentd.verifySsl | quote }}
Expand Down
2 changes: 1 addition & 1 deletion deploy/helm/sumologic/conf/logs/logs.output.conf
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
data_type logs
log_key log
endpoint "#{ENV['SUMO_ENDPOINT_LOGS']}"
endpoint "#{ENV['SUMO_ENDPOINT_LOGS_SOURCE']}"
verify_ssl {{ .Values.fluentd.verifySsl | quote }}
log_format {{ .Values.fluentd.logs.output.logFormat | quote }}
add_timestamp {{ .Values.fluentd.logs.output.addTimestamp | quote }}
Expand Down
27 changes: 16 additions & 11 deletions deploy/helm/sumologic/conf/metrics/metrics.conf
Original file line number Diff line number Diff line change
Expand Up @@ -47,14 +47,19 @@
</filter>
{{- .Values.fluentd.metrics.extraOutputPluginConf | nindent 4 }}

{{ include "utils.metrics.match" (dict "Values" .Values "Match" "prometheus.metrics.apiserver**" "Id" "sumologic.endpoint.metrics.apiserver" "Endpoint" "SUMO_ENDPOINT_METRICS_APISERVER" "Storage" .Values.fluentd.buffer.filePaths.metrics.apiserver) | nindent 4 }}
{{ include "utils.metrics.match" (dict "Values" .Values "Match" "prometheus.metrics.kubelet**" "Id" "sumologic.endpoint.metrics.kubelet" "Endpoint" "SUMO_ENDPOINT_METRICS_KUBELET" "Storage" .Values.fluentd.buffer.filePaths.metrics.kubelet) | nindent 4 }}
{{ include "utils.metrics.match" (dict "Values" .Values "Match" "prometheus.metrics.container**" "Id" "sumologic.endpoint.metrics.container" "Endpoint" "SUMO_ENDPOINT_METRICS_KUBELET" "Storage" .Values.fluentd.buffer.filePaths.metrics.container) | nindent 4 }}
{{ include "utils.metrics.match" (dict "Values" .Values "Match" "prometheus.metrics.controller-manager**" "Id" "sumologic.endpoint.metrics.kube.controller.manager" "Endpoint" "SUMO_ENDPOINT_METRICS_KUBE_CONTROLLER_MANAGER" "Storage" .Values.fluentd.buffer.filePaths.metrics.controller) | nindent 4 }}
{{ include "utils.metrics.match" (dict "Values" .Values "Match" "prometheus.metrics.scheduler**" "Id" "sumologic.endpoint.metrics.kube.scheduler" "Endpoint" "SUMO_ENDPOINT_METRICS_KUBE_SCHEDULER" "Storage" .Values.fluentd.buffer.filePaths.metrics.scheduler) | nindent 4 }}
{{ include "utils.metrics.match" (dict "Values" .Values "Match" "prometheus.metrics.state**" "Id" "sumologic.endpoint.metrics.kube.state" "Endpoint" "SUMO_ENDPOINT_METRICS_KUBE_STATE" "Storage" .Values.fluentd.buffer.filePaths.metrics.state) | nindent 4 }}
{{ include "utils.metrics.match" (dict "Values" .Values "Match" "prometheus.metrics.node**" "Id" "sumologic.endpoint.metrics.node.exporter" "Endpoint" "SUMO_ENDPOINT_METRICS_NODE_EXPORTER" "Storage" .Values.fluentd.buffer.filePaths.metrics.node) | nindent 4 }}
{{ include "utils.metrics.match" (dict "Values" .Values "Match" "prometheus.metrics.control-plane**" "Id" "sumologic.endpoint.metrics.control.plane" "Endpoint" "SUMO_ENDPOINT_METRICS_CONTROL_PLANE" "Storage" .Values.fluentd.buffer.filePaths.metrics.control_plane) | nindent 4 }}
{{ include "utils.metrics.match" (dict "Values" .Values "Match" "prometheus.metrics**" "Id" "sumologic.endpoint.metrics" "Endpoint" "SUMO_ENDPOINT_METRICS" "Storage" .Values.fluentd.buffer.filePaths.metrics.default) | nindent 4 }}

</label>
{{- $ctx := . }}
{{- $out_keys := list "" -}}
{{- range $key, $value := .Values.fluentd.metrics.output -}}
{{- $out_keys = append $out_keys (printf "%d_%d_%s" (len ( $value.weight | int | toString )) ($value.weight | int) $key) -}}
{{- end -}}
{{- $out_keys := slice $out_keys 1 }}
{{- range $_, $key := $out_keys | sortAlpha }}
{{- $name := join "_" (slice (splitList "_" $key) 2) }}
{{- $source := index $ctx.Values.fluentd.metrics.output $name }}
{{- $endpoint := $name }}
{{- if $source.source -}}
{{- $endpoint = $source.source -}}
{{- end -}}
{{ include "utils.metrics.match" (dict "Context" $ctx "Drop" $source.drop "Tag" $source.tag "Id" $source.id "Endpoint" (include "terraform.sources.name_metrics" $endpoint) "Storage" (index $ctx.Values.fluentd.buffer.filePaths.metrics $name) ) | nindent 4 }}
{{- end }}
</label>
15 changes: 6 additions & 9 deletions deploy/helm/sumologic/conf/setup/setup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -15,15 +15,12 @@ terraform init

# Sumo Collector and HTTP sources
terraform import sumologic_collector.collector "$COLLECTOR_NAME"
terraform import sumologic_http_source.default_metrics_source "$COLLECTOR_NAME/(default-metrics)"
terraform import sumologic_http_source.apiserver_metrics_source "$COLLECTOR_NAME/apiserver-metrics"
terraform import sumologic_http_source.events_source "$COLLECTOR_NAME/events"
terraform import sumologic_http_source.kube_controller_manager_metrics_source "$COLLECTOR_NAME/kube-controller-manager-metrics"
terraform import sumologic_http_source.kube_scheduler_metrics_source "$COLLECTOR_NAME/kube-scheduler-metrics"
terraform import sumologic_http_source.kube_state_metrics_source "$COLLECTOR_NAME/kube-state-metrics"
terraform import sumologic_http_source.kubelet_metrics_source "$COLLECTOR_NAME/kubelet-metrics"
terraform import sumologic_http_source.logs_source "$COLLECTOR_NAME/logs"
terraform import sumologic_http_source.node_exporter_metrics_source "$COLLECTOR_NAME/node-exporter-metrics"
{{ range $key, $source := .Values.sumologic.sources }}
terraform import sumologic_http_source.{{ template "terraform.sources.name_metrics" $key }} "$COLLECTOR_NAME/{{ $source.value }}"
{{- end }}
terraform import sumologic_http_source.{{ template "terraform.sources.name" "logs" }} "$COLLECTOR_NAME/logs"
terraform import sumologic_http_source.{{ template "terraform.sources.name" "events" }} "$COLLECTOR_NAME/events"


# Kubernetes Namespace and Secret
terraform import kubernetes_namespace.sumologic_collection_namespace {{ .Release.Namespace }}
Expand Down
81 changes: 18 additions & 63 deletions deploy/helm/sumologic/conf/setup/sumo-k8s.tf
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
{{- $logs := (dict "value" "logs" "endpoint" "logs" )}}
{{- $events := (dict "value" "events" "endpoint" "events" "category" true )}}
variable "cluster_name" {
type = string
default = "{{ template "sumologic.clusterNameReplaceSpaceWithDash" . }}"
Expand All @@ -21,15 +23,11 @@ variable "namespace_name" {
}

locals {
default-metrics-source-name = "(default-metrics)"
apiserver-metrics-source-name = "apiserver-metrics"
events-source-name = "events"
kube-controller-manager-metrics-source-name = "kube-controller-manager-metrics"
kube-scheduler-metrics-source-name = "kube-scheduler-metrics"
kube-state-metrics-source-name = "kube-state-metrics"
kubelet-metrics-source-name = "kubelet-metrics"
logs-source-name = "logs"
node-exporter-metrics-source-name = "node-exporter-metrics"
{{- range $key, $source := .Values.sumologic.sources }}
{{ template "terraform.sources.local" (dict "Name" (include "terraform.sources.name_metrics" $key) "Value" $source.name) }}
{{- end }}
{{ template "terraform.sources.local" (dict "Name" (include "terraform.sources.name" "logs") "Value" "logs") }}
{{ template "terraform.sources.local" (dict "Name" (include "terraform.sources.name" "events") "Value" "events") }}
}

provider "sumologic" {}
Expand All @@ -41,51 +39,12 @@ resource "sumologic_collector" "collector" {
}
}

resource "sumologic_http_source" "default_metrics_source" {
name = local.default-metrics-source-name
collector_id = "${sumologic_collector.collector.id}"
}

resource "sumologic_http_source" "apiserver_metrics_source" {
name = local.apiserver-metrics-source-name
collector_id = "${sumologic_collector.collector.id}"
}

resource "sumologic_http_source" "events_source" {
name = local.events-source-name
category = {{ if .Values.fluentd.events.sourceCategory }}{{ .Values.fluentd.events.sourceCategory | quote }}{{- else}}{{ "\"${var.cluster_name}/${local.events-source-name}\"" }}{{- end}}
collector_id = "${sumologic_collector.collector.id}"
}

resource "sumologic_http_source" "kube_controller_manager_metrics_source" {
name = local.kube-controller-manager-metrics-source-name
collector_id = "${sumologic_collector.collector.id}"
}

resource "sumologic_http_source" "kube_scheduler_metrics_source" {
name = local.kube-scheduler-metrics-source-name
collector_id = "${sumologic_collector.collector.id}"
}

resource "sumologic_http_source" "kube_state_metrics_source" {
name = local.kube-state-metrics-source-name
collector_id = "${sumologic_collector.collector.id}"
}

resource "sumologic_http_source" "kubelet_metrics_source" {
name = local.kubelet-metrics-source-name
collector_id = "${sumologic_collector.collector.id}"
}

resource "sumologic_http_source" "logs_source" {
name = local.logs-source-name
collector_id = "${sumologic_collector.collector.id}"
}

resource "sumologic_http_source" "node_exporter_metrics_source" {
name = local.node-exporter-metrics-source-name
collector_id = "${sumologic_collector.collector.id}"
}
{{- $ctx := .Values -}}
{{- range $key, $source := .Values.sumologic.sources }}
{{ include "terraform.sources.resource" (dict "Name" (include "terraform.sources.name_metrics" $key) $key "Source" $source "Context" $ctx) | nindent 2 }}
{{- end }}
{{ include "terraform.sources.resource" (dict "Name" (include "terraform.sources.name" "logs") "Source" $logs "Context" $ctx) | nindent 2 }}
{{ include "terraform.sources.resource" (dict "Name" (include "terraform.sources.name" "events") "Source" $events "Context" $ctx) | nindent 2 }}

provider "kubernetes" {
{{- $ctx := .Values -}}
Expand Down Expand Up @@ -125,15 +84,11 @@ resource "kubernetes_secret" "sumologic_collection_secret" {
}

data = {
endpoint-events = "${sumologic_http_source.events_source.url}"
endpoint-logs = "${sumologic_http_source.logs_source.url}"
endpoint-metrics = "${sumologic_http_source.default_metrics_source.url}"
endpoint-metrics-apiserver = "${sumologic_http_source.apiserver_metrics_source.url}"
endpoint-metrics-kube-controller-manager = "${sumologic_http_source.kube_controller_manager_metrics_source.url}"
endpoint-metrics-kube-scheduler = "${sumologic_http_source.kube_scheduler_metrics_source.url}"
endpoint-metrics-kube-state = "${sumologic_http_source.kube_state_metrics_source.url}"
endpoint-metrics-kubelet = "${sumologic_http_source.kubelet_metrics_source.url}"
endpoint-metrics-node-exporter = "${sumologic_http_source.node_exporter_metrics_source.url}"
{{ range $key, $source := .Values.sumologic.sources -}}
{{ include "terraform.sources.data" (dict "Endpoint" (include "terraform.sources.config-map-variable" (dict "Context" $ctx "Name" $key)) "Name" (include "terraform.sources.name_metrics" $key)) }}
{{ end -}}
{{ include "terraform.sources.data" (dict "Endpoint" (include "terraform.sources.config-map-variable" (dict "Context" $ctx "Name" "logs" "Endpoint" "endpoint-logs")) "Name" (include "terraform.sources.name" "logs")) }}
{{ include "terraform.sources.data" (dict "Endpoint" (include "terraform.sources.config-map-variable" (dict "Context" $ctx "Name" "events" "Endpoint" "endpoint-events")) "Name" (include "terraform.sources.name" "events")) }}
}

type = "Opaque"
Expand Down
142 changes: 136 additions & 6 deletions deploy/helm/sumologic/templates/_helpers.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -353,27 +353,157 @@ Generate metrics match configuration
Example usage (as one line):

{{ include "utils.metrics.match" (dict
"Values" .Values
"Match" "prometheus.metrics.kubelet"
"Values" .
"Tag" "prometheus.metrics.kubelet"
"Endpoint" "SUMO_ENDPOINT_METRICS"
"Storage" .Values.fluentd.buffer.filePaths.metrics.default
"Id" sumologic.endpoint.metrics
)}}
*/}}
{{- define "utils.metrics.match" -}}
<match {{ .Match }}>
<match {{ .Tag }}>
{{- if .Drop }}
@type null
{{- else }}
@type sumologic
@id {{ .Id }}
endpoint "#{ENV['{{ .Endpoint }}']}"
{{- .Values.fluentd.metrics.outputConf | nindent 2 }}
sumo_client {{ include "sumologic.sumo_client" .Context | quote }}
endpoint "#{ENV['{{ include "terraform.sources.endpoint" .Endpoint}}']}"
{{- .Context.Values.fluentd.metrics.outputConf | nindent 2 }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does .Context do? Is it for the kubectl context?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to pass k8s context to have access to .Values, .Context in this template is simply . from kubernetes.
. is override in range functions and templates, so there is need to pass it as a variable (or a dict value for templates, because they get only one argument). Here is flow (btw it covers both cases):


{{ include "utils.metrics.match" (dict "Context" $ctx "Drop" $source.drop "Tag" $source.tag "Id" $source.id "Endpoint" (include "terraform.sources.name_metrics" $endpoint) "Storage" (index $ctx.Values.fluentd.buffer.filePaths.metrics $name) ) | nindent 4 }}

<buffer>
{{- if or .Values.fluentd.persistence.enabled (eq .Values.fluentd.buffer.type "file") }}
{{- if or .Context.Values.fluentd.persistence.enabled (eq .Context.Values.fluentd.buffer.type "file") }}
@type file
path {{ .Storage }}
{{- else }}
@type memory
{{- end }}
@include buffer.output.conf
</buffer>
{{- end }}
</match>
{{ end -}}

{{/*
Convert source name to terraform name:
* converts all `-` to `_`
* adds `_source` suffix

Example usage:

{{ include "terraform.sources.name" $source }}

*/}}
{{- define "terraform.sources.name" -}}
{{ replace "-" "_" . }}_source
{{- end -}}

{{/*
Convert source name to terraform metric name:
* converts all `-` to `_`
* adds `_metrics_source` suffix

Example usage:

{{ include "terraform.sources.name_metrics" $source }}

*/}}
{{- define "terraform.sources.name_metrics" -}}
{{ replace "-" "_" . }}_metrics_source
{{- end -}}

{{/*
Generate endpoint variable string for given string

Example usage:

{{ include "terraform.sources.endpoint" "logs" }}

*/}}
{{- define "terraform.sources.endpoint" -}}
SUMO_ENDPOINT_{{ replace "-" "_" . | upper }}
{{- end -}}

{{/*
Generate endpoint variable string for given string

Example usage:

{{ include "terraform.sources.endpoint" "logs" }}

*/}}
{{- define "terraform.sources.endpoint_name" -}}
{{ printf "endpoint-%s" . }}
{{- end -}}

{{/*
Generate line for local terraform section
* `terraform.sources.local = value`

Example usage:

{{ include "terraform.sources.local" $source }}

*/}}
{{- define "terraform.sources.local" -}}
{{ printf "%-43s = \"%s\"" .Name .Value }}
{{- end -}}

{{/*
Generate line for data terraform section

Example usage:

{{ include "terraform.sources.data" (dict "Endpoint" "enpoint-default-metrics" "Name" "default") }}

*/}}
{{- define "terraform.sources.data" -}}
{{ printf "%-41s = \"${sumologic_http_source.%s.url}\"" .Endpoint .Name }}
{{- end -}}

{{/*
Generate resource sections

Example usage:

{{ include "terraform.sources.resource" (dict "Source" $source "Context" $ctx) }}

*/}}
{{- define "terraform.sources.resource" -}}
{{- $source := .Source -}}
{{- $ctx := .Context -}}
resource "sumologic_http_source" "{{ .Name }}" {
name = local.{{ .Name }}
collector_id = "${sumologic_collector.collector.id}"
{{- if $source.category }}
category = {{ if $ctx.fluentd.events.sourceCategory }}{{ $ctx.fluentd.events.sourceCategory | quote }}{{- else}}{{ "\"${var.cluster_name}/${local.events_source}\"" }}{{- end}}
{{- end }}
{{- if $source.fields }}
{{- range $fkey, $fvalue := $source.fields }}
{{ $fkey }} = "{{ $fvalue }}"
{{- end -}}
{{ end }}
}
{{- end -}}

{{/*
get configuration variable name for sources confg map

Example usage:

{{ include "terraform.sources.config-map-variable" (dict "Context" .Values "Name" $name "Endpoint" $endpoint) }}

*/}}
{{- define "terraform.sources.config-map-variable" -}}
{{- $name := .Name -}}
{{- $ctx := .Context -}}
{{- $endpoint := .Endpoint -}}
{{- if not $endpoint -}}
{{- $source := (index $ctx.sumologic.sources "default") -}}
{{- if (index $ctx.sumologic.sources .Name "config-name") -}}
{{- $endpoint = index $ctx.sumologic.sources .Name "config-name" -}}
{{- else -}}
{{- $endpoint = printf "endpoint-%s" (include "terraform.sources.name_metrics" $name) -}}
{{- end -}}
{{- end -}}
{{ $endpoint }}
{{- end -}}
4 changes: 2 additions & 2 deletions deploy/helm/sumologic/templates/events-statefulset.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -66,11 +66,11 @@ spec:
initialDelaySeconds: 30
periodSeconds: 5
env:
- name: SUMO_ENDPOINT_EVENTS
- name: {{ template "terraform.sources.endpoint" (include "terraform.sources.name" "events") }}
valueFrom:
secretKeyRef:
name: sumologic
key: endpoint-events
key: {{ template "terraform.sources.config-map-variable" (dict "Context" .Values "Endpoint" "endpoint-events") }}
{{- if .Values.fluentd.events.extraEnvVars }}
{{ toYaml .Values.fluentd.events.extraEnvVars | nindent 10 }}
{{- end }}
Expand Down
Loading