Add a new label for scraping metrics

[kasia-kujawa]

Description

Fixes #896

Testing performed

• ci/build.sh
• Redeploy fluentd and fluentd-events pods
• Confirm events, logs, and metrics are coming in

[sumo-drosiek]

I would add sumologic/scrape: true annotation instead of name based. In the prometheus we can set multiple matchLabels to scrape by, so it will be eg.:

          matchLabels:
            app.kubernetes.io/name: fluent-bit
            sumologic.com/scrape: true

[kasia-kujawa]

I would add sumologic/scrape: true annotation instead of name based. In the prometheus we can set multiple matchLabels to scrape by, so it will be eg.:

          matchLabels:
            app.kubernetes.io/name: fluent-bit
            sumologic.com/scrape: true

Annotations are not used to identify and select objects. , see this.

I tested with annotations and it does not work, target for metics is not correctly selected.
My changes:

% git diff
diff --git a/deploy/helm/sumologic/values.yaml b/deploy/helm/sumologic/values.yaml
index f67d22f9..040e147d 100644
--- a/deploy/helm/sumologic/values.yaml
+++ b/deploy/helm/sumologic/values.yaml
@@ -772,9 +772,8 @@ fluent-bit:
 
   ## Configuration for fluent-bit service
   service:
-    ## Add custom labels to fluent-bit service
-    labels:
-      sumologic.com/scrape: fluent-bit
+    annotations:
+      sumologic.com/scrape: "true"
 
   ## Add custom pod labels to fluent-bit daemonset pods
   podLabels: {}
@@ -1063,7 +1062,8 @@ kube-prometheus-stack:
             - $(NAMESPACE)
         selector:
           matchLabels:
-            sumologic.com/scrape: fluent-bit
+            app.kubernetes.io/name: fluent-bit
+            sumologic.com/scrape: "true"
       - name: collection-sumologic-otelcol
         additionalLabels:
           sumologic.com/app: otelcol

Created resources:

$ kubectl describe servicemonitors collection-fluent-bit
Name:         collection-fluent-bit
Namespace:    sumologic
Labels:       app=collection-fluent-bit
              app.kubernetes.io/managed-by=Helm
              chart=kube-prometheus-stack-12.0.2
              heritage=Helm
              release=collection
Annotations:  meta.helm.sh/release-name: collection
              meta.helm.sh/release-namespace: sumologic
API Version:  monitoring.coreos.com/v1
Kind:         ServiceMonitor
Metadata:
  Creation Timestamp:  2020-11-30T08:19:00Z
  Generation:          5
  Managed Fields:
    API Version:  monitoring.coreos.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:meta.helm.sh/release-name:
          f:meta.helm.sh/release-namespace:
        f:labels:
          .:
          f:app:
          f:app.kubernetes.io/managed-by:
          f:chart:
          f:heritage:
          f:release:
      f:spec:
        .:
        f:endpoints:
        f:namespaceSelector:
          .:
          f:matchNames:
        f:selector:
          .:
          f:matchLabels:
            .:
            f:app.kubernetes.io/name:
            f:sumologic.com/scrape:
    Manager:         Go-http-client
    Operation:       Update
    Time:            2020-11-30T10:06:19Z
  Resource Version:  18397
  Self Link:         /apis/monitoring.coreos.com/v1/namespaces/sumologic/servicemonitors/collection-fluent-bit
  UID:               04df55e7-7f71-4228-a1e5-53435c99079c
Spec:
  Endpoints:
    Path:  /api/v1/metrics/prometheus
    Port:  http
  Namespace Selector:
    Match Names:
      $(NAMESPACE)
  Selector:
    Match Labels:
      app.kubernetes.io/name:  fluent-bit
      sumologic.com/scrape:    true
Events:                        <none>



$ kubectl describe svc collection-fluent-bit
Name:              collection-fluent-bit
Namespace:         sumologic
Labels:            app.kubernetes.io/instance=collection
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=fluent-bit
                   app.kubernetes.io/version=1.6.3
                   helm.sh/chart=fluent-bit-0.7.4
Annotations:       meta.helm.sh/release-name: collection
                   meta.helm.sh/release-namespace: sumologic
                   sumologic.com/scrape: true
Selector:          app.kubernetes.io/instance=collection,app.kubernetes.io/name=fluent-bit
Type:              ClusterIP
IP:                10.152.183.87
Port:              http  2020/TCP
TargetPort:        http/TCP
Endpoints:         10.1.57.18:2020
Session Affinity:  None
Events:            <none>

[kasia-kujawa]

I'll add label "sumologic.com/scrape: true"

[sumo-drosiek]

Shouldn't be this way? 🤔

Suggested change

	sumologic.com/scrape: "true" # label used by fluent-bit service monitor
	sumologic.com/scrape: "true" # label used by fluent-bit service monitor