Metricbeat Kubernetes module and metricsets docs proposal (#17636)

metricbeat/docs/modules/kubernetes.asciidoc

@@ -5,7 +5,7 @@ This file is generated! See scripts/mage/docs_collector.go
 [[metricbeat-module-kubernetes]]
 == Kubernetes module
 
-This module fetches metrics from Kubernetes several components:
+As one of the main pieces provided for Kubernetes monitoring, this module is capable of fetching metrics from several components:
 
 - https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/[kubelet]
 - https://github.com/kubernetes/kube-state-metrics[kube-state-metrics]
@@ -14,11 +14,51 @@ This module fetches metrics from Kubernetes several components:
 - https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/[scheduler]
 - https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/[proxy]
 
-All metricsets with the `state_` prefix require `hosts` field pointing to kube-state-metrics
-service within the cluster, while the rest should be pointed to kubelet service. Check the
-example configuration on how to do it.
+Some of the previous components are running on each of the Kubernetes nodes (like `kubelet` or `proxy`) while others provide a single cluster-wide endpoint. This is important to determine the optimal configuration and running strategy for the different metricsets included in the module.
 
-The default metricsets are `container`, `node`, `pod`, `system` and `volume`.
+For a complete reference on how to configure and run this module on Kubernetes as part of a `DaemonSet` and a `Deployment`, there's a complete example manifest available in <<running-on-kubernetes, Running Metricbeat on Kubernetes>> document.
+
+[float]
+=== Kubernetes endpoints and metricsets
+
+Kubernetes module is a bit complex as its internal metricsets require access to a wide variety of endpoints.
+
+This section highlights and introduces some groups of metricsets with similar endpoint access needs. For more details on the metricsets see `configuration example` and the `metricsets` sections below.
+
+[float]
+==== node / system / pod / container / module
+
+The default metricsets `container`, `node`, `pod`, `system` and `volume` require access to the `kubelet endpoint` in each of the Kubernetes nodes, hence it's recommended to include them as part of a `Metricbeat DaemonSet` or standalone Metricbeats running on the hosts.
+
+Depending on the version and configuration of Kubernetes nodes, `kubelet` might provide a read only http port (typically 10255), which is used in some configuration examples. But in general, and lately, this endpoint requires SSL (`https`) access (to port 10250 by default) and token based authentication.
+
+[float]
+==== state_* and event
+
+All metricsets with the `state_` prefix require `hosts` field pointing to `kube-state-metrics
+service` within the cluster. As the service provides cluster-wide metrics, there's no need to fetch them per node, hence the recommendation is to run these metricsets as part of a `Metricbeat Deployment` with one only replica.
+
+Note: Kube-state-metrics is not deployed by default in Kubernetes. For these cases the instructions for its deployment are available https://github.com/kubernetes/kube-state-metrics#kubernetes-deployment[here]. Generally `kube-state-metrics` runs a `Deployment` and is accessible via a service called `kube-state-metrics` on `kube-system` namespace, which will be the service to use in our configuration.
+
+[float]
+==== apiserver
+
+The apiserver metricset requires access to the Kubernetes API, which should be easily available in all Kubernetes environments. Depending on the Kubernetes configuration, the API access might require SSL (`https`) and token based authentication.
+
+[float]
+==== proxy
+
+The proxy metricset requires access to the proxy endpoint in each of Kubernetes nodes, hence it's recommended to configure it as a part of a `Metricbeat DaemonSet`.
+
+[float]
+==== scheduler and controllermanager
+
+These metricsets require access to the Kubernetes `controller-manager` and `scheduler` endpoints. By default, these pods run only on master nodes, and they are not exposed via a Service, but there are different strategies available for its configuration:
+
+- Create `Kubernetes Services` to make `kube-controller-manager` and `kube-scheduler` available and configure the metricsets to point to these services as part of a Metricbeat `Deployment`.
+- Use `Autodiscovery` functionality as part of a Metricbeat DaemonSet and include the metricsets in a conditional template applied for the specific pods.
+
+Note: In some "As a Service" Kubernetes implementations, like `GKE`, the master nodes or even the pods running on the masters won't be visible. In these cases it won't be possible to use `scheduler` and `controllermanager` metricsets.
 
 [float]
 === Kubernetes RBAC
@@ -88,7 +128,6 @@ roleRef:
   kind: ClusterRole
   name: metricbeat
   apiGroup: rbac.authorization.k8s.io
-[source,yaml]
 ----
 
 
@@ -172,7 +211,8 @@ metricbeat.modules:
     - state_persistentvolume
     - state_persistentvolumeclaim
     - state_storageclass
-  period: 10s
+    # Uncomment this to get k8s events:
+    #- event  period: 10s
   hosts: ["kube-state-metrics:8080"]
 
   # Enriching parameters:
@@ -183,19 +223,19 @@ metricbeat.modules:
   # and if not present it will fall back to InCluster
   #kube_config: ~/.kube/config
 
-# Kubernetes events
-- module: kubernetes
-  enabled: true
-  metricsets:
-    - event
-
 # Kubernetes API server
-# (when running metricbeat as a pod)
+# (when running metricbeat as a deployment)
 - module: kubernetes
   enabled: true
   metricsets:
     - apiserver
   hosts: ["https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"]
+  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
+  ssl.certificate_authorities:
+    - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+  period: 30s
+
+
 
 # Kubernetes proxy server
 # (when running metricbeat locally at hosts or as a daemonset + host network)
@@ -207,7 +247,7 @@ metricbeat.modules:
   period: 10s
 
 # Kubernetes controller manager
-# (URL and deployment will need to be modified to match the controller manager deployment)
+# (URL and deployment method should be adapted to match the controller manager deployment / service / endpoint)
 - module: kubernetes
   enabled: true
   metricsets:
@@ -216,7 +256,7 @@ metricbeat.modules:
   period: 10s
 
 # Kubernetes scheduler
-# (URL and deployment method will need to be adapted to match scheduler deployment)
+# (URL and deployment method should be adapted to match scheduler deployment / service / endpoint)
 - module: kubernetes
   enabled: true
   metricsets:

metricbeat/metricbeat.reference.yml

@@ -503,7 +503,8 @@ metricbeat.modules:
     - state_persistentvolume
     - state_persistentvolumeclaim
     - state_storageclass
-  period: 10s
+    # Uncomment this to get k8s events:
+    #- event  period: 10s
   hosts: ["kube-state-metrics:8080"]
 
   # Enriching parameters:
@@ -514,19 +515,19 @@ metricbeat.modules:
   # and if not present it will fall back to InCluster
   #kube_config: ~/.kube/config
 
-# Kubernetes events
-- module: kubernetes
-  enabled: true
-  metricsets:
-    - event
-
 # Kubernetes API server
-# (when running metricbeat as a pod)
+# (when running metricbeat as a deployment)
 - module: kubernetes
   enabled: true
   metricsets:
     - apiserver
   hosts: ["https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"]
+  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
+  ssl.certificate_authorities:
+    - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+  period: 30s
+
+
 
 # Kubernetes proxy server
 # (when running metricbeat locally at hosts or as a daemonset + host network)
@@ -538,7 +539,7 @@ metricbeat.modules:
   period: 10s
 
 # Kubernetes controller manager
-# (URL and deployment will need to be modified to match the controller manager deployment)
+# (URL and deployment method should be adapted to match the controller manager deployment / service / endpoint)
 - module: kubernetes
   enabled: true
   metricsets:
@@ -547,7 +548,7 @@ metricbeat.modules:
   period: 10s
 
 # Kubernetes scheduler
-# (URL and deployment method will need to be adapted to match scheduler deployment)
+# (URL and deployment method should be adapted to match scheduler deployment / service / endpoint)
 - module: kubernetes
   enabled: true
   metricsets:

metricbeat/module/kubernetes/_meta/config.reference.yml

@@ -40,7 +40,8 @@
     - state_persistentvolume
     - state_persistentvolumeclaim
     - state_storageclass
-  period: 10s
+    # Uncomment this to get k8s events:
+    #- event  period: 10s
   hosts: ["kube-state-metrics:8080"]
 
   # Enriching parameters:
@@ -51,19 +52,19 @@
   # and if not present it will fall back to InCluster
   #kube_config: ~/.kube/config
 
-# Kubernetes events
-- module: kubernetes
-  enabled: true
-  metricsets:
-    - event
-
 # Kubernetes API server
-# (when running metricbeat as a pod)
+# (when running metricbeat as a deployment)
 - module: kubernetes
   enabled: true
   metricsets:
     - apiserver
   hosts: ["https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"]
+  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
+  ssl.certificate_authorities:
+    - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+  period: 30s
+
+
 
 # Kubernetes proxy server
 # (when running metricbeat locally at hosts or as a daemonset + host network)
@@ -75,7 +76,7 @@
   period: 10s
 
 # Kubernetes controller manager
-# (URL and deployment will need to be modified to match the controller manager deployment)
+# (URL and deployment method should be adapted to match the controller manager deployment / service / endpoint)
 - module: kubernetes
   enabled: true
   metricsets:
@@ -84,7 +85,7 @@
   period: 10s
 
 # Kubernetes scheduler
-# (URL and deployment method will need to be adapted to match scheduler deployment)
+# (URL and deployment method should be adapted to match scheduler deployment / service / endpoint)
 - module: kubernetes
   enabled: true
   metricsets:

metricbeat/module/kubernetes/_meta/docs.asciidoc

@@ -1,4 +1,4 @@
-This module fetches metrics from Kubernetes several components:
+As one of the main pieces provided for Kubernetes monitoring, this module is capable of fetching metrics from several components:
 
 - https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/[kubelet]
 - https://github.com/kubernetes/kube-state-metrics[kube-state-metrics]
@@ -7,11 +7,51 @@ This module fetches metrics from Kubernetes several components:
 - https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/[scheduler]
 - https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/[proxy]
 
-All metricsets with the `state_` prefix require `hosts` field pointing to kube-state-metrics
-service within the cluster, while the rest should be pointed to kubelet service. Check the
-example configuration on how to do it.
+Some of the previous components are running on each of the Kubernetes nodes (like `kubelet` or `proxy`) while others provide a single cluster-wide endpoint. This is important to determine the optimal configuration and running strategy for the different metricsets included in the module.
 
-The default metricsets are `container`, `node`, `pod`, `system` and `volume`.
+For a complete reference on how to configure and run this module on Kubernetes as part of a `DaemonSet` and a `Deployment`, there's a complete example manifest available in <<running-on-kubernetes, Running Metricbeat on Kubernetes>> document.
+
+[float]
+=== Kubernetes endpoints and metricsets
+
+Kubernetes module is a bit complex as its internal metricsets require access to a wide variety of endpoints.
+
+This section highlights and introduces some groups of metricsets with similar endpoint access needs. For more details on the metricsets see `configuration example` and the `metricsets` sections below.
+
+[float]
+==== node / system / pod / container / module
+
+The default metricsets `container`, `node`, `pod`, `system` and `volume` require access to the `kubelet endpoint` in each of the Kubernetes nodes, hence it's recommended to include them as part of a `Metricbeat DaemonSet` or standalone Metricbeats running on the hosts.
+
+Depending on the version and configuration of Kubernetes nodes, `kubelet` might provide a read only http port (typically 10255), which is used in some configuration examples. But in general, and lately, this endpoint requires SSL (`https`) access (to port 10250 by default) and token based authentication.
+
+[float]
+==== state_* and event
+
+All metricsets with the `state_` prefix require `hosts` field pointing to `kube-state-metrics
+service` within the cluster. As the service provides cluster-wide metrics, there's no need to fetch them per node, hence the recommendation is to run these metricsets as part of a `Metricbeat Deployment` with one only replica.
+
+Note: Kube-state-metrics is not deployed by default in Kubernetes. For these cases the instructions for its deployment are available https://github.com/kubernetes/kube-state-metrics#kubernetes-deployment[here]. Generally `kube-state-metrics` runs a `Deployment` and is accessible via a service called `kube-state-metrics` on `kube-system` namespace, which will be the service to use in our configuration.
+
+[float]
+==== apiserver
+
+The apiserver metricset requires access to the Kubernetes API, which should be easily available in all Kubernetes environments. Depending on the Kubernetes configuration, the API access might require SSL (`https`) and token based authentication.
+
+[float]
+==== proxy
+
+The proxy metricset requires access to the proxy endpoint in each of Kubernetes nodes, hence it's recommended to configure it as a part of a `Metricbeat DaemonSet`.
+
+[float]
+==== scheduler and controllermanager
+
+These metricsets require access to the Kubernetes `controller-manager` and `scheduler` endpoints. By default, these pods run only on master nodes, and they are not exposed via a Service, but there are different strategies available for its configuration:
+
+- Create `Kubernetes Services` to make `kube-controller-manager` and `kube-scheduler` available and configure the metricsets to point to these services as part of a Metricbeat `Deployment`.
+- Use `Autodiscovery` functionality as part of a Metricbeat DaemonSet and include the metricsets in a conditional template applied for the specific pods.
+
+Note: In some "As a Service" Kubernetes implementations, like `GKE`, the master nodes or even the pods running on the masters won't be visible. In these cases it won't be possible to use `scheduler` and `controllermanager` metricsets.
 
 [float]
 === Kubernetes RBAC
@@ -81,7 +121,6 @@ roleRef:
   kind: ClusterRole
   name: metricbeat
   apiGroup: rbac.authorization.k8s.io
-[source,yaml]
 ----
 
 

metricbeat/module/kubernetes/apiserver/_meta/docs.asciidoc

@@ -1 +1,32 @@
-This is the `apiserver` metricset of the Kubernetes module.
+This is the `apiserver` metricset of the Kubernetes module, in charge of retrieving metrics from the Kubernetes API (available at `/metrics`).
+
+This metricset needs access to the `apiserver` component of Kubernetes, accessible typically by any POD via the `kubernetes.default` service or via environment variables (`KUBERNETES_SERVICE_HOST` and `KUBERNETES_SERVICE_PORT`).
+
+When the API uses https, the pod will need to authenticate using its default token and trust the server using the appropiate CA file.
+
+Configuration example using https and token based authentication:
+
+```yaml
+- module: kubernetes
+  enabled: true
+  metricsets:
+    - apiserver
+  hosts: ["https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"]
+  #hosts: ["https://kubernetes.default"]
+  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
+  ssl.certificate_authorities:
+    - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+  period: 30s
+```
+
+In order to access the `/metrics` path of the API service, some Kubernetes environments might require the following permission to be added to a ClusterRole.
+
+```yaml
+rules:
+- nonResourceURLs:
+  - /metrics
+  verbs:
+  - get
+```
+
+The previous configuration and RBAC requirement is available in the complete example manifest proposed in <<running-on-kubernetes, Running Metricbeat on Kubernetes>> document.

x-pack/metricbeat/metricbeat.reference.yml

@@ -780,7 +780,8 @@ metricbeat.modules:
     - state_persistentvolume
     - state_persistentvolumeclaim
     - state_storageclass
-  period: 10s
+    # Uncomment this to get k8s events:
+    #- event  period: 10s
   hosts: ["kube-state-metrics:8080"]
 
   # Enriching parameters:
@@ -791,19 +792,19 @@ metricbeat.modules:
   # and if not present it will fall back to InCluster
   #kube_config: ~/.kube/config
 
-# Kubernetes events
-- module: kubernetes
-  enabled: true
-  metricsets:
-    - event
-
 # Kubernetes API server
-# (when running metricbeat as a pod)
+# (when running metricbeat as a deployment)
 - module: kubernetes
   enabled: true
   metricsets:
     - apiserver
   hosts: ["https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"]
+  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
+  ssl.certificate_authorities:
+    - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+  period: 30s
+
+
 
 # Kubernetes proxy server
 # (when running metricbeat locally at hosts or as a daemonset + host network)
@@ -815,7 +816,7 @@ metricbeat.modules:
   period: 10s
 
 # Kubernetes controller manager
-# (URL and deployment will need to be modified to match the controller manager deployment)
+# (URL and deployment method should be adapted to match the controller manager deployment / service / endpoint)
 - module: kubernetes
   enabled: true
   metricsets:
@@ -824,7 +825,7 @@ metricbeat.modules:
   period: 10s
 
 # Kubernetes scheduler
-# (URL and deployment method will need to be adapted to match scheduler deployment)
+# (URL and deployment method should be adapted to match scheduler deployment / service / endpoint)
 - module: kubernetes
   enabled: true
   metricsets: