From ea625d8a12e233f0a6a3ac0069982803883af401 Mon Sep 17 00:00:00 2001 From: Sunny Date: Tue, 22 Aug 2023 21:20:42 +0000 Subject: [PATCH 1/4] Update custom-metrics docs - New path to KSM config. - Updated example config to include the suspend metrics label. Signed-off-by: Sunny --- content/en/flux/monitoring/custom-metrics.md | 52 +++++++++++--------- 1 file changed, 30 insertions(+), 22 deletions(-) diff --git a/content/en/flux/monitoring/custom-metrics.md b/content/en/flux/monitoring/custom-metrics.md index 40b0cebd1..8899162ec 100644 --- a/content/en/flux/monitoring/custom-metrics.md +++ b/content/en/flux/monitoring/custom-metrics.md @@ -16,29 +16,27 @@ as these are more useful for the people administering and maintaining Flux. Most of the time, the users of Flux who interact with Flux through the Flux custom resources want to know about the resources they work with. For example, the state of GitRepositories and their branches or tag references. These metrics can -be scraped by using [kube-state-metrics -(KSM)](https://github.com/kubernetes/kube-state-metrics), which is part of the -[kube-prometheus-stack](https://github.com/prometheus-operator/kube-prometheus). -KSM can be configured to add custom labels to the resource metrics, for example, -some value from the status of a resource or some arbitrary value like a team -name, department name, etc. +be scraped by using [kube-state-metrics (KSM)][kube-state-metrics], which is +part of the [kube-prometheus-stack][kube-prometheus-stack]. KSM can be +configured to add custom labels to the resource metrics, for example, some value +from the status of a resource or some arbitrary value like a team name, department name, etc. ## Set up kube-state-metrics Kube-state-metrics can be installed along with the whole monitoring stack using kube-prometheus-stack. The -[fluxcd/flux2-monitoring-example](https://github.com/fluxcd/flux2-monitoring-example) -repository contains example configurations for deploying and configuring -kube-prometheus-stack to monitor Flux. These configurations will be discussed in -detail in the following sections to show how they can be customized. Refer to -[Monitoring with Prometheus](monitoring.md) for detailed installation -instructions. - -The [Helm chart values for -kube-prometheus-stack](https://github.com/fluxcd/flux2-monitoring-example/blob/main/kube-prometheus-stack/release.yaml) -configure KSM to run in `custom-resource-state-only` mode. In this state, KSM -will not collect metrics for any of the Kubernetes core resources. The `rbac` -section provides KSM access to list and watch Flux custom resources. If +[fluxcd/flux2-monitoring-example][monitoring-example-repo] repository contains +example configurations for deploying and configuring kube-prometheus-stack to +monitor Flux. These configurations will be discussed in detail in the following +sections to show how they can be customized. + +The Kube-prometheus-stack Helm chart is used to install the monitoring stack. +The kube-state-metrics related configuration in the chart values exists in a +separate file called +[kube-state-metrics-config.yaml](https://github.com/fluxcd/flux2-monitoring-example/blob/main/monitoring/controllers/kube-prometheus-stack/kube-state-metrics-config.yaml). +It configures KSM to run in `custom-resource-state-only` mode. In this state, +KSM will not collect metrics for any of the Kubernetes core resources. The +`rbac` section provides KSM access to list and watch Flux custom resources. If image-reflector-controller and image-automation-controllers are not used, the API group (`image.toolkit.fluxcd.io`) and resources (`imagerepositories`, `imagepolicies`, `imageupdateautomations`) can be removed. The @@ -50,8 +48,8 @@ kube-apiserver and exporting them as configured. ## Adding custom metrics The example `customResourceState` values used in the above setup add a metric -called `gotk_resource_info` with labels `name`, `exported_namespace`, and -`ready`. +called `gotk_resource_info` with labels `name`, `exported_namespace`, +`suspended`, `ready`, etc. ```yaml - name: "resource_info" @@ -63,7 +61,9 @@ called `gotk_resource_info` with labels `name`, `exported_namespace`, and name: [metadata, name] labelsFromPath: exported_namespace: [metadata, namespace] + suspended: [spec, suspend] ready: [status, conditions, "[type=Ready]", status] + ... ``` This provides the current state of the Flux resources. It can be used to monitor @@ -95,6 +95,7 @@ customResourceState: name: [metadata, name] labelsFromPath: exported_namespace: [metadata, namespace] + suspended: [spec, suspend] ready: [status, conditions, "[type=Ready]", status] - name: "helmrelease_version_info" help: "The version information of helm release resource." @@ -176,6 +177,7 @@ customResourceState: name: [metadata, name] labelsFromPath: exported_namespace: [metadata, namespace] + suspended: [spec, suspend] ready: [status, conditions, "[type=Ready]", status] branch: [spec, ref, branch] ... @@ -191,5 +193,11 @@ It contains the `ownedBy="teamA"`, `department="baz"` and `branch="main"` labels. Similarly, more custom labels can be added depending on the need. Refer to the [kube-state-metrics custom-resource state configuration -docs](https://github.com/kubernetes/kube-state-metrics/blob/main/docs/customresourcestate-metrics.md) -to learn more about customizing the metrics. +docs][ksm-customresourcestate-metrics] to learn more about customizing the +metrics. + + +[kube-state-metrics]: https://github.com/kubernetes/kube-state-metrics +[monitoring-example-repo]: https://github.com/fluxcd/flux2-monitoring-example +[kube-prometheus-stack]: https://github.com/prometheus-operator/kube-prometheus +[ksm-customresourcestate-metrics]: https://github.com/kubernetes/kube-state-metrics/blob/main/docs/customresourcestate-metrics.md From d147cdc41f12d134ac2237395bbec07eef99b705 Mon Sep 17 00:00:00 2001 From: Sunny Date: Tue, 22 Aug 2023 21:14:47 +0000 Subject: [PATCH 2/4] Add metrics docs Signed-off-by: Sunny --- content/en/flux/monitoring/metrics.md | 191 ++++++++++++++++++++++---- 1 file changed, 162 insertions(+), 29 deletions(-) diff --git a/content/en/flux/monitoring/metrics.md b/content/en/flux/monitoring/metrics.md index 27403f018..31fa2cdf6 100644 --- a/content/en/flux/monitoring/metrics.md +++ b/content/en/flux/monitoring/metrics.md @@ -5,65 +5,198 @@ description: "How to monitor Flux with Prometheus Operator and Grafana" weight: 1 --- -## Reconcile metrics +Flux has native support for [Prometheus][prometheus] metrics to provide insights +into the state of the Flux components. These can be used to set up monitoring +for the Flux controllers. In addition, Flux Custom Resource metrics can also +be collected leveraging tools like [kube-state-metrics][kube-state-metrics]. +This document provides information about Flux metrics that can be used to set up +monitoring, with some examples. -Ready status metrics: +The [fluxcd/flux2-monitoring-example][monitoring-example-repo] repository +provides a ready-made example setup to get started with monitoring Flux. It is +recommended to set up the monitoring example before continuing with this +document to follow along. Before getting into the monitoring setup, the +following sections will describe the kinds of metrics that can be collected for +Flux. -```sh -gotk_reconcile_condition{kind, name, namespace, type="Ready", status="True"} -gotk_reconcile_condition{kind, name, namespace, type="Ready", status="False"} -gotk_reconcile_condition{kind, name, namespace, type="Ready", status="Unknown"} -``` - -Suspend status metrics: +## Controller metrics -```sh -gotk_suspend_status{kind, name, namespace} -``` +The default installation of Flux controllers export Prometheus metrics at +port `8080` in the standard `/metrics` path. These metrics are about the inner +workings of the controllers. -Time spent reconciling: +Flux resource reconciliation duration metrics: -```sh +``` gotk_reconcile_duration_seconds_bucket{kind, name, namespace, le} gotk_reconcile_duration_seconds_sum{kind, name, namespace} gotk_reconcile_duration_seconds_count{kind, name, namespace} ``` -## Control plane metrics +Cache event metrics: + +``` +gotk_cache_events_total{event_type, name, namespace} +``` Controller CPU and memory usage: -```sh +``` process_cpu_seconds_total{namespace, pod} container_memory_working_set_bytes{namespace, pod} ``` Kubernetes API usage: -```shell +``` rest_client_requests_total{namespace, pod} ``` Controller runtime: -```shell +``` workqueue_longest_running_processor_seconds{name} controller_runtime_reconcile_total{controller, result} ``` -## Setup monitoring with kube-prom-stack +In addition, many other Go runtime and [controller-runtime +metrics][controller-runtime-metrics] are also exported. -Flux uses [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) -to provide a monitoring stack made out of: +## Resource metrics -* **Prometheus Operator** - manages Prometheus clusters atop Kubernetes -* **Prometheus** - collects metrics from the Flux controllers and Kubernetes API -* **Grafana** dashboards - displays the Flux control plane resource usage and reconciliation stats -* **kube-state-metrics** - generates metrics about the state of the Kubernetes objects +Metrics for the Flux custom resources can be used to monitor the deployment of +workloads. Since the use case for these metrics may vary depending on the +needs, it's hard to decide which fields of the resources would be useful to the +users. Hence, these metrics are not exported by the Flux controllers themselves +but can be collected and exported by using other tools that can read the custom +resource state from the kube-apiserver. One such tool is [kube-state-metrics +(KSM)][kube-state-metrics]. KSM is also deployed as part of +[kube-prometheus-stack][kube-prometheus-stack] and is used to export the metrics +of kubernetes core resources. It can be configured to also collect custom +resource metrics. The monitoring setup in +[flux2-monitoring-example][monitoring-example-repo] uses KSM to collect and +export Flux custom resource metrics. -### Alert manager examples +In the [example monitoring setup][monitoring-example-repo], the metric +`gotk_resource_info` provides information about the current state of Flux +resources. -## Flux Grafana dashboards - -### Grafana annotations +``` +gotk_resource_info{customresource_group, customresource_kind, customresource_version, exported_namespace, name, ready, suspended, ...} +``` +- `customresource_group` is the API group of the resource, for example + `source.toolkit.fluxcd.io` for the Flux source API. +- `customresource_kind` is the kind of the resource, for example a + `GitRepository` source. +- `customresource_version` is the API version of the resource, for example `v1`. +- `exported_namespace` is the namespace of the resource. +- `name` is the name of the resource. +- `ready` shows the readiness of the resource. +- `suspended` shows if the resource's reconciliation is suspended. + +These are some of the common labels that are present in metrics for all the +kinds of resources. In addition, there are a few resource kind specific labels. +See the following table for a list of labels associated with specific resource +kind. + +| Resource Kind | Labels | +| --- | --- | +| Kustomization | `revision`, `source_name` | +| HelmRelease | `revision`, `chart_name`, `chart_source_name` | +| GitRepository | `revision`, `url` | +| Bucket | `revision`, `endpoint`, `bucket_name` | +| HelmRepository | `revision`, `url` | +| HelmChart | `revision`, `chart_name`, `chart_version` | +| OCIRepository | `revision`, `url` | +| Receiver | `webhook_path` | +| ImageRepository | `image` | +| ImagePolicy | `source_name` | +| ImageUpdateAutomation | `source_name` | + +{{< note >}} +The above metric may have extra labels after being collected in Prometheus. This +may be due to the default Prometheus scrape configuration used by +kube-prometheus-stack. Since they are about the kube-state-metrics service and +not about Flux itself, they can be ignored. +{{< /note >}} + +`gotk_resource_info` is an example of a metric used to collect information about +the Flux resources. This metric can be customized to add more labels, or more +such metrics can also be created by changing the kube-state-metrics custom +resource state configuration. Please see [Flux custom Prometheus +metrics][custom-metrics] for details about them. + +### :warning: Deprecated resource metrics + +Prior to Flux v2.1.0, the individual Flux controllers used to export resource +metrics that they managed. They have been deprecated for custom metrics using +kube-state-metrics. + +Users of the deprecated metrics `gotk_reconcile_condition` and +`gotk_suspend_status` can find the same information in the new +`gotk_resource_info` metric exported using kube-state-metrics. If needed, an +equivalent of `gotk_reconcile_condition` and `gotk_suspend_status` can be +created as a custom metric using the kube-state-metrics custom resource state +configuration. Please see [Flux custom Prometheus +metrics][custom-metrics] for details. + +## Monitoring setup + +In the [monitoring example repository][monitoring-example-repo], the monitoring configurations can be found in the +[`monitoring/`](https://github.com/fluxcd/flux2-monitoring-example/tree/main/monitoring) +directory. `monitoring/controllers/` directory contains the configurations for +deploying kube-prometheus-stack and loki-stack. We'll discuss +kube-prometheus-stack below. For Flux log collection using Loki, refer to the +[Flux logs](/flux/monitoring/logs/) docs. + +The configuration in the `monitoring/controllers/kube-prometheus-stack/` +directory creates a HelmRepository of type OCI for the [prometheus-community +helm charts](https://github.com/prometheus-community/helm-charts) and a +HelmRelease to deploy the `kube-prometheus-stack` chart in the `monitoring` +namespace. This installs all the monitoring components in the `monitoring` +namespace. Please see the +[values](https://github.com/fluxcd/flux2-monitoring-example/blob/main/monitoring/controllers/kube-prometheus-stack/release.yaml) +used for the chart deployment and modify them accordingly. + +The chart values used for configuring kube-state-metrics are in the file +[`kube-state-metrics-config.yaml`](https://github.com/fluxcd/flux2-monitoring-example/blob/main/monitoring/controllers/kube-prometheus-stack/kube-state-metrics-config.yaml), +as seen in the +[kustomization.yaml](https://github.com/fluxcd/flux2-monitoring-example/blob/main/monitoring/controllers/kube-prometheus-stack/kustomization.yaml), +which uses a kustomize ConfigMap generator to put the configurations in a +ConfigMap and use the chart values from the ConfigMap. +These values are merged with the inline chart values in the HelmRelease. +Kube-state-metrics values are in a separate file to make it easier to customize +the metrics it collects; refer to the [Flux custom Prometheus +metrics][custom-metrics] docs to see how they are used. Once +deployed with these values, the kube-state-metrics starts collecting and +exporting the Flux resource metrics. + +To configure Prometheus to scrape Flux controller metrics, a +[PodMonitor](https://github.com/fluxcd/flux2-monitoring-example/blob/main/monitoring/configs/podmonitor.yaml) +is used that selects all the Flux controller Pods and sets the metrics endpoint +to the `http-prom` port. Once created, the prometheus-operator will +automatically configure Prometheus to scrape the Flux controller metrics. + +### Flux Grafana dashboards + +The [example monitoring setup][monitoring-example-repo] provides two example +Grafana dashboards in +[`monitoring/configs/dashboards`](https://github.com/fluxcd/flux2-monitoring-example/tree/main/monitoring/configs/dashboards) +that use the Flux controller and resource metrics. The Flux Cluster Stats +dashboard shows the overall state of the Flux Sources and Cluster Reconcilers. +The Flux Control Plane dashboard shows the statistics of the various components +that constitute the Flux Control Plane and their operational metrics. + +[Insert screenshots of the grafana dashboards] + +More custom metrics can be created and used in the dashboards for monitoring +Flux. + + +[kube-state-metrics]: https://github.com/kubernetes/kube-state-metrics +[prometheus]: https://prometheus.io/ +[monitoring-example-repo]: https://github.com/fluxcd/flux2-monitoring-example +[kube-prometheus-stack]: https://github.com/prometheus-operator/kube-prometheus +[controller-runtime-metrics]: https://book.kubebuilder.io/reference/metrics-reference +[custom-metrics]: /flux/monitoring/custom-metrics/ From cda6b6a3320b3eb950ae834cfc738248a365e4e1 Mon Sep 17 00:00:00 2001 From: Sunny Date: Tue, 22 Aug 2023 21:15:13 +0000 Subject: [PATCH 3/4] Add logs docs Signed-off-by: Sunny --- content/en/flux/monitoring/logs.md | 108 +++++++++++++++++++++++++---- 1 file changed, 94 insertions(+), 14 deletions(-) diff --git a/content/en/flux/monitoring/logs.md b/content/en/flux/monitoring/logs.md index b0e374556..26d491561 100644 --- a/content/en/flux/monitoring/logs.md +++ b/content/en/flux/monitoring/logs.md @@ -5,11 +5,43 @@ description: "How to monitor the Flux logs with Loki and Grafana" weight: 3 --- -The Flux controllers follow the Kubernetes structured logging conventions. +The Flux controllers follow the Kubernetes structured logging conventions. These +logs can be collected and analyzed to monitor the operations of the controllers. + +The [fluxcd/flux2-monitoring-example][monitoring-example-repo] repository +provides a ready-made example setup to get started with monitoring Flux, which +includes [Loki-stack][loki-stack] to collect logs from all the Flux controllers +and explore them using Grafana. It is recommended to set up the monitoring +example before continuing with this document to follow along. Before getting +into Loki and Grafana setup, the following sections will describe the Flux logs +and how to interpret them. + +## Controller logs + +The default installation of Flux controllers write logs to `stderr` in JSON +format at the `info` log level. This can be configured using the +`--log-encoding` and `--log-level` flags in the controllers. Refer to the +[flux-system +kustomization](https://github.com/fluxcd/flux2-monitoring-example/blob/main/clusters/test/flux-system/kustomization.yaml) +for an example of how to patch the Flux controllers with flags. The following +example patch snippet can be appended to the existing set of patches to add a +log level flag and change the log level of the controller to `debug`. + +```yaml +- op: add + path: /spec/template/spec/containers/0/args/- + value: --log-level="debug" +``` + +{{< note >}} +The patch configuration in the example only applies to a few targeted +controllers. Update the patch target to apply this to other controllers too. +{{< /note >}} -## Structured logging +### Structured logging -The Flux controllers logs are written to `stderr` in JSON format, with the following common tags: +The Flux controllers support structured logging with the following common +labels: - `level` can be `debug`, `info` or `error` - `ts` timestamp in the ISO 8601 format @@ -21,9 +53,7 @@ The Flux controllers logs are written to `stderr` in JSON format, with the follo - `namespace` The Flux CR namespace - `reconcileID` the UID of the Flux reconcile operation -### Samples - -Sample of a `info` log produced by kustomize-controller: +Sample of an `info` log produced by kustomize-controller: ```json { @@ -61,19 +91,69 @@ Sample of an `error` log produced by kustomize-controller: } ``` -## Log inspection with kubectl - -```shell -kubectl -n flux-system logs deploy/kustomize-controller +The log labels shown above can be used to query for specific types of logs. For +example, error logs can be queried using the `error` label, the output of +successful reconciliation of Kustomization can be queried using the `output` +label, the logs about a specific controller can be queried using the +`controllerKind` label. + +## Querying logs associated with resources + +For querying logs associated with particular resources, the `flux logs` CLI +command can be used. It connects to the cluster, fetches the relevant Flux logs, +and filters them based on the query request. For example, to list the logs +associated with Kustomization `monitoring-configs`: + +```console +$ flux logs --kind=Kustomization --name=monitoring-configs --namespace=flux-system --since=1m +... +2023-08-22T18:35:45.292Z info Kustomization/monitoring-configs.flux-system - All dependencies are ready, proceeding with reconciliation +2023-08-22T18:35:45.348Z info Kustomization/monitoring-configs.flux-system - server-side apply completed +2023-08-22T18:35:45.380Z info Kustomization/monitoring-configs.flux-system - Reconciliation finished in 88.208385ms, next run in 1h0m0s ``` +Refer to the [`flux logs`](/flux/cmd/flux_logs/) CLI docs to learn more about +it. + ## Log aggregation with Grafana Loki -To install Grafana Loki and Promtail in the `monitoring` namespace, apply the -[manifests/monitoring/loki-stack](https://github.com/fluxcd/flux2/tree/main/manifests/monitoring/loki-stack). +In the [monitoring example repository][monitoring-example-repo], the monitoring +configurations can be found in the +[`monitoring/`](https://github.com/fluxcd/flux2-monitoring-example/tree/main/monitoring) +directory. `monitoring/controllers/` directory contains the configurations for +deploying kube-prometheus-stack and loki-stack. We'll discuss loki-stack below. +For Flux metrics collection using Prometheus, refer to the [Flux Prometheus +metrics](/flux/monitoring/metrics/) docs. + +The configuration in the +[`monitoring/controllers/loki-stack`](https://github.com/fluxcd/flux2-monitoring-example/tree/main/monitoring/controllers/loki-stack) +directory creates a HelmRepository for the [Grafana +helm-charts](https://github.com/grafana/helm-charts) and a HelmRelease to +deploy the `loki-stack` chart in the `monitoring` namespace. Please see the +[values](https://github.com/fluxcd/flux2-monitoring-example/blob/main/monitoring/controllers/loki-stack/release.yaml) +used for the chart and modify them accordingly. + +Once deployed, [Loki][loki] and [Promtail][promtail] Pods get created, and Loki +is added as a data source in Grafana. Promtail aggregates the logs from all the +Pods in every node and sends them to Loki. Grafana can be used to query the logs +from Loki and analyze them. Refer to the [LogQL docs][logql] to see examples of +queries and learn more about querying logs. ### Grafana dashboard -Control plane logs [http://localhost:3000/d/flux-logs](http://localhost:3000/d/flux-logs/flux-logs): +The [example monitoring setup][monitoring-example-repo] provides a Grafana +dashboard in +[`monitoring/configs/dashboards/logs.json`](https://github.com/fluxcd/flux2-monitoring-example/tree/main/monitoring/configs/dashboards/logs.json) +that queries and shows logs from all the Flux controllers. + +[Insert screenshot of the grafana logs dashboard] + +This can be used to browse logs from all the Flux controllers in a centralized +manner. + -![Control plane logs dashboard](/img/logs-dashboard.png) +[monitoring-example-repo]: https://github.com/fluxcd/flux2-monitoring-example +[loki-stack]: https://github.com/grafana/helm-charts/tree/main/charts/loki-stack +[loki]: https://grafana.com/docs/loki/latest/ +[promtail]: https://grafana.com/docs/loki/latest/clients/promtail/ +[logql]: https://grafana.com/docs/loki/latest/logql/ From f4df7854448c2f2f088d3b0ec186de7b0f07af0f Mon Sep 17 00:00:00 2001 From: Sunny Date: Tue, 22 Aug 2023 21:15:25 +0000 Subject: [PATCH 4/4] Add events docs Signed-off-by: Sunny --- content/en/flux/monitoring/events.md | 133 +++++++++++++++++++++++++-- 1 file changed, 126 insertions(+), 7 deletions(-) diff --git a/content/en/flux/monitoring/events.md b/content/en/flux/monitoring/events.md index 51f441a6c..82c56e7fa 100644 --- a/content/en/flux/monitoring/events.md +++ b/content/en/flux/monitoring/events.md @@ -5,11 +5,19 @@ description: "How to monitor the Flux events" weight: 4 --- -The Flux controllers emit Kubernetes events for every reconciliation operation. +The Flux controllers emit [Kubernetes events][kubernetes-events] during the +reconciliation operation to provide information about the object being +reconciled. Unlike logs, events are always associated with an object, which is a +Flux resource in this case. Events are supplemental data that can be used along +with logs to provide a complete picture of controllers' operations. Some of +the events emitted by Flux controllers are also used to send notifications. +See the [Alerts docs](/flux/monitoring/alerts/) to learn more about the Flux +Alerts based on events from controllers. In the following sections, we will go +through the Flux events and how to interpret them. ## Kubernetes events -The Flux controllers events contain the following fields: +The Flux controller events about a resource contain the following fields: - `type` can be `Normal` or `Warning` - `firstTimestamp` timestamp in the ISO 8601 format @@ -18,11 +26,11 @@ The Flux controllers events contain the following fields: - `reason` short machine understandable string - `involvedObject` the API version, kind, name and namespace of the Flux object - `metadata.annotations` the Flux specific metadata e.g. source revision -- `source.component` the Flux controller name +- `source.component` the Flux controller name where the event originated from. -### Samples +### Examples -Sample of a `Normal` event produced by kustomize-controller: +Example of a `Normal` event produced by kustomize-controller: ```json { @@ -52,8 +60,119 @@ Sample of a `Normal` event produced by kustomize-controller: } ``` +In the above example: +- The event is about a `Kustomization` named `flux-system` in the `flux-system` + namespace, indicated by the `involvedObject` field. +- The event originates from `kustomize-controller`, indicated by the + `source.component` field. +- The event is a `Normal` type event about a successful reconciliation, + indicated by the `reason` and `message` fields. +- The `metadata.annotations` field `kustomize.toolkit.fluxcd.io/revision` + contains information about the source revision that was successfully applied + as a result of successful reconciliation of the Kustomization. + +Example of a `Warning` event produced by source-controller: + +```json +{ + "apiVersion": "v1", + "count": 4, + "eventTime": null, + "firstTimestamp": "2023-08-22T20:24:06Z", + "involvedObject": { + "apiVersion": "source.toolkit.fluxcd.io/v1", + "kind": "GitRepository", + "name": "podinfo", + "namespace": "default", + "resourceVersion": "1284973", + "uid": "2c2ed1da-556f-4793-863d-7d96e8bab3f5" + }, + "kind": "Event", + "lastTimestamp": "2023-08-22T20:24:18Z", + "message": "failed to checkout and determine revision: unable to clone 'https://github.com/stefanprodan/podinfo': couldn't find remote ref \"refs/tags/v1.8.9\"", + "metadata": { + "creationTimestamp": "2023-08-22T20:24:06Z", + "name": "podinfo.177dce48bc7db3a4", + "namespace": "default", + "resourceVersion": "1285016", + "uid": "3c8f568a-c99b-4279-8093-6ef08fae325b" + }, + "reason": "GitOperationFailed", + "reportingComponent": "", + "reportingInstance": "", + "source": { + "component": "source-controller" + }, + "type": "Warning" +} +``` + +In the above example: +- The event is about a `GitRepository` named `podinfo` in the `default` + namespace, indicated by the `involvedObject` field. +- The event originates from `source-controller`, indicated by the + `source.component` field. +- The event is a `Warning` type event about a failed Git operation, indicated by + the `reason` and `message` fields. + ## Events inspection with kubectl -```shell -kubectl events -n monitoring --for helmreleaase/kube-prom-stack +The events associated with a Flux resource can be queried using `kubectl events` +command: + +```console +$ kubectl events -n flux-system --for kustomization/flux-system +LAST SEEN TYPE REASON OBJECT MESSAGE +58m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 448.00332ms, next run in 10m0s +48m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 486.826649ms, next run in 10m0s +38m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 502.282127ms, next run in 10m0s +28m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 543.745587ms, next run in 10m0s +18m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 465.177441ms, next run in 10m0s +8m27s Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 494.543068ms, next run in 10m0s +``` + +This shows all the events associated with the queried resource in an hour. + +## Events inspection with flux CLI + +The events associated with a Flux resource can be queried using the `flux +events` CLI command: + +```console +$ flux events --for Kustomization/flux-system +LAST SEEN TYPE REASON OBJECT MESSAGE +52m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 506.467ms, next run in 10m0s +42m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 531.072726ms, next run in 10m0 +32m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 506.673992ms, next run in 10m0 +22m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 512.255817ms, next run in 10m0 +12m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 507.521248ms, next run in 10m0 +2m31s Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 448.00332ms, next run in 10m0s +``` + +This can also be used to watch all the events issues by the Flux controllers +across all the namespaces: + +```console +$ flux events --all-namespaces --watch +NAMESPACE LAST SEEN TYPE REASON OBJECT MESSAGE +flux-system 34m (x3 over 154m) Normal GitOperationSucceeded GitRepository/flux-system no changes since last reconcilation: observed revision 'main@sha1:4d768edba5d409feb60870dd3b0ac0d307299898' +flux-system 54m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 486.814878ms, next run in 10m0s +flux-system 44m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 486.203813ms, next run in 10m0s +flux-system 34m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 512.160373ms, next run in 10m0s +flux-system 24m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 543.806383ms, next run in 10m0s +flux-system 14m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 524.293527ms, next run in 10m0s +flux-system 4m5s Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 522.671955ms, next run in 10m0s +flux-system 47s Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 523.892245ms, next run in 10m0s +flux-system 34m Normal ReconciliationSucceeded Kustomization/monitoring-configs Reconciliation finished in 104.609707ms, next run in 1h0m0s +flux-system 42s Normal ReconciliationSucceeded Kustomization/monitoring-configs Reconciliation finished in 90.70521ms, next run in 1h0m0s +flux-system 34m Normal ReconciliationSucceeded Kustomization/monitoring-controllers Reconciliation finished in 118.651968ms, next run in 1h0m0s +flux-system 39s Normal ReconciliationSucceeded Kustomization/monitoring-controllers Reconciliation finished in 132.34839ms, next run in 1h0m0s +monitoring 34m (x3 over 154m) Normal ArtifactUpToDate HelmChart/monitoring-kube-prometheus-stack artifact up-to-date with remote revision: '48.3.3' +monitoring 34m (x3 over 154m) Normal ArtifactUpToDate HelmChart/monitoring-loki-stack artifact up-to-date with remote revision: '2.9.11' ``` + +Refer to the [`flux events`](/flux/cmd/flux_events/) CLI docs to learn more +about it. + + +[kubernetes-events]: https://kubernetes.io/docs/reference/kubernetes-api/cluster-resources/event-v1/