Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[kubernetes] Add last_terminated_reason_timestamp metric #39200

Merged
merged 5 commits into from
Apr 26, 2024
Merged

[kubernetes] Add last_terminated_reason_timestamp metric #39200

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
743246a
add last_terminated_reason_timestamp metric
tetianakravchenko Apr 24, 2024
a1672a6
Update CHANGELOG.next.asciidoc
tetianakravchenko Apr 25, 2024
180afdf
revert the metric name to status.last_terminated_reason_timestamp
tetianakravchenko Apr 25, 2024
90d82b6
rename field to last_terminated_timestamp
tetianakravchenko Apr 26, 2024
4617806
Update CHANGELOG.next.asciidoc
tetianakravchenko Apr 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -275,6 +275,7 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
- Add new memory/cgroup metrics to Kibana module {pull}37232[37232]
- Add SSL support to mysql module {pull}37997[37997]
- Add SSL support for aerospike module {pull}38126[38126]
- Add last_terminated_reason_timestamp metric in kubernetes module {issue}3802[3802]
tetianakravchenko marked this conversation as resolved.
Show resolved Hide resolved


*Metricbeat*
Expand Down
10 changes: 10 additions & 0 deletions metricbeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45985,6 +45985,16 @@ type: keyword

--

*`kubernetes.container.status.last_terminated_reason_timestamp`*::
+
--
Last terminated time (epoch) of a container


type: double

--


*`kubernetes.container.cpu.limit.cores`*::
+
Expand Down
3 changes: 3 additions & 0 deletions metricbeat/module/kubernetes/_meta/test/KSM/ksm.v2.12.0.plain
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -552,10 +552,13 @@ kube_pod_container_state_started{namespace="kube-system",pod="kube-proxy-45qj9",
kube_pod_container_state_started{namespace="kube-system",pod="kube-state-metrics-5bcd4898-bntgt",uid="38cad684-d5e5-4187-8a98-2999c0f5b252",container="kube-state-metrics"} 1.713873336e+09
# HELP kube_pod_container_status_last_terminated_reason Describes the last reason the container was in terminated state.
# TYPE kube_pod_container_status_last_terminated_reason gauge
kube_pod_container_status_last_terminated_reason{namespace="kube-system",pod="kube-scheduler-kind-control-plane",uid="82154115-17ff-4943-8088-879d4d045b52",container="kube-scheduler",reason="Error"} 1
# HELP kube_pod_container_status_last_terminated_exitcode Describes the exit code for the last container in terminated state.
# TYPE kube_pod_container_status_last_terminated_exitcode gauge
kube_pod_container_status_last_terminated_exitcode{namespace="kube-system",pod="kube-scheduler-kind-control-plane",uid="82154115-17ff-4943-8088-879d4d045b52",container="kube-scheduler"} 1
# HELP kube_pod_container_status_last_terminated_timestamp Last terminated time for a pod container in unix timestamp.
# TYPE kube_pod_container_status_last_terminated_timestamp gauge
kube_pod_container_status_last_terminated_timestamp{namespace="kube-system",pod="kube-scheduler-kind-control-plane",uid="82154115-17ff-4943-8088-879d4d045b52",container="kube-scheduler"} 1.71393586e+09
# HELP kube_pod_container_status_ready [STABLE] Describes whether the containers readiness check succeeded.
# TYPE kube_pod_container_status_ready gauge
kube_pod_container_status_ready{namespace="default",pod="hello-28564556-gkqsk",uid="888e4e15-5ffc-40cd-826d-af822fee80dd",container="hello"} 0
Expand Down
2 changes: 1 addition & 1 deletion metricbeat/module/kubernetes/fields.go
View file
Open in desktop

Large diffs are not rendered by default.

4 changes: 4 additions & 0 deletions metricbeat/module/kubernetes/state_container/_meta/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,10 @@
type: keyword
description: >
The last reason the container was in terminated state (Completed, ContainerCannotRun, Error or OOMKilled).
- name: last_terminated_reason_timestamp
type: double
description: >
Last terminated time (epoch) of a container
- name: cpu
type: group
fields:
Expand Down
52 changes: 27 additions & 25 deletions metricbeat/module/kubernetes/state_container/_meta/test/ksm.v2.12.0.plain.expected
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -482,9 +482,9 @@
{
"RootFields": {
"container": {
"id": "cd52120cda244a98ed01e9259381cbcd439bc087115d93c7573bada504dbb620",
"id": "1be2795038ca3a3a291a6aca5d7ecdfaf0b326453c74e471bab3064822f0bdfb",
"image": {
"name": "registry.k8s.io/kube-scheduler-amd64:v1.29.1"
"name": "registry.k8s.io/coredns/coredns:v1.11.1"
},
"runtime": "containerd"
}
Expand All @@ -495,7 +495,7 @@
"name": "kind-control-plane"
},
"pod": {
"name": "kube-scheduler-kind-control-plane"
"name": "coredns-76f75df574-v8skx"
}
},
"MetricSetFields": {
Expand All @@ -504,8 +504,16 @@
"cores": 0.1
}
},
"id": "cd52120cda244a98ed01e9259381cbcd439bc087115d93c7573bada504dbb620",
"name": "kube-scheduler",
"id": "1be2795038ca3a3a291a6aca5d7ecdfaf0b326453c74e471bab3064822f0bdfb",
"memory": {
"limit": {
"bytes": 178257920
},
"request": {
"bytes": 73400320
}
},
"name": "coredns",
"status": {
"phase": "running",
"ready": true,
Expand All @@ -526,9 +534,9 @@
{
"RootFields": {
"container": {
"id": "1be2795038ca3a3a291a6aca5d7ecdfaf0b326453c74e471bab3064822f0bdfb",
"id": "9a40af12036875eb414555deeca2a09e73f37ccb7c8bbf787bfe9616cbbcbd98",
"image": {
"name": "registry.k8s.io/coredns/coredns:v1.11.1"
"name": "registry.k8s.io/kube-apiserver-amd64:v1.29.1"
},
"runtime": "containerd"
}
Expand All @@ -539,25 +547,17 @@
"name": "kind-control-plane"
},
"pod": {
"name": "coredns-76f75df574-v8skx"
"name": "kube-apiserver-kind-control-plane"
}
},
"MetricSetFields": {
"cpu": {
"request": {
"cores": 0.1
}
},
"id": "1be2795038ca3a3a291a6aca5d7ecdfaf0b326453c74e471bab3064822f0bdfb",
"memory": {
"limit": {
"bytes": 178257920
},
"request": {
"bytes": 73400320
"cores": 0.25
}
},
"name": "coredns",
"id": "9a40af12036875eb414555deeca2a09e73f37ccb7c8bbf787bfe9616cbbcbd98",
"name": "kube-apiserver",
"status": {
"phase": "running",
"ready": true,
Expand All @@ -578,9 +578,9 @@
{
"RootFields": {
"container": {
"id": "9a40af12036875eb414555deeca2a09e73f37ccb7c8bbf787bfe9616cbbcbd98",
"id": "cd52120cda244a98ed01e9259381cbcd439bc087115d93c7573bada504dbb620",
"image": {
"name": "registry.k8s.io/kube-apiserver-amd64:v1.29.1"
"name": "registry.k8s.io/kube-scheduler-amd64:v1.29.1"
},
"runtime": "containerd"
}
Expand All @@ -591,18 +591,20 @@
"name": "kind-control-plane"
},
"pod": {
"name": "kube-apiserver-kind-control-plane"
"name": "kube-scheduler-kind-control-plane"
}
},
"MetricSetFields": {
"cpu": {
"request": {
"cores": 0.25
"cores": 0.1
}
},
"id": "9a40af12036875eb414555deeca2a09e73f37ccb7c8bbf787bfe9616cbbcbd98",
"name": "kube-apiserver",
"id": "cd52120cda244a98ed01e9259381cbcd439bc087115d93c7573bada504dbb620",
"name": "kube-scheduler",
"status": {
"last_terminated_reason": "Error",
"last_terminated_reason_timestamp": 1713935860,
"phase": "running",
"ready": true,
"restarts": 0
Expand Down
58 changes: 30 additions & 28 deletions metricbeat/module/kubernetes/state_container/_meta/testdata/ksm.v2.12.0.plain-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -583,9 +583,9 @@
},
{
"container": {
"id": "cd52120cda244a98ed01e9259381cbcd439bc087115d93c7573bada504dbb620",
"id": "54c671509ea7b99312164a5a49297e48eb0b724a8cb2459d2d3917a25ce0c585",
"image": {
"name": "registry.k8s.io/kube-scheduler-amd64:v1.29.1"
"name": "registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.12.0"
},
"runtime": "containerd"
},
Expand All @@ -596,13 +596,8 @@
},
"kubernetes": {
"container": {
"cpu": {
"request": {
"cores": 0.1
}
},
"id": "cd52120cda244a98ed01e9259381cbcd439bc087115d93c7573bada504dbb620",
"name": "kube-scheduler",
"id": "54c671509ea7b99312164a5a49297e48eb0b724a8cb2459d2d3917a25ce0c585",
"name": "kube-state-metrics",
"status": {
"phase": "running",
"ready": true,
Expand All @@ -614,7 +609,7 @@
"name": "kind-control-plane"
},
"pod": {
"name": "kube-scheduler-kind-control-plane"
"name": "kube-state-metrics-5bcd4898-bntgt"
}
},
"metricset": {
Expand All @@ -628,9 +623,9 @@
},
{
"container": {
"id": "54c671509ea7b99312164a5a49297e48eb0b724a8cb2459d2d3917a25ce0c585",
"id": "1be2795038ca3a3a291a6aca5d7ecdfaf0b326453c74e471bab3064822f0bdfb",
"image": {
"name": "registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.12.0"
"name": "registry.k8s.io/coredns/coredns:v1.11.1"
},
"runtime": "containerd"
},
Expand All @@ -641,8 +636,21 @@
},
"kubernetes": {
"container": {
"id": "54c671509ea7b99312164a5a49297e48eb0b724a8cb2459d2d3917a25ce0c585",
"name": "kube-state-metrics",
"cpu": {
"request": {
"cores": 0.1
}
},
"id": "1be2795038ca3a3a291a6aca5d7ecdfaf0b326453c74e471bab3064822f0bdfb",
"memory": {
"limit": {
"bytes": 178257920
},
"request": {
"bytes": 73400320
}
},
"name": "coredns",
"status": {
"phase": "running",
"ready": true,
Expand All @@ -654,7 +662,7 @@
"name": "kind-control-plane"
},
"pod": {
"name": "kube-state-metrics-5bcd4898-bntgt"
"name": "coredns-76f75df574-v8skx"
}
},
"metricset": {
Expand All @@ -668,9 +676,9 @@
},
{
"container": {
"id": "1be2795038ca3a3a291a6aca5d7ecdfaf0b326453c74e471bab3064822f0bdfb",
"id": "cd52120cda244a98ed01e9259381cbcd439bc087115d93c7573bada504dbb620",
"image": {
"name": "registry.k8s.io/coredns/coredns:v1.11.1"
"name": "registry.k8s.io/kube-scheduler-amd64:v1.29.1"
},
"runtime": "containerd"
},
Expand All @@ -686,17 +694,11 @@
"cores": 0.1
}
},
"id": "1be2795038ca3a3a291a6aca5d7ecdfaf0b326453c74e471bab3064822f0bdfb",
"memory": {
"limit": {
"bytes": 178257920
},
"request": {
"bytes": 73400320
}
},
"name": "coredns",
"id": "cd52120cda244a98ed01e9259381cbcd439bc087115d93c7573bada504dbb620",
"name": "kube-scheduler",
"status": {
"last_terminated_reason": "Error",
"last_terminated_reason_timestamp": 1713935860,
"phase": "running",
"ready": true,
"restarts": 0
Expand All @@ -707,7 +709,7 @@
"name": "kind-control-plane"
},
"pod": {
"name": "coredns-76f75df574-v8skx"
"name": "kube-scheduler-kind-control-plane"
}
},
"metricset": {
Expand Down
17 changes: 9 additions & 8 deletions metricbeat/module/kubernetes/state_container/state_container.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,14 +59,15 @@ var (
},
)),

"kube_pod_container_status_ready": p.BooleanMetric("status.ready"),
"kube_pod_container_status_restarts_total": p.Metric("status.restarts"),
"kube_pod_container_status_running": p.KeywordMetric("status.phase", "running"),
"kube_pod_container_status_terminated": p.KeywordMetric("status.phase", "terminated"),
"kube_pod_container_status_waiting": p.KeywordMetric("status.phase", "waiting"),
"kube_pod_container_status_terminated_reason": p.LabelMetric("status.reason", "reason"),
"kube_pod_container_status_waiting_reason": p.LabelMetric("status.reason", "reason"),
"kube_pod_container_status_last_terminated_reason": p.LabelMetric("status.last_terminated_reason", "reason"),
"kube_pod_container_status_ready": p.BooleanMetric("status.ready"),
"kube_pod_container_status_restarts_total": p.Metric("status.restarts"),
"kube_pod_container_status_running": p.KeywordMetric("status.phase", "running"),
"kube_pod_container_status_terminated": p.KeywordMetric("status.phase", "terminated"),
"kube_pod_container_status_waiting": p.KeywordMetric("status.phase", "waiting"),
"kube_pod_container_status_terminated_reason": p.LabelMetric("status.reason", "reason"),
"kube_pod_container_status_waiting_reason": p.LabelMetric("status.reason", "reason"),
"kube_pod_container_status_last_terminated_reason": p.LabelMetric("status.last_terminated_reason", "reason"),
"kube_pod_container_status_last_terminated_timestamp": p.Metric("status.last_terminated_reason.sec"),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here you map kube_pod_container_status_last_terminated_timestamp ksm metric to elasticsearch field kubernetes.container.status.last_terminated_reason.sec. But then , the field you declare in the fields.yaml is last_terminated_reason_timestamp. Am I missing something?

Copy link
Contributor Author

@tetianakravchenko tetianakravchenko Apr 25, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my mistake, reverted this - 180afdf

It is a leftover, I was considering other naming, not really sure regarding the name last_terminated_reason_timestamp. For example in state_cronjob - https://github.com/elastic/beats/blob/main/metricbeat/module/kubernetes/state_cronjob/state_cronjob.go#L33-L34 is used *.sec for time metrics, but then I figured out that there is not really a pattern for time metrics, for example:
job - time.* (type of the field: data)
statefulset - no time or sec in name (type: long)
storageclass - no time or sec in name (type: date)
namespace - created.sec, type: double

do you have any thoughts on naming?
maybe use last_terminated (similar to *.created with the type: double) ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have the field status.last_terminated_reason that corresponds to kube_pod_container_status_last_terminated_reason.
If we follow the same convention for kube_pod_container_status_last_terminated_timestamp let's name the field status.last_terminated_timestamp. There is no need to add the reason in the field name as it is just the timestamp of the last termination. It is not related with the reason.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done in 90d82b6

},

Labels: map[string]p.LabelMap{
Expand Down
Loading