Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add kube_pod_container_status_last_terminated_timestamp metric #2291

Merged
merged 4 commits into from
Mar 25, 2024
Merged

feat: add kube_pod_container_status_last_terminated_timestamp metric #2291

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
51ccb66
feat: add kube_pod_container_status_last_terminated_timestamp metric
tetianakravchenko Dec 28, 2023
8876d40
add kube_pod_container_status_last_terminated_timestamp to the unit t…
tetianakravchenko Jan 22, 2024
dbdb77c
fix benchmark test; add missing kube_pod_container_status_last_termin…
tetianakravchenko Jan 22, 2024
393430e
fix the description of the 'kube_pod_container_status_last_terminated…
tetianakravchenko Mar 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/pod-metrics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
| kube_pod_container_status_terminated_reason | Gauge | Describes the reason the container is currently in terminated state | | `container`=&lt;container-name&gt; <br> `pod`=&lt;pod-name&gt; <br> `namespace`=&lt;pod-namespace&gt; <br> `reason`=&lt;container-terminated-reason&gt; <br> `uid`=&lt;pod-uid&gt; | EXPERIMENTAL | - |
| kube_pod_container_status_last_terminated_reason | Gauge | Describes the last reason the container was in terminated state | | `container`=&lt;container-name&gt; <br> `pod`=&lt;pod-name&gt; <br> `namespace`=&lt;pod-namespace&gt; <br> `reason`=&lt;last-terminated-reason&gt; <br> `uid`=&lt;pod-uid&gt; | EXPERIMENTAL | - |
| kube_pod_container_status_last_terminated_exitcode | Gauge | Describes the exit code for the last container in terminated state. | | `container`=&lt;container-name&gt; <br> `pod`=&lt;pod-name&gt; <br> `namespace`=&lt;pod-namespace&gt; <br> `uid`=&lt;pod-uid&gt; | EXPERIMENTAL | - |
| kube_pod_container_status_last_terminated_timestamp | Gauge | Last terminated time for a pod container, when container was terminated with kube_pod_container_status_last_terminated_reason, in unix timestamp. | | `container`=&lt;container-name&gt; <br> `pod`=&lt;pod-name&gt; <br> `namespace`=&lt;pod-namespace&gt; <br> `uid`=&lt;pod-uid&gt; | EXPERIMENTAL | - |
| kube_pod_container_status_ready | Gauge | Describes whether the containers readiness check succeeded | | `container`=&lt;container-name&gt; <br> `pod`=&lt;pod-name&gt; <br> `namespace`=&lt;pod-namespace&gt; <br> `uid`=&lt;pod-uid&gt; | STABLE | - |
| kube_pod_status_initialized_time | Gauge | Time when the pod is initialized. | seconds | `pod`=&lt;pod-name&gt; <br> `namespace`=&lt;pod-namespace&gt; <br> `uid`=&lt;pod-uid&gt; | EXPERIMENTAL |
| kube_pod_status_ready_time | Gauge | Time when pod passed readiness probes. | seconds | `pod`=&lt;pod-name&gt; <br> `namespace`=&lt;pod-namespace&gt; <br> `uid`=&lt;pod-uid&gt; | EXPERIMENTAL |
Expand Down
27 changes: 27 additions & 0 deletions internal/store/pod.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ func podMetricFamilies(allowAnnotationsList, allowLabelsList []string) []generat
createPodContainerStateStartedFamilyGenerator(),
createPodContainerStatusLastTerminatedReasonFamilyGenerator(),
createPodContainerStatusLastTerminatedExitCodeFamilyGenerator(),
createPodContainerStatusLastTerminatedTimestampFamilyGenerator(),
createPodContainerStatusReadyFamilyGenerator(),
createPodContainerStatusRestartsTotalFamilyGenerator(),
createPodContainerStatusRunningFamilyGenerator(),
Expand Down Expand Up @@ -375,6 +376,32 @@ func createPodContainerStatusLastTerminatedExitCodeFamilyGenerator() generator.F
)
}

func createPodContainerStatusLastTerminatedTimestampFamilyGenerator() generator.FamilyGenerator {
return *generator.NewFamilyGeneratorWithStability(
"kube_pod_container_status_last_terminated_timestamp",
"Last terminated time for a pod container, when container was terminated with kube_pod_container_status_last_terminated_reason, in unix timestamp.",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"Last terminated time for a pod container, when container was terminated with kube_pod_container_status_last_terminated_reason, in unix timestamp.",
"Last terminated time for a pod container in unix timestamp.",

Is when container was terminated with kube_pod_container_status_last_terminated_reason necessary here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @rexagod , thank you for the feedback! I've tried to be more precise in the description, but I think you are right, there is no need to keep it.
Cleaned it up in this commit - 393430e, please have a look

metric.Gauge,
basemetrics.ALPHA,
"",
wrapPodFunc(func(p *v1.Pod) *metric.Family {
ms := make([]*metric.Metric, 0, len(p.Status.ContainerStatuses))
for _, cs := range p.Status.ContainerStatuses {
if cs.LastTerminationState.Terminated != nil {
ms = append(ms, &metric.Metric{
LabelKeys: []string{"container"},
LabelValues: []string{cs.Name},
Value: float64(cs.LastTerminationState.Terminated.FinishedAt.Unix()),
})
}
}

return &metric.Family{
Metrics: ms,
}
}),
)
}

func createPodContainerStatusReadyFamilyGenerator() generator.FamilyGenerator {
return *generator.NewFamilyGeneratorWithStability(
"kube_pod_container_status_ready",
Expand Down
26 changes: 24 additions & 2 deletions internal/store/pod_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -690,7 +690,6 @@ func TestPodStore(t *testing.T) {
},
},
{

Obj: &v1.Pod{
ObjectMeta: metav1.ObjectMeta{
Name: "pod6",
Expand Down Expand Up @@ -720,6 +719,9 @@ func TestPodStore(t *testing.T) {
Terminated: &v1.ContainerStateTerminated{
Reason: "OOMKilled",
ExitCode: 137,
FinishedAt: metav1.Time{
Time: time.Unix(1501779547, 0),
},
},
},
},
Expand All @@ -729,6 +731,7 @@ func TestPodStore(t *testing.T) {
Want: `
# HELP kube_pod_container_status_last_terminated_reason Describes the last reason the container was in terminated state.
# HELP kube_pod_container_status_last_terminated_exitcode Describes the exit code for the last container in terminated state.
# HELP kube_pod_container_status_last_terminated_timestamp Last terminated time for a pod container, when container was terminated with kube_pod_container_status_last_terminated_reason, in unix timestamp.
# HELP kube_pod_container_status_running [STABLE] Describes whether the container is currently in running state.
# HELP kube_pod_container_status_terminated [STABLE] Describes whether the container is currently in terminated state.
# HELP kube_pod_container_status_terminated_reason Describes the reason the container is currently in terminated state.
Expand All @@ -737,6 +740,7 @@ func TestPodStore(t *testing.T) {
# HELP kube_pod_container_state_started [STABLE] Start time in unix timestamp for a pod container.
# TYPE kube_pod_container_status_last_terminated_reason gauge
# TYPE kube_pod_container_status_last_terminated_exitcode gauge
# TYPE kube_pod_container_status_last_terminated_timestamp gauge
# TYPE kube_pod_container_status_running gauge
# TYPE kube_pod_container_status_terminated gauge
# TYPE kube_pod_container_status_terminated_reason gauge
Expand All @@ -749,10 +753,12 @@ func TestPodStore(t *testing.T) {
kube_pod_container_status_waiting{container="container7",namespace="ns6",pod="pod6",uid="uid6"} 0
kube_pod_container_status_last_terminated_reason{container="container7",namespace="ns6",pod="pod6",reason="OOMKilled",uid="uid6"} 1
kube_pod_container_status_last_terminated_exitcode{container="container7",namespace="ns6",pod="pod6",uid="uid6"} 137
kube_pod_container_status_last_terminated_timestamp{container="container7",namespace="ns6",pod="pod6",uid="uid6"} 1.501779547e+09
`,
MetricNames: []string{
"kube_pod_container_status_last_terminated_reason",
"kube_pod_container_status_last_terminated_exitcode",
"kube_pod_container_status_last_terminated_timestamp",
"kube_pod_container_status_running",
"kube_pod_container_state_started",
"kube_pod_container_status_terminated",
Expand Down Expand Up @@ -790,6 +796,9 @@ func TestPodStore(t *testing.T) {
Terminated: &v1.ContainerStateTerminated{
Reason: "DeadlineExceeded",
ExitCode: 143,
FinishedAt: metav1.Time{
Time: time.Unix(1501779547, 0),
},
},
},
},
Expand All @@ -799,6 +808,7 @@ func TestPodStore(t *testing.T) {
Want: `
# HELP kube_pod_container_status_last_terminated_exitcode Describes the exit code for the last container in terminated state.
# HELP kube_pod_container_status_last_terminated_reason Describes the last reason the container was in terminated state.
# HELP kube_pod_container_status_last_terminated_timestamp Last terminated time for a pod container, when container was terminated with kube_pod_container_status_last_terminated_reason, in unix timestamp.
# HELP kube_pod_container_status_running [STABLE] Describes whether the container is currently in running state.
# HELP kube_pod_container_state_started [STABLE] Start time in unix timestamp for a pod container.
# HELP kube_pod_container_status_terminated [STABLE] Describes whether the container is currently in terminated state.
Expand All @@ -807,6 +817,7 @@ func TestPodStore(t *testing.T) {
# HELP kube_pod_container_status_waiting_reason [STABLE] Describes the reason the container is currently in waiting state.
# TYPE kube_pod_container_status_last_terminated_exitcode gauge
# TYPE kube_pod_container_status_last_terminated_reason gauge
# TYPE kube_pod_container_status_last_terminated_timestamp gauge
# TYPE kube_pod_container_status_running gauge
# TYPE kube_pod_container_state_started gauge
# TYPE kube_pod_container_status_terminated gauge
Expand All @@ -816,6 +827,7 @@ func TestPodStore(t *testing.T) {
kube_pod_container_state_started{container="container7",namespace="ns7",pod="pod7",uid="uid7"} 1.501777018e+09
kube_pod_container_status_last_terminated_exitcode{container="container7",namespace="ns7",pod="pod7",uid="uid7"} 143
kube_pod_container_status_last_terminated_reason{container="container7",namespace="ns7",pod="pod7",reason="DeadlineExceeded",uid="uid7"} 1
kube_pod_container_status_last_terminated_timestamp{container="container7",namespace="ns7",pod="pod7",uid="uid7"} 1.501779547e+09
kube_pod_container_status_running{container="container7",namespace="ns7",pod="pod7",uid="uid7"} 1
kube_pod_container_status_terminated{container="container7",namespace="ns7",pod="pod7",uid="uid7"} 0
kube_pod_container_status_waiting{container="container7",namespace="ns7",pod="pod7",uid="uid7"} 0
Expand All @@ -825,6 +837,7 @@ func TestPodStore(t *testing.T) {
"kube_pod_container_state_started",
"kube_pod_container_status_terminated",
"kube_pod_container_status_terminated_reason",
"kube_pod_container_status_last_terminated_timestamp",
"kube_pod_container_status_waiting",
"kube_pod_container_status_last_terminated_reason",
"kube_pod_container_status_last_terminated_exitcode",
Expand Down Expand Up @@ -2210,6 +2223,9 @@ func BenchmarkPodStore(b *testing.B) {
},
LastTerminationState: v1.ContainerState{
Terminated: &v1.ContainerStateTerminated{
FinishedAt: metav1.Time{
Time: time.Unix(1501779547, 0),
},
Reason: "OOMKilled",
ExitCode: 137,
},
Expand All @@ -2227,6 +2243,9 @@ func BenchmarkPodStore(b *testing.B) {
},
LastTerminationState: v1.ContainerState{
Terminated: &v1.ContainerStateTerminated{
FinishedAt: metav1.Time{
Time: time.Unix(1501779547, 0),
},
Reason: "OOMKilled",
ExitCode: 137,
},
Expand All @@ -2244,6 +2263,9 @@ func BenchmarkPodStore(b *testing.B) {
},
LastTerminationState: v1.ContainerState{
Terminated: &v1.ContainerStateTerminated{
FinishedAt: metav1.Time{
Time: time.Unix(1501779547, 0),
},
Reason: "OOMKilled",
ExitCode: 137,
},
Expand All @@ -2253,7 +2275,7 @@ func BenchmarkPodStore(b *testing.B) {
},
}

expectedFamilies := 53
expectedFamilies := 54
for n := 0; n < b.N; n++ {
families := f(pod)
if len(families) != expectedFamilies {
Expand Down
6 changes: 6 additions & 0 deletions pkg/app/server_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -209,6 +209,7 @@ func TestFullScrapeCycle(t *testing.T) {
# HELP kube_pod_container_state_started [STABLE] Start time in unix timestamp for a pod container.
# HELP kube_pod_container_status_last_terminated_exitcode Describes the exit code for the last container in terminated state.
# HELP kube_pod_container_status_last_terminated_reason Describes the last reason the container was in terminated state.
# HELP kube_pod_container_status_last_terminated_timestamp Last terminated time for a pod container, when container was terminated with kube_pod_container_status_last_terminated_reason, in unix timestamp.
# HELP kube_pod_container_status_ready [STABLE] Describes whether the containers readiness check succeeded.
# HELP kube_pod_container_status_restarts_total [STABLE] The number of container restarts per container.
# HELP kube_pod_container_status_running [STABLE] Describes whether the container is currently in running state.
Expand Down Expand Up @@ -261,6 +262,7 @@ func TestFullScrapeCycle(t *testing.T) {
# TYPE kube_pod_container_state_started gauge
# TYPE kube_pod_container_status_last_terminated_exitcode gauge
# TYPE kube_pod_container_status_last_terminated_reason gauge
# TYPE kube_pod_container_status_last_terminated_timestamp gauge
# TYPE kube_pod_container_status_ready gauge
# TYPE kube_pod_container_status_restarts_total counter
# TYPE kube_pod_container_status_running gauge
Expand Down Expand Up @@ -323,6 +325,7 @@ kube_pod_container_resource_requests{namespace="default",pod="pod0",uid="abc-0",
kube_pod_container_resource_requests{namespace="default",pod="pod0",uid="abc-0",container="pod1_con2",node="node1",resource="memory",unit="byte"} 2e+08
kube_pod_container_status_last_terminated_exitcode{namespace="default",pod="pod0",uid="abc-0",container="pod1_con1"} 137
kube_pod_container_status_last_terminated_reason{namespace="default",pod="pod0",uid="abc-0",container="pod1_con1",reason="OOMKilled"} 1
kube_pod_container_status_last_terminated_timestamp{namespace="default",pod="pod0",uid="abc-0",container="pod1_con1"} 1.501779547e+09
kube_pod_container_status_ready{namespace="default",pod="pod0",uid="abc-0",container="pod1_con1"} 0
kube_pod_container_status_ready{namespace="default",pod="pod0",uid="abc-0",container="pod1_con2"} 0
kube_pod_container_status_restarts_total{namespace="default",pod="pod0",uid="abc-0",container="pod1_con1"} 0
Expand Down Expand Up @@ -834,6 +837,9 @@ func pod(client *fake.Clientset, index int) error {
},
LastTerminationState: v1.ContainerState{
Terminated: &v1.ContainerStateTerminated{
FinishedAt: metav1.Time{
Time: time.Unix(1501779547, 0),
},
Reason: "OOMKilled",
ExitCode: 137,
},
Expand Down