Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delete stale metrics on object delete #753

Merged
merged 1 commit into from
Aug 15, 2023
Merged

Delete stale metrics on object delete #753

merged 1 commit into from
Aug 15, 2023

Conversation

darkowlzz
Copy link
Contributor

@darkowlzz darkowlzz commented Aug 10, 2023

Depends on fluxcd/pkg#612

Use the metrics helper to record all the metrics. Metrics helper ensures that the metrics for deleted objects are deleted as well.

Move all the metrics recording to be performed at the very end of the reconciliation. Realtime metrics for readiness is no longer recorded as it will be removed in a future version for CRD metrics collected using kube-state-metrics. Updating the object status with realtime readiness should provide the readiness to CRD metrics watchers.

HelmReleaseReconciler.reconcileDelete() is modified to receive a pointer HelmRelease object so that any modifications on the object is reflected on the object instance that's passed to the metrics recorder. This is not needed for HelmReleaseReconciler.reconcile() as it returns a new copy of the object that's saved in the same object variable, overwriting the object instance with the updates.

Before this change, the following metrics continued to be exported even after the associated object is deleted for HelmRelease:

gotk_reconcile_condition{kind="HelmRelease",name="podinfo",namespace="default",status="False",type="Ready"} 0
gotk_reconcile_condition{kind="HelmRelease",name="podinfo",namespace="default",status="True",type="Ready"} 1
gotk_reconcile_condition{kind="HelmRelease",name="podinfo",namespace="default",status="Unknown",type="Ready"} 0
...
gotk_reconcile_duration_seconds_bucket{kind="HelmRelease",name="podinfo",namespace="default",le="0.01"} 0
gotk_reconcile_duration_seconds_bucket{kind="HelmRelease",name="podinfo",namespace="default",le="0.038363583488692544"} 1
gotk_reconcile_duration_seconds_bucket{kind="HelmRelease",name="podinfo",namespace="default",le="0.1471764538093883"} 1
gotk_reconcile_duration_seconds_bucket{kind="HelmRelease",name="podinfo",namespace="default",le="0.5646216173286169"} 2
gotk_reconcile_duration_seconds_bucket{kind="HelmRelease",name="podinfo",namespace="default",le="2.166090855590701"} 2
gotk_reconcile_duration_seconds_bucket{kind="HelmRelease",name="podinfo",namespace="default",le="8.309900738254731"} 2
gotk_reconcile_duration_seconds_bucket{kind="HelmRelease",name="podinfo",namespace="default",le="31.879757075478317"} 2
gotk_reconcile_duration_seconds_bucket{kind="HelmRelease",name="podinfo",namespace="default",le="122.30217221643493"} 2
gotk_reconcile_duration_seconds_bucket{kind="HelmRelease",name="podinfo",namespace="default",le="469.19495946736544"} 2
gotk_reconcile_duration_seconds_bucket{kind="HelmRelease",name="podinfo",namespace="default",le="1799.9999999999986"} 2
gotk_reconcile_duration_seconds_bucket{kind="HelmRelease",name="podinfo",namespace="default",le="+Inf"} 2
gotk_reconcile_duration_seconds_sum{kind="HelmRelease",name="podinfo",namespace="default"} 0.227111736
...
gotk_suspend_status{kind="HelmRelease",name="podinfo",namespace="default"} 0

With this change, they get deleted once the associated object is deleted.

Also, the metrics helper no longer exports ConditionDelete for readiness metrics.

@darkowlzz darkowlzz added the enhancement New feature or request label Aug 10, 2023
@darkowlzz darkowlzz force-pushed the stale-metrics branch 3 times, most recently from a47aac9 to 82a5325 Compare August 11, 2023 14:21
Use the metrics helper to record all the metrics. Metrics helpers
ensures that the metrics for deleted objects are deleted as well.

Move all the metrics recording to be performed at the very end of the
reconciliation. Realtime metrics for readiness is no longer recorded as
it will be removed in a future version for CRD metrics collected using
kube-state-metrics. Updating the object status with realtime readiness
should provide the readiness to CRD metrics watchers.

`HelmReleaseReconciler.reconcileDelete()` is modified to receive a
pointer HelmRelease object so that any modifications on the object is
reflected on the object instance that's passed to the metrics recorder.
This is not needed for `HelmReleaseReconciler.reconcile()` as it returns
a new copy of the object that's saved in the same object variable,
overwriting the object instance with the updates.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
@stefanprodan stefanprodan merged commit a8c15d3 into main Aug 15, 2023
6 checks passed
@stefanprodan stefanprodan deleted the stale-metrics branch August 15, 2023 08:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants