Add l4netlb metrics #1678

kl52752 · 2022-02-24T13:54:57Z

No description provided.

k8s-ci-robot · 2022-02-24T13:55:06Z

Hi @kl52752. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kl52752 · 2022-02-24T13:55:43Z

/assign @bowei

freehan · 2022-03-02T19:04:28Z

/ok-to-test

freehan · 2022-03-03T02:33:27Z

pkg/l4lb/l4netlbcontroller.go

+	case loadbalancers.SyncTypeDelete:
+		// if service is successfully deleted, remove it from cache
+		if result.Error == nil {
+			klog.V(6).Infof("External L4 Loadbalancer for Service %s deleted, removing its state from metrics cache", namespacedName)


freehan · 2022-03-03T02:34:46Z

pkg/l4lb/metrics/metrics.go

+		prometheus.HistogramOpts{
+			Name: L4netlbLatencyMetricName,
+			Help: "Latency of an L4 NetLB Sync",
+			// custom buckets - [30s, 60s, 120s, 240s(4min), 480s(8min), 960s(16m), +Inf]


You are sure the smallest bucket contains <30s is good enough?

you are right, we need to run more tests

In my experiment, 33 seconds was the shortest time. But most times was around 40-45 sec.

Changed to start with 15s

freehan · 2022-03-03T02:36:35Z

pkg/l4lb/l4netlbcontroller.go

+	}
+	switch result.SyncType {
+	case loadbalancers.SyncTypeCreate, loadbalancers.SyncTypeUpdate:
+		klog.V(6).Infof("External L4 Loadbalancer for Service %s ensured, updating its state %v in metrics cache", namespacedName, result.MetricsState)


V(4)? why V(6)?

freehan · 2022-03-03T02:39:18Z

pkg/l4lb/metrics/metrics.go

+		},
+		l4LBSyncLatencyMetricsLabels,
+	)
+	l4NetLBSyncErrorCount = prometheus.NewCounterVec(


did you guys found out how the counter got handled in the case of master container restart?

Yes, I have jus checked the following metrics:

number_of_ingresses
number_of_service_ports

They were NOT exported with value "0", just after registration in prometheus
They were exported once I created load balancers.

I am not sure if this is a prometheus feature or prom-to-sd

It seems values are exported only after we set some value

I thought errors would be logged as a delta?

cezarygerard · 2022-03-04T10:53:26Z

pkg/test/utils.go

+func GetL4NetLBErrorMetric(t *testing.T) *L4LBErrorMetricInfo {
+	return getL4LBErrorMetric(t, metrics.L4netlbErrorMetricName)
+}
+
 func getL4LBErrorMetric(t *testing.T, metricName string) *L4LBErrorMetricInfo {


typo L4LBE
L4ELB

:) It is L4-LB-Error i can change to getL4lbErrorMetric?

cezarygerard · 2022-03-04T10:54:03Z

pkg/test/utils.go

@@ -342,6 +342,11 @@ func GetL4ILBLatencyMetric(t *testing.T) *L4LBLatencyMetricInfo {
 	return getL4LatencyMetric(t, metrics.L4ilbLatencyMetricName)
 }

+// GetL4NetLBLatencyMetric gets the current state of the l4_netlb_sync_duration_seconds metric.
+func GetL4NetLBLatencyMetric(t *testing.T) *L4LBLatencyMetricInfo {


this should return error rather than pass 't'
or t.Helper() a least
and preferably renamed to MustGetL4NetLBLatencyMetric
go/go-style/decisions#must-functions

ack, and no change?

sorry, forget to push, now it is done

cezarygerard · 2022-03-04T12:38:45Z

pkg/loadbalancers/l4netlb_test.go

@@ -81,6 +96,16 @@ func checkAnnotations(result *L4NetLBSyncResult, l4netlb *L4NetLB) error {
 	return nil
 }

+func checkMetrics(m metrics.L4NetLBServiceState, isManaged, isPremium bool) error {


let's put this at the end of file

cezarygerard · 2022-03-04T12:41:09Z

pkg/loadbalancers/l4netlb_test.go

+	}
+}
+
+func createUsersStatusIP(fakeGCE *gce.Cloud, region string) {


createUsersStatusIP I don't get a name

what UsersStatus means?

it was supposed to be Static :D

cezarygerard · 2022-03-04T12:44:19Z

pkg/loadbalancers/l4netlb_test.go

+const (
+	managed      = true
+	unmanaged    = false
+	premium      = true


premiumTier?

cezarygerard · 2022-03-04T12:44:44Z

pkg/loadbalancers/l4netlb_test.go

+	managed      = true
+	unmanaged    = false
+	premium      = true
+	standard     = false


standardTier?

cezarygerard · 2022-03-04T12:44:57Z

pkg/loadbalancers/l4netlb_test.go

 	"k8s.io/ingress-gce/pkg/test"
 	"k8s.io/ingress-gce/pkg/utils"
 	namer_util "k8s.io/ingress-gce/pkg/utils/namer"
 	"k8s.io/legacy-cloud-providers/gce"
 )

+const (
+	managed      = true


managedIP or managedAddress

cezarygerard · 2022-03-04T12:45:15Z

pkg/loadbalancers/l4netlb_test.go

 	"k8s.io/ingress-gce/pkg/test"
 	"k8s.io/ingress-gce/pkg/utils"
 	namer_util "k8s.io/ingress-gce/pkg/utils/namer"
 	"k8s.io/legacy-cloud-providers/gce"
 )

+const (
+	managed      = true
+	unmanaged    = false


unmanagedIP or unmanagedAddress

cezarygerard · 2022-03-04T13:00:42Z

pkg/metrics/metrics.go

@@ -124,6 +134,8 @@ type ControllerMetrics struct {
 	negMap map[string]NegServiceState
 	// l4ILBServiceMap is a map between service key and L4 ILB service state.
 	l4ILBServiceMap map[string]L4ILBServiceState
+	// l4NetLBServiceMap is a map between service key and L4 NetLB service state.
+	l4NetLBServiceMap map[string]L4NetLBServiceState


maybe it would be better to use sync.Map
for every map in ControllerMetrics
maybe leave a
// todo ...

still not todo...

I added TODO in line 143

//TODO(kl52752) remove mutex and change map to sync.map sync.Mutex

If you leave a TODO, file an issue. Don't attach to a person..

cezarygerard · 2022-03-04T13:12:08Z

pkg/l4lb/l4netlbcontroller.go

+	case loadbalancers.SyncTypeCreate, loadbalancers.SyncTypeUpdate:
+		klog.V(6).Infof("External L4 Loadbalancer for Service %s ensured, updating its state %v in metrics cache", namespacedName, result.MetricsState)
+		lc.ctx.ControllerMetrics.SetL4NetLBService(namespacedName, result.MetricsState)
+		l4metrics.PublishNetLBSyncMetrics(result.Error == nil, result.SyncType, result.GCEResourceInError, utils.GetErrorType(result.Error), result.StartTime)


why not pass result struct as argument
l4metrics.PublishNetLBSyncMetrics(result)

it this couses circular dependency then maybe L4NetLBSyncResult is declared in wrong package?

utils.GetErrorType(result.Error)
can also be called inside l4metrics.PublishNetLBSyncMetrics(

I will do it in new review because I think that we can separate fields that are needed only for metrics from sync result and I think that it will be bigger change. WDYT?

cezarygerard

please address or apply all my comments

kl52752 · 2022-03-15T09:13:55Z

/retest

cezarygerard · 2022-03-15T11:20:23Z

/lgtm

k8s-ci-robot · 2022-03-15T11:20:43Z

@cezarygerard: changing LGTM is restricted to collaborators

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

freehan · 2022-03-15T23:59:36Z

pkg/test/utils.go

@@ -338,19 +338,23 @@ type L4LBLatencyMetricInfo struct {
 }

 // GetL4ILBLatencyMetric gets the current state of the l4_ilb_sync_duration_seconds metric.
-func GetL4ILBLatencyMetric(t *testing.T) *L4LBLatencyMetricInfo {
-	return getL4LatencyMetric(t, metrics.L4ilbLatencyMetricName)
+func GetL4ILBLatencyMetric() (error, *L4LBLatencyMetricInfo) {


golang convention is to put the error as the last returning value.

freehan · 2022-03-15T23:59:43Z

pkg/test/utils.go

 }

-func getL4LatencyMetric(t *testing.T, metricName string) *L4LBLatencyMetricInfo {
+// GetL4NetLBLatencyMetric gets the current state of the l4_netlb_sync_duration_seconds metric.


freehan · 2022-03-15T23:59:51Z

pkg/test/utils.go

+	return getL4LatencyMetric(metrics.L4netlbLatencyMetricName)
+}
+
+func getL4LatencyMetric(metricName string) (error, *L4LBLatencyMetricInfo) {


freehan · 2022-03-16T00:00:00Z

pkg/test/utils.go

@@ -416,16 +420,21 @@ type L4LBErrorMetricInfo struct {
 }

 // GetL4ILBErrorMetric gets the current state of the l4_ilb_sync_error_count.
-func GetL4ILBErrorMetric(t *testing.T) *L4LBErrorMetricInfo {
-	return getL4LBErrorMetric(t, metrics.L4ilbErrorMetricName)
+func GetL4ILBErrorMetric() (error, *L4LBErrorMetricInfo) {


freehan

LGTM overall
just nits

freehan · 2022-03-19T00:34:13Z

pkg/l4lb/metrics/metrics.go

@@ -58,12 +60,30 @@ var (
 		},
 		l4LBSyncErrorMetricLabels,
 	)
+	l4NetLBSyncLatency = prometheus.NewHistogramVec(


do you mind just add a brief comment on the metrics?

Easier to understand

yes done, and I fixed the buckets count because previously I changes it to 15s but for ILB metrics 😄

freehan

/lgtm
/approve

k8s-ci-robot · 2022-03-21T22:22:45Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cezarygerard, freehan, kl52752

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [freehan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

bowei · 2022-03-22T06:05:32Z

pkg/l4lb/l4netlbcontroller.go

 	listers "k8s.io/client-go/listers/core/v1"
 	"k8s.io/client-go/tools/cache"
 	"k8s.io/ingress-gce/pkg/annotations"
 	"k8s.io/ingress-gce/pkg/backends"
 	"k8s.io/ingress-gce/pkg/context"
 	"k8s.io/ingress-gce/pkg/controller/translator"
 	"k8s.io/ingress-gce/pkg/instances"
+	l4metrics "k8s.io/ingress-gce/pkg/l4lb/metrics"


don't alias packages unless it is required due to a conflict

bowei · 2022-03-22T06:06:15Z

pkg/l4lb/l4netlbcontroller.go

@@ -107,6 +109,7 @@ func NewL4NetLBController(
 			}
 		},
 	})
+	//TODO change to component name "l4netlb-controller"


If you leave a todo, you need to file an issue and link it

TODO(): short description.

Otherwise finish the work and don't leave a TODO in the code.

this comment will be removed in #1683

bowei · 2022-03-22T06:07:10Z

pkg/l4lb/l4netlbcontroller.go

 	if lc.needsDeletion(svc) {
 		klog.V(3).Infof("Deleting L4 External LoadBalancer resources for service %s", key)
 		result := lc.garbageCollectRBSNetLB(key, svc)
 		if result == nil {
 			return nil
 		}
+		lc.publishMetrics(result, namespacedName)


Why not change this method to take a types.NamespacesName instead of a string? Then you don't have to have this outside the method.

bowei · 2022-03-22T06:08:45Z

pkg/l4lb/l4netlbcontroller.go

+// publishMetrics sets controller metrics for NetLB services and pushes NetLB metrics based on sync type.
+func (lc *L4NetLBController) publishMetrics(result *loadbalancers.L4NetLBSyncResult, namespacedName string) {
+	if result == nil {
+		return


Is this something that we expect? Should we log an error? Why will this happen?
There is no explanation of why this is an expected value for the parameter.

If we do not expect it to be nil, do not silently ignore it.

removed, this check is done before we call publishMetrics

bowei · 2022-03-22T06:10:37Z

pkg/l4lb/l4netlbcontroller.go

+	case loadbalancers.SyncTypeCreate, loadbalancers.SyncTypeUpdate:
+		klog.V(4).Infof("External L4 Loadbalancer for Service %s ensured, updating its state %v in metrics cache", namespacedName, result.MetricsState)
+		lc.ctx.ControllerMetrics.SetL4NetLBService(namespacedName, result.MetricsState)
+		l4metrics.PublishNetLBSyncMetrics(result.Error == nil, result.SyncType, result.GCEResourceInError, utils.GetErrorType(result.Error), result.StartTime)


It will be easier to reason about if we have

PublishSyncSuccess()

PublishSyncError()

instead of exposing so many parameters to the user. then PublishNetLBSyncMetrics() can be an internal method.

bowei · 2022-03-22T06:11:20Z

pkg/l4lb/metrics/metrics.go

-	statusError            = "error"
-	L4ilbLatencyMetricName = "l4_ilb_sync_duration_seconds"
-	L4ilbErrorMetricName   = "l4_ilb_sync_error_count"
+	statusSuccess            = "success"


if these are reference in only one place, we don't need to create a named constant symbol for the value.

it is used by ILB and NetLB

bowei · 2022-03-22T06:11:48Z

pkg/l4lb/metrics/metrics.go

+			Name: L4netlbLatencyMetricName,
+			Help: "Latency of an L4 NetLB Sync",
+			// custom buckets - [15s, 30s, 60s, 120s, 240s(4min), 480s(8min), 960s(16m), +Inf]
+			Buckets: prometheus.ExponentialBuckets(15, 2, 6),


Can we have more buckets? They are low cost and better than missing out of the data.

bowei · 2022-03-22T06:13:00Z

pkg/l4lb/metrics/metrics.go

@@ -87,3 +111,25 @@ func publishL4ILBSyncLatency(success bool, syncType string, startTime time.Time)
 func publishL4ILBSyncErrorCount(syncType, gceResource, errorType string) {
 	l4ILBSyncErrorCount.WithLabelValues(syncType, gceResource, errorType).Inc()
 }
+
+// PublishL4NetLBSyncMetrics exports metrics related to the L4 NetLB sync.
+func PublishNetLBSyncMetrics(success bool, syncType, gceResource, errType string, startTime time.Time) {


see comment above

bowei · 2022-03-22T06:14:26Z

pkg/loadbalancers/l4netlb.go

@@ -153,6 +152,12 @@ func (l4netlb *L4NetLB) EnsureFrontend(nodeNames []string, svc *corev1.Service)
 		result.Annotations[annotations.UDPForwardingRuleKey] = fr.Name
 	}
 	result.Status = &corev1.LoadBalancerStatus{Ingress: []corev1.LoadBalancerIngress{{IP: fr.IPAddress}}}
+	if fr.NetworkTier == cloud.NetworkTierPremium.ToGCEValue() {


result.MetricsState.IsPremiumTier = fr.NetworkTier == cloud.NetworkTierPremium.ToGCEValue()

bowei · 2022-03-22T06:14:49Z

pkg/loadbalancers/l4netlb.go

+	if fr.NetworkTier == cloud.NetworkTierPremium.ToGCEValue() {
+		result.MetricsState.IsPremiumTier = true
+	}
+	if ipAddrType == IPAddrManaged {


result.MetricsState.IsManagedIP = ipAddrType == IPAddrManaged

bowei · 2022-03-22T06:15:59Z

pkg/loadbalancers/l4netlb_test.go

 	"k8s.io/ingress-gce/pkg/test"
 	"k8s.io/ingress-gce/pkg/utils"
 	namer_util "k8s.io/ingress-gce/pkg/utils/namer"
 	"k8s.io/legacy-cloud-providers/gce"
 )

+const (
+	managedIP    = true


[minor] personally prefer checkMetrics(..., /managedIP/true...) for documentation than adding symbols to the package scope.

if this is a minor comment then I preferred not to change it, it is used in few places and iMO it is more readable than comment

bowei · 2022-03-22T06:16:31Z

pkg/metrics/l4metrics_test.go

@@ -23,6 +23,10 @@ import (
 )

 const (
+	premium             = true


see comment above

bowei · 2022-03-22T06:18:28Z

pkg/metrics/metrics.go

@@ -124,21 +134,25 @@ type ControllerMetrics struct {
 	negMap map[string]NegServiceState
 	// l4ILBServiceMap is a map between service key and L4 ILB service state.
 	l4ILBServiceMap map[string]L4ILBServiceState
+	// l4NetLBServiceMap is a map between service key and L4 NetLB service state.
+	l4NetLBServiceMap map[string]L4NetLBServiceState
 	// pscMap is a map between the service attachment key and PSC state
 	pscMap map[string]pscmetrics.PSCState
 	// ServiceMap track the number of services in this cluster
 	serviceMap map[string]struct{}


Can we document why this is a set? Can we just use the set data structure directly?

bowei · 2022-03-22T06:19:49Z

pkg/metrics/metrics.go

+	defer im.Unlock()
+
+	if im.l4NetLBServiceMap == nil {
+		klog.Fatalf("L4 Net LB Metrics failed to initialize correctly.")


Do we expect this to be common vs just having a panic() when accessing the nil map?

Otherwise, to be consistent, we should be checking nil in way more places...

It is very unlikely that the table will be empty because it is initialized in context and the l4netlb controller is created from the context so the table should already be initialized. But since metric controller and l4netlb controller are independent it's good to have this check if sth will change in the future.
I didn't put this check into delete func because the idea is to call set service first.

bowei · 2022-03-22T06:22:34Z

pkg/metrics/metrics.go

+	im.Lock()
+	defer im.Unlock()
+	klog.V(4).Infof("Computing L4 NetLB usage metrics from service state map: %#v", im.l4NetLBServiceMap)
+	counts := map[feature]int{


why use a map vs just a struct?

type ... struct {
serviceCount int
statisIPCount int
...
}

etc vs allocating a dynamic map

Also, if we are using a dynamic map in this way, it will be better to wrap it so it's more consistent to deal with

type l4MetricsMap map[feature]int

func (m* l4MetricsMap) init() {
m[l4Net...] = 0 ...
}

We use this map to set labels and its values to metric code
l4NetLBCount.With(prometheus.Labels{label: feature.String()}).Set(float64(count))

I added type for a map

This change included fixes for comments for closed PRs. kubernetes#1675 kubernetes#1678

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Feb 24, 2022

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Feb 24, 2022

k8s-ci-robot requested review from freehan and thockin February 24, 2022 13:55

k8s-ci-robot assigned bowei Feb 24, 2022

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Mar 2, 2022

freehan reviewed Mar 3, 2022

View reviewed changes

freehan self-assigned this Mar 3, 2022

cezarygerard reviewed Mar 4, 2022

View reviewed changes

kl52752 force-pushed the metrics-l4-rbs branch from c5a8650 to 08ef19b Compare March 4, 2022 14:15

kl52752 requested a review from freehan March 10, 2022 14:13

cezarygerard reviewed Mar 10, 2022

View reviewed changes

kl52752 force-pushed the metrics-l4-rbs branch 2 times, most recently from ec7f4d8 to 4b47b04 Compare March 15, 2022 08:56

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 15, 2022

freehan reviewed Mar 19, 2022

View reviewed changes

Add l4netlb metrics

4a0fc05

kl52752 force-pushed the metrics-l4-rbs branch from 4b47b04 to 4a0fc05 Compare March 21, 2022 08:09

kl52752 requested a review from freehan March 21, 2022 08:09

freehan reviewed Mar 21, 2022

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 21, 2022

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 21, 2022

k8s-ci-robot merged commit cc29101 into kubernetes:master Mar 21, 2022

bowei reviewed Mar 22, 2022

View reviewed changes

kl52752 deleted the metrics-l4-rbs branch March 24, 2022 10:17

kl52752 added a commit to kl52752/ingress-gce that referenced this pull request Mar 24, 2022

Review fixes

269316e

This change included fixes for comments for closed PRs. kubernetes#1675 kubernetes#1678

kl52752 added a commit to kl52752/ingress-gce that referenced this pull request Mar 24, 2022

Review fixes

24f3085

This change included fixes for comments for closed PRs. kubernetes#1675 kubernetes#1678

kl52752 added a commit to kl52752/ingress-gce that referenced this pull request Mar 24, 2022

Review fixes

2d81e48

This change included fixes for comments for closed PRs. kubernetes#1675 kubernetes#1678

kl52752 mentioned this pull request Mar 24, 2022

Review fixes #1685

Merged

kl52752 added a commit to kl52752/ingress-gce that referenced this pull request Mar 25, 2022

Review fixes

f5541e3

This change included fixes for comments for closed PRs. kubernetes#1675 kubernetes#1678

kl52752 added a commit to kl52752/ingress-gce that referenced this pull request Mar 25, 2022

Review fixes

cf91e35

This change included fixes for comments for closed PRs. kubernetes#1675 kubernetes#1678

kl52752 added a commit to kl52752/ingress-gce that referenced this pull request Apr 14, 2022

Review fixes

4cd7bbc

This change included fixes for comments for closed PRs. kubernetes#1675 kubernetes#1678

kl52752 added a commit to kl52752/ingress-gce that referenced this pull request Apr 21, 2022

Review fixes

6a5fcbe

This change included fixes for comments for closed PRs. kubernetes#1675 kubernetes#1678

Add l4netlb metrics #1678

Add l4netlb metrics #1678

Conversation

kl52752 commented Feb 24, 2022

k8s-ci-robot commented Feb 24, 2022

kl52752 commented Feb 24, 2022

freehan commented Mar 2, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kl52752 Mar 15, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cezarygerard left a comment

Choose a reason for hiding this comment

kl52752 commented Mar 15, 2022

cezarygerard commented Mar 15, 2022

k8s-ci-robot commented Mar 15, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freehan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freehan left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Mar 21, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kl52752 Mar 15, 2022 •

edited

Loading