Error log when running kuma multizone with 2 instances

[lukidzi]

What happened?

When running kuma multizone deployment with 2 instances on global you can notice an error log every 5min and 10 seconds. It is probably because of a timeout on the connection. We should check it and try to figure out if we can fix it.

ERROR	intercp.catalog.heartbeat	could not heartbeat the leader	{"error": "could not send a heartbeat to a leader: rpc error: code = Canceled desc = grpc: the client connection is closing", "errorVerbose": "rpc error: code = Canceled desc = grpc: the client connection is closing\ncould not send a heartbeat to a leader\ngh.neting.cc/kumahq/kuma/pkg/intercp/catalog.(*heartbeatComponent).heartbeat\n\tgh.neting.cc/kumahq/kuma@v0.0.0-20230127105205-446a49676bf8/pkg/intercp/catalog/heartbeat_component.go:90\ngh.neting.cc/kumahq/kuma/pkg/intercp/catalog.(*heartbeatComponent).Start\n\tgh.neting.cc/kumahq/kuma@v0.0.0-20230127105205-446a49676bf8/pkg/intercp/catalog/heartbeat_component.go:57\ngh.neting.cc/kumahq/kuma/pkg/plugins/bootstrap/k8s.componentRunnableAdaptor.Start\n\tgh.neting.cc/kumahq/kuma@v0.0.0-20230127105205-446a49676bf8/pkg/plugins/bootstrap/k8s/plugin.go:214\nsigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1\n\tsigs.k8s.io/controller-runtime@v0.14.1/pkg/manager/runnable_group.go:219\nruntime.goexit\n\truntime/asm_amd64.s:1571"}

2023-01-30T14:02:08.919Z	INFO	intercp.client.pool	closing connection due to lack of activity	{"url": "grpcs://ADDRESS_IP:5683"}

[jakubdyszkiewicz]

Triage: visible on Global CP on GKE

[github-actions]

This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed.
If you think this issue is still relevant, please comment on it or attend the next triage meeting.

[lahabana]

Highly related a user has reported seeing:

ERROR	intercp.catalog.heartbeat	could not heartbeat the leader	{"error": "could not send a heartbeat to a leader: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.18.14.4:5683: i/o timeout\

It may be related to timeout yes. Here are some timestamps => every 20s
2023-06-01T15:30:06.424Z
2023-06-01T15:30:26.426Z
2023-06-01T15:30:46.428Z
2023-06-01T15:31:06.429Z
2023-06-01T15:31:26.430Z

[lahabana]

This was fixed recently with #7323