KDS extra errors on fresh startup

[lahabana]

Description

When you connect a zone and there's already global state there's a KDS NACK because we try to create the resources before the mesh.

This is not an actual problem as it's retried shortly after but creates noisy logs.

2023-12-01T08:50:14.498Z	INFO	kds-delta-zone	creating a new resource from upstream	{"type": "Secret", "name": "external-service-locality-lb-no-egress.ca-builtin-key-ca-1", "mesh": "external-service-locality-lb-no-egress"}
2023-12-01T08:50:14.607Z	INFO	kds-delta-zone	creating a new resource from upstream	{"type": "Secret", "name": "dataplane-token-signing-key-external-service-locality-lb-no-egress-1", "mesh": "external-service-locality-lb-no-egress"}
2023-12-01T08:50:14.614Z	INFO	kds-delta-zone	creating a new resource from upstream	{"type": "Secret", "name": "external-service-locality-lb.ca-builtin-cert-ca-1", "mesh": "external-service-locality-lb"}
2023-12-01T08:50:14.618Z	INFO	kds-delta-zone	creating a new resource from upstream	{"type": "Secret", "name": "external-service-locality-lb.ca-builtin-key-ca-1", "mesh": "external-service-locality-lb"}
2023-12-01T08:50:14.622Z	INFO	kds-delta-zone	creating a new resource from upstream	{"type": "Secret", "name": "dataplane-token-signing-key-external-service-locality-lb-1", "mesh": "external-service-locality-lb"}
2023-12-01T08:50:14.626Z	INFO	kds-delta-zone	creating a new resource from upstream	{"type": "Secret", "name": "external-service-locality-lb-no-egress.ca-builtin-cert-ca-1", "mesh": "external-service-locality-lb-no-egress"}
2023-12-01T08:50:14.630Z	INFO	kds-delta-zone	creating a new resource from upstream	{"type": "Retry", "name": "retry-all-external-service-locality-lb-no-egress-48cc2dbcb5d4456v", "mesh": "external-service-locality-lb-no-egress"}
2023-12-01T08:50:14.633Z	INFO	kds-delta-zone	error during callback received, sending NACK	{"kds-version": "v2", "err": "failed to create k8s resource: admission webhook \"owner-reference.kuma-admission.kuma.io\" denied the request: Mesh.kuma.io \"external-service-locality-lb-no-egress\" not found", "errVerbose": "admission webhook \"owner-reference.kuma-admission.kuma.io\" denied the request: Mesh.kuma.io \"external-service-locality-lb-no-egress\" not found\nfailed to create k8s resource\ngithub.com/kumahq/kuma/pkg/plugins/resources/k8s.(*KubernetesStore).Create\n\tgithub.com/kumahq/kuma/pkg/plugins/resources/k8s/store.go:76\ngithub.com/kumahq/kuma/pkg/core/resources/store.(*paginationStore).Create\n\tgithub.com/kumahq/kuma/pkg/core/resources/store/pagination_store.go:30\ngithub.com/kumahq/kuma/pkg/metrics/store.(*MeteredStore).Create\n\tgithub.com/kumahq/kuma/pkg/metrics/store/store.go:44\ngithub.com/kumahq/kuma/pkg/core/resources/store.(*customizableResourceStore).Create\n\tgithub.com/kumahq/kuma/pkg/core/resources/store/customizable_store.go:44\ngithub.com/kumahq/kuma/pkg/kds/v2/store.(*syncResourceStore).Sync.func2\n\tgithub.com/kumahq/kuma/pkg/kds/v2/store/sync.go:219\ngithub.com/kumahq/kuma/pkg/core/resources/store.InTx\n\tgithub.com/kumahq/kuma/pkg/core/resources/store/transactions.go:46\ngithub.com/kumahq/kuma/pkg/kds/v2/store.(*syncResourceStore).Sync\n\tgithub.com/kumahq/kuma/pkg/kds/v2/store/sync.go:196\ngithub.com/kumahq/kuma/pkg/kds/zone.Setup.func2.ZoneSyncCallback.func3\n\tgithub.com/kumahq/kuma/pkg/kds/v2/store/sync.go:304\ngithub.com/kumahq/kuma/pkg/kds/v2/client.(*kdsSyncClient).Receive\n\tgithub.com/kumahq/kuma/pkg/kds/v2/client/kds_client.go:90\ngithub.com/kumahq/kuma/pkg/kds/zone.Setup.func2.1\n\tgithub.com/kumahq/kuma/pkg/kds/zone/components.go:137\nruntime.goexit\n\truntime/asm_arm64.s:1197"}
2023-12-01T08:50:14.633Z	INFO	kds-delta-zone	creating a new resource from upstream	{"type": "Mesh", "name": "external-service-locality-lb", "mesh": ""}
2023-12-01T08:50:14.639Z	INFO	kds-delta-zone	creating a new resource from upstream	{"type": "Mesh", "name": "external-service-locality-lb-no-egress", "mesh": ""}
2023-12-01T08:50:14.643Z	INFO	kds-delta-zone	creating a new resource from upstream	{"type": "TrafficRoute", "name": "route-all-external-service-locality-lb-no-egress-xcx75fx9424v42fw", "mesh": "external-service-locality-lb-no-egress"}
2023-12-01T08:50:14.647Z	INFO	kds-delta-zone	creating a new resource from upstream	{"type": "TrafficRoute", "name": "route-all-external-service-locality-lb-zc22v4wb2252c44w", "mesh": "external-service-locality-lb"}
2023-12-01T08:50:14.651Z	INFO	kds-delta-zone	creating a new resource from upstream	{"type": "TrafficPermission", "name": "allow-all-external-service-locality-lb-wf6xzz4bfcb68dbc", "mesh": "external-service-locality-lb"}
2023-12-01T08:50:14.654Z	INFO	kds-delta-zone	creating a new resource from upstream	{"type": "TrafficPermission", "name": "allow-all-external-service-locality-lb-no-egress-25wd488w2xdw5444", "mesh": "external-service-locality-lb-no-egress"}
2023-12-01T08:50:14.657Z	INFO	kds-delta-zone	creating a new resource from upstream	{"type": "Timeout", "name": "timeout-all-external-service-locality-lb-7x44zxfv7d84c4b5", "mesh": "external-service-locality-lb"}
2023-12-01T08:50:14.660Z	INFO	kds-delta-zone	creating a new resource from upstream	{"type": "Timeout", "name": "timeout-all-external-service-locality-lb-no-egress-f44757zzdcfzwx2f", "mesh": "external-service-locality-lb-no-egress"}
2023-12-01T08:50:14.663Z	INFO	kds-delta-zone	creating a new resource from upstream	{"type": "CircuitBreaker", "name": "circuit-breaker-all-external-service-locality-lb-no-egress-zb9c99xd4wx95622", "mesh": "external-service-locality-lb-no-egress"}
2023-12-01T08:50:14.666Z	INFO	kds-delta-zone	creating a new resource from upstream	{"type": "CircuitBreaker", "name": "circuit-breaker-all-external-service-locality-lb-ff2dzzffdx52w7wv", "mesh": "external-service-locality-lb"}
2023-12-01T08:50:14.669Z	INFO	kds-delta-zone	creating a new resource from upstream	{"type": "Retry", "name": "retry-all-external-service-locality-lb-no-egress-48cc2dbcb5d4456v", "mesh": "external-service-locality-lb-no-egress"}
2023-12-01T08:50:14.673Z	INFO	kds-delta-zone	creating a new resource from upstream	{"type": "Retry", "name": "retry-all-external-service-locality-lb-2b777wdbzbcv2722", "mesh": "external-service-locality-lb"}
2023-12-01T08:50:14.743Z	INFO	xds.secrets	generating certificate	{"ZoneEgress": {"Mesh":"external-service-locality-lb","Name":"kuma-egress-79877697cd-jm4kd.kuma-system"}, "reason": "mTLS is enabled and DP hasn't received a certificate yet"}

[jakubdyszkiewicz]

Triage: if it's impossible to fix, we can at least add more context into a log itself.

[github-actions]

This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed.
If you think this issue is still relevant, please comment on it or attend the next triage meeting.

[lahabana]

@lobkovilya there is some level of ordering in go-control-plane now right?

[lobkovilya]

yes, there is some kind of ordering of envoy resources, but there is no support for custom resources https://github.com/envoyproxy/go-control-plane/blob/main/pkg/cache/types/types.go#L45

[lahabana]

Does that work in DeltaXDS too @lobkovilya ?

[lobkovilya]

Yes, implemented here envoyproxy/go-control-plane#752

[lahabana]

Ok IIRC @jakubdyszkiewicz argued this can't work exactly well because of the following reasons:

1. The order needs to be reversed for deletion
2. This only sends things in a specific order, it doesn't wait for ack of the first resources before sending the rest. So it may reduce the occurence but it won't remove the error completely.

In any case we discussed having the list of resource types be overridable in such use cases. I believe that'd be quite good.

[github-actions]

This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed.
If you think this issue is still relevant, please comment on it or attend the next triage meeting.

[github-actions]

This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed.
If you think this issue is still relevant, please comment on it or attend the next triage meeting.