fix(xds): make sure ADS are ordered

[jakubdyszkiewicz]

Checklist prior to review

Use ordered ADS. We noticed that in edge case we can hit out of order EDS which results in cluster being never updated.

This PR updates go-control-plane to the newest version and enables ordered ADS functionality. At this moment we cannot rely on upstream go-control-plane because

• our fork has a fix for KDS Option for the delta server to override the channel capacity and avoid deadlocks go-control-plane#3
• as well as fix for ordered ADS avoid write to close channel go-control-plane#5

We now can ditch "resource warming forcer". Because of ordered responses, we can just bump the EDS version whenever we update CDS, go-control-plane will handle the rest.
Forcer might have been a culprit here as well. After we just used a new go-control-plane and kept the forcer, we hit one CI fail with warming clusters. After we got rid of it, 3 full runs passed.

TODO: Remember to also change the replace in Kong Mesh after merge.

• Link to relevant issue as well as docs and UI issues --
• This will not break child repos: it doesn't hardcode values (.e.g "kumahq" as a image registry) and it will work on Windows, system specific functions like syscall.Mkfifo have equivalent implementation on the other OS --
• Tests (Unit test, E2E tests, manual test on universal and k8s) --
  • Don't forget ci/ labels to run additional/fewer tests
• Do you need to update UPGRADE.md? --
• Does it need to be backported according to the backporting policy? (this GH action will add "backport" label based on these file globs, if you want to prevent it from adding the "backport" label use no-backport-autolabel label) --

[jakubdyszkiewicz]

on universal suite

2024-09-26T13:16:04.464Z	INFO	xds.dp-lifecycle	waiting for deregister proxy	{"proxyType": "dataplane", "proxyKey": {"Mesh":"dp-auth","Name":"dp-01"}, "streamID": 108, "waitFor": "0s"}
2024-09-26T13:16:04.464Z	INFO	xds.dp-lifecycle	deregister proxy	{"proxyType": "dataplane", "proxyKey": {"Mesh":"dp-auth","Name":"dp-01"}, "streamID": 108}
2024-09-26T13:16:04.475Z	INFO	xds.reconcile	config has changed	{"proxyName": "egress", "mesh": "", "versions": ["78e4d42f-5c92-426e-8b37-2302991502a0", "5cdbd604-2150-4c9f-a582-0d7ec8683129"]}
2024-09-26T13:16:04.591Z	INFO	xds.status-tracker	proxy disconnected	{"streamID": 111, "proxyName": "", "mesh": "", "subscriptionID": "bdc84fa3-ffc6-4411-9f95-f62c47e37165"}
panic: send on closed channel

goroutine 10572 [running]:
github.com/envoyproxy/go-control-plane/pkg/server/sotw/v3.(*server).process.func1()
	github.com/envoyproxy/go-control-plane@v0.12.0/pkg/server/sotw/v3/xds.go:100 +0x1b
created by github.com/envoyproxy/go-control-plane/pkg/server/sotw/v3.(*server).process in goroutine 10568
	github.com/envoyproxy/go-control-plane@v0.12.0/pkg/server/sotw/v3/xds.go:99 +0xaff

stderr:

edit: fixed in our fork of go control plane

[jakubdyszkiewicz]

While this can fix a potential Cluster stuck in warming state, we are not sure yet if we want to backport it. Aside of ordered ADS it includes all the changes of upgraded go-control-plane. We want to first try it more and then potentially backport to all supported versions.