latest capa-controller is not working properly with our cluster-eks and spawns infinite number of VPC due some error

[AndiDog]

during network creation it will fail and that cause the process to restart and create VPC again

Probably related:

E1213 13:17:04.251075       1 controller.go:324] "Reconciler error" err="failed to reconcile network for AWSManagedControlPlane org-giantswarm/vac0eks: failed to patch conditions: AWSManagedControlPlane.controlplane.cluster.x-k8s.io \"vac0eks\" is invalid: spec.network.subnets[7]: Duplicate value: map[string]interface {}{\"id\":\"\"}" controller="awsmanagedcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="AWSManagedControlPlane" AWSManagedControlPlane="org-giantswarm/vac0eks" namespace="org-giantswarm" name="vac0eks" reconcileID="12841c2f-7435-4f3b-9106-782a05e97702"

[AndiDog]

The mess on the test AWS account is cleaned up. I don't have an effort estimate yet for CAPA, given how its AWSManagedControlPlane reconciler has zero unit tests covering the Reconcile or reconcileNormal functions and testability (e.g. dependency injection for mockability) must first be added.

[AndiDog]

The bug is very basic: CAPA creates a VPC, fails to store it for whatever reason. On next reconciliation, CAPA pretends it doesn't know anything about the VPC (which it really doesn't without making AWS requests) and happily creates a new one. In our case of repeated errors, this happens again and again. I implemented a basic unit test for EKS, and VPC creation idempotence (applies to EC2 and EKS based clusters alike). Upstream PR coming up soon.

[AndiDog]

Unfortunately, my pending PR kubernetes-sigs/cluster-api-provider-aws#4637 blocks opening the follow-up which adds the EKS unit test.

However I managed to extract the fix for the blatant bug in a small, separate PR kubernetes-sigs/cluster-api-provider-aws#4723 so we can go on nevertheless and fix the terrifying issue.

[AndiDog]

Image pull errors fixed via giantswarm/cluster-api-provider-aws-app#211, so now this should be done (except for developer-reserved MCs where Flux is paused, currently golem).

[AndiDog]

The storage error will be solved via #2870.