✨ Support adding custom secondary VPC CIDR blocks in AWSCluster

[AndiDog]

What type of PR is this?

/kind feature

What this PR does / why we need it:

For Cilium in AWS ENI mode, or other CNIs that can use a separate CIDR block e.g. for pod IPs, we need to support creating those CIDRs like we do for EKS.

I kept IPv6 out for now since most users won't be able to use it, but kept the structure field naming flexible to accommodate for it later.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):

n/a

Checklist:

• squashed commits
• includes documentation
• includes emojis
• adds unit tests
• adds or updates e2e tests

Release note:

Support adding custom secondary VPC CIDR blocks in `AWSCluster`

[AndiDog]

/test pull-cluster-api-provider-aws-e2e
/test pull-cluster-api-provider-aws-e2e-eks

[nrb]

/test pull-cluster-api-provider-aws-e2e

[AndiDog]

Marking as draft again since there seem to be some implications and assumptions regarding subnets that CAPA has. I'll first dig into that and update the PR.

[AndiDog]

Working now. I included and extended the fix https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/4800/files since CAPA isn't supposed to choose any secondary CIDR subnet for creating nodes.

[nrb]

/lgtm

The verify job just needs a make generate run, and the API diff doesn't look concerning.

[AndiDog]

I ran make generate and my commit is up to date with that. But it seems the YAML output is slightly different on my machine compared to CI. I'll check the tool versions and update here.

[AndiDog]

@nrb Looks fine now after I rebased onto main

[nrb]

/lgtm

[AndiDog]

I added the two permissions

ec2:AssociateVpcCidrBlock
ec2:DisassociateVpcCidrBlock

to func (t Template) ControllersPolicy() *iamv1.PolicyDocument because they were only listed for EKS until now.

[AndiDog]

make test works for me locally, let's try again

/test pull-cluster-api-provider-aws-test

[cnmcavoy]

/lgtm

[AndiDog]

Rebased. @cnmcavoy can you take a look again (was already LGTM)? And @Ankitasw @richardcase I need an approval as well.

[cnmcavoy]

/lgtm

[AndiDog]

I had to rebase this once again due to the new field SubnetSchema which gave easily resolvable merge conflicts.

@richardcase @Ankitasw do you have time to look into this PR? It was already reviewed, but not approved.

[nrb]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nrb

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [nrb]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment