Replace NLBs on subnets change

[mikkeloscar]

This implements an implicit feature to migrate NLBs when the subnets specified for the NLB changes.

This is done because for NLBs you can't change the subnets once the NLB has been created. This is unlike ALBs where you can freely change the subnets (and wait up to a week for it to take affect 🤷). This enables simple migration between subnets with the assumption that it's no problem to replace an NLB with a new one.

The way this works is that during the sync it checks if the existing managed NLBs have the subnets in sync. If not the NLB stack is ignored and the ingresses that were previously mapped to the NLB stack will now be mapped to a new one. The old NLB stack will become "empty" and after 1 hour (default) it will be cleaned up.

[AlexanderYastrebov]

👍

[mikkeloscar]

Need to properly test it in a cluster before this is ready.

[mikkeloscar]

This change doesn't yet work correctly. The problem is that when subnets are changed then the existing NLB stacks are updated with new subnets which is not possible and leads to those stack going into Rollback, this way they can't be successfully decommissioned.

Solution is to ensure that once an NLB stack is created the subnets are never changed on updates. I will work on this when I find the time 😅