Possible failure to suspend instances on license expired

[hardillb]

Current Behavior

User had trial EE license, went on vacation and license expired and instances marked as suspended.
on return applied new EE trial license, but could not restart existing instances.

Investigation at the k8s level showed Deployments/Service/Ingress still existed (but DB showed suspended).

Failed to start, because entries already exist.

Expected Behavior

Instances should actually be suspended and all K8s entries removed.

Steps To Reproduce

• Deploy with EE license
• Start instances
• Expire license
• Add new License
• Try and restart existing instances

Environment

• FlowFuse version: 2.8.0
• Node.js version: 18.x
• npm version: 9.x
• Platform/OS: K8s
• Browser: NA

Have you provided an initial effort estimate for this issue?

I have provided an initial effort estimate

[hardillb]

I've added more debug (#4773) to this, but it's going to be hard to test.

Best option I can think is to spin up a test instance on k8s and give it a license with only a day's life and then let it expire.

Will try and find a machine I can run single node k8s cluster on to test