Fix flaky tests

[petrutlucian94]

This PR fixes a few integration tests that occasionally fail:

test_skip_services_stop_on_remove

The test doesn't wait for all nodes to become ready, however it removes one out of 3 nodes and expects 2 nodes to be ready.

Sometimes this assertion fails as one of the nodes never actually gets to "ready" state, so there remains just one ready node instead of two.

We'll fix this flaky test by first ensuring that all cluster nodes are ready before proceeding to remove nodes.

node removal fails immediately after bootstraping the cluster

k8sd sometimes fails when requested to remove nodes immediately after bootstrapping the cluster. It seems that it takes a little
longer for trust store changes to be propagated to all nodes, which should probably be fixed on the microcluster side.

For now, we'll perform some retries.

failed to POST /k8sd/cluster/remove: failed to delete cluster member
k8s-integration-c1aee0-2: No truststore entry found for node with name
"k8s-integration-c1aee0-2"

While at it, we're adding some flake8 rules to the ignore list since the linter cannot properly handle formatted strings.

reduce the scope of session_instance

session_instance is a fixture that allows sharing the same k8s environment for multiple tests, aiming to reduce test duration.

The problem is that some tests interfere with each other and fail.

We could try to improve the test cleanup, ensuring that the shared environment is brought back to a "clean" state when the test finishes, however we'd most probably continue to have flaky tests.

To address this, we're reducing the scope of session_instance to function scope. Since it provides a single node cluster with all features enabled, we'll rename it to "aio_instance".

Isolating the tests may also allow us to run them in parallel, which should reduce the overall test duration (note that there are some concerns around Cilium though).

leaked disk loopback devices

The tests currently leak PVC loopback devices and eventually fail when there are no available loopback devices. We'll ensure that the loopback devices are cleaned up when removing the snap. At the same time, we'll update the test fixture to remove the snap before cleaning up the instance.

Trivy failures due to API rate imiting

The Trivy jobs often fail, being unable to download images due to rate limiting. To solve this, we'll use a different registry.

Leaked containerd mounts prevent the k8s-snap from being removed

We're updating the k8s-snap remove hook to clean up remaining containerd mounts.

Increase timeouts

In some cases, the k8s nodes do not come online in time, for which reason we'll increase the timeouts.

shim processes leaked in "strict" mode

In "strict" mode, the snap remove hook is unable to:

• terminate processes
• remove network namespaces
• list mounts

This happens despite having the necessary snap plugs, which indicates a possible snap issue.

The netns assertions are already commented out. For the time being, we'll also comment out the code that checks if the shim processes
where terminated since this is expected to fail in "strict".

This issue came up as we started removing the k8s snap after every test.

[petrutlucian94]

After dropping the session scoped instance, the Github Action workers are running out of disk space.

[claudiubelu]

After dropping the session scoped instance, the Github Action workers are running out of disk space.

GitHub action runners have plenty of space, though they are a bit bloated with tools that may be useful in the most common scenarios. It makes sense to have them there overall, they won't have to be installed over and over, but it is detrimental in cases such as this.

However, there is a solution for such cases: remove the bloat: https://github.com/canonical/k8s-workflows/blob/main/.github/workflows/build_rocks.yaml#L200-L209. I've used something like this as well, you can easily free 20-30GB. If you need more, the action has more options.

[bschimke95]

LGTM, thanks!

[petrutlucian94]

There might be some problems with the k8s snap removal on "strict". It times out while waiting for some processes to stop and I see that the k8s services are still running.

There's an inline note on this along with some checks that are commented out:

k8s-snap/tests/integration/tests/test_cleanup.py

Lines 22 to 55 in 851387c

	LOG.info("Waiting for shims to go away...")
	util.stubbornly(retries=5, delay_s=5).on(instance).until(
	lambda p: all(
	x not in p.stdout.decode()
	for x in ["containerd-shim", "cilium", "coredns", "/pause"]
	)
	).exec(["ps", "-fea"])
	LOG.info("Waiting for kubelet and containerd mounts to go away...")
	util.stubbornly(retries=5, delay_s=5).on(instance).until(
	lambda p: all(
	x not in p.stdout.decode()
	for x in ["/var/lib/kubelet/pods", "/run/containerd/io.containerd"]
	)
	).exec(["mount"])
	# NOTE(neoaggelos): Temporarily disable this as it fails on strict.
	# For details, `snap changes` then `snap change $remove_k8s_snap_change`.
	# Example output follows:
	#
	# 2024-02-23T14:10:42Z ERROR ignoring failure in hook "remove":
	# -----
	# ...
	# ip netns delete cni-UUID1
	# Cannot remove namespace file "/run/netns/cni-UUID1": Device or resource busy
	# ip netns delete cni-UUID2
	# Cannot remove namespace file "/run/netns/cni-UUID2": Device or resource busy
	# ip netns delete cni-UUID3
	# Cannot remove namespace file "/run/netns/cni-UUID3": Device or resource busy
	# LOG.info("Waiting for CNI network namespaces to go away...")
	# util.stubbornly(retries=5, delay_s=5).on(instance).until(
	# lambda p: "cni-" not in p.stdout.decode()
	# ).exec(["ip", "netns", "list"])

.

[petrutlucian94]

We still have tests that time out waiting for k8s nodes to be ready even after doubling the timeouts, I'll have to take a closer look.

INFO     test_util.util:util.py:66 Attempt 59/60 failed. Error: Failed to meet condition
INFO     test_util.util:util.py:69 Retrying in 5 seconds...
DEBUG    test_util.harness.lxd:lxd.py:220 Execute command ['k8s', 'kubectl', 'get', 'node', 'k8s-integration-8358f1-13', '--no-headers'] in instance k8s-integration-863dd6-12
DEBUG    test_util.util:util.py:36 Execute command lxc shell k8s-integration-863dd6-12 -- bash -c 'k8s kubectl get node k8s-integration-8358f1-13 --no-headers' (kwargs={'capture_output': True, 'check': True})

[berkayoz]

LGTM