feat: shutdown race resilience

[stevenh]

A significant rewrite to ensure that we don't suffer from shutdown race conditions as the prune condition is met and additional resources are being created.

Previously this would remove resources that were still in use, now we retry if we detect new resources have been created within a window of the prune condition triggering.

This supports the following new environment configuration settings:

• RYUK_REMOVE_RETRIES - The number of times to retry removing a resource.
• RYUK_REQUEST_TIMEOUT - The timeout for any Docker requests.
• RYUK_RETRY_OFFSET - The offset added to the start time of the prune pass that is used as the minimum resource creation time

[mdelapenya]

For some reason, I need to set up the loadConfig tests with RYUK_VERBOSE=false in order to make the tests pass locally:

tests := map[string]struct {
		setEnv   func(*testing.T)
		expected config
	}{
		"defaults": {
+			setEnv: func(t *testing.T) {
+				t.Helper()
+				t.Setenv("RYUK_VERBOSE", "false")
+			},
			expected: config{
				ConnectionTimeout:   time.Minute,
				Port:                8080,
				ReconnectionTimeout: time.Second * 10,
				RemoveRetries:       10,
				RequestTimeout:      time.Second * 10,
				RetryOffset:         -time.Second,
				ShutdownTimeout:     time.Minute * 10,
			},
		},
		"custom": {
			setEnv: func(t *testing.T) {
				t.Helper()
				t.Setenv("RYUK_PORT", "1234")
				t.Setenv("RYUK_CONNECTION_TIMEOUT", "2s")
				t.Setenv("RYUK_RECONNECTION_TIMEOUT", "3s")
				t.Setenv("RYUK_REQUEST_TIMEOUT", "4s")
				t.Setenv("RYUK_REMOVE_RETRIES", "5")
				t.Setenv("RYUK_RETRY_OFFSET", "-6s")
				t.Setenv("RYUK_SHUTDOWN_TIMEOUT", "7s")
+				t.Setenv("RYUK_VERBOSE", "false")
			},
			expected: config{
				Port:                1234,
				ConnectionTimeout:   time.Second * 2,
				ReconnectionTimeout: time.Second * 3,
				RequestTimeout:      time.Second * 4,
				RemoveRetries:       5,
				RetryOffset:         -time.Second * 6,
				ShutdownTimeout:     time.Second * 7,
			},
		},
	}

A bug in the env library for default booleans?

[mdelapenya]

Thanks for opening this @stevenh. I see a lot of changes here, which I'd suggest splitting in different PRs:

• config refactor
• run function

Ryuk is a critical piece in the Testcontainers ecosystem, so updating it should be done very carefully to not affect all the related projects.

Would you mind separating this big PR into multiple ones? I can help you out with it if needed.

Thanks!

[stevenh]

For some reason, I need to set up the loadConfig tests with RYUK_VERBOSE=false in order to make the tests pass locally:

A bug in the env library for default booleans?

Interesting what's the error you get?

[stevenh]

For some reason, I need to set up the loadConfig tests with RYUK_VERBOSE=false in order to make the tests pass locally:

snip...

A bug in the env library for default booleans?

If you had environment vars set then the test would have failed as it was expecting a clean environment.

I've updated the test to clean the current environment before it starts, which could fix the issue you saw.

[mdelapenya]

Hi @stevenh, thanks for this PR, I've started the review and I see that there are some changes that would deserve its own PR:

• the addition of a library to read the env vars. Is there any other library you considered for that? I like this one is simple and with zero deps.
• the bumps for the different Go versions and tools (golangci-lint)
• the real refactor

Could you separate them into three? I can help out with the two first so you can focus on the real meat here

[stevenh]

Ok so little more involved due to go bump needed new linter so we need to merge in the following order:

1. chore: bump versions #160
2. chore: simply config #159 -- Also includes version bump so it complies
3. feat: shutdown race resilience #141 -- Will need a rebase after the config merge
4. ci: enhanced linting #158 -- Currently erroring, but no point fixing

I didn't consider other env config libraries, as I've used that one a bunch of times and has worked well, but if there is something better happy to swap.

[mdelapenya]

Given #159 has been merged, could you please rebase this one? 🙏

[mdelapenya]

LGTM, thanks!

I like the changes about the vocabulary: I'll miss the ryuk names, although I admit that using the names and words in this PR is now more inclusive and more maintainable for the long term, so thanks for this.

Once merged, as we are building the images for the commit, we can immediately test this image in testcontainers-go, using the commit-sha as docker tag (see https://hub.docker.com/r/testcontainers/ryuk/tags)

[stevenh]

@mdelapenya added some docker cli based tests for container, network, image and volume reaping so should be good to go.

[mdelapenya]

One comment regarding shelling out to the docker binary.

Other than that, will take the final view this weekend for final approval 🙏 I'm doing conferences next week, so I'll try to do before that.

[mdelapenya]

LGTM, great PR, thanks for adding clarity in the reap process.

Cheers!