fix(reaper): refactor to allow retries and fix races #2728
Conversation
✅ Deploy Preview for testcontainers-go ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
|
Depends on #2738 |
stevenh
force-pushed
the
fix/reaper-retries-race
branch
from
0dc52af
to
0b62224
Compare
mdelapenya
force-pushed
the
fix/reaper-retries-race
branch
from
d3c9c9a
to
0589878
Compare
|
@mdelapenya reminder this is dependent on #2728 which is why it has missing components still. I'll rebase once #2738 is merged. |
stevenh
force-pushed
the
fix/reaper-retries-race
branch
from
e7b948d
to
5de803c
Compare
|
Rebased and picking up again, hopefully wont require many fixes. Will convert from draft when ready. |
stevenh
force-pushed
the
fix/reaper-retries-race
branch
3 times, most recently
from
bf961f0
to
7649e0c
Compare
There was a problem hiding this comment.
I only added a few super minor comments, other than that, my only concern is about the implications of these changes with the reuse mode we are proposing in #2768
I'm fine merging this, although I'd like to discuss with you about reuse more in depth, as it could apply to the reaper code as well.
stevenh
force-pushed
the
fix/reaper-retries-race
branch
from
7649e0c
to
4182991
Compare
|
@stevenh is there anything I can do to move forward with this PR? |
stevenh
force-pushed
the
fix/reaper-retries-race
branch
from
4182991
to
7bc2a87
Compare
|
Blocked on #2786 |
Refactor how the reaper is created to allow for proper retries of temporary errors such as container not found issues during startup or shutdown races. This eliminates the use of sync.Once which wasn't solving the problem at hand and replaces it with a singleton spawner with locked access. Wrap reaper errors so we can determine the cause of failures more easily. Fix race condition in port wait when loading from container by always waiting for the port first. Remove unnecessary use of buffering and invalid retry logic in reaper connection handling which could never recover correctly from a partial read. Move the reaper creation just before connections are established in compose to ensure its still running when the Connect calls are made. Previously the reaper was requested in NewDockerComposeWith which means it could have already shutdown before connections are made during the later sections of Up if the startup took over 1 minute. This was causing consistent failures for: TestDockerComposeAPIWithWaitLogStrategy Ensure that resource labels are correct so that resources aren't reaped when the reaper is disabled by excluding session id when reaper is disabled. Error when creating a reaper when the config says it's disabled so that we avoid hard to debug issues because a reaper is running when it shouldn't be.
stevenh
force-pushed
the
fix/reaper-retries-race
branch
from
7bc2a87
to
12c4483
Compare
| func GenericLabels() map[string]string { | ||
| return core.DefaultLabels(core.SessionID()) | ||
| } | ||
|
|
||
| // AddGenericLabels adds the generic labels to target. | ||
| func AddGenericLabels(target map[string]string) { |
There was a problem hiding this comment.
I think we can leverage the core.MergeCustomLabels(dst, src) function addedin #2775
| } | ||
|
|
||
| // AddDefaultLabels adds the default labels for sessionID to target. | ||
| func AddDefaultLabels(sessionID string, target map[string]string) { |
There was a problem hiding this comment.
Same here: #2775 added a very similar method to merge dst and src labels
| if !config.Read().RyukDisabled { | ||
| labels[LabelSessionID] = sessionID |
There was a problem hiding this comment.
I'd not remove the session ID, as this label could be used in the backend to connect test sessions with remote VM executions.
Refactor how the reaper is created to allow for proper retries of temporary errors such as container not found issues during startup or shutdown races.
This eliminates the use of sync.Once which wasn't solving the problem at hand and replaces it with a singleton spawner with locked access.
Wrap reaper errors so we can determine the cause of failures more easily.
Fix race condition in port wait when loading from container by always waiting for the port first.
Remove unnecessary use of buffering and invalid retry logic in reaper connection handling which could never recover correctly from a partial read.
Move the reaper creation just before connections are established in compose to ensure its still running when the Connect calls are made.
Previously the reaper was requested in NewDockerComposeWith which means it could have already shutdown before connections are made during the later sections of Up if the startup took over 1 minute.
This was causing consistent failures for:
TestDockerComposeAPIWithWaitLogStrategy
Ensure that resource labels are correct so that resources aren't reaped when the reaper is disabled by excluding session id when reaper is disabled.
Error when creating a reaper when the config says it's disabled so that we avoid hard to debug issues because a reaper is running when it shouldn't be.