update post-render to retry updating service account on error #269
Conversation
|
I'm not a fan of naive retry loops but I couldn't find any way to check if KIND was done updating the SA before we try to add our keys and label. Any thoughts? @RothAndrew @jeff-mccoy ? |
|
It's definitely a bandaid fix, but I don't have any objections at this point in time. |
|
I'll take a look in a couple hours if that's okay--I might have an idea. |
|
@YrrepNoj I am unable to reproduce the original issue so having a hard time validating this. When you get a chance could you try this instead? This is how the K8s E2E tests do it and I think is a bit cleaner than a retry loop. Thanks! |
|
@jeff-mccoy It doesn't seem to be happening nearly as often but I got the error message after about 10-15 attempts My assumption is that KIND will create a blank/minimalistic ServiceAccount and then create a background job to update it with some metadata or something. Since this package is a lot more lightweight than any of the other packages I guess it just happens to run into this issue. I got this result on a Ubuntu 18.04 machine with an Intel CPU. Zarf binary was built with version tag |
|
Sorry which package is doing this? |
|
data-injection |
RothAndrew
force-pushed
the
multi-distro-support
branch
from
614b4d3 to
4c802d3
Compare
jeff-mccoy
force-pushed
the
multi-distro-support
branch
2 times, most recently
from
614b4d3 to
53101f9
Compare
We experienced a race condition when updating the ServiceAccount for a newly created namespace. Sometimes KIND distro would error because it was still updating the SA at the same time we were.
YrrepNoj
force-pushed
the
retry-updating-service-account
branch
from
58c7321 to
f51f413
Compare
|
@jeff-mccoy all cleaned up ;) |
We experienced a race condition when updating the ServiceAccount for a newly created namespace. Sometimes KIND distro would error because it was still updating the SA at the same time we were.
We experienced a race condition when updating the ServiceAccount for a newly created namespace. Sometimes KIND distro would error because it was still updating the SA at the same time we were. Signed-off-by: Jeff McCoy <code@jeffm.us>
We experienced a race condition when updating the ServiceAccount for a newly created namespace. Sometimes KIND distro would error because it was still updating the SA at the same time we were. Signed-off-by: Jeff McCoy <code@jeffm.us>
We experienced a race condition when updating the ServiceAccount for a newly created namespace.
Sometimes KIND distro would error because it was still updating the SA at the same time we were.
Without this change we were sometimes seeing the following error on KIND clusters:
DEBUG unable to update the default service account for the demo namespace: Operation cannot be fulfilled on serviceaccounts "default": the object has been modified; please apply your changes to the latest version and try againI've tested this several times (more than 5..) locally (macOS and Linux) and have not run into the ImagePullBackoff because of an un-updated ServiceAccount.