increase nodeadm network resilience

[ndbaker1]

Issue #, if available:

Description of changes:

increase retries and total retry window for imds client

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Testing Done

See this guide for recommended testing for PRs. Some tests may not apply. Completing tests and providing additional validation steps are not required, but it is recommended and may reduce review time and time to merge.

[ndbaker1]

by default imds.New uses the standard retryer but sets MaxBackoff as 1 and does not change the MaxAttempts from 3, so we're just mimicking this and bumping the attempts

[ndbaker1]

/ci

[github-actions]

@ndbaker1 roger that! I've dispatched a workflow. 👍

[github-actions]

@ndbaker1 the workflow that you requested has completed. 🎉

AMI variant	Build	Test
1.23 / al2	success ✅	failure ❌
1.23 / al2023	success ✅	success ✅
1.24 / al2	success ✅	failure ❌
1.25 / al2	success ✅	success ✅
1.25 / al2023	success ✅	success ✅
1.26 / al2	success ✅	success ✅
1.26 / al2023	success ✅	success ✅
1.27 / al2	success ✅	failure ❌
1.27 / al2023	success ✅	success ✅
1.28 / al2	success ✅	success ✅
1.28 / al2023	success ✅	success ✅
1.29 / al2	success ✅	failure ❌
1.29 / al2023	success ✅	success ✅

[ndbaker1]

/ci

[github-actions]

@ndbaker1 roger that! I've dispatched a workflow. 👍

[github-actions]

@ndbaker1 the workflow that you requested has completed. 🎉

AMI variant	Build	Test
1.23 / al2	success ✅	success ✅
1.23 / al2023	success ✅	success ✅
1.24 / al2	success ✅	success ✅
1.24 / al2023	success ✅	success ✅
1.25 / al2	success ✅	success ✅
1.25 / al2023	success ✅	success ✅
1.26 / al2	success ✅	success ✅
1.26 / al2023	success ✅	success ✅
1.27 / al2	success ✅	success ✅
1.27 / al2023	success ✅	success ✅
1.28 / al2	success ✅	success ✅
1.28 / al2023	success ✅	success ✅
1.29 / al2	success ✅	success ✅
1.29 / al2023	success ✅	success ✅

[ndbaker1]

bumped some account limits and reran the ci :)

[cartermckinnon]

I'm curious how much of a bottleneck this will create in the systemd unit tree. Can you grab a before and after?

systemd-analyze plot > tree.svg

[cartermckinnon]

I guess I'm mostly interested in whether cloud-init.service already has some kind of dependency on one of the network targets:

systemctl list-dependencies cloud-init

[ndbaker1]

silly me got the network-online.target swapped with network-online.service 🙃,
getting the data now, will follow up offline

[cartermckinnon]

@Issacwww I don't disagree, we'd have to move the definition for this dimension of the matrix out of the workflow file into a variable passed by the bot. I'll see what I can do 👍

[ndbaker1]

the network-online.target already pulls in cloud-init as a successor, so this introduces some conflicts with the target

$ systemctl show network-online.target -p After
After=network.target systemd-networkd-wait-online.service cloud-init.service

so per the network online docs we should just exercise normal retries. this PR should be more resilient than the existing AL2 retry, which is 10 tries with a linear 1s backoff