Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expand retry conditions for K8 logs #818

Merged
merged 1 commit into from
Aug 28, 2023

Conversation

AaronH88
Copy link
Contributor

Fixes: ansible/awx#14293
Expanded retry conditions for K8 logs to retry at least 5 times regardless of the error returned. This is due to K8 api timings where the GOAWAY error may not be sent or sent already before log streams get attached.

@AaronH88 AaronH88 requested a review from TheRealHaoLiu August 15, 2023 15:53
@AaronH88 AaronH88 force-pushed the goaway_catch_update branch from 625a5d2 to 5eb8c0f Compare August 16, 2023 10:53
@AaronH88 AaronH88 force-pushed the goaway_catch_update branch 2 times, most recently from 46cabf7 to 2df4f86 Compare August 22, 2023 10:03
@AaronH88 AaronH88 force-pushed the goaway_catch_update branch from 2df4f86 to 978f1d6 Compare August 22, 2023 15:43
@sonarcloud
Copy link

sonarcloud bot commented Aug 22, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@@ -275,7 +255,7 @@ func (kw *kubeUnit) kubeLoggingWithReconnect(streamWait *sync.WaitGroup, stdout
break
}
}
if err != nil {
if err != nil && remainingRetries > 0 {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think the additional conditional && remainingRetries > 0 is needed here

I think the only way that the code reach this point is if we exhausted the retries?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fosterseth am i right?

Copy link
Member

@fosterseth fosterseth Aug 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agreed. I don't think this conditional will ever hit

Copy link
Contributor Author

@AaronH88 AaronH88 Aug 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed, i think Seth is right, now with the changes this condition will never be hit, the first err != nil will be triggered and this will never be called.

I have removed the whole check.

@AaronH88 AaronH88 force-pushed the goaway_catch_update branch 2 times, most recently from 9591f7e to b6c9e6c Compare August 23, 2023 08:41
Comment on lines 279 to 282
*stdoutErr = err
kw.Error("Error reading from pod %s/%s: %s", podNamespace, podName, err)

return
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these code will still need to be executed when the retries is exhausted

@AaronH88 AaronH88 force-pushed the goaway_catch_update branch 3 times, most recently from 9c83f22 to 4295e36 Compare August 28, 2023 09:41
@AaronH88 AaronH88 force-pushed the goaway_catch_update branch from 4295e36 to a931ac4 Compare August 28, 2023 10:35
@AaronH88 AaronH88 merged commit 93d2e12 into ansible:devel Aug 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

AWX jobs can't tolerate the EKS scale out
5 participants