-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pipeline controller shouldn't retry creating pod when the error cannot be mitigated by retry #4092
Comments
hey @jialindai !
I'm wondering, how would the pipeline controller know if that situation couldn't be mitigated? i.e. what if more quota became available in the namespace later? It sounds like that won't happen in your case, but I think it could for someone else (if I'm wrong maybe you can provide some more details about your setup - e.g. is there some way to conclusively know that quota won't be available?) You might also fine #734 interesting, which is all about scheduling in resource constrained environments - in that case we intentionally retry and backoff, waiting until resources are available. For your specific case, it might make sense for you to create a controller (or maybe even cron?) which observes PipelineRuns in the state you are describing and cancels them. |
Issues go stale after 90d of inactivity. /lifecycle stale Send feedback to tektoncd/plumbing. |
Stale issues rot after 30d of inactivity. /lifecycle rotten Send feedback to tektoncd/plumbing. |
Rotten issues close after 30d of inactivity. /close Send feedback to tektoncd/plumbing. |
@tekton-robot: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Expected Behavior
When executing one pipelinerun, sometimes the pod creation will fail due to error which cannot be mitigated by retrying. In this case, pipeline controller should simply fail the pipelinerun without retrying creating pod.
In my case, such error is pod creation failure due to not enough quota in namespace.
Actual Behavior
Pipeline controller keep trying to create pod even there is no enough quota in namespace.
Steps to Reproduce the Problem
Additional Info
Kubernetes version:
Output of
kubectl version
:1.18
The text was updated successfully, but these errors were encountered: