Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some triggered Tekton jobs should have resource requests/limits #1122

Open
abayer opened this issue Jul 5, 2022 · 6 comments
Open

Some triggered Tekton jobs should have resource requests/limits #1122

abayer opened this issue Jul 5, 2022 · 6 comments
Assignees
Labels
area/dogfooding Indicates an issue on dogfooding (aka using Pipeline to test Pipeline) lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness.

Comments

@abayer
Copy link
Contributor

abayer commented Jul 5, 2022

The ones I notice right now are the plumbing-image-build and pull-pipeline-kind-k8s-v1-21-e2e PR PipelineRuns, and the build-and-push-test-runner cronjob triggered PipelineRun. I've seen the test-runner image builds cause OOMs on their nodes, and the plumbing-image-build one I'm looking at right now is at over 5gb memory used. The pull-pipeline-kind-k8s-v1-21-e2e pods that I've seen have ranged between 2 and 4gb memory used.

None of them (or any of the other Tekton PipelineRuns, for that matter) have any requests or limits configured, so they can end up on the same node, or a node with one of the other high memory usage pods always running in the cluster (i.e., prometheus and kafka) and cause problems. Given that dogfooding is hardcoded to 5 n1-standard-4s, with ~13gb allocatable memory, it's pretty easy for just a few of the high memory pods to end up on the same node and swamp it.

@dibyom dibyom added the area/dogfooding Indicates an issue on dogfooding (aka using Pipeline to test Pipeline) label Aug 17, 2022
@tekton-robot
Copy link
Contributor

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

@tekton-robot tekton-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 16, 2022
@tekton-robot
Copy link
Contributor

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

@tekton-robot tekton-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Dec 16, 2022
@tekton-robot
Copy link
Contributor

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

@tekton-robot
Copy link
Contributor

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@afrittoli
Copy link
Member

/remove-lifecycle rotten

@afrittoli afrittoli reopened this Feb 3, 2023
@tekton-robot tekton-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Feb 3, 2023
@github-project-automation github-project-automation bot moved this from Done to In Progress in Tekton Community Roadmap Feb 3, 2023
@afrittoli
Copy link
Member

/lifecycle frozen

@tekton-robot tekton-robot added the lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. label Feb 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/dogfooding Indicates an issue on dogfooding (aka using Pipeline to test Pipeline) lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness.
Projects
Status: In Progress
Development

No branches or pull requests

4 participants