Affinity assistant only takes first task's resource requirements into account when choosing a node

[abayer]

Expected Behavior

The affinity assistant should either take the resource requirements of all Tasks in the PipelineRun into account when choosing the first node, or it should be flexible enough to allow later TaskRuns to end up on different nodes if the initial node can't satisfy the requirements.

Actual Behavior

The affinity assistant ends up on the node used by the first TaskRun, and subsequent TaskRuns are also tied to that node. This can cause problems when the initial node has sufficient resources for the first TaskRun, but not the second - the second TaskRun's pod will end up just sitting in pending until either the initial node has sufficient resources or the PipelineRun gets deleted.

Steps to Reproduce the Problem

1. Run a PipelineRun with an initial Task with no resource requirements, and a second Task with more resource requirements than a given node in the cluster will have available (due to other pods running on that node, etc).
2. Keep running the PipelineRun until the first TaskRun's pod ends up on the constrained node.
3. Watch the second TaskRun's pod sit in pending forever

Additional Info

• Kubernetes version:

  Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.1", GitCommit:"4485c6f18cee9a5d3c3b4e523bd27972b1b53892", GitTreeState:"clean", BuildDate:"2019-07-18T14:27:12Z", GoVersion:"go1.12.7", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.11-gke.5", GitCommit:"baccd25d44f1a0d06ad3190eb508784efbb990a5", GitTreeState:"clean", BuildDate:"2020-06-25T22:55:26Z", GoVersion:"go1.13.9b4", Compiler:"gc", Platform:"linux/amd64"}

• Tekton Pipeline version:

  Output of tkn version or kubectl get pods -n tekton-pipelines -l app=tekton-pipelines-controller -o=jsonpath='{.items[0].metadata.labels.version}'

Client version: 0.10.0
Pipeline version: v0.14.3
Triggers version: unknown

[jlpettersson]

Expected Behavior

...or it should be flexible enough to allow later TaskRuns to end up on different nodes if the initial node can't satisfy the requirements.

The intention with the Affinity Assistant is to exactly prevent this, because if two TaskRuns using the same volume is scheduled to two different Availability Zones in a regional cluster - the PipelineRun will be deadlocked and can not continue. An example of such situation is the e2e-test that I had to remove #2546 (that was before the affinity assistant existed).

Actual Behavior

The affinity assistant ends up on the node used by the first TaskRun, and subsequent TaskRuns are also tied to that node.

This is the intention by the Affinity Assistant. However, it can be disabled.

This can cause problems when the initial node has sufficient resources for the first TaskRun, but not the second - the second TaskRun's pod will end up just sitting in pending until either the initial node has sufficient resources or the PipelineRun gets deleted.

One solution to this may be to use dedicated Nodes for Tekton workload. E.g. using NodeSelector

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[vdemeester]

/remove-lifecycle stale

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[jlpettersson]

/remove-lifecycle stale

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[jlpettersson]

/remove-lifecycle rotten

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.