Support Task-level Resources Requirements on TaskRun: Part #1

Required fields and related webhook validations are added to support a user to configure compute resources for TaskRun which will significantly reduce the over-asked resources amount configured by the Step-level.
austinzhao-go · Jun 16, 2022 · 445529f · 445529f
1 parent d48cfdd
commit 445529f
Show file tree

Hide file tree

Showing 12 changed files with 411 additions and 7 deletions.
diff --git a/docs/compute-resources.md b/docs/compute-resources.md
@@ -42,7 +42,9 @@ request will be 6.
 Since the sidecar container has no CPU limit, this is treated as the highest CPU limit.
 Therefore, the pod will have no effective CPU limit.
 
-## Task Resource Requirements
+## Task-level Compute Resources Configuration
+
+**([alpha only](https://github.com/tektoncd/pipeline/blob/main/docs/install.md#alpha-features))**
 
 Tekton allows users to specify resource requirements of [`Steps`](./tasks.md#defining-steps),
 which run sequentially. However, the pod's effective resource requirements are still the
@@ -52,6 +54,107 @@ requirements for `Step`containers, they must be treated as if they are running i
 Tekton adjusts `Step` resource requirements to comply with [LimitRanges](#limitrange-support).
 [ResourceQuotas](#resourcequota-support) are not currently supported.
 
+Instead of specifying resource requirements on each `Step`, users can choose to specify resource requirements at the Task-level. If users specify a Task-level resource request, it will ensure that the kubelet reserves only that amount of resources to execute the `Task`'s `Steps`.
+If users specify a Task-level resource limit, no `Step` may use more than that amount of resources.
+
+Each of these details is explained in more depth below.
+
+Some points to note:
+
+- Task-level resource requests and limits do not apply to sidecars which can be configured separately.
+- Users may not configure the Task-level and Step-level resource requirements (requests/limits)  simultaneously.
+
+### Configure Task-level Compute Resources
+
+Task-level resource requirements can be configured in `TaskRun.ComputeResources`, or `PipelineRun.TaskRunSpecs.ComputeResources`.
+
+e.g.
+
+```yaml
+apiVersion: tekton.dev/v1beta1
+kind: TaskRun
+metadata:
+  name: foo 
+spec:
+  computeResources:
+    requests:
+      cpu: 1 
+    limits:
+      cpu: 2
+```
+
+### Specify Task-level Compute Resources in TaskRun/PipelineRun
+
+Users can specify compute resources either at the Step-level or the Task-level in `TaskRun`.
+
+To specify compute resources at the Step-level, use `TaskRun.StepOverrides.Resources`.
+To specify compute resources at the Task-level, use `TaskRun.ComputeResources` or `PipelineRun.TaskRunSpecs.ComputeResources`.
+
+e.g.
+
+Using `TaskRun.Resources`:
+
+```yaml
+kind: TaskRun
+spec:
+  computeResources:
+    requests:
+      cpu: 2
+```
+
+Rejected when configuring `TaskRun.StepOverrides.Resources` and `TaskRun.ComputeResources`:  
+
+```yaml
+kind: TaskRun
+spec:
+  stepOverrides:
+    - name: foo
+      resources:
+        requests:
+          cpu: 1
+  computeResources:
+    requests:
+      cpu: 2 
+```
+
+Rejected when configuring `PipelineRun.TaskRunSpecs.StepOverrides.Resources` and `PipelineRun.TaskRunSpecs.ComputeResources`:  
+
+```yaml
+kind: PipelineRun
+spec:
+  taskRunSpecs:
+    - pipelineTaskName: foo
+      stepOverrides:
+        - name: foo 
+          resources:
+            requests:
+              cpu: 1 
+      computeResources:
+        requests:
+          cpu: 2
+```
+
+### Configure Resource Requirements with Sidecar
+
+Users can specify compute resources separately for a sidecar while configuring task-level resource requirements on TaskRun.
+
+e.g.
+
+```yaml
+kind: TaskRun
+spec:
+  sidecarOverrides:
+    - name: sidecar
+      resources:
+        requests:
+          cpu: 750m
+        limits:
+          cpu: 1
+  computeResources:
+    requests:
+      cpu: 2
+```
+
 ## LimitRange Support
 
 Kubernetes allows users to configure [LimitRanges]((https://kubernetes.io/docs/concepts/policy/limit-range/)),
@@ -217,6 +320,113 @@ spec:
 Here, the minimum of the "max" values is the output "max" value, and likewise for "default" and "defaultRequest".
 The maximum of the "min" values is the output "min" value.
 
+### Configure Task-level Compute Resources on TaskRun with LimitRange
+
+**([alpha only](https://github.com/tektoncd/pipeline/blob/main/docs/install.md#alpha-features))**
+
+User can use `LimitRange` to configure min or max compute resources for a `Task` or `Step`.
+
+(Note: The Task-level compute resources configuration is an alpha feature at this moment.)
+
+Here are examples with a task-level resource requirements:
+
+e.g.
+
+Applying min:
+
+```yaml
+kind: LimitRange
+spec:
+  limits:
+    - min:
+        cpu: 200m
+---
+kind: Task
+spec:
+  steps:
+    - name: step1 # applied with min
+    - name: step2 # applied with min
+    - name: step3 # not applied
+---
+kind: TaskRun
+spec:
+  computeResources:
+    requests:
+      cpu: 1
+```
+
+| Step name | CPU request |
+| --------- | ----------- |
+| step1     | 800m        |
+| step2     | 200m        |
+| step3     | N/A         |
+
+Here the 800m on step1 comes from `200m + (1 - 200m * 2)`.
+
+Applying max:
+
+```yaml
+kind: LimitRange
+spec:
+  limits:
+    - max:
+        cpu: 800m
+---
+kind: Task
+spec:
+  steps:
+    - name: step1 # applied with max
+    - name: step2 # applied with max
+    - name: step3 # not applied
+---
+kind: TaskRun
+spec:
+  computeResources:
+    requests:
+      cpu: 1
+```
+
+| Step name | CPU request |
+| --------- | ----------- |
+| step1     | 333m        |
+| step2     | 333m        |
+| step3     | 333m        |
+
+Here the 333m comes from `1 / 3` with number rounding to leave decimals out.
+
+Applying min and max:
+
+```yaml
+kind: LimitRange
+spec:
+  limits:
+    - min:
+        cpu: 200m
+    - max:
+        cpu: 700m
+---
+kind: Task
+spec:
+  steps:
+    - name: step1 # applied with min and max
+    - name: step2 # applied with min and max
+    - name: step3 # not applied
+---
+kind: TaskRun
+spec:
+  computeResources:
+    requests:
+      cpu: 1
+```
+
+| Step name | CPU request |
+| --------- | ----------- |
+| step1     | 400m        |
+| step2     | 400m        |
+| step3     | 200m        |  
+
+Here the 400m comes from the min `200` + the spread out `(1 - 200m * 2) / 3`.
+
 ## ResourceQuota Support
 
 Kubernetes allows users to define [ResourceQuotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/),

diff --git a/docs/install.md b/docs/install.md
@@ -427,6 +427,7 @@ Features currently in "alpha" are:
 | [Step and Sidecar Overrides](./taskruns.md#overriding-task-steps-and-sidecars)                        | [TEP-0094](https://github.com/tektoncd/community/blob/main/teps/0094-specifying-resource-requirements-at-runtime.md) |                                                                      |                             |
 | [Matrix](./matrix.md)                                                                                 | [TEP-0090](https://github.com/tektoncd/community/blob/main/teps/0090-matrix.md)                                      |                                                                      |                             |
 | [Embedded Statuse](pipelineruns.md#configuring-usage-of-taskrun-and-run-embedded-statuses)                                                                                  | [TEP-0100](https://github.com/tektoncd/community/blob/main/teps/0100-embedded-taskruns-and-runs-status-in-pipelineruns.md)                                      |                                                                      |                             |
+| [Task-level Resource Requirements](compute-resources.md#task-level-compute-resources-configuration)                                                                                  | [TEP-0104](https://github.com/tektoncd/community/blob/main/teps/0104-tasklevel-resource-requirements.md)                                      |
 
 ## Configuring High Availability
 

diff --git a/docs/taskruns.md b/docs/taskruns.md
@@ -324,6 +324,39 @@ Each Step in a Task can specify its resource requirements. See
 [Defining `Steps`](tasks.md#defining-steps). Resource requirements defined in Steps and Sidecars
 may be overridden by a TaskRun's StepOverrides and SidecarOverrides.
 
+### Specifying Task-level `ComputeResources`
+
+**([alpha only](https://github.com/tektoncd/pipeline/blob/main/docs/install.md#alpha-features))**
+
+Task-level compute resources can be configured in `TaskRun.ComputeResources`, or `PipelineRun.TaskRunSpecs.ComputeResources`.
+
+e.g.
+
+```yaml
+apiVersion: tekton.dev/v1beta1
+kind: Task
+metadata:
+  name: task
+spec:
+  steps:
+    - name: foo
+---
+apiVersion: tekton.dev/v1beta1
+kind: TaskRun
+metadata:
+  name: taskrun 
+spec:
+  taskRef:
+    name: task
+  computeResources:
+    requests:
+      cpu: 1 
+    limits:
+      cpu: 2
+```
+
+Further details and examples could be found in [Compute Resources in Tekton](https://github.com/tektoncd/pipeline/blob/main/docs/compute-resources.md).
+
 ### Specifying a `Pod` template
 
 You can specify a [`Pod` template](podtemplates.md) configuration that will serve as the configuration starting

diff --git a/pkg/apis/pipeline/v1beta1/openapi_generated.go b/pkg/apis/pipeline/v1beta1/openapi_generated.go
diff --git a/pkg/apis/pipeline/v1beta1/pipelinerun_types.go b/pkg/apis/pipeline/v1beta1/pipelinerun_types.go
@@ -20,6 +20,7 @@ import (
 	"context"
 	"time"
 
+	corev1 "k8s.io/api/core/v1"
 	"k8s.io/apimachinery/pkg/runtime"
 
 	"github.com/tektoncd/pipeline/pkg/apis/config"
@@ -563,6 +564,9 @@ type PipelineTaskRunSpec struct {
 
 	// +optional
 	Metadata *PipelineTaskMetadata `json:"metadata,omitempty"`
+
+	// Compute resources to use for this TaskRun
+	ComputeResources *corev1.ResourceRequirements `json:"computeResources,omitempty"`
 }
 
 // GetTaskRunSpec returns the task specific spec for a given

diff --git a/pkg/apis/pipeline/v1beta1/pipelinerun_validation.go b/pkg/apis/pipeline/v1beta1/pipelinerun_validation.go
@@ -195,6 +195,9 @@ func validateTaskRunSpec(ctx context.Context, trs PipelineTaskRunSpec) (errs *ap
 		if trs.SidecarOverrides != nil {
 			errs = errs.Also(validateSidecarOverrides(trs.SidecarOverrides).ViaField("sidecarOverrides"))
 		}
+		if trs.ComputeResources != nil {
+			errs = errs.Also(validateTaskRunComputeResources(trs.ComputeResources, trs.StepOverrides))
+		}
 	} else {
 		if trs.StepOverrides != nil {
 			errs = errs.Also(apis.ErrDisallowedFields("stepOverrides"))