Support Task-level Resources Requirements on TaskRun: Part #1

Required fields and related webhook validations are added to support a user to configure compute resources for TaskRun which will significantly reduce the over-asked resources amount configured by the Step-level.
tektoncd · Jun 16, 2022 · 3c668a9 · 3c668a9
1 parent d48cfdd
commit 3c668a9
Show file tree

Hide file tree

Showing 11 changed files with 410 additions and 7 deletions.
diff --git a/docs/compute-resources.md b/docs/compute-resources.md
@@ -42,7 +42,9 @@ request will be 6.
 Since the sidecar container has no CPU limit, this is treated as the highest CPU limit.
 Therefore, the pod will have no effective CPU limit.
 
-## Task Resource Requirements
+## Task-level Compute Resources Configuration
+
+**([alpha only](https://github.com/tektoncd/pipeline/blob/main/docs/install.md#alpha-features))**
 
 Tekton allows users to specify resource requirements of [`Steps`](./tasks.md#defining-steps),
 which run sequentially. However, the pod's effective resource requirements are still the
@@ -52,6 +54,107 @@ requirements for `Step`containers, they must be treated as if they are running i
 Tekton adjusts `Step` resource requirements to comply with [LimitRanges](#limitrange-support).
 [ResourceQuotas](#resourcequota-support) are not currently supported.
 
+Instead of specifying resource requirements on each `Step`, users can choose to specify resource requirements at the Task-level. If users specify a Task-level resource request, it will ensure that the kubelet reserves only that amount of resources to execute the `Task`'s `Steps`.
+If users specify a Task-level resource limit, no `Step` may use more than that amount of resources.
+
+Each of these details is explained in more depth below.
+
+Some points to note:
+
+- Task-level resource requests and limits do not apply to sidecars which can be configured separately.
+- Users may not configure the Task-level and Step-level resource requirements (requests/limits)  simultaneously.
+
+### Configure Task-level Compute Resources
+
+Task-level resource requirements can be configured in `TaskRun.ComputeResources`, or `PipelineRun.TaskRunSpecs.ComputeResources`.
+
+e.g.
+
+```yaml
+apiVersion: tekton.dev/v1beta1
+kind: TaskRun
+metadata:
+  name: foo 
+spec:
+  computeResources:
+    requests:
+      cpu: 1 
+    limits:
+      cpu: 2
+```
+
+### Specify Task-level Compute Resources in TaskRun/PipelineRun
+
+Users can specify compute resources either at the Step-level or the Task-level in `TaskRun`.
+
+To specify compute resources at the Step-level, use `TaskRun.StepOverrides.Resources`.
+To specify compute resources at the Task-level, use `TaskRun.ComputeResources` or `PipelineRun.TaskRunSpecs.ComputeResources`.
+
+e.g.
+
+Using `TaskRun.Resources`:
+
+```yaml
+kind: TaskRun
+spec:
+  computeResources:
+    requests:
+      cpu: 2
+```
+
+Rejected when configuring `TaskRun.StepOverrides.Resources` and `TaskRun.ComputeResources`:  
+
+```yaml
+kind: TaskRun
+spec:
+  stepOverrides:
+    - name: foo
+      resources:
+        requests:
+          cpu: 1
+  computeResources:
+    requests:
+      cpu: 2 
+```
+
+Rejected when configuring `PipelineRun.TaskRunSpecs.StepOverrides.Resources` and `PipelineRun.TaskRunSpecs.ComputeResources`:  
+
+```yaml
+kind: PipelineRun
+spec:
+  taskRunSpecs:
+    - pipelineTaskName: foo
+      stepOverrides:
+        - name: foo 
+          resources:
+            requests:
+              cpu: 1 
+      computeResources:
+        requests:
+          cpu: 2
+```
+
+### Configure Resource Requirements with Sidecar
+
+Users can specify compute resources separately for a sidecar while configuring task-level resource requirements on TaskRun.
+
+e.g.
+
+```yaml
+kind: TaskRun
+spec:
+  sidecarOverrides:
+    - name: sidecar
+      resources:
+        requests:
+          cpu: 750m
+        limits:
+          cpu: 1
+  computeResources:
+    requests:
+      cpu: 2
+```
+
 ## LimitRange Support
 
 Kubernetes allows users to configure [LimitRanges]((https://kubernetes.io/docs/concepts/policy/limit-range/)),
@@ -217,6 +320,113 @@ spec:
 Here, the minimum of the "max" values is the output "max" value, and likewise for "default" and "defaultRequest".
 The maximum of the "min" values is the output "min" value.
 
+### Configure Task-level Compute Resources on TaskRun with LimitRange
+
+**([alpha only](https://github.com/tektoncd/pipeline/blob/main/docs/install.md#alpha-features))**
+
+User can use `LimitRange` to configure min or max compute resources for a `Task` or `Step`.
+
+(Note: The Task-level compute resources configuration is an alpha feature at this moment.)
+
+Here are examples with a task-level resource requirements:
+
+e.g.
+
+Applying min:
+
+```yaml
+kind: LimitRange
+spec:
+  limits:
+    - min:
+        cpu: 200m
+---
+kind: Task
+spec:
+  steps:
+    - name: step1 # applied with min
+    - name: step2 # applied with min
+    - name: step3 # not applied
+---
+kind: TaskRun
+spec:
+  computeResources:
+    requests:
+      cpu: 1
+```
+
+| Step name | CPU request |
+| --------- | ----------- |
+| step1     | 800m        |
+| step2     | 200m        |
+| step3     | N/A         |
+
+Here the 800m on step1 comes from `200m + (1 - 200m * 2)`.
+
+Applying max:
+
+```yaml
+kind: LimitRange
+spec:
+  limits:
+    - max:
+        cpu: 800m
+---
+kind: Task
+spec:
+  steps:
+    - name: step1 # applied with max
+    - name: step2 # applied with max
+    - name: step3 # not applied
+---
+kind: TaskRun
+spec:
+  computeResources:
+    requests:
+      cpu: 1
+```
+
+| Step name | CPU request |
+| --------- | ----------- |
+| step1     | 333m        |
+| step2     | 333m        |
+| step3     | 333m        |
+
+Here the 333m comes from `1 / 3` with number rounding to leave decimals out.
+
+Applying min and max:
+
+```yaml
+kind: LimitRange
+spec:
+  limits:
+    - min:
+        cpu: 200m
+    - max:
+        cpu: 700m
+---
+kind: Task
+spec:
+  steps:
+    - name: step1 # applied with min and max
+    - name: step2 # applied with min and max
+    - name: step3 # not applied
+---
+kind: TaskRun
+spec:
+  computeResources:
+    requests:
+      cpu: 1
+```
+
+| Step name | CPU request |
+| --------- | ----------- |
+| step1     | 400m        |
+| step2     | 400m        |
+| step3     | 200m        |  
+
+Here the 400m comes from the min `200` + the spread out `(1 - 200m * 2) / 3`.
+
 ## ResourceQuota Support
 
 Kubernetes allows users to define [ResourceQuotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/),

diff --git a/docs/taskruns.md b/docs/taskruns.md
@@ -324,6 +324,39 @@ Each Step in a Task can specify its resource requirements. See
 [Defining `Steps`](tasks.md#defining-steps). Resource requirements defined in Steps and Sidecars
 may be overridden by a TaskRun's StepOverrides and SidecarOverrides.
 
+### Specifying Task-level `ComputeResources`
+
+**([alpha only](https://github.com/tektoncd/pipeline/blob/main/docs/install.md#alpha-features))**
+
+Task-level compute resources can be configured in `TaskRun.ComputeResources`, or `PipelineRun.TaskRunSpecs.ComputeResources`.
+
+e.g.
+
+```yaml
+apiVersion: tekton.dev/v1beta1
+kind: Task
+metadata:
+  name: task
+spec:
+  steps:
+    - name: foo
+---
+apiVersion: tekton.dev/v1beta1
+kind: TaskRun
+metadata:
+  name: taskrun 
+spec:
+  taskRef:
+    name: task
+  computeResources:
+    requests:
+      cpu: 1 
+    limits:
+      cpu: 2
+```
+
+Further details and examples could be found in [Compute Resources in Tekton](https://github.com/tektoncd/pipeline/blob/main/docs/compute-resources.md).
+
 ### Specifying a `Pod` template
 
 You can specify a [`Pod` template](podtemplates.md) configuration that will serve as the configuration starting

diff --git a/pkg/apis/pipeline/v1beta1/openapi_generated.go b/pkg/apis/pipeline/v1beta1/openapi_generated.go
diff --git a/pkg/apis/pipeline/v1beta1/pipelinerun_types.go b/pkg/apis/pipeline/v1beta1/pipelinerun_types.go
@@ -20,6 +20,7 @@ import (
 	"context"
 	"time"
 
+	corev1 "k8s.io/api/core/v1"
 	"k8s.io/apimachinery/pkg/runtime"
 
 	"github.com/tektoncd/pipeline/pkg/apis/config"
@@ -563,6 +564,9 @@ type PipelineTaskRunSpec struct {
 
 	// +optional
 	Metadata *PipelineTaskMetadata `json:"metadata,omitempty"`
+
+	// Compute resources to use for this TaskRun
+	ComputeResources *corev1.ResourceRequirements `json:"computeResources,omitempty"`
 }
 
 // GetTaskRunSpec returns the task specific spec for a given

diff --git a/pkg/apis/pipeline/v1beta1/pipelinerun_validation.go b/pkg/apis/pipeline/v1beta1/pipelinerun_validation.go
@@ -195,6 +195,9 @@ func validateTaskRunSpec(ctx context.Context, trs PipelineTaskRunSpec) (errs *ap
 		if trs.SidecarOverrides != nil {
 			errs = errs.Also(validateSidecarOverrides(trs.SidecarOverrides).ViaField("sidecarOverrides"))
 		}
+		if trs.ComputeResources != nil {
+			errs = errs.Also(validateTaskRunComputeResources(trs.ComputeResources, trs.StepOverrides))
+		}
 	} else {
 		if trs.StepOverrides != nil {
 			errs = errs.Also(apis.ErrDisallowedFields("stepOverrides"))