Support Task-level Resources Requirements on TaskRun: Part #1

Required fields and related webhook validations are added to support a user to configure compute resources for TaskRun which will significantly reduce the over-asked resources amount configured by the Step-level.
austinzhao-go · Jun 17, 2022 · e67fb1c · e67fb1c
1 parent 20f1aa7
commit e67fb1c
Show file tree

Hide file tree

Showing 13 changed files with 344 additions and 7 deletions.
diff --git a/docs/compute-resources.md b/docs/compute-resources.md
@@ -42,7 +42,10 @@ request will be 6.
 Since the sidecar container has no CPU limit, this is treated as the highest CPU limit.
 Therefore, the pod will have no effective CPU limit.
 
-## Task Resource Requirements
+## Task-level Compute Resources Configuration
+
+**([alpha only](https://github.com/tektoncd/pipeline/blob/main/docs/install.md#alpha-features))**
+(This feature is under development and not functional yet. Stay tuned!)
 
 Tekton allows users to specify resource requirements of [`Steps`](./tasks.md#defining-steps),
 which run sequentially. However, the pod's effective resource requirements are still the
@@ -52,6 +55,86 @@ requirements for `Step`containers, they must be treated as if they are running i
 Tekton adjusts `Step` resource requirements to comply with [LimitRanges](#limitrange-support).
 [ResourceQuotas](#resourcequota-support) are not currently supported.
 
+Instead of specifying resource requirements on each `Step`, users can choose to specify resource requirements at the Task-level. If users specify a Task-level resource request, it will ensure that the kubelet reserves only that amount of resources to execute the `Task`'s `Steps`.
+If users specify a Task-level resource limit, no `Step` may use more than that amount of resources.
+
+Each of these details is explained in more depth below.
+
+Some points to note:
+
+- Task-level resource requests and limits do not apply to sidecars which can be configured separately.
+- Users may not configure the Task-level and Step-level resource requirements (requests/limits)  simultaneously.
+
+### Configure Task-level Compute Resources
+
+Task-level resource requirements can be configured in `TaskRun.ComputeResources`, or `PipelineRun.TaskRunSpecs.ComputeResources`.
+
+e.g.
+
+```yaml
+apiVersion: tekton.dev/v1beta1
+kind: TaskRun
+metadata:
+  name: foo 
+spec:
+  computeResources:
+    requests:
+      cpu: 1 
+    limits:
+      cpu: 2
+```
+
+The following TaskRun will be rejected, because it configures both step-level and task-level compute resource requirements:
+
+```yaml
+kind: TaskRun
+spec:
+  stepOverrides:
+    - name: foo
+      resources:
+        requests:
+          cpu: 1
+  computeResources:
+    requests:
+      cpu: 2 
+```
+
+```yaml
+kind: PipelineRun
+spec:
+  taskRunSpecs:
+    - pipelineTaskName: foo
+      stepOverrides:
+        - name: foo 
+          resources:
+            requests:
+              cpu: 1 
+      computeResources:
+        requests:
+          cpu: 2
+```
+
+### Configure Resource Requirements with Sidecar
+
+Users can specify compute resources separately for a sidecar while configuring task-level resource requirements on TaskRun.
+
+e.g.
+
+```yaml
+kind: TaskRun
+spec:
+  sidecarOverrides:
+    - name: sidecar
+      resources:
+        requests:
+          cpu: 750m
+        limits:
+          cpu: 1
+  computeResources:
+    requests:
+      cpu: 2
+```
+
 ## LimitRange Support
 
 Kubernetes allows users to configure [LimitRanges]((https://kubernetes.io/docs/concepts/policy/limit-range/)),

diff --git a/docs/install.md b/docs/install.md
@@ -427,6 +427,7 @@ Features currently in "alpha" are:
 | [Step and Sidecar Overrides](./taskruns.md#overriding-task-steps-and-sidecars)                        | [TEP-0094](https://github.com/tektoncd/community/blob/main/teps/0094-specifying-resource-requirements-at-runtime.md) |                                                                      |                             |
 | [Matrix](./matrix.md)                                                                                 | [TEP-0090](https://github.com/tektoncd/community/blob/main/teps/0090-matrix.md)                                      |                                                                      |                             |
 | [Embedded Statuses](pipelineruns.md#configuring-usage-of-taskrun-and-run-embedded-statuses)           | [TEP-0100](https://github.com/tektoncd/community/blob/main/teps/0100-embedded-taskruns-and-runs-status-in-pipelineruns.md) |                                                                |                             |
+| [Task-level Resource Requirements](compute-resources.md#task-level-compute-resources-configuration)   | [TEP-0104](https://github.com/tektoncd/community/blob/main/teps/0104-tasklevel-resource-requirements.md)             |                                                                      |                             |
 
 ## Configuring High Availability
 

diff --git a/docs/pipelineruns.md b/docs/pipelineruns.md
@@ -14,6 +14,7 @@ weight: 500
       - [Tekton Bundles](#tekton-bundles)
       - [Remote Pipelines](#remote-pipelines)
     - [Specifying <code>Resources</code>](#specifying-resources)
+    - [Specifying Task-level `ComputeResources`](#specifying-task-level-computeresources)
     - [Specifying <code>Parameters</code>](#specifying-parameters)
       - [Propagated Parameters](#propagated-parameters)
         - [Scope and Precedence](#scope-and-precedence)
@@ -260,6 +261,40 @@ spec:
 until their respective `Pods` or the entire `PipelineRun` are deleted. This also applies
 to all `persistentVolumeClaims` generated internally.
 
+### Specifying Task-level `ComputeResources`
+
+**([alpha only](https://github.com/tektoncd/pipeline/blob/main/docs/install.md#alpha-features))**
+(This feature is under development and not functional yet. Stay tuned!)
+
+Task-level compute resources can be configured in `PipelineRun.TaskRunSpecs.ComputeResources` or `TaskRun.ComputeResources`.
+
+e.g.
+
+```yaml
+apiVersion: tekton.dev/v1beta1
+kind: Pipeline
+metadata:
+  name: pipeline
+spec:
+  tasks:
+    - name: task
+---
+apiVersion: tekton.dev/v1beta1
+kind: PipelineRun
+metadata:
+  name: pipelinerun 
+spec:
+  pipelineRef:
+    name: pipeline
+  taskRunSpecs:
+    - pipelineTaskName: task
+      computeResources:
+        requests:
+          cpu: 2
+```
+
+Further details and examples could be found in [Compute Resources in Tekton](https://github.com/tektoncd/pipeline/blob/main/docs/compute-resources.md).
+
 ### Specifying `Parameters`
 
 (See also [Specifying Parameters in Tasks](tasks.md#specifying-parameters))

diff --git a/docs/taskruns.md b/docs/taskruns.md
@@ -18,6 +18,7 @@ weight: 300
     - [Extra Parameters](#extra-parameters)
   - [Specifying `Resources`](#specifying-resources)
   - [Specifying `Resource` limits](#specifying-resource-limits)
+  - [Specifying Task-level `ComputeResources`](#specifying-task-level-computeresources)
   - [Specifying a `Pod` template](#specifying-a-pod-template)
   - [Specifying `Workspaces`](#specifying-workspaces)
   - [Specifying `Sidecars`](#specifying-sidecars)
@@ -324,6 +325,40 @@ Each Step in a Task can specify its resource requirements. See
 [Defining `Steps`](tasks.md#defining-steps). Resource requirements defined in Steps and Sidecars
 may be overridden by a TaskRun's StepOverrides and SidecarOverrides.
 
+### Specifying Task-level `ComputeResources`
+
+**([alpha only](https://github.com/tektoncd/pipeline/blob/main/docs/install.md#alpha-features))**
+(This feature is under development and not functional yet. Stay tuned!)
+
+Task-level compute resources can be configured in `TaskRun.ComputeResources`, or `PipelineRun.TaskRunSpecs.ComputeResources`.
+
+e.g.
+
+```yaml
+apiVersion: tekton.dev/v1beta1
+kind: Task
+metadata:
+  name: task
+spec:
+  steps:
+    - name: foo
+---
+apiVersion: tekton.dev/v1beta1
+kind: TaskRun
+metadata:
+  name: taskrun 
+spec:
+  taskRef:
+    name: task
+  computeResources:
+    requests:
+      cpu: 1 
+    limits:
+      cpu: 2
+```
+
+Further details and examples could be found in [Compute Resources in Tekton](https://github.com/tektoncd/pipeline/blob/main/docs/compute-resources.md).
+
 ### Specifying a `Pod` template
 
 You can specify a [`Pod` template](podtemplates.md) configuration that will serve as the configuration starting

diff --git a/pkg/apis/pipeline/v1beta1/openapi_generated.go b/pkg/apis/pipeline/v1beta1/openapi_generated.go
diff --git a/pkg/apis/pipeline/v1beta1/pipelinerun_types.go b/pkg/apis/pipeline/v1beta1/pipelinerun_types.go
@@ -20,6 +20,7 @@ import (
 	"context"
 	"time"
 
+	corev1 "k8s.io/api/core/v1"
 	"k8s.io/apimachinery/pkg/runtime"
 
 	"github.com/tektoncd/pipeline/pkg/apis/config"
@@ -540,6 +541,9 @@ type PipelineTaskRunSpec struct {
 
 	// +optional
 	Metadata *PipelineTaskMetadata `json:"metadata,omitempty"`
+
+	// Compute resources to use for this TaskRun
+	ComputeResources *corev1.ResourceRequirements `json:"computeResources,omitempty"`
 }
 
 // GetTaskRunSpec returns the task specific spec for a given

diff --git a/pkg/apis/pipeline/v1beta1/pipelinerun_validation.go b/pkg/apis/pipeline/v1beta1/pipelinerun_validation.go
@@ -195,6 +195,9 @@ func validateTaskRunSpec(ctx context.Context, trs PipelineTaskRunSpec) (errs *ap
 		if trs.SidecarOverrides != nil {
 			errs = errs.Also(validateSidecarOverrides(trs.SidecarOverrides).ViaField("sidecarOverrides"))
 		}
+		if trs.ComputeResources != nil {
+			errs = errs.Also(validateTaskRunComputeResources(trs.ComputeResources, trs.StepOverrides))
+		}
 	} else {
 		if trs.StepOverrides != nil {
 			errs = errs.Also(apis.ErrDisallowedFields("stepOverrides"))

diff --git a/pkg/apis/pipeline/v1beta1/pipelinerun_validation_test.go b/pkg/apis/pipeline/v1beta1/pipelinerun_validation_test.go
@@ -610,7 +610,32 @@ func TestPipelineRunSpec_Invalidate(t *testing.T) {
 		},
 		wantErr:     apis.ErrMissingField("taskRunSpecs[0].sidecarOverrides[0].name"),
 		withContext: enableAlphaAPIFields,
+	}, {
+		name: "invalid both step-level (stepOverrides.resources) and task-level (taskRunSpecs.resources) resource requirements configured",
+		spec: v1beta1.PipelineRunSpec{
+			PipelineRef: &v1beta1.PipelineRef{Name: "pipeline"},
+			TaskRunSpecs: []v1beta1.PipelineTaskRunSpec{
+				{
+					PipelineTaskName: "pipelineTask",
+					StepOverrides: []v1beta1.TaskRunStepOverride{{
+						Name: "stepOverride",
+						Resources: corev1.ResourceRequirements{
+							Requests: corev1.ResourceList{corev1.ResourceMemory: corev1resources.MustParse("1Gi")},
+						}},
+					},
+					ComputeResources: &corev1.ResourceRequirements{
+						Requests: corev1.ResourceList{corev1.ResourceMemory: corev1resources.MustParse("2Gi")},
+					},
+				},
+			},
+		},
+		wantErr: apis.ErrMultipleOneOf(
+			"taskRunSpecs[0].stepOverrides.resources",
+			"taskRunSpecs[0].computeResources",
+		),
+		withContext: enableAlphaAPIFields,
 	}}
+
 	for _, ps := range tests {
 		t.Run(ps.name, func(t *testing.T) {
 			ctx := context.Background()
@@ -627,8 +652,9 @@ func TestPipelineRunSpec_Invalidate(t *testing.T) {
 
 func TestPipelineRunSpec_Validate(t *testing.T) {
 	tests := []struct {
-		name string
-		spec v1beta1.PipelineRunSpec
+		name        string
+		spec        v1beta1.PipelineRunSpec
+		withContext func(context.Context) context.Context
 	}{{
 		name: "PipelineRun without pipelineRef",
 		spec: v1beta1.PipelineRunSpec{
@@ -641,10 +667,53 @@ func TestPipelineRunSpec_Validate(t *testing.T) {
 				}},
 			},
 		},
+	}, {
+		name: "valid task-level (taskRunSpecs.resources) resource requirements configured",
+		spec: v1beta1.PipelineRunSpec{
+			PipelineRef: &v1beta1.PipelineRef{Name: "pipeline"},
+			TaskRunSpecs: []v1beta1.PipelineTaskRunSpec{{
+				PipelineTaskName: "pipelineTask",
+				StepOverrides: []v1beta1.TaskRunStepOverride{{
+					Name: "stepOverride",
+				}},
+				ComputeResources: &corev1.ResourceRequirements{
+					Requests: corev1.ResourceList{corev1.ResourceMemory: corev1resources.MustParse("2Gi")},
+				},
+			}},
+		},
+		withContext: enableAlphaAPIFields,
+	}, {
+		name: "valid sidecar and task-level (taskRunSpecs.resources) resource requirements configured",
+		spec: v1beta1.PipelineRunSpec{
+			PipelineRef: &v1beta1.PipelineRef{Name: "pipeline"},
+			TaskRunSpecs: []v1beta1.PipelineTaskRunSpec{{
+				PipelineTaskName: "pipelineTask",
+				StepOverrides: []v1beta1.TaskRunStepOverride{{
+					Name: "stepOverride",
+				}},
+				ComputeResources: &corev1.ResourceRequirements{
+					Requests: corev1.ResourceList{corev1.ResourceMemory: corev1resources.MustParse("2Gi")},
+				},
+				SidecarOverrides: []v1beta1.TaskRunSidecarOverride{{
+					Name: "sidecar",
+					Resources: corev1.ResourceRequirements{
+						Requests: corev1.ResourceList{
+							corev1.ResourceMemory: corev1resources.MustParse("4Gi"),
+						},
+					},
+				}},
+			}},
+		},
+		withContext: enableAlphaAPIFields,
 	}}
+
 	for _, ps := range tests {
 		t.Run(ps.name, func(t *testing.T) {
-			if err := ps.spec.Validate(context.Background()); err != nil {
+			ctx := context.Background()
+			if ps.withContext != nil {
+				ctx = ps.withContext(ctx)
+			}
+			if err := ps.spec.Validate(ctx); err != nil {
 				t.Error(err)
 			}
 		})

diff --git a/pkg/apis/pipeline/v1beta1/swagger.json b/pkg/apis/pipeline/v1beta1/swagger.json
@@ -1394,6 +1394,10 @@
       "description": "PipelineTaskRunSpec  can be used to configure specific specs for a concrete Task",
       "type": "object",
       "properties": {
+        "computeResources": {
+          "description": "Compute resources to use for this TaskRun",
+          "$ref": "#/definitions/v1.ResourceRequirements"
+        },
         "metadata": {
           "$ref": "#/definitions/v1beta1.PipelineTaskMetadata"
         },
@@ -2433,6 +2437,10 @@
       "description": "TaskRunSpec defines the desired state of TaskRun",
       "type": "object",
       "properties": {
+        "computeResources": {
+          "description": "Compute resources to use for this TaskRun",
+          "$ref": "#/definitions/v1.ResourceRequirements"
+        },
         "debug": {
           "$ref": "#/definitions/v1beta1.TaskRunDebug"
         },