Skip to content

Commit

Permalink
Support Task-level Resources Requirements on TaskRun: Part #1
Browse files Browse the repository at this point in the history
Required fields and related webhook validations are added to support
a user to configure compute resources for TaskRun which will significantly
reduce the over-asked resources amount configured by the Step-level.
  • Loading branch information
austinzhao-go committed Jun 17, 2022
1 parent 20f1aa7 commit e67fb1c
Show file tree
Hide file tree
Showing 13 changed files with 344 additions and 7 deletions.
85 changes: 84 additions & 1 deletion docs/compute-resources.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,10 @@ request will be 6.
Since the sidecar container has no CPU limit, this is treated as the highest CPU limit.
Therefore, the pod will have no effective CPU limit.

## Task Resource Requirements
## Task-level Compute Resources Configuration

**([alpha only](https://github.com/tektoncd/pipeline/blob/main/docs/install.md#alpha-features))**
(This feature is under development and not functional yet. Stay tuned!)

Tekton allows users to specify resource requirements of [`Steps`](./tasks.md#defining-steps),
which run sequentially. However, the pod's effective resource requirements are still the
Expand All @@ -52,6 +55,86 @@ requirements for `Step`containers, they must be treated as if they are running i
Tekton adjusts `Step` resource requirements to comply with [LimitRanges](#limitrange-support).
[ResourceQuotas](#resourcequota-support) are not currently supported.

Instead of specifying resource requirements on each `Step`, users can choose to specify resource requirements at the Task-level. If users specify a Task-level resource request, it will ensure that the kubelet reserves only that amount of resources to execute the `Task`'s `Steps`.
If users specify a Task-level resource limit, no `Step` may use more than that amount of resources.

Each of these details is explained in more depth below.

Some points to note:

- Task-level resource requests and limits do not apply to sidecars which can be configured separately.
- Users may not configure the Task-level and Step-level resource requirements (requests/limits) simultaneously.

### Configure Task-level Compute Resources

Task-level resource requirements can be configured in `TaskRun.ComputeResources`, or `PipelineRun.TaskRunSpecs.ComputeResources`.

e.g.

```yaml
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
name: foo
spec:
computeResources:
requests:
cpu: 1
limits:
cpu: 2
```
The following TaskRun will be rejected, because it configures both step-level and task-level compute resource requirements:
```yaml
kind: TaskRun
spec:
stepOverrides:
- name: foo
resources:
requests:
cpu: 1
computeResources:
requests:
cpu: 2
```
```yaml
kind: PipelineRun
spec:
taskRunSpecs:
- pipelineTaskName: foo
stepOverrides:
- name: foo
resources:
requests:
cpu: 1
computeResources:
requests:
cpu: 2
```
### Configure Resource Requirements with Sidecar
Users can specify compute resources separately for a sidecar while configuring task-level resource requirements on TaskRun.
e.g.
```yaml
kind: TaskRun
spec:
sidecarOverrides:
- name: sidecar
resources:
requests:
cpu: 750m
limits:
cpu: 1
computeResources:
requests:
cpu: 2
```
## LimitRange Support
Kubernetes allows users to configure [LimitRanges]((https://kubernetes.io/docs/concepts/policy/limit-range/)),
Expand Down
1 change: 1 addition & 0 deletions docs/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -427,6 +427,7 @@ Features currently in "alpha" are:
| [Step and Sidecar Overrides](./taskruns.md#overriding-task-steps-and-sidecars) | [TEP-0094](https://github.com/tektoncd/community/blob/main/teps/0094-specifying-resource-requirements-at-runtime.md) | | |
| [Matrix](./matrix.md) | [TEP-0090](https://github.com/tektoncd/community/blob/main/teps/0090-matrix.md) | | |
| [Embedded Statuses](pipelineruns.md#configuring-usage-of-taskrun-and-run-embedded-statuses) | [TEP-0100](https://github.com/tektoncd/community/blob/main/teps/0100-embedded-taskruns-and-runs-status-in-pipelineruns.md) | | |
| [Task-level Resource Requirements](compute-resources.md#task-level-compute-resources-configuration) | [TEP-0104](https://github.com/tektoncd/community/blob/main/teps/0104-tasklevel-resource-requirements.md) | | |
## Configuring High Availability
Expand Down
35 changes: 35 additions & 0 deletions docs/pipelineruns.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ weight: 500
- [Tekton Bundles](#tekton-bundles)
- [Remote Pipelines](#remote-pipelines)
- [Specifying <code>Resources</code>](#specifying-resources)
- [Specifying Task-level `ComputeResources`](#specifying-task-level-computeresources)
- [Specifying <code>Parameters</code>](#specifying-parameters)
- [Propagated Parameters](#propagated-parameters)
- [Scope and Precedence](#scope-and-precedence)
Expand Down Expand Up @@ -260,6 +261,40 @@ spec:
until their respective `Pods` or the entire `PipelineRun` are deleted. This also applies
to all `persistentVolumeClaims` generated internally.

### Specifying Task-level `ComputeResources`

**([alpha only](https://github.com/tektoncd/pipeline/blob/main/docs/install.md#alpha-features))**
(This feature is under development and not functional yet. Stay tuned!)

Task-level compute resources can be configured in `PipelineRun.TaskRunSpecs.ComputeResources` or `TaskRun.ComputeResources`.

e.g.

```yaml
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
name: pipeline
spec:
tasks:
- name: task
---
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
name: pipelinerun
spec:
pipelineRef:
name: pipeline
taskRunSpecs:
- pipelineTaskName: task
computeResources:
requests:
cpu: 2
```

Further details and examples could be found in [Compute Resources in Tekton](https://github.com/tektoncd/pipeline/blob/main/docs/compute-resources.md).

### Specifying `Parameters`

(See also [Specifying Parameters in Tasks](tasks.md#specifying-parameters))
Expand Down
35 changes: 35 additions & 0 deletions docs/taskruns.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ weight: 300
- [Extra Parameters](#extra-parameters)
- [Specifying `Resources`](#specifying-resources)
- [Specifying `Resource` limits](#specifying-resource-limits)
- [Specifying Task-level `ComputeResources`](#specifying-task-level-computeresources)
- [Specifying a `Pod` template](#specifying-a-pod-template)
- [Specifying `Workspaces`](#specifying-workspaces)
- [Specifying `Sidecars`](#specifying-sidecars)
Expand Down Expand Up @@ -324,6 +325,40 @@ Each Step in a Task can specify its resource requirements. See
[Defining `Steps`](tasks.md#defining-steps). Resource requirements defined in Steps and Sidecars
may be overridden by a TaskRun's StepOverrides and SidecarOverrides.

### Specifying Task-level `ComputeResources`

**([alpha only](https://github.com/tektoncd/pipeline/blob/main/docs/install.md#alpha-features))**
(This feature is under development and not functional yet. Stay tuned!)

Task-level compute resources can be configured in `TaskRun.ComputeResources`, or `PipelineRun.TaskRunSpecs.ComputeResources`.

e.g.

```yaml
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: task
spec:
steps:
- name: foo
---
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
name: taskrun
spec:
taskRef:
name: task
computeResources:
requests:
cpu: 1
limits:
cpu: 2
```

Further details and examples could be found in [Compute Resources in Tekton](https://github.com/tektoncd/pipeline/blob/main/docs/compute-resources.md).

### Specifying a `Pod` template

You can specify a [`Pod` template](podtemplates.md) configuration that will serve as the configuration starting
Expand Down
16 changes: 14 additions & 2 deletions pkg/apis/pipeline/v1beta1/openapi_generated.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions pkg/apis/pipeline/v1beta1/pipelinerun_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ import (
"context"
"time"

corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/runtime"

"github.com/tektoncd/pipeline/pkg/apis/config"
Expand Down Expand Up @@ -540,6 +541,9 @@ type PipelineTaskRunSpec struct {

// +optional
Metadata *PipelineTaskMetadata `json:"metadata,omitempty"`

// Compute resources to use for this TaskRun
ComputeResources *corev1.ResourceRequirements `json:"computeResources,omitempty"`
}

// GetTaskRunSpec returns the task specific spec for a given
Expand Down
3 changes: 3 additions & 0 deletions pkg/apis/pipeline/v1beta1/pipelinerun_validation.go
Original file line number Diff line number Diff line change
Expand Up @@ -195,6 +195,9 @@ func validateTaskRunSpec(ctx context.Context, trs PipelineTaskRunSpec) (errs *ap
if trs.SidecarOverrides != nil {
errs = errs.Also(validateSidecarOverrides(trs.SidecarOverrides).ViaField("sidecarOverrides"))
}
if trs.ComputeResources != nil {
errs = errs.Also(validateTaskRunComputeResources(trs.ComputeResources, trs.StepOverrides))
}
} else {
if trs.StepOverrides != nil {
errs = errs.Also(apis.ErrDisallowedFields("stepOverrides"))
Expand Down
75 changes: 72 additions & 3 deletions pkg/apis/pipeline/v1beta1/pipelinerun_validation_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -610,7 +610,32 @@ func TestPipelineRunSpec_Invalidate(t *testing.T) {
},
wantErr: apis.ErrMissingField("taskRunSpecs[0].sidecarOverrides[0].name"),
withContext: enableAlphaAPIFields,
}, {
name: "invalid both step-level (stepOverrides.resources) and task-level (taskRunSpecs.resources) resource requirements configured",
spec: v1beta1.PipelineRunSpec{
PipelineRef: &v1beta1.PipelineRef{Name: "pipeline"},
TaskRunSpecs: []v1beta1.PipelineTaskRunSpec{
{
PipelineTaskName: "pipelineTask",
StepOverrides: []v1beta1.TaskRunStepOverride{{
Name: "stepOverride",
Resources: corev1.ResourceRequirements{
Requests: corev1.ResourceList{corev1.ResourceMemory: corev1resources.MustParse("1Gi")},
}},
},
ComputeResources: &corev1.ResourceRequirements{
Requests: corev1.ResourceList{corev1.ResourceMemory: corev1resources.MustParse("2Gi")},
},
},
},
},
wantErr: apis.ErrMultipleOneOf(
"taskRunSpecs[0].stepOverrides.resources",
"taskRunSpecs[0].computeResources",
),
withContext: enableAlphaAPIFields,
}}

for _, ps := range tests {
t.Run(ps.name, func(t *testing.T) {
ctx := context.Background()
Expand All @@ -627,8 +652,9 @@ func TestPipelineRunSpec_Invalidate(t *testing.T) {

func TestPipelineRunSpec_Validate(t *testing.T) {
tests := []struct {
name string
spec v1beta1.PipelineRunSpec
name string
spec v1beta1.PipelineRunSpec
withContext func(context.Context) context.Context
}{{
name: "PipelineRun without pipelineRef",
spec: v1beta1.PipelineRunSpec{
Expand All @@ -641,10 +667,53 @@ func TestPipelineRunSpec_Validate(t *testing.T) {
}},
},
},
}, {
name: "valid task-level (taskRunSpecs.resources) resource requirements configured",
spec: v1beta1.PipelineRunSpec{
PipelineRef: &v1beta1.PipelineRef{Name: "pipeline"},
TaskRunSpecs: []v1beta1.PipelineTaskRunSpec{{
PipelineTaskName: "pipelineTask",
StepOverrides: []v1beta1.TaskRunStepOverride{{
Name: "stepOverride",
}},
ComputeResources: &corev1.ResourceRequirements{
Requests: corev1.ResourceList{corev1.ResourceMemory: corev1resources.MustParse("2Gi")},
},
}},
},
withContext: enableAlphaAPIFields,
}, {
name: "valid sidecar and task-level (taskRunSpecs.resources) resource requirements configured",
spec: v1beta1.PipelineRunSpec{
PipelineRef: &v1beta1.PipelineRef{Name: "pipeline"},
TaskRunSpecs: []v1beta1.PipelineTaskRunSpec{{
PipelineTaskName: "pipelineTask",
StepOverrides: []v1beta1.TaskRunStepOverride{{
Name: "stepOverride",
}},
ComputeResources: &corev1.ResourceRequirements{
Requests: corev1.ResourceList{corev1.ResourceMemory: corev1resources.MustParse("2Gi")},
},
SidecarOverrides: []v1beta1.TaskRunSidecarOverride{{
Name: "sidecar",
Resources: corev1.ResourceRequirements{
Requests: corev1.ResourceList{
corev1.ResourceMemory: corev1resources.MustParse("4Gi"),
},
},
}},
}},
},
withContext: enableAlphaAPIFields,
}}

for _, ps := range tests {
t.Run(ps.name, func(t *testing.T) {
if err := ps.spec.Validate(context.Background()); err != nil {
ctx := context.Background()
if ps.withContext != nil {
ctx = ps.withContext(ctx)
}
if err := ps.spec.Validate(ctx); err != nil {
t.Error(err)
}
})
Expand Down
8 changes: 8 additions & 0 deletions pkg/apis/pipeline/v1beta1/swagger.json
Original file line number Diff line number Diff line change
Expand Up @@ -1394,6 +1394,10 @@
"description": "PipelineTaskRunSpec can be used to configure specific specs for a concrete Task",
"type": "object",
"properties": {
"computeResources": {
"description": "Compute resources to use for this TaskRun",
"$ref": "#/definitions/v1.ResourceRequirements"
},
"metadata": {
"$ref": "#/definitions/v1beta1.PipelineTaskMetadata"
},
Expand Down Expand Up @@ -2433,6 +2437,10 @@
"description": "TaskRunSpec defines the desired state of TaskRun",
"type": "object",
"properties": {
"computeResources": {
"description": "Compute resources to use for this TaskRun",
"$ref": "#/definitions/v1.ResourceRequirements"
},
"debug": {
"$ref": "#/definitions/v1beta1.TaskRunDebug"
},
Expand Down
Loading

0 comments on commit e67fb1c

Please sign in to comment.