Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add workspace types for Task and TaskRun with validation #1639

Merged
merged 1 commit into from
Dec 17, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion docs/pipelineruns.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ Creation of a `PipelineRun` will trigger the creation of
- [Service account](#service-account)
- [Service accounts](#service-accounts)
- [Pod Template](#pod-template)
- [Workspaces](#workspaces)
- [Cancelling a PipelineRun](#cancelling-a-pipelinerun)
- [Examples](https://github.com/tektoncd/pipeline/tree/master/examples/pipelineruns)
- [Logs](logs.md)
Expand All @@ -27,7 +28,7 @@ following fields:

- Required:
- [`apiVersion`][kubernetes-overview] - Specifies the API version, for example
`tekton.dev/v1alpha1`.
`tekton.dev/v1alpha1`
- [`kind`][kubernetes-overview] - Specify the `PipelineRun` resource object.
- [`metadata`][kubernetes-overview] - Specifies data to uniquely identify the
`PipelineRun` resource object, for example a `name`.
Expand Down Expand Up @@ -265,6 +266,12 @@ spec:
claimName: my-volume-claim
```

## Workspaces

It is not yet possible to specify [workspaces](tasks.md#workspaces) via `Pipelines`
or `PipelineRuns`, so `Tasks` requiring `workspaces` cannot be used with them until
[#1438](https://github.com/tektoncd/pipeline/issues/1438) is completed.

## Cancelling a PipelineRun

In order to cancel a running pipeline (`PipelineRun`), you need to update its
Expand Down
7 changes: 7 additions & 0 deletions docs/pipelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ This document defines `Pipelines` and their capabilities.

- [Syntax](#syntax)
- [Declared resources](#declared-resources)
- [Workspaces][#declared-workspaces]
- [Parameters](#parameters)
- [Pipeline Tasks](#pipeline-tasks)
- [From](#from)
Expand Down Expand Up @@ -72,6 +73,12 @@ spec:
type: image
```

### Declared Workspaces

It is not yet possible to specify [workspaces](tasks.md#workspaces) via `Pipelines`
or `PipelineRuns`, so `Tasks` requiring `workspaces` cannot be used with them until
[#1438](https://github.com/tektoncd/pipeline/issues/1438) is completed.

### Parameters

`Pipeline`s can declare input parameters that must be supplied to the `Pipeline`
Expand Down
43 changes: 42 additions & 1 deletion docs/taskruns.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ A `TaskRun` runs until all `steps` have completed or until a failure occurs.
- [Overriding where resources are copied from](#overriding-where-resources-are-copied-from)
- [Service Account](#service-account)
- [Pod Template](#pod-template)
- [Workspaces](#workspaces)
- [Status](#status)
- [Steps](#steps)
- [Cancelling a TaskRun](#cancelling-a-taskrun)
Expand Down Expand Up @@ -57,7 +58,9 @@ following fields:
to configure the default timeout.
- [`podTemplate`](#pod-template) - Specifies a subset of
[`PodSpec`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.15/#pod-v1-core)
configuration that will be used as the basis for the `Task` pod.
configuration that will be used as the basis for the `Task` pod.
- [`workspaces`](#workspaces) - Specify the actual volumes to use for the
[workspaces](tasks.md#workspaces) declared by a `Task`

[kubernetes-overview]:
https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects/#required-fields
Expand Down Expand Up @@ -227,7 +230,45 @@ spec:
claimName: my-volume-claim
```

## Workspaces

For a `TaskRun` to execute [a `Task` that declares `workspaces`](tasks.md#workspaces),
at runtime you need to map the `workspaces` to actual physical volumes with
`workspaces`. Values in `workspaces` are
[`Volumes`](https://kubernetes.io/docs/tasks/configure-pod-container/configure-volume-storage/), however currently we only support a subset of `VolumeSources`:

* [`emptyDir`](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir)
* [`persistentVolumeClaim`](https://kubernetes.io/docs/concepts/storage/volumes/#persistentvolumeclaim)

_If you need support for a `VolumeSource` not listed here
[please open an issue](https://github.com/tektoncd/pipeline/issues) or feel free to
[contribute a PR](https://github.com/tektoncd/pipeline/blob/master/CONTRIBUTING.md)._


If the declared `workspaces` are not provided at runtime, the `TaskRun` will fail
with an error.

For example to provide an existing PVC called `mypvc` for a `workspace` called
`myworkspace` declared by the `Pipeline`, using the `my-subdir` folder which already exists
on the PVC (there will be an error if it does not exist):

```yaml
workspaces:
- name: myworkspace
persistentVolumeClaim:
claimName: mypvc
subPath: my-subdir
```

Or to use [`emptyDir`](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir) for the same `workspace`:

```yaml
workspaces:
- name: myworkspace
emptyDir: {}
```

_For a complete example see [workspace.yaml](../examples/taskruns/workspace.yaml)._

## Status

Expand Down
72 changes: 67 additions & 5 deletions docs/tasks.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ entire Kubernetes cluster.
- [Outputs](#outputs)
- [Controlling where resources are mounted](#controlling-where-resources-are-mounted)
- [Volumes](#volumes)
- [Container Template **deprecated**](#step-template)
- [Workspaces](#workspaces)
- [Step Template](#step-template)
- [Variable Substitution](#variable-substitution)
- [Examples](#examples)
Expand Down Expand Up @@ -77,6 +77,8 @@ following fields:
created by your `Task`
- [`volumes`](#volumes) - Specifies one or more volumes that you want to make
available to your `Task`'s steps.
- [`workspaces`](#workspaces) - Specifies paths at which you expect volumes to
be mounted and available
- [`stepTemplate`](#step-template) - Specifies a `Container` step
definition to use as the basis for all steps within your `Task`.
- [`sidecars`](#sidecars) - Specifies sidecar containers to run alongside
Expand Down Expand Up @@ -132,7 +134,6 @@ the body of a `Task`.

If multiple `steps` are defined, they will be executed in the same order as they
are defined, if the `Task` is invoked by a [`TaskRun`](taskruns.md).

Each `steps` in a `Task` must specify a container image that adheres to the
[container contract](./container-contract.md). For each of the `steps` fields,
or container images that you define:
Expand Down Expand Up @@ -377,6 +378,44 @@ For example, use volumes to accomplish one of the following common tasks:
unsafe_. Use [kaniko](https://github.com/GoogleContainerTools/kaniko) instead.
This is used only for the purposes of demonstration.

### Workspaces

`workspaces` are a way of declaring volumes you expect to be made available to your
executing `Task` and the path to make them available at. They are similar to
[`volumes`](#volumes) but allow you to enforce at runtime that the volumes have
been attached and [allow you to specify subpaths](taskruns.md#workspaces) in the volumes
to attach.

The volume will be made available at `/workspace/myworkspace`, or you can override
this with `mountPath`. The value at `mountPath` can be anywhere on your pod's filesystem.
The path will be available via [variable substitution](#variable-substitution) with
`$(workspaces.myworkspace.path)`.

The actual volumes must be provided at runtime
[in the `TaskRun`](taskruns.md#workspaces).
In a future iteration ([#1438](https://github.com/tektoncd/pipeline/issues/1438))
it [will be possible to specify these in the `PipelineRun`](pipelineruns.md#workspaces)
as well.

For example:

```yaml
spec:
steps:
- name: write-message
image: ubuntu
script: |
#!/usr/bin/env bash
set -xe
echo hello! > $(workspaces.messages.path)/message
workspaces:
- name: messages
description: The folder where we write the message to
mountPath: /custom/path/relative/to/root
```

_For a complete example see [workspace.yaml](../examples/taskruns/workspace.yaml)._

### Step Template

Specifies a [`Container`](https://kubernetes.io/docs/concepts/containers/)
Expand Down Expand Up @@ -464,9 +503,17 @@ has been created to track this bug.

### Variable Substitution

`Tasks` support string replacement using values from all [`inputs`](#inputs) and
[`outputs`](#outputs).
`Tasks` support string replacement using values from:

* [Inputs and Outputs](#input-and-output-substitution)
* [Array params](#variable-substitution-with-parameters-of-type-array)
* [`workspaces`](#variable-substitution-with-workspaces)
* [`volumes`](#variable-substitution-with-volumes)

#### Input and Output substitution

[`inputs`](#inputs) and [`outputs`](#outputs) attributes can be used in replacements,
including [`params`](#params) and [`resources`](./resources.md#variable-substitution).

Input parameters can be referenced in the `Task` spec using the variable substitution syntax below,
where `<name>` is the name of the parameter:
Expand All @@ -477,7 +524,7 @@ $(inputs.params.<name>)

Param values from resources can also be accessed using [variable substitution](./resources.md#variable-substitution)

#### Variable Substitution with Parameters of Type `Array`
##### Variable Substitution with Parameters of Type `Array`

Referenced parameters of type `array` will expand to insert the array elements in the reference string's spot.

Expand Down Expand Up @@ -520,6 +567,21 @@ A valid reference to the `build-args` parameter is isolated and in an eligible f
args: ["build", "$(inputs.params.build-args)", "additonalArg"]
```

#### Variable Substitution with Workspaces

Paths to a `Task's` declared [workspaces](#workspaces) can be substituted with:

```
$(workspaces.myworkspace.path)
```

Since the name of the `Volume` is not known until runtime and is randomized, you can also
substitute the volume name with:

```
$(workspaces.myworkspace.volume)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

interesting, what's an expected usage of this variable?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

erm good question! i think @vdemeester wanted it so maybe he has a use case XD

oh here's a crazy one: you could make the volume available to a sidecar? maybe a terrible idea 🤷‍♀️

even in the absence of a great use case tho i think the fact that we wont know what it is until runtime seems like it makes sense to let the Task author have access to it if they want it?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same though as @bobcatfish 😝

```

#### Variable Substitution within Volumes

Task volume names and different
Expand Down
2 changes: 1 addition & 1 deletion examples/taskruns/custom-volume.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ spec:
image: ubuntu
script: |
#!/usr/bin/env bash
cat /short/and/stout/file
cat /short/and/stout/file | grep stuff
volumeMounts:
- name: custom
mountPath: /short/and/stout
Expand Down
68 changes: 68 additions & 0 deletions examples/taskruns/workspace.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-pvc
spec:
resources:
requests:
storage: 5Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
---
apiVersion: tekton.dev/v1alpha1
kind: TaskRun
metadata:
generateName: custom-volume-
spec:
workspaces:
- name: custom
persistentVolumeClaim:
claimName: my-pvc
subPath: my-subdir
- name: custom2
persistentVolumeClaim:
claimName: my-pvc
- name: custom3
emptyDir: {}
subPath: testing
taskSpec:
steps:
- name: write
image: ubuntu
script: |
#!/usr/bin/env bash
set -xe
echo $(workspaces.custom.volume) > $(workspaces.custom.path)/foo
- name: read
image: ubuntu
script: |
#!/usr/bin/env bash
set -xe
cat $(workspaces.custom.path)/foo | grep $(workspaces.custom.volume)
- name: write2
image: ubuntu
script: |
#!/usr/bin/env bash
set -xe
echo $(workspaces.custom2.path) > $(workspaces.custom2.path)/foo
- name: read2
image: ubuntu
script: |
#!/usr/bin/env bash
cat $(workspaces.custom2.path)/foo | grep $(workspaces.custom2.path)
- name: write3
image: ubuntu
script: |
#!/usr/bin/env bash
echo $(workspaces.custom3.path) > $(workspaces.custom3.path)/foo
- name: read3
image: ubuntu
script: |
#!/usr/bin/env bash
cat $(workspaces.custom3.path)/foo | grep $(workspaces.custom3.path)
workspaces:
- name: custom
- name: custom2
mountPath: /foo/bar/baz
- name: custom3
22 changes: 22 additions & 0 deletions pkg/apis/pipeline/paths.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
/*
Copyright 2019 The Tekton Authors

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package pipeline

const (
// WorkspaceDir is the root directory used for PipelineResources and (by default) Workspaces
WorkspaceDir = "/workspace"
)
5 changes: 2 additions & 3 deletions pkg/apis/pipeline/v1alpha1/git_resource.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,11 @@ import (
"strconv"
"strings"

"github.com/tektoncd/pipeline/pkg/apis/pipeline"
"github.com/tektoncd/pipeline/pkg/names"
corev1 "k8s.io/api/core/v1"
)

const WorkspaceDir = "/workspace"

var (
gitSource = "git-source"
)
Expand Down Expand Up @@ -145,7 +144,7 @@ func (s *GitResource) GetInputTaskModifier(_ *TaskSpec, path string) (TaskModifi
Image: s.GitImage,
Command: []string{"/ko-app/git-init"},
Args: args,
WorkingDir: WorkspaceDir,
WorkingDir: pipeline.WorkspaceDir,
// This is used to populate the ResourceResult status.
Env: []corev1.EnvVar{{
Name: "TEKTON_RESOURCE_NAME",
Expand Down
3 changes: 2 additions & 1 deletion pkg/apis/pipeline/v1alpha1/pull_request_resource.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ import (
"fmt"
"strings"

"github.com/tektoncd/pipeline/pkg/apis/pipeline"
"github.com/tektoncd/pipeline/pkg/names"
corev1 "k8s.io/api/core/v1"
)
Expand Down Expand Up @@ -138,7 +139,7 @@ func (s *PullRequestResource) getSteps(mode string, sourcePath string) []Step {
Image: s.PRImage,
Command: []string{"/ko-app/pullrequest-init"},
Args: args,
WorkingDir: WorkspaceDir,
WorkingDir: pipeline.WorkspaceDir,
Env: evs,
}}}
}
Loading