Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow Pipelines to be used with different Resources #248

Merged
merged 5 commits into from
Nov 22, 2018
Merged

Allow Pipelines to be used with different Resources #248

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
8333cb0
Fix markdown linting issues
bobcatfish Nov 19, 2018
c45d62e
Allow Pipelines to be used with different Resources
bobcatfish Nov 19, 2018
5e8cadc
Remove `version` from TaskRun resource binding
bobcatfish Nov 21, 2018
f59f95d
Add docs on how to use git revisions and image digests
bobcatfish Nov 21, 2018
7e2c673
Remove kubebuilder `make` reminders about regenerating code
bobcatfish Nov 22, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
282 changes: 170 additions & 112 deletions docs/using.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
* [How do I create a new Pipeline?](#creating-a-pipeline)
* [How do I make a Task?](#creating-a-task)
* [How do I make Resources?](#creating-resources)
* [How do I run a Pipeline?](#running-a-task)
* [How do I run a Pipeline?](#running-a-pipeline)
* [How do I run a Task on its own?](#running-a-task)
* [How do I troubleshoot a PipelineRun?](#troubleshooting)

Expand Down Expand Up @@ -48,9 +48,9 @@ To create a Task, you must:
* Define the inputs and outputs of the `Task` as [`Resources`](./Concepts.md#pipelineresources)
* Create a `Step` for each action you want to take in the `Task`

`Steps` are images which comply with the [image contract](#image-contract).
`Steps` are images which comply with the [container contract](#container-contract).

### Image Contract
### Container Contract

Each container image used as a step in a [`Task`](#task) must comply with a specific
contract.
Expand Down Expand Up @@ -106,15 +106,13 @@ steps:
value: 'world'
```

### Images Conventions
### Conventions

* `/workspace`: If an input is provided, the default working directory will be
`/workspace` and this will be shared across `steps` (note that in
[#123](https://github.com/knative/build-pipeline/issues/123) we will add supprots for multiple input workspaces)
* `/builder/home`: This volume is exposed to steps via `$HOME`.
* Credentials attached to the Build's service account may be exposed as Git or
Docker credentials as outlined
[in the auth docs](https://github.com/knative/docs/blob/master/build/auth.md#authentication).
* `/workspace/<resource-name>`: [`PipelineResources` are made available in this mounted dir](#creating-resources)
* `/builder/home`: This volume is exposed to steps via `$HOME`.
* Credentials attached to the Build's service account may be exposed as Git or
Docker credentials as outlined
[in the auth docs](https://github.com/knative/docs/blob/master/build/auth.md#authentication).

### Templating

Expand All @@ -136,10 +134,46 @@ ${inputs.params.NAME}

## Running a Pipeline

1. To run your `Pipeline`, create a new `PipelineRun` which links your `Pipeline` to the
`PipelineParams` it will run with.
2. Creation of a `PipelineRun` will trigger the creation of [`TaskRuns`](#running-a-task)
for each `Task` in your pipeline.
In order to run a Pipeline, you will need to provide:

1. A Pipeline to run (see [creating a Pipeline](#creating-a-pipeline))
2. `PipelineParams` which specify the service account to use and where to put logs (see [the example `PipelineParams`](examples/pipelineparams.yaml))
3. The `PipelineResources` to use with this Pipeline.

On its own, a `Pipeline` declares what `Tasks` to run, and what order to run them in
(implied by [`providedBy`](#providedby)). When running a `Pipeline`, you will
need to specify the `PipelineResources` to use with it. One `Pipeline` may need to be run
with different `PipelineResources` in cases such as:

* When triggering the run of a `Pipeline` against a pull request, the triggering system must
specify the commitish of a git `PipelineResource` to use
* When invoking a `Pipeline` manually against one's own setup, one will need to ensure that
one's own github fork (via the git `PipelineResource`), image registry (via the image
`PipelineResource`) and kubernetes cluster (via the cluster `PipelineResource`).

Specify the `PipelineResources` in the PipelineRun using the `resources` section, for example:

```yaml
resources:
- name: push-kritis
inputs:
- key: workspace
resourceRef:
name: kritis-resources-git
outputs:
- key: builtImage
resourceRef:
name: kritis-resources-image
```

This example section says:

* For the `Task` in the `Pipeline` called `push-kritis`
* For the input called `workspace`, use the existing resource called `kritis-resources-git`
* For the output called `builtImage`, use the existing resource called `kritis-resources-image`

Creation of a `PipelineRun` will trigger the creation of [`TaskRuns`](#running-a-task)
for each `Task` in your pipeline.

See [the example PipelineRun](../examples/invocations/kritis-pipeline-run.yaml).

Expand All @@ -154,120 +188,144 @@ See [the example TaskRun](../examples/invocations/run-kritis-test.yaml).

## Creating Resources

We current support these `PipelineResources`:

* [Git resource](#git-resource)
* [Image resource](#image-resource)
* [Cluster resource](#cluster-resource)

When used as inputs, these resources will be made available in a mounted directory called `/workspace`
at the path /workspace/<resource-name>`.

### Git Resource

Git resource represents a [git](https://git-scm.com/) repository, that containes
Git resource represents a [git](https://git-scm.com/) repository, that contains
the source code to be built by the pipeline. Adding the git resource as an input
to a task will clone this repository and allow the task to perform the required
actions on the contents of the repo.

Use the following example to understand the syntax and strucutre of a Git Resource
To create a git resource using the `PipelineResource` CRD:

```yaml
apiVersion: pipeline.knative.dev/v1alpha1
kind: PipelineResource
metadata:
name: wizzbang-git
namespace: default
spec:
type: git
params:
- name: url
value: https://github.com/wizzbangcorp/wizzbang.git
- name: revision
value: master
```

Params that can be added are the following:

1. Create a git resource using the `PipelineResource` CRD
1. `url`: represents the location of the git repository, you can use this to change the repo, e.g. [to use a fork](#using-a-fork)
1. `revision`: Git [revision](https://git-scm.com/docs/gitrevisions#_specifying_revisions) (branch, tag, commit SHA or ref) to clone.
You can use this to control what commit [or branch](#using-a-branch) is used. _If no revision is specified, the resource will default to `latest` from `master`._

```
apiVersion: pipeline.knative.dev/v1alpha1
kind: PipelineResource
metadata:
name: wizzbang-git
namespace: default
spec:
type: git
params:
- name: url
value: https://github.com/wizzbangcorp/wizzbang.git
- name: Revision
value: master
```

Params that can be added are the following:

1. URL: represents the location of the git repository
1. Revision: Git [revision](https://git-scm.com/docs/gitrevisions#_specifying_revisions ) (branch, tag, commit SHA or ref) to clone. If no revision is specified, the resource will default to `latest` from `master`

2. Use the defined git resource in a `Task` definition:

```
apiVersion: pipeline.knative.dev/v1alpha1
kind: Task
metadata:
name: build-push-task
namespace: default
spec:
inputs:
resources:
- name: wizzbang-git
type: git
params:
- name: pathToDockerfile
outputs:
resources:
- name: builtImage
type: image
steps:
- name: build-and-push
image: gcr.io/my-repo/my-imageg
args:
- --repo=${inputs.resources.wizzbang-git.url}
```

3. And finally set the version in the `TaskRun` definition:

```
apiVersion: pipeline.knative.dev/v1alpha1
kind: TaskRun
metadata:
name: build-push-task-run
namespace: default
spec:
taskRef:
name: build-push-task
inputs:
resourcesVersion:
- resourceRef:
name: wizzbang-git
apiVersion: HEAD
outputs:
resources:
- name: builtImage
type: image
# Optional, indicate a serviceAccount for authentication.
serviceAccount: build-bot
```
#### Using a fork

The `Url` parameter can be used to point at any git repository, for example to use a GitHub fork at master:

```yaml
spec:
type: git
params:
- name: url
value: https://github.com/bobcatfish/wizzbang.git
```

#### Using a branch

The `revision` can be any [git commit-ish (revision)](https://git-scm.com/docs/gitrevisions#_specifying_revisions).
You can use this to create a git `PipelineResource` that points at a branch, for example:

```yaml
spec:
type: git
params:
- name: url
value: https://github.com/wizzbangcorp/wizzbang.git
- name: revision
value: some_awesome_feature
```

To point at a pull request, you can use [the pull requests's branch](https://help.github.com/articles/checking-out-pull-requests-locally/):

```yaml
spec:
type: git
params:
- name: url
value: https://github.com/wizzbangcorp/wizzbang.git
- name: revision
value: refs/pull/52525/head
```

### Image Resource

An Image resource represents an image that lives in a remote repository. It is usually used as [a `Task` `output`](concepts.md#task) for
`Tasks` that build images. This allows the same `Tasks` to be used to generically push to any registry.

Params that can be added are the following:

1. `url`: The complete path to the image, including the registry and the image tag
2. `digest`: The [image digest](https://success.docker.com/article/images-tagging-vs-digests) which uniquely identifies a
particular build of an image with a particular tag.

For example:

```yaml
apiVersion: pipeline.knative.dev/v1alpha1
kind: PipelineResource
metadata:
name: kritis-resources-image
namespace: default
spec:
type: image
params:
- name: url
value: gcr.io/staging-images/kritis
```

### Cluster Resource

Cluster Resource represents a Kubernetes cluster other than the current cluster the pipleine
CRD is running on. A common use case for this resource is to deploy your application/function
on different clusters.
Cluster Resource represents a Kubernetes cluster other than the current cluster the pipleine
CRD is running on. A common use case for this resource is to deploy your application/function
on different clusters.

The resource will use the provided parameters to create a [kubeconfig](https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/) file
that can be used by other steps in the pipeline task to access the target cluster. The kubeconfig will be
The resource will use the provided parameters to create a [kubeconfig](https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/) file
that can be used by other steps in the pipeline task to access the target cluster. The kubeconfig will be
placed in `/workspace/<your-cluster-name>/kubeconfig` on your task container

The Cluster resource has the following parameters:
* Name: The name of the Resource is also given to cluster, will be used in the kubeconfig and also as part of the path to the kubeconfig file
* URL (required): Host url of the master node
* Username (required): the user with access to the cluster
* Password: to be used for clusters with basic auth
* Token: to be used for authenication, if present will be used ahead of the password
* Insecure: to indicate server should be accessed without verifying the TLS certificate.
* CAData (required): holds PEM-encoded bytes (typically read from a root certificates bundle).

* Name: The name of the Resource is also given to cluster, will be used in the kubeconfig and also as part of the path to the kubeconfig file
* URL (required): Host url of the master node
* Username (required): the user with access to the cluster
* Password: to be used for clusters with basic auth
* Token: to be used for authenication, if present will be used ahead of the password
* Insecure: to indicate server should be accessed without verifying the TLS certificate.
* CAData (required): holds PEM-encoded bytes (typically read from a root certificates bundle).

Note: Since only one authentication technique is allowed per user, either a token or a password should be provided, if both are provided, the password will be ignored.

The following example shows the syntax and structure of a Cluster Resource

```
```yaml
apiVersion: pipeline.knative.dev/v1alpha1
kind: PipelineResource
metadata:
name: test-cluster
spec:
spec:
type: cluster
params:
- name: url
value: https://10.10.10.10 # url to the cluster master node
value: https://10.10.10.10 # url to the cluster master node
- name: cadata
value: LS0tLS1CRUdJTiBDRVJ.....
- name: token
Expand All @@ -278,7 +336,7 @@ For added security, you can add the sensetive information in a Kubernetes [Secre

For example, create a secret like the following example

```
```yaml
apiVersion: v1
kind: Secret
metadata:
Expand All @@ -288,32 +346,32 @@ data:
tokenkey: ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbX....M2ZiCg==
```

and then apply secrets to the cluster resource
and then apply secrets to the cluster resource

```
```yaml
apiVersion: pipeline.knative.dev/v1alpha1
kind: PipelineResource
metadata:
name: test-cluster
spec:
spec:
type: cluster
params:
- name: url
value: https://10.10.10.10
value: https://10.10.10.10
- name: username
value: admin
secrets:
value: admin
secrets:
- fieldName: token
secretKey: tokenKey
secretName: target-cluster-secrets
- fieldName: cadata
secretKey: cadataKey
secretName: target-cluster-secrets
secretName: target-cluster-secrets
```

Example usage of the cluster resource in a task

```
```yaml
apiVersion: pipeline.knative.dev/v1alpha1
kind: Task
metadata:
Expand All @@ -327,7 +385,7 @@ spec:
- name: dockerimage
type: image
- name: testcluster
type: cluster
type: cluster
steps:
- name: deploy
image: image-wtih-kubectl
Expand Down
Loading