-
Notifications
You must be signed in to change notification settings - Fork 220
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add TEP-0044: Composing Tasks with Tasks ➰
This TEP describes some difficulty folks have run into, especially when trying to use Tekton Pipelines without PipelineResources: if you want to use the functionality of multiple Tasks, you need to put them together in a Pipeline and share data between them with volumes. It could be a useful abstraction, and more efficient, to make it possible to combine Tasks together without scheudling them on different pods.
- Loading branch information
1 parent
3072a20
commit 01cdf06
Showing
2 changed files
with
138 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,137 @@ | ||
| --- | ||
| status: proposed | ||
| title: Composing Tasks with Tasks | ||
| creation-date: '2021-01-22' | ||
| last-updated: '2021-01-22' | ||
| authors: | ||
| - '@bobcatfish' | ||
| --- | ||
|
|
||
| # TEP-0044: Composing Tasks with Tasks | ||
|
|
||
| <!-- toc --> | ||
| - [Summary](#summary) | ||
| - [Motivation](#motivation) | ||
| - [PipelineResources](#pipelineresources) | ||
| - [Goals](#goals) | ||
| - [Non-Goals](#non-goals) | ||
| - [Use Cases (optional)](#use-cases-optional) | ||
| - [Requirements](#requirements) | ||
| - [References (optional)](#references-optional) | ||
|
|
||
| ## Summary | ||
|
|
||
| This TEP describes a gap in composability that isn't currently fully solved by Tasks or Pipelines | ||
| (tho PipelineResources do solve some of it!). | ||
|
|
||
| Ideally by addressing the issues described here we'd make it possible to create more efficient | ||
| Pipelines and make it easier to use a Task easily via a TaskRun, and we'd address some of | ||
| [the problems PipelineResources are trying to solve](https://docs.google.com/document/d/1KpVyWi-etX00J3hIz_9HlbaNNEyuzP6S986Wjhl3ZnA/edit#) | ||
| making it more clear whether or not they are an abstraction we want to keep. | ||
|
|
||
| [#1673: Pipeline Resources Redesign](https://github.com/tektoncd/pipeline/issues/1673) | ||
|
|
||
| ## Motivation | ||
|
|
||
| Currently the only way to combine Tasks together is in a Pipeline. If you combine | ||
| Tasks in a Pipeline and they need to share data (beyond a simple | ||
| [result](https://github.com/tektoncd/pipeline/blob/master/docs/pipelines.md#using-results)), | ||
| you'll need to provision a PVC or do some other similar, cloud specific thing, | ||
| to [make a volume available](https://github.com/tektoncd/pipeline/blob/master/docs/workspaces.md#specifying-volumesources-in-workspaces) | ||
| that can be shared between them. | ||
|
|
||
| Some problems with this: | ||
| * PVCs add additional overhead, both in speed and in management (somewhat | ||
| mitigated by | ||
| [letting tekton create and delete volumes for you](https://github.com/tektoncd/pipeline/blob/master/docs/workspaces.md#volumeclaimtemplate) | ||
| * Sometimes you just want to run a Task with a teeny bit more functionality, | ||
| e.g. you want to use a task to build images, but you want to clone source | ||
| with git first, and it seems like overkill to have to construct a pipeline | ||
|
|
||
| ### PipelineResources | ||
|
|
||
| This kind of compositional functionality was being (somewhat! read on!) provided | ||
| by [PipelineResources](https://github.com/tektoncd/pipeline/blob/master/docs/resources.md). | ||
| [We did not bring these types to beta](https://github.com/tektoncd/pipeline/blob/master/docs/resources.md#why-arent-pipelineresources-in-beta). | ||
| Issues that don't let PipelineResources (as is) solve these problems are: | ||
| * PipelineResource "outputs" do not run when the steps of the Task fail (or if | ||
| a previous PipelienResource "output" fails) (see [unit test use case](#use-cases-optional)) | ||
| * PipelineResources do not use Tasks (which is something we could change), meaning | ||
| you cannot use them to compose Tasks together, you need to build PipelineResources | ||
| * Adding new PipelineResources currently needs to be done in the Tekton Pipelines controller, | ||
| tho there have been several attempts to propose ways to make it extensible: | ||
| * [PipelineResources 2 Uber Design Doc](https://docs.google.com/document/d/1euQ_gDTe_dQcVeX4oypODGIQCAkUaMYQH5h7SaeFs44/edit) | ||
| * [Tekton Pipeline Resource Extensibility](https://docs.google.com/document/d/1rcMG1cIFhhixMSmrT734MBxvH3Ghre9-4S2wbzodPiU/edit) | ||
| * All of this is even though PipelineResources are _very_ similar to Tasks in that they are | ||
| collections of steps (which [get injected before and after a Task's steps](https://github.com/tektoncd/pipeline/issues/1838) | ||
| * Tasks have to declare in advance what PipelineResources they need; you can't decide to use | ||
| a PipelineResource with a Task after it has been written (e.g. in a TaskRun or a Pipeline) | ||
| and you can't mix types of PipelineResource (e.g. if a Task declares it needs a | ||
| [git PipelineResources](https://github.com/tektoncd/pipeline/blob/master/docs/resources.md#git-resource) | ||
| it can't use a different kind of PipelineResource to provide those files (tho you can avoid | ||
| doing the clone multiple times using [from](https://github.com/tektoncd/pipeline/blob/master/docs/pipelines.md#using-the-from-parameter)) | ||
|
|
||
| ### Goals | ||
|
|
||
| - Make it possible to combine Tasks together so that you can run multiple | ||
| Tasks as "one unit" (see [Requirements](#requirements): | ||
| - At runtime in a TaskRun | ||
| - At Pipeline authoring time (which would ultimately be realized through TaskRuns) | ||
| - Add some of [the features we don't have without PipelineResources](https://docs.google.com/document/d/1KpVyWi-etX00J3hIz_9HlbaNNEyuzP6S986Wjhl3ZnA/edit#) | ||
| to Tekton Pipelines (without requiring use of PipelineResources) | ||
|
|
||
| ### Non-Goals | ||
|
|
||
| - Composing Tasks within Tasks at [Task authoring time](https://github.com/tektoncd/community/blob/master/design-principles.md#reusability). | ||
| We could decide to include this if we have some use cases that need it; for now avoiding | ||
| this allows us to avoid many layers of nesting (i.e. Task1 uses Task2 uses Task3, etc.) | ||
| or even worse, recursion (Task 1 uses Task 2 uses Task 1...) | ||
| - One way to support this later could be via | ||
| - Supporting this composition at runtime in a PipelineRun (not quite sure what that | ||
| would look like, I guess you could refer to tasks in the Pipeline and specify | ||
| Tasks to run with them) | ||
| - Completely replacing PipelineResources: we could decide to solve this by improving PipelineResources, | ||
| or we could add a new feature via this TEP and still continue to support PipelineResources | ||
| (since they provide [more functionality than just composition](https://docs.google.com/document/d/1KpVyWi-etX00J3hIz_9HlbaNNEyuzP6S986Wjhl3ZnA/edit#)) | ||
|
|
||
| ### Use Cases (optional) | ||
|
|
||
| - A user wants to use catalog Tasks to checkout code, run unit tests and upload results, | ||
| and does not want to incur the additional overhead (and performance impact) of creating | ||
| volume based workspaces to share data between them in a Pipeline. e.g. specifically | ||
| cloning with [git-clone](https://github.com/tektoncd/catalog/tree/master/task/git-clone/0.2), | ||
| running tests with [golang-test](https://github.com/tektoncd/catalog/tree/master/task/golang-test/0.1) | ||
| and uploading results with [gcs-upload](https://github.com/tektoncd/catalog/tree/master/task/gcs-upload/0.1). | ||
| - A user wants to use a Task from the catalog with a git repo and doesn't want the extra | ||
| overhead of using a Pipeline, they just want to make a TaskRun, | ||
| e.g. [@mattmoor's feedback on PipelineResources and the Pipeline beta](https://twitter.com/mattomata/status/1251378751515922432)) | ||
| where he wants to checkout some code and [use the kaniko task](https://github.com/tektoncd/catalog/tree/master/task/kaniko/0.1) | ||
| without having to fiddle with volumes | ||
|
|
||
| ## Requirements | ||
|
|
||
| - Tasks can be composed together run as "one unit": | ||
| - Must be able to share data without requiring | ||
| an external volume (i.e. probably the containers that make them up are run within | ||
| the same pod) | ||
| - All Tasks should run even if one fails | ||
| - This is to support use cases such as uploading test results, even if the test | ||
| Task failed | ||
|
|
||
| ## References (optional) | ||
|
|
||
| * [Tekton PipelineResources Beta Notes](https://docs.google.com/document/d/1Et10YdBXBe3o2x6lCfTindFnuBKOxuUGESLb__t11xk/edit) | ||
| * [Why aren't PipelineResources in beta?](https://github.com/tektoncd/pipeline/blob/master/docs/resources.md#why-arent-pipelineresources-in-beta) | ||
| * [@mattmoor's feedback on PipelineResources and the Pipeline beta](https://twitter.com/mattomata/status/1251378751515922432)) | ||
| * [PipelineResources 2 Uber Design Doc](https://docs.google.com/document/d/1euQ_gDTe_dQcVeX4oypODGIQCAkUaMYQH5h7SaeFs44/edit) | ||
| * Task specialization: | ||
| * [Specializing Tasks - Vision & Goals](https://docs.google.com/document/d/1G2QbpiMUHSs4LOqcNaIRswcdvoy8n7XuhTV8tXdcE7A/edit) | ||
| * [Task specialization - most appealing options?](https://docs.google.com/presentation/d/12QPKFTHBZKMFbgpOoX6o1--HyGqjjNJ7own6KqM-s68/edit#slide=id.p) | ||
| * Issues: | ||
| * [Pipeline Resources Redesign](https://github.com/tektoncd/pipeline/issues/1673) | ||
| * [#1838 Extract Pre/Post Steps from PipelineResources 2 design into Tasks](https://github.com/tektoncd/pipeline/issues/1838) | ||
| * [Abstract task and nested tasks](https://github.com/tektoncd/pipeline/issues/1796) | ||
| * Oldies but goodies: | ||
| * [Link inputs and outputs without using volumes](https://github.com/tektoncd/pipeline/issues/617) | ||
| * [Design PipelineResource extensibility](https://github.com/tektoncd/pipeline/issues/238) | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters