diff --git a/teps/0044-composing-tasks-with-tasks.md b/teps/0044-decouple-task-composition-from-scheduling.md similarity index 85% rename from teps/0044-composing-tasks-with-tasks.md rename to teps/0044-decouple-task-composition-from-scheduling.md index e9958d919..b4accda94 100644 --- a/teps/0044-composing-tasks-with-tasks.md +++ b/teps/0044-decouple-task-composition-from-scheduling.md @@ -1,91 +1,64 @@ --- status: proposed -title: Composing Tasks with Tasks +title: Decouple Task Composition from Scheduling creation-date: '2021-01-22' -last-updated: '2021-01-22' +last-updated: '2021-02-08' authors: - '@bobcatfish' --- -# TEP-0044: Composing Tasks with Tasks +# TEP-0044: Decouple Task Composition from Scheduling - [Summary](#summary) - [Motivation](#motivation) - - [PipelineResources](#pipelineresources) - [Goals](#goals) - [Non-Goals](#non-goals) - [Use Cases (optional)](#use-cases-optional) - [Overlap with TEP-0046](#overlap-with-tep-0046) - [Requirements](#requirements) - [References (optional)](#references-optional) + - [PipelineResources](#pipelineresources) ## Summary -This TEP describes a gap in composability that isn't currently fully solved by Tasks or Pipelines. - -Ideally by addressing the issues described here we'd make it possible to create more efficient -Pipelines and make it easier to use a Task easily via a TaskRun, and we'd address some of -[the problems PipelineResources are trying to solve](https://docs.google.com/document/d/1KpVyWi-etX00J3hIz_9HlbaNNEyuzP6S986Wjhl3ZnA/edit#) -making it more clear whether or not they are an abstraction we want to keep. - -[#1673: Pipeline Resources Redesign](https://github.com/tektoncd/pipeline/issues/1673) +This TEP addresses how the current concept of a Task embodies both a reusable unit of logic but is also a unit used to +schedule execution. ## Motivation -Currently the only way to combine Tasks together is in a Pipeline. If you combine -Tasks in a Pipeline and they need to share data (beyond a simple -[result](https://github.com/tektoncd/pipeline/blob/master/docs/pipelines.md#using-results)), +* Tasks are a reusable unit combining one or more steps (container images) together and defining an interface to those + steps via params, workspaces and results +* Tasks can be combined in a Pipeline +* When a Pipeline is invoked via a PipelineRun, each Task is executed as a separate pod. + +This means that choices made around Task design (e.g. creating a Task that encapsulates a git clone and a separate Task +to run go unit tests) directly impact the performance and overhead involved in executing the Tasks. For example +if the git clone task wants to share data with the unit test task, beyond a simple +[result](https://github.com/tektoncd/pipeline/blob/master/docs/pipelines.md#using-results), it needs to you'll need to provision a PVC or do some other similar, cloud specific storage, to [make a volume available](https://github.com/tektoncd/pipeline/blob/master/docs/workspaces.md#specifying-volumesources-in-workspaces) -that can be shared between them. - -PVCs add additional overhead, both in speed and in management (somewhat -mitigated by [letting tekton create and delete volumes for you](https://github.com/tektoncd/pipeline/blob/master/docs/workspaces.md#volumeclaimtemplate) -but the PVCs still slow down the overall execution. - -### PipelineResources - -This kind of compositional functionality was being (somewhat! read on!) provided -by [PipelineResources](https://github.com/tektoncd/pipeline/blob/master/docs/resources.md). -[We did not bring these types to beta](https://github.com/tektoncd/pipeline/blob/master/docs/resources.md#why-arent-pipelineresources-in-beta). -Issues that don't let PipelineResources (as is) solve these problems are: -* PipelineResource "outputs" do not run when the steps of the Task fail (or if - a previous PipelienResource "output" fails) (see [unit test use case](#use-cases-optional)) -* PipelineResources do not use Tasks (which is something we could change), meaning - you cannot use them to compose Tasks together, you need to build PipelineResources - * Adding new PipelineResources currently needs to be done in the Tekton Pipelines controller, - tho there have been several attempts to propose ways to make it extensible: - * [PipelineResources 2 Uber Design Doc](https://docs.google.com/document/d/1euQ_gDTe_dQcVeX4oypODGIQCAkUaMYQH5h7SaeFs44/edit) - * [Tekton Pipeline Resource Extensibility](https://docs.google.com/document/d/1rcMG1cIFhhixMSmrT734MBxvH3Ghre9-4S2wbzodPiU/edit) - * All of this is even though PipelineResources are _very_ similar to Tasks in that they are - collections of steps (which [get injected before and after a Task's steps](https://github.com/tektoncd/pipeline/issues/1838) -* Tasks have to declare in advance what PipelineResources they need; you can't decide to use - a PipelineResource with a Task after it has been written (e.g. in a TaskRun or a Pipeline) - and you can't mix types of PipelineResource (e.g. if a Task declares it needs a - [git PipelineResources](https://github.com/tektoncd/pipeline/blob/master/docs/resources.md#git-resource) - it can't use a different kind of PipelineResource to provide those files (tho you can avoid - doing the clone multiple times using [from](https://github.com/tektoncd/pipeline/blob/master/docs/pipelines.md#using-the-from-parameter)) +that can be shared between them, and running the second Task will be delayed by the overhead of scheduling a second pod. ### Goals - Make it possible to combine Tasks together so that you can run multiple - Tasks as "one unit" (see [Requirements](#requirements) at authoring time + Tasks together and have control over the scheduling overhead (i.e. pods and volumes required) at authoring time in a way that can be reused (e.g. in a Pipeline) - Add some of [the features we don't have without PipelineResources](https://docs.google.com/document/d/1KpVyWi-etX00J3hIz_9HlbaNNEyuzP6S986Wjhl3ZnA/edit#) to Tekton Pipelines (without requiring use of PipelineResources), specifically **Task adapters/specialization** - - If possible we can provide a migration path from PipelineResources to the solution proposed here, where applicable ### Non-Goals -- Composing Tasks within Tasks at [Task authoring time](https://github.com/tektoncd/community/blob/master/design-principles.md#reusability). +- Updating the Task CRD to allow Tasks to reference other Tasks + at [Task authoring time](https://github.com/tektoncd/community/blob/master/design-principles.md#reusability). We could decide to include this if we have some use cases that need it; for now avoiding this allows us to avoid many layers of nesting (i.e. Task1 uses Task2 uses Task3, etc.) or even worse, recursion (Task 1 uses Task 2 uses Task 1...) - Completely replacing PipelineResources: we could decide to solve this by improving PipelineResources, or we could add a new feature via this TEP and still continue to support PipelineResources (since they provide [more functionality than just composition](https://docs.google.com/document/d/1KpVyWi-etX00J3hIz_9HlbaNNEyuzP6S986Wjhl3ZnA/edit#)) -- This was previously a use case we were targetting but we've decided to descope the TEP slightly, +- This was previously a use case we were targeting but we've decided to descope the TEP slightly, though if we end up solving this problem as well, that's a bonus: * A user wants to use a Task from the catalog with a git repo and doesn't want the extra overhead of using a Pipeline, they just want to make a TaskRun, @@ -101,9 +74,11 @@ Issues that don't let PipelineResources (as is) solve these problems are: cloning with [git-clone](https://github.com/tektoncd/catalog/tree/master/task/git-clone/0.2), running tests with [golang-test](https://github.com/tektoncd/catalog/tree/master/task/golang-test/0.1) and uploading results with [gcs-upload](https://github.com/tektoncd/catalog/tree/master/task/gcs-upload/0.1). -- An organziation does not want to use PVCs at all; for example perhaps they have decided +- An organization does not want to use PVCs at all; for example perhaps they have decided on uploading to and downloading from buckets in the cloud (e.g. GCS) - An organization is willing to use PVCs to some extent but needs to put limits on their use +- A user has decided that the overhead required in spinning up multiple pods is too much and wants to be able to have + more control over this. ### Overlap with TEP-0046 @@ -144,8 +119,9 @@ Where they differ: ## Requirements -- Tasks can be composed together run as "one unit": +- Tasks can be composed and run together: - Must be able to share data without requiring a volume external to the pod + - Must be possible to run multiple Tasks as one pod - It should be possible to have Tasks that run even if others fail; i.e. the Task can be run on the same pod as another Task that fails - This is to support use cases such as uploading test results, even if the test @@ -171,3 +147,26 @@ Where they differ: * Oldies but goodies: * [Link inputs and outputs without using volumes](https://github.com/tektoncd/pipeline/issues/617) * [Design PipelineResource extensibility](https://github.com/tektoncd/pipeline/issues/238) + +### PipelineResources + +This kind of compositional functionality was being (somewhat! read on!) provided +by [PipelineResources](https://github.com/tektoncd/pipeline/blob/master/docs/resources.md). +[We did not bring these types to beta](https://github.com/tektoncd/pipeline/blob/master/docs/resources.md#why-arent-pipelineresources-in-beta). +Issues that don't let PipelineResources (as is) solve these problems are: +* PipelineResource "outputs" do not run when the steps of the Task fail (or if + a previous PipelienResource "output" fails) (see [unit test use case](#use-cases-optional)) +* PipelineResources do not use Tasks (which is something we could change), meaning + you cannot use them to compose Tasks together, you need to build PipelineResources + * Adding new PipelineResources currently needs to be done in the Tekton Pipelines controller, + tho there have been several attempts to propose ways to make it extensible: + * [PipelineResources 2 Uber Design Doc](https://docs.google.com/document/d/1euQ_gDTe_dQcVeX4oypODGIQCAkUaMYQH5h7SaeFs44/edit) + * [Tekton Pipeline Resource Extensibility](https://docs.google.com/document/d/1rcMG1cIFhhixMSmrT734MBxvH3Ghre9-4S2wbzodPiU/edit) + * All of this is even though PipelineResources are _very_ similar to Tasks in that they are + collections of steps (which [get injected before and after a Task's steps](https://github.com/tektoncd/pipeline/issues/1838) +* Tasks have to declare in advance what PipelineResources they need; you can't decide to use + a PipelineResource with a Task after it has been written (e.g. in a TaskRun or a Pipeline) + and you can't mix types of PipelineResource (e.g. if a Task declares it needs a + [git PipelineResources](https://github.com/tektoncd/pipeline/blob/master/docs/resources.md#git-resource) + it can't use a different kind of PipelineResource to provide those files (tho you can avoid + doing the clone multiple times using [from](https://github.com/tektoncd/pipeline/blob/master/docs/pipelines.md#using-the-from-parameter)) diff --git a/teps/README.md b/teps/README.md index 8409e593c..ce56e1a02 100644 --- a/teps/README.md +++ b/teps/README.md @@ -169,7 +169,7 @@ This is the complete list of Tekton teps: |[TEP-0037](0037-remove-gcs-fetcher.md) | Remove `gcs-fetcher` image | implementing | 2021-01-27 | |[TEP-0039](0039-add-variable-retries-and-retrycount.md) | Add Variable `retries` and `retry-count` | proposed | 2021-01-31 | |[TEP-0040](0040-ignore-step-errors.md) | Ignore Step Errors | proposed | 2021-02-04 | -|[TEP-0044](0044-composing-tasks-with-tasks.md) | Composing Tasks with Tasks | proposed | 2021-01-22 | +|[TEP-0044](0044-decouple-task-composition-from-scheduling.md) | Decouple Task Composition from Scheduling | proposed | 2021-01-22 | |[TEP-0045](0045-whenexpressions-in-finally-tasks.md) | WhenExpressions in Finally Tasks | implementable | 2021-01-28 | |[TEP-0050](0050-ignore-task-failures.md) | Ignore Task Failures | proposed | 2021-02-19 | |[TEP-0051](0051-ppc64le-architecture-support.md) | ppc64le Support | proposed | 2021-01-28 |