SHIP-0022: Build Strategy Volumes

This proposal will allow build strategy authors to declare volumes which
can be shared across build steps and build runs. The build strategy will
declare the mount point for the volume which will be fixed across all
build steps. Volumes can either have a fixed volume source, or an
overridable volume source that can be set in a `Build` or `BuildRun`.

Volumes declared in a build strategy are translated into the following:

1. Volume mounts for each task step that has a corresponding build step
2. A volume that is added to the TaskRun's pod template

This feature will replace the implict creation of emptyDir volumes in build
strategies. Build strategies will need to explicitly declare emptyDir
volumes moving forward.

Tekton's Workspaces feature was purposefully rejected in this proposal,
the reasons outlined in the "Alternatives" section. Other drawbacks and
notable documentation requirements were also outlined.

ships/0022-build-strategy-volumes.md

@@ -0,0 +1,208 @@
+<!--
+Copyright The Shipwright Contributors
+
+SPDX-License-Identifier: Apache-2.0
+-->
+
+---
+title: build-strategy-volumes
+authors:
+  - "@adambkaplan"
+reviewers:
+  - "@otaviof"
+  - "@HeavyWombat"
+  - "@imjasonh"
+approvers:
+  - "@sbose78"
+  - "@SaschaSchwarze0"
+creation-date: 2021-08-18
+last-updated: 2021-10-05
+status: implementable
+see-also: []
+replaces: []
+superseded-by: []
+---
+
+# Build Strategy Volumes
+
+## Release Signoff Checklist
+
+- [x] Enhancement is `implementable`
+- [x] Design details are appropriately documented from clear requirements
+- [x] Test plan is defined
+- [x] Graduation criteria for dev preview, tech preview, GA
+- [ ] User-facing documentation is created in [docs](/docs/)
+
+## Open Questions [optional]
+
+TBD
+
+## Summary
+
+This proposal will allow build strategy authors to declare volumes which can be shared across build steps and build runs.
+The build strategy will declare the mount point for the volume which will be fixed across all build steps.
+Volumes can either have a fixed volume source, or an overridable volume source that can be set in a `Build` or `BuildRun`.
+
+## Motivation
+
+Shipwright builds have two limitations that the build APIs do not address:
+
+1. Builds cannot use persistent storage
+2. Build strategy authors cannot share information across build steps, outside of the cloned source code.
+
+Persistent storage is a core primitive that is needed to add caching mechanisms to builds.
+Caches for container image layers and dependency artifacts are commonly used to improve the performance of application builds.
+
+### Goals
+
+- Build strategies can declare volumes whose data can be reused across steps.
+- Developers can use persistent volumes in a `BuildRun`.
+
+### Non-Goals
+
+- Securing volume usage/access with admission control.
+- Allow developers to inject arbitrary volumes into `BuildRun`s.
+
+## Proposal
+
+### User Stories
+
+#### Share information in build steps
+
+As a build strategy author
+I want to declare volumes in my build steps
+So that I can share information across build steps
+
+#### Use persistent volumes in build runs
+
+As a developer
+I want to declare persistent volume claims in my Builds
+So that I can cache resources like image layers and artifacts for subsequent BuildRuns
+
+### Implementation Notes
+
+The `BuildStrategy` API will be enhanced so that volumes can be declared alongside the build strategy steps.
+The volumes will provide fixed mount points that will be used in all steps, along with a default source for the underlying filesystem.
+Volumes can be marked `optional` indicating that the volume is not needed for a `BuildRun` to execute.
+Volumes can also indicate if the source can be overrode by a `Build` or `BuildRun`.
+
+Both the `Build` and `BuildRun` APIs will be enhanced to allow volumes to be specified.
+The volume name referenced in the `Build` or `BuildRun` must be declared in the build strategy.
+
+#### Deprecate Implicit emptyDir Volumes
+
+Shipwright currently creates an implicit `emtpyDir` volume if one or more build steps declare a volume mount.
+This behavior should be deprecated as a prerequsite to releasing this feature.
+Implicit emptyDir volumes can then be removed when this feature is released.
+
+#### Strategy Volumes API
+
+`BuildStrategy`:
+
+```yaml
+metadata:
+  name: cached-docker-build
+spec:
+  buildSteps:
+  - name: build
+    image: quay.io/my-org/my-builder:latest
+    ...
+  volumes:
+  - name: build-metadata
+    description: "Build metadata"
+    mountPath: /home/build/metadata # mount path for every build step
+    volumeSource:
+      emptyDir: {}
+  - name: image-cache
+    description: "Container image cache"
+    optional: true
+    mountPath: /var/lib/containers/cache
+    volumeSource:
+      overridable: true # indicates the volume source can be different in a Build or BuildRun
+      emptyDir: {}
+```
+
+`Build` and `BuildRun`:
+
+```yaml
+spec:
+  ...
+  strategy:
+    name: cached-docker-build
+    kind: BuildStrategy
+  volumes:
+  - name: image-cache
+    volumeSource:
+      persistentVolumeClaim:
+        name: pvc-image-cache
+```
+
+#### TaskRun Generation
+
+When Shipwright generates the `TaskRun` for the `BuildRun`, volumes are created as follows:
+
+0. For each volume in the build strategy, a corresponding `volumeMount` is added to each build step.
+1. For each volume in the build strategy, a corresponding `volume` is created in the TaskRun's [pod template](https://tekton.dev/docs/pipelines/taskruns/#specifying-a-pod-template).
+2. If the `Build` or `BuildRun` specify a different volume source than what is declared in the build strategy, the volume source in the pod template is updated accordingly, with the `BuildRun` taking precedence over the `Build`.
+3. If the `BuildRun` attempts to override a volume whose source does not have `overridable: true`, the `BuildRun` should fail.
+
+### Test Plan
+
+Testing will ensure that basic scenarios are covered:
+
+1. Volume mounting Secrets and ConfigMaps
+2. Volume mounting Persistent Volumes
+3. Verifying BuildRuns succeed or fail of the `overridible` attribute is set to true/false.
+
+### Release Criteria
+
+#### Removing a deprecated feature [if necessary]
+
+The implicit generation of `emptyDir` volumes needs to be deprecated.
+Afterwards, this feature can be introduced and the implicit generation of `emptyDir` volumes can be removed.
+
+#### Upgrade Strategy [if necessary]
+
+On upgrade, the volume feature is enabled by default, and the implicit creation of `emptyDir` volumes will be removed.
+
+### Risks and Mitigations
+
+Security is a concern with volumes, especially if arbitrary `HostPath` volume mounts are allowed in the API.
+The [Pod Security Admission plugin](https://kubernetes.io/docs/concepts/security/pod-security-admission/)
+is a means to mitigate this issue, as it allows risky volume mounts to be blocked per namespace.
+Shipwright builds should document how this admission plugin and the volumes feature interact.
+
+## Drawbacks
+
+Volumes support adds complexity to the API that may require developers to fully inspect the referenced build strategies.
+We have assumed with other proposals that developers should not be expected to interact fully with `BuildStrategy` objects.
+That said, volumes could be grouped with build strategy parameters as features that an "intermediate" user of Shipwright would interact with.
+
+This current proposal also assumes that every volume would be present in every build step.
+This shares the same drawback as Tekton's workspaces feature.
+Tekton has an [alpha feature](https://tekton.dev/docs/pipelines/workspaces/#isolated-workspaces) which allows workspaces to be isolated to specific task steps.
+In theory we could pursue a similar feature if that proves useful in upstream Tekton.
+
+## Alternatives
+
+We could implement similar capabilities by exposing Tekton's [Workspaces](https://tekton.dev/docs/pipelines/workspaces/) feature set directly.
+This would potentially leak our abstraction of Tekton, breaking our practice of keeping Tekton an implementation detail.
+Tekton workspaces also allow the [default filesystem](https://tekton.dev/docs/pipelines/workspaces/#setting-a-default-taskrun-workspace-binding) for a workspace to be configured cluster-wide.
+If a Shipwright user wanted to change the default workspace filesystem type, they would need to change Tekton's configuration.
+
+This proposal purposefully doesn't use Tekton workspaces for the following reasons:
+
+- Workspaces only support limited types of volumes (`secret`, `configMap`, `pvc` and equivalent).
+  Other volume types may be desired for build strategies that are granted elevated privileges, such as `HostPath`.
+- Workspaces allow the volume source type to be deferred, and the default type to be configured per cluster.
+  This would make debugging difficult if Shipwright build strategies assumed the volume defaulted to an `emptyDir`,
+  but a cluster configured Tekton to provision PersistentVolumes by default instead.
+
+## Infrastructure Needed [optional]
+
+No new infrastructure.
+
+## Implementation History
+
+2021-08-18: Provisional SHIP proposal
+2021-10-05: Updated to implementable