Add workspaces support for the config-artifact-pvc configmap

[ghost]

Expected Behavior

PipelineResources currently use a PVC (or bucket) configured using the config-artifact-pvc configmap to quietly store data when it's shared between tasks. This uses an internal package called artifact_storage that silently mounts the PVC or syncs to a GCS bucket and uses that to transfer files from one task to the next. We'd like to replicate similar behaviour but have it available for the more general purpose workspaces implementation that pipelines now has.

This issue is for discussion of whether we want to support this same mechanism for controlling storage with Workspaces. The simplest, easiest UX might be for workspaces to just automatically support the configmap and "just work" when the user specifies a "legacyArtifactPVC" workspace, something like this:

# in a pipelinerun
spec:
  workspaces:
  - name: code-workspace
    legacyArtifactStorage: {}

Actual Behavior

Workspaces don't currently know about the config-artifact-pvc configmap and each workspace needs its volume to be completely specified as part of the Run it's bound in.

Additional Info

See the Workspaces volumeClaimTemplate issue for another approach to this problem. We may want both that and this, though this is also fair game for discussion here.

[vdemeester]

/kind feature

[jlpettersson]

I am trying to understand / get a clear use case here.

PipelineResources currently use a PVC (or bucket) configured using the config-artifact-pvc configmap to quietly store data when it's shared between tasks.

Here it sounds like the use case is for handling cached dependencies/artifiacts. Or is the use case for using a workspace to pass files between tasks without a PVC?

This uses an internal package called artifact_storage that silently mounts the PVC or syncs to a GCS bucket and uses that to transfer files from one task to the next.

Here it sounds like a use case requiring both GCS bucket and a PVC. The use case could also be to use GCS bucket in the pipeline - have a task that syncs data from bucket to an existing workspace in the task/pipeline, and also a task for uploading data?

[ghost]

Ah, apologies, I see where the confusion comes from. The intended use case is passing files between Tasks in a Pipeline. PipelineResources have this existing behaviour: an output resource in Task A of a Pipeline can be "linked" to an input resource of Task B in the same Pipeline and the files of the PipelineResource will "magically" be copied between the Tasks when the PipelineRun executes. A PVC or a GCS bucket is used as the medium by which the files are transferred between the Tasks in the Pipeline. The configuration for the PVC or GCS bucket is stored in the config-artifact-pvc ConfigMap in the tekton-pipelines namespaces.

This issue exists to discuss whether we provide support in Workspaces for the existing config-artifact-pvc to help users transition from PipelineResources to Workspaces. The YAML for a workspace using this legacy copy system might look like this:

# in a pipelinerun
spec:
  workspaces:
  - name: code-workspace
    legacyArtifactStorage: {}

On seeing this workspace the reconciler could run the same code path as it would were it "linking" PipelineResources, and copy files through that medium.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[vdemeester]

/remove-lifecycle rotten
/remove-lifecycle stale
/reopen

[tekton-robot]

@vdemeester: Reopened this issue.

In response to this:

/remove-lifecycle rotten
/remove-lifecycle stale
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[vdemeester]

/remove-lifecycle stale

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[ghost]

I'm going to let this one rot. We've got several designs supporting inter-Task storage going on at the moment which don't rely on the legacy pipeline resources artifact configuration stuff.

[ghost]

/close

[tekton-robot]

@sbwsg: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.