Support to mark a PVC/workspace as "secondary"

[jlpettersson]

Feature request

Add a field so that a PVC/workspace can be marked as "secondary" - so that it would be possible to use more than one PVC for Tasks when using the Affinity Assistant or the custom scheduler #3052

Use case

It is not working well to support more than one PVC per TaskRun in a Pipeline in a regional cluster. The current implementation of the Affinity Assistant also prohibit this up front. The custom scheduler that eventually should replace the Affinity Assistant also must use the same constraint - to avoid deadlocks when volumes are in different AZs.

But by marking additional PVCs as "secondary" - the Affinity Assistant and the custom scheduler could ignore those PVCs. This "field" should probably be in the WorkspaceBinding.

[imjasonh]

This sounds like something that would need a TEP to fully design it.

It's unclear to me what the semantics of secondary are in the case where a task wants to mount both a source and configuratiton PVC workspace -- which one should the user choose as "secondary", and how would they make that decision?

Does "secondary" mean effectively "mount the other one first, then try to mount this one (or many secondary workspaces)"? Do we expect a "tertiary" option, or some general notion of workspace priority?

(Sorry if these are dumb questions, I don't think I understand how this would help with workspace PVC scheduling)