parallel / serial execution policies in build v2

[gabemontero]

Today build v1 allows for specification of these run policies at the build level

• parallel
• pure serial
• no-op'ing intermediate build request while serial that have queued up, and only run the latest request

Tekton has

• specification of dependencies between tasks in a pipeline, so one task has to run after another .. so a form of serialization at that level
• pipeline executions always run in parallel

build v2 should surface some flavor / combinations of these in its API ... either leverage the tekton features as is, or add build v1 type controls in the build v2 controller.

[sbose78]

Gabe, are you talking about providing a way to specify

• Running Builds in parallel/Serial, or
• Running BuildSteps in a Build in parallel/serial ?

[gabemontero]

Builds in parallel / serial is what buildv1 has today and what I'm referring to

Is it possible to provide the same with tekton as a base given what it provides? Or perhaps the question is "how expensive" would it be to do it given tekton as a base vs. "how desired" such a feature is ?

Given what tekton provides, feels like it would have to be achieved at the buildv2 operator level, where it monitors the state of each buildv2 build under a buildv2 buildconfig (assuming analogous API concepts in buildv2).

Right now I only see build and build strategy in https://github.com/redhat-developer/build/blob/master/pkg/apis/build/v1alpha1/build_types.go .. no "buildconfig"

So how would you group "builds" such that you decide if they run in parallel or serial?

Maybe the build specs get an "identifier" and you group different builds based on the fact that they use the same build spec?

stuff to sort out if it is decided buildv2 supports this .... otherwise this falls into the buildv2 vs buildv1 comparison doc page.

[adambkaplan]

The need for this is related to our trigger stories (#51, #52). Otherwise it is safer for us to assume users can control when builds get executed.

[qu1queee]

@gabemontero take for example the following scenario:

A user defines a Build X. Then the same user cooks two BuildRuns that reference Build X.

from the above, is up to the user to define when should those BuildRuns get trigger ( same as @adambkaplan mentioned). The user can opt for a serial approach or a parallel approach by some sort of means. E.g. in https://github.com/homeport/build-load buildruns are triggered in parallel.

[gabemontero]

yep @qu1queee .... and fwiw @adambkaplan 's #56 (comment) stemmed from a team meeting we had to discuss this particular item and how best, if at all, shipwright provides something equivalent

most likely then when work on those trigger related stories evolve, they'll consider the parallel/serial element when a user is not directly responsible triggering buildruns, and when they do, we can close this issue in favor of where those are headed

[gabemontero]

Update: upstream tekton, as part of moving toward v1 of the API, have registered issues 3563 and 3052 (I am intentionally avoiding putting an actual link to avoid cross referencing for now to avoid clutter on the tekton side) that are in this space

[qu1queee]

Can we close this issue in favor of #56 (comment) and #56 (comment) ? Or do we want to keep this as a reminder of the ongoing Tekton efforts mentioned in #56 (comment) ?

[gabemontero]

Yeah let's keep this as a reminder i.e. monitoring tekton. FWIW it is already in the POST GA milestone

[gabemontero]

So update on this one since the end of 2020 / beginning on 2021.

After following literally 6 or 7 opened and close links in upstream tekton, the current open / upstream tekton TEP in this space is TEP 44, decouple task composition from scheduling https://github.com/tektoncd/community/blob/main/teps/0044-decouple-task-composition-from-scheduling.md

However, a Task / Pipeline based solution of course does not work for Shipwright, given we only leverage TaskRun with tekton right now. We are considering CustomTasks for some integration with various external components with shipwright builds, we have no plans at this time to move off of a TaskRun only approach for the basic build strategy - build - build run flow.

So at this time, possible solutions here do not include IMO direct upstream tekton exploitation, and only include

• the shipwright community building a solution as part of the shipwright offering
• or consumers of shipwright employing their own approach with associated tooling, including some of the approaches noted by other community members in this issue

We do have a burgeoning blog post set that describes third party integrations, like what was recently done for leveraging OPA gatekeeper. So depending on how much this comes up as consumption of shipwright increases, firming up and documenting how to achieve desired levels of serial or parallel execution by what you define the existing build strategy / build / buildrun framework feels like a logical first step before taking on implementing something.

But of course the folks subscribed here chime in and discuss as you desire :-)