TEP-0048 Pipeline as code

[chmouel]

We would like to propose a opiniated flow to support the "Pipeline as code"
technique with tekton.

The pipelines as code emphasizes that the configuration of delivering pipelines
that build, test and deploy our applications or infrastructure should be treated
as code; they should be placed under source control and modularized in reusable
components with automated testing and deployment.

This TEP kicks off the initial discussion on the design and the scope of what we
want to achieve.

Signed-off-by: Chmouel Boudjnah chmouel@redhat.com

[bobcatfish]

/assign

[wlynch]

/cc @wlynch

[vdemeester]

/cc @vdemeester

[tekton-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign bobcatfish
You can assign the PR to them by writing /assign @bobcatfish in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• teps/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[pritidesai]

assigning it to @skaegi

[pritidesai]

/assign skaegi

[FogDong]

Suggested change

	* We assume the user does not need access to the cluster,
	* We assume the user does not need access to the cluster.

[chmouel]

Thanks 👍🏻

[bobcatfish]

@chmouel i think the proposal section needs a lot more details before we can review it - e.g. examples of the templates we'd provide - and we'd want to explore alternatives as well

for other TEPs we've been taking an approach where we merge the the TEP without the proposal initially, so we can all agree about the problem being tackled, and then we update it with design details - what do you think about taking that approach here? from my perspective id be nearly ready to merge the problem statement part of the TEP

[bobcatfish]

something else that might be interesting to mention as motivation is that this is a feature supported by so so many other CI/CD systems

[bobcatfish]

@chmouel @FogDong I spent a bit of time looking at some other systems and whether or not most of them allow you to express both pipelines + triggering information as "code" (i.e. use version control as the source of truth) and it seems like most of them do

In this doc I took a really quick (and maybe inaccurate, plz add any corrections) look at:

• Github actions
• aws codepipeline
• azure pipeline
• jenkins (im the least sure i found the right info for this one)
• circleci
• travis
• harness
• bamboo
• gitlab

I wanted to look at more, including Openshift pipelines ( + @vdemeester @imjasonh ?) but didnt get a chance!

plz feel free to add to this / expand on it / correct it / use anything useful in this TEP

[chmouel]

Thanks for doing this @bobcatfish

Yeah a lot of this has been focused on Pull request CI (pre-merge) and not much on CD and post-merge CI.

I definitively think it's in scope, there is Jenkins-X who supports this too with lighthouse which I find interesting.

We will definitively needs to do a bit of thinking on how to integrate this with triggers.

(and fyi openshift BuildConfig, openshift pipelines doesn't support triggering as code)

[bobcatfish]

is it possible to be specific about which primitives it doesn't take care of?

[vdemeester]

In general, all of them, aka the TEP solely focuses on Pipeline def in the code being picked up. If it helps implementing GitOps operation/primites good but ain't a goal so far.

[chmouel]

yep, there is quite a lot of definitions and interpretation of GitOPS, maybe PAC (pipelines as code) can help for only a subset of it but that's not the goal

[bobcatfish]

can you explain a bit more about what assuming single tenancy means here and why it's a requirement?

(is this a duplicate of the line above?)

[vdemeester]

I think it's the initial assumption, to simplify the design a bit, @afrittoli @chmouel ?

[chmouel]

yep 👍🏻

I wasn't sure about this at first, since multi tenancy is definitively something I would like to have, but there is quite a bit to talk about on multi tenancy support with tekton and it's not necessary a first class citizen on vanilla kubernetes too.

[bobcatfish]

i dont exactly disagree with this requirement, but it seems to be specific to a particular solution (e.g. if we solved this by building support into triggers or pipelines, this might not be relevant?)

[vdemeester]

That's true but work on "pipeline as code" can start / take place in parallel of getting this in triggers (or pipeline 😓 ). I think it's a requirement to "bootstrap" the work on the topic (giving time to implement this correctly if need be in tekton components).

[chmouel]

I agree with @vdemeester

I don't think we need to make "triggering as code" a requirement for pipeline as code.

[bobcatfish]

I don't think we need to make "triggering as code" a requirement for pipeline as code.

can we expand on this a bit? i DO think that we should aim for this - @chmouel you pointed out some other solutions that do not allow triggering as code, but i strongly feel that this is where we want to get to (solutions that don't currently provide it i'm guessing would if they could - and probably will eventually)

maybe one way to avoid expanding the scope of what you are planning too far is if we could include a requirement that for whatever solutions we look at, we at least consider how they could be expanded to support triggering as code? <-- this depends on whether we want what we explore here to be the tekton recommended way of doing pipelines as code, or whether we want this to be one of many options that tekton provides. if this is "the one tekton way" then i think we need to be a lot more thorough

[vdemeester]

Tekton here is very "generic". The Tekton Pipeline component is not aware of them (and shouldn't, ever, probabl)

[chmouel]

Rephrases as "Tekton controller"

[vdemeester]

In general, all of them, aka the TEP solely focuses on Pipeline def in the code being picked up. If it helps implementing GitOps operation/primites good but ain't a goal so far.

[vdemeester]

I think it's the initial assumption, to simplify the design a bit, @afrittoli @chmouel ?

[vdemeester]

That's true but work on "pipeline as code" can start / take place in parallel of getting this in triggers (or pipeline 😓 ). I think it's a requirement to "bootstrap" the work on the topic (giving time to implement this correctly if need be in tekton components).

[vdemeester]

We will need more detailled alternative approach here at some point

[chmouel]

What kind of details would you like to see ? Alternative implementation or alternative design achieving tekton as code ?

The alternative design is I guess an ad-hoc approach, like the way we do right know which is not really tekton as code i guess.

[bobcatfish]

@vdemeester: I also think it really shouldn't be at the tektoncd/pipeline level 👼🏼

This is something where @vdemeester and I differ BUT I'm wondering if one way we could avoid having to address that difference of opinion head on is to add a requirement to this TEP (which i think might already be implied) that whatever solution we have here, it doesn't preclude other solutions: i.e. say we add a "pipeline as code" top level project, folks can choose whether to use that or not, and there could potentially be other pipeline as code solutions later in addition to that one.

If we do want to make this TEP about the one true Tekton blessed way of doing pipelines as code then maybe Vincent and I will have to discuss this more sooner

[afrittoli]

/test pull-community-teps-lint

[afrittoli]

/test pull-community-teps-lint

[afrittoli]

/test pull-community-teps-lint

[dash1291]

I'm curious how the authors visualize this happening on the dashboard. Right now there's not a lot of filters other than namespaces and labels. Are these filters going to be used to find different variants of same pipeline in the same UI or would this involve some restructuring of the UI as well?

[chmouel]

We chatted among RHT colleagues on how we wanted to present this if we wanted to expose pipeline as a code on the openshift console, which probably would be a bit the same for dashboard.

I would imagine, there would be a repository first view, where each repository can have a pipeline as a code status, with the different runs and link to logs.

Not so different from what we have on github actions and other CI i guess.

[bobcatfish]

it would be a bit of a leap for the tekton dashboard to become aware of repositories - but it would be nice! reminds me of tektoncd/dashboard#1612 which is suggesting a grouping mechanism that could group triggering info + pipelines - i think @dibyom might be planning to look into proposing a grouping mechanism like this

[chmouel]

@bobcatfish

for other TEPs we've been taking an approach where we merge the the TEP without the proposal initially, so we can all agree about the problem being tackled, and then we update it with design details - what do you think about taking that approach here? from my perspective id be nearly ready to merge the problem statement part of the TEP

I think that's the approach we wanted to take with @afrittoli, I initially drafter a lot of implementation details in this TEP and andrea convince to let this TEP be more alive so we can agree before on the problem statements and then start to do TEP updates on each implementation details.

[chmouel]

Sorry taking so much time replying on this but let's take this forward

[chmouel]

Rephrases as "Tekton controller"

[chmouel]

Thanks for doing this @bobcatfish

Yeah a lot of this has been focused on Pull request CI (pre-merge) and not much on CD and post-merge CI.

I definitively think it's in scope, there is Jenkins-X who supports this too with lighthouse which I find interesting.

We will definitively needs to do a bit of thinking on how to integrate this with triggers.

(and fyi openshift BuildConfig, openshift pipelines doesn't support triggering as code)

[chmouel]

yep, there is quite a lot of definitions and interpretation of GitOPS, maybe PAC (pipelines as code) can help for only a subset of it but that's not the goal

[chmouel]

I agree with @vdemeester

I don't think we need to make "triggering as code" a requirement for pipeline as code.

[chmouel]

yep 👍🏻

I wasn't sure about this at first, since multi tenancy is definitively something I would like to have, but there is quite a bit to talk about on multi tenancy support with tekton and it's not necessary a first class citizen on vanilla kubernetes too.

[chmouel]

Thanks 👍🏻

[chmouel]

We chatted among RHT colleagues on how we wanted to present this if we wanted to expose pipeline as a code on the openshift console, which probably would be a bit the same for dashboard.

I would imagine, there would be a repository first view, where each repository can have a pipeline as a code status, with the different runs and link to logs.

Not so different from what we have on github actions and other CI i guess.

[chmouel]

What kind of details would you like to see ? Alternative implementation or alternative design achieving tekton as code ?

The alternative design is I guess an ad-hoc approach, like the way we do right know which is not really tekton as code i guess.

[bobcatfish]

In addressing the gap around "config as code" in tekton I feel strongly that we don't want to leave triggering configuration out (at least in the long run). If the idea is to automatically pick up the content of .tekton, I'm not sure how we could leave the triggering bit out of it? (e.g. are the contents of .tekton applied on every PR?) (maybe elaborating on the requirements around the .tekton dir will help make this clear)

I wanted to point out the TEP that @sbwsg has opened recently (#389) as potentially a way to let people treat their pipelines and tasks as code - in monday's API working group he demoed a way to allow folks to reference Tasks that are stored in git, or a gcs bucket, or oci or wherever. Another thing that might help me understand the scope of this TEP would be to understand - if we did have the functionality Scott demoed, what would this TEP be adding? (Some automation around the .tekton folder it sounds like?)

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[jerop]

/lifecycle frozen

there's a working group actively making progress on this