"Build Number" for PIpelineRuns scoped to the Pipeline

[bigkevmcd]

Feature request

Most Integration Services have a concept of a sequential, human-readable build number.

Jenkins has BUILD_NUMBER https://wiki.jenkins.io/display/JENKINS/Building+a+software+project#Buildingasoftwareproject-belowJenkinsSetEnvironmentVariables
Travis has TRAVIS_BUILD_NUMBER https://docs.travis-ci.com/user/environment-variables/
Circle has CIRCLE_BUILD_NUM https://circleci.com/docs/2.0/env-vars/#built-in-environment-variables
Bamboo has bamboo.buildNumber https://confluence.atlassian.com/bamboo/bamboo-variables-289277087.html

This sort of functionality is not currently available in PipelineRuns, and while, timestamps are useful, humans like numbers that they can relate to.

I spoke to a Jenkins user recently, and they use the build-number for tagging artifacts (including image builds), but there's really no similar functionality in Tekton.

Use case

Being able to use something like $(context.pipelineRun.buildNumber) in an environment variable would make this easier.

Presumably the user would control when the sequence resets, and arguably be able to set a specific value on the sequence (similar to database sequences).

[vdemeester]

To summarize my views on this (based on multiple discussions) : build number sequences are a "user" concept aka what makes a set of pipelinerun or taskrun part of a given sequence is something the user defines, not tektoncd/pipeline.

That said, as of today, it is not possible to emulate that behavior with Tekton, even with tektoncd/community#147 (integration TEP), we need few things to allow such a construct.

• Exposing more information as variables, in context : metadata.uid is a really useful one as it defines the unique uid (set by k8s) of the given *Run.
• Having a way to pass runtime information and annotation as environment variables (to mimic other tools like Jenkins, travis, …). This is tracked by Provide mechanism to set environment variables on TaskRuns #1606 (comment)

With those two "fixed", an integration (in the sense of the proposed TEP) or a webhook (or something else) could be responsible for providing those "build number" based on user rules.

I spoke to a Jenkins user recently, and they use the build-number for tagging artifacts (including image builds), but there's really no similar functionality in Tekton.

If you are able to pass the "build number" as parameter to you task, then it is possible. The core question is, who is responsible for defining the sequence. And there, I think it's another component than tektoncd/pipeline itself (same as triggers for event handling for example). That component could be a tektoncd project too 😛

[bobcatfish]

Exposing more information as variables, in context : metadata.uid is a really useful one as it defines the unique uid (set by k8s) of the given *Run.

I like the idea of providing a uid - i think the uid feature in triggers has been really useful and I've run into a few places where I've wanted to use it as well.

Particularly recently trying to create a task that does what our end to end tests do:

# Build a resource name based on $E2E_BASE_NAME, a suffix and $BUILD_NUMBER.
# Restricts the name length to 40 chars (the limit for resource names in GCP).
# Name will have the form $E2E_BASE_NAME-<PREFIX>$BUILD_NUMBER.
# Parameters: $1 - name suffix
function build_resource_name() {
  local prefix=${E2E_BASE_NAME}-$1
  local suffix=${BUILD_NUMBER}
  # Restrict suffix length to 20 chars
  if [[ -n "${suffix}" ]]; then
    suffix=${suffix:${#suffix}<20?0:-20}
  fi
  local name="${prefix:0:20}${suffix}"
  # Ensure name doesn't end with "-"
  echo "${name%-}"
}

They use the BUILD_NUMBER from prow to create a unique resource name: but here is where I want to introduce some caution and suggest we want to be careful with this.

If you look at the prow announcements you can see:

November 6, 2017 plank:0.55 provides Pods with the $BUILD_ID variable as well as the $BUILD_NUMBER variable. The latter is deprecated and will be removed in a future version.

There's an indication that BUILD_NUMBER is deprecated and would be removed, back in 2017. Note that in 2020 we are using this today and also there is really no way that Prow can know who is using that variable and for what; and providing it as an environment variable makes this harder b/c it's impossible to say how many scripts/tools/images have been build assuming that variable will exist.

So I want to make sure we are careful not to make things too auto magical and as much as possible enforce Tasks being clear about what they are using and where i came from - which I think something like metadata.uid would allow!

[vdemeester]

@bobcatfish k8s do create a metadata.uid when a resource is created 🙃

[bigkevmcd]

In my head, I was envisaging some sort of resource that incremented, and that a PipelineRun could reference to get a number from.

I do think that it could be done via an external integration, perhaps annotating the PipelineRun, assuming that we could make annotations available to the context (to be exposed for use), I guess my only concern is that there could be a race, between the external component adding the annotation, and the value being required.

[bobcatfish]

That sounds almost like something that could be the job of the thing creating the PipelineRun, and could be reflected in both the name of the PipelineRun and potentially the parameter.

If we felt like it was the tekton controller's job to keep track of this, it would have to start storing a count of how many runs of each pipeline have occurred, and if we scaled out the controller, they'd need to coordinate between each other to determine the number.

Maybe you can provide more info on your requirements: does it need to be an incrementing number or could a unique id + a timestamp do the trick?

[dlorenc]

I agree with @bobcatfish - an incrementing number is very hard to get right in a distributed system. I think we can probably make a simpler form of the uid that might be easier to read/remember.