E2E Tests are always run from the master branch

[barkbay]

When running e2e tests against a branch we should use the e2e tests from this specific branch.

Otherwise some E2E tests in master may fail because they require some features which are not available in the tested branch.

This is the root cause of #3055

[thbkrkr]

I think we can make a branch specifier (git commit/branch/tag) parameter at the job level to use it for the git checkout in the e2e tests jobs jjbb definitions:

     parameters:
+      - string:
+        name: BRANCH_SPECIFIER
+        default: master
+        description: Git branch specifier (<branchName>,
+          <tagName>, <gitCommit>, etc.)
       - string:
           name: JKS_PARAM_OPERATOR_IMAGE
           description: "ECK Docker image"
@@ -16,7 +21,7 @@
         - git:
             url: https://github.com/elastic/cloud-on-k8s
             branches:
-              - master
+              - $BRANCH_SPECIFIER

And we propagate the git commit in the build (release and nightly snapshots) pipeline that triggers these jobs:

                 build job: 'cloud-on-k8s-e2e-tests-xxx',
-                    parameters: [string(name: 'JKS_PARAM_OPERATOR_IMAGE', value: operatorImage)],
+                    parameters: [
+                        string(name: 'JKS_PARAM_OPERATOR_IMAGE', value: operatorImage),
+                        string(name: 'BRANCH_SPECIFIER', value: GIT_COMMIT)
+                    ],
                     wait: false

[sebgl]

+1, it's nice since it means we can also manually run those job and specify the git commit we want to use.

[thbkrkr]

Quick-update: the previous diff I wrote is wrong as it only concerns the sha1 to checkout the Jenkinsfile. We need to work at the pipeline level (the Jenkinsfile) and to override the default checkout, thing we already did to have a retry. In short, this is not so easy.

[thbkrkr]

Actually it can work but only if we switch lightweight-checkout to false. With lightweight-checkout set to true, the Jenkinsfile is retrieved using the API while false implies a git clone.
So the downside of this is that we add a git clone.

An alternative to avoid the extra git clone is to explicitly call the checkout directive to pass it the parametrized branch specifier:

checkout([$class: 'GitSCM', 
    branches: [[name: params.branch_specifier ]],
    extensions: [[$class: 'WipeWorkspace']],
    userRemoteConfigs: [[
        credentialsId: 'f6c7695a-671e-4f4f-a331-acdce44ff9ba',
        url: 'https://github.com/elastic/cloud-on-k8s',
        refspec: params.branch_specifier
    ]]
])

Whatever the solution chosen, we can reduce the number of checkouts. We can use stash/unstash in the jobs where we run stages in parallel with one checkout scm by stage (the kind, stack-versions, snapshots and gke jobs). Moreover I think we have some extra unnecessary checkout in the ocp, eks and aks jobs as there is already a default checkout.

[thbkrkr]

Let's start with the simpler solution despite the extra git clone.
I will remove the extra unnecessary checkout and reduce the number of checkouts in separate pull requests.