Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding version overrides ENVs for e2e and individual tools #46763

Merged
merged 1 commit into from
Dec 15, 2023
Merged

Adding version overrides ENVs for e2e and individual tools #46763

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions ...step-registry/openshift-qe/cluster-density-v2/openshift-qe-cluster-density-v2-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,10 @@ source ./venv_qe/bin/activate
ES_PASSWORD=$(cat "/secret/password")
ES_USERNAME=$(cat "/secret/username")


git clone https://github.com/cloud-bulldozer/e2e-benchmarking --depth=1
REPO_URL="https://github.com/cloud-bulldozer/e2e-benchmarking";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we add this repo url as an env variable as well so that in the future we can specify it in the config files if we ever want to test a pr/someone else repo/branch

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Testing PRs is out of scope for this change. It needs to be taken through this JIRA: https://issues.redhat.com/browse/PERFSCALE-2731
  • And the idea is to use only tags for testing but not branches going forward as they are error prone. (because as soon as a PR gets merged branch gets updated with those changes whereas release tag does not)

LATEST_TAG=$(curl -s "https://api.github.com/repos/cloud-bulldozer/e2e-benchmarking/releases/latest" | jq -r '.tag_name');
TAG_OPTION="--branch $(if [ "$E2E_VERSION" == "default" ]; then echo "$LATEST_TAG"; else echo "$E2E_VERSION"; fi)";
git clone $REPO_URL $TAG_OPTION --depth 1
pushd e2e-benchmarking/workloads/kube-burner-ocp-wrapper
export WORKLOAD=cluster-density-v2

Expand Down
6 changes: 5 additions & 1 deletion ...or/step-registry/openshift-qe/cluster-density-v2/openshift-qe-cluster-density-v2-ref.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,12 @@ ref:
name: ocp-qe-perfscale-ci
tag: latest
env:
- name: E2E_VERSION
default: "v2.1.2"
documentation: |-
Override the e2e version
- name: KUBE_BURNER_VERSION
default: "1.7.12"
default: "default"
documentation: |-
Override the kube burner version
- name: CHURN
Expand Down
5 changes: 4 additions & 1 deletion ci-operator/step-registry/openshift-qe/conc-builds/openshift-qe-conc-builds-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,10 @@ ES_USERNAME=$(cat "/secret/username")

export KUBE_BURNER_URL="https://github.com/cloud-bulldozer/kube-burner/releases/download/v0.17.3/kube-burner-0.17.3-Linux-x86_64.tar.gz"

git clone https://github.com/cloud-bulldozer/e2e-benchmarking/ --depth=1
REPO_URL="https://github.com/cloud-bulldozer/e2e-benchmarking";
LATEST_TAG=$(curl -s "https://api.github.com/repos/cloud-bulldozer/e2e-benchmarking/releases/latest" | jq -r '.tag_name');
TAG_OPTION="--branch $(if [ "$E2E_VERSION" == "default" ]; then echo "$LATEST_TAG"; else echo "$E2E_VERSION"; fi)";
git clone $REPO_URL $TAG_OPTION --depth 1
pushd e2e-benchmarking/workloads/kube-burner
export WORKLOAD=concurrent-builds

Expand Down
4 changes: 4 additions & 0 deletions ci-operator/step-registry/openshift-qe/conc-builds/openshift-qe-conc-builds-ref.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,10 @@ ref:
cpu: 100m
memory: 100Mi
env:
- name: E2E_VERSION
default: "v2.1.2"
documentation: |-
Override the e2e version
- name: BUILD_LIST
default: "1 8 15 30 45 60 75"
documentation: |-
Expand Down
5 changes: 4 additions & 1 deletion ...ep-registry/openshift-qe/control-plane-smoke/openshift-qe-control-plane-smoke-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,10 @@ pushd /tmp
python -m virtualenv ./venv_qe
source ./venv_qe/bin/activate

git clone https://github.com/cloud-bulldozer/e2e-benchmarking --depth=1
REPO_URL="https://github.com/cloud-bulldozer/e2e-benchmarking";
LATEST_TAG=$(curl -s "https://api.github.com/repos/cloud-bulldozer/e2e-benchmarking/releases/latest" | jq -r '.tag_name');
TAG_OPTION="--branch $(if [ "$E2E_VERSION" == "default" ]; then echo "$LATEST_TAG"; else echo "$E2E_VERSION"; fi)";
git clone $REPO_URL $TAG_OPTION --depth 1
pushd e2e-benchmarking/workloads/kube-burner-ocp-wrapper

sleep 60;
Expand Down
6 changes: 5 additions & 1 deletion .../step-registry/openshift-qe/control-plane-smoke/openshift-qe-control-plane-smoke-ref.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,12 @@ ref:
name: ocp-qe-gcp-sa
mount_path: /ga-gsheet
env:
- name: E2E_VERSION
default: "v2.1.2"
documentation: |-
Override the e2e version
- name: KUBE_BURNER_VERSION
default: "1.7.12"
default: "default"
documentation: |-
Override the kube burner version
- name: GC
Expand Down
5 changes: 4 additions & 1 deletion ci-operator/step-registry/openshift-qe/ingress-perf/openshift-qe-ingress-perf-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,10 @@ ES_PASSWORD=$(cat "/secret/password")
ES_USERNAME=$(cat "/secret/username")

# Clone the e2e repo
git clone https://github.com/cloud-bulldozer/e2e-benchmarking
REPO_URL="https://github.com/cloud-bulldozer/e2e-benchmarking";
LATEST_TAG=$(curl -s "https://api.github.com/repos/cloud-bulldozer/e2e-benchmarking/releases/latest" | jq -r '.tag_name');
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we had issues on jenins being able to consistently connect and get the latest tag. I wonder if we will hit that here as well. Might need another option

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 Also, do we even want a latest tag vs latest i.e. git clone of main branch?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That shouldn't ideally happen here as it is just hitting the endpoint and getting the tag_name. Where as previous command had some processing like below

$(curl -s https://api.github.com/repos/cloud-bulldozer/kube-burner/releases/latest | jq -r '.assets | map(select(.name | test("linux-x86_64"))) | .[0].browser_download_url')

And it used to run tar on top of it to extract binaries too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if someone wants a specific tag, they can override E2E_VERSION in their test or else latest would be picked by default.

TAG_OPTION="--branch $(if [ "$E2E_VERSION" == "default" ]; then echo "$LATEST_TAG"; else echo "$E2E_VERSION"; fi)";
git clone $REPO_URL $TAG_OPTION --depth 1
pushd e2e-benchmarking/workloads/ingress-perf

# ES Configuration
Expand Down
8 changes: 8 additions & 0 deletions ci-operator/step-registry/openshift-qe/ingress-perf/openshift-qe-ingress-perf-ref.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,14 @@ ref:
name: ocp-qe-perfscale-es
mount_path: /secret
env:
- name: E2E_VERSION
default: "v2.1.2"
documentation: |-
Override the e2e version
- name: INGRESS_PERF_VERSION
default: "default"
documentation: |-
Override the ingress-perf version
- name: CONFIG
default: "config/standard.yml"
documentation: |-
Expand Down
6 changes: 5 additions & 1 deletion ci-operator/step-registry/openshift-qe/loaded-upgrade/openshift-qe-loaded-upgrade-ref.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,12 @@ ref:
cpu: 100m
memory: 100Mi
env:
- name: E2E_VERSION
default: "v2.1.2"
documentation: |-
Override the e2e version
- name: KUBE_BURNER_VERSION
default: "1.7.12"
default: "default"
documentation: |-
Override the kube burner version
- name: ENABLE_FORCE
Expand Down
5 changes: 4 additions & 1 deletion ci-operator/step-registry/openshift-qe/network-perf/openshift-qe-network-perf-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,10 @@ source ./venv_qe/bin/activate
ES_PASSWORD=$(cat "/secret/password")
ES_USERNAME=$(cat "/secret/username")

git clone https://github.com/cloud-bulldozer/e2e-benchmarking --depth=1
REPO_URL="https://github.com/cloud-bulldozer/e2e-benchmarking";
LATEST_TAG=$(curl -s "https://api.github.com/repos/cloud-bulldozer/e2e-benchmarking/releases/latest" | jq -r '.tag_name');
TAG_OPTION="--branch $(if [ "$E2E_VERSION" == "default" ]; then echo "$LATEST_TAG"; else echo "$E2E_VERSION"; fi)";
git clone $REPO_URL $TAG_OPTION --depth 1
pushd e2e-benchmarking/workloads/network-perf-v2

# Clean up resources from possible previous tests.
Expand Down
8 changes: 8 additions & 0 deletions ci-operator/step-registry/openshift-qe/network-perf/openshift-qe-network-perf-ref.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,14 @@ ref:
name: ocp-qe-perfscale-ci
tag: latest
env:
- name: E2E_VERSION
default: "v2.1.2"
documentation: |-
Override the e2e version
- name: NETPERF_VERSION
default: "default"
documentation: |-
Override the k8s netperf
commands: openshift-qe-network-perf-commands.sh
timeout: 4h
credentials:
Expand Down
5 changes: 4 additions & 1 deletion ...perator/step-registry/openshift-qe/network-policy/openshift-qe-network-policy-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,10 @@ ES_PASSWORD=$(cat "/secret/password")
ES_USERNAME=$(cat "/secret/username")


git clone https://github.com/cloud-bulldozer/e2e-benchmarking --depth=1
REPO_URL="https://github.com/cloud-bulldozer/e2e-benchmarking";
LATEST_TAG=$(curl -s "https://api.github.com/repos/cloud-bulldozer/e2e-benchmarking/releases/latest" | jq -r '.tag_name');
TAG_OPTION="--branch $(if [ "$E2E_VERSION" == "default" ]; then echo "$LATEST_TAG"; else echo "$E2E_VERSION"; fi)";
git clone $REPO_URL $TAG_OPTION --depth 1
pushd e2e-benchmarking/workloads/kube-burner

export JOB_TIMEOUT=${JOB_TIMEOUT:=21600}
Expand Down
6 changes: 5 additions & 1 deletion ci-operator/step-registry/openshift-qe/network-policy/openshift-qe-network-policy-ref.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,12 @@ ref:
commands: openshift-qe-network-policy-commands.sh
timeout: 8h0m0s
env:
- name: E2E_VERSION
default: "v2.1.2"
documentation: |-
Override the e2e version
- name: KUBE_BURNER_VERSION
default: "1.7.12"
default: "default"
documentation: |-
Override the kube burner version
- name: WORKLOAD
Expand Down
5 changes: 4 additions & 1 deletion ci-operator/step-registry/openshift-qe/network-smoke/openshift-qe-network-smoke-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,10 @@ pushd /tmp
python -m virtualenv ./venv_qe
source ./venv_qe/bin/activate

git clone https://github.com/cloud-bulldozer/e2e-benchmarking --depth=1
REPO_URL="https://github.com/cloud-bulldozer/e2e-benchmarking";
LATEST_TAG=$(curl -s "https://api.github.com/repos/cloud-bulldozer/e2e-benchmarking/releases/latest" | jq -r '.tag_name');
TAG_OPTION="--branch $(if [ "$E2E_VERSION" == "default" ]; then echo "$LATEST_TAG"; else echo "$E2E_VERSION"; fi)";
git clone $REPO_URL $TAG_OPTION --depth 1
pushd e2e-benchmarking/workloads/network-perf-v2

# Clean up
Expand Down
9 changes: 9 additions & 0 deletions ci-operator/step-registry/openshift-qe/network-smoke/openshift-qe-network-smoke-ref.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,5 +13,14 @@ ref:
requests:
cpu: 100m
memory: 100Mi
env:
- name: E2E_VERSION
default: "v2.1.2"
documentation: |-
Override the e2e version
- name: NETPERF_VERSION
default: "default"
documentation: |-
Override the netperf version
documentation: >-
This step runs the perfscale network-perf smoke workload in the deployed cluster
5 changes: 4 additions & 1 deletion ...tor/step-registry/openshift-qe/node-density-cni/openshift-qe-node-density-cni-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,10 @@ source ./venv_qe/bin/activate
ES_PASSWORD=$(cat "/secret/password")
ES_USERNAME=$(cat "/secret/username")

git clone https://github.com/cloud-bulldozer/e2e-benchmarking
REPO_URL="https://github.com/cloud-bulldozer/e2e-benchmarking";
LATEST_TAG=$(curl -s "https://api.github.com/repos/cloud-bulldozer/e2e-benchmarking/releases/latest" | jq -r '.tag_name');
TAG_OPTION="--branch $(if [ "$E2E_VERSION" == "default" ]; then echo "$LATEST_TAG"; else echo "$E2E_VERSION"; fi)";
git clone $REPO_URL $TAG_OPTION --depth 1
pushd e2e-benchmarking/workloads/kube-burner-ocp-wrapper
export WORKLOAD=node-density-cni

Expand Down
6 changes: 5 additions & 1 deletion ...erator/step-registry/openshift-qe/node-density-cni/openshift-qe-node-density-cni-ref.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,12 @@ ref:
name: ocp-qe-perfscale-es
mount_path: /secret
env:
- name: E2E_VERSION
default: "v2.1.2"
documentation: |-
Override the e2e version
- name: KUBE_BURNER_VERSION
default: "1.7.12"
default: "default"
documentation: |-
Override the kube burner version
- name: PODS_PER_NODE
Expand Down
5 changes: 4 additions & 1 deletion ...step-registry/openshift-qe/node-density-heavy/openshift-qe-node-density-heavy-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,10 @@ ES_USERNAME=$(cat "/secret/username")
GSHEET_KEY_LOCATION="/ga-gsheet/gcp-sa-account"
export GSHEET_KEY_LOCATION

git clone https://github.com/cloud-bulldozer/e2e-benchmarking
REPO_URL="https://github.com/cloud-bulldozer/e2e-benchmarking";
LATEST_TAG=$(curl -s "https://api.github.com/repos/cloud-bulldozer/e2e-benchmarking/releases/latest" | jq -r '.tag_name');
TAG_OPTION="--branch $(if [ "$E2E_VERSION" == "default" ]; then echo "$LATEST_TAG"; else echo "$E2E_VERSION"; fi)";
git clone $REPO_URL $TAG_OPTION --depth 1
pushd e2e-benchmarking/workloads/kube-burner-ocp-wrapper
export WORKLOAD=node-density-heavy

Expand Down
6 changes: 5 additions & 1 deletion ...or/step-registry/openshift-qe/node-density-heavy/openshift-qe-node-density-heavy-ref.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,12 @@ ref:
name: ocp-qe-gcp-sa
mount_path: /ga-gsheet
env:
- name: E2E_VERSION
default: "v2.1.2"
documentation: |-
Override the e2e version
- name: KUBE_BURNER_VERSION
default: "1.7.12"
default: "default"
documentation: |-
Override the kube burner version
- name: PODS_PER_NODE
Expand Down
5 changes: 4 additions & 1 deletion ci-operator/step-registry/openshift-qe/node-density/openshift-qe-node-density-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,10 @@ source ./venv_qe/bin/activate
ES_PASSWORD=$(cat "/secret/password")
ES_USERNAME=$(cat "/secret/username")

git clone https://github.com/cloud-bulldozer/e2e-benchmarking --depth=1
REPO_URL="https://github.com/cloud-bulldozer/e2e-benchmarking";
LATEST_TAG=$(curl -s "https://api.github.com/repos/cloud-bulldozer/e2e-benchmarking/releases/latest" | jq -r '.tag_name');
TAG_OPTION="--branch $(if [ "$E2E_VERSION" == "default" ]; then echo "$LATEST_TAG"; else echo "$E2E_VERSION"; fi)";
git clone $REPO_URL $TAG_OPTION --depth 1
pushd e2e-benchmarking/workloads/kube-burner-ocp-wrapper
export WORKLOAD=node-density

Expand Down
6 changes: 5 additions & 1 deletion ci-operator/step-registry/openshift-qe/node-density/openshift-qe-node-density-ref.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,12 @@ ref:
cpu: 100m
memory: 100Mi
env:
- name: E2E_VERSION
default: "v2.1.2"
documentation: |-
Override the e2e version
- name: KUBE_BURNER_VERSION
default: "1.7.12"
default: "default"
documentation: |-
Override the kube burner version
- name: PODS_PER_NODE
Expand Down
5 changes: 4 additions & 1 deletion ci-operator/step-registry/openshift-qe/router-perf/openshift-qe-router-perf-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,10 @@ ES_USERNAME=$(cat "/secret/username")
GSHEET_KEY_LOCATION="/ga-gsheet/gcp-sa-account"
export GSHEET_KEY_LOCATION

git clone https://github.com/cloud-bulldozer/e2e-benchmarking --depth=1
REPO_URL="https://github.com/cloud-bulldozer/e2e-benchmarking";
LATEST_TAG=$(curl -s "https://api.github.com/repos/cloud-bulldozer/e2e-benchmarking/releases/latest" | jq -r '.tag_name');
TAG_OPTION="--branch $(if [ "$E2E_VERSION" == "default" ]; then echo "$LATEST_TAG"; else echo "$E2E_VERSION"; fi)";
git clone $REPO_URL $TAG_OPTION --depth 1
pushd e2e-benchmarking/workloads/router-perf-v2
# ES configuration
export ES_SERVER="https://$ES_USERNAME:$ES_PASSWORD@search-ocp-qe-perf-scale-test-elk-hcm7wtsqpxy7xogbu72bor4uve.us-east-1.es.amazonaws.com"
Expand Down
4 changes: 4 additions & 0 deletions ci-operator/step-registry/openshift-qe/router-perf/openshift-qe-router-perf-ref.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,10 @@ ref:
name: ocp-qe-gcp-sa
mount_path: /ga-gsheet
env:
- name: E2E_VERSION
default: "v2.1.2"
documentation: |-
Override the e2e version
- name: WORKLOAD_TYPE
default: router-perf
documentation: |-
Expand Down