Build and run functional tests all locally

The `jenkins/runlocal` script now builds a Pbench Server RPM from the
checked-out sources in the current working tree, builds the Pbench
Server CI container, and runs the functional tests against the CI
container image.

As a result, `jenkins/runlocal` no longer pushes images to a registry.

It was necessary to support alternate base images for server container
to allow for fully local builds.  Prior to this change, if one tried to
use `jenkins/runlocal` without setting `BASE_IMAGE`, the default of
`UBI:9` would result in an error if the system on which this script was
invoked did not have a RHEL entitlement.

This change simplifies things so that the RPM built is truly local to
the environment, and using the installed environment, derives the
`BASE_IMAGE` is to work with the built RPM.

For example, if running on a Fedora 37 box, a Fedora 37 RPM would be
built and installed using the `fedora:37` base image.

NOTE, we only support defaulting to a `ubi9:latest` or `centos:stream9`
image in those respective environments.

We now only perform "hard" cleanup on INT, QUIT, and TERM signals.  If a
functional test is successful, and `--cleanup` has been requested, we
will have already removed the POD and container.  There is no need to
also run the "hard" cleanup script.

Further, we emit a note when `--cleanup` is not requested after the
functional tests have finished to alert the user that the POD and
container are still running.  We also emit the print statements for the
clean up being performed.

All arguments to `jenkins/runlocal` are passed along to
`jenkins/run-server-unit-tests` as a simple way to support the
`--cleanup` option on `jenkins/runlocal`.

We also:

 * Fail running server functional tests if pod creation fails
 * Remove the need for the `jenkins/runner`
 * Remove the unnecessary use of `--format=docker` in the `buildah
   from` command
 * Drop specific linting references, add `isort` in `README.md`
 * Use local file to keep CI container image name
   This allows us to entirely remove all references to a specific
   container registry from the code so that developers can create a
   local file in which to store the registry want to use in parallel
   with what the CI jobs already do with Jenkins credentials.
 * Correct a comment in `jenkins/Pipeline.gy`
 * Correct `GITTOP` definition location
 * Use `/bin/bash -e` consistently, adding it to `run-pbench-in-a-can`

README.md

@@ -163,12 +163,13 @@ our CI jobs.
 
 ### Python formatting
 
-This project uses the [flake8==3.8.3](http://flake8.pycqa.org/en/latest) method of
-code style enforcement, linting, and checking.
+This project uses the [flake8](http://flake8.pycqa.org/en/latest) method of code
+style enforcement, linting, and checking.
 
 All python code contributed to pbench must match the style requirements. These
 requirements are enforced by the [pre-commit](https://pre-commit.com) hook
-using the [black==1.19b0](https://github.com/psf/black) Python code formatter.
+using the [black](https://github.com/psf/black) Python code formatter and the
+[isort](https://github.com/pycqa/isort) Python import sorter.
 
 ### Use pre-commit to set automatic commit requirements
 
@@ -210,3 +211,17 @@ we build, with the following exceptions for tag names:
     for `Major`.`Minor` released images
 
   * `<SHA1 git hash>` (9 characters) - commit hash of the checked out code
+
+### References to Container Image Repositories
+The operation of our functional tests, the Pbench Server "in-a-can" used in
+the functional tests, and other verification and testing environments use
+container images from public repositories and non-public ones.  The CI jobs
+obtain references to those repositories using Jenkins credentials.  When a
+developer runs those same jobs locally, you can create two files with the
+appropriate contents locally:
+
+  * `${HOME}/.config/pbench/ci_registry.name`
+  * `${HOME}/.config/pbench/public_registry.name`
+
+If those files are not provided local execution will report an error when those
+values are missing.

build.sh

@@ -30,6 +30,8 @@ black --check .
 flake8 .
 isort --check .
 ( cd dashboard && npx eslint --max-warnings 0 "src/**" )
+# We need to invoke the alembic check with host networking so that it can reach
+# the PostgreSQL pod it creates.
 EXTRA_PODMAN_SWITCHES="--network host" jenkins/run tox -e alembic-check
 set +x
 

jenkins/Makefile

@@ -25,9 +25,6 @@ RPMBUILD_BASEIMAGE_DEFAULTS = \
 # All Fedora images are based on Fedora 36
 FEDORA_BASE_IMAGE_REF = quay.io/fedora/fedora:36
 
-# Image registry for storing RPM build container images
-RPMBUILD_IMAGE_REPO = images.paas.redhat.com/pbench
-
 # Templates for the various distributions' base images.  For the moment, for a
 # given distribution, all of the versions can be found by tweaking the image
 # tag; so, to generate the appropriate repository for the base image, we just
@@ -69,13 +66,13 @@ push-rpmbuild-all: $(RPMBUILD_BASEIMAGE_DEFAULTS:%=push-rpmbuild-%)
 image-rpmbuild-all: $(RPMBUILD_BASEIMAGE_DEFAULTS:%=image-rpmbuild-%)
 
 push-ci:
-	buildah push localhost/pbench-ci-fedora:${BRANCH} quay.io/pbench/pbench-ci-fedora:${BRANCH}
+	buildah push quay.io/pbench/pbench-ci-fedora:${BRANCH}
 
 image-ci: ci.fedora.Dockerfile
-	buildah bud -f ci.fedora.Dockerfile -t localhost/pbench-ci-fedora:${BRANCH}
+	buildah bud -f ci.fedora.Dockerfile -t quay.io/pbench/pbench-ci-fedora:${BRANCH}
 
 ${_DISTROS:%=push-rpmbuild-%}: push-rpmbuild-%:
-	buildah push localhost/pbench-rpmbuild:${*} ${RPMBUILD_IMAGE_REPO}/pbench-rpmbuild:${*}
+	buildah push ${PB_CONTAINER_REG}/pbench-rpmbuild:${*}
 
 image-rpmbuild-rhel-7 : PKGMGR = yum
 image-rpmbuild-centos-7 : PKGMGR = yum
@@ -86,7 +83,7 @@ ${_DISTROS:%=image-rpmbuild-%}: image-rpmbuild-%:
 		buildah run $$container ${PKGMGR} install -y ${RPMBUILD_PACKAGES} && \
 		buildah run $$container ${PKGMGR} clean all && \
 		buildah run $$container python3 -m pip install jinja2-cli && \
-		buildah commit $$container localhost/pbench-rpmbuild:${*}
+		buildah commit $$container ${PB_CONTAINER_REG}/pbench-rpmbuild:${*}
 
 ci.fedora.Dockerfile: ci.Dockerfile.j2
 	jinja2 ci.Dockerfile.j2 \

jenkins/Pipeline.gy

@@ -11,6 +11,7 @@ pipeline {
         PB_CI_REGISTRY=credentials('c3e2d737-0e56-4c1e-a945-d86bc644384c')
         PB_CI_REG_CRED=credentials('12b404ca-3036-4960-9929-979148b9e49a')
         PB_ORG_NAME="pbench"
+        PB_CONTAINER_REG="${PB_CI_REGISTRY}/${PB_ORG_NAME}"
         // If we are executing for a PR, as opposed to executing for a branch
         // like `main`, the CHANGE_ID will be defined to the PR number.  If it
         // is defined we'll use its value for the container image tag;
@@ -45,7 +46,7 @@ pipeline {
             steps {
                 sh 'buildah login -u="${PB_CI_REG_CRED_USR}" -p="${PB_CI_REG_CRED_PSW}" ${PB_CI_REGISTRY}'
                 sh 'bash -ex ./server/pbenchinacan/container-build.sh'
-                sh 'buildah push localhost/${PB_SERVER_IMAGE_NAME}:${PB_IMAGE_TAG} ${PB_CI_REGISTRY}/${PB_ORG_NAME}/${PB_SERVER_IMAGE_NAME}:${PB_IMAGE_TAG}'
+                sh 'buildah push ${PB_CONTAINER_REG}/${PB_SERVER_IMAGE_NAME}:${PB_IMAGE_TAG}'
             }
         }
         stage('Build the Pbench Agent Containers') {
@@ -56,12 +57,11 @@ pipeline {
                 sh 'buildah login -u="${PB_CI_REG_CRED_USR}" -p="${PB_CI_REG_CRED_PSW}" ${PB_CI_REGISTRY}'
                 sh '/usr/bin/python3 -m pip install --user jinja2-cli'
                 sh 'make -C agent/containers/images CI_RPM_ROOT=${WORKSPACE_TMP} clean ${PB_AGENT_DISTRO}-everything'
-                sh 'make -C agent/containers/images IMAGE_REPO=${PB_CI_REGISTRY}/${PB_ORG_NAME} IMAGE_TAG=${PB_IMAGE_TAG} ${PB_AGENT_DISTRO}-push-ci'
+                sh 'make -C agent/containers/images IMAGE_REPO=${PB_CONTAINER_REG} IMAGE_TAG=${PB_IMAGE_TAG} ${PB_AGENT_DISTRO}-push-ci'
             }
         }
         stage('Deploy server and run functional tests') {
             environment {
-                PB_CONTAINER_REG="${PB_CI_REGISTRY}"
                 PB_DASHBOARD_DIR="${WORKSPACE}/dashboard/build/"
                 PB_SERVER_IMAGE_TAG="${PB_IMAGE_TAG}"
             }
@@ -84,11 +84,11 @@ pipeline {
     }
     post {
         success {
-            // Note that jenkins/run-pytests is executed inside the container
-            // while the Cobertura plug-in is executed natively, so this poses
-            // a challenge in terms of finding the coverage report file; we
-            // finesse this by assuming that it will be in the current
-            // directory in both environments.
+            // Note that `build.sh` is executed inside the container while the
+            // Cobertura plug-in is executed natively, so this poses a challenge
+            // in terms of finding the coverage report file; we finesse this by
+            // assuming that it will be in the current directory in both
+            // environments.
             step([$class: 'CoberturaPublisher',
                 autoUpdateHealth: false,
                 autoUpdateStability: false,

jenkins/podman

@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+#!/bin/bash
 
 # A simple wrapper for the podman command which handles invocation inside a container
 

jenkins/run

@@ -140,23 +140,25 @@ echo "Starting container with ${PODMAN}:
         WORKSPACE=${WORKSPACE}
         ${GIT_BASE_VOLUME:+GIT_BASE_VOLUME=${GIT_BASE_VOLUME}}"
 ${PODMAN} run \
+    --init \
     --userns=keep-id \
     --security-opt label=disable \
     --volume ${WORKSPACE_TMP}:${HOME_DIR} \
     --volume ${WORKSPACE}:${HOME_DIR}/pbench \
     --volume ${PODMAN_SOCK}:${PODMAN_SOCK} \
     ${GIT_BASE_VOLUME} \
     -w ${HOME_DIR}/pbench \
+    --env CONTAINER_HOST=unix:/${PODMAN_SOCK} \
     --env HOME=${HOME_DIR} \
+    --env PB_CONTAINER_REG=${PB_CONTAINER_REG} \
     --env USER=${USER_NAME} \
-    --env CONTAINER_HOST=unix:/${PODMAN_SOCK} \
     --env WORKSPACE=${WORKSPACE} \
     --env WORKSPACE_TMP=${WORKSPACE_TMP} \
     --ulimit nofile=65536:65536 \
     --rm \
     ${EXTRA_PODMAN_SWITCHES} \
     ${_image} \
-    jenkins/runner "${@}"
+    "${@}"
 
 rc=$?
 

jenkins/run-server-func-tests

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-export EXTRA_PODMAN_SWITCHES=${EXTRA_PODMAN_SWITCHES:-"--pull=always"}
+export EXTRA_PODMAN_SWITCHES=${EXTRA_PODMAN_SWITCHES:-"--pull=newer"}
 export IMAGE_KIND=${IMAGE_KIND:-"fedora"}
 export IMAGE_REPO=${IMAGE_REPO:-"quay.io/pbench"}
 export IMAGE_ROLE=${IMAGE_ROLE:-"ci"}
@@ -9,22 +9,31 @@ export PB_SERVER_IMAGE_TAG=${PB_SERVER_IMAGE_TAG:-"$(cat jenkins/branch.name)"}
 export PB_POD_NAME=${PB_POD_NAME:-"pbench-in-a-can_${PB_SERVER_IMAGE_TAG}"}
 export PB_SERVER_CONTAINER_NAME=${PB_SERVER_CONTAINER_NAME:-"${PB_POD_NAME}-pbenchserver"}
 
+server/pbenchinacan/run-pbench-in-a-can
+if [[ ${?} -ne 0 ]]; then
+    echo "Failure to start Pbench-in-a-Can pod" >&2
+    exit 1
+fi
+
 function cleanup {
     # Remove the pod which we just created and ran; remove any dangling
     # containers; and then remove any dangling images.
+    echo "Forcefully removing the Pbench Server container ..." >&2
     podman rm --force --ignore ${1}
+    echo "Forcefully removing the Pbench Support Services pod ..." >&2
     podman pod rm --force --ignore ${2}
+    echo "Pruning containers ..." >&2
     podman container prune -f
+    echo "Performing container cleanup ..." >&2
     podman container cleanup --all --rm
+    echo "Pruning images ..." >&2
     podman image prune -f
 }
 
 if [[ ${1} == "--cleanup" ]]; then
-    trap "cleanup ${PB_SERVER_CONTAINER_NAME} ${PB_POD_NAME}" INT QUIT EXIT
+    trap "cleanup ${PB_SERVER_CONTAINER_NAME} ${PB_POD_NAME}" INT QUIT TERM
 fi
 
-server/pbenchinacan/run-pbench-in-a-can
-
 SERVER_URL="http://localhost:8080"
 SERVER_API_ENDPOINTS="${SERVER_URL}/api/v1/endpoints"
 
@@ -65,23 +74,26 @@ if [[ ${exit_status} -ne 0 ]]; then
     printf -- "\n--- journalctl dump ---\n\n"
 fi
 if [[ -z ${1} ]]; then
-    # No clean up requested.
+    echo "No clean up requested ... Pbench Server container and support services pod likely still running!"
     exit ${exit_status}
 fi
 
 # Remove the server container
+echo "Stopping Pbench Server container ..."
 podman stop ${PB_SERVER_CONTAINER_NAME}
 stop_status=${?}
 if [[ ${exit_status} -eq 0 ]]; then
     exit_status=${stop_status}
 fi
 
+echo "Stopping Pbench Support Services pod ..."
 podman pod stop ${PB_POD_NAME}
 stop_status=${?}
 if [[ ${exit_status} -eq 0 ]]; then
     exit_status=${stop_status}
 fi
 
+echo "Removing Pbench Support Services pod ..."
 podman pod rm ${PB_POD_NAME}
 stop_status=${?}
 if [[ ${exit_status} -eq 0 ]]; then

jenkins/runlocal

@@ -1,29 +1,52 @@
 #!/bin/bash -e
 
-# Build the RPM and then build the containers, pushing only the CI image
-# to the registry, and then run the fuctional tests against the built
-# container image.
+# Build the Pbench Server RPM and container, and run functional tests locally.
+# Requires a Fedora, CentOS, or RHEL environment to run.
 
-export PB_CONTAINER_REG=images.paas.redhat.com
-export PB_ORG_NAME=pbench
+# NOTE WELL: By default, when the functional tests are run, the infrastructure
+# pod and Pbench Server container are left running by default.  Add the switch,
+# `--cleanup` to direct `jenkins/run-server-func-tests` to cleanup when
+# finished (success or failure).
+
+# Build the pbench-server RPM locally, then build the containers locally, and
+# then run the functional tests against the locally built CI container image.
 export PB_SERVER_IMAGE_NAME=pbench-server
+
+# We use the current user name as the tag to avoid any conflict with what the CI
+# environment does.
 export PB_SERVER_IMAGE_TAG=${USER}
-export PB_DASHBOARD_DIR=$(pwd)/dashboard/build
 
+# We use the image pull policy of `never` here to ensure our locally built image
+# is used by the pod.
+export PB_SERVER_IMAGE_PULL_POLICY=never
+
+# Create an RPM from the current source tree and double check it exists.
 make -C server/rpm clean rpm
+export RPM_PATH=${HOME}/rpmbuild/RPMS/noarch/pbench-server-*.rpm
+ls -ld ${RPM_PATH}
+
+source /etc/os-release
+
+if [[ -z ${BASE_IMAGE} ]]; then
+    major=${VERSION_ID%%.*}
+    if [[ ${ID} == "fedora" ]]; then
+        # Any Fedora is okay.
+        BASE_IMAGE=${ID}:${major}
+    elif [[ ${ID} == "centos" && "${major}" == "9" ]]; then
+        # Only CentOS 9 is supported
+        BASE_IMAGE=${ID}:stream${major}
+    elif [[ ${ID} == "rhel" && "" == "9" ]]; then
+        # Only RHEL 9 is supported
+        BASE_IMAGE=ubi${major}:latest
+    else
+        echo "Unsupported local OS, ${ID}:${VERSION_ID}" >&2
+        exit 1
+    fi
+    export BASE_IMAGE
+fi
+
+# Build the canned Pbench Server container from the RPM built above.
+server/pbenchinacan/container-build.sh
 
-# Typically, one logs in to a container registery with automated scripts using
-# an "application" token.  When using quay.io based container registries, the
-# application token uses the user name `$app` and the token is provided as the
-# password.
-#
-# For example, if you generate a token and set the value to the environment
-# variable named, __LOGIN_SECRET__, then the `buildah login` command below
-# demonstrates how to login with that token.
-#
-# $ buildah login -u='$app' -p="${__LOGIN_SECRET__}" ${PB_CONTAINER_REG}
-
-RPM_PATH=${HOME}/rpmbuild/RPMS/noarch/pbench-server-*.rpm bash -ex ./server/pbenchinacan/container-build.sh
-buildah push localhost/${PB_SERVER_IMAGE_NAME}:${PB_SERVER_IMAGE_TAG} ${PB_CONTAINER_REG}/${PB_ORG_NAME}/${PB_SERVER_IMAGE_NAME}:${PB_SERVER_IMAGE_TAG}
-
-jenkins/run-server-func-tests
+# Run the functional tests using the locally built image.
+jenkins/run-server-func-tests "${@}"

lib/pbench/server/database/alembic.check

@@ -2,6 +2,8 @@
 
 pb_root=${PWD}  # This script is run from the Pbench root; remember where that is
 
+PB_CONTAINER_REG=${PB_CONTAINER_REG:-$(<${HOME}/.config/pbench/ci_registry.name)}
+
 echo "Starting PostgreSQL container" >&2
 ${pb_root}/jenkins/podman run --name postgresql-alembic \
     --detach \
@@ -24,7 +26,7 @@ ${pb_root}/jenkins/podman run --name postgresql-alembic \
     --env 'POSTGRESQL_USER=pbench' \
     --env 'POSTGRESQL_PASSWORD=pbench' \
     --env 'POSTGRESQL_DATABASE=pbench' \
-    images.paas.redhat.com/pbench/postgresql-13:latest container-entrypoint run-postgresql
+    ${PB_CONTAINER_REG}/postgresql-13:latest container-entrypoint run-postgresql
 
 trap "echo 'Stopping PostgreSQL container' >&2 ; ${pb_root}/jenkins/podman stop postgresql-alembic" INT ABRT QUIT EXIT
 

server/pbenchinacan/container-build.sh

@@ -1,20 +1,23 @@
-#!/usr/bin/env bash -e
+#!/bin/bash -e
 #
 # This script builds a container which, when run, starts a Pbench Server.
 #
 
 #+
 # Configuration definition section.
 #-
+GITTOP=${GITTOP:-$(git rev-parse --show-toplevel)}
 
 BASE_IMAGE=${BASE_IMAGE:-registry.access.redhat.com/ubi9:latest}
 PB_SERVER_IMAGE_NAME=${PB_SERVER_IMAGE_NAME:-"pbench-server"}
 PB_SERVER_IMAGE_TAG=${PB_SERVER_IMAGE_TAG:-$(< ${GITTOP}/jenkins/branch.name)}
 RPM_PATH=${RPM_PATH:-/root/sandbox/rpmbuild/RPMS/noarch/pbench-server-*.rpm}
 KEYCLOAK_CLIENT_SECRET=${KEYCLOAK_CLIENT_SECRET:-"client-secret"}
 
+# Default target registry to use.
+PB_CONTAINER_REG=${PB_CONTAINER_REG:-$(<${HOME}/.config/pbench/ci_registry.name)}
+
 # Locations on the host
-GITTOP=${GITTOP:-$(git rev-parse --show-toplevel)}
 PBINC_SERVER=${GITTOP}/server
 PBINC_INACAN=${PBINC_SERVER}/pbenchinacan
 
@@ -37,19 +40,21 @@ fi
 # Container build section.
 #-
 
-# Open a copy of the base container.  Docker format is required in order to set
-# the hostname.
-container=$(buildah from --format=docker ${BASE_IMAGE})
+# Open a copy of the base container.
+container=$(buildah from ${BASE_IMAGE})
 
 buildah config \
     --label maintainer="Pbench Maintainers <pbench@googlegroups.com>" \
     $container
 
 buildah copy $container ${RPM_PATH} /tmp/pbench-server.rpm
 buildah run $container dnf update -y
-buildah run $container dnf install -y --setopt=tsflags=nodocs \
+if [[ "${BASE_IMAGE}" == *"ubi9:latest" || "${BASE_IMAGE}" == *"centos:stream9" ]]; then
+    buildah run $container dnf install -y --nodocs \
         https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
-buildah run $container dnf install -y /tmp/pbench-server.rpm nginx less rsyslog rsyslog-mmjsonparse
+fi
+buildah run $container dnf install -y --nodocs \
+    /tmp/pbench-server.rpm nginx less rsyslog rsyslog-mmjsonparse
 buildah run $container dnf clean all
 buildah run $container rm -f /tmp/pbench-server.rpm
 
@@ -98,5 +103,5 @@ buildah run $container mkdir -p -m 0755  \
 buildah run $container cp /usr/share/nginx/html/404.html /usr/share/nginx/html/50x.html /srv/pbench/public_html/
 buildah run $container chown --recursive pbench:pbench /srv/pbench
 
-# Create the container image
-buildah commit $container localhost/${PB_SERVER_IMAGE_NAME}:${PB_SERVER_IMAGE_TAG}
+# Create the container image.
+buildah commit $container ${PB_CONTAINER_REG}/${PB_SERVER_IMAGE_NAME}:${PB_SERVER_IMAGE_TAG}