DAOS-16262 build: Add weekly bullseye build and test GHA #15287

phender · 2024-10-09T20:09:35Z

Automate weekly build with Bullseye and VM Functionality test.

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-test: true

Required-githooks: true

Before requesting gatekeeper:

Two review approvals and any prior change requests have been resolved.
Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
Commit messages follows the guidelines outlined here.
Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

Automate weekly build with Bullseye and VM Functionality test. Skip-unit-tests: true Skip-fault-injection-test: true Skip-test: true Required-githooks: true Signed-off-by: Phil Henderson <phillip.henderson@intel.com>

github-actions · 2024-10-09T20:09:56Z

Ticket title is 'Automate weekly build with Bullseye with simple VM functional test'
Status is 'In Review'
https://daosio.atlassian.net/browse/DAOS-16262

.github/workflows/code-coverage-test.yml

phender · 2024-10-09T20:23:11Z

.github/workflows/code-coverage-test.yml

+  JENKINS_URL: https://build.hpdd.intel.com/
+
+on:
+  pull_request:             # Remove after testing


Running with this for now for testing. We don't want to run this on every PR so the plan is to remove this line in the final push.

brianjmurrell · 2024-10-09T20:28:04Z

Ha ha. I just added a couple of comments to #15206 which seems to be some kind of copy of this one?

brianjmurrell

Great to see this being added to GHA instead of Jenkins! Awesome job on jumping in and doing it this way!

brianjmurrell · 2024-10-09T20:36:21Z

.github/workflows/code-coverage-test.yml

+    - cron: "0 0 * * 6"     # Every Saturday
+
+concurrency:
+  group: code-coverage-${{ github.head_ref  || github.run_id }}


Good, you caught this one then.

brianjmurrell · 2024-10-09T20:46:22Z

.github/workflows/code-coverage-test.yml

As mentioned elsewhere, it's a shame to see this being an almost duplicate of https://github.com/daos-stack/daos/blob/master/.github/workflows/rpm-build-and-test.yml and while I have been working on workflow reusability, I wonder if whatever is different in this copy can be handled in the build and functional test matrices so that all of this can be merged with https://github.com/daos-stack/daos/blob/master/.github/workflows/rpm-build-and-test.yml.

The matrix ability of GHA seems to be quite a bit better than Jenkins.

Yes it would ideal not have to reproduce code. It sounds like composite actions is what we want. Let me try adding some for building RPMs and running Functional (VM) tests.

Yes it would ideal not have to reproduce code. It sounds like composite actions is what we want.

Yes, it is. That is already implemented in #14177 along with a number of other RPMs in other repos.

Since I am in the process of finalizing that PR and it's associates and getting ready to land them, would it be possible for you to rebase this PR on #14177 and it's associated PRs to further validate the concepts that I am trying to employ there? This would be a poster-child for that. Indeed, it might be that some of the work in this PR (I'm thinking splitting out the Functional test jobs for example) needs to be pushed up into #14177 and it's friends and it would be nicer to know that before landing it all.

If you don't have the time, I can fork this PR and do the rebase/validation to see what needs changing in my PRs to support it to further flesh those out.

All of that said, I was actually looking for a solution along the lines of not [re-]creating any more workflows of the same kind of build+test. I was/am hoping we can use the same single workflow for all of the scheduled testing just by permuting within the one single workflow.

This seems useful in that regard.

Although, I was hoping that https://github.com/daos-stack/daos/actions would have greater flexibility than Jenkins does at being able to filter to specific types of runs but it seems it doesn't get more granular than event:schedule.

So that is, we can get a bit better than Jenkins and isolate the landings to a branch (i.e. master from the scheduled runs but we cannot isolate each different schedule and given we might have no choice but to run them on master. 😦

That leaves release/* as being an even bigger problem.

We might have to use reusable workflows that checkout the branch to be tested. 😠

github-actions · 2024-10-09T22:03:17Z

Functional on EL 8.8 Test Results (old)

131 tests 127 ✅ 1h 36m 44s ⏱️
41 suites 4 💤
41 files 0 ❌

Results for commit 0905646.

Required-githooks: true Signed-off-by: Phil Henderson <phillip.henderson@intel.com>

.github/workflows/code-coverage-test.yml

Required-githooks: true Signed-off-by: Phil Henderson <phillip.henderson@intel.com>

daosbuild1 · 2024-10-11T21:51:04Z

Test stage Unit Test on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15287/4/display/redirect

daosbuild1 · 2024-10-11T21:51:05Z

Test stage Unit Test bdev with memcheck on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15287/4/display/redirect

daosbuild1 · 2024-10-11T21:51:06Z

Test stage Unit Test bdev on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15287/4/display/redirect

daosbuild1 · 2024-10-11T21:51:07Z

Test stage Unit Test with memcheck on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15287/4/display/redirect

Required-githooks: true Signed-off-by: Phil Henderson <phillip.henderson@intel.com>

brianjmurrell · 2024-10-15T19:03:07Z

.github/workflows/code-coverage-test.yml

+on:
+  pull_request:             # Remove after testing
+  schedule:
+    - cron: "0 0 * * 6"     # Every Saturday


If the intention is to run this on a branch other than master like we do with Jenkins, I think this restriction:

Scheduled workflows will only run on the default branch.

from https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows#schedule might be a problem.

Required-githooks: true Signed-off-by: Phil Henderson <phillip.henderson@intel.com>

.github/workflows/code-coverage-test.yml

+      statuses: write
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4.1.1


Required-githooks: true Signed-off-by: Phil Henderson <phillip.henderson@intel.com>

daosbuild1 · 2024-10-17T18:45:18Z

Test stage Unit Test with memcheck on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15287/7/display/redirect

daosbuild1 · 2024-10-17T18:45:19Z

Test stage Unit Test on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15287/7/display/redirect

daosbuild1 · 2024-10-17T18:45:20Z

Test stage NLT on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15287/7/display/redirect

daosbuild1 · 2024-10-17T18:45:21Z

Test stage Unit Test bdev with memcheck on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-15287/7/display/redirect

Required-githooks: true Signed-off-by: Phil Henderson <phillip.henderson@intel.com>

.github/workflows/code-coverage-test.yml

+       needs.build-rpms.result == 'skipped')
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4.1.1


Required-githooks: true Signed-off-by: Phil Henderson <phillip.henderson@intel.com>

github-actions · 2024-10-18T19:04:24Z

Functional on EL 8.6 Test Results (old)

0 tests 0 ✅ 0s ⏱️
0 suites 0 💤
0 files 0 ❌

Results for commit cc7116a.

♻️ This comment has been updated with latest results.

Required-githooks: true Signed-off-by: Phil Henderson <phillip.henderson@intel.com>

brianjmurrell · 2024-10-21T18:32:11Z

.github/actions/build-rpm/action.yml

+    - name: Publish RPMs
+      uses: actions/upload-artifact@v4
+      with:
+        # yamllint disable-line rule:line-length
+        name: ${{ steps.build-vars.outputs.distro_name }} ${{ steps.build-vars.outputs.distro_version }} RPM repository
+        path: ${{ inputs.REPO_PATH}}${{ github.run_number }}/artifact/artifacts/${{ inputs.distro }}


You can't do this. Per SDL, we are not allowed to distribute unsigned binaries.

This is the same code as https://github.com/daos-stack/daos/blob/master/.github/workflows/rpm-build-and-test.yml#L301-L305. Doesn't it just add the RPMs to Artifactory so that the functional test stages can install those RPMs?

This is the same code as master/.github/workflows/rpm-build-and-test.yml#L301-L305.

Which is being replaced by https://github.com/daos-stack/actions-lib/pull/1/files#diff-46a90c32d59a22431144045d82ce732bc5a498ea20229b697bc2fd2154899662R423-R428 and https://github.com/daos-stack/daos/pull/14177/files#diff-681a2eb462568f244d2fd51eff96c3d9945e33dec02ee489c680a742a0bdcef4R79-R85 which you can see comments out that upload.

Doesn't it just add the RPMs to Artifactory so that the functional test stages can install those RPMs?

No. That uploads them to the artifacts in GitHub, where anyone can download them. This is the part that puts us off-side of the SDL.

Moreover, we don't put RPMs from PRs into Artifactory. With Jenkins, we install them from the artifacts of the Jenkins job itself and in GHA they are stored in this location https://github.com/daos-stack/daos/pull/14177/files#diff-681a2eb462568f244d2fd51eff96c3d9945e33dec02ee489c680a742a0bdcef4R79-R85 by ci/rpm/create_repo.sh.

DAOS-16262 build: Add weekly bullseye build and test GHA

0905646

Automate weekly build with Bullseye and VM Functionality test. Skip-unit-tests: true Skip-fault-injection-test: true Skip-test: true Required-githooks: true Signed-off-by: Phil Henderson <phillip.henderson@intel.com>

phender requested a review from dinghwah October 9, 2024 20:09

github-advanced-security bot found potential problems Oct 9, 2024

View reviewed changes

phender mentioned this pull request Oct 9, 2024

DAOS-16285 build: Support passing a repo when provisioning a cluster #15238

Closed

18 tasks

phender requested a review from brianjmurrell October 9, 2024 20:14

phender commented Oct 9, 2024

View reviewed changes

brianjmurrell reviewed Oct 9, 2024

View reviewed changes

brianjmurrell self-requested a review October 9, 2024 20:42

brianjmurrell reviewed Oct 9, 2024

View reviewed changes

phender added 3 commits October 11, 2024 15:04

Merge branch 'master' into pahender/DAOS-16285_master2

fbbfa79

Avoid code re-use with compositre actions.

b973d2c

Required-githooks: true Signed-off-by: Phil Henderson <phillip.henderson@intel.com>

Updates.

94163e2

Required-githooks: true Signed-off-by: Phil Henderson <phillip.henderson@intel.com>

github-advanced-security bot found potential problems Oct 11, 2024

View reviewed changes

.github/workflows/code-coverage-test.yml Fixed Show fixed Hide fixed

.github/workflows/code-coverage-test.yml Fixed Show fixed Hide fixed

phender added 2 commits October 11, 2024 17:36

Build-rpm action updates

007d69f

Required-githooks: true Signed-off-by: Phil Henderson <phillip.henderson@intel.com>

Fix actions syntax

a3d7efd

Required-githooks: true Signed-off-by: Phil Henderson <phillip.henderson@intel.com>

Using composite actions.

ae4bf42

Required-githooks: true Signed-off-by: Phil Henderson <phillip.henderson@intel.com>

brianjmurrell reviewed Oct 15, 2024

View reviewed changes

phender added 2 commits October 16, 2024 15:41

Merge branch 'master' into pahender/DAOS-16285_master2

09ccb80

Attempt to resolve action errors.

f3d71a3

Required-githooks: true Signed-off-by: Phil Henderson <phillip.henderson@intel.com>

github-advanced-security bot found potential problems Oct 17, 2024

View reviewed changes

.github/workflows/code-coverage-test.yml

statuses: write

steps:

- name: Checkout code

uses: actions/checkout@v4.1.1

Check warning

Code scanning / Scorecard

Pinned-Dependencies Medium

score is 1: GitHub-owned GitHubAction not pinned by hash
Click Remediation section below to solve this issue

phender added 2 commits October 17, 2024 14:41

Add missing 'shell: bash'.

3520c1a

Required-githooks: true Signed-off-by: Phil Henderson <phillip.henderson@intel.com>

Fix missing required property.

5c5ee7d

Required-githooks: true Signed-off-by: Phil Henderson <phillip.henderson@intel.com>

phender added 4 commits October 17, 2024 14:46

Fix build-variables.

ccc327b

Required-githooks: true Signed-off-by: Phil Henderson <phillip.henderson@intel.com>

More fixes.

caf8861

Required-githooks: true Signed-off-by: Phil Henderson <phillip.henderson@intel.com>

Updates.

cb2c7ab

Required-githooks: true Signed-off-by: Phil Henderson <phillip.henderson@intel.com>

functional-test updates.

8a76f64

Required-githooks: true Signed-off-by: Phil Henderson <phillip.henderson@intel.com>

github-advanced-security bot found potential problems Oct 18, 2024

View reviewed changes

phender added 2 commits October 18, 2024 12:51

Fix secrets error.

c782bbc

Required-githooks: true Signed-off-by: Phil Henderson <phillip.henderson@intel.com>

Pass secret as input to composite action.

1d6949b

Required-githooks: true Signed-off-by: Phil Henderson <phillip.henderson@intel.com>

Fix typo.

cc7116a

Required-githooks: true Signed-off-by: Phil Henderson <phillip.henderson@intel.com>

brianjmurrell requested changes Oct 21, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DAOS-16262 build: Add weekly bullseye build and test GHA #15287

DAOS-16262 build: Add weekly bullseye build and test GHA #15287

phender commented Oct 9, 2024

github-actions bot commented Oct 9, 2024 •

edited

Loading

phender Oct 9, 2024

brianjmurrell commented Oct 9, 2024

brianjmurrell left a comment

brianjmurrell Oct 9, 2024

brianjmurrell Oct 9, 2024

phender Oct 11, 2024

brianjmurrell Oct 15, 2024 •

edited

Loading

brianjmurrell Oct 15, 2024 •

edited

Loading

github-actions bot commented Oct 9, 2024

daosbuild1 commented Oct 11, 2024

daosbuild1 commented Oct 11, 2024

daosbuild1 commented Oct 11, 2024

daosbuild1 commented Oct 11, 2024

brianjmurrell Oct 15, 2024

daosbuild1 commented Oct 17, 2024

daosbuild1 commented Oct 17, 2024

daosbuild1 commented Oct 17, 2024

daosbuild1 commented Oct 17, 2024

github-actions bot commented Oct 18, 2024 •

edited

Loading

brianjmurrell Oct 21, 2024

phender Oct 21, 2024

brianjmurrell Oct 21, 2024

DAOS-16262 build: Add weekly bullseye build and test GHA #15287

Are you sure you want to change the base?

DAOS-16262 build: Add weekly bullseye build and test GHA #15287

Conversation

phender commented Oct 9, 2024

Before requesting gatekeeper:

Gatekeeper:

github-actions bot commented Oct 9, 2024 • edited Loading

Choose a reason for hiding this comment

brianjmurrell commented Oct 9, 2024

brianjmurrell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brianjmurrell Oct 15, 2024 • edited Loading

Choose a reason for hiding this comment

brianjmurrell Oct 15, 2024 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Oct 9, 2024

Functional on EL 8.8 Test Results (old)

daosbuild1 commented Oct 11, 2024

daosbuild1 commented Oct 11, 2024

daosbuild1 commented Oct 11, 2024

daosbuild1 commented Oct 11, 2024

Choose a reason for hiding this comment

daosbuild1 commented Oct 17, 2024

daosbuild1 commented Oct 17, 2024

daosbuild1 commented Oct 17, 2024

daosbuild1 commented Oct 17, 2024

github-actions bot commented Oct 18, 2024 • edited Loading

Functional on EL 8.6 Test Results (old)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Oct 9, 2024 •

edited

Loading

brianjmurrell Oct 15, 2024 •

edited

Loading

brianjmurrell Oct 15, 2024 •

edited

Loading

github-actions bot commented Oct 18, 2024 •

edited

Loading