pr-tester improvements to provide more reliable and better coverage

[andrew-m-leonard]

Following a discussion and review of the current pr-tester, the following suggestions are worth consideration for improvements:

• Currently only runs limited platforms and versions:
  • jdk8,11,17,21 : More versions ?
  • aarch64-linux, x64-linux, x64-mac, x64-windows, alpine-x64-linux : More platforms?
  • Add s390x, riscv? Anything else? Solaris? AIX?
    Why not build them all?
    Space and time complexity
    Runs with an explicit command
    Saves the artifacts
    We can run a set that covers the versions and platforms well, but it doesn’t need to be the entire set
• Does not currently run any AQA-tests, thus does not test any changes to AQA test invocation
  • Should run the smoke tests and maybe one of the quick suites (sanity.openjdk)
  • Reproducible build test?
• PRs within temurin-build are limited to testing on github action runners, could we extend the pr-tester ("run tests") feature to the temurin-build repo as well?

[sxa]

Further build test idea: Run a test to verify reproducibility of an existing GA build to ensure we haven't broken that in any way.

[sxa]

Also wonder if there's any value in having another keyword trigger run tests nogen or similar which would avoid the 30 minutes of job regeneration if you know that you've only changed pipeline code and not the configurations? Some of the issues I had while testing #1117 would have shown up very quickly, but each test had to wait over 40 minutes before hitting the pipeline code that I had changed:

Also noting that it's not possible to re-run the lower level jobs e.g. https://ci.adoptium.net/job/build-scripts-pr-tester/job/build-test/job/openjdk8-pipeline/ or https://ci.adoptium.net/job/build-scripts-pr-tester/job/build-test/job/jobs/job/jdk21u/job/jdk21u-linux-x64-temurin/ unless you change ciReference or CI_REF and USER_REMOTE_CONFIGS/branch as appropriate (along with the branch specifier in the job configuration for both)

If you try to rebuild the top level PR tester in jenkins to avoid clogging up the comments in the PR you get:

Also:   hudson.remoting.ProxyException: org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: e3813ad0-48e5-4e11-a635-a424bacc6fcf
hudson.remoting.ProxyException: groovy.lang.MissingPropertyException: No such property: ghprbActualCommit for class: WorkflowScript

[sxa]

Current PR testing pipeline does not test the signing phase, so any problems in there (such as with mac as I hit when doing some of the recent refactoring) does not get picked up (Ref).

[sxa]

Current PR testing does not test any cross compiled builds such as Windows/aarch64, so any problems with those are not picked up correctly. This is an scenario which we should aim to include (Maybe do a mix of windows on x64 and aarch64 on the different versions?) (Ref)