[CI] Stop testing quay.io images and always build from source

[zakkak]

No description provided.

[jerboaa]

• Stop testing 23.1 builds from source, 23.1 is now being constantly
  tested against Quarkus main on the Quarkus repository.

Where would I be able to look at those test results? Do you have a link to the changeset that adds support for 23.1 on Quarkus?

[jerboaa]

I'm not sure I agree with removing 23.1 build and test support. We surely want to continue testing that branch with updated JDK 21 EA builds, mandrel-integration-tests and so on (which Quarkus presumably doesn't do?!)? 23.1 is the latest LTS Mandrel release until GraalVM for JDK 25 comes out? A weekly test run doesn't hurt too much for this.

[zakkak]

Where would I be able to look at those test results? Do you have a link to the changeset that adds support for 23.1 on Quarkus?

That would be quarkusio/quarkus#36457 and since the time it got merged all Quarkus PRs are tested with 23.1 (from quay.io)

I'm not sure I agree with removing 23.1 build and test support. We surely want to continue testing that branch with updated JDK 21 EA builds, mandrel-integration-tests and so on (which Quarkus presumably doesn't do?!)?

The same still holds for 22.3 and 23.0, right?

23.1 is the latest LTS Mandrel release until GraalVM for JDK 25 comes out? A weekly test run doesn't hurt too much for this.

Well they add up...

We are interested in testing:

1. Q 2.13 branch with:
   1. Mandrel 22.3 quay.io image (to catch regressions due to base image updates)
   2. Mandrel 22.3 built from source using the latest JDK 17-EA (to catch regressions due to changes in the JDK)
2. Q 3.2 branch with:
   1. Mandrel 23.0 quay.io image (to catch regressions due to base image updates)
   2. Mandrel 23.0 built from source using the latest JDK 17-EA (to catch regressions due to changes in the JDK)
3. Q main branch with:
   1. Mandrel 23.0 quay.io image (to catch regressions due to base image updates)
   2. Mandrel 23.0 built from source using the latest JDK 17-EA (to catch regressions due to changes in the JDK)
   3. Mandrel 23.1 built from source using the latest JDK 21-EA (to catch regressions due to changes in the JDK)

In total that's 7 configurations each taking about 5 hours...

At this point it might be worth setting up the weekly GH actions to also create a builder-image
and test all moving parts in a single run. That would leave us with 4 configurations. WDYT?

[jerboaa]

We are interested in testing:

1. Q 2.13 branch with:
   
   1. Mandrel 22.3 quay.io image (to catch regressions due to base image updates)
   2. Mandrel 22.3 built from source using the latest JDK 17-EA (to catch regressions due to changes in the JDK)

2. Q 3.2 branch with:
   
   1. Mandrel 23.0 quay.io image (to catch regressions due to base image updates)
   2. Mandrel 23.0 built from source using the latest JDK 17-EA (to catch regressions due to changes in the JDK)

3. Q main branch with:
   
   1. Mandrel 23.0 quay.io image (to catch regressions due to base image updates)
   2. Mandrel 23.0 built from source using the latest JDK 17-EA (to catch regressions due to changes in the JDK)
   3. Mandrel 23.1 built from source using the latest JDK 21-EA (to catch regressions due to changes in the JDK)

In total that's 7 configurations each taking about 5 hours...

At this point it might be worth setting up the weekly GH actions to also create a builder-image and test all moving parts in a single run. That would leave us with 4 configurations. WDYT?

I'm not sure I agree on this test plan. I've yet to see a change in the base image updates to cause a major headache for us. So I'd skip those tests. I'm also not sure why we care about Q main + 23.0 combo. I thought it's 3.2 only at this point. That leaves us with Q main + 23.1. Since 23.1 isn't used in any Quarkus release yet, that needs the most testing, IMO. The Quarkus CI isn't good enough, since it uses released bits (Mandrel and JDK), so we don't catch EA JDK build issues (which we've definitely seen). So for weekly CI testing the focus, in my opinion, should be:

1. Q 2.13 branch with 22.3 built from source and EA JDK 17.
2. Q 3.2 branch with 23.0 built from source and EA JDK 17.
3. Q main with 23.1 built from source and EA JDK 21.

[zakkak]

I'm not sure I agree on this test plan. I've yet to see a change in the base image updates to cause a major headache for us. So I'd skip those tests.

That would probably be fine indeed given that these images are supposed to be used out in the field and we will learn sooner or later if something broke.

I'm also not sure why we care about Q main + 23.0 combo. I thought it's 3.2 only at this point. That leaves us with Q main + 23.1. Since 23.1 isn't used in any Quarkus release yet, that needs the most testing, IMO.

AFAIK it's still to be decided wheter the next Quarkus LTS release will move to 23.1 (i.e. JDK 21) or stick to 23.0 (i.e. JDK 17), thus why we need to keep testing it at least for the time being.

The Quarkus CI isn't good enough, since it uses released bits (Mandrel and JDK), so we don't catch EA JDK build issues (which we've definitely seen). So for weekly CI testing the focus, in my opinion, should be:

Q 2.13 branch with 22.3 built from source and EA JDK 17.
Q 3.2 branch with 23.0 built from source and EA JDK 17.
Q main with 23.1 built from source and EA JDK 21.

OK as long as we keep 23.0 for Q main. @Karm WDYT?

[jerboaa]

I'm also not sure why we care about Q main + 23.0 combo. I thought it's 3.2 only at this point. That leaves us with Q main + 23.1. Since 23.1 isn't used in any Quarkus release yet, that needs the most testing, IMO.

AFAIK it's still to be decided wheter the next Quarkus LTS release will move to 23.1 (i.e. JDK 21) or stick to 23.0 (i.e. JDK 17), thus why we need to keep testing it at least for the time being.

Fair enough.

The Quarkus CI isn't good enough, since it uses released bits (Mandrel and JDK), so we don't catch EA JDK build issues (which we've definitely seen). So for weekly CI testing the focus, in my opinion, should be:

Q 2.13 branch with 22.3 built from source and EA JDK 17.
Q 3.2 branch with 23.0 built from source and EA JDK 17.
Q main with 23.1 built from source and EA JDK 21.

OK as long as we keep 23.0 for Q main.

Sure.

[jerboaa]

Looks fine.

[zakkak]

22.3 and 23.0 failures are due to #608 and #607 respectively