Unify Y-build and I-build configurations and their tests

[HannesWell]

Recently I have reworked the definition of I-build tests to reduce duplication and to make them more unified and simpler.
This has the effect that the definition of Y-build tests has now significantly diverged from the I-build tests and I intended to apply the same simplifications for Y-build tests as well.

And while looking into all the definitions of I-builds and Y-builds I noticed that they have a lot in common and wonder respectively think we should try to unify their definitions and try to define both pipelines from one source, while considering the points where they are different.
But this questions what exactly the differences between I-builds and Y-builds are? What I have found so far is

• Y-builds built another branch than the master, at least for jdt.core the development branch of ECJ supporting the next Java release
• Y-build tests have one linux setup that uses the latest EA JDK instead of the latest release
• The results of Y-builds are published to another website than I-builds
• Y-builds publish their build p2-repo e.g. to
  https://download.eclipse.org/eclipse/updates/4.35-Y-builds/
  instead of https://download.eclipse.org/eclipse/updates/4.35-I-builds/
• Y-builds don't publish to a Maven snapshot repository (I assume)

Together with #1950 this should to further simplify the overall releng.
Of course a unification would have to happen in multiple steps.

An alternative solution would be to just develop ECJ's support for upcoming java versions on the master branch and stop using another branch. AFAICT this has already been discussed respectively is currently discussed but will not be implemented soon.

The questions I have to the JDT-team:

• Are you fine with the proposed general goal?
• Who of you is interested to discuss the details that come along the path of making this happen?

@jarthana, @stephan-herrmann or @iloveeclipse are you interested in discussing this or can you tell who is interested?

Community

• I understand suggesting an enhancement doesn't mandate anyone to implement it. Other contributors may consider this suggestion, or not, at their own convenience. The most efficient way to get it fixed is that I implement it myself and contribute it back as a good quality patch to the project.

[stephan-herrmann]

I am interested in seeing successful Y-builds with useful test result, ideally with no big gaps around the times of release, or infra changes. If unification helps to this end, then surely I fully agree with the goals, and your description looks correct to me.

I don't, however, have much insight into the inner workings of I-builds nor Y-builds, so I don't know in which detail discussion I could help. I'm already looking into P-builds, which is about all I can contribute in terms of releng builds.

And while it's correct that the matter of master vs BETA_JAVA24 (all of jdt, not just jdt.core) is being discussed widely and for a long time already, it's pretty clear that the mere fact of branching will stay for the foreseeable future, for more than one reason.

[HannesWell]

I am interested in seeing successful Y-builds with useful test result, ideally with no big gaps around the times of release, or infra changes. If unification helps to this end, then surely I fully agree with the goals, and your description looks correct to me.

It will simplify the maintenance of the Y-build and will make it quicker to apply changes and it will also mean that the basic configurations that are meant to be equal, stay equal. So I think this could be a contribution to that overall goal.

I don't, however, have much insight into the inner workings of I-builds nor Y-builds, so I don't know in which detail discussion I could help. I'm already looking into P-builds, which is about all I can contribute in terms of releng builds.

It's mainly when I encounter differences to the I-build and I have to find out if they are really necessary or not. Then I hope to get an answer from you or somebody else from the JDT-team. But the technical Jenkins details I can hopefully handle by myself.

And while it's correct that the matter of master vs BETA_JAVA24 (all of jdt, not just jdt.core) is being discussed widely and for a long time already, it's pretty clear that the mere fact of branching will stay for the foreseeable future, for more than one reason.

Yes, I just wanted to mention that option, knowing that it's at the moment not feasible. :)

[HannesWell]

For your information I have now done the following further step to unify Y-build and I-build tests:

• [Build] Unify Y-build test job names to schema os-arch-javaVersion #2633
• [Y-Build-Tests] Define Mac and Windows jobs as Jenkins-pipeline #2634
• [Y-Build-Tests] Simplify and unify job definitions #2635

The tasks executed respectively the behavior should be the same, but please let me know if you encounter any problem upon the next execution. Respectively if you don't plan to run another Y-build soon, do you mind if I start one?

Does the JDT team mind if obsolete Y-build test jobs, i.e. those that are no longer in use, are removed from https://ci.eclipse.org/releng/job/YPBuilds/ or do you need them for some time for reference?

[HannesWell]

For your information I have now done the following further step to unify Y-build and I-build tests:

To me it looks like the first execution after these changes were ok, without regressions due to infrastructure changes.

@stephan-herrmann or @jarthana I have two questions about the Y-build setup:

1. Is there a reason why Y-build tests don't run on windows? Is it just to save precious computation time on the windows machine?

2. Would it be possible to use the jdk installations provided by default for Eclipse Jenkins instances in Y-build tests on Linux?
   At the moment specific open-jdks are download for each build:
   

   eclipse.platform.releng.aggregator/JenkinsJobs/YBuilds/Y_unit_linux.groovy

   Lines 5 to 7 in 768f042

   	[javaVersion: 17, javaHome: '''installJDK('https://download.java.net/java/GA/jdk17.0.2/dfd4a8d0985749f896bed50d7138ee7f/8/GPL/openjdk-17.0.2_linux-x64_bin.tar.gz')''' ],
   	[javaVersion: 21, javaHome: '''installJDK('https://download.java.net/java/GA/jdk21/fd2272bbf8e04c3dbaee13770090416c/35/GPL/openjdk-21_linux-x64_bin.tar.gz')''' ],
   	[javaVersion: 24, javaHome: '''installJDK('https://download.java.net/java/early_access/jdk24/18/GPL/openjdk-24-ea+18_linux-x64_bin.tar.gz')''' ]

   
   But EF Jenkins provides all these JDKs already, even EA builds for not yet released versions:
   See 'openjdk-ea-latest' at https://github.com/eclipse-cbi/jiro/wiki/Tools-(JDK,-Maven,-Ant)#openjdk
   @fredg02 can you tell how often the EA builds are updated for all Jenkins instances?

To the JDT-devs:
Would you be fine with changing to the provided 'openjdk-ea-latest' for the Y-build
Java 24 tests and switch to temurin for all others?
Just like all other builds:
https://github.com/eclipse-cbi/jiro/wiki/Tools-(JDK,-Maven,-Ant)#eclipse-temurin

[stephan-herrmann]

Does the JDT team mind if obsolete Y-build test jobs, i.e. those that are no longer in use, are removed from https://ci.eclipse.org/releng/job/YPBuilds/ or do you need them for some time for reference?

IMHO old jobs can be removed once the current set of jobs (build & test) is functional.

Let's perhaps just wait for one more +1 from @jarthana or @mpalat

[stephan-herrmann]

1. Is there a reason why Y-build tests don't run on windows? Is it just to save precious computation time on the windows machine?

I'll have to pass this one, as I don't know the reason.

2. Would it be possible to use the jdk installations provided by default for Eclipse Jenkins instances in Y-build tests on Linux?
   At the moment specific open-jdks are download for each build:
   

   eclipse.platform.releng.aggregator/JenkinsJobs/YBuilds/Y_unit_linux.groovy

   Lines 5 to 7 in 768f042

   	[javaVersion: 17, javaHome: '''installJDK('https://download.java.net/java/GA/jdk17.0.2/dfd4a8d0985749f896bed50d7138ee7f/8/GPL/openjdk-17.0.2_linux-x64_bin.tar.gz')''' ],
   	[javaVersion: 21, javaHome: '''installJDK('https://download.java.net/java/GA/jdk21/fd2272bbf8e04c3dbaee13770090416c/35/GPL/openjdk-21_linux-x64_bin.tar.gz')''' ],
   	[javaVersion: 24, javaHome: '''installJDK('https://download.java.net/java/early_access/jdk24/18/GPL/openjdk-24-ea+18_linux-x64_bin.tar.gz')''' ]

But EF Jenkins provides all these JDKs already, even EA builds for not yet released versions:
See 'openjdk-ea-latest' at https://github.com/eclipse-cbi/jiro/wiki/Tools-(JDK,-Maven,-Ant)#openjdk
@fredg02 can you tell how often the EA builds are updated for all Jenkins instances?

IMHO, the specific JDK version is not relevant here, as long as we have some "good" EA build of the current stream.

To the JDT-devs: Would you be fine with changing to the provided 'openjdk-ea-latest' for the Y-build Java 24 tests and switch to temurin for all others? Just like all other builds: https://github.com/eclipse-cbi/jiro/wiki/Tools-(JDK,-Maven,-Ant)#eclipse-temurin

No objections from my side.

[jarthana]

Does the JDT team mind if obsolete Y-build test jobs, i.e. those that are no longer in use, are removed from https://ci.eclipse.org/releng/job/YPBuilds/ or do you need them for some time for reference?

IMHO old jobs can be removed once the current set of jobs (build & test) is functional.

Let's perhaps just wait for one more +1 from @jarthana or @mpalat

No objections from me either.

1. Is there a reason why Y-build tests don't run on windows? Is it just to save precious computation time on the windows machine?

Just checked with @MohananRahul who says that it's configured but not configured to run automatically.

To the JDT-devs: Would you be fine with changing to the provided 'openjdk-ea-latest' for the Y-build Java 24 tests and switch to temurin for all others? Just like all other builds: https://github.com/eclipse-cbi/jiro/wiki/Tools-(JDK,-Maven,-Ant)#eclipse-temurin

I agree with Stephan. No problem for me.

[jarthana]

1. Is there a reason why Y-build tests don't run on windows? Is it just to save precious computation time on the windows machine?

Just checked with @MohananRahul who says that it's configured but not configured to run automatically.

Rahul just dug this up for me, thanks Rahul!
#457
Still doesn't explain why, though.

[fredg02]

@fredg02 can you tell how often the EA builds are updated for all Jenkins instances?

They are mostly updated on demand.