[Build] Simplify bootstrapping of I-build tests and fetch configuration files directly from git

[HannesWell]

Instead of fetching the test-configuration files from eclipse.downloads and push it there in each I-build, simply get the test-configuration files directly from the eclipse.platform.releng.aggregator git repository, using partial-clones for improved performance:

• https://git-scm.com/docs/partial-clone
• https://github.blog/open-source/git/get-up-to-speed-with-partial-clone-and-shallow-clone/

This avoids the need to pack and upload the test-configuration zip-file and also simplifies fetching the test-configuration for other branches, like maintenance branches or to test changes.

This also removes from the getEBuilder.xml ant script the possibility to download the test-configuration zip file and to use a local zip.
Furthermore this removes from the I-/Y-builds the creation and publication of the test-configuration zip-file.

As a consequence remove the definitions of the properties EBUILDER_HASH and downloadURL that are not necessary anymore to be passed to the getEBuilder ant-script. And remove the eclipseStream property that now can be computed by the 'getEBuilder' ant-script.
Also rename the folder containing the getEBuilder.xml from to just bootstrap since Hudson is not used anymore.

[HannesWell]

Does anyone know where exactly the push of the folder
https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/tree/master/production
to https://download.eclipse.org/eclipse/relengScripts/production/ happens?
When searching this repository for relengScripts, I didn't found a corresponding location.

[HannesWell]

I requested from the infra-team to install git on all test machines, which is a prerequisite for this change:

• https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/5369

[HannesWell]

With the latest unification of Y-build-tests and smoke tests this now required many files less to adjust.
I'll have to run a few final verification tests, but this is getting close to completion.

[HannesWell]

And can anybody tell which job or part (maybe of the I-build?) transfers the content of
https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/tree/e9fa506d45f771d882d38eb6b79caa804491d733/production
into https://download.eclipse.org/eclipse/relengScripts/production/ ?
Looking at the modification times displayed at the file-server, it looks like it happens less frequently than an I-build.

Once that job/step is identified and this is submitted, I think it should be stopped and the current content should be deleted.
Or are there consumers, like maintenance-builds that cannot be migrated to the new source? I.e. using just
curl -o getEBuilder.xml https://raw.githubusercontent.com/eclipse-platform/eclipse.platform.releng.aggregator/refs/heads/master/production/testScripts/bootstrap/getEBuilder.xml

[MohananRahul]

And can anybody tell which job or part (maybe of the I-build?) transfers the content of https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/tree/e9fa506d45f771d882d38eb6b79caa804491d733/production into https://download.eclipse.org/eclipse/relengScripts/production/ ? Looking at the modification times displayed at the file-server, it looks like it happens less frequently than an I-build.

Once that job/step is identified and this is submitted, I think it should be stopped and the current content should be deleted. Or are there consumers, like maintenance-builds that cannot be migrated to the new source? I.e. using just curl -o getEBuilder.xml https://raw.githubusercontent.com/eclipse-platform/eclipse.platform.releng.aggregator/refs/heads/master/production/testScripts/bootstrap/getEBuilder.xml

I found a scheduled Job https://ci.eclipse.org/releng/job/Publish_shell_scripts_for_releng_jobs/ which publish scripts.

[akurtakov]

Please keep in mind https://bugs.eclipse.org/bugs/show_bug.cgi?id=578862#c1 . This rate limit is what forced us to copy scripts to download - to not hit the download limit.

[HannesWell]

I found a scheduled Job https://ci.eclipse.org/releng/job/Publish_shell_scripts_for_releng_jobs/ which publish scripts.

Thanks! That looks promising.

Please keep in mind https://bugs.eclipse.org/bugs/show_bug.cgi?id=578862#c1 . This rate limit is what forced us to copy scripts to download - to not hit the download limit.

That's of course a problem. Especially since we cannot control what other projects do. But I see multiple solutions:

• continue to put to put just the getEBuilder script at downloads.eclipse
• check weather the rate-limit can be increased since the EF now has a GH Enterprise account? @fredg02 can you tell this?
• stash/archive the getEBuilder.xml in the triggering Jenkins job and retrieve it from there.

[fredg02]

check weather the rate-limit can be increased since the EF now has a GH Enterprise account? @fredg02 can you tell this?

GitHub is generally quite shy about their rate-limiting. See github/docs#8031, for example.

In your case, I would suggest trying authenticated calls as described in https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/912#note_570713.

[HannesWell]

I found a scheduled Job https://ci.eclipse.org/releng/job/Publish_shell_scripts_for_releng_jobs/ which publish scripts.

I have now created #2645 to perform the deployment as part of the master branch build, similar to the deployment of the parent-pom.

In your case, I would suggest trying authenticated calls as described in https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/912#note_570713.

Thanks for that hint. That would be an option, but since I would also like to keep it simple at the caller side I think we should just continue with the Eclipse download-server approach, probably reducing the number of stored files to the necessary minimum. And with #2645, maintaining that space should be simpler and better to understand.

[HannesWell]

I would also like to keep it simple at the caller side I think we should just continue with the Eclipse download-server approach, probably reducing the number of stored files to the necessary minimum. And with #2645, maintaining that space should be simpler and better to understand.

Now that #2645 is available I have implemented it and took the opportunity of the rename to shorten the path to the getEBuilder.xml script and made the deployment more fine-grained, i.e. skip not necessary files. I assume that on maintenance branches the tests are not executed.
But I have to test that it works as desired.

Additionally this also needs the following change, respectively the part of it to install git into the docker-images:

• Install git and procps package in all docker-images #2678

Once that's submitted, this should be ready after a final round of testing.