Ensure all Windows build machines have same compiler level

[pshipton]

Platform:
Windows

Architecture:
any

@jdekonin mentioned the Windows playbooks install the latest VS2017 compiler. Meaning that if build machines are created at different times, they could have different levels of the VS2017 compiler on them. Ideally all build machines should have the same level of VS2017, so the build isn't affected by what machine it lands on.

Not sure if the same issue applies to VS2010.

[sxa]

Do you have a proposal for how to implement this? And is this a theoretical issue or something you've seen an issue with?

[pshipton]

My proposal is to fix at some known version for all machines, until such time as all machines are updated to a different known version. This a theoretical issue. If there is some difference in the code generated from different compiles, it could cause a lot of confusion if the behavior changes based on the machine where the build was compiled.

[sxa]

My question was more whether you knew of an easy way to fix it - we pull the installer from https://aka.ms/vs/15/release/vs_community.exe

We might be able (subject to licensing) to copy a fixed version of that executable somewhere and run that instead of downloading it from the original source each time, but that would also be subject to whether MS's installer allows the downloading of earlier versions.

[pshipton]

I believe @jdekonin mentioned there is a way to get particular versions.

[jdekonin]

Not for getting the older versions. I have @Kevin-Bonilla looking into whether that is possible.

From my msdn account I don't see any download for older versions, so unless there is a method via the installer or someone has an "offline installer installation", I don't see any options to accomplish this yet.

[ghost]

Not for getting the older versions. I have @Kevin-Bonilla looking into whether that is possible.

From my msdn account I don't see any download for older versions, so unless there is a method via the installer or someone has an "offline installer installation", I don't see any options to accomplish this yet.

Yeah I still need to do some further research to see if this is still possible. I tried digging through some DevOps microsoft tool as well called Visual Studio Dev Essentials but that product as well only gives me the Visual Studio Installer which would install the latest 15.9.22

[sxa]

Copying in my comment from #1354 as I don't want conversations fragmented across issues
This would need wider approval I think before doing it than just me.
Are we also confident that such a solution is allowed within the terms of the license agreement for the free compiler download?

[pshipton]

It's possible to create an offline cache of a particular version. Then all machines should get that version until there is a decision to switch to a different one.
https://docs.microsoft.com/en-us/visualstudio/install/create-an-offline-installation-of-visual-studio?view=vs-2017

[pshipton]

I believe all the OpenJ9 machines were updated to 15.9.22, @jdekonin will need to confirm. However the latest version is now 15.9.23 so you can't get 15.9.22 any more unless you already have an offline cache for it.

[jdekonin]

OpenJ9 machines are at 15.9.22 using a offline installer for VS Pro, not the Community version as corporate machines can't be using the free version.

[pshipton]

@jdekonin The OpenJ9 machines used publicly at https://ci.eclipse.org/openj9?

[jdekonin]

Yes these machines are also using the same VS Pro. I checked with my licensing contact and she advised we could use the same for internal IBM and the IBM Cloud Windows instances.

This was more of a convenience issue as my team supports not only these 6 windows machines but an internal IBM machine build/test farm that cannot use the VS Community version.

[sxa]

Here at adopt we use the community version as installed by the playbooks. Unless this can be done through the community version this /may/ be a non-starter ... Although we do have some MSFT people on the team but I have a slight preference for using the same freely available tools that others may use our scripts with.

[pshipton]

The community version is listed in the instructions for creating an offline cache.
https://docs.microsoft.com/en-us/visualstudio/install/create-an-offline-installation-of-visual-studio?view=vs-2017

[sxa]

Bumping this as from a secure (SSDF) perspective and from a perspective of trying to recover MSVS_2013 this would be beneficial

[andrew-m-leonard]

This is a potential problem for "reproducible builds", as any difference in compiler, even a minor build version, introduces binary differences... Without the exact same build level of VisualStudio, you cannot reproduce a build.

@gdams fyi

[gdams]

@andrew-m-leonard take a look at how it's done at the OpenJDK project (https://github.com/openjdk/jdk/blob/87340fd5408d89d9343541ff4fcabde83548a598/.github/workflows/build-windows.yml#L100)

Essentially we need to set a specific version of msvc toolset and we should then have identical compiler levels on all windows machines.

[andrew-m-leonard]

So i've done some test installs and builds, and if I install vs_Community19, then select say build tools 14.27, it seems to also install the latest 14.29 if I do a straight build.
However, I can select 14.27 directly to build openjdk by using the --with-msvc-toolset-version=14.27 configure arg.
Which is good, however it does not allow you to select "fix update levels", eg.14.27.111, which is on most of the Adoptium windows nodes currently. On a new node if you select 14.27 you get 14.27.120 currently.

For reproducibility it seems to imply we will need to cache the install ?

[sxa]

@andrew-m-leonard What's your feeling on trying to switch the older JDK releases over to building on Windows Server 2022 before the release later this month?

[andrew-m-leonard]

So I guess we need to do this at some point, and can't foresee there being a problem. The only thing to question, is capacity of 2022 nodes for all versions? or do we just provision via Azure?

[sxa]

The intention world be to start switching all of the other build machines we have over to 2022 if we make the change. We're potentially building the versions at once for any given version (x32/x64/Arm64) and we have two available at the moment: https://ci.adoptium.net/label/build&&win2022/

[sxa]

Done for JDK20+ by moving to server 2022 build machines - we should look to progress this with earlier versions after the July releases

[sxa]

@steelhead31 has confirmed that the wix label can be applied to the win2022 nodes and still produce usable installers so we may now aim to simply reprovision some of the older ones with win2022 and the fixed compiler instead of adjusting them.

[sxa]

We currently have 1/5 online machine labelled build&win2012

Noting that JDK8 and perhaps more are still built using those labels so they are currently all being funnelled to that one machine so it is currently a single point of failure (Although we can bring others back online if required)

[steelhead31]

As per this issue #3158 and this PR adoptium/ci-jenkins-pipelines#771 all Temurin JDK builds have been migrated to Windows Server 2022. These machines were all built using the infrastructure playbook, and as such have a consistent Visual Studio 2017 layout applied to them.