ARM64 CI Leg fails due to missing python3

[adamsitnik]

So far all the ARM64 machines had python3 installed, now they don't.

Related issues:

https://github.com/dotnet/core-eng/issues/5338
https://github.com/dotnet/core-eng/issues/5220

[billwert]

Those issues seem fairly old - does anyone know why ARM64 went away? @ulisesh ?

[jorive]

@billwert What do you mean? They are one day and 16 days old?

[ulisesh]

Those issues seem fairly old - does anyone know why ARM64 went away? @ulisesh ?

The queue Ubuntu.1604.Arm64.Open was moved from running on bare-metal servers to run on containers. The docker images we are using don't include Python3 making Adam's jobs fail. We are working to add Python3

[michellemcdaniel]

Malena is working on adding python3 to all of the queues (windows and linux), so when that work is in, we should be good.

Because this is a machine configuration issues, I would suggest we disable the Arm64 runs until this work is in so that PRs aren't blocked (in particular the dependency update jobs)

[billwert]

@adiaaida good thinking. I was just working on this - check out #351.

[adamsitnik]

@ulisesh @adiaaida any update on this?

In #382 I am introducing a new benchmark that should not be built or run on Arm and it would be great if the CI could tell me if I do not break anything.

[michellemcdaniel]

I believe the change has been checked in and all of the machines (all queues) should have python3 installed on them.

[adamsitnik]

@adiaaida thanks! I will try to re-enable the runs for ARM then

[adamsitnik]

@ulisesh @adiaaida The python3 is still missing on some machines, I tried to re-enable the ARM runs in #387 and failed:

https://mc.dot.net/#/user/dotnet-performance/pr~2Fdotnet~2Fperformance~2Frefs~2Fpull~2F387~2Fmerge/test~2Fperformance_arm64_corefx_netcoreapp3.0_micro~2F/20190315.2/workItem/WorkItem/wilogs

Executed on ubuntu1604openprod-ddarm64s046-0019
chmod: cannot access '/home/helixbot/dotnetbuild/work/fcdf44c5-c163-4bc4-a258-40eef1495cf3/Work/391091c3-37c5-45f1-93f1-6006ec9aed4f/Exec/*.sh': No such file or directory
+ python3 scripts/benchmarks_ci.py --csproj src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture arm64 -f netcoreapp3.0 --bdn-arguments=--allCategories corefx --iterationCount 1 --warmupCount 0 --invocationCount 1 --unrollFactor 1 --strategy ColdStart --stopOnFirstError true --benchview-submission-name .NET Performance - private - merge 387 --benchview-machinepool perfsnake --generate-benchview-data --benchview-run-type private --dotnet-compilation-mode NoTiering
/home/helixbot/dotnetbuild/work/fcdf44c5-c163-4bc4-a258-40eef1495cf3/Payload/scripts/a69f18a8d62e45a39588d77d51bace4e/execute.sh: 4: /home/helixbot/dotnetbuild/work/fcdf44c5-c163-4bc4-a258-40eef1495cf3/Payload/scripts/a69f18a8d62e45a39588d77d51bace4e/execute.sh: python3: not found
+ _commandExitCode=127
+ exit 127

https://mc.dot.net/#/user/dotnet-performance/pr~2Fdotnet~2Fperformance~2Frefs~2Fpull~2F387~2Fmerge/test~2Fperformance_arm64_coreclr_netcoreapp3.0_micro~2F/20190315.1/workItem/WorkItem/wilogs

Executed on ubuntu1604openprod-ddarm64s046-0013
chmod: cannot access '/home/helixbot/dotnetbuild/work/ac3a6691-5c2c-408c-ba94-beada4fbcab0/Work/3f3df429-0584-44e2-8f52-c8b435be543a/Exec/*.sh': No such file or directory
+ python3 scripts/benchmarks_ci.py --csproj src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture arm64 -f netcoreapp3.0 --bdn-arguments=--allCategories coreclr --iterationCount 1 --warmupCount 0 --invocationCount 1 --unrollFactor 1 --strategy ColdStart --stopOnFirstError true --benchview-submission-name .NET Performance - private - merge 387 --benchview-machinepool perfsnake --generate-benchview-data --benchview-run-type private --dotnet-compilation-mode NoTiering
/home/helixbot/dotnetbuild/work/ac3a6691-5c2c-408c-ba94-beada4fbcab0/Payload/scripts/6bd0f092d3414e8698ef52349cb8e97f/execute.sh: 4: /home/helixbot/dotnetbuild/work/ac3a6691-5c2c-408c-ba94-beada4fbcab0/Payload/scripts/6bd0f092d3414e8698ef52349cb8e97f/execute.sh: python3: not found
+ _commandExitCode=127
+ exit 127

[adamsitnik]

It's not an issue anymore