Illegal Instruction occurs when executing on Ubuntu 16.04 ARM64 using 2.2.100-preview1-008633

[bbusacker]

This blocks the ability to build applications on this arm64 platform.

Steps to reproduce

It can be easily reproduced by typing "dotnet new" on the command-line

It can also be reproduced by attempting to build vsts-agent on Ubuntu 16.04 Arm64 platform. dev.sh need dotcli library version changed from 2.0.3 to 2.2.100-preview1-008633.

Expected behavior

Build fails and illegal instructions is printed on the console and also in dmesg

Actual behavior

No illegal instruction should occur

Environment data

dotnet --info output:

root@dmx-yic1:~/vsts-agent/src# ../_dotnetsdk/2.2.100-preview1-008633/dotnet --info
.NET Core SDK (reflecting any global.json):
Version: 2.2.100-preview1-008633
Commit: 8c937a0

Runtime Environment:
OS Name: ubuntu
OS Version: 16.04
OS Platform: Linux
RID: ubuntu.16.04-x64
Base Path: /root/vsts-agent/_dotnetsdk/2.2.100-preview1-008633/sdk/2.2.100-preview1-008633/

Host (useful for support):
Version: 2.1.0-preview3-26411-06
Commit: 8faa8fcfcf

.NET Core SDKs installed:
2.2.100-preview1-008633 [/root/vsts-agent/_dotnetsdk/2.2.100-preview1-008633/sdk]

.NET Core runtimes installed:
Microsoft.NETCore.App 2.1.0-preview3-26411-06 [/root/vsts-agent/_dotnetsdk/2.2.100-preview1-008633/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download

output of uname -a:

root@dmx-yic1:~/vsts-agent/src# uname -a
Linux dmx-yic1 3.18.14-eng #4287 SMP PREEMPT Thu Nov 16 00:57:06 MST 2017 aarch64 aarch64 aarch64 GNU/Linux

Let me know if you need any other details

[livarcocc]

@steveharter can you help look into this?

[steveharter]

cc @janvorli.

@weshaggard Do we have a supported matrix of ARM64? The current core-setup build for ARM64 just says "Linux (arm64) (for glibc based OS)" and does not have Ubuntu, etc specific builds.

@bbusacker do you get the same errors during "dotnet new" and when building vsts-agent?

[janvorli]

@bbusacker This looks strange in the dotnet --info output:
RID: ubuntu.16.04-x64
It should have arm64 and not x64 in it. That could explain the invalid instruction thing.

As a workaround, can you try to build and publish for arm64 on an x64 Linux machine and just copy the output to the arm64 one and see if it works? (run dotnet publish -r linux-arm64 and then grab the contents of the publish folder - the publish will print the path to the console).

@steveharter, where does the dotnet --info command get the architecture from?

[steveharter]

where does the dotnet --info command get the architecture from?

From System.Runtime.InteropServices.RuntimeEnvironment.GetRuntimeIdentifier(): https://github.com/dotnet/core-setup/blob/9dff3ce12574699e6d4d31722047f775d92d05fd/src/managed/Microsoft.DotNet.PlatformAbstractions/RuntimeEnvironment.cs

Unfortunately that currently appears to hard-code "x64" for any 64-bit arch; I'll log an issue for that.

[steveharter]

Actually, there is a check after calling RuntimeEnvironment.GetRuntimeIdentifier: https://github.com/dotnet/cli/blob/56c10f65c71e070116ca92f55598015a7f7713c1/src/dotnet/Program.cs#L302

that says:

// if the current RID isn't supported by the shared framework, display the RID the CLI was
// built with instead, so the user knows which RID they should put in their "runtimes" section

I don't have a Linux VM available at the moment to check if "ubuntu.16.04-x64" is in the framework's RID graph to know if this code is executed. The RID graph is in Microsoft.NETCore.App.deps.json

[bbusacker]

@steveharter The cli appears to run until it hits this exception. It starts to buid vsts-agent until it hits this exception. I believe this issue occurs in many of the cli commands new, msbuild, etc.

[bbusacker]

@steveharter If you can point to instructions on how to build and publish for arm on an x64 Linux machine, I'll gladly do it. Never mind I see you gave me a hint above. I will build and run the publish commands. i'll let you know the status shortly.

[steveharter]

@livarcocc @nguerrera because of the bug in RuntimeEnvironment.GetRuntimeIdentifier, could RID fallback not be working correctly, and perhaps installing the x64 packages instead of arm64?

[livarcocc]

I find that unlikely. The RID fallback happens in NuGet, based on the specified rid of the app (which should say linux-arm64) and the Rid graph in the Platforms package.

[janvorli]

@bbusacker I've already described how to build on x64 to target arm64 in my comment above:
run dotnet publish -r linux-arm64 and then grab the contents of the publish folder - the publish will print the path to the console

[bbusacker]

Everything build successful except for the following warnings:

` "/home/bbusacker/dotnetcli/cli/build.proj" (default target) (1) ->
"/home/bbusacker/dotnetcli/cli/src/redist/redist.csproj" (Publish target) (2) ->
(CrossgenPublishDir target) ->
CROSSGEN : warning : Method not found: 'Void Microsoft.DiaSymReader.ISymUnmanagedWriter8._VtblGap1_33()'. while resolving 0x60007b6 - Microsoft.DiaSymReader.ISymUnmanagedWriter8._VtblGap1_33. [/home/bbusacker/dotnetcli/cli/src/redist/redist.csproj]

   "/home/bbusacker/dotnetcli/cli/build.proj" (default target) (1) ->
   "/home/bbusacker/dotnetcli/cli/src/redist/redist.csproj" (Publish target) (2) ->
     CROSSGEN : warning : Method not found: 'Void Microsoft.Cci.ISymUnmanagedWriter7._VtblGap1_33()'. while resolving 0x60004d1 - Microsoft.Cci.ISymUnmanagedWriter7._VtblGap1_33. [/home/bbusacker/dotnetcli/cli/src/redist/redist.csproj]


   "/home/bbusacker/dotnetcli/cli/build.proj" (default target) (1) ->
   (CreateTestAssetPackageNuPkgs target) -> 
     /home/bbusacker/dotnetcli/cli/bin/2/linux-x64/dotnet/sdk/2.2.100-preview1-008633/Microsoft.Common.CurrentVersion.targets(818,5): warning MSB3539: The value of the property "BaseIntermediateOutputPath" was modified after it was used by MSBuild which can lead to unexpected build results. Tools such as NuGet will write outputs to the path specified by the "MSBuildProjectExtensionsPath" instead. To set this property, you must do so before Microsoft.Common.props is imported, for example by using Directory.Build.props.  For more information, please visit https://go.microsoft.com/fwlink/?linkid=869650 [/home/bbusacker/dotnetcli/cli/TestAssets/TestPackages/dotnet-dependency-tool-invoker/dotnet-dependency-tool-invoker.csproj] [/home/bbusacker/dotnetcli/cli/build.proj]


   "/home/bbusacker/dotnetcli/cli/build.proj" (default target) (1) ->
   (TestDebuild target) -> 
     /home/bbusacker/dotnetcli/cli/build/package/Installer.DEB.proj(178,7): warning MSB3073: The command "/usr/bin/env debuild -h" exited with code 127. [/home/bbusacker/dotnetcli/cli/build.proj]

4 Warning(s)

`

I'll now publish the arm64 version and test it.

[bbusacker]

@janvorli

This is what I get when I try to use the command-line you gave me:

`bbusacker@ubuntu:~/dotnetcli/cli$ ./bin/2/linux-x64/dotnet/dotnet publish -r linux-arm64
Microsoft (R) Build Engine version 15.7.145.53551 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

MSBUILD : error MSB1011: Specify which project or solution file to use because this folder contains more than one project or solution file.
`

[bbusacker]

@janvorli

So I tried something different. I used the following command-line on my Ubuntu-16.04.x64 machine:

./build.sh --linux-portable --skip-prereqs --architecture arm64 --targets Default /p:CLIBUILD_SKIP_TESTS=true

then I copied the resulting tar.gz file from packages to the Ubuntu-16.04.arm64 machine and untarred it.

This is the result when called with new:

ufs@dmx-yic1:~$ ./dotnet new
Getting ready...
Illegal instruction

[janvorli]

@bbusacker I was thinking that you wanted to try to build the simple hello world app produced by the dotnet new to verify that things work. So I was assuming you would do:
On a x64 Linux:

dotnet new console
dotnet publish -r linux-arm64

Then you'd copy the files in the publish dir (path reported by the publish step) to your arm64 linux device and try to run the published app there (you would start it by the app name, not the dotnet tool).

This would show whether the issue is in the arm64 cli or somewhere deeper.

[bbusacker]

@janvorli

After adding this: <RuntimeFrameworkVersion>2.1.0-preview2*</RuntimeFrameworkVersion>

to console.csproj

I was able to run

dotnet publish -r linux-arm64

Copying this output to the target ran without the illegal instruction error.

[livarcocc]

I see. The 2.2.100 SDK is still targetting a very old runtime by default. We have a PR to fix that that we need to get to. That old implicit runtime did not have support for arm64, it seems, which is why it failed.

Closing this issue now.