[Alpine Linux] `s390x` platform support #2839

ayakael · 2022-04-17T01:19:56Z

Description

Dotnet6 on Fedora supports s390x. Dotnet6 on Alpine should do so, too.

Major roadblocks

As there are no dotnet6 prebuilt bootstraps release for s390x, we have to build our own by crosscompiling from x86_64 to s390x. Crosscompilation scripts was kindly provided by the Fedora team, and added to a git repo on Apine's Gitlab instance for further work.

Hypothetical steps

clone aports, and cd aports/community/dotnet6-stage0
abuild deps to install build dependencies of dotnet
apk add quilt mono dotnet6-sdk
git clone https://gitlab.alpinelinux.org/ayakael/dotnet6-s390x (original version here)
cd dotnet6-s390x && ./dotnet-s390x-prepare
./dotnet-s390x-build

cc. @nekopsykose @uweigand @omajid

The text was updated successfully, but these errors were encountered:

dotnet-issue-labeler · 2022-04-17T01:19:59Z

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

ayakael · 2022-04-17T01:20:44Z

Current issue: As is currently the case, crosscompilation of runtime fails due to clang13 not having elf64_s390 emulation. How to get this working? Having no experience of crosscompilation, I'd appreciate if anyone has pointers for this.

ayakael · 2022-04-17T01:58:47Z

I made another attempt. Considering no s390x support from lld, I set LD=s390x-linux-gnu-ld and tried making a binutils-cross package for s390x by modifying binutils-cross-embedded aport to target s390x-linux-gnu. New errros:

  Invoking "/var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/eng/native/gen-buildsys.sh" "/var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/src/coreclr" "/var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/artifacts/obj/coreclr/Linux.s390x.Release" s390x clang "" "0" Release ""  -DCLR_CMAKE_PGO_INSTRUMENT=0 -DCLR_CMAKE_OPTDATA_PATH= -DCLR_CMAKE_PGO_OPTIMIZE=1 -DFEATURE_DISTRO_AGNOSTIC_SSL=1 
  WARN: Specific version of clang not found, falling back to use the one in PATH.
  ~/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/artifacts/obj/coreclr/Linux.s390x.Release ~/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/src/coreclr
  Not searching for unused variables given on the command line.
  loading initial cache file /var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/eng/native/tryrun.cmake
  -- The C compiler identification is Clang 13.0.1
  -- The CXX compiler identification is Clang 13.0.1
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - failed
  -- Check for working C compiler: /usr/bin/clang
  -- Check for working C compiler: /usr/bin/clang - broken
  CMake Error at /usr/share/cmake/Modules/CMakeTestCCompiler.cmake:69 (message):
    The C compiler
  
      "/usr/bin/clang"
  
    is not able to compile a simple test program.
  
    It fails with the following output:
  
      Change Dir: /var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/artifacts/obj/coreclr/Linux.s390x.Release/CMakeFiles/CMakeTmp
      
      Run Build Command(s):/usr/bin/make -f Makefile cmTC_30712/fast && /usr/bin/make  -f CMakeFiles/cmTC_30712.dir/build.make CMakeFiles/cmTC_30712.dir/build
      make[1]: Entering directory '/var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/artifacts/obj/coreclr/Linux.s390x.Release/CMakeFiles/CMakeTmp'
      Building C object CMakeFiles/cmTC_30712.dir/testCCompiler.c.o
      /usr/bin/clang --target=s390x-linux-gnu --gcc-toolchain=//usr --sysroot=/    -MD -MT CMakeFiles/cmTC_30712.dir/testCCompiler.c.o -MF CMakeFiles/cmTC_30712.dir/testCCompiler.c.o.d -o CMakeFiles/cmTC_30712.dir/testCCompiler.c.o -c /var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/artifacts/obj/coreclr/Linux.s390x.Release/CMakeFiles/CMakeTmp/testCCompiler.c
      Linking C executable cmTC_30712
      /usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_30712.dir/link.txt --verbose=1
      /usr/bin/clang --target=s390x-linux-gnu --gcc-toolchain=//usr --sysroot=/ -Wl,--rpath-link=//lib/s390x-linux-gnu -Wl,--rpath-link=//usr/lib/s390x-linux-gnu -Wl,--rpath-link=//lib/s390x-linux-gnu -Wl,--rpath-link=//usr/lib/s390x-linux-gnu  CMakeFiles/cmTC_30712.dir/testCCompiler.c.o -o cmTC_30712 
      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: Relocations in generic ELF (EM: 62)
      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: Relocations in generic ELF (EM: 62)
      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: Relocations in generic ELF (EM: 62)
      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: Relocations in generic ELF (EM: 62)
      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: Relocations in generic ELF (EM: 62)
      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: Relocations in generic ELF (EM: 62)
      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: Relocations in generic ELF (EM: 62)
      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: Relocations in generic ELF (EM: 62)
      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: error adding symbols: file in wrong format
      clang-13: error: linker command failed with exit code 1 (use -v to see invocation)
      make[1]: *** [CMakeFiles/cmTC_30712.dir/build.make:100: cmTC_30712] Error 1
      make[1]: Leaving directory '/var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/artifacts/obj/coreclr/Linux.s390x.Release/CMakeFiles/CMakeTmp'
      make: *** [Makefile:127: cmTC_30712/fast] Error 2
      
      
  
    
  
    CMake will not be able to correctly generate this project.
  Call Stack (most recent call first):
    CMakeLists.txt:16 (project)
  
  
  -- Configuring incomplete, errors occurred!
  See also "/var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/artifacts/obj/coreclr/Linux.s390x.Release/CMakeFiles/CMakeOutput.log".
  See also "/var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/artifacts/obj/coreclr/Linux.s390x.Release/CMakeFiles/CMakeError.log".
  ~/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/src/coreclr
  ~/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/artifacts/obj/coreclr/Linux.s390x.Release ~/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/src/coreclr
  Executing make   iltools  -j 24
  make: *** No rule to make target 'iltools'.  Stop.
  ~/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/src/coreclr
  Failed to build "CoreCLR component".

omajid · 2022-04-17T04:15:28Z

cc @BahaVv

uweigand · 2022-04-19T10:40:42Z

This seems to be a problem with the cross-compile environment.

Specifically:

      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: Relocations in generic ELF (EM: 62)

EM 62 is EM_X86_64, which the s390x specific ld doesn't understand. This is probably a problem with the cross-compiler configuration, since the compiler apparently attempts to pull in the host version of crt1.o into a cross-link, instead of the appropriate target version.

crt1.o is part of the libc package, so the question is whether/where you've installed the target libc. In my original Ubuntu-based scripts, multiple architecture versions of libc (and other libraries) can be installed into the same root directory, because of Debian's multi-architecture file system layout. This doesn't work on most other distros, so there you usually have to use something like a sysroot environment.

I see in the repo you've linked above that in dotnet-s390x-build you have this line:

ROOTFS_DIR=/ ./build.sh -arch s390x -c ${RUNTIME_CONF} -cross -clang

ROOTFS_DIR=/ is correct on Ubuntu because of the multi-arch file system layout mentioned above. On other distros, this is nearly certain to be incorrect. Instead, you'll have to create a sysroot directory containing target libraries somewhere, and then point ROOTFS_DIR to that directory.

ayakael · 2022-04-19T13:29:16Z

That strikes me as making total sense. I'll explore this further and come back to you.

ayakael · 2022-08-22T10:01:59Z

Okay, finally found time to setup a crossbuild environement on Alpine. I've managed to get parts of runtime built, but musl is apparently utterly unhappy with dotnet's mono. Many errors such as function-like macro '__GLIBC_PREREQ' is not defined and variable 'is_enum' set but not used [-Wunused-but-set-variable]. I assume I need to go through the build and turn all Werrors off, as I don't see any deal breaker erros.

uweigand · 2022-08-22T10:19:34Z

Okay, finally found time to setup a crossbuild environement on Alpine. I've managed to get parts of runtime built, but musl is apparently utterly unhappy with dotnet's mono. Many errors such as function-like macro '__GLIBC_PREREQ' is not defined and variable 'is_enum' set but not used [-Wunused-but-set-variable]. I assume I need to go through the build and turn all Werrors off, as I don't see any deal breaker erros.

Yes, it's never been built against musl, so there may well be some issues. The __GLIBC_PREREQ most likely comes from this:

#elif defined(__s390x__)

#define MONO_ARCH_HAS_MONO_CONTEXT 1

#include <sys/ucontext.h>

#if __GLIBC_PREREQ(2, 26)
typedef ucontext_t MonoContext;
#else
typedef struct ucontext MonoContext;
#endif

so this needs to be fixed for musl.

ayakael · 2022-08-23T09:13:40Z

Indeed, I'll have to clean up mono for musl. Considering that, I've setup a development protocol for crossbuilding on Alpine.

(done) Port your s390x crossbuild script to APKBUILD, allowing Alpine's abuild to manage dependencies. The goal is to have one APKBUILD for crossbuilding to any Alpine arches without prebuilt SDKs. See repo
Build non-crossbuild x64 build of runtime, roslyn, sdk, aspnetcore and installer. This will iron out any non-crossbuild problems with the build process.
Crossbuild to aarch64. Since I've already ironed out linux-musl-arm64 issues and patched that, this will iron out any crossbuild related issues.
Crossbuild to armhf. This will allow me to bootstrap armhf which is not available right now.
Crossbuild to s390x. Given all of the above, whatever issues I'll encounter will have to do with getting s390x working on musl.

I'm currently at step 2. I've succesfully got runtime built, but I'm stuck on roslyn due to the following:

CSC : error CS1705: Assembly 'Microsoft.CodeAnalysis' with identity 'Microsoft.CodeAnalysis, Version=3.8.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' uses 'System.Collections.Immutable, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' which has a higher version than referenced assembly 'System.Collections.Immutable' with identity 'System.Collections.Immutable, Version=1.1.36.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' [/home/user/projects/dotnet6-cross/src/source-build-tarball-6.0.108/src/roslyn/src/Tools/Source/CompilerGeneratorTools/Source/CSharpSyntaxGenerator/CSharpSyntaxGenerator.csproj]

Roslyn's restore seems really bent on using 1.1.36, any idea what lever I need to switch?

ayakael · 2022-08-23T09:17:52Z

Note that for the above I'm trying to build 6.0.108 / 6.0.8, as fixing 6.0.100 on current Alpine would be a lot of work for nothing.

uweigand · 2022-08-23T11:15:04Z

Looking at your repo, I see things like this:

_build_roslyn() {
        # without these exports, build still attempts a download of dotnet sdk
        export _InitializeDotNetCli="$_cli_root"
        export DOTNET_INSTALL_DIR="$_cli_root"
        export NUGET_PACKAGES="$_nugetdir"

        cd "$builddir"/src/roslyn
        ./eng/build.sh --restore --build --pack -c $_conf
        for I in artifacts/packages/$_conf/*/*.nupkg; do
                $_nuget add $I -Source "$_packagesdir"
        done
}

The point of my cross-build scripts was that you should allow downloading the SDK. Since it's an Intel-based cross-build, those will be available, and they'll provide all the exact versions that the build needs.

Once you've used these scripts to create an initial bootstrap SDK, you can then use that bootstrap SDK to (natively) build the actual final version of the SDK. This would typically use the source-build method, with doesn't require any additional downloads.

ayakael · 2022-08-23T12:10:44Z

Looking at your repo, I see things like this:
_build_roslyn() {
        # without these exports, build still attempts a download of dotnet sdk
        export _InitializeDotNetCli="$_cli_root"
        export DOTNET_INSTALL_DIR="$_cli_root"
        export NUGET_PACKAGES="$_nugetdir"

        cd "$builddir"/src/roslyn
        ./eng/build.sh --restore --build --pack -c $_conf
        for I in artifacts/packages/$_conf/*/*.nupkg; do
                $_nuget add $I -Source "$_packagesdir"
        done
}
The point of my cross-build scripts was that you should allow downloading the SDK. Since it's an Intel-based cross-build, those will be available, and they'll provide all the exact versions that the build needs.

Once you've used these scripts to create an initial bootstrap SDK, you can then use that bootstrap SDK to (natively) build the actual final version of the SDK. This would typically use the source-build method, with doesn't require any additional downloads.

Unfortunately whether we pull dotnet v6.0.100 from Microsoft or use Alpine's dotnet 6.0.108 package, the result is the same. Maybe its a matter of building v6.0.100, as it was in your script originally. I have terrible memories of that release, it was really problematic to build on musl.

uweigand · 2022-08-23T12:26:59Z

Unfortunately whether we pull dotnet v6.0.100 from Microsoft or use Alpine's dotnet 6.0.108 package, the result is the same. Maybe its a matter of building v6.0.100, as it was in your script originally. I have terrible memories of that release, it was really problematic to build on musl.

Hmm, I haven't tried building 6.0.108 myself yet. I'll see if I can get it to work.

ayakael · 2022-08-23T12:34:35Z

Unfortunately whether we pull dotnet v6.0.100 from Microsoft or use Alpine's dotnet 6.0.108 package, the result is the same. Maybe its a matter of building v6.0.100, as it was in your script originally. I have terrible memories of that release, it was really problematic to build on musl.

Hmm, I haven't tried building 6.0.108 myself yet. I'll see if I can get it to work.

Appreciate your help :) Next thing I should try is figuring out what flags source-build applies to roslyn's build

  <ItemGroup>
    <!--
      From roslyn Versions.props:
        The version of Roslyn we build Source Generators against that are built in this
         repository. This must be lower than MicrosoftNetCompilersToolsetVersion,
         but not higher than our minimum dogfoodable Visual Studio version, or else
         the generators we build would load on the command line but not load in IDEs.
      In source-build these don't need to be pinned and can use the source-built versions since it doesn't
      need to support VS.
    -->
    <ExtraPackageVersionPropsPackageInfo Include="SourceGeneratorMicrosoftCodeAnalysisVersion" Version="%24(MicrosoftCodeAnalysisCommonVersion)" />
    <ExtraPackageVersionPropsPackageInfo Include="SourceBuildLiftedSystemCollectionsImmutableVersion" Version="%24(SystemCollectionsImmutableVersion)" />
    <ExtraPackageVersionPropsPackageInfo Include="SourceBuildLiftedSystemReflectionMetadataVersion" Version="%24(SystemReflectionMetadataVersion)" />
    <ExtraPackageVersionPropsPackageInfo Include="SourceBuildLiftedSystemRuntimeCompilerServicesUnsafeVersion" Version="%24(SystemRuntimeCompilerServicesUnsafeVersion)" />
    <ExtraPackageVersionPropsPackageInfo Include="SourceBuildLiftedSystemTextEncodingCodePagesVersion" Version="%24(SystemTextEncodingCodePagesVersion)" />
  </ItemGroup

this from repos/roslyn.proj seems relevant.

uweigand · 2022-08-23T20:28:53Z

Hmm, I wasn't seeing any issue with building roslyn. To be specific, I'm using the version included in 6.0.108, which is Microsoft.NETCore.Compilers.4.0.1-1.22181.2.nupkg etc. built from the roslyn repo commit 487283bcd8d66693091f2800dcf1c8ae37cccdee, using a build command along the lines of:

./eng/build.sh --restore --build --pack -c Release /p:VersionSuffix=1.22181.2

ayakael · 2022-08-23T22:36:48Z

Silly me! It was caused by 0001-lift-version-of-Microsoft.CodeAnalysis.Common-depend.patch from source-build. I pulled roslyn's source code from the cached tarball produced by installer. After reversing that patch, it works. Looking back, I should've pulled the source code directly, but I have limited bandwidth right now so it would've expensive.

ayakael · 2022-08-27T08:32:46Z

Now at the stage of doing a control crossbuild to armhf, skipping aarch64 due to lack of time. I get this error when building runtime:

  /var/build/aports/main/dotnet6-cross/src/source-build-tarball-6.0.108/src/runtime/artifacts/bin/coreclr/Linux.arm.Release/ilasm: line 1: syntax error: unexpected word (expecting ")")
/var/build/aports/main/dotnet6-cross/src/nuget/microsoft.net.sdk.il/6.0.0-rc.1.21415.6/targets/Microsoft.NET.Sdk.IL.targets(145,5): error MSB3073: The command ""/var/build/aports/main/dotnet6-cross/src/source-build-tarball-6.0.108/src/runtime/artifacts/bin/coreclr/Linux.arm.Release/ilasm" -QUIET -NOLOGO -OPTIMIZE -DLL  -DEBUG=OPT -INCLUDE="/var/build/aports/main/dotnet6-cross/src/source-build-tarball-6.0.108/src/runtime/artifacts/obj/System.Runtime.CompilerServices.Unsafe/net6.0-Release/version/" -OUTPUT="/var/build/aports/main/dotnet6-cross/src/source-build-tarball-6.0.108/src/runtime/artifacts/obj/System.Runtime.CompilerServices.Unsafe/net6.0-Release/System.Runtime.CompilerServices.Unsafe.dll" -KEY="/var/build/aports/main/dotnet6-cross/src/nuget/microsoft.dotnet.arcade.sdk/6.0.0-beta.22161.1/tools/snk/MSFT.snk" System.Runtime.CompilerServices.Unsafe.il" exited with code 2. [/var/build/aports/main/dotnet6-cross/src/source-build-tarball-6.0.108/src/runtime/src/libraries/System.Runtime.CompilerServices.Unsafe/src/System.Runtime.CompilerServices.Unsafe.ilproj]

Any idea what might cause this?

EDIT:

aarch64 has a similar error:

  /var/build/aports/main/dotnet6-cross/src/source-build-tarball-6.0.108/src/runtime/artifacts/bin/coreclr/Linux.arm64.Release/ilasm: line 0: syntax error: unterminated quoted string
/var/build/aports/main/dotnet6-cross/src/nuget/microsoft.net.sdk.il/6.0.0-rc.1.21415.6/targets/Microsoft.NET.Sdk.IL.targets(145,5): error MSB3073: The command ""/var/build/aports/main/dotnet6-cross/src/source-build-tarball-6.0.108/src/runtime/artifacts/bin/coreclr/Linux.arm64.Release/ilasm" -QUIET -NOLOGO -OPTIMIZE -DLL  -DEBUG=OPT -INCLUDE="/var/build/aports/main/dotnet6-cross/src/source-build-tarball-6.0.108/src/runtime/artifacts/obj/System.Runtime.CompilerServices.Unsafe/net6.0-Release/version/" -OUTPUT="/var/build/aports/main/dotnet6-cross/src/source-build-tarball-6.0.108/src/runtime/artifacts/obj/System.Runtime.CompilerServices.Unsafe/net6.0-Release/System.Runtime.CompilerServices.Unsafe.dll" -KEY="/var/build/aports/main/dotnet6-cross/src/nuget/microsoft.dotnet.arcade.sdk/6.0.0-beta.22161.1/tools/snk/MSFT.snk" System.Runtime.CompilerServices.Unsafe.il" exited with code 2. [/var/build/aports/main/dotnet6-cross/src/source-build-tarball-6.0.108/src/runtime/src/libraries/System.Runtime.CompilerServices.Unsafe/src/System.Runtime.CompilerServices.Unsafe.ilproj]

EDIT 2
Linking /bin/sh to /bin/bash rather than Alpine's default /bin/busybox changed the error messages, now:

cannot execute binary file: Exec format error
/var/build/aports/main/dotnet6-cross/src/nuget/microsoft.net.sdk.il/6.0.0-rc.1.21415.6/targets/Microsoft.NET.Sdk.IL.targets(145,5): error MSB3073: The command ""/var/build/aports/main/dotnet6-cross/src/source-build-tarball-6.0.108/src/runtime/artifacts/bin/coreclr/Linux.arm.Release/ilasm" -QUIET -NOLOGO -OPTIMIZE -DLL  -DEBUG=OPT -INCLUDE="/var/build/aports/main/dotnet6-cross/src/source-build-tarball-6.0.108/src/runtime/artifacts/obj/System.Runtime.CompilerServices.Unsafe/net6.0-Release/version/" -OUTPUT="/var/build/aports/main/dotnet6-cross/src/source-build-tarball-6.0.108/src/runtime/artifacts/obj/System.Runtime.CompilerServices.Unsafe/net6.0-Release/System.Runtime.CompilerServices.Unsafe.dll" -KEY="/var/build/aports/main/dotnet6-cross/src/nuget/microsoft.dotnet.arcade.sdk/6.0.0-beta.22161.1/tools/snk/MSFT.snk" System.Runtime.CompilerServices.Unsafe.il" exited with code 126. [/var/build/aports/main/dotnet6-cross/src/source-build-tarball-6.0.108/src/runtime/src/libraries/System.Runtime.CompilerServices.Unsafe/src/System.Runtime.CompilerServices.Unsafe.ilproj]

Thus ilasm is being built for arm, but not for x64...

EDIT 3
Fixed by setting ILAsmToolPath to a path containing an x64 ilasm.

ayakael · 2022-08-28T10:23:05Z

Terrific news! armhf crossbuild was a success- although I am now experiencing issues with the first full build of source-build due to build requiring stable versions of ILAsm, ILDAsm, and TestHost. Unfortunately, despite what your script expects, those nupkgs were generated in the NonShipping packages folder. Indeed, the version is 6.0.8-dev. With the /p:DotNetUseShippingVersions=true flag, I got it to 6.0.8-dev.22428.1, but unfortunately source build still does not recognize that as stable. I tried setting /p:PreReleaseVersionLabel=servicing /p:PB_VersionStamp=servicing /p:PackageVersionStamp=servicing for dev to become servicing to no avail. I'm sure it's a matter of putting in the correct flag. Would anyone have any pointers?

EDIT 1
No such luck when setting /p:VersionSuffix=22363 and executing sed 's|<IsShipping>false</IsShipping>|<IsShipping>true</IsShipping>|' -i src/coreclr/.nuget/Directory.Build.props to force Shipping.

EDIT 2
Fixed by manually modifying the nupkg file to set version to 6.0.0 in nuspec. source-build doesn't take anything else for those packages. It'd be nice to have another way to do this, but at least this hack seems to work.

ayakael · 2022-09-18T13:06:45Z

I got Mono fixed, and managed to crossbuild an SDK on dotnet7 :) Right now I just need an s390x VM to work out the kinks. Any idea where I might find that?

uweigand · 2022-09-20T08:21:57Z

I got Mono fixed, and managed to crossbuild an SDK on dotnet7 :) Right now I just need an s390x VM to work out the kinks. Any idea where I might find that?

You can get access to your own s390x VM here: https://linuxone.cloud.marist.edu

If the default setup of those machines doesn't meet your requirements, let me know and we can work something out.

ayakael · 2022-09-26T20:36:04Z

I made a request last week, and awaiting response from their team :). I've been moving along via our CI pipeline, but the build has been really unstable. Every build fails for different reason, so it makes me think that the issue is with the CI.

ayakael · 2022-09-29T05:38:19Z

Got the s390x VM setup, and it looks like dotnet6 is pretty stable build wise - the bootstrap can build on itself (thus only runtime, roslyn, sdk, aspnet, installer). As for the full sdk, I'm getting a bunch of CompilerServer: fatal error - server reports different hash version than build task errors despite setting UseSharedCompilation to false.

uweigand · 2022-09-29T13:57:19Z

Got the s390x VM setup, and it looks like dotnet6 is pretty stable build wise - the bootstrap can build on itself (thus only runtime, roslyn, sdk, aspnet, installer). As for the full sdk, I'm getting a bunch of CompilerServer: fatal error - server reports different hash version than build task errors despite setting UseSharedCompilation to false.

That's strange, the compile server normally works fine for me. This may be a problem with running different dotnet versions at the same time? Are there any old compile server tasks still left running?

ayakael · 2022-09-29T19:25:41Z

Got the s390x VM setup, and it looks like dotnet6 is pretty stable build wise - the bootstrap can build on itself (thus only runtime, roslyn, sdk, aspnet, installer). As for the full sdk, I'm getting a bunch of CompilerServer: fatal error - server reports different hash version than build task errors despite setting UseSharedCompilation to false.

That's strange, the compile server normally works fine for me. This may be a problem with running different dotnet versions at the same time? Are there any old compile server tasks still left running?

The error occurs in a pipeline environment, so it can't be due to running different dotnet versions. It only occurs on s390x.

ayakael · 2022-09-29T20:42:41Z

Fortunately, setting UseSharedCompilation=false in problematic repos/*.proj files does the trick. Next error:
NETSDK1084: There is no application host available for the specified RuntimeIdentifier 'linux-musl-s390x'. . This was to be expected, which is why I crosscompiled runtime.linux-musl-s390x.Microsoft.NETCore.DotNetAppHost.6.0.9.nupkg and Microsoft.NETCore.App.Host.linux-musl-s390x.6.0.9.nupkg. Though note TestHost didn't seem to want to be crosscompiled.

ayakael · 2022-09-30T18:52:00Z

Fixed the apphost issue by doing as fedora did and disabling use of apphost in various components.

Unfortunately the CompileServer issue is basically unsurmountable on full source-build. While I could get the bootstrap components (runtime, roslyn, sdk, aspnetcore, installer) to build, the full stack doesn't pickup UseSharedCompilation=false consistently. source-build is currently the most problematic given it being a collection of various components. What could break the compile server like this for linux-musl-s390x?

@omajid, have you encountered this error when porting to fedora?

CompilerServer: fatal error - server reports different hash version than build task

ayakael · 2022-10-01T08:10:18Z

Disabling shared compilation via

local _sharedcompArray="
   Microsoft.CSharp.CurrentVersion.targets
   Microsoft.VisualBasic.CurrentVersion.targets
   Sdks/Microsoft.NET.Sdk.Razor/targets/Microsoft.NET.Sdk.Razor.Component.targets
   Sdks/Microsoft.NET.Sdk.Razor/targets/Microsoft.NET.Sdk.Razor.Compilation.targets
   Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets
   Roslyn/Microsoft.Managed.Core.targets
"
for i in $_sharedcompArray; do
  sed -i 's|<UseSharedCompilation>true</UseSharedCompilation>|<UseSharedCompilation>false</UseSharedCompilation>|' "$_cli_root"/sdk/*/$i
done

where _cli_root equals location of bootstrap seems to force the false value on this flag accross the board.

Although now build of newtonsoft-json fails with many errors of:

/home/build/dotnet6/community/dotnet6-build/src/bootstrap/sdk/6.0.109/Roslyn/Microsoft.CSharp.Core.targets(75,5): error : [ERROR] FATAL UNHANDLED EXCEPTION: System.InsufficientExecutionStackException: Insufficient stack to continue executing the program safely. This can happen from having too many functions on the call stack or function on the stack using too much stack space. [/home/build/dotnet6/community/dotnet6-build/src/dotnet-v6.0.109/src/source-build/artifacts/source-build/self/src/src/newtonsoft-json/Src/Newtonsoft.Json/Newtonsoft.Json.csproj]

The most recent version of newtonsoft-json gives the same error.

ayakael · 2022-10-01T22:46:34Z

Upgrading to 6.0.401, thus upgrading roslyn, seems to fix the shared compilation errors. The stack trace errors are still an issue though, suggesting that something is broken with my crosscompiled sdk.

edit Nope, shared compilation still broken... there's something fundamentally broken with my crossgen and I can't find what it is. But, one thing is for sure, both dotnet7 and dotnet6 give me System.InsufficientExecutionStackException: Insufficient stack to continue executing the program safely. errors

ayakael · 2022-10-03T15:27:34Z

It turns out the issues that I'm encountering have less to do with s390x and more to do with mono-backed runtime being broken on musl. The stack issue is being tracked here. The shared compilation issue only exists on dotnet6, thus suggesting it has already been fixed and needing backporting. Shared compilation issue tracked here

ayakael · 2022-10-10T16:20:46Z

The stack issues seem to have been fixed. The current roadblock is a mono bug on musl making build of fsharp fail. It is being tracked here: dotnet/runtime#76805. I suspect it is the last bug to deal with given fsharp being one of the last components to build.

ayakael · 2022-10-22T01:01:28Z

Moving right along now, it seems like we got a full source-build build for .NET 6.0 on s390x. For .NET 7.0, there seems to be a regression:

The process cannot access the file '/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/artifacts/obj/System.IO.Ports' because it is being used by another process

Indeed, it doesn't take long for this kind of error to appear in CI. I set ulimit -n 8192 to no avail.

ayakael · 2022-10-25T13:18:16Z

Closing as s390x support is now merged in dotnet6.

uweigand · 2022-10-25T13:35:08Z

That's really excellent news! Thanks for your efforts in making this happen, @ayakael !

ayakael · 2022-10-25T13:42:43Z

That's really excellent news! Thanks for your efforts in making this happen, @ayakael !

Thanks for your help in making it happen ^^ There is still one issue facing .net7 on s390x, but that's going to be tracked in #3020

Now that mono is fixed for musl, ppc64le support is trivial on dotnet7 (we even got a bootstrap crossbuilt already). Our crossbuild aware package builder, designed off of your s390x script, makes it much easier to port to new platforms.

If anyone should want to model a script around our packabe builder, it reads very much like a shell script and it is available here

ayakael · 2024-01-18T20:36:05Z

I'm reopening this issue as s390x and ppc64le on Alpine is still plagued by stability problems, namely issues related to processes accessing the same file and process stalls which seem to be related to roslyn. These are hard to reproduce and I do not have the time / experience to work on this further. For anyone who wants to work further on this, I make the bootstraps available at: https://lab.ilot.io/ayakael/dotnet-stage0/-/releases

dotnet-issue-labeler bot added the untriaged label Apr 17, 2022

MichaelSimons added area-product-experience Improvements in the end-user's product experience and removed untriaged labels Apr 20, 2022

uweigand mentioned this issue Aug 29, 2022

Add support for the s390x architecture dotnet/runtime#34195

Closed

ayakael closed this as completed Oct 25, 2022

huoqifeng mentioned this issue Jan 18, 2024

Make s390x and multi-arch fedora PeerPod images via mkosi confidential-containers/cloud-api-adaptor#1640

Open

8 tasks

ayakael reopened this Jan 18, 2024

[Alpine Linux] s390x platform support #2839

[Alpine Linux] s390x platform support #2839

Comments

ayakael commented Apr 17, 2022 • edited Loading

Description

Major roadblocks

Hypothetical steps

dotnet-issue-labeler bot commented Apr 17, 2022

ayakael commented Apr 17, 2022

ayakael commented Apr 17, 2022

omajid commented Apr 17, 2022

uweigand commented Apr 19, 2022

ayakael commented Apr 19, 2022

ayakael commented Aug 22, 2022

uweigand commented Aug 22, 2022

ayakael commented Aug 23, 2022 • edited Loading

ayakael commented Aug 23, 2022 • edited Loading

uweigand commented Aug 23, 2022

ayakael commented Aug 23, 2022

uweigand commented Aug 23, 2022

ayakael commented Aug 23, 2022

uweigand commented Aug 23, 2022

ayakael commented Aug 23, 2022

ayakael commented Aug 27, 2022 • edited Loading

ayakael commented Aug 28, 2022 • edited Loading

ayakael commented Sep 18, 2022

uweigand commented Sep 20, 2022

ayakael commented Sep 26, 2022

ayakael commented Sep 29, 2022

uweigand commented Sep 29, 2022

ayakael commented Sep 29, 2022

ayakael commented Sep 29, 2022

ayakael commented Sep 30, 2022

ayakael commented Oct 1, 2022

ayakael commented Oct 1, 2022 • edited Loading

ayakael commented Oct 3, 2022 • edited Loading

ayakael commented Oct 10, 2022

ayakael commented Oct 22, 2022

ayakael commented Oct 25, 2022

uweigand commented Oct 25, 2022

ayakael commented Oct 25, 2022

ayakael commented Jan 18, 2024

[Alpine Linux] `s390x` platform support #2839

[Alpine Linux] `s390x` platform support #2839

ayakael commented Apr 17, 2022 •

edited

Loading

ayakael commented Aug 23, 2022 •

edited

Loading

ayakael commented Aug 23, 2022 •

edited

Loading

ayakael commented Aug 27, 2022 •

edited

Loading

ayakael commented Aug 28, 2022 •

edited

Loading

ayakael commented Oct 1, 2022 •

edited

Loading

ayakael commented Oct 3, 2022 •

edited

Loading