About crosscompile julia with aarch64

[billfang2024]

We crosscompile julia V1.10.6 on aarch64,met this problem,

My host setting is :

Linux 0ea0ffc92e59 5.10.0-153.56.0.134.oe2203sp2.x86_64 #1 SMP Wed May 29 14:04:28 CST 2024 x86_64 x86_64 x86_64 GNU/Linux

crosscompile setting is ：

SRCDIR=`pwd`
GCCDIR="${SRCDIR}/musl-cross-make/output/bin"
JULIADIR="${SRCDIR}/julia-1.10.6"
target_host=aarch64-linux-musleabi

export AR=$GCCDIR/$target_host-ar
export AS=$GCCDIR/$target_host-as
export CC=$GCCDIR/$target_host-gcc
export CXX=$GCCDIR/$target_host-g++
export FORTRAN=$GCCDIR/$target_host-gfortran
export LD=$GCCDIR/$target_host-ld
export RANLIB=$GCCDIR/$target_host-ranlib
export STRIP=$GCCDIR/$target_host-strip
export OBJDUMP=$GCCDIR/$target_host-objdump
export OBJCOPY=$GCCDIR/$target_host-objcopy
export NM=$GCCDIR/$target_host-nm

export CPPFLAGS="-I${JULIADIR}/usr/include"
export CFLAGS="-fPIC -D__MUSL__=1 -I${JULIADIR}/usr/include"
export CXXFLAGS="-fPIC -D__MUSL__=1 -I${JULIADIR}/usr/include"
export LDFLAGS="-L${JULIADIR}/usr/lib"

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${JULIADIR}/usr/lib

/bin/sh: ../../../bin/llvm-tblgen: cannot execute binary file: Exec format error
include/llvm/IR/CMakeFiles/intrinsics_gen.dir/build.make:153: recipe for target 'include/llvm/IR/Attributes.inc' failed
make[4]: *** [include/llvm/IR/Attributes.inc] Error 126
CMakeFiles/Makefile2:9265: recipe for target 'include/llvm/IR/CMakeFiles/intrinsics_gen.dir/all' failed
make[3]: *** [include/llvm/IR/CMakeFiles/intrinsics_gen.dir/all] Error 2
Makefile:155: recipe for target 'all' failed
make[2]: *** [all] Error 2
/home/openharmony/ohpc-app-julia/julia-1.10.6/deps/llvm.mk:268: recipe for target 'scratch/llvm-julia-15.0.7-10/build_Release/build-compiled' failed
make[1]: *** [scratch/llvm-julia-15.0.7-10/build_Release/build-compiled] Error 2
Makefile:79: recipe for target 'julia-deps' failed

[inkydragon]

Based on the Supported platforms for julia and my observations of the mainline CI:

• aarch64-linux-gnu is a Tire 2 platform, and it builds and passed tests on master
  https://buildkite.com/julialang/julia-master/builds/43152#0193dd96-9f61-4730-940a-c319d0788199
• x86-64-linux-musl is a Tire 2 platform, but it rarely builds successfully, and I occasionally see successful builds on the master branch.
  https://buildkite.com/julialang/julia-master/builds/43152#0193dd96-a400-4a1b-aeb0-5167fca389c9

Then you are trying to build aarch64-linux-musleabi, which is not yet listed among the supported platforms.
My suggestion is to try cross-compiling for aarch64-linux-gnu first.

A better way for supporting new platforms is to compile from source on the target platform (without pre-built dependencies)
One issue mentioned that julia doesn't have good cross-compilation support yet. Not sure things have improved though.
#30338

---

llvm-tblgen: cannot execute binary file: Exec format error
...
recipe for target 'scratch/llvm-julia-15.0.7-10/build_Release/build-compiled' failed

The error occurred during the build of LLVM.
One of my guesses is: llvm-tblgen needs to be executed on the host, and the architecture of the cross-compiled executable doesn't match.
You can confirm this by cross-compile the official version of LLVM 15 (without the julia-specific patch).

Maybe related issue: NixOS/nixpkgs#39170, NixOS/nixpkgs#40602

[billfang2024]

Thanks you very much.
We have a question
Why call cross platform commands on the host
Is it because our environment is not properly configured
Is it possible that the compilation parameters passed into LLVM are inappropriate？
############################

file deps/scratch/llvm-julia-15.0.7-10/build_Release/bin/llvm-tblgen
deps/scratch/llvm-julia-15.0.7-10/build_Release/bin/llvm-tblgen: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-aarch64.so.1, not stripped
############################
include/llvm/IR/Attributes.inc: /home/openharmony/ohpc-app-julia/julia-1.10.6/deps/srccache/llvm-julia-15.0.7-10/llvm/include/llvm/IR/Attributes.td
@$(CMAKE_COMMAND) -E cmake_echo_color --switch=$(COLOR) --blue --bold --progress-dir=/home/openharmony/ohpc-app-julia/julia-1.10.6/deps/scratch/llvm-julia-15.0.7-10/build_Release/CMakeFiles --progress-num=$(CMAKE_PROGRESS_1) "Building Attributes.inc..."
cd /home/openharmony/ohpc-app-julia/julia-1.10.6/deps/scratch/llvm-julia-15.0.7-10/build_Release/include/llvm/IR &&
../../../bin/llvm-tblgen -gen-attrs
-I /home/openharmony/ohpc-app-julia/julia-1.10.6/deps/srccache/llvm-julia-15.0.7-10/llvm/include/llvm/IR
-I/home/openharmony/ohpc-app-julia/julia-1.10.6/deps/scratch/llvm-julia-15.0.7-10/build_Release/include
-I/home/openharmony/ohpc-app-julia/julia-1.10.6/deps/srccache/llvm-julia-15.0.7-10/llvm/include
/home/openharmony/ohpc-app-julia/julia-1.10.6/deps/srccache/llvm-julia-15.0.7-10/llvm/include/llvm/IR/Attributes.td
--write-if-changed -o
/home/openharmony/ohpc-app-julia/julia-1.10.6/deps/scratch/llvm-julia-15.0.7-10/build_Release/include/llvm/IR/Attributes.inc
############################

[inkydragon]

Why call cross platform commands on the host

I don't know. Maybe ask the LLVM community.

The LLVM documentation does mention this:

You may also want to set the LLVM_NATIVE_TOOL_DIR option - pointing at a directory with prebuilt LLVM tools (llvm-tblgen, clang-tblgen etc) for the build host, allowing you to them reuse them if available.

https://llvm.org/docs/HowToCrossCompileLLVM.html#configuring-cmake

The options there should be straightforward, except maybe for LLVM_TABLEGEN and CLANG_TABLEGEN. They have to be specified, because these binaries need to run on the host, but the build compiles them for the target, so it cannot use what it just build.

https://stackoverflow.com/a/61771495

[giordano]

Why are you building LLVM from source? Did you set USE_BINARYBUILDER=0 or something similar? We can successfully cross-compile Julia (without building the sysimage though) for aarch64-linux-musl in Yggdrasil, but there we build all the dependencies separately, and for LLVM we need to do a complicated dance to first build the host tools with the host toolchain and then the library for the target (of course this would be much simpler if CMake was a sensible build system and allowed to specify two different toolchains for the host and the target, but unfortunately it isn't).

But as mentioned above, Julia on Musl unfortunately doesn't work very well due to a couple of major issues (#40556, #52707), so even if you managed to solve compilation issues in LLVM you'd have loads of other issues at runtime (if you ever get to build Julia at all, since it crashes very often due to limited stack size available with this libc).

[billfang2024]

Thank you very much.
We'll digest it slowly, it's a bit complicated~