[libjulia] add or update 1.12-DEV, 1.11-DEV, 1.10

[fingolfin]

No description provided.

[fingolfin]

Should this be 1.11? 1.12? It's not clear to me given that 1.10.8 already contained 1.11-dev... Huh

[benlorenz]

I think at some point we gave up with this number and just kept bumping the patch version. Since it is only used as BuildDependency there is no need for compat statements anyway.
Those packages that need a specific version just fix that exact version and that's about it.

[fingolfin]

macOS builds in Julia >= 1.10 failing now.

Weird also mingw builds for Julia 1.6.3 and 1.7.0 ?!? Has something changed in that regard on Yggdrasil? Error is e.g.

[14:50:28] /workspace/destdir/lib/libuv.a(libuv_la-process.o): In function `uv__kill':
[14:50:28] /workspace/srcdir/libuv/src/win/process.c:1292: undefined reference to `_imp__CoTaskMemFree@4'
[14:50:28] collect2: error: ld returned 1 exit status
[14:50:28] make[2]: *** [Makefile:108: /workspace/srcdir/julia-1.7.0/src/flisp/flisp.exe] Error 1
[14:50:28] make[2]: Leaving directory '/workspace/srcdir/julia-1.7.0/src/flisp'

Aha, libuv; and for 1.6.3 and 1.7.0 we don't specify a version for those. Let's see.

[fingolfin]

Pushed a change which might or might not help 1.6 and 1.7 with mingw, by hardcoding JLL versions there, too.

For darwin, I see these errors:

[14:57:50]  x86_64-apple-darwin14-clang -mmacosx-version-min=10.14 -m64 -O3 -g -DDEP_LIBS="\"libgcc_s.1.dylib:julia/libopenlibm.dylib:@:@libjulia-internal.1.10.dylib:@libjulia-codegen.1.10.dylib:\"" -std=gnu11 -pipe -fPIC -fno-strict-aliasing -D_FILE_OFFSET_BITS=64 -I/workspace/srcdir/julia-1.10.0/src -I/workspace/srcdir/julia-1.10.0/src -I/workspace/srcdir/julia-1.10.0/src/support -I/workspace/srcdir/julia-1.10.0/usr/include -ffreestanding /workspace/srcdir/julia-1.10.0/cli/trampolines/trampolines_x86_64.S -c -o loader_trampolines.o
[14:57:50] error: /opt/x86_64-apple-darwin14/bin/x86_64-apple-darwin14-nm: invalid argument -D
[14:57:50] Usage: /opt/x86_64-apple-darwin14/bin/x86_64-apple-darwin14-nm [-agnopruUmxjlfAPL[s segname sectname] [-] [-t format] [[-arch <arch_flag>] ...] [file ...]

and then a bit later

[14:57:50]  x86_64-apple-darwin14-clang -mmacosx-version-min=10.14 -m64  -std=gnu11 -pipe -fPIC -fno-strict-aliasing -D_FILE_OFFSET_BITS=64 -I/workspace/srcdir/julia-1.10.0/src -I/workspace/srcdir/julia-1.10.0/src -I/workspace/srcdir/julia-1.10.0/src/support -I/workspace/srcdir/julia-1.10.0/usr/include -ffreestanding -shared -O3 -g -DDEP_LIBS="\"libgcc_s.1.dylib:julia/libopenlibm.dylib:@:@libjulia-internal.1.10.dylib:@libjulia-codegen.1.10.dylib:\"" ./loader_lib.o ./loader_trampolines.o -o /workspace/srcdir/julia-1.10.0/usr/lib/libjulia.1.10.0.dylib -Wl,-compatibility_version,1.10 -Wl,-current_version,1.10.0  -ffreestanding -L/workspace/srcdir/julia-1.10.0/usr/lib -L/workspace/srcdir/julia-1.10.0/usr/lib -lSystem
[14:57:50] ld64.lld: error: ./loader_trampolines.o: unhandled file type
[14:57:50] clang-16: error: linker command failed with exit code 1 (use -v to see invocation)
[14:57:50] make[1]: *** [Makefile:117: /workspace/srcdir/julia-1.10.0/usr/lib/libjulia.1.10.0.dylib] Error 1
[14:57:50] make[1]: Leaving directory '/workspace/srcdir/julia-1.10.0/cli'
[14:57:50] make: *** [Makefile:94: julia-cli-release] Error 2
[14:57:50] make: *** Waiting for unfinished jobs....

---

Also this (harmless) warnings pops up a lot:

[14:57:50] make[1]: xcode-select: No such file or directory

Harmless, but suggests a possible improvement in the Julia build system

[benlorenz]

The loader_trampolines.o error might be a race condition because we are building the debug and non-debug version at the same time. Usually this disappears when re-running the job, see e.g. #7484 (comment).

[fingolfin]

Mingw builds working now.

Darwin/macOS jobs still failing.

@benlorenz Thanks for the hint about loader_trampolines.o , I can try a workaround then.

The x86_64-apple-darwin14-nm: invalid argument -D error also remains.
It comes from src/Makefile:

LLVM_SHLIB_SYMBOL_VERSION := $(shell nm -D --with-symbol-versions $(shell $(LLVM_CONFIG_HOST) --libfiles --link-shared | awk '{print $1; exit}') | \
                               grep _ZN4llvm3Any6TypeId | head -n 1 | sed -e 's/.*@//')

I wonder what's going on there? Both GNU nm as well as llvm-nm support it. Where does x86_64-apple-darwin14-nm come from? @giordano ?

[fingolfin]

Let's see if my patch from JuliaLang/julia#53437 does the job

[giordano]

Where does x86_64-apple-darwin14-nm come from?

I'm not entirely sure, but it may be cctools, we use an old-ish version:

Yggdrasil/0_RootFS/gcc_sources.jl

Lines 158 to 164 in 7fc6b66

	# MacOS doesn't actually use binutils, it uses cctools
	binutils_sources = [
	GitSource("https://github.com/tpoechtrager/apple-libtapi.git",
	"a66284251b46d591ee4a0cb4cf561b92a0c138d8"),
	GitSource("https://github.com/tpoechtrager/cctools-port.git",
	"634a084377ee2e2932c66459b0396edf76da2e9f"),
	]

[fingolfin]

Chasing down nm:

sandbox:${WORKSPACE} # which nm
/opt/bin/x86_64-apple-darwin14-libgfortran3-cxx03/nm
sandbox:${WORKSPACE} # cat /opt/bin/x86_64-apple-darwin14-libgfortran3-cxx03/nm
#!/bin/bash
# This compiler wrapper script brought into existence by `generate_compiler_wrappers!()`

if [ "x${SUPER_VERBOSE}" = "x" ]; then
    vrun() { "$@"; }
else
    vrun() { echo -e "\e[96m$@\e[0m" >&2; "$@"; }
fi

ARGS=( "$@" )
PRE_FLAGS=()
POST_FLAGS=()


if [[ " ${ARGS[@]} " == *"-march="* ]]; then
    echo "BinaryBuilder: Cannot force an architecture via -march" >&2
    exit 1
fi

vrun ${CCACHE} /opt/x86_64-apple-darwin14/bin/x86_64-apple-darwin14-nm "${PRE_FLAGS[@]}" "${ARGS[@]}" "${POST_FLAGS[@]}"
sandbox:${WORKSPACE} # file /opt/x86_64-apple-darwin14/bin/x86_64-apple-darwin14-nm
/opt/x86_64-apple-darwin14/bin/x86_64-apple-darwin14-nm: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, with debug_info, not stripped
sandbox:${WORKSPACE} # strings /opt/x86_64-apple-darwin14/bin/x86_64-apple-darwin14-nm | fgrep -i cc
access
cctools-949.0.1
@(#)PROGRAM:ld  PROJECT:cctools-949.0.1
malformed object (LC_VERSION_MIN_MACOSX is set but the LC_BUILD_VERSION load command is not for MACCATALYST)
malformed object (the two LC_BUILD_VERSION load commands are not for MACOS and MACCATALYST)
GCC: (Alpine 9.3.0) 9.3.0
access

So yeah, cctools.

[fingolfin]

But since we override LLVM_SHLIB_SYMBOL_VERSION anyway, it doesn't matter. Still, it is confusing to see. Ah well.

[fingolfin]

Darwin patch helped. Now 1.10 builds again. but new errors on 1.11 (and 1.12, which at this point is still quite similar):

[17:09:43] make[2]: Leaving directory '/workspace/srcdir/julia/src/flisp'
[17:09:43]  x86_64-apple-darwin14-clang++ -mmacosx-version-min=10.14 -m64 -shared  -pipe -fPIC -fno-rtti -std=c++17 -pedantic -D_FILE_OFFSET_BITS=64   -O0 -DJL_DEBUG_BUILD -fstack-protector -g -D_GNU_SOURCE -I. -I/workspace/srcdir/julia/src -I/workspace/srcdir/julia/src/flisp -I/workspace/srcdir/julia/src/support -I/workspace/destdir/include -I/workspace/srcdir/julia/usr/include -I/workspace/srcdir/julia/deps/valgrind -Wall -Wno-strict-aliasing -fno-omit-frame-pointer -fvisibility=hidden -fno-common -Wno-comment -Wpointer-arith -Wundef -Wno-return-type-c-linkage -DJL_BUILD_ARCH='"x86_64"' -DJL_BUILD_UNAME='"Darwin"' -I -DLLVM_SHLIB "-DJL_SYSTEM_IMAGE_PATH=\"../lib/julia/sys-debug.dylib\"" "-DJL_LIBJULIA_SONAME=\"@rpath/libjulia-debug.1.11.dylib\"" ./jltypes.dbg.obj ./gf.dbg.obj ./typemap.dbg.obj ./smallintset.dbg.obj ./ast.dbg.obj ./builtins.dbg.obj ./module.dbg.obj ./interpreter.dbg.obj ./symbol.dbg.obj ./dlload.dbg.obj ./sys.dbg.obj ./init.dbg.obj ./task.dbg.obj ./array.dbg.obj ./genericmemory.dbg.obj ./staticdata.dbg.obj ./toplevel.dbg.obj ./jl_uv.dbg.obj ./datatype.dbg.obj ./simplevector.dbg.obj ./runtime_intrinsics.dbg.obj ./precompile.dbg.obj ./jloptions.dbg.obj ./mtarraylist.dbg.obj ./threading.dbg.obj ./scheduler.dbg.obj ./stackwalk.dbg.obj ./gc.dbg.obj ./gc-debug.dbg.obj ./gc-pages.dbg.obj ./gc-stacks.dbg.obj ./gc-alloc-profiler.dbg.obj ./gc-page-profiler.dbg.obj ./method.dbg.obj ./jlapi.dbg.obj ./signal-handling.dbg.obj ./safepoint.dbg.obj ./timing.dbg.obj ./subtype.dbg.obj ./rtutils.dbg.obj ./gc-heap-snapshot.dbg.obj ./crc32c.dbg.obj ./APInt-C.dbg.obj ./processor.dbg.obj ./ircode.dbg.obj ./opaque_closure.dbg.obj ./codegen-stubs.dbg.obj ./coverage.dbg.obj ./runtime_ccall.dbg.obj -Wl,-rpath,'@loader_path/' -o /workspace/srcdir/julia/usr/lib/libjulia-internal-debug.1.11.0.dylib -L/workspace/destdir/lib -Wl,-compatibility_version,1.11 -Wl,-current_version,1.11.0 -L/workspace/srcdir/julia/usr/lib -L/workspace/srcdir/julia/usr/lib -Xlinker -all_load ./flisp/libflisp-debug.a -Xlinker -all_load ./support/libsupport-debug.a -ljulia-debug -Xlinker -all_load /workspace/destdir/lib/libuv.a -Xlinker -all_load -lutf8proc   -L/workspace/destdir/lib -lLLVM  -framework CoreFoundation -Wl,-U,__dyld_atfork_parent -Wl,-U,__dyld_atfork_prepare -Wl,-U,__dyld_dlopen_atfork_parent -Wl,-U,__dyld_dlopen_atfork_prepare -Wl,-U,_jl_image_pointers -Wl,-U,_jl_system_image_data -Wl,-U,_jl_system_image_size
[17:09:43] ld64.lld: error: undefined symbol: operator new(unsigned long, std::align_val_t)
[17:09:43] >>> referenced by new:230 (/opt/x86_64-apple-darwin14/x86_64-apple-darwin14/sys-root/usr/include/c++/v1/new:230)
[17:09:43] >>>               ./gc-heap-snapshot.dbg.obj:(symbol std::__1::__libcpp_allocate(unsigned long, unsigned long)+0x37)
[17:09:43] 
[17:09:43] ld64.lld: error: undefined symbol: operator delete(void*, std::align_val_t)
[17:09:43] >>> referenced by new:312 (/opt/x86_64-apple-darwin14/x86_64-apple-darwin14/sys-root/usr/include/c++/v1/new:312)
[17:09:43] >>>               ./processor.dbg.obj:(symbol void std::__1::_DeallocateCaller::__do_call<std::align_val_t>(void*, std::align_val_t)+0x19)
[17:09:43] clang-16: error: linker command failed with exit code 1 (use -v to see invocation)
[17:09:43] make[1]: *** [Makefile:397: /workspace/srcdir/julia/usr/lib/libjulia-internal-debug.1.11.0.dylib] Error 1
[17:09:43] make[1]: Leaving directory '/workspace/srcdir/julia/src'
[17:09:43] make: *** [Makefile:97: julia-src-debug] Error 2
[17:09:44]  ---> make USE_CROSS_FLISP=1 NO_GIT=1 LDFLAGS="${LDFLAGS}" CFLAGS="${CFLAGS}" CXXFLAGS="${CXXFLAGS}" -j${nproc} VERBOSE=1 ${MAKE_TARGET} DSYMUTIL=true
[17:09:44]  ---> make USE_CROSS_FLISP=1 NO_GIT=1 LDFLAGS="${LDFLAGS}" CFLAGS="${CFLAGS}" CXXFLAGS="${CXXFLAGS}" -j${nproc} VERBOSE=1 ${MAKE_TARGET} DSYMUTIL=true
[17:09:44] Previous command exited with 2
[17:09:44] Child Process exited, exit code 2

[giordano]

Try with clang_use_lld=false, although this error doesn't feel like it should depend on the linker (hopefully?).

[benlorenz]

Reading https://reviews.llvm.org/D129198, I think we might need to do the osx framework update dance to get these symbols (to 10.13 or newer, I think the default is 10.12 on x86_64). Not really sure in what form the compiler is supposed to deal with this but if I understand some related comments correct then this operator is defined in the headers but not in the libc++ for 10.12.

Setting -fno-aligned-allocation also seems to work, at least when building x86_64-apple-darwin-julia_version+1.12.0 locally:

    # macos 10.12 has aligned alloc in the header but not in libc++
    # leading to linker errors
    if [[ "${target}" == *x86_64-apple* ]]; then
        CXXFLAGS=-fno-aligned-allocation
    fi

Not totally sure how this will affect the downstream packages, maybe the CXXFLAGS approach is better since then the symbols should not appear anymore, otherwise we might end up needing to update the osx framework for all downstream recipes as well?

[fingolfin]

@benlorenz thanks that did the trick