Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade mariner images to LLVM 16 #840

Merged
merged 3 commits into from
Apr 11, 2023
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/cbl-mariner/2.0/cross/amd64-alpine/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ ARG ROOTFS_DIR=/crossrootfs/x64
RUN /scripts/eng/common/cross/build-rootfs.sh x64 alpine3.13 --skipunmount


FROM mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-crossdeps-amd64-local
FROM mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-crossdeps-llvm-local
ARG ROOTFS_DIR=/crossrootfs/x64

COPY --from=builder $ROOTFS_DIR $ROOTFS_DIR
32 changes: 27 additions & 5 deletions src/cbl-mariner/2.0/cross/amd64/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,33 @@ ARG ROOTFS_DIR=/crossrootfs/x64
RUN /scripts/eng/common/cross/build-rootfs.sh x64 xenial --skipunmount


FROM mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-crossdeps-amd64-local
# Build compiler-rt profile library for PGO instrumentation
RUN mkdir compiler-rt_build && cd compiler-rt_build && \
cmake ../llvm-project.src/compiler-rt \
-DCOMPILER_RT_BUILD_PROFILE=ON \
-DCOMPILER_RT_BUILD_BUILTINS=OFF \
-DCOMPILER_RT_BUILD_CRT=OFF \
-DCOMPILER_RT_BUILD_MEMPROF=OFF \
-DCOMPILER_RT_BUILD_GWP_ASAN=OFF \
-DCOMPILER_RT_BUILD_SANITIZERS=OFF \
-DCOMPILER_RT_BUILD_XRAY=OFF \
-DCOMPILER_RT_BUILD_LIBFUZZER=OFF \
-DCOMPILER_RT_BUILD_ORC=OFF \
\
-DCMAKE_C_COMPILER=clang \
-DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld" \
-DCMAKE_C_COMPILER_TARGET="x86_64-linux-gnu" \
-DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_FLAGS="-v --sysroot=$ROOTFS_DIR" && \
make -j $(getconf _NPROCESSORS_ONLN)

RUN LLVM_VERSION=16.0.0 LLVM_VERSION_MAJOR="${LLVM_VERSION%%.*}" && \
mkdir -p $ROOTFS_DIR/usr/lib/llvm-${LLVM_VERSION_MAJOR}/lib/clang/${LLVM_VERSION}/lib/linux/ && \
cp compiler-rt_build/lib/linux/libclang_rt.profile-x86_64.a $ROOTFS_DIR/usr/lib/llvm-${LLVM_VERSION_MAJOR}/lib/clang/${LLVM_VERSION}/lib/linux/


FROM mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-crossdeps-llvm-local
ARG ROOTFS_DIR=/crossrootfs/x64

COPY --from=builder $ROOTFS_DIR $ROOTFS_DIR

RUN LLVM_VERSION=12.0.1 LLVM_VERSION_MAJOR="${LLVM_VERSION%%.*}" && \
mkdir -p $ROOTFS_DIR/usr/lib/llvm-${LLVM_VERSION_MAJOR}/lib/clang/${LLVM_VERSION}/lib/linux/ && \
cp /usr/lib/clang/${LLVM_VERSION}/lib/linux/libclang_rt.profile-x86_64.a $ROOTFS_DIR/usr/lib/llvm-${LLVM_VERSION_MAJOR}/lib/clang/${LLVM_VERSION}/lib/linux/
19 changes: 8 additions & 11 deletions src/cbl-mariner/2.0/cross/arm/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -7,34 +7,31 @@ RUN /scripts/eng/common/cross/build-rootfs.sh arm xenial --skipunmount

# Build compiler-rt profile library for PGO instrumentation
RUN mkdir compiler-rt_build && cd compiler-rt_build && \
BUILD_FLAGS="-v --sysroot=$ROOTFS_DIR" \
TARGET_TRIPLE=arm-linux-gnueabihf && \
cmake ../llvm-project.src/compiler-rt \
-DCOMPILER_RT_BUILD_PROFILE=ON \
-DCOMPILER_RT_BUILD_BUILTINS=OFF \
-DCOMPILER_RT_BUILD_CRT=OFF \
-DCOMPILER_RT_BUILD_MEMPROF=OFF \
-DCOMPILER_RT_BUILD_GWP_ASAN=OFF \
-DCOMPILER_RT_BUILD_SANITIZERS=OFF \
-DCOMPILER_RT_BUILD_XRAY=OFF \
-DCOMPILER_RT_BUILD_LIBFUZZER=OFF \
-DCOMPILER_RT_BUILD_ORC=OFF \
\
-DCMAKE_C_COMPILER=clang \
-DCMAKE_CXX_COMPILER=clang++ \
-DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld" \
-DCMAKE_C_COMPILER_TARGET=${TARGET_TRIPLE} \
-DCMAKE_CXX_COMPILER_TARGET=${TARGET_TRIPLE} \
-DLLVM_CONFIG_PATH=llvm-config \
-DCMAKE_C_COMPILER_TARGET="arm-linux-gnueabihf" \
-DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX=$ROOTFS_DIR/usr \
-DCMAKE_C_FLAGS="${BUILD_FLAGS}" \
-DCMAKE_CXX_FLAGS="${BUILD_FLAGS}" && \
-DCMAKE_C_FLAGS="-v --sysroot=$ROOTFS_DIR" && \
make -j $(getconf _NPROCESSORS_ONLN)

RUN LLVM_VERSION=12.0.1 LLVM_VERSION_MAJOR="${LLVM_VERSION%%.*}" && \
RUN LLVM_VERSION=16.0.0 LLVM_VERSION_MAJOR="${LLVM_VERSION%%.*}" && \
mkdir -p $ROOTFS_DIR/usr/lib/llvm-${LLVM_VERSION_MAJOR}/lib/clang/${LLVM_VERSION}/lib/linux/ && \
cp compiler-rt_build/lib/linux/libclang_rt.profile-armhf.a $ROOTFS_DIR/usr/lib/llvm-${LLVM_VERSION_MAJOR}/lib/clang/${LLVM_VERSION}/lib/linux/


FROM mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-crossdeps-arm-local
FROM mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-crossdeps-llvm-local
ARG ROOTFS_DIR=/crossrootfs/arm

COPY --from=builder $ROOTFS_DIR $ROOTFS_DIR
2 changes: 1 addition & 1 deletion src/cbl-mariner/2.0/cross/arm64-alpine/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ ARG ROOTFS_DIR=/crossrootfs/arm64
RUN /scripts/eng/common/cross/build-rootfs.sh arm64 alpine3.13 --skipunmount


FROM mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-crossdeps-arm-local
FROM mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-crossdeps-llvm-local
ARG ROOTFS_DIR=/crossrootfs/arm64

COPY --from=builder $ROOTFS_DIR $ROOTFS_DIR
19 changes: 8 additions & 11 deletions src/cbl-mariner/2.0/cross/arm64/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -5,34 +5,31 @@ RUN /scripts/eng/common/cross/build-rootfs.sh arm64 xenial --skipunmount

# Build compiler-rt profile library for PGO instrumentation
RUN mkdir compiler-rt_build && cd compiler-rt_build && \
BUILD_FLAGS="-v --sysroot=$ROOTFS_DIR" \
TARGET_TRIPLE=aarch64-linux-gnu && \
cmake ../llvm-project.src/compiler-rt \
-DCOMPILER_RT_BUILD_PROFILE=ON \
-DCOMPILER_RT_BUILD_BUILTINS=OFF \
-DCOMPILER_RT_BUILD_CRT=OFF \
-DCOMPILER_RT_BUILD_MEMPROF=OFF \
-DCOMPILER_RT_BUILD_GWP_ASAN=OFF \
-DCOMPILER_RT_BUILD_SANITIZERS=OFF \
-DCOMPILER_RT_BUILD_XRAY=OFF \
-DCOMPILER_RT_BUILD_LIBFUZZER=OFF \
-DCOMPILER_RT_BUILD_ORC=OFF \
\
-DCMAKE_C_COMPILER=clang \
-DCMAKE_CXX_COMPILER=clang++ \
-DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld" \
-DCMAKE_C_COMPILER_TARGET=${TARGET_TRIPLE} \
-DCMAKE_CXX_COMPILER_TARGET=${TARGET_TRIPLE} \
-DLLVM_CONFIG_PATH=llvm-config \
-DCMAKE_C_COMPILER_TARGET="aarch64-linux-gnu" \
-DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX=$ROOTFS_DIR/usr \
-DCMAKE_C_FLAGS="${BUILD_FLAGS}" \
-DCMAKE_CXX_FLAGS="${BUILD_FLAGS}" && \
-DCMAKE_C_FLAGS="-v --sysroot=$ROOTFS_DIR" && \
make -j $(getconf _NPROCESSORS_ONLN)

RUN LLVM_VERSION=12.0.1 LLVM_VERSION_MAJOR="${LLVM_VERSION%%.*}" && \
RUN LLVM_VERSION=16.0.0 LLVM_VERSION_MAJOR="${LLVM_VERSION%%.*}" && \
mkdir -p $ROOTFS_DIR/usr/lib/llvm-${LLVM_VERSION_MAJOR}/lib/clang/${LLVM_VERSION}/lib/linux/ && \
cp compiler-rt_build/lib/linux/libclang_rt.profile-aarch64.a $ROOTFS_DIR/usr/lib/llvm-${LLVM_VERSION_MAJOR}/lib/clang/${LLVM_VERSION}/lib/linux/


FROM mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-crossdeps-arm-local
FROM mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-crossdeps-llvm-local
ARG ROOTFS_DIR=/crossrootfs/arm64

COPY --from=builder $ROOTFS_DIR $ROOTFS_DIR
24 changes: 8 additions & 16 deletions src/cbl-mariner/2.0/cross/x86/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,43 +1,35 @@
FROM mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-crossdeps-builder-local AS builder
ARG ROOTFS_DIR=/crossrootfs/x86

# Provides LLVMConfig.cmake, needed to build compiler-rt. Only needed on x86.
# x64 doesn't need this because we use compiler-rt from the mariner package.
# arm/arm64 don't need this because LLVMConfig.cmake is provided by the LLVM we build from source.
RUN tdnf install -y llvm-devel

RUN /scripts/eng/common/cross/build-rootfs.sh x86 xenial --skipunmount

# Build compiler-rt profile library for PGO instrumentation
RUN mkdir compiler-rt_build && cd compiler-rt_build && \
BUILD_FLAGS="-v --sysroot=$ROOTFS_DIR" \
TARGET_TRIPLE=i386-linux-gnu && \
cmake ../llvm-project.src/compiler-rt \
-DCOMPILER_RT_BUILD_PROFILE=ON \
-DCOMPILER_RT_BUILD_BUILTINS=OFF \
-DCOMPILER_RT_BUILD_CRT=OFF \
-DCOMPILER_RT_BUILD_MEMPROF=OFF \
-DCOMPILER_RT_BUILD_GWP_ASAN=OFF \
-DCOMPILER_RT_BUILD_SANITIZERS=OFF \
-DCOMPILER_RT_BUILD_XRAY=OFF \
-DCOMPILER_RT_BUILD_LIBFUZZER=OFF \
-DCOMPILER_RT_BUILD_ORC=OFF \
\
-DCMAKE_C_COMPILER=clang \
-DCMAKE_CXX_COMPILER=clang++ \
-DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld" \
-DCMAKE_C_COMPILER_TARGET=${TARGET_TRIPLE} \
-DCMAKE_CXX_COMPILER_TARGET=${TARGET_TRIPLE} \
-DLLVM_CONFIG_PATH=llvm-config \
-DCMAKE_C_COMPILER_TARGET="i386-linux-gnu" \
-DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX=$ROOTFS_DIR/usr \
-DCMAKE_C_FLAGS="${BUILD_FLAGS}" \
-DCMAKE_CXX_FLAGS="${BUILD_FLAGS}" && \
-DCMAKE_C_FLAGS="-v --sysroot=$ROOTFS_DIR" && \
make -j $(getconf _NPROCESSORS_ONLN)

RUN LLVM_VERSION=12.0.1 LLVM_VERSION_MAJOR="${LLVM_VERSION%%.*}" && \
RUN LLVM_VERSION=16.0.0 LLVM_VERSION_MAJOR="${LLVM_VERSION%%.*}" && \
mkdir -p $ROOTFS_DIR/usr/lib/llvm-${LLVM_VERSION_MAJOR}/lib/clang/${LLVM_VERSION}/lib/linux/ && \
cp compiler-rt_build/lib/linux/libclang_rt.profile-i386.a $ROOTFS_DIR/usr/lib/llvm-${LLVM_VERSION_MAJOR}/lib/clang/${LLVM_VERSION}/lib/linux/


FROM mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-crossdeps-amd64-local
FROM mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-crossdeps-llvm-local
ARG ROOTFS_DIR=/crossrootfs/x86

COPY --from=builder $ROOTFS_DIR $ROOTFS_DIR
6 changes: 0 additions & 6 deletions src/cbl-mariner/2.0/crossdeps-amd64/Dockerfile

This file was deleted.

6 changes: 3 additions & 3 deletions src/cbl-mariner/2.0/crossdeps-builder/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -38,11 +38,11 @@ RUN wget https://releases.llvm.org/release-keys.asc && \
gpg --output release-keys.gpg --dearmor release-keys.asc && \
rm release-keys.asc && \
# 2. Download llvm sources and signature, and verify signature
LLVM_VERSION=12.0.1 && \
LLVM_VERSION=16.0.0 && \
wget -O llvm-project.src.tar.xz.sig https://github.com/llvm/llvm-project/releases/download/llvmorg-${LLVM_VERSION}/llvm-project-${LLVM_VERSION}.src.tar.xz.sig && \
echo "6cc956d622a7d3d746de0d71d8ca616a6c291e2c561703ac7a9535f38b999955 llvm-project.src.tar.xz.sig" | sha256sum -c && \
echo "2fa1ae520c6756fb53ede6bd9359f7e40fe5d0bfdbd04c359f3b8ce22958a896 llvm-project.src.tar.xz.sig" | sha256sum -c && \
wget -O llvm-project.src.tar.xz https://github.com/llvm/llvm-project/releases/download/llvmorg-${LLVM_VERSION}/llvm-project-${LLVM_VERSION}.src.tar.xz && \
echo "129cb25cd13677aad951ce5c2deb0fe4afc1e9d98950f53b51bdcfb5a73afa0e llvm-project.src.tar.xz" | sha256sum -c && \
echo "9a56d906a2c81f16f06efc493a646d497c53c2f4f28f0cb1f3c8da7f74350254 llvm-project.src.tar.xz" | sha256sum -c && \
gpg --keyring /release-keys.gpg --verify llvm-project.src.tar.xz.sig && \
rm llvm-project.src.tar.xz.sig

Expand Down
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

crossdeps-builder has extra contents (mostly sources and intermediate build outputs) that shouldn't be included in the final image, so we use it to build llvm, then only copy what we need out of it (llvm binaries, compiler-rt profile shared lib). crossdeps-llvm is a shared base image for the other cross-ARCH images, since it has the llvm cross-compiler. It is only one layer on top of crossdeps. crossdeps is used as the base image for both crossdeps-llvm and crossdeps-builder.

File renamed without changes.
24 changes: 4 additions & 20 deletions src/cbl-mariner/manifest.json
Original file line number Diff line number Diff line change
Expand Up @@ -77,29 +77,13 @@
{
"platforms": [
{
"dockerfile": "src/cbl-mariner/2.0/crossdeps-amd64",
"dockerfile": "src/cbl-mariner/2.0/crossdeps-llvm",
"os": "linux",
"osVersion": "cbl-mariner2.0",
"tags": {
"cbl-mariner-2.0-crossdeps-amd64-$(System:TimeStamp)-$(System:DockerfileGitCommitSha)": {},
"cbl-mariner-2.0-crossdeps-amd64$(FloatingTagSuffix)": {},
"cbl-mariner-2.0-crossdeps-amd64-local": {
"isLocal": true
}
}
}
]
},
{
"platforms": [
{
"dockerfile": "src/cbl-mariner/2.0/crossdeps-arm",
"os": "linux",
"osVersion": "cbl-mariner2.0",
"tags": {
"cbl-mariner-2.0-crossdeps-arm-$(System:TimeStamp)-$(System:DockerfileGitCommitSha)": {},
"cbl-mariner-2.0-crossdeps-arm$(FloatingTagSuffix)": {},
"cbl-mariner-2.0-crossdeps-arm-local": {
"cbl-mariner-2.0-crossdeps-llvm-$(System:TimeStamp)-$(System:DockerfileGitCommitSha)": {},
"cbl-mariner-2.0-crossdeps-llvm$(FloatingTagSuffix)": {},
"cbl-mariner-2.0-crossdeps-llvm-local": {
"isLocal": true
}
}
Expand Down