feat: devbox (#5772)

This introduces a base build image in `./build-images`. The idea is that this image contains a complete environment to bootstrap aztec-packages. It's available for both x86 and arm, and thus can be used to have a stable and consistent development environment on all operating systems that have docker installed. The Dockerfile is a multi-stage image with 3 interesting targets: * `build`: The base build image - this can be used by CI to build any part of the project. It contains just the tools needed to successfully bootstrap. * `devbox`: Extends `build` and includes all sorts of useful developer tools. This container can be run independently via a `./build-images/run.sh` script. Vscode can launch it via it's "Dev Containers" plugin, or github/vscode can launch it within codespaces, the configuration is at `.devcontainer/dev/devcontainer.json`. * `sysbox`: Is similar to `devbox` but is designed to be run using nestybox's sysbox runtime, like internal aztec developers use on mainframe. An important thing to note, is that this approach removes all toolchain installations from bootstrap scripts, as it's expected for the right things to be available in your PATH (or in the case of wasi-sdk, at `/opt/wasi-sdk`). The `bootstrap.sh` script now checks all toolchains are available and the right version at start. You can also run this command: ``` % ./bootstrap.sh check Toolchains look good! 🎉 ``` If there's an issue, a hint is provided on how to resolve, but the engineer is encouraged to just use the dev container. What's currently checked for: * Utils: `jq`, `parallel`, `awk`, `git`, `curl`. * `cmake` min 3.24. * `clang-16`. * Rust 1.74. * Wasi SDK 22 at `/opt/wasi-sdk`. * `forge` and `anvil` at commit `de33b6a`. * `node` min 18.19. * `yarn` and `solhint`. Aztec internal engineers who use sysbox on mainframe, should run a `sudo halt` at their convenience, wait for the box to restart, and they should have a setup that passes the above check. If somethings wrong, you may have some custom configuration in your home dot files that needs removing. Otherwise contact me. **Your hostkey to your sysbox may have changed! Get your IP from `~/.ssh/config` and do an e.g.** ``` ssh-keygen -R 173.26.1.1 ``` Other notable changes: * wasi-sdk is now version 22.0. This means we have a single sdk for both threaded and unthreaded wasms (they still need to be built independently however. TODO: Investigate if we can build once and link twice). It's expected to be installed at `/opt/wasi-sdk`. * Ensured we're using version 16 of both `clangd` and `clang-format`. We should always be explicit about the version we're using. We should probably upgrade to clang/d/format 18 soon as that's what's distributed in ubuntu noble, and is also what the wasi-sdk uses. * Update Earthly builds to use noble. * Some cpp files formatting change with the shift from 15 to 16. * Removed the `parallel_for` `moody` implementation. Had to update various bits of threading code to be properly ignored when building non threaded wasm. Maybe we can just always build the threaded wasm, and just link it twice with shared memory enable/disabled? Not sure. * When running the dev container on a mac, with aztec-packages mounted from the host filesystem, there is a known bug around incremental rust builds, and fractional file modification timestamps. A script runs after building noir-repo to normalise these timestamps, if it's detected that the host is a mac. * Purge a load of unneeded stuff when building aztec prod image, to reduce container size. We do similar for end-to-end tests because they need to be serialized from Earthly to Docker and that's currently slow. * Simplified some end-to-end Earthly config. * Removed all test specific timeouts in favour of just launching jest with a 5m test timeout.
AztecProtocol · May 1, 2024 · 72321f9 · 72321f9
1 parent cfe1b05
commit 72321f9
Show file tree

Hide file tree

Showing 123 changed files with 1,241 additions and 5,691 deletions.
diff --git a/.devcontainer/dev/devcontainer.json b/.devcontainer/dev/devcontainer.json
@@ -1,17 +1,22 @@
 {
-  "image": "node:lts-bookworm-slim",
-  "features": {
-    "ghcr.io/devcontainers/features/docker-in-docker:2": {}
+  "name": "Development",
+  "build": {
+    "dockerfile": "../../build-images/Dockerfile",
+    "context": "../../build-images",
+    "target": "devbox"
   },
-  "postCreateCommand": "curl -s install.aztec.network | VERSION=master NON_INTERACTIVE=1 BIN_PATH=/usr/local/bin bash -s",
-  "customizations": {
-    "vscode": {
-      "settings": {},
-      "extensions": [
-        "noir-lang.vscode-noir"
-      ]
-    }
-  },
-  "workspaceMount": "source=${localWorkspaceFolder},target=/root/workspace,type=bind",
-  "workspaceFolder": "/root/workspace"
+  "containerUser": "aztec-dev",
+  // ubuntu:noble is currently not supported.
+  // Can possibly workaround cherry-picking from here:
+  //   https://github.com/devcontainers/features/blob/main/src/docker-in-docker/install.sh
+  //
+  // "image": "aztecprotocol/codespace",
+  // "features": {
+  //   "docker-in-docker": {
+  //     "version": "latest",
+  //     "moby": true,
+  //     "dockerDashComposeVersion": "v1"
+  //   }
+  // },
+  "mounts": ["source=devbox-home,target=/home/aztec-dev,type=volume"]
 }
diff --git a/.github/workflows/ci-arm.yml b/.github/workflows/ci-arm.yml
@@ -32,7 +32,10 @@ jobs:
     needs: setup
     runs-on: master-arm
     steps:
-      - {uses: actions/checkout@v4, with: { ref: "${{ github.event.pull_request.head.sha }}"}}
+      - {
+          uses: actions/checkout@v4,
+          with: { ref: "${{ github.event.pull_request.head.sha }}" },
+        }
       - uses: ./.github/ci-setup-action
         with:
           dockerhub_password: "${{ secrets.DOCKERHUB_PASSWORD }}"
@@ -48,15 +51,18 @@ jobs:
     needs: build
     runs-on: master-arm
     steps:
-      - {uses: actions/checkout@v4, with: { ref: "${{ github.event.pull_request.head.sha }}"}}
+      - {
+          uses: actions/checkout@v4,
+          with: { ref: "${{ github.event.pull_request.head.sha }}" },
+        }
       - uses: ./.github/ci-setup-action
         with:
           dockerhub_password: "${{ secrets.DOCKERHUB_PASSWORD }}"
           concurrency_key: e2e-master-arm-e2e-tests
       - name: Test
         working-directory: ./yarn-project/end-to-end/
         timeout-minutes: 15
-        run: earthly -P --no-output +uniswap-trade-on-l1-from-l2 --e2e_mode=cache
+        run: earthly -P --no-output +uniswap-trade-on-l1-from-l2
 
   notify:
     needs: [e2e]

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -76,7 +76,7 @@ jobs:
       - name: Test
         working-directory: ./yarn-project/end-to-end/
         timeout-minutes: 25
-        run: earthly-ci -P --secret AWS_ACCESS_KEY_ID=${{ secrets.AWS_ACCESS_KEY_ID }} --secret AWS_SECRET_ACCESS_KEY=${{ secrets.AWS_SECRET_ACCESS_KEY }} --no-output +${{ matrix.test }} --e2e_mode=cache
+        run: earthly-ci -P --secret AWS_ACCESS_KEY_ID=${{ secrets.AWS_ACCESS_KEY_ID }} --secret AWS_SECRET_ACCESS_KEY=${{ secrets.AWS_SECRET_ACCESS_KEY }} --no-output +${{ matrix.test }}
 
   bench-summary:
     needs: e2e

diff --git a/.vscode/extensions.json b/.vscode/extensions.json
@@ -37,7 +37,7 @@
     // Displays code coverage report information within vscode
     "ryanluker.vscode-coverage-gutters",
     // Spell checking
-    "streetsidesoftware.code-spell-checker",
+    "streetsidesoftware.code-spell-checker"
     // End C++/Circuits extensions
     ///////////////////////////////////////
   ],
@@ -58,7 +58,7 @@
     // Most features are disabled in `settings.json`
     // which confict with `clangd`
     // Since we ignore GDB, we no longer need this extension
-    "ms-vscode.cpptools",
+    "ms-vscode.cpptools"
     // End C++/Circuits unwanted extensions
     ///////////////////////////////////////
   ]

diff --git a/.vscode/settings.json b/.vscode/settings.json
@@ -136,7 +136,7 @@
   // Clangd. Note that this setting may be overridden by user settings
   // to the default value "clangd".
   //
-  "clangd.path": "clangd-15",
+  "clangd.path": "clangd-16",
   //
   // C/C++ (should be disabled)
   //
@@ -165,6 +165,5 @@
     "**/target/**": true,
     "**/l1-contracts/lib/**": true,
     "**/barretenberg/cpp/build*/**": true
-  },
-  "cmake.sourceDirectory": "/mnt/user-data/adam/aztec-packages/barretenberg/cpp"
+  }
 }
diff --git a/Earthfile b/Earthfile
@@ -1,5 +1,5 @@
 VERSION 0.8
-FROM ubuntu:lunar
+FROM ubuntu:noble
 
 build-ci:
     BUILD ./avm-transpiler/+build
@@ -16,15 +16,12 @@ build-ci:
     BUILD ./yarn-project/+end-to-end
     BUILD ./yarn-project/+aztec
 
-build-ci-small:
-    BUILD ./yarn-project/end-to-end/+e2e-escrow-contract
-
 build:
     # yarn-project has the entry point to Aztec
     BUILD ./yarn-project/+build
 
 test-end-to-end:
-    BUILD ./yarn-project/end-to-end/+test-all
+    BUILD ./yarn-project/end-to-end+e2e-tests
 
 bench:
   RUN echo hi

diff --git a/avm-transpiler/Earthfile b/avm-transpiler/Earthfile
@@ -1,18 +1,17 @@
 VERSION 0.8
 IMPORT ../noir AS noir
-# we rely on noir source, which this image has
-FROM noir+nargo
 
-# move noir contents to /usr/src/noir
-RUN mv /usr/src /noir && mkdir /usr/src && mv /noir /usr/src
-# work in avm-transpiler
-WORKDIR /usr/src/avm-transpiler
+source:
+  # we rely on noir source, which this image has
+  FROM noir+nargo
 
-RUN apt-get update && apt-get install -y git
+  # move noir contents to /usr/src/noir
+  RUN mv /usr/src /noir && mkdir /usr/src && mv /noir /usr/src
+  # work in avm-transpiler
+  WORKDIR /usr/src/avm-transpiler
 
-COPY --dir scripts src Cargo.lock Cargo.toml rust-toolchain.toml .
+  COPY --dir scripts src Cargo.lock Cargo.toml rust-toolchain.toml .
 
-source:
   # for debugging rebuilds
   RUN echo CONTENT HASH $(find . -type f -exec sha256sum {} ';' | sort | sha256sum | awk '{print $1}') | tee .content-hash
 
@@ -21,7 +20,7 @@ build:
     RUN ./scripts/bootstrap_native.sh
     SAVE ARTIFACT target/release/avm-transpiler avm-transpiler
 
-run: 
+run:
     #TODO needed?
     FROM ubuntu:focal
     COPY +build/avm-transpiler /usr/src/avm-transpiler

diff --git a/barretenberg/cpp/CMakePresets.json b/barretenberg/cpp/CMakePresets.json
@@ -256,7 +256,7 @@
       "generator": "Ninja",
       "toolchainFile": "cmake/toolchains/wasm32-wasi.cmake",
       "environment": {
-        "WASI_SDK_PREFIX": "${sourceDir}/src/wasi-sdk",
+        "WASI_SDK_PREFIX": "/opt/wasi-sdk",
         "CC": "$env{WASI_SDK_PREFIX}/bin/clang",
         "CXX": "$env{WASI_SDK_PREFIX}/bin/clang++",
         "AR": "$env{WASI_SDK_PREFIX}/bin/llvm-ar",

diff --git a/barretenberg/cpp/Earthfile b/barretenberg/cpp/Earthfile
@@ -1,80 +1,13 @@
 VERSION 0.8
 
-build-base:
-    ARG TARGETARCH
-    FROM --platform=linux/$TARGETARCH ubuntu:lunar
-    RUN apt-get update && apt-get install -y \
-        build-essential \
-        curl \
-        git \
-        cmake \
-        lsb-release \
-        wget \
-        software-properties-common \
-        gnupg \
-        ninja-build \
-        npm \
-        libssl-dev \
-        jq \
-        bash \
-        libstdc++6 \
-        clang-format \
-        clang-16
-
-    IF [ $TARGETARCH = arm64 ]
-        # publish arm after, assumes x86 already exists, becomes multiplatform image
-        SAVE IMAGE --push aztecprotocol/bb-ubuntu-lunar
-        FROM --platform=linux/amd64 aztecprotocol/bb-ubuntu-lunar:x86-latest
-        SAVE IMAGE --push aztecprotocol/bb-ubuntu-lunar
-    ELSE
-        SAVE IMAGE --push aztecprotocol/bb-ubuntu-lunar:x86-latest
-    END
-
-build-wasi-sdk-image:
-    WORKDIR /
-    RUN git clone --recursive https://github.com/WebAssembly/wasi-sdk.git
-    WORKDIR /wasi-sdk
-    RUN git checkout 9389ea5eeec98afc61039683ae92c6147fee9c54
-    ENV NINJA_FLAGS=-v
-    ENV MAKEFLAGS="-j$(nproc)"
-    RUN make build/llvm.BUILT
-    RUN make build/wasi-libc.BUILT
-    RUN make build/compiler-rt.BUILT
-    RUN make build/libcxx.BUILT
-    RUN make build/config.BUILT
-    SAVE ARTIFACT build/install/opt/wasi-sdk
-
-build-wasi-sdk:
-    ARG TARGETARCH
-    # Wrapper just exists share files.
-    FROM scratch
-    WORKDIR /usr/src
-    COPY +get-wasi-sdk-image/wasi-sdk wasi-sdk
-    SAVE ARTIFACT wasi-sdk
-    SAVE IMAGE --push aztecprotocol/cache:wasi-sdk-threads-v21.0-$TARGETARCH
-
-get-wasi-sdk-threads:
-    ARG TARGETARCH
-    # If this is failing, we need to run earthly --push +build-wasi-sdk
-    FROM aztecprotocol/cache:wasi-sdk-threads-v21.0-$TARGETARCH
-    SAVE ARTIFACT wasi-sdk
-
-get-wasi-sdk:
-    # NOTE: currently only works with x86
-    # TODO Align with above
-    FROM +source
-    COPY ./scripts/install-wasi-sdk.sh ./scripts/
-    RUN ./scripts/install-wasi-sdk.sh
-    # TODO better output name to mirror wasi-sdk
-    SAVE ARTIFACT src/wasi-sdk-20.0 wasi-sdk
-
 wasmtime:
-    FROM aztecprotocol/bb-ubuntu-lunar
+    FROM ubuntu:noble
+    RUN apt update && apt install -y curl xz-utils
     RUN curl https://wasmtime.dev/install.sh -sSf | bash
     SAVE ARTIFACT /root/.wasmtime/bin/wasmtime
 
 source:
-    FROM aztecprotocol/bb-ubuntu-lunar
+    FROM ../../build-images+build
     WORKDIR /usr/src/barretenberg
     # cpp source
     COPY --dir src/barretenberg src/CMakeLists.txt src
@@ -106,19 +39,17 @@ preset-wasm:
         FROM +preset-wasm-threads
         SAVE ARTIFACT build/bin
     ELSE
-        COPY +get-wasi-sdk/wasi-sdk src/wasi-sdk
         RUN cmake --preset wasm -Bbuild && cmake --build build --target barretenberg.wasm
-        RUN src/wasi-sdk/bin/llvm-strip ./build/bin/barretenberg.wasm
+        RUN /opt/wasi-sdk/bin/llvm-strip ./build/bin/barretenberg.wasm
         SAVE ARTIFACT build/bin
         SAVE IMAGE --cache-hint
     END
 
 preset-wasm-threads:
     FROM +source
-    COPY +get-wasi-sdk-threads/wasi-sdk src/wasi-sdk
     RUN cmake --preset wasm-threads -Bbuild && cmake --build build --target barretenberg.wasm
     # TODO(https://github.com/AztecProtocol/barretenberg/issues/941) We currently do not strip barretenberg threaded wasm, for stack traces.
-    # RUN src/wasi-sdk/bin/llvm-strip ./build/bin/barretenberg.wasm
+    # RUN /opt/wasi-sdk/bin/llvm-strip ./build/bin/barretenberg.wasm
     SAVE ARTIFACT build/bin
 
 preset-gcc:
@@ -189,7 +120,7 @@ bench-binaries:
 # Runs on the bench image, sent from the builder runner
 bench-ultra-honk:
     BUILD +wasmtime # prefetch
-    FROM +source
+    FROM ubuntu:noble
     COPY --dir +bench-binaries/* .
     # install SRS needed for proving
     COPY --dir ./srs_db/+build/. srs_db
@@ -201,7 +132,7 @@ bench-ultra-honk:
 
 bench-client-ivc:
     BUILD +wasmtime # prefetch
-    FROM +source
+    FROM ubuntu:noble
     COPY --dir +bench-binaries/* .
     # install SRS needed for proving
     COPY --dir ./srs_db/+build/. srs_db

diff --git a/barretenberg/cpp/bootstrap.sh b/barretenberg/cpp/bootstrap.sh
@@ -31,9 +31,6 @@ fi
 # Download ignition transcripts.
 (cd ./srs_db && ./download_ignition.sh 0)
 
-# Install wasi-sdk.
-./scripts/install-wasi-sdk.sh
-
 # Attempt to just pull artefacts from CI and exit on success.
 [ -n "${USE_CACHE:-}" ] && ./bootstrap_cache.sh && exit
 
@@ -82,19 +79,19 @@ AVAILABLE_MEMORY=0
 case "$(uname)" in
   Linux*)
     # Check available memory on Linux
-    AVAILABLE_MEMORY=$(awk '/MemFree/ { printf $2 }' /proc/meminfo)
+    AVAILABLE_MEMORY=$(awk '/MemTotal/ { printf $2 }' /proc/meminfo)
     ;;
   *)
     echo "Parallel builds not supported on this operating system"
     ;;
 esac
 # This value may be too low.
 # If builds fail with an amount of free memory greater than this value then it should be increased.
-MIN_PARALLEL_BUILD_MEMORY=32000000
+MIN_PARALLEL_BUILD_MEMORY=32854492
 
 if [[ AVAILABLE_MEMORY -lt MIN_PARALLEL_BUILD_MEMORY ]]; then
   echo "System does not have enough memory for parallel builds, falling back to sequential"
-  build_native 
+  build_native
   build_wasm
   build_wasm_threads
 else

diff --git a/barretenberg/cpp/cmake/threading.cmake b/barretenberg/cpp/cmake/threading.cmake
@@ -4,7 +4,7 @@ if(MULTITHREADING)
     add_link_options(-pthread)
     if(WASM)
         add_compile_options(--target=wasm32-wasi-threads)
-        add_link_options(--target=wasm32-wasi-threads)
+        add_link_options(--target=wasm32-wasi-threads -Wl,--shared-memory)
     endif()
     #add_compile_options(-fsanitize=thread)
     #add_link_options(-fsanitize=thread)

diff --git a/barretenberg/cpp/dockerfiles/Dockerfile.x86_64-linux-clang b/barretenberg/cpp/dockerfiles/Dockerfile.x86_64-linux-clang
@@ -20,7 +20,7 @@ RUN wget https://apt.llvm.org/llvm.sh && chmod +x llvm.sh && ./llvm.sh 16
 
 WORKDIR /usr/src/barretenberg/cpp
 COPY . .
-# Build bb binary and targets needed for benchmarking. 
+# Build bb binary and targets needed for benchmarking.
 # Everything else is built as part linux-clang-assert.
 # Benchmark targets want to run without asserts, so get built alongside bb.
 RUN cmake --preset clang16

diff --git a/barretenberg/cpp/dockerfiles/Dockerfile.x86_64-linux-clang-assert b/barretenberg/cpp/dockerfiles/Dockerfile.x86_64-linux-clang-assert
@@ -14,10 +14,9 @@ RUN apt update && apt install -y \
   libssl-dev \
   jq \
   bash \
-  libstdc++6 \
-  clang-format
+  libstdc++6
 
-RUN wget https://apt.llvm.org/llvm.sh && chmod +x llvm.sh && ./llvm.sh 16
+RUN wget https://apt.llvm.org/llvm.sh && chmod +x llvm.sh && ./llvm.sh 16 && apt install -y clang-format-16
 
 WORKDIR /usr/src/barretenberg/cpp
 COPY . .

diff --git a/barretenberg/cpp/format.sh b/barretenberg/cpp/format.sh
@@ -4,22 +4,22 @@ set -e
 if [ "$1" == "staged" ]; then
   echo Formatting barretenberg staged files...
   for FILE in $(git diff-index --diff-filter=d --relative --cached --name-only HEAD | grep -e '\.\(cpp\|hpp\|tcc\)$'); do
-    clang-format -i $FILE
+    clang-format-16 -i $FILE
     sed -i.bak 's/\r$//' $FILE && rm ${FILE}.bak
     git add $FILE
   done
 elif [ "$1" == "check" ]; then
   for FILE in $(find ./src -iname *.hpp -o -iname *.cpp -o -iname *.tcc | grep -v src/msgpack-c); do
-    clang-format --dry-run --Werror $FILE
+    clang-format-16 --dry-run --Werror $FILE
   done
 elif [ -n "$1" ]; then
   for FILE in $(git diff-index --relative --name-only $1 | grep -e '\.\(cpp\|hpp\|tcc\)$'); do
-    clang-format -i $FILE
+    clang-format-16 -i $FILE
     sed -i.bak 's/\r$//' $FILE && rm ${FILE}.bak
   done
 else
   for FILE in $(find ./src -iname *.hpp -o -iname *.cpp -o -iname *.tcc | grep -v src/msgpack-c); do
-    clang-format -i $FILE
+    clang-format-16 -i $FILE
     sed -i.bak 's/\r$//' $FILE && rm ${FILE}.bak
   done
 fi