-
Notifications
You must be signed in to change notification settings - Fork 4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore: build noctilucent WASM library in a container #26123
Changes from 6 commits
647bc17
eecf5e8
58769b2
ae51139
3609fb2
ad45f49
8ea0070
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -39,6 +39,3 @@ test/integ/cli/*.d.ts | |
.DS_Store | ||
|
||
junit.xml | ||
|
||
# Exclude the noctilucent WASM package | ||
lib/vendor/noctilucent/ |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -15,30 +15,20 @@ cat > build-info.json <<HERE | |
} | ||
HERE | ||
|
||
# Build noctilucent package | ||
( | ||
# Check out the submodule if it's not there already | ||
if [ ! -f "vendor/noctilucent/Cargo.toml" ]; then | ||
git -C ./vendor clone https://github.com/iph/noctilucent.git | ||
fi | ||
|
||
# update the package to the pinned commit hash | ||
git -C vendor/noctilucent reset --hard HEAD | ||
git -C vendor/noctilucent fetch && git -C vendor/noctilucent checkout 6da7c9fade55f8443bba7b8fdfcd4ebfe5208fb1 | ||
|
||
# Install wasm-pack if it's not there already | ||
if ! command -v wasm-pack >/dev/null 2>/dev/null; then | ||
echo "installing wasm-pack, this may take a while..." | ||
cargo install wasm-pack | ||
fi | ||
|
||
pkgroot=$(cd $(dirname -- "$0") && pwd) | ||
|
||
cd vendor/noctilucent | ||
wasm-pack build --target nodejs \ | ||
--out-dir="${pkgroot}/lib/vendor/noctilucent" \ | ||
--out-name=index | ||
|
||
cd ../../lib/vendor/noctilucent | ||
rm package.json | ||
) | ||
# Build noctilucent package in a Docker/Finch VM | ||
NOCTILUCENT_GIT="https://github.com/iph/noctilucent.git" | ||
NOCTILUCENT_COMMIT_ID="6da7c9fade55f8443bba7b8fdfcd4ebfe5208fb1" | ||
if [ "$(cat lib/vendor/noctilucent/.version 2>/dev/null || echo '')" == "${NOCTILUCENT_GIT}:${NOCTILUCENT_COMMIT_ID}" ] | ||
then | ||
echo "⏭️ Noctilucent WASM binary is up-to date, skipping build..." | ||
echo "ℹ️ Delete lib/vendor/noctilucent/.version to force a rebuild." | ||
Comment on lines
+23
to
+24
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ❤️ clever There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. But: wouldn't a git submodule do the same trick, and/or releasing the Noctilucent WASM package to NPMJS by itself? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. we were originally using git submodules but it turned out codepipeline didn't like that very much so we had to just get it ourselves There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Submodules come with their own set of issues. If it's required for a build, we should not put it into a submodule. The |
||
else | ||
echo "⏳ Building Noctilucent WASM binary for embedding... This will take a while..." | ||
${CDK_DOCKER:-docker} build --rm \ | ||
--build-arg NOCTILUCENT_GIT="${NOCTILUCENT_GIT}" \ | ||
--build-arg NOCTILUCENT_COMMIT_ID="${NOCTILUCENT_COMMIT_ID}" \ | ||
--file lib/vendor/noctilucent/Dockerfile \ | ||
--target wasm \ | ||
--output type=local,dest=lib/vendor/noctilucent \ | ||
lib/vendor/noctilucent | ||
fi |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
# Ignore all files in this directory except the Dockerfile | ||
/* | ||
!/.gitignore | ||
!/Dockerfile |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,41 @@ | ||
FROM public.ecr.aws/debian/debian:buster-slim as build | ||
|
||
# Install basic pre-requisites | ||
RUN apt-get update \ | ||
&& apt-get install -y build-essential curl git libssl-dev openssl pkg-config zsh | ||
|
||
# Make sure we use the correct shell going forward | ||
SHELL ["/bin/zsh", "-c"] | ||
|
||
# Install Rustup | ||
ENV RUSTUP_HOME=/usr/local/rustup | ||
ENV CARGO_HOME=/usr/local/cargo | ||
RUN set -eo pipefail \ | ||
&& curl -fSsL "https://sh.rustup.rs" | sh -s -- -y --no-modify-path --profile=minimal \ | ||
&& echo "source ${CARGO_HOME}/env" >> /etc/profile.d/cargo.sh \ | ||
&& chmod -R a+rw ${CARGO_HOME} | ||
ENV PATH=$PATH:${CARGO_HOME}/bin | ||
|
||
# Install Node | ||
RUN curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \ | ||
&& apt-get install -y nodejs | ||
|
||
# Install wasm-pack | ||
RUN cargo install wasm-opt wasm-pack | ||
|
||
ARG NOCTILUCENT_GIT | ||
ARG NOCTILUCENT_COMMIT_ID | ||
|
||
# Check out noctilucent | ||
RUN git clone "${NOCTILUCENT_GIT}" "${TMPDIR}/noctilucent" \ | ||
&& git -C "${TMPDIR}/noctilucent" checkout -b wasm "${NOCTILUCENT_COMMIT_ID}" | ||
|
||
# Build noctilucent to WASM | ||
RUN cd "${TMPDIR}/noctilucent" \ | ||
&& wasm-pack build --target=nodejs --out-name=index --out-dir=/wasm-out \ | ||
&& rm --force /wasm-out/.gitignore /wasm-out/README.md /wasm-out/package.json \ | ||
&& echo "${NOCTILUCENT_GIT}:${NOCTILUCENT_COMMIT_ID}" > /wasm-out/.version | ||
|
||
#################################################################################################### | ||
FROM scratch as wasm | ||
COPY --from=build /wasm-out / |
This file was deleted.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree with @rix0rrr this feels like we are reinventing versioning and releases.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To clarify - this is just a stepping stone for now... We're trying to avoid wasting time on nom logistics to focus on the product itself. We (the folks working on
migrate
) are going to own the upkeep of this until it transitions to something our automation is able to deal with...