Skip to content

Commit

Permalink
chore: build noctilucent WASM library in a container (aws#26123)
Browse files Browse the repository at this point in the history
Due to the addition of noctilucent to cdk, contributors needed to download rust/rustup to be able to build the cdk.

This uses the pre-existing dependency on Docker/Finch to containerize the process in order to not incur any further dependencies for contributors to manage.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
  • Loading branch information
HBobertz authored and lukey-aleios committed Jun 30, 2023
1 parent 2e6cde8 commit bc409ba
Show file tree
Hide file tree
Showing 7 changed files with 68 additions and 49 deletions.
8 changes: 4 additions & 4 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -207,7 +207,7 @@ eval $(gp env -e)

### Amazon CodeCatalyst Dev Environments

Dev Environments are cloud-based development environments.
Dev Environments are cloud-based development environments.
[Amazon CodeCatalyst](https://aws.amazon.com/codecatalyst/) allows you to checkout your linked Github
repositories in your Dev Environments with your favorite local IDEs such as VSCode or JetBrains.

Expand All @@ -220,7 +220,7 @@ $ NODE_OPTIONS=--max-old-space-size=8192 npx lerna run build --scope=aws-cdk-lib

You may [configure your Dev Env](https://docs.aws.amazon.com/codecatalyst/latest/userguide/devenvironment-devfile.html) with the `devfile.yaml` to further customize your Dev Env for CDK development.

Read the links below for more details:
Read the links below for more details:
- [Dev Environments in CodeCatalyst](https://docs.aws.amazon.com/codecatalyst/latest/userguide/devenvironment.html)
- [Using GitHub repositories in CodeCatalyst](https://docs.aws.amazon.com/codecatalyst/latest/userguide/extensions-github.html)
- [Setting up to use the AWS CLI with CodeCatalyst](https://docs.aws.amazon.com/codecatalyst/latest/userguide/set-up-cli.html)
Expand Down Expand Up @@ -429,7 +429,7 @@ $ npx cdk -a test/aws-eks/test/sample.js deploy

This allows you to iterate your development and ensure a minimal sample app would successfully deploy as you expect.
You have the freedom to interact with it just as a common CDK app such as viewing differences with `npx cdk diff`
or pass context variables with `npx cdk deploy -c`. You can rapidly iterate your testing with repeated deployments
or pass context variables with `npx cdk deploy -c`. You can rapidly iterate your testing with repeated deployments
by importing existing resource such as existing VPC. This can save a lot of time and help you focus on the core changes.

```ts
Expand All @@ -439,7 +439,7 @@ const vpc = ec2.Vpc.fromLookup(stack, 'Vpc', { isDefault: true });
As this is for testing only, do not commit `sample.ts` and `sample.js` to your PR branch.

Alternatively, you can write this test as a new integration test like `integ.my-test.ts` and deploy it
using `yarn integ --no-clean`. This may be useful when you need to publish a new
using `yarn integ --no-clean`. This may be useful when you need to publish a new
integration test:

```console
Expand Down
3 changes: 0 additions & 3 deletions packages/aws-cdk/.gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,3 @@ test/integ/cli/*.d.ts
.DS_Store

junit.xml

# Exclude the noctilucent WASM package
lib/vendor/noctilucent/
4 changes: 2 additions & 2 deletions packages/aws-cdk/.npmignore
Original file line number Diff line number Diff line change
Expand Up @@ -29,5 +29,5 @@ tsconfig.json
**/cdk.out
junit.xml

# exclude noctilucent source
/vendor/noctilucent/
generate.sh
lib/vendor/noctilucent/Dockerfile
44 changes: 17 additions & 27 deletions packages/aws-cdk/generate.sh
Original file line number Diff line number Diff line change
Expand Up @@ -15,30 +15,20 @@ cat > build-info.json <<HERE
}
HERE

# Build noctilucent package
(
# Check out the submodule if it's not there already
if [ ! -f "vendor/noctilucent/Cargo.toml" ]; then
git -C ./vendor clone https://github.com/iph/noctilucent.git
fi

# update the package to the pinned commit hash
git -C vendor/noctilucent reset --hard HEAD
git -C vendor/noctilucent fetch && git -C vendor/noctilucent checkout 6da7c9fade55f8443bba7b8fdfcd4ebfe5208fb1

# Install wasm-pack if it's not there already
if ! command -v wasm-pack >/dev/null 2>/dev/null; then
echo "installing wasm-pack, this may take a while..."
cargo install wasm-pack
fi

pkgroot=$(cd $(dirname -- "$0") && pwd)

cd vendor/noctilucent
wasm-pack build --target nodejs \
--out-dir="${pkgroot}/lib/vendor/noctilucent" \
--out-name=index

cd ../../lib/vendor/noctilucent
rm package.json
)
# Build noctilucent package in a Docker/Finch VM
NOCTILUCENT_GIT="https://github.com/iph/noctilucent.git"
NOCTILUCENT_COMMIT_ID="6da7c9fade55f8443bba7b8fdfcd4ebfe5208fb1"
if [ "$(cat lib/vendor/noctilucent/.version 2>/dev/null || echo '')" == "${NOCTILUCENT_GIT}:${NOCTILUCENT_COMMIT_ID}" ]
then
echo "⏭️ Noctilucent WASM binary is up-to date, skipping build..."
echo "ℹ️ Delete lib/vendor/noctilucent/.version to force a rebuild."
else
echo "⏳ Building Noctilucent WASM binary for embedding... This will take a while..."
${CDK_DOCKER:-docker} build --rm \
--build-arg NOCTILUCENT_GIT="${NOCTILUCENT_GIT}" \
--build-arg NOCTILUCENT_COMMIT_ID="${NOCTILUCENT_COMMIT_ID}" \
--file lib/vendor/noctilucent/Dockerfile \
--target wasm \
--output type=local,dest=lib/vendor/noctilucent \
lib/vendor/noctilucent
fi
4 changes: 4 additions & 0 deletions packages/aws-cdk/lib/vendor/noctilucent/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# Ignore all files in this directory except the Dockerfile
/*
!/.gitignore
!/Dockerfile
41 changes: 41 additions & 0 deletions packages/aws-cdk/lib/vendor/noctilucent/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
FROM public.ecr.aws/debian/debian:buster-slim as build

# Install basic pre-requisites
RUN apt-get update \
&& apt-get install -y build-essential curl git libssl-dev openssl pkg-config zsh

# Make sure we use the correct shell going forward
SHELL ["/bin/zsh", "-c"]

# Install Rustup
ENV RUSTUP_HOME=/usr/local/rustup
ENV CARGO_HOME=/usr/local/cargo
RUN set -eo pipefail \
&& curl -fSsL "https://sh.rustup.rs" | sh -s -- -y --no-modify-path --profile=minimal \
&& echo "source ${CARGO_HOME}/env" >> /etc/profile.d/cargo.sh \
&& chmod -R a+rw ${CARGO_HOME}
ENV PATH=$PATH:${CARGO_HOME}/bin

# Install Node
RUN curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
&& apt-get install -y nodejs

# Install wasm-pack
RUN cargo install wasm-opt wasm-pack

ARG NOCTILUCENT_GIT
ARG NOCTILUCENT_COMMIT_ID

# Check out noctilucent
RUN git clone "${NOCTILUCENT_GIT}" "${TMPDIR}/noctilucent" \
&& git -C "${TMPDIR}/noctilucent" checkout -b wasm "${NOCTILUCENT_COMMIT_ID}"

# Build noctilucent to WASM
RUN cd "${TMPDIR}/noctilucent" \
&& wasm-pack build --target=nodejs --out-name=index --out-dir=/wasm-out \
&& rm --force /wasm-out/.gitignore /wasm-out/README.md /wasm-out/package.json \
&& echo "${NOCTILUCENT_GIT}:${NOCTILUCENT_COMMIT_ID}" > /wasm-out/.version

####################################################################################################
FROM scratch as wasm
COPY --from=build /wasm-out /
13 changes: 0 additions & 13 deletions packages/aws-cdk/vendor/README.md

This file was deleted.

0 comments on commit bc409ba

Please sign in to comment.