Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: build noctilucent WASM library in a container #26123

Merged
merged 7 commits into from
Jun 28, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -207,7 +207,7 @@ eval $(gp env -e)

### Amazon CodeCatalyst Dev Environments

Dev Environments are cloud-based development environments.
Dev Environments are cloud-based development environments.
[Amazon CodeCatalyst](https://aws.amazon.com/codecatalyst/) allows you to checkout your linked Github
repositories in your Dev Environments with your favorite local IDEs such as VSCode or JetBrains.

Expand All @@ -220,7 +220,7 @@ $ NODE_OPTIONS=--max-old-space-size=8192 npx lerna run build --scope=aws-cdk-lib

You may [configure your Dev Env](https://docs.aws.amazon.com/codecatalyst/latest/userguide/devenvironment-devfile.html) with the `devfile.yaml` to further customize your Dev Env for CDK development.

Read the links below for more details:
Read the links below for more details:
- [Dev Environments in CodeCatalyst](https://docs.aws.amazon.com/codecatalyst/latest/userguide/devenvironment.html)
- [Using GitHub repositories in CodeCatalyst](https://docs.aws.amazon.com/codecatalyst/latest/userguide/extensions-github.html)
- [Setting up to use the AWS CLI with CodeCatalyst](https://docs.aws.amazon.com/codecatalyst/latest/userguide/set-up-cli.html)
Expand Down Expand Up @@ -429,7 +429,7 @@ $ npx cdk -a test/aws-eks/test/sample.js deploy

This allows you to iterate your development and ensure a minimal sample app would successfully deploy as you expect.
You have the freedom to interact with it just as a common CDK app such as viewing differences with `npx cdk diff`
or pass context variables with `npx cdk deploy -c`. You can rapidly iterate your testing with repeated deployments
or pass context variables with `npx cdk deploy -c`. You can rapidly iterate your testing with repeated deployments
by importing existing resource such as existing VPC. This can save a lot of time and help you focus on the core changes.

```ts
Expand All @@ -439,7 +439,7 @@ const vpc = ec2.Vpc.fromLookup(stack, 'Vpc', { isDefault: true });
As this is for testing only, do not commit `sample.ts` and `sample.js` to your PR branch.

Alternatively, you can write this test as a new integration test like `integ.my-test.ts` and deploy it
using `yarn integ --no-clean`. This may be useful when you need to publish a new
using `yarn integ --no-clean`. This may be useful when you need to publish a new
integration test:

```console
Expand Down
3 changes: 0 additions & 3 deletions packages/aws-cdk/.gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,3 @@ test/integ/cli/*.d.ts
.DS_Store

junit.xml

# Exclude the noctilucent WASM package
lib/vendor/noctilucent/
4 changes: 2 additions & 2 deletions packages/aws-cdk/.npmignore
Original file line number Diff line number Diff line change
Expand Up @@ -29,5 +29,5 @@ tsconfig.json
**/cdk.out
junit.xml

# exclude noctilucent source
/vendor/noctilucent/
generate.sh
lib/vendor/noctilucent/Dockerfile
44 changes: 17 additions & 27 deletions packages/aws-cdk/generate.sh
Original file line number Diff line number Diff line change
Expand Up @@ -15,30 +15,20 @@ cat > build-info.json <<HERE
}
HERE

# Build noctilucent package
(
# Check out the submodule if it's not there already
if [ ! -f "vendor/noctilucent/Cargo.toml" ]; then
git -C ./vendor clone https://github.com/iph/noctilucent.git
fi

# update the package to the pinned commit hash
git -C vendor/noctilucent reset --hard HEAD
git -C vendor/noctilucent fetch && git -C vendor/noctilucent checkout 6da7c9fade55f8443bba7b8fdfcd4ebfe5208fb1

# Install wasm-pack if it's not there already
if ! command -v wasm-pack >/dev/null 2>/dev/null; then
echo "installing wasm-pack, this may take a while..."
cargo install wasm-pack
fi

pkgroot=$(cd $(dirname -- "$0") && pwd)

cd vendor/noctilucent
wasm-pack build --target nodejs \
--out-dir="${pkgroot}/lib/vendor/noctilucent" \
--out-name=index

cd ../../lib/vendor/noctilucent
rm package.json
)
# Build noctilucent package in a Docker/Finch VM
NOCTILUCENT_GIT="https://github.com/iph/noctilucent.git"
NOCTILUCENT_COMMIT_ID="6da7c9fade55f8443bba7b8fdfcd4ebfe5208fb1"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with @rix0rrr this feels like we are reinventing versioning and releases.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To clarify - this is just a stepping stone for now... We're trying to avoid wasting time on nom logistics to focus on the product itself. We (the folks working on migrate) are going to own the upkeep of this until it transitions to something our automation is able to deal with...

if [ "$(cat lib/vendor/noctilucent/.version 2>/dev/null || echo '')" == "${NOCTILUCENT_GIT}:${NOCTILUCENT_COMMIT_ID}" ]
then
echo "⏭️ Noctilucent WASM binary is up-to date, skipping build..."
echo "ℹ️ Delete lib/vendor/noctilucent/.version to force a rebuild."
Comment on lines +23 to +24
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️ clever

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But: wouldn't a git submodule do the same trick, and/or releasing the Noctilucent WASM package to NPMJS by itself?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we were originally using git submodules but it turned out codepipeline didn't like that very much so we had to just get it ourselves

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Submodules come with their own set of issues. If it's required for a build, we should not put it into a submodule.

The noctilucent-wasm npm package is still an option.

else
echo "⏳ Building Noctilucent WASM binary for embedding... This will take a while..."
${CDK_DOCKER:-docker} build --rm \
--build-arg NOCTILUCENT_GIT="${NOCTILUCENT_GIT}" \
--build-arg NOCTILUCENT_COMMIT_ID="${NOCTILUCENT_COMMIT_ID}" \
--file lib/vendor/noctilucent/Dockerfile \
--target wasm \
--output type=local,dest=lib/vendor/noctilucent \
lib/vendor/noctilucent
fi
4 changes: 4 additions & 0 deletions packages/aws-cdk/lib/vendor/noctilucent/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# Ignore all files in this directory except the Dockerfile
/*
!/.gitignore
!/Dockerfile
41 changes: 41 additions & 0 deletions packages/aws-cdk/lib/vendor/noctilucent/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
FROM public.ecr.aws/debian/debian:buster-slim as build

# Install basic pre-requisites
RUN apt-get update \
&& apt-get install -y build-essential curl git libssl-dev openssl pkg-config zsh

# Make sure we use the correct shell going forward
SHELL ["/bin/zsh", "-c"]

# Install Rustup
ENV RUSTUP_HOME=/usr/local/rustup
ENV CARGO_HOME=/usr/local/cargo
RUN set -eo pipefail \
&& curl -fSsL "https://sh.rustup.rs" | sh -s -- -y --no-modify-path --profile=minimal \
&& echo "source ${CARGO_HOME}/env" >> /etc/profile.d/cargo.sh \
&& chmod -R a+rw ${CARGO_HOME}
ENV PATH=$PATH:${CARGO_HOME}/bin

# Install Node
RUN curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
&& apt-get install -y nodejs

# Install wasm-pack
RUN cargo install wasm-opt wasm-pack

ARG NOCTILUCENT_GIT
ARG NOCTILUCENT_COMMIT_ID

# Check out noctilucent
RUN git clone "${NOCTILUCENT_GIT}" "${TMPDIR}/noctilucent" \
&& git -C "${TMPDIR}/noctilucent" checkout -b wasm "${NOCTILUCENT_COMMIT_ID}"

# Build noctilucent to WASM
RUN cd "${TMPDIR}/noctilucent" \
&& wasm-pack build --target=nodejs --out-name=index --out-dir=/wasm-out \
&& rm --force /wasm-out/.gitignore /wasm-out/README.md /wasm-out/package.json \
&& echo "${NOCTILUCENT_GIT}:${NOCTILUCENT_COMMIT_ID}" > /wasm-out/.version

####################################################################################################
FROM scratch as wasm
COPY --from=build /wasm-out /
13 changes: 0 additions & 13 deletions packages/aws-cdk/vendor/README.md

This file was deleted.