Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: bootstrap_docker skips build it can pull from ecr. #2545

Merged
merged 111 commits into from
Sep 27, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
111 commits
Select commit Hold shift + click to select a range
04d5136
Initial cci_gen work.
charlielye Sep 16, 2023
27ef8da
Intial
charlielye Sep 16, 2023
06ae9b1
more
charlielye Sep 16, 2023
3d55c7c
Merge remote-tracking branch 'origin/master' into cl/ypb_cleanup
charlielye Sep 16, 2023
a26f1f4
wip
charlielye Sep 16, 2023
61b2b80
wip
charlielye Sep 17, 2023
14a334f
wip
charlielye Sep 17, 2023
63a7d3c
wip
charlielye Sep 17, 2023
cdc2b65
fix
charlielye Sep 17, 2023
c7c5baf
Add yq bin.
charlielye Sep 17, 2023
d3f44c8
Move to native version of yq. Can also now remove jq from build-system.
charlielye Sep 17, 2023
4d818d1
fix
charlielye Sep 17, 2023
d4f5a81
debug
charlielye Sep 17, 2023
36c8251
fix: PATH and bb.js ref
ludamad Sep 18, 2023
3d52670
fix: PATH and bb.js ref
ludamad Sep 18, 2023
7a6e238
fix: PATH and bb.js ref
ludamad Sep 18, 2023
e72e347
Merge remote-tracking branch 'origin/master' into cl/ypb_cleanup
ludamad Sep 19, 2023
a447ffe
fix: pass debug env to spot [debug ci]
ludamad Sep 19, 2023
054efef
tinkering [debug ci]
ludamad Sep 19, 2023
7b93f83
Merge remote-tracking branch 'origin/master' into cl/ypb_cleanup
ludamad Sep 19, 2023
648c910
fix: yarn-project-base-deps not in manifest [debug ci]
ludamad Sep 19, 2023
617f0ac
fix: dont set CACHE_FROM if try_fetch_image fails
ludamad Sep 19, 2023
da5b6d9
Format private-token
ludamad Sep 19, 2023
ff0d592
Lint private-token
ludamad Sep 19, 2023
f1f649e
chore: speed up yarn-project step
ludamad Sep 19, 2023
daf365c
Revert change
ludamad Sep 19, 2023
23c1a93
yarn-project Dockerfile update
ludamad Sep 19, 2023
79ea225
yarn-project Dockerfile update
ludamad Sep 19, 2023
7d491c2
Fix script call.
charlielye Sep 19, 2023
0813f83
wip
charlielye Sep 20, 2023
c01660a
Use r6a instances to build. Higher cpu count for bb. Allow [ci rebuil…
charlielye Sep 20, 2023
ebe6bb3
Run cci_gen. [ci rebuild]
charlielye Sep 20, 2023
36de7fb
Simplify ci cmd stuff. Rollback cpu count to see if it actually helpe…
charlielye Sep 20, 2023
28da340
Forgot to cci_gen... [ci rebuild]
charlielye Sep 20, 2023
5701341
Add timestamps. yolo [ci rebuild]
charlielye Sep 20, 2023
46e468c
Add timestamps. yolo [ci rebuild]
charlielye Sep 20, 2023
6d9378b
Remove redundant timestamps
ludamad Sep 20, 2023
54f89f2
fix: rebuild flag unbound
ludamad Sep 20, 2023
961dd5c
fix: use fetching commit message
ludamad Sep 20, 2023
173b5ee
wip [ci rebuild]
charlielye Sep 20, 2023
576c8f2
Merge remote-tracking branch 'origin/master' into cl/ypb_cleanup
charlielye Sep 20, 2023
5018c7e
Remove cci_gen stuff for now.
charlielye Sep 21, 2023
2128754
Fix
charlielye Sep 21, 2023
b87c3c1
wip
charlielye Sep 21, 2023
fe3390f
README
charlielye Sep 21, 2023
0d227a8
Formatting
charlielye Sep 21, 2023
951bf36
Disable blank tests as not passing.
charlielye Sep 21, 2023
a766965
fix
charlielye Sep 21, 2023
d22b550
fix
charlielye Sep 21, 2023
914e2b9
fix
charlielye Sep 21, 2023
4784413
fix
charlielye Sep 21, 2023
c4abeac
fix
charlielye Sep 21, 2023
644c8f2
fix
charlielye Sep 21, 2023
1e0e6ba
ypb down to 730MB
charlielye Sep 23, 2023
3596ee8
Smaller builds using pnp (at last second), and some incantations.
charlielye Sep 24, 2023
30e5e86
Comments.
charlielye Sep 24, 2023
c95c1f2
Bump ubuntu machine executor image to 2204
charlielye Sep 24, 2023
5738974
Fix wait on missing image.
charlielye Sep 24, 2023
d8ecba6
Revert Dockerfile.dockerignore to work with Docker v20. Can move to m…
charlielye Sep 24, 2023
c8da4d2
Fix sandbox dockerfile
charlielye Sep 24, 2023
b3afd23
Fix canary Dockerfile.
charlielye Sep 24, 2023
3e5a985
Add tests to own job.
charlielye Sep 24, 2023
6d05422
Make tests actually run...
charlielye Sep 24, 2023
2238e78
Add ecr_login
charlielye Sep 24, 2023
a10336c
Fix browser.
charlielye Sep 24, 2023
0c7a83d
Make yarn-project use some rebuild patterns. May need tweaking. Fix a…
charlielye Sep 24, 2023
2434d9a
Try and get browser test working again.
charlielye Sep 24, 2023
b1a8f4b
Fix canary.
charlielye Sep 24, 2023
a03f341
Merge remote-tracking branch 'origin/master' into cl/ypb_cleanup
charlielye Sep 25, 2023
3b5a942
fix: ypb deploy (#2464)
ludamad Sep 25, 2023
1a77721
chore: yarn project test breakup (#2484)
ludamad Sep 25, 2023
47a83d2
Merge branch 'master' into cl/ypb_cleanup
ludamad Sep 25, 2023
d0446aa
fix: cimg
ludamad Sep 25, 2023
46b3ab4
fix: cimg
ludamad Sep 25, 2023
2037278
[ci debug]
ludamad Sep 25, 2023
4e0cb82
Merge remote-tracking branch 'origin/cl/ypb_cleanup' into cl/ybp_smaller
charlielye Sep 25, 2023
c01cae3
fix: Use calculate_image_uri
ludamad Sep 25, 2023
6322f5c
[ci debug]
ludamad Sep 25, 2023
71ca071
ecr login [ci debug]
ludamad Sep 25, 2023
4376ca9
Merge remote-tracking branch 'origin/cl/ypb_cleanup' into cl/ybp_smaller
charlielye Sep 25, 2023
a79cd39
fix: dont require tag
ludamad Sep 25, 2023
5507089
fix: dont require tag
ludamad Sep 25, 2023
b8b23cd
Fix merge. yarn-project on spot.
charlielye Sep 25, 2023
5fe0019
Merge remote-tracking branch 'origin/cl/ypb_cleanup' into cl/ybp_smaller
charlielye Sep 25, 2023
50cf1a2
Fix canary deploy build.
charlielye Sep 25, 2023
5901208
Merge remote-tracking branch 'origin/master' into cl/ybp_smaller
charlielye Sep 25, 2023
8219860
Fix merge oops.
charlielye Sep 25, 2023
12a07a2
Merge branch 'master' into cl/ybp_smaller
charlielye Sep 25, 2023
03129c1
Parallel wasm/wasm-threads build.
charlielye Sep 25, 2023
a1b3ccc
Shift pnpify into final artefact, separate formatting again. [ci rebu…
charlielye Sep 25, 2023
eee400c
Merge [ci rebuild]
charlielye Sep 25, 2023
0510b1e
[ci rebuild yarn-project]
charlielye Sep 25, 2023
fdde4b9
Simpler build that doesn't push/pull to leverage layer cache.
charlielye Sep 25, 2023
9d6f022
Attempt to handle ' in commit msg.
charlielye Sep 25, 2023
91fd342
[ci rebuild yarn-project]
charlielye Sep 25, 2023
d39a46f
[ci rebuild yarn-project]
charlielye Sep 25, 2023
1f898e3
[ci rebuild yarn-project]
charlielye Sep 25, 2023
05bb816
[ci rebuild]
charlielye Sep 25, 2023
d400647
Add yarn-project as doc dep.
charlielye Sep 25, 2023
a3f2fab
Fix docs
charlielye Sep 26, 2023
2497b24
Add file.
charlielye Sep 26, 2023
a11e2d0
Remove pnp. Gains not worth it.
charlielye Sep 26, 2023
21506f6
Merge remote-tracking branch 'origin/master' into cl/ybp_smaller
charlielye Sep 26, 2023
6d827ce
Slim stage yarn-project.
charlielye Sep 26, 2023
1ff4e45
Formatting.
charlielye Sep 26, 2023
d16ad40
Merge branch 'cl/ybp_smaller' into cl/no_build_layer_cache
charlielye Sep 26, 2023
b495771
boostrap_docker will skip builds it can pull from ecr
charlielye Sep 26, 2023
7dd3c5a
Consider working tree when computing content hash (on request)
charlielye Sep 26, 2023
736be1b
Merge remote-tracking branch 'origin/master' into cl/bootstrap_docker…
charlielye Sep 26, 2023
6dcf540
Favour rebuilding changed over computing new hash.
charlielye Sep 27, 2023
b67799a
Color!
charlielye Sep 27, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 44 additions & 4 deletions build-system/scripts/build_local
Original file line number Diff line number Diff line change
Expand Up @@ -63,15 +63,55 @@ for E in "${PROJECTS[@]}"; do
fi
fi

# TODO: Move everything to aztecprotocol/*
DEPLOY_IMAGE_URI=$ECR_DEPLOY_URL/$REPO:latest
CACHE_IMAGE_URI=$(calculate_image_uri $REPO)
ARR=(${CACHE_IMAGE_URI//:/ })
REPO_URI=${ARR[0]}
TAG=${ARR[1]}

GREEN="\033[32m"
BLUE="\033[94m"
YELLOW="\033[93m"
BOLD="\033[1m"
RESET="\033[0m"

echo
echo
echo
echo "*** Building $PROJECT_DIR_NAME:$DOCKERFILE -> $REPO ***"
echo "*** Context: $PWD"
echo -e "${BOLD}Repository${RESET}: ${BOLD}${BLUE}$REPO${RESET}"
echo -e "${BOLD}Cache Image URI${RESET}: $REPO_URI:$TAG"
echo -e "${BOLD}Deploy Image URI${RESET}: $DEPLOY_IMAGE_URI"
echo -e "${BOLD}Dockerfile${RESET}: $DOCKERFILE"
echo -e "${BOLD}Context${RESET}: $PWD"
echo

docker build ${ADDITIONAL_ARGS:-} --build-arg ARG_COMMIT_HASH=$COMMIT_HASH -f $DOCKERFILE -t $ECR_DEPLOY_URL/$REPO:latest .
docker tag $ECR_DEPLOY_URL/$REPO:latest aztecprotocol/$REPO:latest
GREP_PATTERN=$(query_manifest rebuildPatterns $REPO | tr '\n' '|')
# Remove the trailing '|'.
GREP_PATTERN=${GREP_PATTERN%|}

# Ensure index is updated before we call git diff-index.
git status > /dev/null

if (git diff-index --name-only HEAD; git ls-files --others --exclude-standard) | grep -qE "$GREP_PATTERN"; then
echo -e "${YELLOW}Project or dependency has local modifications! Building...${RESET}"
docker build ${ADDITIONAL_ARGS:-} --build-arg ARG_COMMIT_HASH=$COMMIT_HASH -f $DOCKERFILE -t $DEPLOY_IMAGE_URI .
else
if docker image ls --format "{{.Repository}}:{{.Tag}}" | grep -q -w $CACHE_IMAGE_URI; then
echo -e "${GREEN}Image exists locally. Tagging as $DEPLOY_IMAGE_URI${RESET}"
docker tag $CACHE_IMAGE_URI $DEPLOY_IMAGE_URI
else
if ecr_login && image_exists $REPO_URI $TAG ; then
docker pull $CACHE_IMAGE_URI
else
docker build ${ADDITIONAL_ARGS:-} --build-arg ARG_COMMIT_HASH=$COMMIT_HASH -f $DOCKERFILE -t $CACHE_IMAGE_URI .
docker tag $CACHE_IMAGE_URI $DEPLOY_IMAGE_URI
fi
fi
fi

# Retag for aztecprotocol dockerhub.
docker tag $DEPLOY_IMAGE_URI aztecprotocol/$REPO:latest

if [ "$PROJECT_DIR_NAME" = "$TARGET_PROJECT" ]; then
if [ -n "$LAUNCH" ]; then
Expand Down
3 changes: 2 additions & 1 deletion build-system/scripts/calculate_content_hash
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,10 @@ REBUILD_PATTERNS=$(query_manifest rebuildPatterns $REPOSITORY)

AWK_PATTERN=$(echo $REBUILD_PATTERNS | sed 's/ /|/g')
cd "$(git rev-parse --show-toplevel)"

# an example line is
# 100644 da9ae2e020ea7fe3505488bbafb39adc7191559b 0 yarn-project/world-state/tsconfig.json
# this format is beneficial as it grabs the hashes from git efficiently
# we will next filter by our rebuild patterns
# then we pipe the hash portion of each file to git hash-object to produce our content hash
git ls-tree -r $COMMIT_HASH | awk -v pattern="($AWK_PATTERN)" '$4 ~ pattern {print $3}' | git hash-object --stdin
git ls-tree -r $COMMIT_HASH | awk -v pattern="($AWK_PATTERN)" '$4 ~ pattern {print $3}' | git hash-object --stdin
1 change: 0 additions & 1 deletion build-system/scripts/calculate_image_uri
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,5 @@ REPOSITORY=$1
# Assume we might be calling this directly without build system in path
export ROOT_PATH=$(git rev-parse --show-toplevel)
export PATH="$PATH:$ROOT_PATH/build-system/scripts:$ROOT_PATH/build-system/bin"
export LD_LIBRARY_PATH="$ROOT_PATH/build-system/lib"
CONTENT_HASH=$(calculate_content_hash $REPOSITORY)
echo "278380418400.dkr.ecr.us-east-2.amazonaws.com/$REPOSITORY:cache-$CONTENT_HASH"