feat: CI/local S3 build cache

[ludamad]

What this PR achieves: Optional S3 caching in Earthly files with a 2nd layer of file caching (except in C). A future step would be to use this cache locally for fast bootstrapping. Currently implemented for barretenberg and noir.

If AWS credentials are available with this script, S3 will be used as a download source. Otherwise, just the local minio cache will be used. If both are used, minio will act like a pull-through cache for S3 and provide a two-tiered caching solution good for when earthly is being really forgetful and you are using artifacts that have not changed in a while.

USAGE NOTE: After this PR, you will want to run on scripts/setup-earthly-local.sh or equivalent if not using zsh. This will create an alias for earthly as earthly-local in the repo, which sets up scripts and starts the (currently always on) file server.

USAGE NOTE: Because we use git to reliably make content hashes that aren't prone to changing due to temporary files, we require all changes to be committed before we content hashes will be used. The cache will not be used at all if there are staged changes. Taking feedback on this.

I recommend doing something like git commit -am 'sync' && earthly ... if changing files in a loop (possibly with --amend). Remember to git add new wanted files or you may get old cache.

USAGE NOTE: If you do not want AWS secrets in the env for every program you can figure out your own wrapper alias, perhaps reading from files

EARTHFILE DEV NOTE: There is a little bit of a dance to grab rebuild patterns and run with the git context on the local machine. To enable this, WORKDIR needs to match the repo layout, some adjustments thusly

Bundled:

• Just make build the root of the dependency tree in github. This should fix a lot of contention issues

Followups:

• Cache protocol circuits and verification keys

[github-actions]

Changes to public function bytecode sizes

Generated at commit: 9e06a73ad7904c637f78bc21561ce9bb82bcec54, compared to commit: 6a132906ec8653cec7f30af2e008c8881d42db46

🧾 Summary (100% most significant diffs)

Program	Bytecode size in bytes (+/-)	%
AvmTest::public_dispatch	+378 ❌	+0.46%
FPC::public_dispatch	+28 ❌	+0.23%
Auth::public_dispatch	+21 ❌	+0.13%
NFT::public_dispatch	+28 ❌	+0.11%
Uniswap::public_dispatch	+28 ❌	+0.08%
TokenBlacklist::public_dispatch	-56 ✅	-0.06%
TokenBridge::public_dispatch	-28 ✅	-0.08%
Lending::public_dispatch	-860 ✅	-1.37%

---

Full diff report 👇

Program	Bytecode size in bytes (+/-)	%
AvmTest::public_dispatch	83,276 (+378)	+0.46%
FPC::public_dispatch	12,260 (+28)	+0.23%
Auth::public_dispatch	16,544 (+21)	+0.13%
NFT::public_dispatch	26,558 (+28)	+0.11%
Uniswap::public_dispatch	36,528 (+28)	+0.08%
TokenBlacklist::public_dispatch	93,563 (-56)	-0.06%
TokenBridge::public_dispatch	36,379 (-28)	-0.08%
Lending::public_dispatch	61,701 (-860)	-1.37%

[github-actions]

Changes to circuit sizes

Generated at commit: 9e06a73ad7904c637f78bc21561ce9bb82bcec54, compared to commit: 2d5ae664964b66c4b617965fe85488e95706a8d3

🧾 Summary (100% most significant diffs)

Program	ACIR opcodes (+/-)	%	Circuit size (+/-)	%
rollup_base	0 ➖	0.00%	+383 ❌	+0.01%
rollup_merge	0 ➖	0.00%	+6 ❌	+0.00%
parity_root	0 ➖	0.00%	+10 ❌	+0.00%
rollup_block_root	0 ➖	0.00%	+6 ❌	+0.00%
rollup_block_root_final	0 ➖	0.00%	+6 ❌	+0.00%
rollup_root	0 ➖	0.00%	+4 ❌	+0.00%
rollup_block_merge	0 ➖	0.00%	+4 ❌	+0.00%

---

Full diff report 👇

Program	ACIR opcodes (+/-)	%	Circuit size (+/-)	%
rollup_base	423,026 (0)	0.00%	3,734,646 (+383)	+0.01%
rollup_merge	1,483 (0)	0.00%	2,751,436 (+6)	+0.00%
parity_root	2,114 (0)	0.00%	5,486,736 (+10)	+0.00%
rollup_block_root	1,966 (0)	0.00%	4,121,162 (+6)	+0.00%
rollup_block_root_final	1,966 (0)	0.00%	4,121,162 (+6)	+0.00%
rollup_root	18,844 (0)	0.00%	2,787,661 (+4)	+0.00%
rollup_block_merge	18,860 (0)	0.00%	2,787,693 (+4)	+0.00%