Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(CI): contracts compilers cached in a docker image #1888

Closed
wants to merge 24 commits into from

Conversation

tomg10
Copy link
Contributor

@tomg10 tomg10 commented May 8, 2024

No description provided.

mm-zk and others added 13 commits April 24, 2024 12:20
Signed-off-by: tomg10 <lemures64@gmail.com>
Signed-off-by: tomg10 <lemures64@gmail.com>
Signed-off-by: tomg10 <lemures64@gmail.com>
Signed-off-by: tomg10 <lemures64@gmail.com>
Signed-off-by: tomg10 <lemures64@gmail.com>
Signed-off-by: tomg10 <lemures64@gmail.com>
Signed-off-by: tomg10 <lemures64@gmail.com>
Signed-off-by: tomg10 <lemures64@gmail.com>
@tomg10 tomg10 changed the title [DO NOT MERGE] compilers dockerfile Contracts compilers cached in a docker image May 8, 2024
@tomg10 tomg10 changed the title Contracts compilers cached in a docker image feat(CI): contracts compilers cached in a docker image May 8, 2024
tomg10 added 10 commits May 8, 2024 20:45
Signed-off-by: tomg10 <lemures64@gmail.com>
Signed-off-by: tomg10 <lemures64@gmail.com>
Signed-off-by: tomg10 <lemures64@gmail.com>
Signed-off-by: tomg10 <lemures64@gmail.com>
Signed-off-by: tomg10 <lemures64@gmail.com>
Signed-off-by: tomg10 <lemures64@gmail.com>
Signed-off-by: tomg10 <lemures64@gmail.com>
Signed-off-by: tomg10 <lemures64@gmail.com>
@tomg10 tomg10 marked this pull request as ready for review May 8, 2024 20:02
@tomg10 tomg10 requested a review from a team as a code owner May 8, 2024 20:02
@mm-zk
Copy link
Collaborator

mm-zk commented May 9, 2024

i see that in this pr you are simply copying binaries from docker to others.
my original idea was to mount this docker as a volume in our docker compose that we run when we setup tests.

@tomg10 tomg10 requested a review from itegulov May 9, 2024 10:34
@slowli
Copy link
Contributor

slowli commented May 10, 2024

Just to understand: Is using the GitHub cache action not an option? I know that we run everything in Docker Compose, but as @mm-zk said, it should be possible to mount cache as a rw volume in the appropriate place. If this work, it'd be beneficial compared to the implemented approach because it wouldn't be necessary to manually synchronize compiler versions. OTOH, for the cache to work well, its key should reflect its content (so, it should include compiler versions somehow, perhaps in a hashed form).

Copy link
Contributor

@itegulov itegulov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I share @slowli's concern that we will need to manually update the images with this approach. Moreover, it will be very easy to upgrade solc version and forget about this Dockerfile as having the compiler here is not a hard requirement for CI.

Ideally I am envisioning some way to share this env-paths-based directory between builds so kind of what @slowli is talking about above.

Comment on lines +16 to +24
export async function prepareCompilersCache() {
// TODO(tomek) remove line below as soon as there is at least one pushed docker image
await utils.spawn('zk docker build compilers');
await utils.spawn('docker create --name temp matterlabs/compilers:latest2.0');
await utils.spawn('mkdir -p /root/.cache/hardhat-nodejs/compilers-v2');
await utils.spawn('docker cp temp:./.cache/hardhat-nodejs/compilers-v2/ /root/.cache/hardhat-nodejs/');
await utils.spawn('chmod -R +x /root/.cache/hardhat-nodejs/');
await utils.spawn('docker rm temp');
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems very environment-specific. Also, can we at least reuse whatever hardhat is using to derive this directories? env-paths AFAIR

Comment on lines +44 to +49
# This matches COMPILER_REPOSITORY_URL from hardhat-core.
RUN wget -nv -O $COMPILER_DIR/linux-amd64/list.json https://binaries.soliditylang.org/linux-amd64/list.json

RUN (for ver in solc-linux-amd64-v0.8.20+commit.a1b79de6 solc-linux-amd64-v0.8.23+commit.f704f362 solc-linux-amd64-v0.8.24+commit.e11b9ed9; do \
wget -nv -O $COMPILER_DIR/linux-amd64/$ver https://binaries.soliditylang.org/linux-amd64/$ver; \
done)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We also have some arm runners, do you plan to cross-build this to support them too?

@tomg10
Copy link
Contributor Author

tomg10 commented May 16, 2024

Closing this for now until we switch from foundry to hardhat

@tomg10 tomg10 closed this May 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants