Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: split CI into separate moldy jobs #1192

Merged
merged 7 commits into from
Nov 4, 2024
Merged

ci: split CI into separate moldy jobs #1192

merged 7 commits into from
Nov 4, 2024

Conversation

a10y
Copy link
Contributor

@a10y a10y commented Nov 3, 2024

  • use mold linker for building Rust code
  • Splits into several parallel jobs
  • disables one test that was taking ~7 minutes with miri, cutting the miri time down considerably

Shaves ~40% off of CI times

@a10y a10y changed the title [testing] use mold linker for build workflow feat: CI speedups Nov 4, 2024
@a10y a10y changed the title feat: CI speedups ci: split CI into separate moldy jobs Nov 4, 2024
.github/workflows/ci.yml Outdated Show resolved Hide resolved
.github/workflows/ci.yml Outdated Show resolved Hide resolved
.github/workflows/ci.yml Outdated Show resolved Hide resolved
@@ -65,6 +55,60 @@ jobs:
(cd pyvortex && uv run maturin develop)

(cd docs && uv run make html)

rust-docs:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

weirdly the command pyvortex runs also does this

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah make -C docs html builds all the docs, including the Rust docs. We could add a different target like make -C docs full-html and use that in docs.yml.

steps:
- uses: rui314/setup-mold@v1
- uses: actions/checkout@v4
- uses: ./.github/actions/cleanup
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to run cleanup as the first thing in the job, otherwise it evicts installed binaries

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But also maybe we don't need it that much if we have separate jobs?

@a10y a10y marked this pull request as ready for review November 4, 2024 13:33
@robert3005
Copy link
Member

robert3005 commented Nov 4, 2024

Ok @a10y I think we should change the cleanup job if we are splitting things, it's now taking close to 50% of the time. for some of them now. Could probably trim down the list. I went full hog trying to make sure we don't have to do a cleanup midway through

@a10y
Copy link
Contributor Author

a10y commented Nov 4, 2024

Ok I've removed usage of cleanup action from all steps, and have added a new custom action to download a flatc compiler so that we can check all generated files at CI time

@robert3005
Copy link
Member

looks like you do need some level of cleanup of hosted tools otherwise our builds step runs out of disk

@a10y
Copy link
Contributor Author

a10y commented Nov 4, 2024

It was node_modules that was slow to delete. Cleanup times went from 3minutes -> ~10 seconds now

@a10y a10y enabled auto-merge (squash) November 4, 2024 14:26
@a10y a10y disabled auto-merge November 4, 2024 14:26
@a10y a10y merged commit 7846c36 into develop Nov 4, 2024
12 checks passed
@a10y a10y deleted the aduffy/lets-get-moldy branch November 4, 2024 14:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants