-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cleaning and improvements in the Build Pipeline #2873
Cleaning and improvements in the Build Pipeline #2873
Conversation
FYI this is mostly for consideration in the open, branch commits are dirty as I've been running remote builds a lot. Things on my mind:
|
On the subject of committing back the It's a tradeoff. Keeping it will mean running a There are some useful parallels with Rust's cargo as mentioned in the below thread
python-poetry/poetry#7488 |
Subsequent builds will be quicker due to the addition of caching, I may re-run the actions for this ref for this reason, also looks like we got a slow runner. |
Looking good! Let me know when it is ready for for final review! |
👍 no worries, I really just need to test the publication logic, which is hard to do without standing up some PyPi/Docker repo, so I'll do this as soon as I can to test it out on my fork then I'll flag this for review |
So I tested out the publishing logic and fixed a few quirks to do with tag resolution etc. Artifacts have been published successfully to: Would consider this in a review-able state:
|
.github/workflows/tests.yml
Outdated
- { name: "3.10", python: "3.10", os: ubuntu-latest, tox: py310 } | ||
- { name: "3.9", python: "3.9", os: ubuntu-latest, tox: py39 } | ||
# Static analysis and utilities | ||
- { name: "Verification Tests: Ruff", python: "3.11", os: ubuntu-latest, tox: "ruff" } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you can drop the prefixes for the names. "Ruff/Mypy/Python 3.X" are specific enough, and I dont really know what a "verification test" is :P
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And if you really want to have prefixes, maybe "static analysis" is a better prefix?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We don't need prefixes nah -
I actually had them as Static Analysis but then I added the docs
building in so it kinda ruined the fun
I can make it something shorter 🤔
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Addressed in 7d7bfc6
FROM python:3.11-slim AS base | ||
|
||
FROM base AS builder | ||
RUN apt-get update && apt-get install -y git |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's the reason for installing git here? Where is it used?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No idea! This is cruft from the historical docker image, I assumed the building of binaries required it
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It was needed so that we can determine the locust version name (it was used by setuptools_scm), e.g. for use with locust --version
. Does that still work with the new way of building the docker image?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe the Dynamic Versioning plugin for Poetry comes with whatever it needs 🤔
--- but I'll check on the docker image, that a produced version is correct
On the CI, I did make a cut-down docker image which doesn't do any building at all, just takes the already-known-good built assets - the version of this is already resolved in the package (and can be seen in the test-docker-image step)
On the "normal" image, which users can still build end-to-end, I can check that the output of locust --version
is correct inside the docker image 🤔
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It turns out the git binary is a necessity for resolving the version (understandably) even using dunamai
etc. so I guess it gets left in, wanted to have the images as close to each other as possible.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Addressed in 7d7bfc6
(had to allow dist
in the docker context for the CI, so the local one has to remove it (or declare a wheel file name specifically)
FYI @cyberw this brings the Docker image build size back down to the same ~90mb it was previously 🥳 |
I tried to improve the error message for the |
the error went away on retry so 🎊 |
nooo... I didnt squash :-/ |
Overview:
Hey it's the draft PR guy again!
This is a WIP branch where I'm attempting to clean up the build pipeline as per #2857 - which is to say, implementing the hygiene that was omitted from the first iteration of the "get poetry working" work
The goal of this work is not only jank reduction, but:
For @cyberw @andrewbaldwin44 and community
Major
poetry
featuretox
steps for integration testing to allow this to use an externally-built artifactpoetry.lock
file to start caching dependency versions, but this could be reverted, there are tradeoffs I'll outline in the commentsMinor
poetry
tox
test to make sure the docs build worksruff
version markerdist
to be picked up by the Docker context for theci
image