-
Notifications
You must be signed in to change notification settings - Fork 14.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Move javascript compilation to host #25169
Conversation
cc: @pierrejeambrun @bbovenzi - please take a look, I think you will find this change really cool. |
e1a2212
to
225c1e8
Compare
34fe38c
to
dc46ffa
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tested this out locally. Everything seems to be working. But does look like we need a quick rebase.
Cool. Yeah. I will . :) |
Thanks for testing @bbovenzi - > I think this will help a bit with faster iterations and maybe we could also add some more dev-friendly javascript tooling around to attract more contributors and make it easier to start/test/debug Javascript code. Especially with the typescript and React, I have a feeling we could add some tooling to breeze to improve "first-time-experience" for new users. Happy to brainstorm on it and implement any improvements there :) |
dc46ffa
to
1a949e4
Compare
Hm. Interestingly enough - we seem to have not enough space on our machines to get THREE yarn installs :) |
Are we not clearing the cache each time? |
Not when we have three independent yarn builds run in pre-commits - the envs are created in ~/.cache which is rather small it seems, But I am looking into it. |
1a949e4
to
1ffd5a1
Compare
Let's see now :) |
Added more disk space for the instance (later I can optimize it even more). |
Now we are cooking with gas. I will just re-run it with public runners to be absolutely sure the yarn modules will not make it run out of the capacity either. |
Previously, in order to keep consistent development environment we've compiled javascript in the CI image. However we can utilise power of pre-commmit for setting the node environment for all contributors automatically. Instead of compiling the javascript in the image, we can compile it via pre-commit in the host. This can be done thanks to the new python breeze which is far more flexible and can now add execution of compilation of the javascript when needed and using pre-commit environments. Thanks to that, we can vastly simplify the Dockerfiles and scripts that are used to automatically build or signal that the assets need recompilation. We can basically assume that the assets were prepared outside of the image building (and breeze makes sure it happens) The changes: * node.js is not needed in images (neither PROD build nor CI) * no need for multiple asset compilation scripts. All is done via pre-commit environment with `breeze compile-www-assets`` command * lint checks for UI do not need the docker image any more (they are also based on pre-commit environment) * no more checks/warnings when you enter the image * start-airflow command builds the compilation before entering * prepare-airflow-package runs asset compilation before entering docker airflow building.
1ffd5a1
to
f07c3d8
Compare
I have some automation which runs
Is that a bug, or is there a better way for me to be building this? |
A bug. fix is coming |
Asset compilation via setup.py has been broken in apache#25169. This PR fixes it.
PR here: #25201 |
Asset compilation via setup.py has been broken in #25169. This PR fixes it.
Merged. Should be fixed now @MatrixManAtYrService |
Previously, in order to keep consistent development environment we've compiled javascript in the CI image. However we can utilise power of pre-commmit for setting the node environment for all contributors automatically. Instead of compiling the javascript in the image, we can compile it via pre-commit in the host. This can be done thanks to the new python breeze which is far more flexible and can now add execution of compilation of the javascript when needed and using pre-commit environments. Thanks to that, we can vastly simplify the Dockerfiles and scripts that are used to automatically build or signal that the assets need recompilation. We can basically assume that the assets were prepared outside of the image building (and breeze makes sure it happens) The changes: * node.js is not needed in images (neither PROD build nor CI) * no need for multiple asset compilation scripts. All is done via pre-commit environment with `breeze compile-www-assets`` command * lint checks for UI do not need the docker image any more (they are also based on pre-commit environment) * no more checks/warnings when you enter the image * start-airflow command builds the compilation before entering * prepare-airflow-package runs asset compilation before entering docker airflow building. (cherry picked from commit acff129)
When we moved www asset compilation from container to the host in apache#25169, we removed yarn from the image and the old instructions are no longer valid. Instead of in-container yarn dev we expect now to have yarn dev run in the host. It can be done either with pre-commit (if you have no node/yarn installed in your host, pre-commit will automatically install both node and yarn in the right versions, or if you have yarn installed you can run it manually. The 'start-airflow' instructions now remove the in-container instructions and explain those two options you have.
When we moved www asset compilation from container to the host in #25169, we removed yarn from the image and the old instructions are no longer valid. Instead of in-container yarn dev we expect now to have yarn dev run in the host. It can be done either with pre-commit (if you have no node/yarn installed in your host, pre-commit will automatically install both node and yarn in the right versions, or if you have yarn installed you can run it manually. The 'start-airflow' instructions now remove the in-container instructions and explain those two options you have.
When we moved www asset compilation from container to the host in #25169, we removed yarn from the image and the old instructions are no longer valid. Instead of in-container yarn dev we expect now to have yarn dev run in the host. It can be done either with pre-commit (if you have no node/yarn installed in your host, pre-commit will automatically install both node and yarn in the right versions, or if you have yarn installed you can run it manually. The 'start-airflow' instructions now remove the in-container instructions and explain those two options you have. (cherry picked from commit 1544868)
Previously, in order to keep consistent development environment
we've compiled javascript in the CI image. However we can utilise
power of pre-commmit for setting the node environment for all
contributors automatically. Instead of compiling the javascript
in the image, we can compile it via pre-commit in the host.
This can be done thanks to the new python breeze which is far more
flexible and can now add execution of compilation of the javascript when
needed and using pre-commit environments.
Thanks to that, we can vastly simplify the Dockerfiles and scripts that
are used to automatically build or signal that the assets need
recompilation. We can basically assume that the assets were prepared
outside of the image building (and breeze makes sure it happens)
The changes:
via pre-commit environment with `breeze compile-www-assets``
command
(they are also based on pre-commit environment)
docker airflow building.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in newsfragments.