Add Makefile caching helpers for lazy, expensive evaluations #35626
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
staticfloat
added
the
building
staticfloat
force-pushed
the
sf/makefile_caching
branch
from
aac2259
to
5d32089
Compare
Makefiles natively offer two variable expansion modes: immediate and deferred. When expanding variables that require invocations of external programs (such as `llvm-config`) immediate expansion is almost always preferred, as it will run the external command once, exactly when the makefile variable is defined. Deferred mode, on the other hand, will expand the variable every time it is used, running the external program again and again. When the external program is expensive, this cost can slow down the build significantly, however when the external program requires some setup (for instance, when it itself is downloaded through other rules in the Makefile) it cannot always be immediately expanded. To address this, we build a caching layer that allows for deferred expansion, but once it has been expanded once, the variable is replaced with the result of running the command, and further hits to the same variable will return the cached value. (With the slight caveat that an empty result will cause the external command to be run again in the future). As an example usecase, this commit converts our relative path calculation to use a python script and showcases how to cache this operation. This will be used for further JLL stdlib work where the invocation is much more expensive.
staticfloat
force-pushed
the
sf/makefile_caching
branch
from
e8a56a8
to
ad907d5
Compare
KristofferC
reviewed
Nov 20, 2024
| libdir_rel := $(shell $(JULIAHOME)/contrib/relative_path.sh $(bindir) $(libdir)) | ||
| build_private_libdir_rel := $(shell $(JULIAHOME)/contrib/relative_path.sh $(build_bindir) $(build_private_libdir)) | ||
| private_libdir_rel := $(shell $(JULIAHOME)/contrib/relative_path.sh $(bindir) $(private_libdir)) | ||
| datarootdir_rel := $(shell $(JULIAHOME)/contrib/relative_path.sh $(bindir) $(datarootdir)) |
There was a problem hiding this comment.
This is lame imo. You can't grep for where this variable is defined anymore.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Makefiles natively offer two variable expansion modes: immediate and
deferred. When expanding variables that require invocations of external
programs (such as
llvm-config) immediate expansion is almost alwayspreferred, as it will run the external command once, exactly when the
makefile variable is defined. Deferred mode, on the other hand, will
expand the variable every time it is used, running the external program
again and again.
When the external program is expensive, this cost can slow down the
build significantly, however when the external program requires some
setup (for instance, when it itself is downloaded through other rules in
the Makefile) it cannot always be immediately expanded. To address
this, we build a caching layer that allows for deferred expansion, but
once it has been expanded once, the variable is replaced with the result
of running the command, and further hits to the same variable will
return the cached value. (With the slight caveat that an empty result
will cause the external command to be run again in the future).
As an example usecase, this commit converts our relative path
calculation to use a python script and showcases how to cache this
operation. This will be used for further JLL stdlib work where the
invocation is much more expensive.