Memray memory profiler support for mrun command line tool #794
Conversation
… for memory profiler
… memory profiling
…atform specific temp directory
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## main #794 +/- ##
==========================================
+ Coverage 88.12% 88.20% +0.08%
==========================================
Files 44 44
Lines 3562 3587 +25
==========================================
+ Hits 3139 3164 +25
Misses 423 423
☔ View full report in Codecov by Sentry. |
|
This looks useful, thanks @tsmathis! I just have a few comments:
|
|
Thanks for the comments, @rkingsbury! I will address your points before reopening! |
|
I second what @rkingsbury has suggested. Other than that I’m good with merging. |
…quiremnets.txt and setup.py
…-default output dir for memray
…pecify output directory for memray
|
|
|
||
| ## Profiling Memory Usage of Builders | ||
|
|
||
| `mrun` can optionally profile the memory usage of a running builder by using the Memray Python memory profiling tool ([Memray](https://github.com/bloomberg/memray)). To get started, `maggma` will first need to be installed from source ([Maggma installation](https://materialsproject.github.io/maggma/#installation-from-source)) followed by `pip` installing Memray using `pip install memray`, or by installing the optional `maggma` requirements by using `pip install requirements-optional.txt` in the `maggma` base directory. |
There was a problem hiding this comment.
Why is it necessary to install maggma from source in order for this to work? Is it not possible to pip install maggma and then install the optional memray?
There was a problem hiding this comment.
Ah you are right, I will correct that
|
Thanks for addressing comments @tsmathis , this looks great! I just have one small question about the installation requirements; otherwise I think this is ready to merge. |
… doc in favor of just pip installing maggma plus memray
|
Thanks @tsmathis ! |
This PR adds an additional optional flag to
mrun:--memray, or alternatively-m, to enable memory profiling of a builder using the Memray Python memory profiling tool (Memray repo).The profiler supports profiling of both single and forked processes. For example, spawning multiple processes in
mrunwith-nwill signal the profiler to track any forked child process spawned from the parent process.A basic invocation from a terminal would look something like
mrun --memray on builder_dump_file.json.--memrayis a Click boolean parameter (Click) and thus accepts the values1,yes,y,on, andtrue, which convert toTrue, and0,no,n,off, andfalse, which convert toFalse.The output
.binfile produced by Memray is dumped by default into the user'stempdirectory, which is platform/OS dependent. For Linux/MacOS this will be/tmp/and for Windows the target directory will beC:\TEMP\.The output file will have a generic naming pattern as follows:
BUILDER_NAME_PASSED_TO_MRUN + BUILDER_START_DATETIME_ISO.bin, e.g.,builder_dump.json_2023-06-06T16.14.19.272030.bin.Following completion, the
.binoutput file can be converted into a flamegraph using the Memray CLI, i.e,memray flamegraph builder_dump.bin.Further visualization and data transform examples can be found in Memray's documentation (Memray reporters).