Add experimental dashboard to monitor simulations #6137

nilsvu · 2024-06-29T21:10:20Z

Proposed changes

This takes the status data and displays it in a web browser for all running jobs on the cluster, along with lots of plots to show the current state of BBH simulations. It's experimental, in that to use it you have to pip install streamlit by yourself and run bin/python-spectre -m streamlit run tools/Status/Dashboard.py to start the dashboard. There are some instructions in the file.

Here's a glance at what it looks like:

I found this so useful when monitoring BBH simulations that I decided to open this experimental PR.

Upgrade instructions

Code review checklist

The code is documented and the documentation renders correctly. Run
make doc to generate the documentation locally into BUILD_DIR/docs/html.
Then open index.html.
The code follows the stylistic and code quality guidelines listed in the
code review guide.
The PR lists upgrade instructions and is labeled bugfix or
new feature if appropriate.

Further comments

knelli2

These are all optional really. Just some things I noticed. We can't test any of this so once you feel it's ready we can just merge

knelli2 · 2024-07-02T21:53:04Z

tools/Status/Dashboard.py

+            with open(outfile, "r") as open_outfile:
+                st.code("".join(["...\n"] + open_outfile.readlines()[-30:]))


Any way to make this scrollable?

I tried loading the full file but that broke the server :D Added a comment

knelli2 · 2024-07-02T21:55:19Z

tools/Status/Dashboard.py

+# Refresh the page regularly
+if st.toggle("Auto-refresh", True):


I found that if there is an error rending one of the plots of an executable status, then the auto-refresh is somehow disabled. Clicking on a different run brings it back. Any ideas on what to do for this?

Ah ok I moved the auto refresh further up in the script, that should fix it

knelli2 · 2024-07-02T21:56:45Z

tools/Status/ExecutableStatus/ExecutableStatus.py

+          job: A dictionary of job information, including the input file and
+            working directory. See the 'spectre.tools.Status.status' function


fetch_status or render_status to be explicit?

knelli2 · 2024-07-02T22:00:10Z

tools/Status/ExecutableStatus/EvolveGhBinaryBlackHole.py

+                                "L2Norm(PointwiseL2Norm(GaugeConstraint))",
+                                "L2Norm(PointwiseL2Norm(ThreeIndexConstraint))",
+                                "L2Norm(PointwiseL2Norm(FourIndexConstraint))",


Maybe use

[col for col in df.columns if "Constraint" in col]

so that it will work if people stray from the pipeline a bit?

knelli2 · 2024-07-02T22:03:19Z

tools/Status/ExecutableStatus/EvolveGhBinaryBlackHole.py

+        # Eccentricity
+        # (can be enabled once EccentricityControl is available)
+        # st.subheader("Eccentricity")


I think you've updated this section since you opened this. Wanna add those changes in?

It's enabled now since #5895 was merged

knelli2 · 2024-07-02T22:04:55Z

tools/Status/ExecutableStatus/ExecutableStatus.py


 class EllipticStatus(ExecutableStatus):


I think you've also added a dashboard for the elliptic one too since you opened this. Wanna add that? I think the only thing is figuring out how to show the latest control step and not just the first one.

I added the elliptic dashboard. It only shows the convergence for the first control step atm, and also plots the convergence of the control parameter. I'll probably combine these somehow, but haven't decided how yet. Also AMR iterations are still missing from the status.

knelli2

Everything else looks ok

knelli2 · 2024-07-08T18:57:27Z

support/Pipelines/EccentricityControl/EccentricityControl.py

    # Compute separation
-    num_obs = len(ObjectA_centers[:, 0])
-
-    if len(ObjectB_centers[:, 0]) < num_obs:
-        num_obs = len(ObjectB_centers[:, 0])
+    num_obs = min(len(ObjectA_centers[:, 0]), len(ObjectB_centers[:, 0]))


Maybe this commit and the one before should be in a separate PR since they are a bug fix? That way they don't get buried in an experimental PR

knelli2 · 2024-07-15T18:18:44Z

@nilsvu once the tests are fixed, we can merge this

Make sure isort always treats the `spectre` imports as first party. Before this fix, isort could get confused if the source directory was not named "spectre".

Allows to call the function separately from the CLI.

- Fix issue with trajectories having different lengths - Pass in an MPL figure for plots instead of always writing the plot to a file.

nilsvu · 2024-07-15T21:35:19Z

Ok I squashed and rebased. Thanks for reviewing!

knelli2 reviewed Jul 2, 2024

View reviewed changes

nilsvu force-pushed the dashboard3 branch 3 times, most recently from 249873b to 53f1077 Compare July 3, 2024 22:16

knelli2 requested changes Jul 8, 2024

View reviewed changes

nilsvu and others added 2 commits July 15, 2024 14:15

Fix isort config

2313ead

Make sure isort always treats the `spectre` imports as first party. Before this fix, isort could get confused if the source directory was not named "spectre".

Split some CLIs into a free function

afb2933

Allows to call the function separately from the CLI.

nilsvu force-pushed the dashboard3 branch 2 times, most recently from 3876903 to 09402e2 Compare July 15, 2024 21:24

nilsvu and others added 6 commits July 15, 2024 14:31

PlotTrajectories: load time column

4039708

Minor updates to EccentricityControl.py

00935eb

- Fix issue with trajectories having different lengths - Pass in an MPL figure for plots instead of always writing the plot to a file.

Add experimental dashboard to monitor simulations

ec23ab0

Add dashboard for BBH

2261c08

Add dashboard for elliptic executables

c6c8f78

Add CCE executable status and dashboard

f49eeb9

nilsvu force-pushed the dashboard3 branch from 09402e2 to f49eeb9 Compare July 15, 2024 21:31

knelli2 approved these changes Jul 15, 2024

View reviewed changes

knelli2 merged commit 7ae00e7 into sxs-collaboration:develop Jul 15, 2024
22 checks passed

knelli2 added the cli/pybindings Command line interface & Python bindings label Aug 3, 2024

nilsvu deleted the dashboard3 branch August 3, 2024 22:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add experimental dashboard to monitor simulations #6137

Add experimental dashboard to monitor simulations #6137

nilsvu commented Jun 29, 2024

knelli2 left a comment

knelli2 Jul 2, 2024

nilsvu Jul 2, 2024

knelli2 Jul 2, 2024

nilsvu Jul 2, 2024

knelli2 Jul 2, 2024

knelli2 Jul 2, 2024

knelli2 Jul 2, 2024

nilsvu Jul 3, 2024

knelli2 Jul 2, 2024

nilsvu Jul 3, 2024 •

edited

Loading

knelli2 left a comment

knelli2 Jul 8, 2024

knelli2 commented Jul 15, 2024

nilsvu commented Jul 15, 2024

		with open(outfile, "r") as open_outfile:
		st.code("".join(["...\n"] + open_outfile.readlines()[-30:]))

		# Refresh the page regularly
		if st.toggle("Auto-refresh", True):

		job: A dictionary of job information, including the input file and
		working directory. See the 'spectre.tools.Status.status' function

Add experimental dashboard to monitor simulations #6137

Add experimental dashboard to monitor simulations #6137

Conversation

nilsvu commented Jun 29, 2024

Proposed changes

Upgrade instructions

Code review checklist

Further comments

knelli2 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nilsvu Jul 3, 2024 • edited Loading

Choose a reason for hiding this comment

knelli2 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

knelli2 commented Jul 15, 2024

nilsvu commented Jul 15, 2024

nilsvu Jul 3, 2024 •

edited

Loading