Using the built-in python viewer as a non-interactive synchronized viewer?

[breakds]

Hi,

I really like the new viewer in the python bindings that wraps simulate.cc. If my understanding is correct, it is currently serving as an interactive viewer for the users to interact with the models.

I am wondering whether I can use it for another use case. I would like to visualize how a neural network plays with the robot model. In this case, the viewer is in a "view only" mode, and it updates the visualization as the MjModel and MjData step forward with the actions produced by the neural network. As a plus, if I can still use the mouse to change the viewpoint in the viewer it would be great.

Currently when I call viewer.launch(model, data) it will block. If it is started in another process, I am not sure what is the recommended approach to send the update on the MjData to that process.

Thanks!

[yuvaltassa]

I believe you are asking for the passive viewer mode described in the 3rd subsection here.

[breakds]

Thanks for the prompt answer! Actually I tried the 3rd subsection approach, which is viewer.launch_repl. It will start the viewer, but still blocking the original process, and it also shows a python REPL prompts. Should I do this in a different process?

[breakds]

If I understand the code logic here for launch_repl correctly, it still blocks the main process by waiting for the repl thread to join. The is is undesired in the use case I am looking at because the main process need to be used for the actual interaction between Mujoco and the neural network.

[yuvaltassa]

I see, you want a non-blocking passive viewer, correct?

I'm not sure you can do that in a single process in Python, but I might be wrong. @saran-t, is there a recommended MO for this use case?

[breakds]

I see, you want a non-blocking passive viewer, correct?

Yes exactly. Sorry I did not make it clear enough.

[breakds]

I have also tried to put viewer.launch in a separate thread:

def _run_viewer(model, data):
    viewer.launch(model, data, run_physics_thread=False)

self._viewer_thread = threading.Thread(target=_run_viewer, args=(self._mj_model, self._mj_data))
self._viewer_thread.start()

So that simulate.renderloop() within launch() is caged in the thread. The viewer can start, but it does not render anything.

[cemoke]

In m1 mac using jupyter notebook, even the viewer does not start for me inside the thread

[saran-t]

Running the viewer in a thread is problematic, that's why the REPL mode is implemented the way it is.

For the use case that you mentioned, the best thing to do would be to run your agent loop in a separate thread, but then you'd need to do the timekeeping yourself (you almost definitely don't want to just let the loop run as fast as possible), or install your controller into mjcb_control and let the viewer handle the timing as usual.

[aftersomemath]

Here is an example of using the mjcb_control callback to add a controller to the physics simulation run by the viewer (at first glance this sounds sufficient for what you want to do). You can still inject perturbations with the mouse, pause, reload, etc, and the camera can still be moved. Here is a video.

Here is the viewer's physics thread which you could re-implement so that your code controls every aspect of the physics simulation (and thus you can also add a controller and disable perturbations from the viewer).

[breakds]

Thanks a lot @saran-t and @aftersomemath ! I did not know mjcb_control before - sounds like the way to go.
You are right that I do not want to let the loop run as fast as possible. Ideally I would like it to be synced to my main control thread.

It is not as simple for me to hand over the main process to the viewer in the current architecture. However, I think I can still try to run viewer in a subprocess and update the MjData in the subprocess via the the mjcb_control callback and some IPC.

The information and suggestions are really helpful. I will close this issue.