Web page not rendering isaac gym

[OhhTuRnz]

Hi there! I previously worked with this code using ngrok and was able to use the Meta Quest 3S correctly. However, right now I've changed to adb and, while I can connect to the web server, I can't see the Isaac Gym simulation in the web page.

I am also testing it in my localhost:8012 local web page and I experience the same issue. Additionally, the web page raises a warning that says that the maximum reconnect attempts were exceeded. Attaching a screenshot.

[OhhTuRnz]

Solved the issue, I was using Meta Quest 3S and wanted to try adb without ngrok because of the free trial. It assumes that it will use ngrok even if it skips the https or https part, here are some minor changes I did to make it run:

Modified TeleVision.py to support non-grok connection.

def __init__(self, img_shape, shm_name, queue, toggle_streaming, stream_mode="image", cert_file="./cert.pem", key_file="./key.pem", ngrok=False):
   \# ... Previous code

    # Only create SSL context if not in ngrok mode
    ssl_context = None
    if not ngrok:
        try:
            ssl_context = ssl.SSLContext()
            ssl_context.load_cert_chain(cert_file, key_file)
        except Exception as e:
            print(f"SSL setup failed: {e}, falling back to HTTP")

    self.webrtc_process = Process(
        target=web.run_app, 
        args=(app,), 
        kwargs={
            "host": "0.0.0.0", 
            "port": 8080, 
            "ssl_context": ssl_context
        }
    )
   # Previous code

Modified teleop_hand.py to check for websocked losses and recapture the VUER stream.

def step(self):
  try:
      head_mat, left_wrist_mat, right_wrist_mat, left_hand_mat, right_hand_mat = self.processor.process(self.tv)
  except AssertionError as e:
      if "Websocket session is missing" in str(e):
          print("WebSocket connection lost, waiting for reconnection...")
          time.sleep(1)
          return None, None, None, None, None
      raise e`

[atharvamete]

Hey @OhhTuRnz
I'm facing similar issue but was not able to resolve it after modifying teleop_hand.py
I did not change TeleVision.py as running with ngrok=False and stream_mode="webrtc" will execute the code that you gave by default under if stream_mode="webrtc":
Am I missing something? Could you please guide me more on how to resolve this? thanks

[OhhTuRnz]

Hey @atharvamete

The "hotfix" only works for image mode where it just renders the Isaac Gym environment through shared memory to a VUER webXR webpage that allows you to interact with Isaac Gym only using the hands. This is what the code calls stream_mode="image".

However, when running the code in the stream_mode="webrtc" it tries to connect to a server where a zed camera is streamed through that specific webpage. I am right now striving with the code and finding how to do this, because I am not sure if the VUE WebRTC component requires HTTPS, and my hotfix does not work with HTTPS because the page is not loading (when using ngrok it did so maybe I'm not getting something about the forwarding).

[OhhTuRnz]

Anyways I need to still work on that so I will reopen

[atharvamete]

@OhhTuRnz Thanks for responding, however I'm still lost with making hotfix work.
If not using webrtc then webrtc_process.start() won't ever be called, additionally in your fix with args=(app,), did you mean self.app here or app=web.Application()?
Could you please clarify or maybe put your TeleVision.py here so that I can make hotfix work? thanks again.

[OhhTuRnz]

Yeah, sorry I'm still finding out how some things work.

The main change is to use the server without the certificates because the fallback does not work very well. So just go to ngrok=True (they set it up as false constantly prolly cause their research was done on vision pro) or change the if statement to check if not ngrok and not use the certificates.

The RTC server works if you change the IP because they fixed the IP as 192.168.8.102 which is their local IP, changing it to localhost when using HTTP seems to work but I can only see a blank square. Anyway, I think that is because the RTC server they create uses the ZED camera and I am not using that one. Will try to adapt it and see if everything can be ran in localhost.

[OhhTuRnz]

Yeah if you set ngrok as true it will be streamed in http://127.0.0.1:8012 (works with localhost:8012) and if you run webrtc it should display a blank square because the ZED camera is obviously missing. WebRTC runs smoother as stated in #36 so I will try to stream the shared buffer through WebRTC.

[OhhTuRnz]

If you want to open in metaquest just follow #12; install adb and reverse the port tcp:8012 tcp:8012 while allowing connection from your quest