How to add other cameras perspective in one robot

[ClorisUcas]

Thanks for grate work!
I 'm trying to add another camera pespective in the robot by adding another "VisionSensor" in the configuration, but it seems like only support one camera in the robot. Besides, I use "enternal_sensor" to add a camera, but I only get a world view and cannot bind with the robot.
Hope for your useful solutions !!!

[hang-yin]

Hi @ClorisUcas , this is a great question! For example, here's how you can add a wrist camera to Fetch:

  external_sensors:
    - sensor_type: VisionSensor
      name: wrist_camera
      relative_prim_path: /controllable__fetch__robot0/wrist_roll_link/wrist_camera
      modalities: [rgb, depth]
      sensor_kwargs:
        image_height: 128
        image_width: 128
      position: [0, 0, 0.2]
      orientation: [0.707, 0.0, 0.0, 0.707]
      pose_frame: parent

You can specify how you want to position the camera in relation to the link on the robot with the position and orientation args. Please let me know if this works for you!

[ClorisUcas]

Thanks for your helpful response, but there still exist some problem.

I cannot see the change of the viewport2 with robot moving.
By the way, I also want to know how to set the fisheye camera, and how can i get the position and orientation of the camera.
Looking forward to your reply~

[hang-yin]

Hi @ClorisUcas ! Unfortunately, to make this fully functional, we are going to need a little bit of infra change. I'll work on this tomorrow and hopefully get it done before the weekend. I'll let you know, but once that is done, you can use our develop branch - og-develop for your purposes. To get the pose of the camera, you can do something like this:

from omnigibson.sensors import VisionSensor

robot = env.robots[0]
vision_sensors = [s for s in robot.sensors.values() if isinstance(s, VisionSensor)]
camera = vision_sensors[0]

pos, ori = camera.get_position_orientation()

[hang-yin]

Hello @ClorisUcas , I talked to my team this morning, and it turns out that the correct way of adding a wrist camera is by creating a custom robot. I just tried it out myself, and it's super easy! Let me walk you through what I did:

1. Take a look at this tutorial: https://behavior.stanford.edu/omnigibson/tutorials/custom_robot_import.html. We can skip this Convert from URDF to USD step because we already have the USD of all the robots in OmniGibson, we just need to modify them.
2. Go to Process USD Model step 1 and import the USD file we just found.
3. Go to Process USD Model step 5 and add a camera to the link you want, e.g. the gripper link on Fetch.
4. Go to Process USD Model step 6 to save this USD as a separate file! Let's call it fetch_with_wrist_camera.usd
5. Now, we want to make a very small modification to our source code. In this omnigibson/robots/fetch.py file, look for the usd_path property and change it to something like:

    @property
    def usd_path(self):
        return os.path.join("{wherever you saved your custom usd to}/fetch_with_wrist_camera.usd")

Note that you will need to install from source to edit this file.

For your reference, here's an example usd file I got from adding a camera to the gripper link: https://drive.google.com/file/d/1WZJwMkfuHsbDr794LNXcCftj-ovi9dGc/view?usp=sharing

[ClorisUcas]

This solution helps me a lot ^ ^. Thanks a lot for your support!!