Depth and infra sequence number alignment with external trigger

[dongdongbh]

I'm using realsense D435i with my external trigger device, I connect my trigger signal to the D435i sync pin 5 as Multi-Camera configurations.

I want to use the timestamp of my trigger signal to get the exact trigger time of D435 depth and infra capture time. I already have my trigger time stamp with trigger sequence number, I set inter_cam_sync_mode=4, which triggers camera capture pre-trigger signal. Then in realsense-ros warper, I cache all video stream with camera sequence number and try to find corresponding trigger sequence number. Then I get the trigger shift and keep tracking the frame and align with my trigger time stamp. I noticed that when I set the trigger signal as 5Hz, the frame rate of d435 is also 5Hz, which suggests the external trigger worked.

But here is the problem, when the first trigger signal send with trig_seq=0, the depth frame will came first with seq[stream] = 1, and then after trig_seq=1, the seq[stream] = 2 depth frame come, then after trig_seq=2 the depth frame seq[stream] = 3 depth frame come, then the infra1 frame with seq[stream] = 1 come, then the infra2 frame with seq[stream] = 1 come. Here is a table to demonstrate it.

trig_seq	c_seq[depth]	c_seq[infra1]	c_seq[infra12]
0	1	-	-
1	2	-	-
2	3	1	1
3	4	2	2

My problem is, does infra frame with seq[stream] = 1 align with depth frame with seq[stream] = 1? If it aligned, why it comes after more than 600ms after trigger signal? If not, how can I determinate which trig_seq number corresponds which seq[stream]?

I also tested different trigger rate, the seqence of infra shift are also different. I tested if abs(timeshift between camera and trigger) < 1.0 / frame_rate * 0.75, then I view the seqence is aligned, and record the sequence shift for each channel (depth, infra1, infra2), then replace the frame topic timestamp with my trigger time trig_stamp + exposure_time /2. I try it with my VIO algorithm, it works really bad, even worse than the orignal unsynchronized time stamp.

Test environment

Realsense: D435i
Platform: Intel NUC8i7
System: Ubuntu 20.04
ROS: ROS1 noetic
Realsense-ros: v2.3.2
LibRealSense: v2.50.0
Matadata available with patched kernal

@MartyG-RealSense @doronhi

[MartyG-RealSense]

Hi @dongdongbh The External Synchronization (genlock) hardware sync system only timestamp-syncs depth to a trigger in mode 4+ and not infrared. The same is true for the original hardware sync system (modes 1 and 2).

[dongdongbh]

But the frequency of infrared frames also changed with external trigger. If it not triggers the infrared, where is the depth from? I think the depth frame is calculated from two infrared frame. If mode 1, 2, and 4+ only trigger for depth? Does it means the 3 is different? What is 3 for? Is it triggers infrared?

[MartyG-RealSense]

Mode 3 is called Full Slave. Whilst Mode 2 is 'Slave' and only syncs depth, Mode 3 syncs depth and RGB (but not infrared) but it only works with the D415 camera model.

The depth frame is calculated from raw left and right infrared frames in the camera hardware, which are not related to the Infrared and Infrared 2 streams. This is why depth can still be produced even if Infrared and Infrared 2 are not enabled.

[dongdongbh]

So there is no way to hardware trigger infrared frames? Is there a way to get the raw infrared frames timestamps?

The infrared frames frequency also changed with external trigger, so I think they are indeed related to the trigger signal. But the timestamp are different, it's so weird. Or maybe the depth frame and infrared frame with same sequence number are actually produced by same trigger signal with timestamp of depth frame?

[MartyG-RealSense]

The genlock hardware sync system (Mode 4+) synchronizes the depth timestamps of a 'slave' camera with an external trigger. The genlock system's documentation makes no mention of infrared.

A RealSense user at IntelRealSense/librealsense#6195 obtained raw hardware timestamps through camera metadata using frame.get_frame_metadata(RS2_FRAME_METADATA_SENSOR_TIMESTAMP)

[dongdongbh]

I know I can get the raw timestamp of infrared, but it is based on D435's clock. I'm using external trigger, I want to know how the timestamp align with my external trigger, then I can use my external clock. Because the external trigger does has impact on infrared frequency, the infrared frames has same frequency as the trigger signal.

Can you double check with the developer what is actual relationship between the sequence number between depth frame, external trigger and their timestamps. I'm sure they are clearly related, I just want to know what is going on between them.

[MartyG-RealSense]

Support for the genlock sync system has ceased unfortunately and its online documentation has been withdrawn. This is because it was an experimental proof of concept rather than an official, validated Intel feature (modes 1 and 2 are the only ones that are officially supported and validated by Intel). So developer assistance for this external trigger system is now regretfully unavailable.

[dongdongbh]

But the mode 2 has the same problem and is supported

[MartyG-RealSense]

Using an external trigger with mode 2 is possible but significantly more difficult than with genlock because the frequency of the trigger has to be matched extremely closely with the 'actual' FPS of the slave camera (e.g 30.02 FPS instead of 30 FPS). Usually with mode 2, instead of receiving an external trigger the slave is connected by sync cable to a 'master' RealSense camera that outputs a trigger pulse from its sync pin when it is set to mode 1.

As far as mode 1 and 2 hardware sync is concerned, the behaviour of infrared during sync is not considered at all. It relates to synching the slave's depth timestamps with the master's depth timestamps, or with an external trigger pulse if the pulse is able to be tuned finely enough to match with the actual FPS of the slave camera.

[dongdongbh]

How about mode 3, seems mode 3 does not need to be triggered with exact frequency from #2323.

[MartyG-RealSense]

Mode 3 does not require an exactly tuned frequency because it is part of the genlock sync system rather than the original sync system where synchronizng an external trigger to a camera was much more difficult. As mentioned earlier, the RGB sync will not work with non-D415 camera models though.

[dongdongbh]

The behaviour of infrared during sync is not considered at all.

I think the behaviour of infrared should be clarified, because the sync indeed has impact on infrared frames.

[MartyG-RealSense]

I will submit a query to my Intel RealSense colleagues about the influence that sync has on infrared.

[dongdongbh]

Great! Thanks for your help.

[dongdongbh]

After debugging the timestamps, in trigger mode, I found that it seems the RealSense camera first publishes two depth frames with first two trigger signals, then publishes the third depth frame and the first infrared frames together with the same RealSense timestamp. So I guess there are two depth frames generated first, then the following depth and infraed frames are all triggered by subsequent trigger signals.

trig_seq	c_seq[depth]	c_seq[infra1]	c_seq[infra12]	c_time[depth]	c_time[infra12]
0	1	-	-	c_1	-
1	2	-	-	c_2	-
2	3	1	1	c_3	c_3
3	4	2	2	c_4	c_4

I hope that the RealSense team can help confirm this.

[MartyG-RealSense]

I have passed on your latest question to my colleagues. Thank you.

[MartyG-RealSense]

I have discussed your case with my Intel RealSense colleagues. Thanks very much for your patience.

They advised that infrared and depth frames are always synchronized even when hardware sync is active. This is because the depth is derived from infrared frames and so there is no way for them to be out of sync with each other.

[dongdongbh]

How about the sequence number corespondence? Did they really test the trigger mode? On my test, it first generates two depth frames, then third depth frame seems sync with the first infrared frames, and they start show same timestamp. I think the third depth frame corespond the first infrared frames. I want confirm the this. If the first depth frame coresponds the first infrared frame. Then the first infrared frame comes later after two depth frames. In 5 Hz case, it comes around 600ms later than the first depth frame.

[MartyG-RealSense]

I have highlighted your latest query above to my colleagues. Thanks again for your patience.

[dongdongbh]

Any update on this thread?

[MartyG-RealSense]

I will follow up with my Intel RealSense colleagues about your question.

[MartyG-RealSense]

I have received further advice from my colleagues about your question.

---

The frame timestamp on the depth and IR images should be in sync. Frame counter / number is not an ideal way to sync between frames or cameras. The RealSense Viewer can reproduce and show the diff / delta in frame number between depth and IR, so the behavior should be expected. Please see the screenshot below.

[dongdongbh]

Thanks for your update, I want to know if infra_frame_num = depth_frame_num - 2 always hold.

[MartyG-RealSense]

If there is consistently a difference of 2 between depth and infrared then it is reasonable to use that in your project as a predictable rule for frame-count behaviour. This rule does not necessarily apply to every RealSense user's project though, since - as my colleagues advised - the frame number will not be an ideal way to check for sync.

[dongdongbh]

OK, I got it. Because there seeems no other simple way to check for sync, I just use the frame number method for now.

[MartyG-RealSense]

Thanks very much @dongdongbh

[dongdongbh]

My previous issue #2592 said that I want to use these mode to do synchronization with external trigger signal with emitter on-off.

Here I want to report the pattern of depth and infra sequence number alignment in genlock 259 and 260 for whom also interest in these modes.

According my test, in genlock 259 and 260, sequence number aligns as following

Test configuration

Realsense: D435i
System: Ubuntu 20.04
ROS: ROS1 noetic
Realsense-ros: v2.3.2
LibRealSense: v2.50.0

External trigger frequency: 4Hz

<arg name="depth_fps"           default="90"/>
<arg name="infra_fps"           default="90"/>

/camera/stereo_module/emitter_enabled: 1
/camera/stereo_module/global_time_enabled: true

The emitter state get from (int)f.get_frame_metadata(RS2_FRAME_METADATA_FRAME_LASER_POWER_MODE);

Genlock 259 emitter ON_OFF mode

trig_seq	emitter state	c_seq[depth]	c_seq[infra12]	c_time[depth]	c_time[infra12]	dt = c_n -c_{n-1} (ms)
0	1	1		c_1
	1	2		c_2		83.7
	0	3	1(with emmitter on)	c_3	c_3	11.1
1	1	4	2	c_4	c_4	238.9
	0	5	3	c_5	c_5	11.1
2	1	6	4	c_6	c_6	238.9
	0	7	5	c_7	c_7	11.1
3	1	8	6	c_8	c_8	238.9
	0	9	7	c_9	C-9	11.1

Regarding Genlock 259, it is noteworthy that the first infra frame has an 'emitter on' state, which appears unusual. However, the interval between the triggerd frame pair (emitter on and emitter off) is 11.1ms, conforming to the native frame rate of 90 Hz. It is likely that this interval was set according to the native frame rate. Since the trigger frequency is 4Hz, the trigger interval is 250ms, which is equivalent to the sum of 238.9ms and 11.1ms.

Genlock 260 emitter OFF_ON mode

trig_seq	emitter state	c_seq[depth]	c_seq[infra12]	c_time[depth]	c_time[infra12]	dt = c_n -c_{n-1} (ms)
0	0	1		c_1
	0	2		c_2		56.3
	1	3	1	c_3	c_3	11.1
1	0	4	2	c_4	c_4	238.9
	1	5	3	c_5	c_5	11.1
2	0	6	4	c_6	c_6	238.9
	1	7	5	c_7	c_7	11.1
3	0	8	6	c_8	c_8	238.9
	1	9	7	c_9	C-9	11.1

The results indicate that the frames triggered by the first signal is not stable. However, the system starts working as expected from the second signal onwards.

[MartyG-RealSense]

Thanks so much @dongdongbh for sharing such detailed feedback!