Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

T265 stops sending data #6362

Open
dschnabel opened this issue May 7, 2020 · 67 comments
Open

T265 stops sending data #6362

dschnabel opened this issue May 7, 2020 · 67 comments

Comments

@dschnabel
Copy link

dschnabel commented May 7, 2020

Required Info
Camera Model T265 & D435
Firmware Version T265: 0.2.0.926, D435: 05.12.03.00
Operating System & Version Raspbian 10.3
Kernel Version (Linux Only) 4.19.97-v7l+ armv7l
Platform Raspberry Pi 4
SDK Version 2.33.1
Language C++/ROS
Segment Robot

Issue Description

After upgrading from SDK 2.31.0 to 2.33.1 my T265 stops sending pose data when running for some time (1-5 min). This happens if the T265 is run together with the D435 and also if I run the T265 alone (less frequently).

This wasn't a problem in 2.31.0 but I was experiencing crashes in 2.31.0 so I decided to upgrade to the latest stable version (2.33.1).

In 2.33.1 I see no more crashes but I seem to be experiencing a very similar problem as described in #5509 (comment).

I saw some discussion on missing serial numbers but in my case both cameras seem to get detected properly (see my ROS startup logs for T265 and D435).

When I monitor the odom topic from the T265, after a while I don't get any more pose (odom) data:

$ rostopic hz /odom

average rate: 200.380
	min: 0.000s max: 0.099s std dev: 0.01482s window: 50000
average rate: 200.380
	min: 0.000s max: 0.099s std dev: 0.01482s window: 50000
average rate: 200.380
	min: 0.000s max: 0.099s std dev: 0.01482s window: 50000
average rate: 200.315
	min: 0.000s max: 0.099s std dev: 0.01483s window: 50000
no new messages
no new messages
no new messages
no new messages
no new messages

For completeness here are my two ROS launch files:

Is there any fix for this?

@dschnabel dschnabel changed the title T265 stops sending pose data if run together with D435 T265 stops sending pose data May 7, 2020
@dschnabel
Copy link
Author

@RealSenseCustomerSupport @RealSense-Customer-Engineering @MartyG-RealSense

Can you help us? Is this a known issue and if so, is there a workaround or proposed fix? Any additional information we can collect for you too look at?

This problem renders the T265 unusable for our project.

@dschnabel
Copy link
Author

dschnabel commented May 19, 2020

We did some more investigation and it looks like a firmware or a libusb issue.

This is how the pose/gyro/accel is being received from the T265 in librealsense:

  1. A new initial request is generated.
  2. This request gets forwarded to libusb.
  3. When a response arrives the interrupt_callback gets called.
  4. At the end of the callback function a new request is created.

Steps 2-4 are repeated. As long as the callback function _interrupt_callback gets called we receive data. But for some reason, occasionally the callback doesn't get called anymore, even if the request has been dispatched to libusb correctly without any errors. This is when the T265 stops sending pose/gyro/accel data.

We implemented a watchdog thread which monitors the callback and if the callback is not called anymore after some time we call stop_interrupt() followed by a start_interrupt(). We were hoping this would reinitialize the callback and cause the data to show up again. But this didn't help. Only a relaunch of the ROS node helped.

Unfortunately we cannot go deeper since we don't have the firmware of the T265. But our guess is that the T265 just wouldn't respond anymore.

@RealSenseCustomerSupport @RealSense-Customer-Engineering @MartyG-RealSense could you please see what could cause the callback function not getting called anymore? Are there logs I can enable in the T265 firmware?

@dschnabel
Copy link
Author

We ran the ROS node in debug mode. These are the last few lines before T265 stops sending data:

[DEBUG] [1590375454.268119176]: Publish Pose stream
[DEBUG] [1590375454.272547826]: Frame arrived: stream: Gyro ; index: 0 ; Timestamp Domain: Global Time
[DEBUG] [1590375454.272984024]: Frame arrived: stream: Pose ; index: 0 ; Timestamp Domain: Global Time
[DEBUG] [1590375454.273135615]: Publish Pose stream
[DEBUG] [1590375454.273732108]: Frame arrived: stream: Accel ; index: 0 ; Timestamp Domain: Global Time
[DEBUG] [1590375454.278416105]: Frame arrived: stream: Gyro ; index: 0 ; Timestamp Domain: Global Time
[DEBUG] [1590375454.278897526]: Frame arrived: stream: Pose ; index: 0 ; Timestamp Domain: Global Time
[DEBUG] [1590375454.279104097]: Publish Pose stream
[DEBUG] [1590375454.286298009]: Frame arrived: stream: Gyro ; index: 0 ; Timestamp Domain: Global Time
[DEBUG] [1590375454.286856576]: Frame arrived: stream: Pose ; index: 0 ; Timestamp Domain: Global Time
[DEBUG] [1590375454.287164961]: Publish Pose stream
[DEBUG] [1590375454.287998580]: Frame arrived: stream: Gyro ; index: 0 ; Timestamp Domain: Global Time
[DEBUG] [1590375454.288404798]: Frame arrived: stream: Pose ; index: 0 ; Timestamp Domain: Global Time
[DEBUG] [1590375454.288710257]: Publish Pose stream
[DEBUG] [1590375454.289378008]: Frame arrived: stream: Accel ; index: 0 ; Timestamp Domain: Global Time
[DEBUG] [1590375454.292816706]: Frame arrived: stream: Gyro ; index: 0 ; Timestamp Domain: Global Time

After the last line there's no more output.

@dschnabel dschnabel changed the title T265 stops sending pose data T265 stops sending data May 25, 2020
@csr-kick
Copy link

I have also been seeing this exact issue, with the same setup. RPI4 with same OS, SDK version, with T265 and D435i. Happens with both ROS and simple C++ test scripts.

Data comes in from the T265 (using poll_for_frames, try_wait_for_frames, or wait_for_frames) until suddenly the frames just stop. Relaunching whichever program will get data from the device(s) just fine for another short period of time (but I do call hardware_reset() on the T265 as part of my code). The only data stream I am getting from the T265 is pose via: enable_stream(RS2_STREAM_POSE, RS2_FORMAT_6DOF);

Other Observations:

  • Rpi temperature does not correlate with duration before data stops
  • Neither using different cables nor using a powered USB hub helps
  • With both devices plugged in and streaming data, the T265 does seem to fail sooner than just the T265 alone, but that's not always the case. When T265 fails, the D435i continues to function normally.
  • Oddly, fails faster if you just let the sensor sit in one place, but takes longer if you move the device around. The longest 'uptime' I have seen with the sensor was from moving it a little every 60 or so seconds.
  • USB 2.0 or 3.x makes no difference

Personally have not had any success with pipe.stop() and pipe.start(). I tried to see if there was a usb disconnection or such that would account for this issue, but the log file attached does not show any events to support that. It really feels like there is something that crashes or hangs on the T265.

This problem is currently a showstopper for me, which is unfortunate since during the windows when it works the sensor is pretty great. I might try rolling back to 2.31, but based on what I read here and the release notes that may just be trading one set of problems for another. Please help.

udev log.txt

@csr-kick
Copy link

csr-kick commented May 31, 2020

I switched to the development branch (2.35). The D435i is not recognized at all, but the T265 seemed to work better, but still crashes. The D435i shows up properly as a usb device:

Bus 002 Device 004: ID 8086:0b3a Intel Corp.
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 012: ID 8087:0b37 Intel Corp.
Bus 001 Device 004: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

but the SDK refuses to detect it. The rs-enumerate-devices example does not see it, nor do tools like the fw-logger/fw-update.

As a side note, both sensors have always worked when plugged into a windows computer.

@csr-kick
Copy link

csr-kick commented Jun 2, 2020

Using latest development branch at commit 2a79d31 merged by @ev-mp

Compiled and ran the rs-pose example with the line rs2::log_to_console(RS2_LOG_SEVERITY_INFO);

Attached is the log, failure after about 60 seconds of running. The message received after failure is:

02/06 01:04:05,676 INFO [2949632528] (tm-device.cpp:1408) T265 FW message: 13042424847758289392: [0x�/2:721] Host not reading - stopping

Pose_Info_log_output.txt

Anyone have any ideas?

edit: Using RS2_LOG_SEVERITY_DEBUG we get this after the data stops:

02/06 01:23:56,838 DEBUG [2941239824] (tm-device.cpp:1473) T265 time synced, host_ns: 1591055918197446912
 02/06 01:23:56,848 DEBUG [2949632528] (tm-device.cpp:2031) Sending message DEV_GET_AND_CLEAR_EVENT_LOG length 6
 02/06 01:23:56,849 DEBUG [2949632528] (tm-device.cpp:2047) Receiving message with max_response_size 32776
 02/06 01:23:56,849 DEBUG [2949632528] (tm-device.cpp:2062) Received DEV_GET_AND_CLEAR_EVENT_LOG with length 8
 02/06 01:23:56,949 DEBUG [2949632528] (tm-device.cpp:2031) Sending message DEV_GET_AND_CLEAR_EVENT_LOG length 6
 02/06 01:23:56,949 DEBUG [2949632528] (tm-device.cpp:2047) Receiving message with max_response_size 32776
 02/06 01:23:56,950 DEBUG [2949632528] (tm-device.cpp:2062) Received DEV_GET_AND_CLEAR_EVENT_LOG with length 8
 02/06 01:23:57,050 DEBUG [2949632528] (tm-device.cpp:2031) Sending message DEV_GET_AND_CLEAR_EVENT_LOG length 6
 02/06 01:23:57,050 DEBUG [2949632528] (tm-device.cpp:2047) Receiving message with max_response_size 32776
 02/06 01:23:57,050 DEBUG [2949632528] (tm-device.cpp:2062) Received DEV_GET_AND_CLEAR_EVENT_LOG with length 8
 02/06 01:23:57,151 DEBUG [2949632528] (tm-device.cpp:2031) Sending message DEV_GET_AND_CLEAR_EVENT_LOG length 6
 02/06 01:23:57,151 DEBUG [2949632528] (tm-device.cpp:2047) Receiving message with max_response_size 32776
 02/06 01:23:57,151 DEBUG [2949632528] (tm-device.cpp:2062) Received DEV_GET_AND_CLEAR_EVENT_LOG with length 8
 02/06 01:23:57,251 DEBUG [2949632528] (tm-device.cpp:2031) Sending message DEV_GET_AND_CLEAR_EVENT_LOG length 6
 02/06 01:23:57,252 DEBUG [2949632528] (tm-device.cpp:2047) Receiving message with max_response_size 32776
 02/06 01:23:57,252 DEBUG [2949632528] (tm-device.cpp:2062) Received DEV_GET_AND_CLEAR_EVENT_LOG with length 8
 02/06 01:23:57,339 DEBUG [2941239824] (tm-device.cpp:2031) Sending message DEV_GET_TIME length 6
 02/06 01:23:57,339 DEBUG [2941239824] (tm-device.cpp:2047) Receiving message with max_response_size 16
 02/06 01:23:57,339 DEBUG [2941239824] (tm-device.cpp:2062) Received DEV_GET_TIME with length 16
 02/06 01:23:57,340 DEBUG [2941239824] (tm-device.cpp:1473) T265 time synced, host_ns: 1591055918197446912

Also, probably not important but when running at this debug level properly we get a flood of:
02/06 01:16:48,408 DEBUG [2958025232] (frame-archive.h:155) Frame Callback [Pose#1563] overdue. (Duration: 5.015381ms, FPS: 200, Max Duration: 4ms)

Max duration is calculated as 1000 / (frame->get_stream()->get_framerate() + 1);
Is this correct? Feels like it should be more like (1000.f / (frame->get_stream()->get_framerate()) + 1 to get max of 6ms rather than 4ms on a message that comes in at an average of 5ms.

There is also this:

CallbackFinished,Pose,0,DispatchedAt,1591070836901.750732
 02/06 05:07:16,901 DEBUG [3070188800] (frame-archive.h:155) Frame Callback [Pose#0] overdue. (Duration: 1591070836901.750732ms, FPS: 200, Max Duration: 4ms)

Pose #0 is always overdue, every single cycle, with duration being equal to the DispatchedAt time.

@dschnabel
Copy link
Author

Glad to hear I'm not the only one with this problem, @csr-kick. When you tried with the latest dev branch and you got the Host not reading - stopping error, did you have the D435 run in parallel or only the T265?

And is it still true that a pipe.stop()/pipe.start() won't fix it once you get into that state?

@csr-kick
Copy link

csr-kick commented Jun 2, 2020

I can replicate the failure without the D435i plugged in, but it does seem to make it happen faster when it is running. The data I am collecting now is as bare as possible, running the rs-pose example with only the T265 plugged in at all to the Rpi4.

edit:

When I do variations on pipe.stop() and pipe.start() I am not getting any success.

Device Position: 0.142 -0.010 0.104 (meters)
Device Position: 0.142 -0.010 0.104 (meters) 
02/06 01:59:31,648 INFO [2949632528] (tm-device.cpp:1408) T265 FW message: 13042428173785044114: [0x�/2:721] Host not reading - stopping
!*****restarting pipe*****!
 02/06 01:59:32,885 INFO [3070188800] (tm-info.cpp:53) Picked 1/4 devices
 02/06 01:59:32,899 ERROR [3070188800] (handle-libusb.h:95) failed to claim usb interface: 0, error: RS2_USB_STATUS_BUSY
 02/06 01:59:32,899 ERROR [3070188800] (types.h:307) Unable to open device interface

@csr-kick
Copy link

csr-kick commented Jun 2, 2020

@MartyG-RealSense @dorodnic Any ideas/help on this issue? 'Host not responding' error makes me think the SDK stops processing messages or maybe requesting the message in the first place, so the sensor stops as well.

@dschnabel
Copy link
Author

dschnabel commented Jun 2, 2020

@csr-kick I was just thinking, what if you create a new pipe after stopping the old one? This should release any resource that the old pipe holds:

pipe->stop();
pipe = std::make_shared<rs2::pipeline>();
pipe->start();

I'm not home right now so I can't test this myself.


Edit:
Ok so I tested this and it's working! I can recreate the pipe and am getting data again from the T265. This is not ideal but it's a workaround I guess until the root cause has been fixed.

Here's the code that worked for me

rs2::config cfg;
cfg.enable_stream(RS2_STREAM_POSE, RS2_FORMAT_6DOF);
auto pipe = std::make_shared<rs2::pipeline>();
pipe->start(cfg);

while (!ros::isShuttingDown()) {
    try {
        auto frames = pipe->wait_for_frames();
        // do something with the frames
    } catch (const rs2::error & e) {
        std::cout << "T265 crashed. Restarting ..." << std::endl;
        pipe->stop();
        pipe = std::make_shared<rs2::pipeline>();
        pipe->start(cfg);
        std::cout << "... restarted." << std::endl;
    }
}

@csr-kick can you give this a try?

@csr-kick
Copy link

csr-kick commented Jun 3, 2020

Using your code snippet does recover, but I want to try it a few times with the D435i also running. However, this isnt a viable solution beyond basic prototyping.

  1. It takes 15 seconds for that error to be thrown when using pipe.wait_for_frames. Maybe one of the other frame wait methods will be able to fail faster, but any real delay will be a problem for a mobile robot.
  2. When it resets, your back at 0,0,0 meaning you would maybe have to track you actual last pose and then offset after each reset?
  3. How will exporting and importing the localization maps play with this? If you are far from the origin will they actually be able to re-localize? Do you then drop your manual offset after?

It really seems like somewhere in the SDK it stops processing the pose data coming from the sensor. The sensor sees the messages backing up and it stops sending to avoid any issues. The bulk data transfers and time syncing continue to work, which tells me that it is specific to whatever protocol the pose message uses. We also know that none of this happens if you plug into a windows computer and run the realsense viewer app, so its probably low in the code stack.

@dschnabel
Copy link
Author

I agree that this is not a fix and ultimately we need something better than restarting the pipe. I think the workaround can work fairly well though if configured properly but we need to do some more testing.

To your questions:

Re 1: pipe.wait_for_frames() takes a timeout in milliseconds as an optional parameter (see here). So you can override the default 15s timeout with whatever makes sense to you.

Re 2: I might be wrong on this but the way I understand is when you reset the pipe, you are only reconnecting to the T265, you're not hard resetting the camera. In order to hard reset the T265 you'd have to call hardware_reset(). That means your camera will keep the last pose even after you've reset the pipe.

Re 3: Since the T265 didn't hard reset you shouldn't need to re-import the localization map.

Would be nice to have someone from Realsense chime in on this to make sure I'm not mistaken.

@csr-kick
Copy link

csr-kick commented Jun 3, 2020

Thanks, forgot about the timeout. Make sure that you set it initially to a high value to allow the sensor boot up or whatever before dropping the timeout down for normal operations. Then in the catch statement put it back up. My current testing code:

rs2::log_to_console(RS2_LOG_SEVERITY_DEBUG); //verbose, can make this _INFO for less

rs2::config cfg;
cfg.enable_stream(RS2_STREAM_POSE, RS2_FORMAT_6DOF);
auto pipe = std::make_shared<rs2::pipeline>();
pipe->start(cfg);
unsigned int to = 20000;
while (true)
{
    try {
            
        auto frames = pipe->wait_for_frames(to);
        to = 40;
        auto f = frames.first_or_default(RS2_STREAM_POSE);
        // Cast the frame to pose_frame and get its data
        auto pose_data = f.as<rs2::pose_frame>().get_pose_data();
    
        // Print the x, y, z values of the translation, relative to initial position
        std::cout << "\r" << "Device Position: " << std::setprecision(3) << std::fixed << pose_data.translation.x << " " <<
            pose_data.translation.y << " " << pose_data.translation.z << " (meters)" << std::endl;
    }
    catch (const rs2::error& e) {
        to = 20000;
        std::cout << "T265 crashed. Restarting ..." << std::endl;
        pipe->stop();
        pipe = std::make_shared<rs2::pipeline>();
        pipe->start(cfg);
        std::cout << "... restarted." << std::endl;
    }
}

My testing shows that (without loading any maps right now) localization does go back to 0,0,0. In extended testing this has thrown a segfault after quite a while, typically when coming back up. Seems like a pose message from before the pipe.stop comes back after we restart and it doesn't like that.

@RealSenseSupport can we get any sort of statement about this problem?

@csr-kick
Copy link

csr-kick commented Jun 5, 2020

Still working to see if there is some way to actually make the T265 work with the RPI.

I have tried setting up ubuntu 18 on the rpi, and same problem, T265 will eventually stop responding, and it happens in a matter of seconds with the D435 sending depth at 640x360x30fps.

@MartyG-RealSense please can we get some sort of response from intel on this issue.

@MoBaT
Copy link

MoBaT commented Jun 9, 2020

Just wanted to put out there that I'm having the exact same problem and am also restarting the pipeline on failure. I'm on v2.34.1. Would love a fix to this @MartyG-RealSense

@m4xr1sk
Copy link

m4xr1sk commented Jun 15, 2020

Same problem here with the SDK v2.35.2
I also would like to have a fix because otherwise this product is not usable at all.

@alanypf
Copy link

alanypf commented Jul 10, 2020

Hi all,
I have the same problem.
I am using Ubuntu 18.04 server. Librealsense is built from the source with the LTS kernel patch.
The data link broke after a minute or so. The problem only exists on raspberry pi4 for me since if I plug the same SD card into a raspberry pi 3b+, the data links remain healthy. And on raspberry pi 4, neither USB2 nor USB3 ports work.
I hope this helps the developer team.

Thanks.

@RealSenseSupport
Copy link
Collaborator

Thanks for highlighting this. At this time we have moved our focus to our next generation of products and consequentially will not be addressing this T265 issue.

@dschnabel
Copy link
Author

@RealSenseSupport are you dropping support for the T265? I don't see any mention on https://www.intelrealsense.com/tracking-camera-t265

@ArkadiuszNiemiec
Copy link

Same issue here, I did not find an information about T265 not being supported.

@MartyG-RealSense
Copy link
Collaborator

I am now able to provide information about the future of RealSense, as an 'End of Life' notice has been released by Intel. Although some RealSense models are being discontinued, the 400 Series stereo depth cameras are continuing. The official PDF of the End of Life notice explaining the changes is attached below.

PCN118463-00.pdf

@dschnabel
Copy link
Author

Thank you for this update @MartyG-RealSense.

As the T265 will be de-supported, can at least the source code of the firmware be made public? Even if Intel isn't interested in fixing this bug, many other in the community will be eager and capable of tackling this problem if we just had access to the source code.

@MartyG-RealSense
Copy link
Collaborator

Hi @dschnabel Whilst the RealSense SDK is open-source and developers are free to 'fork' it to create their own custom versions that they host on their own GitHub, Intel do not open-source RealSense's firmwares or camera algorithms.

@nixinator
Copy link

i think you just highlighted the problem... maybe i can ask 'do not', to 'to the reason why'?

@MartyG-RealSense
Copy link
Collaborator

@nixinator A reason given in the past has been that such resources are Intel's intellectual property.

@nixinator
Copy link

root cause analysis...complete!

@zhouzhiwen2000
Copy link

@nixinator A reason given in the past has been that such resources are Intel's intellectual property.

If you can't release the source code, can you solve this issue before dropping support? Perhaps you could release part of the source code?

@MartyG-RealSense
Copy link
Collaborator

@zhouzhiwen2000 Development work for the T265 has ended so no further software or firmware updates can be provided for it.

@csr-kick
Copy link

csr-kick commented Sep 4, 2021

@MartyG-RealSense Development work for the T265 has ended so no further software or firmware updates can be provided for it.

This is why no one will ever trust the depth sensors that are still around. Companies cannot base their products on a device where the creator has no 'anchor' product or vested interest in it's success and support. If you have an issue and wake up the next day to find that the supplier has decided to simply walk away rather than do the right thing for their customers, your existence is at risk while theirs is not.

@HanDaSeul
Copy link

HanDaSeul commented Sep 13, 2021

I think I found the solution, which the T265 suddenly stops with RPi using ROS where the RPi is 8GB.
The problem is neither the T265 nor the software(SDK, ROS whatever). It's the lack of power output for USB3.0 port on the RPi.
I guess most of you guys plug at least two items to the RPi usb ports, especially wireless usb wifi adapter and theoretically, there will be no problem in case of power consumption, but it has. I think because of power consumption is too high so one of them tries disconnecting. Or you just using RPi's wifi chip, also you need to consider it.
Here's how did I make it: You need 1 powered usb hub. Some cheap powered usb hubs still have problems, but in my case i used 'ugreen usb 3.0 hub Model name: cr113' with external power. Plug all of USB devices to that usb hub, except t265. Only the T265 is connected to the RPi directly. I guess you can also plug to that usb hub but there's another problem 'when boot up you have to unplug and plug for working'. My T265 is directly connected to RPi using 'uhubctl' so I don't need to unplug for detection.
My RPi's OS is gnome-session but not fully installed ubuntu-desktop. Realsense SDK version is 2.48, not 2.49. I tested how long can RPi handle without losing connection in both ROS and ROS2. On ROS2 my RPi's usb port have t265 and usb hub. And usb hub have wireless usb wifi adapter and RPLidar A1 and takes 5W from external power. But in ROS, I didn't use Lidar(I forgot it) and my usb hub takes 2.5W. I didn't try streaming cameras on both.
The result is both are fine! In the ROS, still keeps connected after 50 minutes and ROS2, I just disconnected after 30 minutes. In ROS2 there's little bit of lag but I don't think that's related to the power.
One more thing. Make sure keep T265 cool. When the T265 is hot, it seemed to lose connection.
Hope it helps you guys!

20210912_175145
Above photo is the power consumption and connected devices testing in ROS2

Screenshot from 2021-09-10 17-11-45
Above screenshot is testing in ROS. See the elapsed time below.

@HanDaSeul
Copy link

HanDaSeul commented Sep 13, 2021

And in the case of the problem unplug then plug after first boot up, why don't T265 try turn off then on itself when launch the node? For example, I saw RPLidar a1 keeps rotate and when I launch the node RPLidar a1 on ROS, it stops rotate for 1~2 seconds and rotate again. Even that Lidar connected to the usb hub, still works.

@silverjoda
Copy link

After plugging in and out about 200 times on the Rpi, I found out that you can use Uhubctl to automatically power cycle all the USB ports after boot, so that the t265 is recognized all the time.

@nixinator
Copy link

nice work and power fluctuation debugging, nice. can i ask what power meter you are using in your screenshot, probably a essential item for robot builders.

@HanDaSeul
Copy link

HanDaSeul commented Sep 13, 2021

nice work and power fluctuation debugging, nice. can i ask what power meter you are using in your screenshot, probably a essential item for robot builders.

@nixinator My bad, it's a photo not a screenshot😂
It's j7-c. When I plug T265 through that usb tester, placing between RPi usb port and T265, t265 is always recognized as being connected to usb 2.0, not 3.0.

@csr-kick
Copy link

@HanDaSeul When I did my testing the RPI4 (4GB) was sitting on my desk with just the T265 plugged in. It happens more frequently when you stress the CPU, and if you stress the USB by also taking in the video streams. When I used a powered USB hub with the D453i it also crashed more frequently, but as you pointed out not all usb hubs can provide the right power. I'll do a quick check with a custom USB hub I have somewhere around here that has its own 5v/5A regulator on it. Can you tell me what USB power analyzer you have? Looks pretty nice.

As we talked earlier in this thread, I wrapped the driver in code that tracks the last pose, restarts the T265 pose stream, then applies the old pose as an offset to the now zero'd poses. Some other trickery with matching depths or lidar scans will relocalize any drift pretty well.

@HanDaSeul
Copy link

HanDaSeul commented Sep 14, 2021

@HanDaSeul When I did my testing the RPI4 (4GB) was sitting on my desk with just the T265 plugged in. It happens more frequently when you stress the CPU, and if you stress the USB by also taking in the video streams. When I used a powered USB hub with the D453i it also crashed more frequently, but as you pointed out not all usb hubs can provide the right power. I'll do a quick check with a custom USB hub I have somewhere around here that has its own 5v/5A regulator on it. Can you tell me what USB power analyzer you have? Looks pretty nice.

As we talked earlier in this thread, I wrapped the driver in code that tracks the last pose, restarts the T265 pose stream, then applies the old pose as an offset to the now zero'd poses. Some other trickery with matching depths or lidar scans will relocalize any drift pretty well.

@csr-kick If i'm right, USB power analyzer = power meter = 'j7-c'.

As you mentioned about stress, I tried overclocking little bit higher and making sure keep cool(didn't exceed 50 celsius degree), but nothing changed. Through the whole test myself, conclusion is 'as long as you are using RPi, don't try streaming in ROS'. But just realsense-viewer, you can try streaming and there's no problem with connection. I still can't understand why. Only I can make it clear is there's difference of using resources like cpu, power whatever.
Before I changed the usb hub to 'CR113', i tried other powered usb hub. Some bad powered usb hubs are not stable and have back-power issue.

I don't have D453i, or kinda depth. That's why I'm trying make wheel odometry model for my project.

@Construkted-Reality
Copy link

I don't think it's a power delivery issue. At least not from my tests:
I'm running only the T265 attached to the Pi 4 (8gb). No other USB devices. I'm running a simple python script that outputs to the screen the T265 pose values to the screen. There is minimal CPU load.

The Pi is sucking back around 0.8-0.9A with the T265 running. The USB power bank I'm using is capable of delivering 2.5A, so there's no bottleneck there.
The T265 still stops at random intervals...

image

@HanDaSeul
Copy link

HanDaSeul commented Sep 14, 2021

@Construkted-Reality I tried several times after I applied my solution, there's no losing connection. Sometimes, having without usb tester is helpful. In my experience, which is not exactly clear tho, powerbank is less stable than power adapter. RPi 4 8GB is recommended 3A. for input. I tested with 3A adapter for RPi, 2A for powered usb hub.

I'm using wireless usb adapter for sending pose data and it takes 0.45A, 5V, which means I turned off RPi's wifi chip. And I disabled fisheye camera streaming, too(maybe you already tried).

@Construkted-Reality
Copy link

@HanDaSeul
I just tried two different 3A power adapters (one was from an LG phone, and another from a Pixel 3 phone) and have experienced the same loss of connection.
I have disabled the camera stream in the python script.

@HanDaSeul
Copy link

HanDaSeul commented Sep 14, 2021

@Construkted-Reality Then, probably only I can say is use wireless usb wifi adapter with powered usb hub. Don't let other usb devices take power from RPi. Let RPi take less power as you can like setting RPi's wifi chip off. And just in case keep T265 cool.
In my case as you see my screenshot, RPi gathers pose data and sends. My laptop receives that pose data, so I can see the t265's movement almost without lag through rviz.

Wait...it looks like the problem is not only the power, but also RPi's wifi chip performance...I didn't test using RPi's wifi chip.

@Construkted-Reality
Copy link

I don't think cooing the T265 has anything to do with it since I've had the T265 work perfectly on a Pi3, and it never disconnects (without any cooling).
I'll do some tests with the wifi module disabled to see if that makes any difference.

@zhouzhiwen2000
Copy link

@Construkted-Reality Then, probably only I can say is use wireless usb wifi adapter with powered usb hub. Don't let other usb devices take power from RPi. Let RPi take less power as you can like setting RPi's wifi chip off. And just in case keep T265 cool.
In my case as you see my screenshot, RPi gathers pose data and sends. My laptop receives that pose data, so I can see the t265's movement almost without lag through rviz.

Wait...it looks like the problem is not only the power, but also RPi's wifi chip performance...I didn't test using RPi's wifi chip.

Try putting more pressure on the RPI USB port while testing, eg. copying a large file from an SSD. It seems that the problem occurs more often when the bus is busy. The SSD can be powered externally to avoid extra power consumption.

@zhouzhiwen2000
Copy link

@Construkted-Reality Then, probably only I can say is use wireless usb wifi adapter with powered usb hub. Don't let other usb devices take power from RPi. Let RPi take less power as you can like setting RPi's wifi chip off. And just in case keep T265 cool.
In my case as you see my screenshot, RPi gathers pose data and sends. My laptop receives that pose data, so I can see the t265's movement almost without lag through rviz.
Wait...it looks like the problem is not only the power, but also RPi's wifi chip performance...I didn't test using RPi's wifi chip.

Try putting more pressure on the RPI USB port while testing, eg. copying a large file from an SSD. It seems that the problem occurs more often when the bus is busy. The SSD can be powered externally to avoid extra power consumption.

I tested using T265 and a USB camera, with the USB cam externally powered, and the problem is still around.

@zhouzhiwen2000
Copy link

For those using T265, the ASUS tinker board seems to be a good choice. It's tested and no such problem is observed. Its price is also similar to RPI4.

@DevepNoName
Copy link

DevepNoName commented Sep 15, 2021

If you guys seach here about T265, that camera doesnt work properly in Raspberry 4 / Nvidia Jetson Nano, lots of people complaining about this (inluding myself), and Intel will not correct this (just shutdown after a while, and have to take out the cable and reconnect again). But works in Raspberry 3(not tested by myself), some people confirmed this.

@DevepNoName
Copy link

#8779

@giangthewalkingman
Copy link

giangthewalkingman commented May 14, 2024

@DevepNoName If you guys seach here about T265, that camera doesnt work properly in Raspberry 4 / Nvidia Jetson Nano, lots of people complaining about this (inluding myself), and Intel will not correct this (just shutdown after a while, and have to take out the cable and reconnect again). But works in Raspberry 3(not tested by myself), some people confirmed this.

no, I 'm facing the similar problem in raspberry pi 3b+

@giangthewalkingman
Copy link

giangthewalkingman commented May 15, 2024

@silverjoda After plugging in and out about 200 times on the Rpi, I found out that you can use Uhubctl to automatically power cycle all the USB ports after boot, so that the t265 is recognized all the time.

did u mean that using uhubctl to manage power so that t265 will be connected all the time without odom losing?

@silverjoda
Copy link

silverjoda commented May 15, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests