Any Possible Way to Get Depth from T265

[mzahana]

Hi.

I know that T265 is currently used as a tracking camera. Although it is a stereo camera, but no depth is provided. Is there any possibly way to get depth from T265?

[MartyG-RealSense]

The official answer in terms of RealSense support for this is of course "no". However, some researchers have used fisheye camera pairs in general (not the T265 ones specifically) to generate a depth map for applications such as self-driving cars. Google "get depth from fisheye camera" for more details.

Here is an example paper:

https://arxiv.org/pdf/1809.06132.pdf

[ev-mp]

@mzahana,
Please check #3951 by @bfulkers-i that demonstrates how to generate depth map with T265 fisheye stereo sensors.

[MartyG-RealSense]

@ev-mp Thanks so much for the link! It's awesome that someone has managed to put the depth from stereo fisheye principle into practice with T265 and also deal with the fisheye distortion factor that otherwise makes depth calculation from fisheye awkward :)

[mzahana]

@ev-mp thanks for the information.

[ShettyHarapanahalli]

If you are using any Nvidia product then you can try to use stereo processing algorithms like one in the link below to get depth from the stereo cameras -

https://docs.nvidia.com/isaac/isaac/packages/stereo_depth/doc/sddnn.html

FYI -
The controller used in T265 has can generate 3d Depth data, Intel can release SW update to extract depth, more information about the processor used is in the link below -

https://movidius-uploads.s3.amazonaws.com/1532512604-1503680554-2016-12-12_VPU_ProductBrief.pdf

Best Regards,
Suman

[mzahana]

@ShettyHarapanahalli thanks for the information. The stereo_depth_dnn, according to my understanding, would run at 10-30 FPS on TX2 which can be power hungry for small robots. I guess a depth camera like D4xx would be better in this case. Also, if depth can be computed on T265 as you mentioned, it would probably solve the power issue and avoiding the use of an extra depth camera. However, I am not sure if the depth quality would be comparable to D4xxs.

[dorodnic]

@ShettyHarapanahalli @mzahana - it is possible to do basic depth in T265 firmware, however, IMHO it would be low priority, because resulting depth quality would not be acceptable for most applications (both because of passive nature of the device and because of the wide field of view)
Optics of T265 are optimised for the task of tracking while these of the D400 - for dense depth generation.
Even if you only need sparse depth for something like collision avoidance, without a projector there is no guaranty you would get any depth from the obstacle.

[ShettyHarapanahalli]

Field of view of T265 is much better compared to D4xx and for many application, we need wide viewing angles the depth would be icing for such applications. Considering obstacle avoidance we need accurate depth just in front of the robot, other places low-quality depth will do the job.

I believe next-generation tracking cameras will have these features -

1. Depth is also provided
2. Integration of GNSS and other technologies to get better localization
3. AI is deployable within the device for object detection and other applications.
4. Add microphones support and wireless connectivity (also can be used for localization)

This can become a visual and audio sensor for projects.

[ShettyHarapanahalli]

Good to see the depth generation from T265 ..
https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/t265_stereo.py

[RealSenseCustomerSupport]

---

Hi @mzahana

With the inclusion of the python example, can we close this issue now?

Thanks

[alexcherpi]

Hi @MartyG-RealSense
You said that we could use standard algorithm such as in (https://arxiv.org/pdf/1809.06132.pdf) to reconstruct the depth map from the right&left images.
But to my knowledge the T265 has a IR projector which projects a dot pattern both left & right cameras can see. I'm afraid that given all these dots in the image standard depth-from-stereo algorithm would lead to errors during the reconstruction.
What do you think? Thanks

[MartyG-RealSense]

Since that PDF was quoted, an example program for getting depth from the T265's left and right fisheye cameras has been added to the SDK. It isn't as good as the depth on a dedicated depth camera such as the D415 or D435 but certainly better than nothing.

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/t265_stereo.py

As far as I know, the T265 does not have an IR projector (which is a separate component from the left and right IR imagers). The purpose of the dot pattern cast by the projector in cameras such as the D415 and D435 is to add texture to objects to aid a camera in reading depth information from a surface. But as the T265 does not have depth sensing by default and has to obtain it with a workaround on the fisheye lenses with the above script, it would not need a projector to enhance depth.

A diagram of the T265 is shown below.