Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Depth Calculator Node in Gen2 #292

Closed
Luxonis-Brandon opened this issue Dec 21, 2020 · 16 comments
Closed

Depth Calculator Node in Gen2 #292

Luxonis-Brandon opened this issue Dec 21, 2020 · 16 comments
Labels
enhancement New feature or request

Comments

@Luxonis-Brandon
Copy link
Contributor

Start with the why:

In the Gen1 pipeline builder of DepthAI, one of the core functionalities that is hard-coded in is the capability to get the depth of an neural-network object detector object. The algorithm averages over the center of the object, with the size of this averaging area selectable via API (via the padding_factor). This can be seen below, where the blue box is where the depth is averaged from:

image

DepthAI then reprojects this depth, using the camera intrinsics, to the location of the object in physical space in X, Y, and Z in meters relative to the center of the right grayscale cameras.

This functionality, however, is in a hard-coded position in the Gen1 pipeline. So it is only useful if the hard-coded placement in the pipeline is useful to the end application. Enabling this functionality as a node in the Gen2 Pipeline Builder (#136) allows this functionality to be used in all sorts of other permutations (including more CV/AI functions being performed prior to getting the object location, say to improve location accuracy). Or for example, being used with object tracking (instead of an object detector), or some other region of interest (ROI) not produced by an object detector, but perhaps some other node (or series of nodes) that produces this ROI.

Move to the how:

Leverage the Gen2 Pipeline Builder (#136) architecture to build a depth calculator node. Which then will work with the stereo node input to generate a depth and 3D coordinates (XYZ in meters) for a given passed-in ROI.

Move to the what:

A node that takes an ROI and calculates the depth of that ROI.

For the initial version of this node, let's just have it run the same algorithm as Gen1, so the padding_factor approach. Later, we can expose other options for the node (for example, maybe some more sophisticated depth-edge detection and averaging only inside that region, or the capability to take in a semantically input (pixel-map) input, but for now let's ignore this and save for later).
@Luxonis-Brandon Luxonis-Brandon added enhancement New feature or request Gen2 labels Dec 21, 2020
@Luxonis-Brandon
Copy link
Contributor Author

@szabi-luxonis has this initially working as a node. So I'm moving this to In-Progress. Examples will be coming in a week or so most likely.

@luiscastro1995
Copy link

Hi!
I am currently running the YoloV4-tiny on my OAK-D using the Gen2 API, however, I am only getting the bounding boxes coordinates. I was wondering if the 3D coordinates of a ROI can already be retrieved using this API? Are there any examples available?

Thank you for your time!

@Luxonis-Brandon
Copy link
Contributor Author

Luxonis-Brandon commented Mar 10, 2021 •
edited
Loading

Hi @luiscastro1995 ,

Great question (and timing). So in Gen2 we just pushed a PR for how to do this.
See luxonis/depthai-python#151 .

So the example included is for MobileNetSSD, but since the same DetectionNetwork API is used (which supports YOLO3/4 and MobileNetSSD), I think this should directly work for YoloV4-tiny.

@szabi-luxonis can confirm though.

Thoughts?

Thanks,
Brandon

@SzabolcsGergely
Copy link
Collaborator

Hi @luiscastro1995 ,

Great question (and timing). So in Gen2 we just pushed a PR for how to do this.
See luxonis/depthai-python#151 .

So the example included is for MobileNetSSD, but since the same DetectionNetwork API is used (which supports YOLO3/4 and MobileNetSSD), I think this should directly work for YoloV4-tiny.

@szabi-luxonis can confirm though.

Thoughts?

Thanks,
Brandon

Yes, it will work by combining recently added example 26/28 with example 22_tiny_yolo_v3_device_side_decoding.py

@luiscastro1995
Copy link

luiscastro1995 commented Mar 11, 2021 via email

@Luxonis-Brandon
Copy link
Contributor Author

Yes, just responded there. Very odd - haven't seen this before.

@luiscastro1995
Copy link

luiscastro1995 commented Mar 18, 2021 via email

@SzabolcsGergely
Copy link
Collaborator

Hi again! I was wondering if you could give me a hint on a problem I am facing now. I am trying to adapt the 26_1_spacial_mobilenet.py example in order to use yolov4-tiny. The relevant changes I have made were: Line 31: spatialDetectionNetwork = pipeline.createYoloSpatialDetectionNetwork() * After Line 67 I have added the following: spatialDetectionNetwork.setNumInferenceThreads(2)spatialDetectionNetwork.setNumClasses(len(labelMap))spatialDetectionNetwork.setCoordinateSize(4)spatialDetectionNetwork.setAnchors(np.array([10,14, 23,27, 37,58, 81,82, 135,169, 344,319]))spatialDetectionNetwork.setAnchorMasks({ "side26": np.array([1,2,3]), "side13": np.array([3,4,5]) })spatialDetectionNetwork.setIouThreshold(0.5) I get errors because the methods setNumClasses, setCoordinateSize, setAnchors, setAnchorMasks and setIouThreshold are not valid. If I remove these method calls the program works but I have no detection whatsoever. I have also changed the labelMap list as in the 22_1 example. My first question is how can I set the YOLO parameters then? My second question and just for a matter of curiosity rather than a real problem is as follows. Right now I am using a self compiled yolov4.tiny blob as I could not find a way to get depthai download one for me. How can I get the blob file that the code is expecting? I thought about building the repository with the test flag enable but for some reason, I got an error regarding the pybind11, here is the log, just in case: ################# CMake Error at cmake/pybind11-mkdoc.cmake:2 (include): include could not find load file: target-public-headers Call Stack (most recent call first): CMakeLists.txt:96 (include) CMake Error at cmake/pybind11-mkdoc.cmake:9 (get_target_property): get_target_property() called with non-existent target "depthai::core". Call Stack (most recent call first): CMakeLists.txt:99 (target_pybind11_mkdoc_setup)
…
-- Configuring incomplete, errors occurred! ################# Sorry for the lengthy questionnaire. Again amazing work and thank you for your patience! Kind regards, Luís Castro
On Thu, Mar 11, 2021 at 5:01 AM Luxonis-Brandon @.> wrote: Yes, just responded there. Very odd - haven't seen this before. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Will create an example for you, I will let you know when it's done.

@SzabolcsGergely
Copy link
Collaborator

SzabolcsGergely commented Mar 18, 2021 •
edited
Loading

Here is the example for tiny-yolo-v3 and v4.
luxonis/depthai-python#163

Install the following depthai library:

python3 -m pip install depthai==2.0.0.1+9cf036c3b0c3266500dbb71f7626e6909a654fda --extra-index-url https://artifacts.luxonis.com/artifactory/luxonis-python-snapshot-local/

Download blob from here:
https://artifacts.luxonis.com/artifactory/luxonis-depthai-data-local/network/tiny-yolo-v4_openvino_2021.2_6shave.blob
Or tiny-yolo-v3 : https://artifacts.luxonis.com/artifactory/luxonis-depthai-data-local/network/tiny-yolo-v3_openvino_2021.2_6shave.blob

Run as:
python3 26_3_spatial_tiny_yolo.py <path_to>/tiny-yolo-v4_openvino_2021.2_6shave.blob

@luiscastro1995
Copy link

luiscastro1995 commented Mar 18, 2021 via email

@Luxonis-Brandon
Copy link
Contributor Author

Awesome, thanks for circling back!

@SzabolcsGergely
Copy link
Collaborator

Implemented in : luxonis/depthai-python#151

@elakkiya-workit
Copy link

elakkiya-workit commented Mar 23, 2021 •
edited
Loading

Hi, This is a great example. I am working with a custom mobile SSD model and trying to adapt to 26_1_spatial_mobilenet.py.

I installed the below dependency

python3 -m pip install depthai==2.0.0.1+9cf036c3b0c3266500dbb71f7626e6909a654fda --extra-index-url https://artifacts.luxonis.com/artifactory/luxonis-python-snapshot-local/

However, I am facing an attribute error.

File "depthhelloWorld.py", line 31, in <module> spatialDetectionNetwork = pipeline.createMobileNetSpatialDetectionNetwork() AttributeError: 'depthai.Pipeline' object has no attribute 'createMobileNetSpatialDetectionNetwork'

@szabi-luxonis

@Luxonis-Brandon
Copy link
Contributor Author

Hi @elakkiya-workit ,

Thanks for the kind words and sorry about the trouble here.

So we actually just did the formal release around this. So I'm thinking there's a good chance that this may solve the problem. I don't know for sure, but I think there's a good chance.

So could you please give this example a try?

https://docs.luxonis.com/projects/api/en/latest/samples/26_1_spatial_mobilenet/

And make sure to do the Setup section from in there to make sure the depthai library is updated to the latest:
https://docs.luxonis.com/projects/api/en/latest/samples/26_1_spatial_mobilenet/#setup

Thoughts?

Thanks,
Brandon

@elakkiya-workit
Copy link

Hi @Luxonis-Brandon

Thank you for sharing these useful links.

I found the problem. There was two different versions of depthai installed in my venv. After clearing them, I'm able to execute the files with depthai==2.0.0.1+12037bb68b00eb5397c946e13db8d58c46e39c45 package.

Appreciate your support.

Thanks
Elakkiya

@Luxonis-Brandon
Copy link
Contributor Author

Thanks for circling back!

And excited to see what you build with OAK!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Luxonis-Brandon @luiscastro1995 @SzabolcsGergely @elakkiya-workit