External code during device inference

[PVSemk]

Hello, I'm instrested in usage of a depth-estimation network, which is written in PyTorch, on your device.
We've faced an issue with an operation (grid_sample) that is not supported by ONNX and OpenVINO Toolkit. We came to the solution of splitting the model into 2 parts (before the operation, after the operation) and launching the external C++ code, implementing the operation, to process the intermediate tensors. Is it possibly to somehow run such a pipeline on your device?

Thanks in advance,
Pavel Semkin

[Luxonis-Brandon]

Hi @PVSemk ,

Yes, this is now becoming possible using our Gen2 Pipeline Builder (#136). We have an initial example implementation of such logic between networks here:
https://github.com/luxonis/depthai-experiments/tree/master/gaze-estimation

The processing between neural networks is done on the host in the example above to implement the processing/logic between each neural network:

So I think this should also be leverage-able for your purposes. We could help you implement it live via our Slack if you are interested (here)

We could also probably take a stab at implementing the pipeline on the device if you are comfortable sharing the neural models.

Thoughts?

Thanks,
Brandon

[Luxonis-Brandon]

And another option is you could actually use OpenCL to implement that custom logic as its own 'network' that could run as a node in the pipeline builder. In this case then no operations would be done on the host.

[Luxonis-Brandon]

Let me know if you have any additional questions @PVSemk otherwise I'm thinking I'll close this issue soon. Thoughts?

Thanks again!
-Brandon