Scripting Support on DepthAI

[Luxonis-Brandon]

Start with the why:

When running DephtAI in an embedded use-case where there is no other processor involved (like here), it is necessary to have custom code run directly on DepthAI for many applications, like implementing rulesets and filtering of meta-data to determine when regions of interest to pass between stages of inference as in #136, or to talk directly with actuators, or implement custom SPI, UART, or I2C communication with sensors or other devices.

Move to the how:

We are now sponsors of microPython!

We have initially compiled it for DepthAI (running directly on the platform), and we will be implementing the capability for microPython to run in nodes in Pipeline Builder Gen2 (#136).

In implementing microPython (see comments in this issue) we discovered that supporting the why here would actually be better served by CPython bindings directly on DepthAI. So we hard-pivoted from microPython to CPython. And this actually allows more functionality, accessible with standard Python language features.

Move to the what:

Implement the capability for microPython CPython to run directly on DepthAI as nodes in the Gen2 Pipeline Builer.

[Luxonis-Brandon]

As an update, we do have microPython now compiled and printing "hello world" from DepthAI.

[rychardeh]

Will you also be including a numpy-like library to help manipulate data between pipeline stages, e.g. https://github.com/v923z/micropython-ulab?

[Luxonis-Brandon]

Good idea. CC: @jonngai

[Luxonis-Brandon]

So @rychardeh - we will be supporting ulab. Great idea and thanks for bringing this up.

[Luxonis-Brandon]

Status update:

To summarize, here’s what we have:
- Individual micropython instances being kicked off in Micropython nodes.
- Our own mpy Module built externally from MicroPython that can be used to call stuff back in C++/Nodes.
- Martin recently added the ability to add multiple inputs/outputs to the node on the host.
Next, here’s what we’re missing:
- A clean way to build everything in a single Makefile.
- The binding between the C python modules and our C++ nodes.
- A way to run compiled python “binaries”.
We're currently experimenting with what we can do with that binding between our node and Micropython. We're going after something to wrap SReceiver. The other big piece is binding something like MSender so the MicroPython node can output to other nodes.

[Luxonis-Brandon]

As another update on this, when trying to get microPython to thread so that it can be used in any number of nodes arbitrarily placed in a pipeline, we ran into many multithreading issues.

We tracked down and fixed many of them, and then found a micropython bug causing reliable periodic crashing. We narrowed it down to specifically when the GC ran and we started more than 1 thread using the python exec() method. We found the best way to deal with it is just to not use exec in threads...

But, in parallel, we investigated using CPython instead of microPython altogether, and it seems to be a better solution to what we are trying to enable here. So the upside is that with CPython this will actually be more powerful. The downside is some our microPython work to date won't be used. But it was useful in learning and getting to a better/more-robust final solution with CPython. And a lot of the intefacing mechanisms we made (which were a lot of the time) are directly reusable, as microPython and CPython are very similar, with micropython just quite stripped down.

So the status now is that we're currently adding bindings for all of our message types (Buffer, ImgFrame, NNData, ImageManipConfig, CameraControl, ImgDetections, SystemInformation) to CPython.

We'll keep everyone updated as we make it further to getting this out as a capability as a node in the Gen2 pipeline builder.

[Luxonis-Brandon]

This CPython implementation is now available for anyone who'd like to play with it:

See the gen2_scripting branch, here: https://github.com/luxonis/depthai-python/tree/gen2_scripting

And an example script is here:
https://github.com/luxonis/depthai-python/blob/gen2_scripting/examples/29_simple_script_camera_control.py

In a nutshell, onboard the device a CPython3.9 implementation is running with many modules removed to not bloat FW size too much.

Each node gets its own subinterpreter and starts executing the given script once the pipeline starts.
On-device bindings are made such that they resemble host side bindings as much as possible, with the bindings module being implicitly imported.

There are 2 special global variables:

• node
• pipeline

These hold current node object and pipeline object. These give you access to eg assets of current node or pipeline node and loggers

For node there is also a io map attribute, which represents host specified IOs:

# Host
myScript.outputs['test1'].link(xlink.input)
xlinkin.output.link(myScript.inputs['in1'])
# Device
node.io['test1'].send(...)
msg = node.io['in1'].get()

IO match host DataOutputQueue and DataInputQueue API.

Right now there are get (and tryGet, tryGetAll, getAll) and send (and trySend) functions available

Feel free to play around and break stuff - we haven't done much stability testing yet :)

You can call node.trace, node.debug, node.warn, node.error, node.critical to print log messages (kinda like log on Android:))

Outside of existing messages (Buffer, ImgFrame, ImgDetections, ...), custom messages must be serialized and added as data to buffer or to other message.
eg. custom messages are not supported, existing ones must be used and data carried inside of them.

There is also setScriptPath([path/to/script.dpy]) which you may use if you prefer to have the script in a separate file.

(Note, python library might not be prebuilt - feel free to compile locally)

Next up in the works is HW access (GPIO, SPI & I2C), while SDCard and other device access should be accessible with standard Python language features.

[nemccarthy]

It looks like the gen2_scripting branch is no more. Is it still possible to test this implementation?

[Erol444]

It looks like the gen2_scripting branch is no more. Is it still possible to test this implementation?

Hello @nemccarthy , we have already merged the gen2_scripting into develop branch (depthai-python repo), closed PR here: luxonis/depthai-python#316

[Luxonis-Brandon]

This is now implemented and documented here.