Project SPIDER (Spiking Perception and processing for Intelligent Detection of pEdestrians on urban Roads)

In Project SPIDER, we propose a novel solution for urban road surveillance using event-based neuromorphic cameras (i.e. Dynamic Vision Sensors - DVS), neural algorithms, and embedded neuromorphic computing platforms (i.e. Brainchip Akida). The solution is described by rapid detection and identification of abnormal activities, typically describing roadside pedestrian and bicyclists dynamics. The initial setup we have build uses a pipeline comprised of 3 main modules:

• Data acquisition module: including the DVS sensor and (optionally) a traditional CMOS camera for acquiring ground truth and a reference of the visual scene used in the data labelling phase
• Data pre-processing module: this module contains a data analysis, recording, replay, and algorithm testing sub-module which uses a custom developed code to control the data acquisition from the DVS, data visualization, data recording, data replay, and basic algorithm testing capabilities. This module is also responsible for the DVS events filtering and (eventually) minimal aggregation of the events in processing bins (i.e. event framing) for the classification algorithms in both algorithmic approaches (i.e. the SNN and the Event-based EM approaches, respectively).
• Model life-cycle module: implementing the two approaches using Spiking Neural Networks and the Event-based Expectation Maximization in EdgeImpulse. The two algorithms we tested are: Object Detection FOMO (Faster Objects, More Objects) using MobileNetV2 0.35 (EdgeImpulse Library) and Event-based Expectation Maximization (custom implementation added in EdgeImpulse)
• System deployment module: FOMO MobileNetV2 deployment and Event-based Expectation Maximization deployment

The datasets used for training and testing the models are available on Zenodo