A spiking neural network model of 3D perception for event-based neuromorphic stereo vision systems
by Marc Osswald, Sio-Hoi Ieng, Ryad Benosman & Giacomo Indiveri
Abstract
Stereo vision is an important feature that enables machine vision systems to perceive their environment in 3D. While machine vision has spawned a variety of software algorithms to solve the stereo-correspondence problem, their implementation and integration in small, fast, and efficient hardware vision systems remains a difficult challenge. Recent advances made in neuromorphic engineering offer a possible solution to this problem, with the use of a new class of event-based vision sensors and neural processing devices inspired by the organizing principles of the brain. Here we propose a radically novel model that solves the stereo-correspondence problem with a spiking neural network that can be directly implemented with massively parallel, compact, low-latency and low-power neuromorphic engineering devices. We validate the model with experimental results, highlighting features that are in agreement with both computational neuroscience stereo vision theories and experimental findings. We demonstrate its features with a prototype neuromorphic hardware system and provide testable predictions on the role of spike-based representations and temporal dynamics in biological stereo vision processing systems. [...]
Detailed view of a horizontal layer of the network. An object is sensed by two eyes and accordingly projected onto their retinal cells. The spiking output of these cells is spatio-temporally correlated (coincidence detectors) and integrated (disparity detectors). The final output encodes a representation of the original scene in disparity space (x, y, d). For the sake of visibility, only a horizontal line of retinal cells, at fixed vertical cyclopean position y, is considered. The corresponding coincidence and disparity detector units, hence, lie within a horizontal plane (spanned by x and d). Only a few units are shown here whereas in the complete network, the units are uniformly distributed over the entire plane. The shaded planes indicate how the network expands vertically over y. More details on how the neurons are connected among each other is provided in the Methods section.
(A) Schematic of the dRDS stimulus for the left and right eye. (B) Ground-truth disparity image. Disparity is encoded by color ranging from near (red) to far (blue). (C) Disparity map generated from accumulated responses of the network while the dRDS stimulus was presented for 1 second.
Full Article Here:
www.nature.com/articles/srep40703
and Special Thanks to
@3Droundabout