Two-Way Street: Efficient VSLAM using Collaborative In-Sensor and Off-Sensor processing

We present a novel implementation of Visual SimultaneousLocalisation and Mapping (VSLAM), whose computation is performedcollaboratively between a sensor-based front-end utilizing efficient in-pixel compute, and a PC-based back-end that sends pose estimationback to the sensor to improve feature matching. The front-end performsthe tasks of point-feature detection and tracking at up to 2000 Hz, whilethe PC back-end conducts the less immediate, higher level tasks, of map-ping and localisation at ≈ 200 Hz. This division of tasks is somewhatreminiscent of biological vision, in which the retina performs low levelcomputations, while the brain handles slower integrative functions. Onlysparse data is exchanged between these two ends, such as feature loca-tions, descriptor data, and sensor pose estimations. Essentially, each sideonly sends data which the other side requires to perform its respectivetask. This is a radical departure from the traditional visual pipeline,where dense image frames are streamed from sensor to PC. By compar-ison, our collaborative compute pipeline is extremely efficient, demon-strating a 10× improvement to speed and latency, > 1000× bandwidthand ∼ 20× reduction in power consumption per processed frame.