• NATALIA SOUSA

World-first In-cabin Monitoring on Neuromorphic Vision Systems

DTS®, a supplier of next-generation audio, imaging and sensing technology and a wholly-owned subsidiary of Xperi Holding Corporation, announced a world-first neuromorphic driver monitoring solution (DMS) powered by Prophesee Metavision® - Event-Based Vision sensor.


The Xperi team maps and adapts in-cabin monitoring (ICM) technologies to existing and future sensor centric trends. Using the raw feed from the Metavision® - Event-Based Vision sensor, the DTS AutoSense* team, developed driver-centric monitoring features such as gaze tracking, head pose, identification, and eyelid opening.


“Intelligent sensing is required by the next-generation in-cabin experience. The critical path to this is a sensor fusion-focused advanced research strategy,” said Petronel Bigioi, chief technology officer, Xperi. “DTS AutoSense’s advanced research team is centred on state-of-the-art sensing solutions, and neuromorphic sensing is one of the technologies that can shape the future of the industry.”



The neuromorphic sensors capture information at an equivalent frame rate of 10,000 frames per second without active illumination. This enables better low light performance for driver monitoring features and never-before-seen capabilities such as saccadic eye movement or micro-expressions, crucial next steps in the next-generation in-cabin experiences.

Xperi relies on an end-to-end data generation and training system built to address the particular needs of the sensor. The training data set was generated based on Xperi’s extensive computer vision infrastructure, reusing ground truth from the visible and near-infrared spectrums, synthesizing an entirely novel approach in neuromorphic based sensing.

“Being able to instantaneously detect the subtlest, almost imperceptible, face and eye motions can be lifesaving,” said Petronel Bigioi. “The low light powerful vision processing capabilities, fast response times and low power operation that this pioneering neuromorphic-based technology application enables makes it ideal for the safety-critical applications of driver monitoring systems, while also broadening the ability to further enhance and personalize the vehicle cabin.”

In addition to excellent low light performance and robustness to various illumination conditions, neuromorphic sensing directly addresses the privacy aspect specific to traditional imaging-based sensing (always-on cameras).

“DTS AutoSense’s innovative AI-enabled approach underscores the ability of Prophesee’s sensor solution to provide OEMs with an effective safety enhancement for vehicles of all types,” said Marc Rousseau, VP Products and Business Development at Prophesee. “We are delighted to collaborate on this first-of-its-kind technology with Xperi and its forward-thinking, high caliber team.”


*DTS AutoSense comprises a Driver Monitoring Solution (DMS) and an Occupancy Monitoring Solution (OMS), the first to be designed into passenger vehicles projected to be on the road in 2021. Working together they provide actionable insights into activity inside the vehicle, including the driver, passengers, pets and objects, to create a better, safer experience.