Research News

Accurate and Efficient 3D Motion Tracking Using Deep Learning

A new sensing method has made tracking movement easier and more efficient. A research group from Tohoku University has captured dexterous 3D motion data from a flexible magnetic flux sensor array, using deep learning and a structure-aware temporal bilateral filter.

"We can now track complex motions with higher accuracy," said Yoshifumi Kitamura, co-author of the study.

Dexterous 3D motion data can be used for multiple purposes: biologists can use the data to record detailed movements of small animals in their living environments, scientists can track the flow of fluids, and researchers can track finger movements and objects being manipulated by users in virtual reality.

Currently, optical cameras are the most prominent method of tracking movements. Yet optical cameras struggle with accuracy and reliability. If a small animal burrows away or if fingers or objects obscure the view, the camera will fail to detect the motion.

Magnetic tracking technology is also used for dexterous motion. However, even state-of-the-art magnetic systems face limitations. The classic tracking method creates bias and magnetic sources have a dead-angle problem or bulky markers.

Captured motion sequences with marker orientation kept vertical to the flux sensor plane and computed with the proposed method and numerical method. The traditional method has a certain bias while the proposed deep-learning approach solves the problem to achieve higher accuracy. ©Tohoku University

The research team invented their new method by applying a deep neural network and a novel structure-aware temporal bilateral filter on a new magnetic tracking principle. First, the neural networks learn the regression from the simulation flux values to the LC coils 3D configuration at any location and orientation.

The new filter further compensates the data to reconstruct smooth and accurate motion. Markers do not require batteries, so observation time can be maximized.

When the motion contains occasional dead-angle data, the result is wrong, but it can be reconstructed with filters. The performance of the proposed structure-aware temporal bilateral filter surpasses any other conventional filters because it leverages raw measurement of the sensors.©Tohoku University

As a result, the new integrated system can track multiple LC coils at 100Hz speed at millimetre level accuracy. Tracking loss due to dead-angle can be reconstructed because of the system's self-learning.

"The application of our research is widespread. Hand motions can be tracked to make creating smooth animations easier, markers can be put into fluids to track its flow, and tracking can be placed on small animals," added Kitamura.

Publication Details:

Title: Reconstruction of Dexterous 3D Motion Data from a Flexible Magnetic Sensor with Deep Learning and Structure-Aware Filtering
Authors: Jiawei Huang, Ryo Sugawara, Kinfung Chu, Taku Komura, Yoshifumi Kitamura
Journal: IEEE Transactions on Visualization and Computer Graphics
DOI: 10.1109/TVCG.2020.3031632

Press release in Japanese


Yoshifumi Kitamura
Deputy Director, Professor, Research Institute of Electrical Communication

Page Top