Facial Expression Reconstruction with Photo-Reflective Sensors Embedded in a Head-Mounted Display
Loading...
Date
2025
Journal Title
Journal ISSN
Volume Title
Publisher
The Eurographics Association
Abstract
Reconstructing the 3D facial expressions of head-mounted display (HMD) wearers is essential for natural avatar communication in virtual reality (VR). Camera-based methods achieve high fidelity but involve heavy processing and privacy risks, whereas non-imaging sensors are lightweight and privacy-preserving but provide only sparse features. We propose a reconstruction system that learns high-dimensional 3D facial representations from camera images during training, but performs inference using only compact photo-reflective sensors embedded in the HMD. This design integrates the expressiveness of camera-based supervision with the efficiency and privacy of sensor-based operation. Experimental results show that our method accurately reconstructs 3D facial expressions from the sensor data, training with diverse wearing conditions is more effective than collecting more data under a single condition, and accuracy further improves with a dedicated mouth-shape predictor and lightweight personalization using small wearer-specific datasets.
Description
CCS Concepts: Human-centered computing → Virtual reality; Interaction devices
@inproceedings{10.2312:egve.20251347,
booktitle = {ICAT-EGVE 2025 - International Conference on Artificial Reality and Telexistence and Eurographics Symposium on Virtual Environments},
editor = {Jorge, Joaquim A. and Sakata, Nobuchika},
title = {{Facial Expression Reconstruction with Photo-Reflective Sensors Embedded in a Head-Mounted Display}},
author = {Nakabayashi, Yuki and Nakamura, Fumihiko and Masai, Katsutoshi and Sugimoto, Maki},
year = {2025},
publisher = {The Eurographics Association},
ISSN = {1727-530X},
ISBN = {978-3-03868-278-3},
DOI = {10.2312/egve.20251347}
}