Facial Expression Reconstruction with Photo-Reflective Sensors Embedded in a Head-Mounted Display

Simple item page
dc.contributor.authorNakabayashi, Yukien_US
dc.contributor.authorNakamura, Fumihikoen_US
dc.contributor.authorMasai, Katsutoshien_US
dc.contributor.authorSugimoto, Makien_US
dc.contributor.editorJorge, Joaquim A.en_US
dc.contributor.editorSakata, Nobuchikaen_US
dc.date.accessioned2025-11-26T09:21:42Z
dc.date.available2025-11-26T09:21:42Z
dc.date.issued2025
dc.description.abstractReconstructing the 3D facial expressions of head-mounted display (HMD) wearers is essential for natural avatar communication in virtual reality (VR). Camera-based methods achieve high fidelity but involve heavy processing and privacy risks, whereas non-imaging sensors are lightweight and privacy-preserving but provide only sparse features. We propose a reconstruction system that learns high-dimensional 3D facial representations from camera images during training, but performs inference using only compact photo-reflective sensors embedded in the HMD. This design integrates the expressiveness of camera-based supervision with the efficiency and privacy of sensor-based operation. Experimental results show that our method accurately reconstructs 3D facial expressions from the sensor data, training with diverse wearing conditions is more effective than collecting more data under a single condition, and accuracy further improves with a dedicated mouth-shape predictor and lightweight personalization using small wearer-specific datasets.en_US
dc.description.sectionheadersMultisensory and Olfactory Interaction
dc.description.seriesinformationICAT-EGVE 2025 - International Conference on Artificial Reality and Telexistence and Eurographics Symposium on Virtual Environments
dc.identifier.doi10.2312/egve.20251347
dc.identifier.isbn978-3-03868-278-3
dc.identifier.issn1727-530X
dc.identifier.pages10 pages
dc.identifier.urihttps://doi.org/10.2312/egve.20251347
dc.identifier.urihttps://diglib.eg.org/handle/10.2312/egve20251347
dc.publisherThe Eurographics Associationen_US
dc.rightsAttribution 4.0 International License
dc.rights.urihttps://creativecommons.org/licenses/by/4.0/
dc.subjectCCS Concepts: Human-centered computing → Virtual reality; Interaction devices
dc.subjectHuman centered computing → Virtual reality
dc.subjectInteraction devices
dc.titleFacial Expression Reconstruction with Photo-Reflective Sensors Embedded in a Head-Mounted Displayen_US
Files
Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
egve20251347.pdf
Size:
8.19 MB
Format:
Adobe Portable Document Format
Download
Collections
ICAT-EGVE2025