InstanceFusion: Real-time Instance-level 3D Reconstruction Using a Single RGBD Camera

Simple item page
dc.contributor.authorLu, Feixiangen_US
dc.contributor.authorPeng, Haotianen_US
dc.contributor.authorWu, Hongyuen_US
dc.contributor.authorYang, Junen_US
dc.contributor.authorYang, Xinhangen_US
dc.contributor.authorCao, Ruizhien_US
dc.contributor.authorZhang, Liangjunen_US
dc.contributor.authorYang, Ruigangen_US
dc.contributor.authorZhou, Binen_US
dc.contributor.editorEisemann, Elmar and Jacobson, Alec and Zhang, Fang-Lueen_US
dc.date.accessioned2020-10-29T18:51:02Z
dc.date.available2020-10-29T18:51:02Z
dc.date.issued2020
dc.description.abstractWe present InstanceFusion, a robust real-time system to detect, segment, and reconstruct instance-level 3D objects of indoor scenes with a hand-held RGBD camera. It combines the strengths of deep learning and traditional SLAM techniques to produce visually compelling 3D semantic models. The key success comes from our novel segmentation scheme and the efficient instancelevel data fusion, which are both implemented on GPU. Specifically, for each incoming RGBD frame, we take the advantages of the RGBD features, the 3D point cloud, and the reconstructed model to perform instance-level segmentation. The corresponding RGBD data along with the instance ID are then fused to the surfel-based models. In order to sufficiently store and update these data, we design and implement a new data structure using the OpenGL Shading Language. Experimental results show that our method advances the state-of-the-art (SOTA) methods in instance segmentation and data fusion by a big margin. In addition, our instance segmentation improves the precision of 3D reconstruction, especially in the loop closure. InstanceFusion system runs 20.5Hz on a consumer-level GPU, which supports a number of augmented reality (AR) applications (e.g., 3D model registration, virtual interaction, AR map) and robot applications (e.g., navigation, manipulation, grasping). To facilitate future research and reproduce our system more easily, the source code, data, and the trained model are released on Github: https://github.com/Fancomi2017/InstanceFusion.en_US
dc.description.number7
dc.description.sectionheadersVision Meets Graphics
dc.description.seriesinformationComputer Graphics Forum
dc.description.volume39
dc.identifier.doi10.1111/cgf.14157
dc.identifier.issn1467-8659
dc.identifier.pages433-445
dc.identifier.urihttps://doi.org/10.1111/cgf.14157
dc.identifier.urihttps://diglib.eg.org:443/handle/10.1111/cgf14157
dc.publisherThe Eurographics Association and John Wiley & Sons Ltd.en_US
dc.subjectComputing methodologies
dc.subjectScene understanding
dc.subjectVision for robotics
dc.subjectPerception
dc.titleInstanceFusion: Real-time Instance-level 3D Reconstruction Using a Single RGBD Cameraen_US
Files
Collections
39-Issue 7