Deep Video-Based Performance Synthesis from Sparse Multi-View Capture
| dc.contributor.author | Chen, Mingjia | en_US |
| dc.contributor.author | Wang, Changbo | en_US |
| dc.contributor.author | Liu, Ligang | en_US |
| dc.contributor.editor | Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon | en_US |
| dc.date.accessioned | 2019-10-14T05:09:38Z | |
| dc.date.available | 2019-10-14T05:09:38Z | |
| dc.date.issued | 2019 | |
| dc.description.abstract | We present a deep learning based technique that enables novel-view videos of human performances to be synthesized from sparse multi-view captures. While performance capturing from a sparse set of videos has received significant attention, there has been relatively less progress which is about non-rigid objects (e.g., human bodies). The rich articulation modes of human body make it rather challenging to synthesize and interpolate the model well. To address this problem, we propose a novel deep learning based framework that directly predicts novel-view videos of human performances without explicit 3D reconstruction. Our method is a composition of two steps: novel-view prediction and detail enhancement. We first learn a novel deep generative query network for view prediction. We synthesize novel-view performances from a sparse set of just five or less camera videos. Then, we use a new generative adversarial network to enhance fine-scale details of the first step results. This opens up the possibility of high-quality low-cost video-based performance synthesis, which is gaining popularity for VA and AR applications. We demonstrate a variety of promising results, where our method is able to synthesis more robust and accurate performances than existing state-of-the-art approaches when only sparse views are available. | en_US |
| dc.description.number | 7 | |
| dc.description.sectionheaders | Image Based Rendering | |
| dc.description.seriesinformation | Computer Graphics Forum | |
| dc.description.volume | 38 | |
| dc.identifier.doi | 10.1111/cgf.13859 | |
| dc.identifier.issn | 1467-8659 | |
| dc.identifier.pages | 543-554 | |
| dc.identifier.uri | https://doi.org/10.1111/cgf.13859 | |
| dc.identifier.uri | https://diglib.eg.org:443/handle/10.1111/cgf13859 | |
| dc.publisher | The Eurographics Association and John Wiley & Sons Ltd. | en_US |
| dc.subject | Computing methodologies | |
| dc.subject | Computer graphics | |
| dc.subject | Image | |
| dc.subject | based rendering | |
| dc.title | Deep Video-Based Performance Synthesis from Sparse Multi-View Capture | en_US |