Real-time Virtual-Try-On from a Single Example Image through Deep Inverse Graphics and Learned Differentiable Renderers
dc.contributor.author | Kips, Robin | en_US |
dc.contributor.author | Jiang, Ruowei | en_US |
dc.contributor.author | Ba, Sileye | en_US |
dc.contributor.author | Duke, Brendan | en_US |
dc.contributor.author | Perrot, Matthieu | en_US |
dc.contributor.author | Gori, Pietro | en_US |
dc.contributor.author | Bloch, Isabelle | en_US |
dc.contributor.editor | Chaine, Raphaëlle | en_US |
dc.contributor.editor | Kim, Min H. | en_US |
dc.date.accessioned | 2022-04-22T06:26:21Z | |
dc.date.available | 2022-04-22T06:26:21Z | |
dc.date.issued | 2022 | |
dc.description.abstract | Augmented reality applications have rapidly spread across online retail platforms and social media, allowing consumers to virtually try-on a large variety of products, such as makeup, hair dying, or shoes. However, parametrizing a renderer to synthesize realistic images of a given product remains a challenging task that requires expert knowledge. While recent work has introduced neural rendering methods for virtual try-on from example images, current approaches are based on large generative models that cannot be used in real-time on mobile devices. This calls for a hybrid method that combines the advantages of computer graphics and neural rendering approaches. In this paper, we propose a novel framework based on deep learning to build a real-time inverse graphics encoder that learns to map a single example image into the parameter space of a given augmented reality rendering engine. Our method leverages self-supervised learning and does not require labeled training data, which makes it extendable to many virtual try-on applications. Furthermore, most augmented reality renderers are not differentiable in practice due to algorithmic choices or implementation constraints to reach real-time on portable devices. To relax the need for a graphics-based differentiable renderer in inverse graphics problems, we introduce a trainable imitator module. Our imitator is a generative network that learns to accurately reproduce the behavior of a given non-differentiable renderer. We propose a novel rendering sensitivity loss to train the imitator, which ensures that the network learns an accurate and continuous representation for each rendering parameter. Automatically learning a differentiable renderer, as proposed here, could be beneficial for various inverse graphics tasks. Our framework enables novel applications where consumers can virtually try-on a novel unknown product from an inspirational reference image on social media. It can also be used by computer graphics artists to automatically create realistic rendering from a reference product image. | en_US |
dc.description.number | 2 | |
dc.description.sectionheaders | Learning for Rendering | |
dc.description.seriesinformation | Computer Graphics Forum | |
dc.description.volume | 41 | |
dc.identifier.doi | 10.1111/cgf.14456 | |
dc.identifier.issn | 1467-8659 | |
dc.identifier.pages | 29-40 | |
dc.identifier.pages | 12 pages | |
dc.identifier.uri | https://doi.org/10.1111/cgf.14456 | |
dc.identifier.uri | https://diglib.eg.org:443/handle/10.1111/cgf14456 | |
dc.publisher | The Eurographics Association and John Wiley & Sons Ltd. | en_US |
dc.subject | CCS Concepts: Computing methodologies --> Computer vision; Machine learning; Computer graphics | |
dc.subject | Computing methodologies | |
dc.subject | Computer vision | |
dc.subject | Machine learning | |
dc.subject | Computer graphics | |
dc.title | Real-time Virtual-Try-On from a Single Example Image through Deep Inverse Graphics and Learned Differentiable Renderers | en_US |
Files
Original bundle
1 - 3 of 3
Loading...
- Name:
- v41i2pp029-040.pdf
- Size:
- 13.68 MB
- Format:
- Adobe Portable Document Format
- Description:
No Thumbnail Available
- Name:
- lipstick_video_examples.mp4
- Size:
- 57.24 MB
- Format:
- Unknown data format