CLIP-based Neural Neighbor Style Transfer for 3D Assets

No Thumbnail Available
Date
2023
Journal Title
Journal ISSN
Volume Title
Publisher
The Eurographics Association
Abstract
We present a method for transferring the style from a set of images to the texture of a 3D object. The texture of an asset is optimized with a differentiable renderer and losses using pretrained deep neural networks. More specifically, we utilize a nearest-neighbor feature matching (NNFM) loss with CLIP-ResNet50 that we extend to support multiple style images. We improve color accuracy and artistic control with an extra loss on user-provided or automatically extracted color palettes. Finally, we show that a CLIP-based NNFM loss provides a different appearance over a VGG-based one by focusing more on textural details over geometric shapes. However, we note that user preference is still subjective.
Description

CCS Concepts: Computing methodologies → Appearance and texture representations; Rasterization; Supervised learning by regression

        
@inproceedings{
10.2312:egs.20231006
, booktitle = {
Eurographics 2023 - Short Papers
}, editor = {
Babaei, Vahid
and
Skouras, Melina
}, title = {{
CLIP-based Neural Neighbor Style Transfer for 3D Assets
}}, author = {
Mishra, Shailesh
and
Granskog, Jonathan
}, year = {
2023
}, publisher = {
The Eurographics Association
}, ISSN = {
1017-4656
}, ISBN = {
978-3-03868-209-7
}, DOI = {
10.2312/egs.20231006
} }
Citation