Semantic Stylization and Shading via Segmentation Atlas utilizing Deep Learning Approaches
Loading...
Date
2024
Journal Title
Journal ISSN
Volume Title
Publisher
The Eurographics Association
Abstract
We present a novel hybrid approach for semantic stylization of surface materials of 3D models while preserving shading. Based on a hybrid approach that builds on directly applying style transfer on the object surface obtained by learning-based or traditional methods such as 3D scanners or structured light systems, thereby overcoming artifacts like halos, ghosting or lacking quality of the geometric representation produced by other 3D stylization methods. For this purpose, our methods involves (i) the initial generation of a segmentation map parameterized over the object surface inferred based on a deep-learning-based foundation model to guide the stylization and shading of different regions of the 3D model, and (ii) a subsequent 2D style transfer that allows the exchange or stylization of surface materials in high quality. By delivering high-quality semantic perceptive reconstructions in a shorter timeframe than current approaches using manual 3D segmentation and stylization, our approach holds significant potential for various application scenarios including creative design, architecture and cultural heritage.
Description
CCS Concepts: Computing methodologies → Computer graphics; Reflectance modeling; Image manipulation; Scene understanding
@inproceedings{10.2312:stag.20241352,
booktitle = {Smart Tools and Applications in Graphics - Eurographics Italian Chapter Conference},
editor = {Caputo, Ariel and Garro, Valeria and Giachetti, Andrea and Castellani, Umberto and Dulecha, Tinsae Gebrechristos},
title = {{Semantic Stylization and Shading via Segmentation Atlas utilizing Deep Learning Approaches}},
author = {Sinha, Saptarshi Neil and Kühn, Paul Julius and Rojtberg, Pavel and Graf, Holger and Kuijper, Arjan and Weinmann, Michael},
year = {2024},
publisher = {The Eurographics Association},
ISSN = {2617-4855},
ISBN = {978-3-03868-265-3},
DOI = {10.2312/stag.20241352}
}