Dressi: A Hardware-Agnostic Differentiable Renderer with Reactive Shader Packing and Soft Rasterization

Takimoto, Yusuke; Sato, Hiroyuki; Takehara, Hikari; Uragaki, Keishiro; Tawara, Takehiro; Liang, Xiao; Oku, Kentaro; Kishimoto, Wataru; Zheng, Bo

Dressi: A Hardware-Agnostic Differentiable Renderer with Reactive Shader Packing and Soft Rasterization

Files

v41i2pp013-027.pdf (19.12 MB)

paper1042_3.mp4 (50.46 MB)

paper1042_4.pdf (2.56 MB)

Date

2022

Authors

Takimoto, Yusuke
Sato, Hiroyuki
Takehara, Hikari
Uragaki, Keishiro
Tawara, Takehiro
Liang, Xiao
Oku, Kentaro
Kishimoto, Wataru
Zheng, Bo

Publisher

The Eurographics Association and John Wiley & Sons Ltd.

Abstract

Differentiable rendering (DR) enables various computer graphics and computer vision applications through gradient-based optimization with derivatives of the rendering equation. Most rasterization-based approaches are built on general-purpose automatic differentiation (AD) libraries and DR-specific modules handcrafted using CUDA. Such a system design mixes DR algorithm implementation and algorithm building blocks, resulting in hardware dependency and limited performance. In this paper, we present a practical hardware-agnostic differentiable renderer called Dressi, which is based on a new full AD design. The DR algorithms of Dressi are fully written in our Vulkan-based AD for DR, Dressi-AD, which supports all primitive operations for DR. Dressi-AD and our inverse UV technique inside it bring hardware independence and acceleration by graphics hardware. Stage packing, our runtime optimization technique, can adapt hardware constraints and efficiently execute complex computational graphs of DR with reactive cache considering the render pass hierarchy of Vulkan. HardSoftRas, our novel rendering process, is designed for inverse rendering with a graphics pipeline. Under the limited functionalities of the graphics pipeline, HardSoftRas can propagate the gradients of pixels from the screen space to far-range triangle attributes. Our experiments and applications demonstrate that Dressi establishes hardware independence, high-quality and robust optimization with fast speed, and photorealistic rendering.

CCS Concepts: Computing methodologies --> Rasterization; Shape inference

        @article{10.1111:cgf.14455
,
journal = {Computer Graphics Forum},
title = {{Dressi: A Hardware-Agnostic Differentiable Renderer with Reactive Shader Packing and Soft Rasterization
}},
author = {Takimoto, Yusuke and 
Sato, Hiroyuki and 
Takehara, Hikari and 
Uragaki, Keishiro and 
Tawara, Takehiro and 
Liang, Xiao and 
Oku, Kentaro and 
Kishimoto, Wataru and 
Zheng, Bo
},
year = {2022
},
publisher = {The Eurographics Association and John Wiley & Sons Ltd.
},
ISSN = {1467-8659
},
DOI = {10.1111/cgf.14455
}
}