A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesis

dc.contributor.authorPoirier-Ginter, Yohanen_US
dc.contributor.authorGauthier, Albanen_US
dc.contributor.authorPhilip, Julienen_US
dc.contributor.authorLalonde, Jean-Françoisen_US
dc.contributor.authorDrettakis, Georgeen_US
dc.contributor.editorGarces, Elenaen_US
dc.contributor.editorHaines, Ericen_US
dc.date.accessioned2024-06-25T10:19:34Z
dc.date.available2024-06-25T10:19:34Z
dc.date.issued2024
dc.description.abstractRelighting radiance fields is severely underconstrained for multi-view data, which is most often captured under a single illumination condition; It is especially hard for full scenes containing multiple objects. We introduce a method to create relightable radiance fields using such single-illumination data by exploiting priors extracted from 2D image diffusion models. We first fine-tune a 2D diffusion model on a multi-illumination dataset conditioned by light direction, allowing us to augment a single-illumination capture into a realistic - but possibly inconsistent - multi-illumination dataset from directly defined light directions. We use this augmented data to create a relightable radiance field represented by 3D Gaussian splats. To allow direct control of light direction for low-frequency lighting, we represent appearance with a multi-layer perceptron parameterized on light direction. To enforce multi-view consistency and overcome inaccuracies we optimize a per-image auxiliary feature vector. We show results on synthetic and real multi-view data under single illumination, demonstrating that our method successfully exploits 2D diffusion model priors to allow realistic 3D relighting for complete scenes.en_US
dc.description.number4
dc.description.sectionheadersRelighting
dc.description.seriesinformationComputer Graphics Forum
dc.description.volume43
dc.identifier.doi10.1111/cgf.15147
dc.identifier.issn1467-8659
dc.identifier.pages14 pages
dc.identifier.urihttps://doi.org/10.1111/cgf.15147
dc.identifier.urihttps://diglib.eg.org/handle/10.1111/cgf15147
dc.publisherThe Eurographics Association and John Wiley & Sons Ltd.en_US
dc.subjectKeywords: NeRF, Radiance Field, Relighting
dc.subjectNeRF
dc.subjectRadiance Field
dc.subjectRelighting
dc.titleA Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesisen_US
Files
Original bundle
Now showing 1 - 3 of 3
No Thumbnail Available
Name:
v43i4_03_cgf15147.pdf
Size:
93.72 MB
Format:
Adobe Portable Document Format
No Thumbnail Available
Name:
paper1038.mp4
Size:
156.74 MB
Format:
Video MP4
No Thumbnail Available
Name:
paper1038.zip
Size:
414.68 MB
Format:
Zip file
Collections