42-Issue 7
Permanent URI for this collection
Browse
Browsing 42-Issue 7 by Issue Date
Now showing 1 - 20 of 57
Results Per Page
Sort Options
Item Multi-scale Iterative Model-guided Unfolding Network for NLOS Reconstruction(The Eurographics Association and John Wiley & Sons Ltd., 2023) Su, Xiongfei; Hong, Yu; Ye, Juntian; Xu, Feihu; Yuan, Xin; Chaine, RaphaĆ«lle; Deng, Zhigang; Kim, Min H.Non-line-of-sight (NLOS) imaging can reconstruct hidden objects by analyzing diffuse reflection of relay surfaces, and is potentially used in autonomous driving, medical imaging and national defense. Despite the challenges of low signal-to-noise ratio (SNR) and ill-conditioned problem, NLOS imaging has developed rapidly in recent years. While deep neural networks have achieved impressive success in NLOS imaging, most of them lack flexibility when dealing with multiple spatial-temporal resolution and multi-scene images in practical applications. To bridge the gap between learning methods and physical priors, we present a novel end-to-end Multi-scale Iterative Model-guided Unfolding (MIMU), with superior performance and strong flexibility. Furthermore, we overcome the lack of real training data with a general architecture that can be trained in simulation. Unlike existing encoder-decoder architectures and generative adversarial networks, the proposed method allows for only one trained model adaptive for various dimensions, such as various sampling time resolution, various spatial resolution and multiple channels for colorful scenes. Simulation and real-data experiments verify that the proposed method achieves better reconstruction results both in quality and quantity than existing methods.Item Robust Novel View Synthesis with Color Transform Module(The Eurographics Association and John Wiley & Sons Ltd., 2023) Kim, Sang Min; Choi, Changwoon; Heo, Hyeongjun; Kim, Young Min; Chaine, RaphaĆ«lle; Deng, Zhigang; Kim, Min H.The advancements of the Neural Radiance Field (NeRF) and its variants have demonstrated remarkable capabilities in generating photo-realistic novel views from a small set of input images. While recent works suggest various techniques and model architectures that enhance speed or reconstruction quality, little attention is paid to exploring the RGB color space of input images. In this paper, we propose a universal color transform module that can maximally harness the captured evidence for the neural networks at hand. The color transform module utilizes an encoder-decoder framework that maps the RGB color space into a new latent space, enhancing the expressiveness of the input domain. We attach the encoder and the decoder at the input and output of a NeRF model of choice, respectively, and jointly optimize them to maintain the cycle consistency of the proposed transform, in addition to minimizing the reconstruction errors in the feature domain. Our comprehensive experiments demonstrate that the learned color space can significantly improve the quality of reconstructions compared to the conventional RGB representation. Its benefits are particularly pronounced in challenging scenarios characterized by low-light environments and scenes with low-textured regions. The proposed color transform pushes the boundaries of limitations in the input domain and offers a promising avenue for advancing the reconstruction capabilities of various neural representations. Source code is available at https://github.com/sangminkim-99/ColorTransformModule.Item DAFNet: Generating Diverse Actions for Furniture Interaction by Learning Conditional Pose Distribution(The Eurographics Association and John Wiley & Sons Ltd., 2023) Jin, Taeil; Lee, Sung-Hee; Chaine, RaphaĆ«lle; Deng, Zhigang; Kim, Min H.We present DAFNet, a novel data-driven framework capable of generating various actions for indoor environment interactions. By taking desired root and upper-body poses as control inputs, DAFNet generates whole-body poses suitable for furniture of various shapes and combinations. To enable the generation of diverse actions, we introduce an action predictor that automatically infers the probabilities of individual action types based on the control input and environment. The action predictor is learned in an unsupervised manner by training Gaussian Mixture Variational Autoencoder (GMVAE). Additionally, we propose a two-part normalizing flow-based pose generator that sequentially generates upper and lower body poses. This two-part model improves motion quality and the accuracy of satisfying conditions over a single model generating the whole body. Our experiments show that DAFNet can create continuous character motion for indoor scene scenarios, and both qualitative and quantitative evaluations demonstrate the effectiveness of our framework.Item Data-guided Authoring of Procedural Models of Shapes(The Eurographics Association and John Wiley & Sons Ltd., 2023) Hossain, Ishtiaque; Shen, I-Chao; Igarashi, Takeo; Kaick, Oliver van; Chaine, RaphaĆ«lle; Deng, Zhigang; Kim, Min H.Procedural models enable the generation of a large amount of diverse shapes by varying the parameters of the model. However, writing a procedural model for replicating a collection of reference shapes is difficult, requiring much inspection of the original and replicated shapes during the development of the model. In this paper, we introduce a data-guided method for aiding a programmer in creating a procedural model to replicate a collection of reference shapes. The user starts by writing an initial procedural model, and the system automatically predicts the model parameters for reference shapes, also grouping shapes by how well they are approximated by the current procedural model. The user can then update the procedural model based on the given feedback and iterate the process. Our system thus automates the tedious process of discovering the parameters that replicate reference shapes, allowing the programmer to focus on designing the high-level rules that generate the shapes. We demonstrate through qualitative examples and a user study that our method is able to speed up the development time for creating procedural models of 2D and 3D man-made shapes.Item D-Cloth: Skinning-based Cloth Dynamic Prediction with a Three-stage Network(The Eurographics Association and John Wiley & Sons Ltd., 2023) Li, Yu Di; Tang, Min; Chen, Xiao Rui; Yang, Yun; Tong, Ruo Feng; An, Bai Lin; Yang, Shuang Cai; Li, Yao; Kou, Qi Long; Chaine, RaphaĆ«lle; Deng, Zhigang; Kim, Min H.We propose a three-stage network that utilizes a skinning-based model to accurately predict dynamic cloth deformation. Our approach decomposes cloth deformation into three distinct components: static, coarse dynamic, and wrinkle dynamic components. To capture these components, we train our three-stage network accordingly. In the first stage, the static component is predicted by constructing a static skinning model that incorporates learned joint increments and skinning weight increments. Then, in the second stage, the coarse dynamic component is added to the static skinning model by incorporating serialized skeleton information. Finally, in the third stage, the mesh sequence stage refines the prediction by incorporating the wrinkle dynamic component using serialized mesh information. We have implemented our network and used it in a Unity game scene, enabling real-time prediction of cloth dynamics. Our implementation achieves impressive prediction speeds of approximately 3.65ms using an NVIDIA GeForce RTX 3090 GPU and 9.66ms on an Intel i7-7700 CPU. Compared to SOTA methods, our network excels in accurately capturing fine dynamic cloth deformations.Item Error-bounded Image Triangulation(The Eurographics Association and John Wiley & Sons Ltd., 2023) Fang, Zhi-Duo; Guo, Jia-Peng; Xiao, Yanyang; Fu, Xiao-Ming; Chaine, RaphaĆ«lle; Deng, Zhigang; Kim, Min H.We propose a novel image triangulation method to reduce the complexity of image triangulation under the color error-bounded constraint and the triangle quality constraint. Meanwhile, we realize a variety of visual effects by supporting different types of triangles (e.g., linear or curved) and color approximation functions (e.g., constant, linear, or quadratic). To adapt to these discontinuous and combinatorial objectives and constraints, we formulate it as a constrained optimization problem that is solved by a series of tailored local remeshing operations. The feasibility and practicability of our method are demonstrated over various types of images, such as organisms, landscapes, portraits and cartoons. Compared to state-of-the-art methods, our method generates far fewer triangles for the same color error or much smaller color errors using the same number of triangles.Item Interactive Authoring of Terrain using Diffusion Models(The Eurographics Association and John Wiley & Sons Ltd., 2023) Lochner, Joshua; Gain, James; Perche, Simon; Peytavie, Adrien; Galin, Eric; GuĆ©rin, Eric; Chaine, RaphaĆ«lle; Deng, Zhigang; Kim, Min H.Generating heightfield terrains is a necessary precursor to the depiction of computer-generated natural scenes in a variety of applications. Authoring such terrains is made challenging by the need for interactive feedback, effective user control, and perceptually realistic output encompassing a range of landforms.We address these challenges by developing a terrain-authoring framework underpinned by an adaptation of diffusion models for conditional image synthesis, trained on real-world elevation data. This framework supports automated cleaning of the training set; authoring control through style selection and feature sketches; the ability to import and freely edit pre-existing terrains, and resolution amplification up to the limits of the source data. Our framework improves on previous machine-learning approaches by: expanding landform variety beyond mountainous terrain to encompass cliffs, canyons, and plains; providing a better balance between terseness and specificity in user control, and improving the fidelity of global terrain structure and perceptual realism. This is demonstrated through drainage simulations and a user study testing the perceived realism for different classes of terrain. The full source code, blender add-on, and pretrained models are available.Item Data-Driven Ink Painting Brushstroke Rendering(The Eurographics Association and John Wiley & Sons Ltd., 2023) Madono, Koki; Simo-Serra, Edgar; Chaine, RaphaĆ«lle; Deng, Zhigang; Kim, Min H.Although digital painting has advanced much in recent years, there is still a significant divide between physically drawn paintings and purely digitally drawn paintings. These differences arise due to the physical interactions between the brush, ink, and paper, which are hard to emulate in the digital domain. Most ink painting approaches have focused on either using heuristics or physical simulation to attempt to bridge the gap between digital and analog, however, these approaches are still unable to capture the diversity of painting effects, such as ink fading or blotting, found in the real world. In this work, we propose a data-driven approach to generate ink paintings based on a semi-automatically collected high-quality real-world ink painting dataset. We use a multi-camera robot-based setup to automatically create a diversity of ink paintings, which allows for capturing the entire process in high resolution, including capturing detailed brush motions and drawing results. To ensure high-quality capture of the painting process, we calibrate the setup and perform occlusion-aware blending to capture all the strokes in high resolution in a robust and efficient way. Using our new dataset, we propose a recursive deep learning-based model to reproduce the ink paintings stroke by stroke while capturing complex ink painting effects such as bleeding and mixing. Our results corroborate the fidelity of the proposed approach to real hand-drawn ink paintings in comparison with existing approaches. We hope the availability of our dataset will encourage new research on digital realistic ink painting techniques.Item An Efficient Self-supporting Infill Structure for Computational Fabrication(The Eurographics Association and John Wiley & Sons Ltd., 2023) Wang, Shengfa; Liu, Zheng; Hu, Jiangbei; Lei, Na; Luo, Zhongxuan; Chaine, RaphaĆ«lle; Deng, Zhigang; Kim, Min H.Efficiently optimizing the internal structure of 3D printing models is a critical focus in the field of industrial manufacturing, particularly when designing self-supporting structures that offer high stiffness and lightweight characteristics. To tackle this challenge, this research introduces a novel approach featuring a self-supporting polyhedral structure and an efficient optimization algorithm. Specifically, the internal space of the model is filled with a combination of self-supporting octahedrons and tetrahedrons, strategically arranged to maximize structural integrity. Our algorithm optimizes the wall thickness of the polyhedron elements to satisfy specific stiffness requirements, while ensuring efficient alignment of the filled structures in finite element calculations. Our approach results in a considerable decrease in optimization time. The optimization process is stable, converges rapidly, and consistently delivers effective results. Through a series of experiments, we have demonstrated the effectiveness and efficiency of our method in achieving the desired design objectivesItem OptCtrlPoints: Finding the Optimal Control Points for Biharmonic 3D Shape Deformation(The Eurographics Association and John Wiley & Sons Ltd., 2023) Kim, Kunho; Uy, Mikaela Angelina; Paschalidou, Despoina; Jacobson, Alec; Guibas, Leonidas J.; Sung, Minhyuk; Chaine, RaphaĆ«lle; Deng, Zhigang; Kim, Min H.We propose OPTCTRLPOINTS, a data-driven framework designed to identify the optimal sparse set of control points for reproducing target shapes using biharmonic 3D shape deformation. Control-point-based 3D deformation methods are widely utilized for interactive shape editing, and their usability is enhanced when the control points are sparse yet strategically distributed across the shape. With this objective in mind, we introduce a data-driven approach that can determine the most suitable set of control points, assuming that we have a given set of possible shape variations. The challenges associated with this task primarily stem from the computationally demanding nature of the problem. Two main factors contribute to this complexity: solving a large linear system for the biharmonic weight computation and addressing the combinatorial problem of finding the optimal subset of mesh vertices. To overcome these challenges, we propose a reformulation of the biharmonic computation that reduces the matrix size, making it dependent on the number of control points rather than the number of vertices. Additionally, we present an efficient search algorithm that significantly reduces the time complexity while still delivering a nearly optimal solution. Experiments on SMPL, SMAL, and DeformingThings4D datasets demonstrate the efficacy of our method. Our control points achieve better template-to-target fit than FPS, random search, and neural-network-based prediction. We also highlight the significant reduction in computation time from days to approximately 3 minutes.Item Semantics-guided Generative Diffusion Model with a 3DMM Model Condition for Face Swapping(The Eurographics Association and John Wiley & Sons Ltd., 2023) Liu, Xiyao; Liu, Yang; Zheng, Yuhao; Yang, Ting; Zhang, Jian; Wang, Victoria; Fang, Hui; Chaine, RaphaĆ«lle; Deng, Zhigang; Kim, Min H.Face swapping is a technique that replaces a face in a target media with another face of a different identity from a source face image. Currently, research on the effective utilisation of prior knowledge and semantic guidance for photo-realistic face swapping remains limited, despite the impressive synthesis quality achieved by recent generative models. In this paper, we propose a novel conditional Denoising Diffusion Probabilistic Model (DDPM) enforced by a two-level face prior guidance. Specifically, it includes (i) an image-level condition generated by a 3D Morphable Model (3DMM), and (ii) a high-semantic level guidance driven by information extracted from several pre-trained attribute classifiers, for high-quality face image synthesis. Although swapped face image from 3DMM does not achieve photo-realistic quality on its own, it provides a strong image-level prior, in parallel with high-level face semantics, to guide the DDPM for high fidelity image generation. The experimental results demonstrate that our method outperforms state-of-the-art face swapping methods on benchmark datasets in terms of its synthesis quality, and capability to preserve the target face attributes and swap the source face identity.Item Deep Shape and SVBRDF Estimation using Smartphone Multi-lens Imaging(The Eurographics Association and John Wiley & Sons Ltd., 2023) Fan, Chongrui; Lin, Yiming; Ghosh, Abhijeet; Chaine, RaphaĆ«lle; Deng, Zhigang; Kim, Min H.We present a deep neural network-based method that acquires high-quality shape and spatially varying reflectance of 3D objects using smartphone multi-lens imaging. Our method acquires two images simultaneously using a zoom lens and a wide angle lens of a smartphone under either natural illumination or phone flash conditions, effectively functioning like a single-shot method. Unlike traditional multi-view stereo methods which require sufficient differences in viewpoint and only estimate depth at a certain coarse scale, our method estimates fine-scale depth by utilising an optical-flow field extracted from subtle baseline and perspective due to different optics in the two images captured simultaneously. We further guide the SVBRDF estimation using the estimated depth, resulting in superior results compared to existing single-shot methods.Item Multi-Level Implicit Function for Detailed Human Reconstruction by Relaxing SMPL Constraints(The Eurographics Association and John Wiley & Sons Ltd., 2023) Ma, Xikai; Zhao, Jieyu; Teng, Yiqing; Yao, Li; Chaine, RaphaĆ«lle; Deng, Zhigang; Kim, Min H.Aiming at enhancing the rationality and robustness of the results of single-view image-based human reconstruction and acquiring richer surface details, we propose a multi-level reconstruction framework based on implicit functions.This framework first utilizes the predicted SMPL model (Skinned Multi-Person Linear Model) as a prior to further predict consistent 2.5D sketches (depth map and normal map), and then obtains a coarse reconstruction result through an Implicit Function fitting network (IF-Net). Subsequently, with a pixel-aligned feature extraction module and a fine IF-Net, the strong constraints imposed by SMPL are relaxed to add more surface details to the reconstruction result and remove noise. Finally, to address the trade-off between surface details and rationality under complex poses, we propose a novel fusion repair algorithm that reuses existing information. This algorithm compensates for the missing parts of the fine reconstruction results with the coarse reconstruction results, leading to a robust, rational, and richly detailed reconstruction. The final experiments prove the effectiveness of our method and demonstrate that it achieves the richest surface details while ensuring rationality. The project website can be found at https://github.com/MXKKK/2.5D-MLIF.Item Enhancing Low-Light Images: A Variation-based Retinex with Modified Bilateral Total Variation and Tensor Sparse Coding(The Eurographics Association and John Wiley & Sons Ltd., 2023) Yang, Weipeng; Gao, Hongxia; Zou, Wenbin; Huang, Shasha; Chen, Hongsheng; Ma, Jianliang; Chaine, RaphaĆ«lle; Deng, Zhigang; Kim, Min H.Low-light conditions often result in the presence of significant noise and artifacts in captured images, which can be further exacerbated during the image enhancement process, leading to a decrease in visual quality. This paper aims to present an effective low-light image enhancement model based on the variation Retinex model that successfully suppresses noise and artifacts while preserving image details. To achieve this, we propose a modified Bilateral Total Variation to better smooth out fine textures in the illuminance component while maintaining weak structures. Additionally, tensor sparse coding is employed as a regularization term to remove noise and artifacts from the reflectance component. Experimental results on extensive and challenging datasets demonstrate the effectiveness of the proposed method, exhibiting superior or comparable performance compared to state-ofthe- art approaches. Code, dataset and experimental results are available at https://github.com/YangWeipengscut/BTRetinex.Item Fabricatable 90Ā° Pop-ups: Interactive Transformation of a 3D Model into a Pop-up Structure(The Eurographics Association and John Wiley & Sons Ltd., 2023) Fujikawa, Junpei; Ijiri, Takashi; Chaine, RaphaĆ«lle; Deng, Zhigang; Kim, Min H.Ninety-degree pop-ups are a type of papercraft on which a three-dimensional (3D) structure pops up when the angle of the base fold is 90Ā°. They are fabricated by cutting and creasing a single sheet of paper. Traditional 90Ā° pop-ups are limited to 3D shapes only comprising planar shapes because they are made of paper. In this paper, we present novel pop-ups, fabricatable 90Ā° pop-ups that employ the 90Ā° pop-up mechanism, consist of components with curved shapes, and can be fabricatable using a 3D printer. We propose a method for converting a 3D model into a fabricatable 90Ā° pop-up. The user first interactively designs a layout of pop-up components, and the system automatically deforms the components using the 3D model. Because the generated pop-ups contain necessary cuts and folds, no additional assembly process is required. To demonstrate the feasibility of the proposed method, we designed and fabricated various 90Ā° pop-ups using a 3D printer.Item Neural Shading Fields for Efficient Facial Inverse Rendering(The Eurographics Association and John Wiley & Sons Ltd., 2023) Rainer, Gilles; Bridgeman, Lewis; Ghosh, Abhijeet; Chaine, RaphaĆ«lle; Deng, Zhigang; Kim, Min H.Given a set of unstructured photographs of a subject under unknown lighting, 3D geometry reconstruction is relatively easy, but reflectance estimation remains a challenge. This is because it requires disentangling lighting from reflectance in the ambiguous observations. Solutions exist leveraging statistical, data-driven priors to output plausible reflectance maps even in the underconstrained single-view, unknown lighting setting. We propose a very low-cost inverse optimization method that does not rely on data-driven priors, to obtain high-quality diffuse and specular, albedo and normal maps in the setting of multi-view unknown lighting. We introduce compact neural networks that learn the shading of a given scene by efficiently finding correlations in the appearance across the face. We jointly optimize the implicit global illumination of the scene in the networks with explicit diffuse and specular reflectance maps that can subsequently be used for physically-based rendering. We analyze the veracity of results on ground truth data, and demonstrate that our reflectance maps maintain more detail and greater personal identity than state-of-the-art deep learning and differentiable rendering methods.Item Structure Learning for 3D Point Cloud Generation from Single RGB Images(The Eurographics Association and John Wiley & Sons Ltd., 2023) Charrada, Tarek Ben; Laga, Hamid; Tabia, Hedi; Chaine, RaphaĆ«lle; Deng, Zhigang; Kim, Min H.3D point clouds can represent complex 3D objects of arbitrary topologies and with fine-grained details. They are, however, hard to regress from images using convolutional neural networks, making tasks such as 3D reconstruction from monocular RGB images challenging. In fact, unlike images and volumetric grids, point clouds are unstructured and thus lack proper parameterization, which makes them difficult to process using convolutional operations. Existing point-based 3D reconstruction methods that tried to address this problem rely on complex end-to-end architectures with high computational costs. Instead, we propose in this paper a novel mechanism that decouples the 3D reconstruction problem from the structure (or parameterization) learning task, making the 3D reconstruction of objects of arbitrary topologies tractable and thus easier to learn. We achieve this using a novel Teacher-Student network where the Teacher learns to structure the point clouds. The Student then harnesses the knowledge learned by the Teacher to efficiently regress accurate 3D point clouds. We train the Teacher network using 3D ground-truth supervision and the Student network using the Teacherā's annotations. Finally, we employ a novel refinement network to overcome the upper-bound performance that is set by the Teacher network. Our extensive experiments on ShapeNet and Pix3D benchmarks, and on in-the-wild images demonstrate that the proposed approach outperforms previous methods in terms of reconstruction accuracy and visual quality.Item Dissection Puzzles Composed of Multicolor Polyominoes(The Eurographics Association and John Wiley & Sons Ltd., 2023) Kita, Naoki; Chaine, RaphaĆ«lle; Deng, Zhigang; Kim, Min H.Dissection puzzles leverage geometric dissections, wherein a set of puzzle pieces can be reassembled in various configurations to yield unique geometric figures. Mathematically, a dissection between two 2D polygons can always be established. Consequently, researchers and puzzle enthusiasts strive to design unique dissection puzzles using the fewest pieces feasible. In this study, we introduce novel dissection puzzles crafted with multi-colored polyominoes. Diverging from the traditional aim of establishing geometric dissection between two 2D polygons with the minimal piece count, we seek to identify a common pool of polyomino pieces with colored faces that can be configured into multiple distinct shapes and appearances. Moreover, we offer a method to identify an optimized sequence for rearranging pieces from one form to another, thus minimizing the total relocation distance. This approach can guide users in puzzle assembly and lessen their physical exertion when manually reconfiguring pieces. It could potentially also decrease power consumption when pieces are reorganized using robotic assistance. We showcase the efficacy of our proposed approach through a wide range of shapes and appearances.Item Robust Distribution-aware Color Correction for Single-shot Images(The Eurographics Association and John Wiley & Sons Ltd., 2023) Dhillon, Daljit Singh J.; Joshi, Parisha; Baron, Jessica; Patterson, Eric K.; Chaine, RaphaĆ«lle; Deng, Zhigang; Kim, Min H.Color correction for photographed images is an ill-posed problem. It is also a crucial initial step towards material acquisition for inverse rendering methods or pipelines. Several state-of-the-art methods rely on reducing color differences for imaged reference color chart blocks of known color values to devise or optimize their solution. In this paper, we first establish through simulations the limitation of this minimality criteria which in principle results in overfitting. Next, we study and propose a few spatial distribution measures to augment the evaluation criteria. Thereafter, we propose a novel patch-based, white-point centric approach that processes luminance and chrominance information separately to improve on the color matching task. We compare our method qualitatively with several state-of-the art methods using our augmented evaluation criteria along with quantitative examinations. Finally, we perform rigorous experiments and demonstrate results to clearly establish the benefits of our proposed method.Item Meso-Skeleton Guided Hexahedral Mesh Design(The Eurographics Association and John Wiley & Sons Ltd., 2023) Viville, Paul; Kraemer, Pierre; Bechmann, Dominique; Chaine, RaphaĆ«lle; Deng, Zhigang; Kim, Min H.We present a novel approach for the generation of hexahedral meshes in a volume domain given its meso-skeleton. This compact representation of the topology and geometry, composed of both curve and surface parts, is used to produce a raw decomposition of the domain into hexahedral blocks. Analysis of the different local configurations of the skeleton leads to the construction of a set of connection surfaces that are used as a scaffold onto which the hexahedral blocks are assembled. These local configurations of the skeleton completely determine the singularities of the final mesh, and by following the skeleton, the geometry of the produced mesh naturally follows the geometry of the domain. Depending on the end user needs, the obtained mesh can be further adapted, refined or optimized, for example to better fit the boundary of the domain. Our algorithm does not involve the resolution of any global problem, most decisions are taken locally and it is thus highly suitable for parallel processing. This efficiency allows the user to stay in the loop for the correction or edition of the meso-skeleton for which a first sketch can be given by an existing automatic extraction algorithm.