Adversarial Texture Optimization from RGB-D Scans.
🔬 Downloadable at: https://arxiv.org/abs/1911.02466. CVPR 2020.
Introduces a new method to construct realistic color textures in RGB-D surface reconstruction problems. Specifically, this paper proposes an approach to produce photorealistic textures for approximate surfaces, even from misaligned images, by learning an objective function that is robust to these errors.
Key idea: to account for misalignments in a learned objective function of the texture optimization.
The key idea of this approach is to learn a patch-based conditional discriminator which guides the texture optimization to be tolerant to misalignments. Our discriminator takes a synthesized view and a real image, and evaluates whether the synthesized one is realistic, under a broadened definition of realism.
From an input RGB-D scan, we optimize for both its texture image and a learned texture objective function characterized by a discriminator network.
For each optimization iteration, we randomly select two input images,
To retrieve a texture, we jointly optimize the texture and the misalignment-tolerant metric. Our view-conditioned adversarial loss function:
The adversarial loss above is difficult to train alone, we add an L1 loss to the texture optimization to provide initial guidance:
And the final objective texture solution:
To enable the optimization of the RGB texture of a 3D model, we leverage a differentiable rendering to generate synthesized 'fake' views.
|Nearest patch loss||For each pixel |
|Perceptual metric||To evaluate perceptual quality.|
|Difference between generated textures and ground truth||Measure this according to sharpness and the average intensity of image gradients, in order to evaluate how robust the generated textures are to blurring artifacts without introducing noise artifacts.|
We cannot use standard image quality metrics such as MSE, PSNR or SSIM - image-quality-assessment, as they assume perfect alignment between target and the ground truth.
Texture generation in case of high camera or geometry errors.
Texture generation under increasing camera or geometry errors.