自單一視角圖像的環境光線預測｜國立清華大學博碩士論文庫

簡易檢索 / 詳目顯示

回結果列表

研究生：	呂尚霖 Lu, Shang-Lin
論文名稱：	自單一視角圖像的環境光線預測 LightDistill: Predicting View-Dependent Lighting from a Single Image
指導教授：	陳煥宗 Chen, Hwann-Tzong
口試委員:	賴尚宏 Lai, Shang-Hong 劉庭祿 Liu, Tyng-Luh
學位類別：	碩士 Master
系所名稱：	電機資訊學院 - 資訊工程學系 Computer Science
論文出版年：	2024
畢業學年度：	112
語文別：	中文
論文頁數：	42
中文關鍵詞：	三維重建、反射分解、光線探測、二維到三維、環境圖、單一圖像
外文關鍵詞：	3D reconstruction, reflection decomposition, lighting estimation, 2D to 3D, environment map, single image
相關次數：	點閱：37 下載：0
分享至:	分享至facebook 分享至twitter

查詢本校圖書館目錄查詢臺灣博碩士論文知識加值系統勘誤回報

我們提出了一種基於學習的方法，用於從單一圖像評估依據視角的環境照明。我們的方法（稱為 LightDistill）學習從可微幾何和紋理分解的框架中提取知識。目標是使用神經網路直接從單一輸入圖像預測環境圖，從而繞過以迭代最佳化求解的需求。我們基於物理的新策略自輸入圖像上取樣像素，並解耦照明顏色與局部光探測的分佈。實驗結果表明，我們提出的方法可以訓練神經網絡，在不到一秒的時間內從單個圖像中有效地導出高質量的環境圖—與耗時的基於優化的其他方法相比有顯著的改進，這些方法通常需要幾分鐘來獲得可比較的結果。

We present a learning-based method for estimating view-dependent environmental lighting from a single image. Our approach (dubbed LightDistill) learns to distill knowledge from a differentiable geometry and texture decomposition framework. The goal is to directly predict the environment map from a single input image using a neural network to bypass the need for solving iterative optimization. Our new physics-based strategy decouples the illumination color and the distribution of a local light probe from a sampled pixel on the input image. The experimental results show that our proposed method can train a neural network to efficiently derive a high-quality environment map from a single image in less than a second—a significant improvement over the timeconsuming optimization-based alternatives that often require a few minutes to obtain comparable results.

List of Tables                        3
List of Figures                        4
摘 要                             6
Abstract                            7
1 Introduction                         8
2 Related work                         10
3 Approach                         13
3.1    LightDistill Phase I: Decomposition under varying illuminations . . . . . . 14
3.1.1 Rendering equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.1.2 Applying nvdiffrec . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.1.3 Loss functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.2    LightDistill Phase II: Learning LightDistill . . . . . . . . . . . . . . . . . . . . . . . 16
3.2.1 Training LightDistill MLP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.2.2 Distribution of light probe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.2.3 Stacking light probes from sampled directions . . . . . . . . . . . . . . . . . . . . 18
3.2.4 Loss functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4 Experiments                         20
4.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.2 Main results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.3 Implementation details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
5 Conclusion                         29
A More Details and Results                     30
A.1 Deformation of geometry on Globe dataset . . . . . . . . . . . . . . . . . . . . . . . . 30
A.2 Gradient vanishing by nvdiffrec optimization . . . . . . . . . . . . . . . . . . . . . . 30
A.3 Structure of LightDistill MLP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
A.4 More qualitative results on ALP datasets . . . . . . . . . . . . . . . . . . . . . . . . . . 32
A.5 Comparisons of reconstructions on Gold dataset. . . . . . . . . . . . . . . . . . . . 33
A.6 More qualitative results on NeRD datasets . . . . . . . . . . . . . . . . . . . . . . . . 34
Bibliography                         40
                                

[1] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, 2022.
[2] M. Boss, R. Braun, V. Jampani, J. T. Barron, C. Liu, and H. P. Lensch. Nerd: Neural reflectance decomposition from image collections. In IEEE International Conference on Computer Vision (ICCV), 2021.
[3] M. Boss, V. Jampani, R. Braun, C. Liu, J. T. Barron, and H. P. Lensch. Neural-pil: Neural pre-integrated lighting for reflectance decomposition. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[4] J. Choi, S. Lee, H. Park, S. Jung, I. Kim, and J. Cho. MAIR: multi-view attention inverse rendering with 3d spatially-varying lighting estimation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, 2023.
[5] S. Fridovich-Keil, A. Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa. Plenoxels: Radiance fields without neural networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 5491–5500. IEEE, 2022.
[6] M. Gardner, Y. Hold-Geoffroy, K. Sunkavalli, C. Gagné, and J. Lalonde. Deep parametric indoor lighting estimation. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, 2019.
[7] M. Garon, K. Sunkavalli, S. Hadap, N. Carr, and J. Lalonde. Fast spatially-varying indoor lighting estimation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, 2019.
[8] J. Hasselgren, N. Hofmann, and J. Munkberg. Shape, light, and material decomposition from images using monte carlo rendering and denoising. In NeurIPS, 2022.
[9] Z. Li, M. Shafiei, R. Ramamoorthi, K. Sunkavalli, and M. Chandraker. Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and SVBRDF from a single image. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, 2020.
[10] Z. Li, Z. Xu, R. Ramamoorthi, K. Sunkavalli, and M. Chandraker. Learning to reconstruct shape and spatially-varying reflectance from a single image. ACM Trans. Graph., 37(6):269, 2018.
[11] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng.
Nerf: Representing scenes as neural radiance fields for view synthesis. In Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I, 2020.
[12] T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 41(4):102:1–102:15, 2022.
[13] J. Munkberg, W. Chen, J. Hasselgren, A. Evans, T. Shen, T. Müller, J. Gao, and S. Fidler. Extracting triangular 3d models, materials, and lighting from images. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, 2022.
[14] A. Sommer, U. Schwanecke, and E. Schömer. Real-time light estimation and neural soft shadows for AR indoor scenarios. J. WSCG, 31(1-2):71–79, 2023.
[15] P. P. Srinivasan, B. Deng, X. Zhang, M. Tancik, B. Mildenhall, and J. T. Barron. Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, 2021.
[16] P. P. Srinivasan, B. Mildenhall, M. Tancik, J. T. Barron, R. Tucker, and N. Snavely. Lighthouse: Predicting lighting volumes for spatially-coherent illumination. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, 2020.
[17] C. Sun, M. Sun, and H. Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, 2022.
[18] G. Wang, Y. Yang, C. C. Loy, and Z. Liu. Stylelight: HDR panorama generation for lighting estimation and editing. In Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV, 2022.
[19] H. Yu, S. Agarwala, C. Herrmann, R. Szeliski, N. Snavely, J. Wu, and D. Sun. Accidental light probes. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, 2023.
[20] K. Zhang, F. Luan, Q. Wang, K. Bala, and N. Snavely. Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, 2021.
[21] X. Zhang, P. P. Srinivasan, B. Deng, P. E. Debevec, W. T. Freeman, and J. T. Barron. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. CoRR, abs/2106.01970, 2021.

簡易檢索 / 詳目顯示

相關論文