research-article
Authors: Ludwic Leonard and Rüdiger Westermann
Volume 119, Issue C
Published: 18 July 2024 Publication History
- 1citation
- 0
- Downloads
Metrics
Total Citations1Total Downloads0Last 12 Months0
Last 6 weeks0
New Citation Alert added!
This alert has been successfully added and will be sent to:
You will be notified whenever a record that you have chosen has been cited.
To manage your alert preferences, click on the button below.
Manage my Alerts
New Citation Alert!
Please log in to your account
- View Options
- References
- Media
- Tables
- Share
Abstract
Image-based reconstruction of a three-dimensional heterogeneous density field in the presence of multiple scattering of light is intrinsically under-constrained. This leads to reconstructions that look similar to the ground truth when rendered, but the recovered field is often far off the real one. We shed light on the sources of uncertainty in the reconstruction process which are responsible for this ambiguity, and propose the following approaches to improve the reconstruction quality: Firstly, we introduce a new path sampling strategy, which yields more accurate estimates of the gradients of the extinction field. Secondly, we build upon the observation that the variance in the loss computation is one source of bias in the optimization process. To reduce this variance in the primal estimator, we propose exploiting temporal coherence by reusing previously rendered images. All this is coupled with a constraint on spatial object occupancy, which restricts the problem to a reconstructed shape prior. In a number of examples we demonstrate that compared to existing approaches the proposed reconstruction pipeline leads to improved accuracy of the reconstructed density fields.
Graphical abstract
Display Omitted
Highlights
•
Reconstruction of volumetric media in the presence of multiple light scattering.
•
Physically based differentiable volume rendering with improved path sampling.
•
Initial solution built with a simpler direction-dependent emission field.
•
Occupancy mask used for constraining the volume in object space.
•
Loss variance is reduced by using an exponential moving average.
References
[1]
Nimier-David M., Müller T., Keller A., Jakob W., Unbiased inverse volume rendering with differential trackers, ACM Trans Graph 41 (4) (2022) 1–20. Publisher: ACM New York, NY, USA.
[2]
Nimier-David M., Vicini D., Zeltner T., Jakob W., Mitsuba 2: A retargetable forward and inverse renderer, ACM Trans Graph 38 (6) (2019) 1–17. Publisher: ACM New York, NY, USA.
[3]
Nimier-David M., Speierer S., Ruiz B., Jakob W., Radiative backpropagation: an adjoint method for lightning-fast differentiable rendering, ACM Trans Graph 39 (4) (2020) 1–15,. Publisher: ACM New York, NY, USA.
Digital Library
[4]
Feldkamp L.A., Davis L.C., Kress J.W., Practical cone-beam algorithm, Josa a 1 (6) (1984) 612–619. Publisher: Optica Publishing Group.
[5]
Natterer F., The mathematics of computerized tomography, SIAM, 2001.
[6]
Kak A.C., Slaney M., Principles of computerized tomographic imaging, SIAM, 2001.
[7]
Gordon R., A tutorial on ART (algebraic reconstruction techniques), IEEE Trans Nucl Sci 21 (3) (1974) 78–93.
[8]
Andersen A.H., Kak A.C., Simultaneous algebraic reconstruction technique (SART): a superior implementation of the ART algorithm, Ultrason Imaging 6 (1) (1984) 81–94.
[9]
Andersen A.H., Algebraic reconstruction in CT from limited views, IEEE Trans Med Imaging 8 (1) (1989) 50–55.
[10]
Katsevich A., Theoretically exact filtered backprojection-type inversion algorithm for spiral CT, SIAM J Appl Math 62 (6) (2002) 2012–2026.
[11]
Katsevich A., An improved exact filtered backprojection algorithm for spiral computed tomography, Adv Appl Math 32 (4) (2004) 681–697.
[12]
Karimi D., Ward R.K., A hybrid stochastic-deterministic gradient descent algorithm for image reconstruction in cone-beam computed tomography, Biomed Phys Eng Express 2 (1) (2016).
[13]
Tang J., Egiazarian K., Golbabaee M., Davies M., The practicality of stochastic optimization in imaging inverse problems, IEEE Trans Comput Imaging 6 (2020) 1471–1485.
[14]
Ntziachristos V., Fluorescence molecular imaging, Annu Rev Biomed Eng 8 (2006) 1–33. Publisher: Annual Reviews.
[15]
Stuker F., Ripoll J., Rudin M., Fluorescence molecular tomography: principles and potential for pharmaceutical research, Pharmaceutics 3 (2) (2011) 229–274. Publisher: MDPI.
[16]
Huang D., Liu Y., Wiscombe W., Determination of cloud liquid water distribution using 3D cloud tomography, J Geophys Res: Atmos 113 (D13) (2008) Publisher: Wiley Online Library.
[17]
LevisA, SchechnerYY, AidesA, DavisAB. Airborne three-dimensional cloud tomography. In: Proceedings of the IEEE international conference on computer vision. 2015, p. 3379–87.
[18]
Sitzmann V., Zollhoefer M., Wetzstein G., Scene representation networks: Continuous 3D-structure-aware neural scene representations, in: Advances in neural information processing systems, vol. 32, 32, Curran Associates, Inc., 2019, URL https://papers.nips.cc/paper_files/paper/2019/hash/b5dc4e5d9b495d0196f61d45b26ef33e-Abstract.html.
[19]
Lombardi S., Simon T., Saragih J., Schwartz G., Lehrmann A., Sheikh Y., Neural volumes: learning dynamic renderable volumes from images, ACM Trans Graph 38 (4) (2019) 1–14,.
Digital Library
[20]
Sitzmann V., Thies J., Heide F., NieBner M., Wetzstein G., Zollhofer M., DeepVoxels: Learning persistent 3D feature embeddings, in: 2019 IEEE/CVF conference on computer vision and pattern recognition, IEEE, Long Beach, CA, USA, ISBN 978-1-72813-293-8, 2019, pp. 2432–2441,. URL https://ieeexplore.ieee.org/document/8953309/.
[21]
Mildenhall B., Srinivasan P.P., Tancik M., Barron J.T., Ramamoorthi R., Ng R., Nerf: Representing scenes as neural radiance fields for view synthesis, Commun ACM 65 (1) (2021) 99–106.
Digital Library
[22]
Gao K., Gao Y., He H., Lu D., Xu L., Li J., NeRF: Neural radiance field in 3D vision, A comprehensive review, 2022, URL http://arxiv.org/abs/2210.00379 arXiv:2210.00379 [cs].
[23]
Fridovich-Keil S., Yu A., Tancik M., Chen Q., Recht B., Kanazawa A., Plenoxels: Radiance fields without neural networks, in: 2022 IEEE/CVF conference on computer vision and pattern recognition, IEEE, New Orleans, LA, USA, ISBN 978-1-66546-946-3, 2022, pp. 5491–5500,. URL https://ieeexplore.ieee.org/document/9880358/.
[24]
Yu A., Li R., Tancik M., Li H., Ng R., Kanazawa A., PlenOctrees for real-time rendering of neural radiance fields, in: 2021 IEEE/CVF international conference on computer vision, IEEE, Montreal, QC, Canada, ISBN 978-1-66542-812-5, 2021, pp. 5732–5741,. URL https://ieeexplore.ieee.org/document/9711398/.
[25]
Lu Y., Jiang K., Levine J.A., Berger M., Compressive neural representations of volumetric scalar fields, in: Computer graphics forum, vol. 40, no. 3, Wiley Online Library, 2021, pp. 135–146.
[26]
Weiss S., Hermüller P., Westermann R., Fast neural representations for direct volume rendering, in: Computer graphics forum, vol. 41. no. 6, Wiley Online Library, 2022, pp. 196–211.
[27]
Zhang K., Riegler G., Snavely N., Koltun V., Nerf++: Analyzing and improving neural radiance fields, 2020, arXiv preprint arXiv:2010.07492.
[28]
Zhu C., Wan R., Shi B., Neural transmitted radiance fields, Adv Neural Inf Process Syst 35 (2022) 38994–39006.
[29]
Zhang Y., Xu T., Yu J., Ye Y., Wang J., Jing Y., et al., NeMF: Inverse volume rendering with neural microflake field, 2023, arXiv preprint arXiv:2304.00782.
[30]
Tong J., Muthu S., Maken F.A., Nguyen C., Li H., Seeing through the glass: Neural 3D reconstruction of object inside a transparent container, 2023, arXiv preprint arXiv:2303.13805.
[31]
Franz E., Solenthaler B., Thuerey N., Global transport for fluid reconstruction with learned self-supervision, in: 2021 IEEE/CVF conference on computer vision and pattern recognition, IEEE, Nashville, TN, USA, ISBN 978-1-66544-509-2, 2021, pp. 1632–1642,. URL https://ieeexplore.ieee.org/document/9577274/.
[32]
Gkioulekas I., Zhao S., Bala K., Zickler T., Levin A., Inverse volume rendering with material dictionaries, ACM Trans Graph 32 (6) (2013) 162:1–162:13.
[33]
Weiss S., Westermann R., Differentiable direct volume rendering, IEEE Trans Vis Comput Graphics 28 (1) (2021) 562–572. Publisher: IEEE.
[34]
Gkioulekas I., Levin A., Zickler T., An Evaluation of computational imaging techniques for heterogeneous inverse scattering, in: Computer vision, Springer International Publishing, 2016, pp. 685–701.
[35]
Donner C., Weyrich T., d’Eon E., Ramamoorthi R., Rusinkiewicz S., A layered, heterogeneous reflectance model for acquiring and rendering human skin, ACM Trans Graph (TOG) 27 (5) (2008) 1–12.
[36]
Zhang C., Wu L., Zheng C., Gkioulekas I., Ramamoorthi R., Zhao S., A differential theory of radiative transfer, ACM Trans Graph 38 (6) (2019) 1–16. Publisher: ACM New York, NY, USA.
[37]
Zhang C., Yu Z., Zhao S., Path-space differentiable rendering of participating media, ACM Trans Graph 40 (4) (2021) 1–15. Publisher: ACM New York, NY, USA.
[38]
Che C., Luan F., Zhao S., Bala K., Gkioulekas I., Inverse transport networks, 2018, arXiv preprint arXiv:1809.10820.
[39]
Vicini D., Speierer S., Jakob W., Path replay backpropagation: differentiating light paths using constant memory and linear time, ACM Trans Graph 40 (4) (2021) 1–14. Publisher: ACM New York, NY, USA.
[40]
Zeltner T., Speierer S., Georgiev I., Jakob W., Monte Carlo estimators for differential light transport, ACM Trans Graph 40 (4) (2021) 1–16. Publisher: ACM New York, NY, USA.
[41]
Kajiya J.T., VonHerzen B.P., Ray tracing volume densities, SIGGRAPH Comput Graph 18 (3) (1984) 165–174.
[42]
KajiyaJT. The Rendering Equation. In: SIGGRAPH ’86. 1986, p. 143–50.
[43]
Pharr M., Jakob W., Humphreys G., Physically based rendering: From theory to implementation, Morgan Kaufmann, 2016.
[44]
Raab M., Seibert D., Keller A., Unbiased global illumination with participating media, in: Monte Carlo and quasi-Monte Carlo methods 2006, Springer, 2008, pp. 591–605.
[45]
Woodco*ck E., Murphy T., Hemmings P., Longworth S., Techniques used in the GEM code for Monte Carlo neutronics calculations in reactors and other systems of complex geometry, in: Proc. Conf. applications of computing methods to reactor problems, vol. 557 and no. 2, Argonne National Laboratory, 1965.
[46]
Veach E., Guibas L., Bidirectional estimators for light transport, in: Photorealistic rendering techniques, Springer, 1995, pp. 145–167.
[47]
Rudin L.I., Osher S., Total variation based image restoration with free local constraints, in: Proceedings of 1st international conference on image processing, vol. 1, IEEE, 1994, pp. 31–35.
[48]
Panin V., Zeng G., Gullberg G., Total variation regulated EM algorithm [SPECT reconstruction], IEEE Trans Nucl Sci 46 (6) (1999) 2202–2210.
[49]
NgAY. Feature selection, L 1 vs. L 2 regularization, and rotational invariance. In: Proceedings of the twenty-first international conference on machine learning. 2004, p. 78.
[50]
FranzE, SolenthalerB, ThuereyN. Global transport for fluid reconstruction with learned self-supervision. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021, p. 1632–42.
[51]
AzinovicD, LiT-M, KaplanyanA, NiessnerM. Inverse Path Tracing for Joint Material and Lighting Estimation. In: The IEEE conference on computer vision and pattern recognition. CVPR, 2019.
[52]
Hasselgren J., Hofmann N., Munkberg J., Shape, light & material decomposition from images using monte carlo rendering and denoising, 2022, arXiv preprint arXiv:2206.03380.
[53]
Karis B., High-quality temporal supersampling, Adv Real-Time Render Games, SIGGRAPH Courses 1 (10.1145) (2014).
[54]
SchiedC, KaplanyanA, WymanC, PatneyA, ChaitanyaCRA, BurgessJ, et al. Spatiotemporal variance-guided filtering: real-time reconstruction for path-traced global illumination. In: Proceedings of high performance graphics. 2017, p. 1–12.
[55]
Huber P.J., Robust estimation of a location parameter, in: Breakthroughs in statistics: Methodology and distribution, Springer, 1992, pp. 492–518.
Cited By
View all
- (2024)Editorial note Computers & Graphics issue 119Computers and Graphics10.1016/j.cag.2024.103927119:COnline publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1016/j.cag.2024.103927
Recommendations
- Image-based rendering of diffuse, specular and glossy surfaces from a single image
SIGGRAPH '01: Proceedings of the 28th annual conference on Computer graphics and interactive techniques
In this paper, we present a new method to recover an approximation of the bidirectional reflectance distribution function (BRDF) of the surfaces present in a real scene. This is done from a single photograph and a 3D geometric model of the scene. The ...
Read More
- Differentiable Heightfield Path Tracing with Accelerated Discontinuities
SIGGRAPH '23: ACM SIGGRAPH 2023 Conference Proceedings
We investigate the problem of accelerating a physically-based differentiable renderer for heightfields based on path tracing with global illumination. On a heightfield with 1 million vertices (1024 × 1024 resolution), our differentiable renderer ...
Read More
- Reparameterizing discontinuous integrands for differentiable rendering
Differentiable rendering has recently opened the door to a number of challenging inverse problems involving photorealistic images, such as computational material design and scattering-aware reconstruction of geometry and materials from photographs. ...
Read More
Comments
Information & Contributors
Information
Published In
Computers and Graphics Volume 119, Issue C
Apr 2024
407 pages
ISSN:0097-8493
Issue’s Table of Contents
The Authors.
Publisher
Pergamon Press, Inc.
United States
Publication History
Published: 18 July 2024
Author Tags
- Differentiable rendering
- Inverse rendering
- Volume reconstruction
Qualifiers
- Research-article
Contributors
Other Metrics
View Article Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations
1
Total Citations
Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Other Metrics
View Author Metrics
Citations
Cited By
View all
- (2024)Editorial note Computers & Graphics issue 119Computers and Graphics10.1016/j.cag.2024.103927119:COnline publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1016/j.cag.2024.103927
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in
Full Access
Get this Publication
Media
Figures
Other
Tables
