Clean Implicit 3D Structure from Noisy 2D STEM Images

CVPR 2022

Hannah Kniesel1 *        Timo Ropinski1        Tim Bergner1        Kavitha Shaga Devan1       
Clarissa Read1        Paul Walther1        Tobias Ritschel2       Pedro Hermosilla1      
1Ulm University  
2University College London  

Paper Code

🔬🖥️✨ Our CVPR2022 paper unveils a method to extract pristine 3D structures from noisy 2D STEM images. Using a NeRF-inspired approach combined with a normalizing flow model, we untangle signal from noise for clean reconstructions from noisy projections. 🌟🔍

Abstract

STEM acquire 2D images of a 3D sample on the scale of individual cell components. Unfortunately, these 2D images can be too noisy to be fused into a useful 3D structure and facilitating good denoisers is challenging due to the lack of clean-noisy pairs. Additionally, representing detailed 3D structure can be difficult even for clean data when using regular 3D grids. Addressing these two limitations, we suggest a differentiable image formation model for STEM, allowing to learn a joint model of 2D sensor noise in STEM together with an implicit 3D model. We show, that the combination of these models outperforms both individually, as well as several baselines on synthetic and real data. We show, that the combination of these models are able to successfully disentangle 3D signal and noise without supervision and outperform at the same time several baselines on synthetic and real data.

Video 🎬

BibTeX


        @inproceedings{kniesel2020clean,
          title={Clean Implicit 3D Structure from Noisy 2D STEM Images},
          author={Kniesel, Hannah and Ropinski, Timo and Bergner, Tim and Shaga Devan, Kavitha and Read, Clarissa and Walther, Paul and Ritschel, Tobias and Hermosilla, Pedro},
          booktitle={Proceedings of IEEE Conference on Computer Vision and Pattern Recognition}
          year={2022}
        }