Zero-shot Depth from Defocus

Depth from Defocus (DfD) is the task of estimating a densemetric depth map from a focus stack. Unlike previous works overfittingto a certain dataset, this paper focuses on the challenging and practi-cal setting of zero-shot generalization. We first propose a new real-worldDfD benchmark ZEDD, which contains 8.3× more scenes and signifi-cantly higher quality images and ground-truth depth maps comparedto previous benchmarks. We also design a novel network architecturenamed FOSSA. FOSSA is a Transformer-based architecture with noveldesigns tailored to the DfD task. The key contribution is a stack atten-tion layer with a focus distance embedding, allowing efficient informa-tion exchange across the focus stack. Finally, we develop a new trainingdata pipeline allowing us to utilize existing large-scale RGBD datasetsto generate synthetic focus stacks. Experiment results on ZEDD andother benchmarks show a significant improvement over the baselines,reducing errors by up to 55.7%. The ZEDD benchmark is released athttps://zedd.cs.princeton.edu. The code and checkpoints are re-leased at https://github.com/princeton-vl/FOSSA.Input: Focus stack and corresponding focus distancesFocus Distance = 0.8m Focus Distance = 1.6m Focus Distance = 3.1m Focus Distance = 8.1mDFF-FV 𝛿! = 0.71 MoGe-2 𝛿! = 0.45 𝛿! = 0.98Existing depth from defocus Monocular depth models suffer Ours Ground Truthmodels are not generalizable from scale ambiguity*Equal contribution.