LSRM: High-Fidelity Object-Centric Reconstruction via Scaled Context Windows

We introduce the Large Sparse Reconstruction Model tostudy how scaling transformer context windows affects feed-forward 3Dreconstruction. Although recent object-centric feed-forward methods pro-duce robust, high-quality reconstructions, they still lag behind dense-view optimization in recovering fine-grained texture and appearance. Weshow that expanding the context window—by substantially increasingthe number of active object and image tokens—narrows this gap andenables high-fidelity 3D object reconstruction and inverse rendering. Toscale effectively, we adapt native sparse attention [68] for 3D reconstruc-tion with three key contributions: (1) an efficient coarse-to-fine pipelinethat focuses computation on informative regions by predicting sparsehigh-resolution residuals; (2) a 3D-aware spatial routing mechanism thatestablishes accurate 2D-3D correspondences using explicit geometric dis-tances rather than standard attention scores; and (3) a custom block-aware sequence-parallel strategy with an All-gather-KV protocol to bal-ance dynamic, sparse workloads across GPUs. As a result, LSRM handles20× more object tokens and >2× more image tokens than prior state-of-the-art (SOTA) methods. Extensive evaluations on standard novel-viewsynthesis benchmarks show substantial gains over the current SOTA,yielding >2.4 dB higher PSNR and >40% lower LPIPS. Furthermore,when extending LSRM to inverse rendering, qualitative and quantitativeevaluations on widely used benchmarks demonstrate consistent improve-ments in texture and geometry details, achieving an LPIPS that matchesor exceeds that of SOTA dense-view optimization methods. Code andmodel weights are available on our project page.