FILT3R: Latent State Adaptive Kalman Filter for Streaming 3D Reconstruction

Streaming 3D reconstruction maintains a persistent latentstate that is updated online from incoming frames, enabling constant-memory inference. A key failure mode is the state update rule: aggressiveoverwrites forget useful history, while conservative updates fail to tracknew evidence, and both behaviors become unstable beyond the traininghorizon. To address this challenge, we propose FILT3R, a training-freelatent filtering layer that casts recurrent state updates as stochastic stateestimation in token space. FILT3R maintains a per-token variance andcomputes a Kalman-style gain that adaptively balances memory reten-tion against new observations. Process noise – governing how much thelatent state is expected to change between frames – is estimated onlinefrom EMA-normalized temporal drift of candidate tokens. Using exten-sive experiments, we demonstrate that FILT3R yields an interpretable,plug-in update rule that generalizes common overwrite and gating poli-cies as special cases. Specifically, we show that gains shrink in stableregimes as uncertainty contracts with accumulated evidence, and risewhen genuine scene change increases process uncertainty, improving long-horizon stability for depth, pose, and 3D reconstruction, compared to theexisting methods.