Fast Spatial Memory with Scalable Elastic Test-Time Training

Large Chunk Test-Time Training (LaCT) has shown strongperformance on long-context 3D reconstruction, but its fully plastic inference-time updates remain vulnerable to catastrophic forgetting and overfit-ting. As a result, LaCT is typically instantiated with a single large chunkspanning the full input sequence, falling short of the broader goal ofhandling arbitrarily long sequences in a single pass. We propose ElasticTest-Time Training inspired by elastic weight consolidation, that sta-bilizes LaCT fast-weight updates with a Fisher-weighted elastic prioraround a maintained anchor state. The anchor evolves as an exponentialmoving average of past fast weights to balance stability and plasticity.Based on this updated architecture, we introduce Fast Spatial Memory(FSM), an efficient and scalable model for 4D reconstruction that learnsspatiotemporal representations from long observation sequences and ren-ders novel view-time combinations. We pre-trained FSM on large-scalecurated 3D/4D data to capture the dynamics and semantics of complexspatial environments. Extensive experiments show that FSM supportsfast adaptation over long sequences and delivers high-quality 3D/4D re-construction with smaller chunks and mitigates the camera-interpolationshortcut. Overall, we hope to advance LaCT beyond the bounded single-chunk setting toward robust multi-chunk adaptation, a necessary stepfor generalization to genuinely longer sequences, while substantially al-leviating the activation-memory bottleneck.