REON-NVS: Real-Time Online Novel-View Synthesis from Sparse-View Videos

Recent advances in online reconstruction of dynamic scenesdemonstrate impressive visual quality, showing potential for real-worldapplications such as VR content streaming. Yet, existing online recon-struction methods still require a large number of input views and time-consuming iterative optimization. Moreover, reliable pose estimation,a prerequisite for NVS, introduces additional delay. In this paper, wepresent REON-NVS, a feedforward online NVS framework for sparse-view input video streams, which goes from pose estimation to novel-viewreconstruction in real time. REON-NVS comprises two main compo-nents. First, our method leverages a feedforward pose estimator andmapper that replace conventional camera pose optimization pipelines.Second, we design a scene reconstructor built upon a state-space model(SSM), which efficiently synthesizes temporally consistent novel-view im-ages by exploiting information from previous frames. To train and eval-uate our approach in realistic in-the-wild streaming scenarios, we intro-duce a new multi-view dynamic scene dataset of 150 dynamic scenescaptured with moving cameras. Extensive experiments demonstrate thatREON-NVS achieves high visual quality while operating in real time (32FPS), validating its applicability in real-world scenarios.