A First Exploration of Neuromorphic OT-CFM for Multi-Speaker VSR

Visual Speech Recognition (VSR) tasks in complex multi-speaker scenarios are severely hindered by rapid head motions, occlu-sions, and subtle lip articulations. Traditional RGB-based methods strug-gle here due to low rates and motion blur of frames. To overcome these,we propose LipsFlow, a neuromorphic-inspired VSR framework thatconverts RGB videos into high-temporal-resolution event streams. Formulti-speaker, we employ ByteTrack tracking and TalkNet active speakerdetection to temporally segment scenes into single-speaker clips, enablingfocused per-speaker analysis. By explicitly capturing microsecond-levelarticulatory dynamics via learnable event-based representations, Lips-Flow achieves inherent robustness against visual degradation. To effi-ciently model these dense event-based features and adapt to speaker-specific articulatory patterns, we introduce Optimal Transport Condi-tional Flow Matching (OT-CFM). It enforces deterministic, straight-line trajectory generation in a semantic latent space, slashing inferencelatency to just two Ordinary Differential Equation (ODE) steps. Further-more, we design a Dual-Level Semantic Supervision mechanism combin-ing token-level BERT weight tying and sentence-level priors to resolvehomophene ambiguities. Validated on competitive benchmarks, LipsFlowachieves a state-of-the-art WER of 22.3% at 240 ms latency, estab-lishing a highly robust and efficient paradigm for event-based VSR.