LEO-Fuse: A Modality- and Task-Agnostic Universal Framework for Multimodal Human Sensing

Robust human-body understanding benefits from fusing het-erogeneous sensors—WiFi CSI, depth, LiDAR, mmWave, and RGB—yeteach modality carries distinct noise characteristics and may be intermit-tently unavailable. Existing fusion methods often assume fixed sensorsets and can collapse onto dominant modalities, reducing robustness un-der missing-modality conditions. We propose LEO-Fuse, a task-general,modality-agnostic Local Expert Orchestration (LEO) framework for uni-versal multimodal human sensing. LEO-Fuse freezes unimodal encoders,aligns modality features in a shared token space, fuses available modal-ity tokens with a mask-aware transformer, and applies task-specific lo-cal routing for downstream tasks including 3D human pose estimation(HPE) and human activity recognition (HAR). For each target task, asingle trained model handles all 31 non-empty subsets of five modalitieswithout retraining. On MM-Fi, LEO-Fuse obtains 75.5 mm mean per-joint position error (MPJPE), averaged over all 31 subsets. For HAR,HPE-trained encoders yield 93.4% average top-1 accuracy, improving to95.7% with HAR-trained encoders. On XRF55, LEO-Fuse reaches up to95.5% HAR accuracy. These results demonstrate scalable and robust uni-versal multimodal human sensing under realistic missing-modality set-tings.