When the City Teaches the Car: Label-Free 3D Perception from Infrastructure

Building robust 3D perception for self-driving still relies heav-ily on large-scale data collection and manual annotation, yet this paradigmbecomes impractical as deployment expands across diverse cities andregions. Meanwhile, modern cities are increasingly instrumented withroadside units (RSUs), static sensors deployed along roads and at inter-sections to monitor traffic. This raises a natural question: can the cityitself help train the vehicle? We propose infrastructure-taught, label-free3D perception, a paradigm in which RSUs act as stationary, unsuper-vised teachers for ego vehicles. Leveraging their fixed viewpoints and re-peated observations, RSUs learn local 3D detectors from unlabeled dataand broadcast predictions to passing vehicles, which are aggregated aspseudo-label supervision for training a standalone ego detector. The re-sulting model requires no infrastructure or communication at test time.We instantiate this idea as a fully label-free three-stage pipeline and con-duct a concept-and-feasibility study in a CARLA-based multi-agent en-vironment. With CenterPoint, our pipeline achieves 82.3% AP for detect-ing vehicles, compared to a fully supervised ego upper bound of 94.4%.We further systematically analyze each stage, evaluate its scalability, anddemonstrate complementarity with existing ego-centric label-free meth-ods. Together, these results suggest that city infrastructure itself canpotentially provide a scalable supervisory signal for autonomous vehi-cles, positioning infrastructure-taught learning as a promising orthogonalparadigm for reducing annotation cost in 3D perception.