Rapidly Deploying On-Device Eye Tracking by Distilling Visual Foundation Models

Eye tracking (ET) plays a critical role in augmented andvirtual reality applications. However, rapidly deploying high-accuracy,on-device gaze estimation for new products remains challenging becausehardware configurations (e.g., camera placement, camera pose, and il-lumination) often change across device generations. Visual foundationmodels (VFMs) excel on natural-image benchmarks and offer a promis-ing path to rapid training and deployment; yet, we find that off-the-shelfVFMs still struggle to reach high accuracy on specialized near-eye in-frared images. To close this gap, we introduce DistillGaze, a frameworkthat distills a VFM using labeled synthetic data and unlabeled real datafor rapid, high-accuracy on-device gaze estimation. DistillGaze proceedsin two stages. First, we adapt a VFM into a domain-specialized teacherusing synthetic gaze labels and unlabeled real images. Synthetic dataprovide scalable, high-quality gaze supervision, while unlabeled real databridges the synthetic-to-real domain gap. Second, we train an on-devicestudent from both teacher guidance and self-training. Evaluated on alarge-scale crowd-sourced dataset spanning more than 2,000 participants,DistillGaze reduces median gaze error by 58.6% relative to synthetic-only baselines while maintaining a lightweight 256K-parameter modelsuitable for real-time on-device deployment. More broadly, DistillGazeoffers an efficient path to training and deploying ET models that adaptto hardware changes, and a recipe for combining synthetic supervisionwith unlabeled real data in on-device regression tasks.