Domain Adaptation with Adaptive Imagination for Visual Reinforcement Learning under Limited Target Data

Sim-to-real transfer remains a major obstacle for reinforce-ment learning (RL), especially for vision-based control where image ob-servations exacerbate the state-distribution shift between simulation andthe real world. Domain adaptation (DA) is a promising remedy for thischallenge. Prior sim-to-real DA works have demonstrated encouragingresults, yet these approaches typically assume substantially more targetdata, which is not available in practice. Indeed, their performance de-grades significantly when the target data budget is reduced. To addressthis challenge, we propose AIDA (Adaptive Imagination for DomainAdaptation), a domain adaptation framework for visual reinforcementlearning that addresses sim-to-real transfer under scarce target datawithout requiring additional interaction with the target environment.Our key idea is adaptive imagination: generating reliable and semanticimagination rollouts to augment limited target data. Specifically, AIDAemploys a distribution-shift-aware discriminator that truncates rolloutswhen imagined transitions drift into low-confidence regions, so that onlyreliable transitions contribute to the augmentation. On these reliabletransitions, AIDA introduces a self-consistency loss that cycles throughstate → image observation → state, penalizing discrepancies betweenthe original and reconstructed states. This provides additional adap-tation signals beyond the scarce target data. Our experiments demon-strate that adaptive imagination effectively truncates unreliable rollouts.By enforcing a self-consistency loss on the resulting reliable transitions,AIDA learns semantically meaningful state representations and outper-forms baselines across five MuJoCo tasks and two Gymnasium-Roboticstasks.