ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control

Human demonstrations provide strong priors for robot ma-nipulation, yet it is non-trivial to transfer them to execute on real robotsdue to the kinematic gap. In dexterous manipulation, it remains chal-lenging to track long-horizon, contact-rich sequences even in simulators:a reference-tracking policy must keep objects on their target trajectorieswhile preserving demonstrated joint motion and contact timing. Exist-ing approaches often rely on hand-crafted reward tuning that requireper-sequence tuning and break under limited interaction budgets. We in-troduce ConTrack, a reinforcement learning (RL) framework that scaleswith tracking data. ConTrack treats object tracking as a constraint andallocates remaining control authority to motion fidelity, which allows itto adapt task–style trade-offs online using a dual-variable update. In ad-dition, ConTrack also stabilizes long-horizon learning with an adaptivemid-trajectory reset library that reuses policy-reachable simulator states.Our qualitative and quantitative results in simulation tracking and realrobot demonstrate that ConTrack improves success and object pose ac-curacy significantly over prior arts while preserving joint and contactfidelity. Website: https://www.lyt0112.com/projects/ConTrack.