RT-RMOT: A Dataset and Framework for RGB-Thermal Referring Multi-Object Tracking

Referring Multi-Object Tracking has attracted increasing at-tention due to its human-friendly interactive characteristics, yet it ex-hibits limitations in low-visibility conditions, such as nighttime, smoke,and other challenging scenarios. To overcome this limitation, we proposea new RGB-Thermal RMOT task, named RT-RMOT, which aimsto fuse RGB appearance features with the illumination robustness ofthe thermal modality to enable all-day referring multi-object tracking.To promote research on RT-RMOT, we construct the first Ref erringMulti-Object Tracking dataset under RGB-Thermal modality, namedRefRT. It contains 388 language descriptions, 1,250 tracked targets,and 166,147 Language-RGB-Thermal (L-RGB-T) triplets. Furthermore,we propose RTrack, a framework built upon a multimodal large lan-guage model (MLLM) that integrates RGB, thermal, and textual fea-tures. Since the initial framework still leaves room for improvement, weintroduce a Group Sequence Policy Optimization (GSPO) strategy tofurther exploit the model’s potential. To alleviate training instabilityduring RL fine-tuning, we introduce a Clipped Advantage Scaling (CAS)strategy to suppress gradient explosion. In addition, we design StructuredOutput Reward and Comprehensive Detection Reward to balance explo-ration and exploitation, thereby improving the completeness and accu-racy of target perception. Extensive experiments on the RefRT datasetdemonstrate the effectiveness of the proposed RTrack framework.