Seeing Touch from Motion: A Unified Modality-Aware Visuo-Tactile Policy with Tactile Motion Correlation

Visuo-Tactile policies leveraging optical tactile sensors haveshown great promise in contact-rich manipulation. These sensors achievehigh spatial resolution and multi-dimensional force sensing by utilizingan internal camera to monitor the deformation of their elastic gel sur-face, thereby indirectly inferring tactile cues. Despite their advantages,extracting fine-grained contact states necessary for contact-rich manipu-lation remains an open challenge. Existing methods typically use eitherraw images or cumulative motion fields to represent tactile cues. However,both are prone to perception ambiguity. Raw tactile images mainly cap-ture appearance changes, while cumulative motion fields only reflect theaggregate gel deformation. Consequently, distinct fine-grained contactstates can exhibit highly similar patterns, making it difficult to explicitlydistinguish subtle contact variations. To address this issue, we explorethe dynamic priors of tactile motion and discover that the correlationbetween transient and cumulative motion can explicitly distinguish fine-grained contact states. Based on this insight, we propose a motion-awaretactile representation to facilitate contact-rich manipulation. Beyond tac-tile representation, effective fusion of tactile and visual modalities is alsocritical. Most existing fusion methods either directly concatenate featuresfrom each modality or train modality-specific networks separately andfuse their outputs. However, these strategies struggle to simultaneouslymodel cross-modal interactions and preserve modality-specific character-istics. In this work, we take advantage of the Mixture-of-Transformersarchitecture and propose a unified modality-aware visuo-tactile policythat captures cross-modal complementarity while maintaining modality-specific properties. Extensive experiments on four challenging contact-rich manipulation tasks indicate the superior performance of our method.