StochasT: Learning with Stochastic Turn Depth for Visual Instruction Tuning

Large Vision-Language Models (LVLMs) rely extensively onVisual Instruction Tuning (VIT) to elicit their multimodal reasoning ca-pabilities. However, we find a discrepancy: VIT often packs multiple lan-guage tasks about the same image for conversational, multi-turn training,whereas existing benchmarks evaluate LVLMs in isolated, single-turn sce-narios. The models can suffer from visual attention decay and contextualoverfitting during multi-turn training, making it hard for them to realizetheir full potential in the mismatched test phase. To close the gap, wepropose learning with Stochastic Turn Depth (StochasT), which stochas-tically groups language tasks for the same image into clusters of vary-ing sizes (turn depth) while preserving their organic order. Hence, whileStochasT draws on Dropout and stochastic depth for ResNets, it doesnot actually drop anything to maximize the utility of the training data.Furthermore, we introduce a challenging, benchmark-agnostic evaluationmechanism based on the Balanced Latin Square to measure LVLMs’ ro-bustness under varying contextual dependencies. Extensive experimentsdemonstrate that StochasT effectively grants LVLMs strong, harmonizedcapabilities for both single-turn and multi-turn use cases. Code is avail-able at: https://yuanqing-ai.github.io/StochasT.