Attention-Logit Steering to Compositional Generalization for Continual VQA

Continual visual question answering (VQA) requires learningnew question types while retaining previously acquired visual ground-ing and reasoning. Existing methods mainly reduce parameter interfer-ence, but do not explicitly control how sequential adaptation changesthe coupling between a question and its visual evidence. We proposeQ-STEER, a task-agnostic framework that separates plasticity from re-tention through a shared question-conditioned controller. For plasticity,probe-based MoE-LoRA self-expansion activates new expert slots onlywhen the current capacity is insufficient. For retention, a KL-based late-layer attention-drift signal conditions a low-rank correction to attentionlogits; the drift is diagnostic rather than an attention-matching loss. Oncontinual VQA v2, Q-STEER obtains 53.68 average performance and4.51 average forgetting, and reaches 51.00/51.48 on Novel/Seen compo-sitional splits. Experiments on a heterogeneous visual-instruction stream,together with seed, ablation, grounding, sensitivity, and overhead anal-yses, support the effectiveness of drift-aware steering and on-demandexpansion.