Audio-Visual Continual Test-Time Adaptation without Forgetting

Audio-visual continual test-time adaptation involves contin-ually adapting a source audio-visual model at test-time, to unlabelednon-stationary domains, where either or both modalities can be distribu-tionally shifted, which hampers online cross-modal learning and eventuallyleads to poor accuracy. While previous works have tackled this problem,we find that SOTA methods suffer from catastrophic forgetting wherethe model’s performance drops well below even the source model dueto continual parameter updates at test-time. In this work, we first showthat adapting only the modality fusion layer to a target domain not onlyimproves performance on that domain but can also enhance performanceon subsequent domains. Based on this strong cross-task transferability ofthe fusion layer’s parameters, we propose a method, AVReCAP, that im-proves test-time performance of the models without access to any sourcedata. Our approach works by using a selective parameter retrieval mech-anism that dynamically retrieves the best fusion layer parameters from abuffer using only a small batch of test data. These parameters are thenintegrated into the model, adapted to the current test distribution, andsaved back for future use. Extensive experiments on benchmark datasetsinvolving unimodal and bimodal corruptions show our proposed AVReCAPsignificantly outperforms existing methods while minimizing catastrophicforgetting.