Efficient Test-Time Adaptation with Cache-based Dynamic Adapter

Waiting List Strategy: A novel enhancement approach for improved robustness in test-time adaptation

Abstract

This project extends the official implementation of TDA (Test-Time Adaptation) by exploring its efficiency, robustness, and flexibility in different real-world scenarios. We focus on improving Vision-Language Models' adaptation capabilities through cache-based dynamic adapters, investigating their performance under various distribution shifts and budget constraints.

Our work builds upon the official paper "Efficient Test-Time Adaptation of Vision-Language Models" and contributes novel insights through comprehensive benchmarking, hyperparameter sensitivity analysis, and innovative enhancement strategies like the Waiting List approach.

🚀 Our Contributions

📌 1. Benchmarking on Diverse Datasets

We evaluated TDA on:

Class Distribution Shifts (CD)
Out-of-Distribution (OOD) test cases

Our comprehensive evaluation on CIFAR-10-C (non-iid stream) provides new insights into the method's robustness.

⚙️ 2. Hyperparameter Sensitivity under Budget Constraints

Performance analysis across:

Adaptation budgets
Stream orderings
Different computational constraints

🧩 3. Waiting List Strategy

We introduced a novel Waiting List strategy to improve robustness:

Showed improvement on ImageNet datasets
Analyzed effectiveness on CIFAR10-C
Provided insights into when and why this strategy works

🔬 Methodology

Our approach builds upon the Test-Time Adaptation framework for Vision-Language Models, with several key innovations:

Cache-based Dynamic Adapters: Efficient adaptation mechanisms that maintain performance while reducing computational overhead
Comprehensive Benchmarking: Systematic evaluation across multiple datasets and distribution shift scenarios
Budget-Conscious Analysis: Real-world constraints consideration in adaptation strategies
Novel Enhancement Strategies: Introduction of waiting list mechanisms for improved robustness

📈 Key Findings

✅ Reproducible Results: All experiments are reproducible and tested under realistic compute constraints
🎯 Improved Robustness: Waiting List strategy shows promise for certain dataset types
⚙️ Hyperparameter Insights: Comprehensive analysis of sensitivity under budget constraints
🧠 Distribution Shift Understanding: Better characterization of when TDA methods work best

📁 Implementation Details

File	Purpose
`tda_cd_benchmark.ipynb`	Benchmarking on class-distribution shifts
`tda_runner_experiments.py`	Hyperparameter tuning and sensitivity analysis
`tda_runner_with_waiting.py`	Waiting list enhancement implementation

BibTeX

@misc{camacho2023tta,
  title={Efficient Test-Time Adaptation with Cache-based Dynamic Adapter: Extensions and Analysis},
  author={Camacho Mohedano, Juan and De Carlo, Andrea and Bolotta, Samuele},
  year={2023},
  institution={University of Trento},
  type={Course Project},
  note={Trends and Applications in Computer Vision}
}