NeurIPS 2026 · Main Track (poster) NeurIPS 2026 · 主会（海报）
Code (MIT)代码（MIT） ↗DAGA: Dynamic Attention-Guided Adaptation for Self-Supervised Vision Transformers DAGA: Dynamic Attention-Guided Adaptation for Self-Supervised Vision TransformersDAGA：面向自监督视觉 Transformer 的动态注意力引导适配
1 National University of Defense Technology (NUDT) · 2 Northwestern Polytechnical University (NPU) 1 国防科技大学（NUDT） · 2 西北工业大学（NPU）
† Equal contribution · * Corresponding author † 共同第一作者（同等贡献） · * 通讯作者
A frozen backbone, steered by its own attention. 冻结的骨干，由自身注意力引导。
Self-supervised vision transformers are strong frozen backbones, yet adapting them to downstream tasks still trades accuracy against the number of trained parameters. Existing parameter-efficient methods apply a fixed-capacity module uniformly to every patch and channel, ignoring the attention structure the backbone has already learned.
自监督视觉 Transformer 是强大的冻结骨干网络，但将其迁移到下游任务时，仍要在精度与可训练参数量之间做取舍。现有参数高效方法以固定容量的模块对所有 token 与通道一视同仁，忽视了骨干网络自身已经学到的注意力结构。
We introduce DAGA, a dynamic attention-guided adaptation framework that turns the backbone's own attention into in-loop guidance for lightweight feature refinement. A shared attention encoder summarizes each layer's attention map and patch features; a dynamic gate generator converts the summary into per-layer guidance; and a feature transformer applies the resulting channel and spatial gates to the frozen features, adding only 1.4% trainable parameters.
我们提出 DAGA——一种动态注意力引导的适配框架，把骨干网络自身的注意力转化为轻量特征细化的闭环引导。共享的注意力编码器将每层的注意力图与 patch 特征压缩为紧凑表示；动态门控生成器再把该表示转换为逐层引导；特征变换器据此得到通道门控与空间门控，并对冻结特征进行调制，仅增加 1.4% 的可训练参数。
With a frozen DINOv3-ViT-B backbone, DAGA reaches 85.90 ± 0.06% ImageNet-1K top-1 (n=3) and matches layer-wise fine-tuning (85.84 ± 0.07%) within seed noise at 18× fewer trained parameters. It improves dense prediction by +2.9 mIoU on ADE20K and +2.1 AP on COCO over the best-tuned static-gating PEFT, adds +12.1 mIoU on ADE20K over the frozen baseline, and lifts ROxford-Medium retrieval by +41.0 mAP. Gains concentrate on attention-rich self-distilled transformers (+4.5 to +5.0 ImageNet-1K on DINOv2, DINOv3 and iBOT) and are marginal on MAE, CLIP, DeiT and MoCo v3 (≤ +0.2); composing with refinement front-ends such as CLIPSelf and SD-MAE recovers most of the gap — refine first, then adapt.
在冻结的 DINOv3-ViT-B 骨干上，DAGA 取得 85.90 ± 0.06% 的 ImageNet-1K top-1 精度（n=3），与逐层微调（85.84 ± 0.07%）在随机种子噪声范围内持平，而可训练参数量少 18 倍。相较调优最佳的静态门控 PEFT，DAGA 在 ADE20K 上提升 +2.9 mIoU、在 COCO 上提升 +2.1 AP；相较冻结基线，ADE20K 提升 +12.1 mIoU，ROxford-Medium 检索提升 +41.0 mAP。增益集中在注意力信息丰富的自蒸馏 Transformer 上（DINOv2、DINOv3 与 iBOT 上的 ImageNet-1K 增益为 +4.5 至 +5.0），而在 MAE、CLIP、DeiT 与 MoCo v3 上较为有限（≤ +0.2）；与 CLIPSelf、SD-MAE 等细化前端组合时，可补回大部分差距——先细化，再适配。
Static adapters ignore what the backbone already knows. 静态适配器忽视了骨干已经学到的东西。
Adapters, LoRA and visual prompts add fixed-capacity modules and treat every patch alike. A self-supervised ViT, however, has already learned where the evidence is for a given query: its attention maps localize the object or region that matters. DAGA does not discard that signal — it feeds the backbone's own attention back into the adaptation loop as guidance, so refinement is conditioned on what the frozen model is already looking at.
Adapter、LoRA 与视觉提示等方法引入固定容量的模块，并对所有 patch 一视同仁。然而自监督 ViT 已经学会在给定查询下证据位于何处：其注意力图会定位真正重要的物体或区域。DAGA 并不丢弃这一信号——它把骨干自身的注意力作为引导重新输入适配环路，使特征细化以冻结模型正在关注的内容为条件。
The qualitative comparison below shows the difference: DAGA's responses concentrate on the queried object (refrigerator, knife, apple; kite, motorcycle, bench), while the baseline spreads attention diffusely over the scene.
下方的定性对比展示了这一差异：DAGA 的响应集中在被查询的物体上（冰箱、刀、苹果；风筝、摩托车、长椅），而基线将注意力弥散在整个场景中。
Three pieces, one loop. 三个部件，一个闭环。
DAGA keeps the backbone frozen and attaches a small module to every block. Each module receives two signals from its block — the attention map and the patch features — and returns refined patch features for the next block.
DAGA 保持骨干冻结，并在每个 Block 上接入一个小模块。每个模块接收该 Block 的两种信号——注意力图与 patch 特征——并输出细化后的 patch 特征，送入下一个 Block。
Attention Encoder注意力编码器
A shared convolutional extractor (Conv–BN–GELU stages) summarizes the attention map and the patch features of each block into a compact feature Z.
共享卷积提取器（Conv–BN–GELU 两级）将每个 Block 的注意力图与 patch 特征压缩为紧凑特征 Z。
Dynamic Gate Generator动态门控生成器
Lightweight layer heads (pool–MLP–LN) turn Z into per-layer guidance I(ℓ), from which a channel gate Gc and a spatial scale S are generated.
轻量层头（pool–MLP–LN）将 Z 转换为逐层引导 I(ℓ)，并由此生成通道门 Gc 与空间尺度 S。
Feature Transformer特征变换器
A small MLP produces the transformed features ψ(F); the channel and spatial gates then modulate them onto the frozen patch features.
小型 MLP 产生变换后的特征 ψ(F)，再由通道门与空间门将其调制到冻结的 patch 特征上。
Gated Refinement门控细化
A residual refinement combines both: the gated term is scaled by a learnable factor and added back to the input features, so every block keeps its pretrained representation.
残差式细化将二者结合：门控项乘以可学习系数后加回输入特征，使每个 Block 保留其预训练表征。
Gated feature refinement. F̂(ℓ) = F(ℓ) + α · Gc · (1 + β · S) ⊙ ψ(F(ℓ)) — the channel gate Gc and spatial scale S are generated from layer-ℓ attention guidance. The whole loop adds 1.4% trainable parameters to a frozen backbone.
门控特征细化。F̂(ℓ) = F(ℓ) + α · Gc · (1 + β · S) ⊙ ψ(F(ℓ))——通道门 Gc 与空间尺度 S 由第 ℓ 层的注意力引导生成。整个环路在冻结骨干上仅增加 1.4% 的可训练参数。
A better Pareto point. 更好的帕累托点。
We compare DAGA with linear probing, LoRA, VPT-Deep, AdaptFormer, ViT-Adapter, layer-wise fine-tuning (Layer-FT) and full fine-tuning, all on a frozen DINOv3-ViT-B backbone.
我们将 DAGA 与线性探测、LoRA、VPT-Deep、AdaptFormer、ViT-Adapter、逐层微调（Layer-FT）以及全量微调进行对比，骨干均为冻结的 DINOv3-ViT-B。
Matches Layer-FT, 18× lighter匹配 Layer-FT，参数少 18 倍
85.90 ± 0.06% versus 85.84 ± 0.07% ImageNet-1K top-1 (n=3), within seed noise at 18× fewer trained parameters.
85.90 ± 0.06% 对 85.84 ± 0.07% 的 ImageNet-1K top-1（n=3），在随机种子噪声范围内持平，而可训练参数量少 18 倍。
Dense prediction稠密预测
+2.9 mIoU on ADE20K and +2.1 AP on COCO over the best-tuned static-gating PEFT; +12.1 mIoU on ADE20K over the frozen baseline.
相较调优最佳的静态门控 PEFT，ADE20K 提升 +2.9 mIoU、COCO 提升 +2.1 AP；相较冻结基线，ADE20K 提升 +12.1 mIoU。
Retrieval and fine-grained recognition检索与细粒度识别
+41.0 mAP on ROxford-Medium retrieval over the frozen baseline; +1.6 on CUB-200 and +3.5 on FGVC-Aircraft.
相较冻结基线，ROxford-Medium 检索提升 +41.0 mAP；CUB-200 提升 +1.6，FGVC-Aircraft 提升 +3.5。
1.4% trainable parameters1.4% 可训练参数
The backbone stays frozen; only the added DAGA modules are trained — 1.4% additional parameters.
骨干保持冻结，仅训练新增的 DAGA 模块——额外参数占 1.4%。
Where it helps — and where it does not. 在何处有效，在何处无效。
The benefit is not uniform across backbones. DAGA gains +4.5 to +5.0 ImageNet-1K on attention-rich self-distilled transformers (DINOv2, DINOv3, iBOT), and is marginal on MAE, CLIP, DeiT and MoCo v3 (≤ +0.2).
增益并非对所有骨干都一致。在注意力信息丰富的自蒸馏 Transformer（DINOv2、DINOv3、iBOT）上，DAGA 取得 +4.5 至 +5.0 的 ImageNet-1K 提升；而在 MAE、CLIP、DeiT 与 MoCo v3 上提升有限（≤ +0.2）。
When composed with refinement front-ends such as CLIPSelf or SD-MAE, DAGA recovers most of the remaining gap: refine first, then adapt.
与 CLIPSelf、SD-MAE 等细化前端组合时，DAGA 可补回大部分剩余差距：先细化，再适配。
| Backbone family骨干类型 | ImageNet-1K top-1 gainImageNet-1K top-1 增益 |
|---|---|
| Attention-rich self-distilled ViTs 注意力信息丰富的自蒸馏 ViT · DINOv2 · DINOv3 · iBOT | +4.5 … +5.0 |
| MAE · CLIP · DeiT · MoCo v3 | ≤ +0.2 |
Gains are reported on ImageNet-1K top-1 with the backbone frozen. 增益均在冻结骨干的 ImageNet-1K top-1 上报告。
Cite this work. 引用本文。
If you find our work or code useful in your research, please cite: 如果我们的工作或代码对您的研究有所帮助，请引用：
@inproceedings{zhou2026daga,
title = {DAGA: Dynamic Attention-Guided Adaptation for Self-Supervised Vision Transformers},
author = {Zhou, Tianjian and Jiang, Jie and Li, Yishan and Zhang, Yifei},
booktitle = {Advances in Neural Information Processing Systems},
year = {2026}
}
NeurIPS 2026 · Main Track (poster) · Code: github.com/Jaywalk18/DAGA (MIT) NeurIPS 2026 · 主会（海报） · 代码：github.com/Jaywalk18/DAGA（MIT）