DINOv1
In this paper, we question if self-supervised learning provides new properties to Vision Transformer (ViT) that stand out compared to convolutional networks (convnets). Beyond the fact that adapting self-supervised methods to this architecture works particularly well, we make the following observations: first, self-supervised ViT features contain explicit information about the semantic segmentation of an image, which does not emerge as clearly with supervised ViTs, nor with convnets. Second, these features are also excellent k-NN classifiers, reaching 78.3% top-1 on ImageNet with a small ViT. Our study also underlines the importance of momentum encoder, multi-crop training, and the use of small patches with ViTs. We implement our findings into a simple self-supervised method, called DINO, which we interpret as a form of self-distillation with no labels. We show the synergy between DINO and ViTs by achieving 80.1% top-1 on ImageNet in linear evaluation with ViT-Base.
TL;DR
Label 없이 self-distillation(Student-Teacher, Teacher=EMA(Student))만으로 ViT를 학습시켰더니, attention map에 객체 경계와 semantic segmentation 정보가 emergent하게 나타나고, 그 feature가 k-NN만으로도 강한 성능을 내는 representation이 된다.
ViT는 strong inductive bias(locality, translation equivariance)가 없어 CNN보다 이 효과가 두드러지는데, self-supervision 하에서는 그 부재가 오히려 structure가 emerge할 여지가 되고 DINO가 meaningful global representation으로 유도하는 regularizer로 작동한다.
결과적으로 task-agnostic하고 object-centric하며 spatially coherent한 representation을 학습해 다양한 downstream task의 foundation이 되었다!
