When Does Contrastive Visual Representation Learning Work?
Research output: Working paper › Preprint › Research
Standard
When Does Contrastive Visual Representation Learning Work? / Cole, Elijah; Yang, Xuan; Wilber, Kimberly; Mac Aodha, Oisin; Belongie, Serge.
arXiv.org, 2022.Research output: Working paper › Preprint › Research
Harvard
APA
Vancouver
Author
Bibtex
}
RIS
TY - UNPB
T1 - When Does Contrastive Visual Representation Learning Work?
AU - Cole, Elijah
AU - Yang, Xuan
AU - Wilber, Kimberly
AU - Mac Aodha, Oisin
AU - Belongie, Serge
PY - 2022
Y1 - 2022
N2 - Recent self-supervised representation learning techniques have largely closed the gap between supervised and unsupervised learning on ImageNet classification. While the particulars of pretraining on ImageNet are now relatively well understood, the field still lacks widely accepted best practices for replicating this success on other datasets. As a first step in this direction, we study contrastive self-supervised learning on four diverse large-scale datasets. By looking through the lenses of data quantity, data domain, data quality, and task granularity, we provide new insights into the necessary conditions for successful self-supervised learning. Our key findings include observations such as: (i) the benefit of additional pretraining data beyond 500k images is modest, (ii) adding pretraining images from another domain does not lead to more general representations, (iii) corrupted pretraining images have a disparate impact on supervised and self-supervised pretraining, and (iv) contrastive learning lags far behind supervised learning on fine-grained visual classification tasks.
AB - Recent self-supervised representation learning techniques have largely closed the gap between supervised and unsupervised learning on ImageNet classification. While the particulars of pretraining on ImageNet are now relatively well understood, the field still lacks widely accepted best practices for replicating this success on other datasets. As a first step in this direction, we study contrastive self-supervised learning on four diverse large-scale datasets. By looking through the lenses of data quantity, data domain, data quality, and task granularity, we provide new insights into the necessary conditions for successful self-supervised learning. Our key findings include observations such as: (i) the benefit of additional pretraining data beyond 500k images is modest, (ii) adding pretraining images from another domain does not lead to more general representations, (iii) corrupted pretraining images have a disparate impact on supervised and self-supervised pretraining, and (iv) contrastive learning lags far behind supervised learning on fine-grained visual classification tasks.
UR - https://arxiv.org/abs/2105.05837
M3 - Preprint
BT - When Does Contrastive Visual Representation Learning Work?
PB - arXiv.org
ER -
ID: 303800508