A Simple Framework for Contrastive Learning of Visual Representations

Ting Chen 1 Simon Kornblith 1 Mohammad Norouzi 1 Geoffrey Hinton 1

Abstract
This paper presents SimCLR: a simple framework
for contrastive learning of visual representations.
We simplify recently proposed contrastive self-
supervised learning algorithms without requiring
specialized architectures or a memory bank. In
order to understand what enables the contrastive
prediction tasks to learn useful representations,
we systematically study the major components of
our framework. We show that (1) composition of
data augmentations plays a critical role in defining
effective predictive tasks, (2) introducing a learn-
able nonlinear transformation between the repre-
sentation and the contrastive loss substantially im-
proves the quality of the learned representations,
Figure 1. ImageNet Top-1 accuracy of linear classifiers trained
and (3) contrastive learning benefits from larger on representations learned with different self-supervised meth-
batch sizes and more training steps compared to ods (pretrained on ImageNet). Gray cross indicates supervised
supervised learning. By combining these findings, ResNet-50. Our method, SimCLR, is shown in bold.
we are able to considerably outperform previous
methods for self-supervised and semi-supervised However, pixel-level generation is computationally expen-
learning on ImageNet. A linear classifier trained sive and may not be necessary for representation learning.
on self-supervised representations learned by Sim- Discriminative approaches learn representations using objec-
CLR achieves 76.5% top-1 accuracy, which is a tive functions similar to those used for supervised learning,
7% relative improvement over previous state-of- but train networks to perform pretext tasks where both the in-
the-art, matching the performance of a supervised puts and labels are derived from an unlabeled dataset. Many
ResNet-50. When fine-tuned on only 1% of the such approaches have relied on heuristics to design pretext
labels, we achieve 85.8% top-5 accuracy, outper- tasks (Doersch et al., 2015; Zhang et al., 2016; Noroozi &
forming AlexNet with 100× fewer labels. 1 Favaro, 2016; Gidaris et al., 2018), which could limit the
generality of the learned representations. Discriminative
approaches based on contrastive learning in the latent space
1. Introduction have recently shown great promise, achieving state-of-the-
art results (Hadsell et al., 2006; Dosovitskiy et al., 2014;
Learning effective visual representations without human Oord et al., 2018; Bachman et al., 2019).
supervision is a long-standing problem. Most mainstream
In this work, we introduce a simple framework for con-
approaches fall into one of two classes: generative or dis-
trastive learning of visual representations, which we call
criminative. Generative approaches learn to generate or
SimCLR. Not only does SimCLR outperform previous work
otherwise model pixels in the input space (Hinton et al.,
(Figure 1), but it is also simpler, requiring neither special-
2006; Kingma & Welling, 2013; Goodfellow et al., 2014).
ized architectures (Bachman et al., 2019; Hénaff et al., 2019)
1
Google Research, Brain Team. Correspondence to: Ting Chen nor a memory bank (Wu et al., 2018; Tian et al., 2019; He
<iamtingchen@google.com>. et al., 2019; Misra & van der Maaten, 2019).
Proceedings of the 37 th International Conference on Machine In order to understand what enables good contrastive repre-
Learning, Vienna, Austria, PMLR 119, 2020. Copyright 2020 by sentation learning, we systematically study the major com-
the author(s). ponents of our framework and show that:
1
Code available at https://github.com/google-research/simclr.
 A Simple Framework for Contrastive Learning of Visual Representations

• Composition of multiple data augmentation operations Maximize agreement

zi zj
is crucial in defining the contrastive prediction tasks that
yield effective representations. In addition, unsupervised g(·) g(·)
contrastive learning benefits from stronger data augmen- ←− Representation −→ hj
hi
tation than supervised learning.
• Introducing a learnable nonlinear transformation be- f (·) f (·)
tween the representation and the contrastive loss substan-
tially improves the quality of the learned representations. x̃i x̃j
• Representation learning with contrastive cross entropy t∼
T T
loss benefits from normalized embeddings and an appro- t ∼
0
x
priately adjusted temperature parameter.
• Contrastive learning benefits from larger batch sizes and
Figure 2. A simple framework for contrastive learning of visual
longer training compared to its supervised counterpart.
representations. Two separate data augmentation operators are
Like supervised learning, contrastive learning benefits sampled from the same family of augmentations (t ∼ T and
from deeper and wider networks. t0 ∼ T ) and applied to each data example to obtain two correlated
We combine these findings to achieve a new state-of-the-art views. A base encoder network f (·) and a projection head g(·)
are trained to maximize agreement using a contrastive loss. After
in self-supervised and semi-supervised learning on Ima-
training is completed, we throw away the projection head g(·) and
geNet ILSVRC-2012 (Russakovsky et al., 2015). Under the
use encoder f (·) and representation h for downstream tasks.
linear evaluation protocol, SimCLR achieves 76.5% top-1
accuracy, which is a 7% relative improvement over previous to obtain hi = f (x̃i ) = ResNet(x̃i ) where hi ∈ Rd is
state-of-the-art (Hénaff et al., 2019). When fine-tuned with the output after the average pooling layer.
only 1% of the ImageNet labels, SimCLR achieves 85.8%
top-5 accuracy, a relative improvement of 10% (Hénaff et al., • A small neural network projection head g(·) that maps
2019). When fine-tuned on other natural image classifica- representations to the space where contrastive loss is
tion datasets, SimCLR performs on par with or better than applied. We use a MLP with one hidden layer to obtain
a strong supervised baseline (Kornblith et al., 2019) on 10 zi = g(hi ) = W (2) σ(W (1) hi ) where σ is a ReLU non-
out of 12 datasets. linearity. As shown in section 4, we find it beneficial to
define the contrastive loss on zi ’s rather than hi ’s.

2. Method • A contrastive loss function defined for a contrastive pre-

diction task. Given a set {x̃k } including a positive pair
2.1. The Contrastive Learning Framework of examples x̃i and x̃j , the contrastive prediction task
aims to identify x̃j in {x̃k }k6=i for a given x̃i .
Inspired by recent contrastive learning algorithms (see Sec-
tion 7 for an overview), SimCLR learns representations We randomly sample a minibatch of N examples and define
by maximizing agreement between differently augmented the contrastive prediction task on pairs of augmented exam-
views of the same data example via a contrastive loss in ples derived from the minibatch, resulting in 2N data points.
the latent space. As illustrated in Figure 2, this framework We do not sample negative examples explicitly. Instead,
comprises the following four major components. given a positive pair, similar to (Chen et al., 2017), we treat
the other 2(N − 1) augmented examples within a minibatch
• A stochastic data augmentation module that transforms
as negative examples. Let sim(u, v) = u> v/kukkvk de-
any given data example randomly resulting in two cor-
note the dot product between `2 normalized u and v (i.e.
related views of the same example, denoted x̃i and x̃j ,
cosine similarity). Then the loss function for a positive pair
which we consider as a positive pair. In this work, we
of examples (i, j) is defined as
sequentially apply three simple augmentations: random
cropping followed by resize back to the original size, ran- exp(sim(zi , zj )/τ )
`i,j = − log P2N , (1)
dom color distortions, and random Gaussian blur. As 1[k6=i] exp(sim(zi , zk )/τ )
k=1
shown in Section 3, the combination of random crop and
color distortion is crucial to achieve a good performance. where 1[k6=i] ∈ {0, 1} is an indicator function evaluating to
1 iff k 6= i and τ denotes a temperature parameter. The fi-
• A neural network base encoder f (·) that extracts repre-
nal loss is computed across all positive pairs, both (i, j)
sentation vectors from augmented data examples. Our
and (j, i), in a mini-batch. This loss has been used in
framework allows various choices of the network archi-
previous work (Sohn, 2016; Wu et al., 2018; Oord et al.,
tecture without any constraints. We opt for simplicity
2018); for convenience, we term it NT-Xent (the normalized
and adopt the commonly used ResNet (He et al., 2016)
temperature-scaled cross entropy loss).
 A Simple Framework for Contrastive Learning of Visual Representations

Algorithm 1 SimCLR’s main learning algorithm.

input: batch size N , constant τ , structure of f , g, T .
for sampled minibatch {xk }N k=1 do D
for all k ∈ {1, . . . , N } do
A B
draw two augmentation functions t ∼ T , t0 ∼ T C
# the first augmentation
x̃2k−1 = t(xk )
h2k−1 = f (x̃2k−1 ) # representation (a) Global and local views. (b) Adjacent views.
z2k−1 = g(h2k−1 ) # projection Figure 3. Solid rectangles are images, dashed rectangles are ran-
# the second augmentation dom crops. By randomly cropping images, we sample contrastive
x̃2k = t0 (xk ) prediction tasks that include global to local view (B → A) or
h2k = f (x̃2k ) # representation adjacent view (D → C) prediction.
z2k = g(h2k ) # projection
end for 2.3. Evaluation Protocol
for all i ∈ {1, . . . , 2N } and j ∈ {1, . . . , 2N } do
Here we lay out the protocol for our empirical studies, which
si,j = zi> zj /(kzi kkzj k) # pairwise similarity
aim to understand different design choices in our framework.
end for
exp(s /τ )
define `(i, j) as `(i, j) = − log P2N 1 i,jexp(s /τ ) Dataset and Metrics. Most of our study for unsupervised
k=1 [k6=i] i,k

L = 2N 1
PN
[`(2k−1, 2k) + `(2k, 2k−1)] pretraining (learning encoder network f without labels)
k=1
update networks f and g to minimize L is done using the ImageNet ILSVRC-2012 dataset (Rus-
end for sakovsky et al., 2015). Some additional pretraining experi-
return encoder network f (·), and throw away g(·) ments on CIFAR-10 (Krizhevsky & Hinton, 2009) can be
found in Appendix B.9. We also test the pretrained results
on a wide range of datasets for transfer learning. To evalu-
Algorithm 1 summarizes the proposed method. ate the learned representations, we follow the widely used
linear evaluation protocol (Zhang et al., 2016; Oord et al.,
2.2. Training with Large Batch Size 2018; Bachman et al., 2019; Kolesnikov et al., 2019), where
To keep it simple, we do not train the model with a memory a linear classifier is trained on top of the frozen base net-
bank (Wu et al., 2018; He et al., 2019). Instead, we vary work, and test accuracy is used as a proxy for representation
the training batch size N from 256 to 8192. A batch size quality. Beyond linear evaluation, we also compare against
of 8192 gives us 16382 negative examples per positive pair state-of-the-art on semi-supervised and transfer learning.
from both augmentation views. Training with large batch Default setting. Unless otherwise specified, for data aug-
size may be unstable when using standard SGD/Momentum mentation we use random crop and resize (with random
with linear learning rate scaling (Goyal et al., 2017). To flip), color distortions, and Gaussian blur (for details, see
stabilize the training, we use the LARS optimizer (You et al., Appendix A). We use ResNet-50 as the base encoder net-
2017) for all batch sizes. We train our model with Cloud work, and a 2-layer MLP projection head to project the
TPUs, using 32 to 128 cores depending on the batch size.2 representation to a 128-dimensional latent space. As the
Global BN. Standard ResNets use batch normaliza- loss, we use NT-Xent, optimized using LARS with learning
tion (Ioffe & Szegedy, 2015). In distributed training with rate of 4.8 (= 0.3 × BatchSize/256) and weight decay of
data parallelism, the BN mean and variance are typically 10−6 . We train at batch size 4096 for 100 epochs.3 Fur-
aggregated locally per device. In our contrastive learning, thermore, we use linear warmup for the first 10 epochs,
as positive pairs are computed in the same device, the model and decay the learning rate with the cosine decay schedule
can exploit the local information leakage to improve pre- without restarts (Loshchilov & Hutter, 2016).
diction accuracy without improving representations. We ad-
dress this issue by aggregating BN mean and variance over 3. Data Augmentation for Contrastive
all devices during the training. Other approaches include Representation Learning
shuffling data examples across devices (He et al., 2019), or
replacing BN with layer norm (Hénaff et al., 2019). Data augmentation defines predictive tasks. While data
augmentation has been widely used in both supervised and
2
With 128 TPU v3 cores, it takes ∼1.5 hours to train our unsupervised representation learning (Krizhevsky et al.,
ResNet-50 with a batch size of 4096 for 100 epochs.
3
Although max performance is not reached in 100 epochs, rea-
sonable results are achieved, allowing fair and efficient ablations.
 A Simple Framework for Contrastive Learning of Visual Representations

(a) Original (b) Crop and resize (c) Crop, resize (and flip) (d) Color distort. (drop) (e) Color distort. (jitter)

(f) Rotate {90◦ , 180◦ , 270◦ } (g) Cutout (h) Gaussian noise (i) Gaussian blur (j) Sobel filtering
Figure 4. Illustrations of the studied data augmentation operators. Each augmentation can transform data stochastically with some internal
parameters (e.g. rotation degree, noise level). Note that we only test these operators in ablation, the augmentation policy used to train our
models only includes random crop (with flip and resize), color distortion, and Gaussian blur. (Original image cc-by: Von.grzanka)

2012; Hénaff et al., 2019; Bachman et al., 2019), it has Crop 33.1 33.9 56.3 46.0 39.9 35.0 30.2 39.2
50
not been considered as a systematic way to define the con-
Cutout 32.2 25.6 33.9 40.0 26.5 25.2 22.4 29.4
trastive prediction task. Many existing approaches define 40
1st transformation

contrastive prediction tasks by changing the architecture. Color 55.8 35.5 18.8 21.0 11.4 16.5 20.8 25.7

For example, Hjelm et al. (2018); Bachman et al. (2019) Sobel 46.2 40.6 20.9 4.0 9.3 6.2 4.2 18.8 30
achieve global-to-local view prediction via constraining the
Noise 38.8 25.8 7.5 7.6 9.8 9.8 9.6 15.5
receptive field in the network architecture, whereas Oord 20

et al. (2018); Hénaff et al. (2019) achieve neighboring view Blur 35.1 25.2 16.6 5.8 9.7 2.6 6.7 14.5

prediction via a fixed image splitting procedure and a con- 10

Rotate 30.0 22.5 20.7 4.3 9.7 6.5 2.6 13.8
text aggregation network. We show that this complexity can
ut l
op lor be ise
Blu
r te ge
be avoided by performing simple random cropping (with Cr
Cu
to Co So No Ro
ta
Av
er
a

resizing) of target images, which creates a family of predic- 2nd transformation

tive tasks subsuming the above mentioned two, as shown in Figure 5. Linear evaluation (ImageNet top-1 accuracy) under in-
Figure 3. This simple design choice conveniently decouples dividual or composition of data augmentations, applied only to
the predictive task from other components such as the neural one branch. For all columns but the last, diagonal entries corre-
network architecture. Broader contrastive prediction tasks spond to single transformation, and off-diagonals correspond to
can be defined by extending the family of augmentations composition of two transformations (applied sequentially). The
and composing them stochastically. last column reflects the average over the row.

3.1. Composition of data augmentation operations is To understand the effects of individual data augmentations
crucial for learning good representations and the importance of augmentation composition, we in-
vestigate the performance of our framework when applying
To systematically study the impact of data augmentation, augmentations individually or in pairs. Since ImageNet
we consider several common augmentations here. One type images are of different sizes, we always apply crop and re-
of augmentation involves spatial/geometric transformation size images (Krizhevsky et al., 2012; Szegedy et al., 2015),
of data, such as cropping and resizing (with horizontal which makes it difficult to study other augmentations in
flipping), rotation (Gidaris et al., 2018) and cutout (De- the absence of cropping. To eliminate this confound, we
Vries & Taylor, 2017). The other type of augmentation consider an asymmetric data transformation setting for this
involves appearance transformation, such as color distortion ablation. Specifically, we always first randomly crop im-
(including color dropping, brightness, contrast, saturation, ages and resize them to the same resolution, and we then
hue) (Howard, 2013; Szegedy et al., 2015), Gaussian blur, apply the targeted transformation(s) only to one branch of
and Sobel filtering. Figure 4 visualizes the augmentations the framework in Figure 2, while leaving the other branch
that we study in this work. as the identity (i.e. t(xi ) = xi ). Note that this asymmet-
 A Simple Framework for Contrastive Learning of Visual Representations

80
Sup. R50(4x)
Sup. R50(2x)
Sup. R50 R50(4x)*
75
R50(2x)*
R50(4x)
R152(2x)
R101(2x)
70
R50* R50(2x)
R34(4x)
R152
R101

Top 1
65 R18(4x)
R50
R34(2x)
(a) Without color distortion. (b) With color distortion.
60
R18(2x)
Figure 6. Histograms of pixel intensities (over all channels) for
different crops of two different images (i.e. two rows). The image
for the first row is from Figure 4. All axes have the same range. 55
R34

50 R18
Color distortion strength
Methods 1/8 1/4 1/2 1 1 (+Blur) AutoAug 0 50 100 150 200 250 300 350 400 450
Number of Parameters (Millions)
SimCLR 59.6 61.0 62.6 63.2 64.5 61.1
Supervised 77.0 76.7 76.5 75.7 75.4 77.1 Figure 7. Linear evaluation of models with varied depth and width.
Models in blue dots are ours trained for 100 epochs, models in red
stars are ours trained for 1000 epochs, and models in green crosses
Table 1. Top-1 accuracy of unsupervised ResNet-50 using linear
are supervised ResNets trained for 90 epochs7 (He et al., 2016).
evaluation and supervised ResNet-505 , under varied color distor-
tion strength (see Appendix A) and other data transformations.
Strength 1 (+Blur) is our default data augmentation policy. shown in Table 1. Stronger color augmentation substan-
tially improves the linear evaluation of the learned unsuper-
ric data augmentation hurts the performance. Nonetheless, vised models. In this context, AutoAugment (Cubuk et al.,
this setup should not substantively change the impact of 2019), a sophisticated augmentation policy found using su-
individual data augmentations or their compositions. pervised learning, does not work better than simple cropping
+ (stronger) color distortion. When training supervised mod-
Figure 5 shows linear evaluation results under individual els with the same set of augmentations, we observe that
and composition of transformations. We observe that no stronger color augmentation does not improve or even hurts
single transformation suffices to learn good representations, their performance. Thus, our experiments show that unsu-
even though the model can almost perfectly identify the pervised contrastive learning benefits from stronger (color)
positive pairs in the contrastive task. When composing aug- data augmentation than supervised learning. Although pre-
mentations, the contrastive prediction task becomes harder, vious work has reported that data augmentation is useful
but the quality of representation improves dramatically. Ap- for self-supervised learning (Doersch et al., 2015; Bachman
pendix B.2 provides a further study on composing broader et al., 2019; Hénaff et al., 2019; Asano et al., 2019), we
set of augmentations. show that data augmentation that does not yield accuracy
One composition of augmentations stands out: random crop- benefits for supervised learning can still help considerably
ping and random color distortion. We conjecture that one with contrastive learning.
serious issue when using only random cropping as data
augmentation is that most patches from an image share a 4. Architectures for Encoder and Head
similar color distribution. Figure 6 shows that color his-
tograms alone suffice to distinguish images. Neural nets 4.1. Unsupervised contrastive learning benefits (more)
may exploit this shortcut to solve the predictive task. There- from bigger models
fore, it is critical to compose cropping with color distortion Figure 7 shows, perhaps unsurprisingly, that increasing
in order to learn generalizable features. depth and width both improve performance. While similar
findings hold for supervised learning (He et al., 2016), we
3.2. Contrastive learning needs stronger data find the gap between supervised models and linear classifiers
augmentation than supervised learning trained on unsupervised models shrinks as the model size
To further demonstrate the importance of the color aug- increases, suggesting that unsupervised learning benefits
mentation, we adjust the strength of color augmentation as more from bigger models than its supervised counterpart.
7
5
Supervised models are trained for 90 epochs; longer training Training longer does not improve supervised ResNets (see
improves performance of stronger augmentation by ∼ 0.5%. Appendix B.3).
 A Simple Framework for Contrastive Learning of Visual Representations

Name Negative loss function Gradient w.r.t. u

T −
exp(uT v + /τ )
uT v + /τ − log v∈{v+ ,v− } exp(uT v/τ ) − v− exp(uZ(u)
)/τ v + v /τ )
/τ v −
P P
NT-Xent (1 − Z(u)

NT-Logistic log σ(uT v + /τ ) + log σ(−uT v − /τ ) (σ(−uT v + /τ ))/τ v + − σ(uT v − /τ )/τ v −

Margin Triplet − max(uT v − − uT v + + m, 0) v + − v − if uT v + − uT v − < m else 0

Table 2. Negative loss functions and their gradients. All input vectors, i.e. u, v + , v − , are `2 normalized. NT-Xent is an abbreviation for
“Normalized Temperature-scaled Cross Entropy”. Different loss functions impose different weightings of positive and negative examples.

70
Representation
60 What to predict? Random guess
h g(h)
Top 1

50 Projection Color vs grayscale 80 99.3 97.4

Linear Rotation 25 67.6 25.6
40 Non-linear Orig. vs corrupted 50 99.5 59.6
None Orig. vs Sobel filtered 50 96.6 56.3
30
32 64 12
8
25
6
51
2 24 48
10 20 Table 3. Accuracy of training additional MLPs on different repre-
Projection output dimensionality sentations to predict the transformation applied. Other than crop
Figure 8. Linear evaluation of representations with different pro- and color augmentation, we additionally and independently add
jection heads g(·) and various dimensions of z = g(h). The rotation (one of {0◦ , 90◦ , 180◦ , 270◦ }), Gaussian noise, and So-
representation h (before projection) is 2048-dimensional here. bel filtering transformation during the pretraining for the last three
rows. Both h and g(h) are of the same dimensionality, i.e. 2048.
4.2. A nonlinear projection head improves the
representation quality of the layer before it be found in Appendix B.4.
We then study the importance of including a projection
head, i.e. g(h). Figure 8 shows linear evaluation results 5. Loss Functions and Batch Size
using three different architecture for the head: (1) identity
5.1. Normalized cross entropy loss with adjustable
mapping; (2) linear projection, as used by several previous
temperature works better than alternatives
approaches (Wu et al., 2018); and (3) the default nonlinear
projection with one additional hidden layer (and ReLU acti- We compare the NT-Xent loss against other commonly used
vation), similar to Bachman et al. (2019). We observe that a contrastive loss functions, such as logistic loss (Mikolov
nonlinear projection is better than a linear projection (+3%), et al., 2013), and margin loss (Schroff et al., 2015). Table
and much better than no projection (>10%). When a pro- 2 shows the objective function as well as the gradient to
jection head is used, similar results are observed regardless the input of the loss function. Looking at the gradient, we
of output dimension. Furthermore, even when nonlinear observe 1) `2 normalization (i.e. cosine similarity) along
projection is used, the layer before the projection head, h, with temperature effectively weights different examples, and
is still much better (>10%) than the layer after, z = g(h), an appropriate temperature can help the model learn from
which shows that the hidden layer before the projection hard negatives; and 2) unlike cross-entropy, other objec-
head is a better representation than the layer after. tive functions do not weigh the negatives by their relative
hardness. As a result, one must apply semi-hard negative
We conjecture that the importance of using the representa-
mining (Schroff et al., 2015) for these loss functions: in-
tion before the nonlinear projection is due to loss of informa-
stead of computing the gradient over all loss terms, one can
tion induced by the contrastive loss. In particular, z = g(h)
compute the gradient using semi-hard negative terms (i.e.,
is trained to be invariant to data transformation. Thus, g can
those that are within the loss margin and closest in distance,
remove information that may be useful for the downstream
but farther than positive examples).
task, such as the color or orientation of objects. By leverag-
ing the nonlinear transformation g(·), more information can To make the comparisons fair, we use the same `2 normaliza-
be formed and maintained in h. To verify this hypothesis, tion for all loss functions, and we tune the hyperparameters,
we conduct experiments that use either h or g(h) to learn and report their best results.8 Table 4 shows that, while
to predict the transformation applied during the pretraining. (semi-hard) negative mining helps, the best result is still
Here we set g(h) = W (2) σ(W (1) h), with the same input much worse than our default NT-Xent loss.
and output dimensionality (i.e. 2048). Table 3 shows h 8
Details can be found in Appendix B.10. For simplicity, we
contains much more information about the transformation only consider the negatives from one augmentation view.
applied, while g(h) loses information. Further analysis can
 A Simple Framework for Contrastive Learning of Visual Representations

Margin NT-Logi. Margin (sh) NT-Logi.(sh) NT-Xent Method Architecture Param (M) Top 1 Top 5
50.9 51.6 57.5 57.9 63.9 Methods using ResNet-50:
Local Agg. ResNet-50 24 60.2 -
Table 4. Linear evaluation (top-1) for models trained with different MoCo ResNet-50 24 60.6 -
loss functions. “sh” means using semi-hard negative mining. PIRL ResNet-50 24 63.6 -
CPC v2 ResNet-50 24 63.8 85.3
SimCLR (ours) ResNet-50 24 69.3 89.0
`2 norm? τ Entropy Contrastive acc. Top 1
Methods using other architectures:
0.05 1.0 90.5 59.7 Rotation RevNet-50 (4×) 86 55.4 -
0.1 4.5 87.8 64.4 BigBiGAN RevNet-50 (4×) 86 61.3 81.9
Yes
0.5 8.2 68.2 60.7 AMDIM Custom-ResNet 626 68.1 -
1 8.3 59.1 58.0 CMC ResNet-50 (2×) 188 68.4 88.2
10 0.5 91.7 57.2 MoCo ResNet-50 (4×) 375 68.6 -
No CPC v2 ResNet-161 (∗) 305 71.5 90.1
100 0.5 92.1 57.0
SimCLR (ours) ResNet-50 (2×) 94 74.2 92.0
Table 5. Linear evaluation for models trained with different choices SimCLR (ours) ResNet-50 (4×) 375 76.5 93.2
of `2 norm and temperature τ for NT-Xent loss. The contrastive
Table 6. ImageNet accuracies of linear classifiers trained on repre-
distribution is over 4096 examples.
sentations learned with different self-supervised methods.
70.0

67.5
Label fraction
Method Architecture 1% 10%
65.0 Top 5

62.5
Supervised baseline ResNet-50 48.4 80.4
Methods using other label-propagation:
Top 1

60.0
Batch size Pseudo-label ResNet-50 51.6 82.4
57.5 256 VAT+Entropy Min. ResNet-50 47.0 83.4
512 UDA (w. RandAug) ResNet-50 - 88.5
55.0 1024 FixMatch (w. RandAug) ResNet-50 - 89.1
2048 S4L (Rot+VAT+En. M.) ResNet-50 (4×) - 91.2
52.5 4096
8192 Methods using representation learning only:
50.0 InstDisc ResNet-50 39.2 77.4
100 200 300 400 500 600 700 800 900 1000
Training epochs BigBiGAN RevNet-50 (4×) 55.2 78.8
PIRL ResNet-50 57.2 83.8
Figure 9. Linear evaluation models (ResNet-50) trained with differ- CPC v2 ResNet-161(∗) 77.9 91.2
ent batch size and epochs. Each bar is a single run from scratch.10 SimCLR (ours) ResNet-50 75.5 87.8
SimCLR (ours) ResNet-50 (2×) 83.0 91.2
We next test the importance of the `2 normalization (i.e. SimCLR (ours) ResNet-50 (4×) 85.8 92.6
cosine similarity vs dot product) and temperature τ in our
Table 7. ImageNet accuracy of models trained with few labels.
default NT-Xent loss. Table 5 shows that without normal-
ization and proper temperature scaling, performance is sig- supervised learning (Goyal et al., 2017), in contrastive learn-
nificantly worse. Without `2 normalization, the contrastive ing, larger batch sizes provide more negative examples,
task accuracy is higher, but the resulting representation is facilitating convergence (i.e. taking fewer epochs and steps
worse under linear evaluation. for a given accuracy). Training longer also provides more
negative examples, improving the results. In Appendix B.1,
5.2. Contrastive learning benefits (more) from larger results with even longer training steps are provided.
batch sizes and longer training
Figure 9 shows the impact of batch size when models are 6. Comparison with State-of-the-art
trained for different numbers of epochs. We find that, when
the number of training epochs is small (e.g. 100 epochs), In this subsection, similar to Kolesnikov et al. (2019); He
larger batch sizes have a significant advantage over the et al. (2019), we use ResNet-50 in 3 different hidden layer
smaller ones. With more training steps/epochs, the gaps widths (width multipliers of 1×, 2×, and 4×). For better
between different batch sizes decrease or disappear, pro- convergence, our models here are trained for 1000 epochs.
vided the batches are randomly resampled. In contrast to Linear evaluation. Table 6 compares our results with previ-
10
A linear learning rate scaling is used here. Figure B.1 shows ous approaches (Zhuang et al., 2019; He et al., 2019; Misra
using a square root learning rate scaling can improve performance & van der Maaten, 2019; Hénaff et al., 2019; Kolesnikov
of ones with small batch sizes. et al., 2019; Donahue & Simonyan, 2019; Bachman et al.,
 A Simple Framework for Contrastive Learning of Visual Representations

Food CIFAR10 CIFAR100 Birdsnap SUN397 Cars Aircraft VOC2007 DTD Pets Caltech-101 Flowers
Linear evaluation:
SimCLR (ours) 76.9 95.3 80.2 48.4 65.9 60.0 61.2 84.2 78.9 89.2 93.9 95.0
Supervised 75.2 95.7 81.2 56.4 64.9 68.8 63.8 83.8 78.7 92.3 94.1 94.2
Fine-tuned:
SimCLR (ours) 89.4 98.6 89.0 78.2 68.1 92.1 87.0 86.6 77.8 92.1 94.1 97.6
Supervised 88.7 98.3 88.7 77.8 67.0 91.4 88.0 86.5 78.8 93.2 94.2 98.0
Random init 88.3 96.0 81.9 77.0 53.7 91.3 84.8 69.4 64.1 82.7 72.5 92.5

Table 8. Comparison of transfer learning performance of our self-supervised approach with supervised baselines across 12 natural image
classification datasets, for ResNet-50 (4×) models pretrained on ImageNet. Results not significantly worse than the best (p > 0.05,
permutation test) are shown in bold. See Appendix B.8 for experimental details and results with standard ResNet-50.

2019; Tian et al., 2019) in the linear evaluation setting (see 7. Related Work
Appendix B.6). Table 1 shows more numerical compar-
isons among different methods. We are able to use standard The idea of making representations of an image agree with
networks to obtain substantially better results compared to each other under small transformations dates back to Becker
previous methods that require specifically designed archi- & Hinton (1992). We extend it by leveraging recent ad-
tectures. The best result obtained with our ResNet-50 (4×) vances in data augmentation, network architecture and con-
can match the supervised pretrained ResNet-50. trastive loss. A similar consistency idea, but for class label
prediction, has been explored in other contexts such as semi-
Semi-supervised learning. We follow Zhai et al. (2019) supervised learning (Xie et al., 2019; Berthelot et al., 2019).
and sample 1% or 10% of the labeled ILSVRC-12 training
datasets in a class-balanced way (∼12.8 and ∼128 images Handcrafted pretext tasks. The recent renaissance of self-
per class respectively). 11 We simply fine-tune the whole supervised learning began with artificially designed pretext
base network on the labeled data without regularization tasks, such as relative patch prediction (Doersch et al., 2015),
(see Appendix B.5). Table 7 shows the comparisons of solving jigsaw puzzles (Noroozi & Favaro, 2016), coloriza-
our results against recent methods (Zhai et al., 2019; Xie tion (Zhang et al., 2016) and rotation prediction (Gidaris
et al., 2019; Sohn et al., 2020; Wu et al., 2018; Donahue & et al., 2018; Chen et al., 2019). Although good results
Simonyan, 2019; Misra & van der Maaten, 2019; Hénaff can be obtained with bigger networks and longer train-
et al., 2019). The supervised baseline from (Zhai et al., ing (Kolesnikov et al., 2019), these pretext tasks rely on
2019) is strong due to intensive search of hyper-parameters somewhat ad-hoc heuristics, which limits the generality of
(including augmentation). Again, our approach significantly learned representations.
improves over state-of-the-art with both 1% and 10% of the Contrastive visual representation learning. Dating back
labels. Interestingly, fine-tuning our pretrained ResNet-50 to Hadsell et al. (2006), these approaches learn represen-
(2×, 4×) on full ImageNet are also significantly better then tations by contrasting positive pairs against negative pairs.
training from scratch (up to 2%, see Appendix B.2). Along these lines, Dosovitskiy et al. (2014) proposes to
Transfer learning. We evaluate transfer learning perfor- treat each instance as a class represented by a feature vector
mance across 12 natural image datasets in both linear evalu- (in a parametric form). Wu et al. (2018) proposes to use
ation (fixed feature extractor) and fine-tuning settings. Fol- a memory bank to store the instance class representation
lowing Kornblith et al. (2019), we perform hyperparameter vector, an approach adopted and extended in several recent
tuning for each model-dataset combination and select the papers (Zhuang et al., 2019; Tian et al., 2019; He et al.,
best hyperparameters on a validation set. Table 8 shows 2019; Misra & van der Maaten, 2019). Other work explores
results with the ResNet-50 (4×) model. When fine-tuned, the use of in-batch samples for negative sampling instead
our self-supervised model significantly outperforms the su- of a memory bank (Doersch & Zisserman, 2017; Ye et al.,
pervised baseline on 5 datasets, whereas the supervised 2019; Ji et al., 2019).
baseline is superior on only 2 (i.e. Pets and Flowers). On Recent literature has attempted to relate the success of their
the remaining 5 datasets, the models are statistically tied. methods to maximization of mutual information between
Full experimental details as well as results with the standard latent representations (Oord et al., 2018; Hénaff et al., 2019;
ResNet-50 architecture are provided in Appendix B.8. Hjelm et al., 2018; Bachman et al., 2019). However, it is not
11 clear if the success of contrastive approaches is determined
The details of sampling and exact subsets can be found in
https://www.tensorflow.org/datasets/catalog/imagenet2012_subset. by the mutual information, or by the specific form of the
contrastive loss (Tschannen et al., 2019).
 A Simple Framework for Contrastive Learning of Visual Representations

We note that almost all individual components of our frame- Chen, T., Sun, Y., Shi, Y., and Hong, L. On sampling strategies
work have appeared in previous work, although the specific for neural network-based collaborative filtering. In Proceed-
instantiations may be different. The superiority of our frame- ings of the 23rd ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining, pp. 767–776, 2017.
work relative to previous work is not explained by any single
design choice, but by their composition. We provide a com- Chen, T., Zhai, X., Ritter, M., Lucic, M., and Houlsby, N. Self-
prehensive comparison of our design choices with those of supervised gans via auxiliary rotation loss. In Proceedings of the
previous work in Appendix C. IEEE Conference on Computer Vision and Pattern Recognition,
pp. 12154–12163, 2019.
8. Conclusion Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi,
A. Describing textures in the wild. In IEEE Conference on
In this work, we present a simple framework and its in- Computer Vision and Pattern Recognition (CVPR), pp. 3606–
stantiation for contrastive visual representation learning. 3613. IEEE, 2014.
We carefully study its components, and show the effects
of different design choices. By combining our findings, Cubuk, E. D., Zoph, B., Mane, D., Vasudevan, V., and Le, Q. V.
we improve considerably over previous methods for self- Autoaugment: Learning augmentation strategies from data. In
Proceedings of the IEEE conference on computer vision and
supervised, semi-supervised, and transfer learning. pattern recognition, pp. 113–123, 2019.
Our approach differs from standard supervised learning on
DeVries, T. and Taylor, G. W. Improved regularization of
ImageNet only in the choice of data augmentation, the use of
convolutional neural networks with cutout. arXiv preprint
a nonlinear head at the end of the network, and the loss func- arXiv:1708.04552, 2017.
tion. The strength of this simple framework suggests that,
despite a recent surge in interest, self-supervised learning Doersch, C. and Zisserman, A. Multi-task self-supervised visual
remains undervalued. learning. In Proceedings of the IEEE International Conference
on Computer Vision, pp. 2051–2060, 2017.

Acknowledgements Doersch, C., Gupta, A., and Efros, A. A. Unsupervised visual

representation learning by context prediction. In Proceedings
We would like to thank Xiaohua Zhai, Rafael Müller and of the IEEE International Conference on Computer Vision, pp.
Yani Ioannou for their feedback on the draft. We are also 1422–1430, 2015.
grateful for general support from Google Research teams in Donahue, J. and Simonyan, K. Large scale adversarial representa-
Toronto and elsewhere. tion learning. In Advances in Neural Information Processing
Systems, pp. 10541–10551, 2019.
References Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E.,
Asano, Y. M., Rupprecht, C., and Vedaldi, A. A critical analysis and Darrell, T. Decaf: A deep convolutional activation feature
of self-supervision, or what we can learn from a single image. for generic visual recognition. In International Conference on
arXiv preprint arXiv:1904.13132, 2019. Machine Learning, pp. 647–655, 2014.

Bachman, P., Hjelm, R. D., and Buchwalter, W. Learning rep- Dosovitskiy, A., Springenberg, J. T., Riedmiller, M., and Brox, T.
resentations by maximizing mutual information across views. Discriminative unsupervised feature learning with convolutional
In Advances in Neural Information Processing Systems, pp. neural networks. In Advances in neural information processing
15509–15519, 2019. systems, pp. 766–774, 2014.

Everingham, M., Van Gool, L., Williams, C. K., Winn, J., and
Becker, S. and Hinton, G. E. Self-organizing neural network that
Zisserman, A. The pascal visual object classes (voc) challenge.
discovers surfaces in random-dot stereograms. Nature, 355
International Journal of Computer Vision, 88(2):303–338, 2010.
(6356):161–163, 1992.
Fei-Fei, L., Fergus, R., and Perona, P. Learning generative visual
Berg, T., Liu, J., Lee, S. W., Alexander, M. L., Jacobs, D. W., models from few training examples: An incremental bayesian
and Belhumeur, P. N. Birdsnap: Large-scale fine-grained visual approach tested on 101 object categories. In IEEE Conference
categorization of birds. In IEEE Conference on Computer Vision on Computer Vision and Pattern Recognition (CVPR) Workshop
and Pattern Recognition (CVPR), pp. 2019–2026. IEEE, 2014. on Generative-Model Based Vision, 2004.

Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, Gidaris, S., Singh, P., and Komodakis, N. Unsupervised represen-
A., and Raffel, C. A. Mixmatch: A holistic approach to semi- tation learning by predicting image rotations. arXiv preprint
supervised learning. In Advances in Neural Information Pro- arXiv:1803.07728, 2018.
cessing Systems, pp. 5050–5060, 2019.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-
Bossard, L., Guillaumin, M., and Van Gool, L. Food-101–mining Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative
discriminative components with random forests. In European adversarial nets. In Advances in neural information processing
conference on computer vision, pp. 446–461. Springer, 2014. systems, pp. 2672–2680, 2014.
 A Simple Framework for Contrastive Learning of Visual Representations

Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Loshchilov, I. and Hutter, F. Sgdr: Stochastic gradient descent
Kyrola, A., Tulloch, A., Jia, Y., and He, K. Accurate, large with warm restarts. arXiv preprint arXiv:1608.03983, 2016.
minibatch sgd: Training imagenet in 1 hour. arXiv preprint
arXiv:1706.02677, 2017. Maaten, L. v. d. and Hinton, G. Visualizing data using t-sne. Jour-
nal of machine learning research, 9(Nov):2579–2605, 2008.
Hadsell, R., Chopra, S., and LeCun, Y. Dimensionality reduction
by learning an invariant mapping. In 2006 IEEE Computer So- Maji, S., Kannala, J., Rahtu, E., Blaschko, M., and Vedaldi, A.
ciety Conference on Computer Vision and Pattern Recognition Fine-grained visual classification of aircraft. Technical report,
(CVPR’06), volume 2, pp. 1735–1742. IEEE, 2006. 2013.

He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for Mikolov, T., Chen, K., Corrado, G., and Dean, J. Efficient esti-
image recognition. In Proceedings of the IEEE conference on mation of word representations in vector space. arXiv preprint
computer vision and pattern recognition, pp. 770–778, 2016. arXiv:1301.3781, 2013.

He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Momentum Misra, I. and van der Maaten, L. Self-supervised learn-
contrast for unsupervised visual representation learning. arXiv ing of pretext-invariant representations. arXiv preprint
preprint arXiv:1911.05722, 2019. arXiv:1912.01991, 2019.

Hénaff, O. J., Razavi, A., Doersch, C., Eslami, S., and Oord, A. Nilsback, M.-E. and Zisserman, A. Automated flower classification
v. d. Data-efficient image recognition with contrastive predictive over a large number of classes. In Computer Vision, Graphics &
coding. arXiv preprint arXiv:1905.09272, 2019. Image Processing, 2008. ICVGIP’08. Sixth Indian Conference
on, pp. 722–729. IEEE, 2008.
Hinton, G. E., Osindero, S., and Teh, Y.-W. A fast learning al-
Noroozi, M. and Favaro, P. Unsupervised learning of visual repre-
gorithm for deep belief nets. Neural computation, 18(7):1527–
sentations by solving jigsaw puzzles. In European Conference
1554, 2006.
on Computer Vision, pp. 69–84. Springer, 2016.
Hjelm, R. D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K.,
Oord, A. v. d., Li, Y., and Vinyals, O. Representation learning with
Bachman, P., Trischler, A., and Bengio, Y. Learning deep repre-
contrastive predictive coding. arXiv preprint arXiv:1807.03748,
sentations by mutual information estimation and maximization.
2018.
arXiv preprint arXiv:1808.06670, 2018.
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. Cats
Howard, A. G. Some improvements on deep convolutional and dogs. In IEEE Conference on Computer Vision and Pattern
neural network based image classification. arXiv preprint Recognition (CVPR), pp. 3498–3505. IEEE, 2012.
arXiv:1312.5402, 2013.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma,
Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.
network training by reducing internal covariate shift. arXiv Imagenet large scale visual recognition challenge. International
preprint arXiv:1502.03167, 2015. journal of computer vision, 115(3):211–252, 2015.
Ji, X., Henriques, J. F., and Vedaldi, A. Invariant information Schroff, F., Kalenichenko, D., and Philbin, J. Facenet: A unified
clustering for unsupervised image classification and segmenta- embedding for face recognition and clustering. In Proceed-
tion. In Proceedings of the IEEE International Conference on ings of the IEEE conference on computer vision and pattern
Computer Vision, pp. 9865–9874, 2019. recognition, pp. 815–823, 2015.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. Simonyan, K. and Zisserman, A. Very deep convolutional
arXiv preprint arXiv:1312.6114, 2013. networks for large-scale image recognition. arXiv preprint
arXiv:1409.1556, 2014.
Kolesnikov, A., Zhai, X., and Beyer, L. Revisiting self-supervised
visual representation learning. In Proceedings of the IEEE Sohn, K. Improved deep metric learning with multi-class n-pair
conference on Computer Vision and Pattern Recognition, pp. loss objective. In Advances in neural information processing
1920–1929, 2019. systems, pp. 1857–1865, 2016.
Kornblith, S., Shlens, J., and Le, Q. V. Do better ImageNet models Sohn, K., Berthelot, D., Li, C.-L., Zhang, Z., Carlini, N., Cubuk,
transfer better? In Proceedings of the IEEE conference on E. D., Kurakin, A., Zhang, H., and Raffel, C. Fixmatch: Simpli-
computer vision and pattern recognition, pp. 2661–2671, 2019. fying semi-supervised learning with consistency and confidence.
arXiv preprint arXiv:2001.07685, 2020.
Krause, J., Deng, J., Stark, M., and Fei-Fei, L. Collecting a
large-scale dataset of fine-grained cars. In Second Workshop on Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D.,
Fine-Grained Visual Categorization, 2013. Erhan, D., Vanhoucke, V., and Rabinovich, A. Going deeper
with convolutions. In Proceedings of the IEEE conference on
Krizhevsky, A. and Hinton, G. Learning multiple layers of features computer vision and pattern recognition, pp. 1–9, 2015.
from tiny images. Technical report, University of Toronto,
2009. URL https://www.cs.toronto.edu/~kriz/ Tian, Y., Krishnan, D., and Isola, P. Contrastive multiview coding.
learning-features-2009-TR.pdf. arXiv preprint arXiv:1906.05849, 2019.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classifi- Tschannen, M., Djolonga, J., Rubenstein, P. K., Gelly, S., and Lu-
cation with deep convolutional neural networks. In Advances in cic, M. On mutual information maximization for representation
neural information processing systems, pp. 1097–1105, 2012. learning. arXiv preprint arXiv:1907.13625, 2019.
 A Simple Framework for Contrastive Learning of Visual Representations

Wu, Z., Xiong, Y., Yu, S. X., and Lin, D. Unsupervised feature
learning via non-parametric instance discrimination. In Proceed-
ings of the IEEE Conference on Computer Vision and Pattern
Recognition, pp. 3733–3742, 2018.

Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. Sun
database: Large-scale scene recognition from abbey to zoo. In
IEEE Conference on Computer Vision and Pattern Recognition
(CVPR), pp. 3485–3492. IEEE, 2010.

Xie, Q., Dai, Z., Hovy, E., Luong, M.-T., and Le, Q. V. Unsu-
pervised data augmentation. arXiv preprint arXiv:1904.12848,
2019.

Ye, M., Zhang, X., Yuen, P. C., and Chang, S.-F. Unsupervised
embedding learning via invariant and spreading instance feature.
In Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, pp. 6210–6219, 2019.
You, Y., Gitman, I., and Ginsburg, B. Large batch training of con-
volutional networks. arXiv preprint arXiv:1708.03888, 2017.

Zhai, X., Oliver, A., Kolesnikov, A., and Beyer, L. S4l: Self-
supervised semi-supervised learning. In The IEEE International
Conference on Computer Vision (ICCV), October 2019.

Zhang, R., Isola, P., and Efros, A. A. Colorful image coloriza-

tion. In European conference on computer vision, pp. 649–666.
Springer, 2016.

Zhuang, C., Zhai, A. L., and Yamins, D. Local aggregation for

unsupervised learning of visual embeddings. In Proceedings
of the IEEE International Conference on Computer Vision, pp.
6002–6012, 2019.