## Physics in a diverse world

### or A Spherical Cow Model of Physics Talent

### Talk at the KITP Conference: Snowmass Theory Frontier

### Feb. 23, 2022

### Howard Georgi

I suspect that many of you, when you think about diversity in physics feel as
I did for much of my life. We say to ourselves something like “I am a good
person and I would like to increase diversity in physics but we face societal
obstacles not of our own making and there is nothing I can do.”

I don’t want this talk to sound judgmental and I certainly don’t want to hold
myself up as a role model. In fact, I am very embarrassed that it took me
almost 45 years to realize that assuming there is nothing I can do is both
morally untenable and very bad for physics. For me, the epiphany came when I
was chairman of the Harvard Physics Department in 1992 and it had to do with
gender diversity. By this time I had been teaching at Harvard for over 15
years and I was very spoiled.

I thought that my job as a physicist was just to do good physics. Helen Quinn
was at Harvard when I arrived as a postdoc and we had many discussions about
the dearth of women in physics. I knew it was a problem but I didn’t see how I
could help.

I thought that my job as a teacher was just to organize a subject in the
deepest and most interesting way (which I enjoyed), come up with devilishly
clever problems (which I also enjoyed) and give lectures (which I was never
very good at). I was lucky to be at an institution where I could get away with
this and where there were always some students who were excited by the
challenge of assimilating my oracular pronouncements and struggling with my
crazy problems. Through no particular fault of my own, I had many spectacular
graduate students whom I thought of as just great young colleagues. Our high
energy theory group in the ’80s included Shelly Glashow and Steve Weinberg, so
we had our pick of students who wanted to do particle phenomenology, and many
of them ended up working for me, including four amazing women: Sally Dawson,
Ann Nelson, Lisa Randall and Liz Simmons. Ann Nelson’s graduate class in
particular was very small with several brilliant women and other diverse and
interesting characters. This class had a big positive influence on the culture
of the department. For example, they contributed to the mental health of
graduate students by initiating the “Puppet Show” in which the second-year
students made anonymous fun of the faculty (each of us represented by a silly
puppet) to give the first-year students the real scoop about the department.
This wonderful tradition is still going on.

From that time on, I spent a lot of time on the graduate admissions committee
trying to ensure that the women applicants were treated fairly. The number of
racial minorities in the applicant pool was still almost zero. But when the
women graduate students tried to explain that there was a real problem for
women in physics, I still did not get it.

Meanwhile I was having a wonderful time teaching undergraduate courses as well
as my graduate courses. Many undergrads hung out in my office and I got to
know many of them, men and women, very well. There were some amazing kids.
Again, some of the women tried to explain how difficult the physics culture
was for them, but it just wasn’t getting through my thick skull.

What happened in 1992, to make a long story short, was that I had some data
broken down by gender that showed that the young men who graduated in physics
loved their time in the Harvard Physics department and the women hated it! And
these were women who loved physics so much that they stuck it out and
graduated in physics in spite of the fact that they felt like outcasts.
Finally I understood what the women had been telling us. They were in an
abusive relationship with the Harvard Physics Department. This was just not
right!

Ever since that time, I have done my best to encourage diversity. The effort
has been extremely rewarding. I will say a bit more about it in a few minutes.

So what is wrong with saying “there is nothing I can do”?

We absolutely do face societal obstacles and we must do everything in our
power to break them down. Many children who could develop into outstanding
scientists cannot even imagine a career as a physicist and far too many
minority kids would not be able to afford to pursue a physics career even if
they dreamed of it. And it is an understatement to say that one of our major
political parties is not trying to change this. I know that many of you are
working hard with your outreach and your teaching to chip away at this
problem.

But we also put up a daunting obstacle of our own making. Perhaps
subconsciously, we are drawn to a metaphor of physics as survival of the
fittest and we look for the apex predator who will aggressively claw a way to
the top of the food chain. So in our teaching and our admissions and in our
hiring and most importantly in our own heads we make ordered lists and we
search for “the best.”

This is not surprising. We are drawn to physics because we want to give real
answers to real questions. We don’t make up our questions. We try to uncover
how the world works at the deepest levels. And we don’t make up our answers.
We are not satisfied unless our understanding is quantitative and expressible
in the language of mathematics and does not depend on how cleverly we express
it in human language. And we submit our answers to the ultimate test -
quantitative comparison with experiment. So naturally, we tend to quantify our
thinking about physicists. Then we say that if white male applicant A is
higher on our ordered list than minority applicant B we must choose A and
there is nothing we can do about diversity. This is absolute nonsense!

I am absolutely ABSOLUTELY NOT suggesting naive affirmative action. Few things
damage the cause of diversity more than choosing a minority applicant who is
not outstanding. The first requirement for any candidate is that they must be
able to do a great job. The trouble is that for physics, one doesn’t know
exactly what the job is and the assumption that we can unambiguously order
outstanding people makes no sense. We physicists should know better than
anyone that there are quantitative questions that do not have answers, like
which of two space-like separated events comes first.

In fact, I believe, there is more to this issue than just removing obstacles
to diversity. The right metaphor, I believe, gives us a powerful incentive to
increase diversity in physics — not just because it is the right thing to do,
but because it is important for physics. Here I want to talk to you
established physicists out there. I know that each of you has incredible
skills that you have worked hard to hone and use productively. But I also
believe strongly that each of you has a DIFFERENT cocktail of skills. If you
are good at people-watching, you will know this. I see this myself in many
ways. Looking inward, I know that there are some things that I am MUCH better
at than other things. Looking outward, I see a similar incredible diversity of
skills in my students and my younger colleagues. And looking back on my long
career, I have been fortunate to work with many great physicists in my own
subfield of particle theory. I have had the privilege of collaborating closely
with almost ninety amazing physicists, including dozens of winners of various
physics prizes. And of course I have gotten to know many people very well
without actually collaborating. I shared an office for a number of years with
Ed Witten which was a humbling experience. All of these people are simply
amazing intellects, but what I have observed is that each one is amazing in a
very unique way. Here I do not mean how they talk about physics. All of these
characters speak the language of physics, whatever natural language they are
comfortable with. But if you get to know them well enough to glimpse what is
going on in their heads, you see wild diversity.

The conclusion I draw from many years of physicist watching is that if you
want to quantify what makes a great physicist, you must use a space with very
many dimensions, a different dimension for each of the very many possible ways
of thinking that may be important for really interesting problems. I sometimes
imagine a spherical cow model of physics talent in N dimensions where N is
large and talent in each dimension increases from 0 at the origin to 1 at the
boundary, the N-dimensional version of the positive octant of a sphere. This,
I learned from Wikipedia, is called an “orthant”. Each point in my
N-dimensional orthant is a possible set of talents for physics. Great
physicists are out near the boundary, far away from the origin. If we assume
that the talents are uniformly distributed, you can see that in my spherical
cow model, the fraction of possible physics talents within $\epsilon$ of the
boundary grows like N times $\epsilon$ for small $\epsilon$. If N is very
large, as I think it is, that means that there is a lot of space near the
boundary! What this suggests to me is that there are a huge number of ways of
being a great physicist and that in turn suggests there are many ways of being
a great physicist that we haven’t seen yet. Diversity is critical to the
future of physics, I believe, because it is imperative to explore this vast
space of physics talent and that means encouraging people who are different
and who think and act differently.

You can quibble with the details of my spherical cow orthant, but I will be
very happy if this helps some of you recognize how damaging it is to rely on
one dimensional ordered lists in dealing with people. If such a list means
anything at all, it represents only some arbitrary one-dimensional projection
from some much higher dimensional space. This is why I think that working for
diversity in physics is not just a moral imperative, it is the most useful
approach for our groups, for our institutions, and for the field of physics
that we all love. To do this effectively, you should keep the spherical cow in
mind.

In your teaching, you probably have to give grades, but I think you should
give your students sub-grades for several ways of excelling in your courses
and you should keep track of each of them separately. This makes it much
easier to encourage and get to know a diverse set of students and to explain
that the final grade is a somewhat arbitrary combination. You should get to
know your students as people and celebrate their uniqueness and try to
encourage them to find and recognize and be proud of their particular
strengths.

In your mentoring, I believe that you should not try to force a younger
colleague into a predetermined mold but instead encourage them to develop
their unique strengths and help them to understand how to display them most
effectively. Sometimes this also means working to help your older colleagues
accept new ways of thinking.

The most important thing you can do for diversity is to hire diverse faculty.
I have seen this effect dramatically for women in my own department. We still
have a long way to go but having a group of outstanding women faculty who are
very different among themselves has made a big change in the culture of the
department and in the morale of the students. I could not have been happier
when three of our women undergrads won Rhodes Scholarships this year. All of
them are very active in our women in physics group. I look forward to the day
when our department is equally diverse racially and culturally. I think it
will happen if we work harder to consider candidates as people with multiple
skills rather than numbers in a ranked list.

There are certainly some very rare super-super-stars who are so far out in
some direction in the space of physics talent that they are clearly unique. If
you can find another Ann Nelson or Ed Witten you should hire her independent
of her minority status. But most of our candidates, like most of us, will just
be “ordinary” good physicists who have a combination of skills and, if they
are lucky, those skills will sometimes be the right skills to solve important
problems. Selection committees need to avoid overly rigid definitions of sub-
field so they can search broadly. They need to avoid confusing aggressiveness
or facileness with ability. They need to apply an appropriate implicit bias
factor for candidates who look exactly like the selection committee and the
current department. They need to avoid relying on the “old-boy network” and to
make a special effort to identify promising candidates who have not come up
through the usual channels. None of this is easy.

At the national level, you need to educate yourselves about the outstanding
minority physicists in your own subfields. Invite them to give important
talks. Nominate them for prizes.

While I care deeply about these issues, I know that my own view is narrow and
personal and also probably outdated in many ways because the landscape is
changing with time. So I am delighted that we have a panel of three young
physicists who will, I hope, give you a more current perspective on these
issues. I am very grateful to them for being willing to share their thoughts.

I can’t give a talk like this without getting choked up at the thought of the
incalculable loss to Physics of my former student Ann Nelson. Ann was one of
the most remarkable of the amazing people I have been privileged to know. I
have said before that while I have had many students who are much better than
I am at many things, Ann was my only student who was much better than I am at
what I do best. But she was not only an amazingly good physicist. She was an
amazingly good person. I am very grateful to Devin for inviting me to honor
Ann’s legacy in this session. Let me close by quoting from her backpage
article in a 2017 Physics Today.

“If your career is established and you are not making an explicit and
continual effort to encourage, mentor, and support all young physicists, to
create a welcoming climate in your department, and to promote the hiring of
diverse faculty members, you are part of the problem. This is a critical issue
of civil rights in our field.”

I have tried to argue here that this is more than a civil rights issue. It is
the best way to ensure that Physics will continue to be great.

So what is your assignment?

Your first job is to do great physics and enjoy it and communicate your
excitement to the next generation. But you owe it to this field that we love
to work to increase its diversity.

The most important thing is to keep at it! This is a job for optimists.
Progress will always be slower than we would like. That is just the way the
world works. But progress will not happen at all unless the good people who
think that there is nothing they can do actually wake up — and start doing.


Vertical Strain-Induced Modification of the Electrical and Spin Properties of
Monolayer MoSi2X4 (X= N, P, As and Sb)

# –ٰ

Shoeib Babaee Touski  Department of Electrical Engineering, Hamedan University
of Technology, Hamedan, Iran  Nayereh Ghobadi<EMAIL_ADDRESS>Department
of Electrical Engineering, University of Zanjan, Zanjan, Iran

###### Abstract

In this work, the electrical and spin properties of monolayer MoSi2X4 (X= N,
P, As, and Sb) under vertical strain are investigated. The band structures
state that MoSi2N4 is an indirect semiconductor, whereas other compounds are
direct semiconductors. The vertical strain has been selected to modify the
electrical properties. The bandgap shows a maximum and decreases for both
tensile and compressive strains. The valence band at K-point displays a large
spin-splitting, whereas the conduction band has a negligible splitting. On the
other hand, the second conduction band has a large spin-splitting and moves
down under vertical strain which leads to a large spin-splitting in both
conduction and valence bands edges. The projected density of states along with
the projected band structure clarifies the origin of these large spin-
splittings. These three spin-splittings can be controlled by vertical strain.

##  I Introduction

The graphene, the first member of two-dimensional (2D) materials, with
honeycomb structure does not exhibit an electronic band gap Neto et al.
(2009). This problem has motivated the researchers to explore and design novel
two-dimensional semiconductors, such as the transition metal dichalcogenides
Radisavljevic et al. (2011), black phosphorene Guan et al. (2014), antimonene
Zhang et al. (2015) and indium selenideMudd et al. (2013). Therefore,
exploring new 2D materials with proper electronic properties for particular
applications is highly demanded. After monolayer TMDC, 2D transition metal
nitride (TMN) have been proposed in the recent years Li et al. (2017); Wang et
al. (2016); Ozdemir (2018). Since TMNs do not contain layered structures, the
fabrication of large-area TMNs monolayers has remained challenging. Recently,
MoSi2N4 monolayer based on TMN and without 3D counterpart structure has been
synthesizedHong et al. (2020). MoSi2N4 is constructed from a MoN2 monolayer
sandwiched between two Si-N monolayers. By analyzing MoSi2N4 monolayer, it can
be realized that with sandwiching a TMDC-type MZ2 monolayer into InSe-type
A2Z2, twelve different structures with MA2Z4 formula have been achieved. They
demonstrate different phases such as semiconductor, topological insulators,
and Ising superconductors Wang et al. (2021). Monolayer MoSi2N4 has been
successfully fabricated using chemical vapor deposition with large size of up
to 15mm$\times$15mmHong et al. (2020). This compound demonstrates high
strength and excellent stability at ambient. MoSi2N4 compound also shows an
elastic constant three times larger than monolayer MoS2. In addition, the
large electron and hole mobility is estimated to be 270/1200 cm2/Vs that is
four to six times higher than MoS2 ones. This high mobility along with
outstanding stability in the ambient environment makes this material a
promising candidate for future applications. Other members of MoSi2N4 family
with MA2Z4 formula (M = Transition metal, A = Si, Ge and Z = N, P, As) have
been extensively investigated Zhong et al. (2021); Wang et al. (2021). These
monolayers exhibit a wide range from semiconducting to metallic properties. In
addition, some compounds with magnetic transition metal elements also
represent magnetic propertiesLi et al. (2021). The MoSi2N4 and WSi2N4
monolayers also show high lattice thermal conductivity for thermoelectric
applications Mortazavi et al. (2021); Yu et al. (2021). Monolayer MoSi2N4 and
its family also have suitable band gaps up to 1.73eV for potential optical
applications in the visible rangeBafekry et al. (2021a); Yao et al. (2021). It
has been anticipated that MoSi2N4 family compounds have a pair of valley
pseudospins similar to monolayer MoS2 Yang et al. (2021). These monolayers
have been also predicted to exceed monolayer MoS2 due to fascinating
valleytronic properties, for instance, multiple-folded valleys in monolayer
MoSi2As4. The lack of inversion symmetry along with strong spin-orbit coupling
from transition metal elements in the MA2Z4 family results in two inequivalent
valleys (K and K’) with sizable spin-splitting and spin-valley coupling Ai et
al. (2021). For example, WSi2N4 demonstrates a large spin-splitting at both
conduction and valence bands at K and K’-valleys together with spin-valley
coupling. It has been confirmed theoretically and experimentally that a
semiconductor to metal transition occurs in multi-layer MoS2 by applying
vertical strain Nayak et al. (2014); Chi et al. (2014); Ghobadi (2019).
Totally, the decrease of interlayer distance may lead to charge redistribution
and semiconductor to metal transition. The out-of-plane strain is presented as
a powerful tool to tune the electrical properties of bilayer MA2Z4 family
Bafekry et al. (2021b). For instance, the bandgap of bilayer MoSi2N4 decreases
with increasing compressive strain and the bandgap closes at the strain of
22$\%$. Such a semiconductor to metal transition also occurs in other MA2Z4
bilayers Zhong et al. (2021). The transition pressure is distributed from 2.18
GPa in CrSi2N4 to 32.04 GPa in TiSi2N4. In another work, the biaxial strain is
used to tune the band gaps of bilayer MoSi2N4 and WSi2N4 Wu et al. (2021). It
has been reported that these compounds demonstrate a direct bandgap at
compressive strain. Furthermore, it has been shown that strain can also modify
the electronic and magnetic properties of VSi2P4 monolayer Guo et al. (2020).
The strain increasing changes the phase of this monolayer from a ferromagnetic
metal to a spin-gapless semiconductor, afterward to a ferromagnetic
semiconductor, and then come back to spin-gapless semiconductor and finally to
a ferromagnetic half-metal. In this work, the effects of vertical strain on
the electrical and spin properties of monolayer MoSi2X4 (X=N, P, As, and Sb)
are investigated. First, the electrical properties such as bandgap, effective
mass, and band edge locations are obtained at equilibrium. In the following,
vertical strain is applied to all materials, and the variation of band gaps,
band edges, and charge densities are explored. Finally, spin-splitting at both
equilibrium and strained samples is studied.

##  II Computational details

In order to investigate the electrical and spin properties of MoSi2X4 (X= N,
P, As and Sb) monolayers, density functional calculations are performed using
the SIESTA package Soler et al. (2002). The generalized gradient approximation
(GGA) with the Perdew-Burke-Ernzerhof (PBE) Perdew and Zunger (1981)
functional is employed for the exchange-correlation term. We have adopted
fully relativistic pseudopotentials and have taken into account the effect of
spin-orbit coupling (SOC). A Monkhorst-Pack k-point grid of $21\times 21\times
1$ is chosen for the unit-cell. The energy cutoff is set to be 200 Ry and a
double-$\zeta$ plus polarization basis-set is used. The total energy is
converged to better than $10^{-5}$ eV and the geometries are fully relaxed
until the force on each atom is less than 0.02 eV/$\mathrm{\AA}$. A vacuum
region of 30 $\mathrm{\AA}$ is added to avoid interactions in the normal
direction. To visualize the atomic structures, XCrySDen package has been used
Kokalj (2003). The out-of-plane strain is defined as
$\varepsilon=(t-t_{0})/t_{0}$, where $t_{0}$ and $t$ are the equilibrium and
deformed compound thickness, respectively. The effective masses are calculated
by using the following equation Touski and Ghobadi (2020); Ghobadi and Touski
(2020),

$m^{*}=\hbar^{2}/\left(\partial^{2}E/\partial k^{2}\right)$ (1)

Here, $\hbar$ is the reduced Planck constant, E and k are the energy and wave
vector of conduction band minimum and valence band maximum.

Figure 1: The schematic of the studied materials with MoSi2X4 formula. The atomic structure of MoSi2N4 is depicted. N atoms are replaced by P, As and Sb in other studied structures. Figure 2: Phonon dispersion relations for four compounds. Figure 3: Two-dimensional map of the conduction and valence bands for all compounds. The first Brillouin zone is indicated for MoSi2N4 as a sample. Top and down rows are for the first conduction and valence bands, respectively. Table 1:  The lattice constant (a), the Mo-X (dMo-X) and Si-X (dSi-X) bond lengths, the vertical distance between Si and X atoms ($\Delta_{Si-X}$), the thickness and the elastic constants (C11 and C12) of MoSi2X4 monolayers.  |  a($\mathrm{\AA}$) |  dMo-X($\mathrm{\AA}$) |  dSi-X($\mathrm{\AA}$) |  $\Delta_{Si-X}$($\mathrm{\AA}$) |  Thickness($\mathrm{\AA}$) |  C11(N/m) |  C12(N/m)
---|---|---|---|---|---|---|---
MoSi2N4 |  2.928 |  2.111 |  1.763 |  0.531 |  7.119 |  472.66 |  142.02
MoSi2P4 |  3.486 |  2.477 |  2.253 |  1.046 |  9.486 |  207.77 |  58.46
MoSi2As4 |  3.633 |  2.583 |  2.366 |  1.136 |  10.021 |  178.03 |  54.75
MoSi2Sb4 |  3.898 |  2.777 |  2.589 |  1.278 |  10.991 |  141.40 |  47.72

##  III Results and discussion

The schematic of the studied compounds is shown in Fig. 1. As it is obvious, a
MoN2 monolayer is sandwiched between two SiN monolayers. The unsaturated Si
atoms in SiN sub-layer bond with unsaturated N atoms in MoN2. The structural
and mechanical properties of four monolayers are presented in Table 1. The
lattice constant of MoSi2N4 is 2.928 $\AA$ that is close to experimental
resultHong et al. (2020). The lattice constants of MoSi2P4 and MoSi2As4 are
also 3.486 and 3.633 $\AA$, respectively, that are compatible with previous
studies Guo et al. (2020); Ai et al. (2021). The lattice constant increases
for the compounds with heavier X elements. This behavior can be observed in
the Mo-X bond length ($d_{Mo-X}$) in MoX2 sub-layer, Si-X bond length, and
buckling height of SiN sub-layer. Thickness of these compounds demonstrates a
high value of 7.119 for MoSi2N4 to 10.991 $\AA$ for MoSi2Sb4. This high
thickness confirms that applying vertical strain is feasible for these
compounds. The elastic constants C11 and C12 are also obtained and reported in
the table which is compatible with previously reported amounts Yao et al.
(2021). These materials are stable and the elastic constants satisfy the Born
criteria stability, $0<C_{11}$, $0<C_{22}$, $C_{12}<C_{11}$ and
$C_{12}<C_{22}$. In addition, the stability of these compounds is studied by
phonon dispersion, see Fig. 2. The positive frequencies confirm the stability
of these structures. The two-dimensional contour plots of the first Brillouin
zone for the valence and conduction bands are depicted in Fig. 3. The
conduction band minimum (CBM) is located at K-valley for all compounds. In the
case of MoSi2N4, K-valley approximately demonstrates isotropic contour in
different directions whereas, anisotropic behavior is obvious for the other
compounds. The anisotropy of K-valley increases for heavier compounds and
MoSi2Sb4 demonstrates the highest anisotropy. At the same time, the valence
band maximum (VBM) is located at $\Gamma$-point in the case of MoSi2N4,
whereas the energy of K-point is close to the $\Gamma$-point. On the other
hand, the VBM of other structures is placed at K-point. MoSi2N4 demonstrates
isotropic contours around $\Gamma$-point whereas the other compounds exhibit
anisotropic behavior. However, K-point in the valence band shows a high
anisotropy with triangular contours for all compounds. The band structures of
the four studied materials are drawn in Fig. 4. MoSi2N4 demonstrates an
indirect bandgap from $\Gamma$-valley in the valence to K-valley in the
conduction band whereas, K-point at the valence band is close to
$\Gamma$-point and contributes to the top of the valence band. Three other
compounds have a direct bandgap at K-point. In these compounds, $\Gamma$-point
at valence band also is close to K-point. On the other hand, K-point displays
a considerable spin-splitting whereas, the splitting vanishes for
$\Gamma$-point due to the high symmetry. The minimum of the second conduction
band with a large spin-splitting is near to the minimum of the first band. In
order to further analyze the electronic properties of the structures, the
projected density of states (PDOS) is also depicted in Fig. 4. The states near
the VBM and CBM are mainly contributed by the Mo atom. X atoms also contribute
to the conduction and valence bands, but their contributions are lower than
the Mo atom.

Figure 4: The band structure along with corresponding PDOS of MoSi2X4
structures. The d-orbital of Mo atom and p-orbitals of Si and X atoms have the
main contributions to the DOS. Figure 5: The band gap variation as a function
of vertical strain. Figure 6: The energies of the K-valleys of the first and
second conduction bands along with K- and $\Gamma$-points of the valence band
versus vertical strain. The up- and down-spin for each band are also
indicated. The range of strain is kept in the semiconducting phase and is
different for various compounds.  Figure 7: Charge distribution for MoSi2P4 as
a sample. The charge is plotted for - and K-point in the valence band and for
the first and second conduction band at K-valley.

The electrical and spin properties of four materials are summarized in Table
2. First, the band gaps with and without spin-orbit coupling consideration are
compared. In the case of MoSi2N4, the band gaps without and with SOC are
approximately the same. These values which are in good agreement with
experimental work Hong et al. (2020), are 1.73 and 1.721 eV, respectively. The
bandgap of MoSi2P4 without(with) SOC is 0.772(0.648) eV that is near to 0.61
eV reported for with SOCAi et al. (2021). The bandgap of MoSi2As4 decreases
from 0.664 to 0.525eV with applying spin-orbit coupling, while the values of
0.6eV and 0.41-0.5 eV are reported for without and with spin-orbit coupling
consideration, respectivelyHong et al. (2020); Guo et al. (2020); Ai et al.
(2021); Li et al. (2020). The SOC has more significant effects on the heavier
compounds so that the bandgap reduces by 0.172 eV for MoSi2Sb4 with applying
SOC. In addition, the conduction and valence band maximum is presented in the
table. In the end, the spin-splitting at K-point in the valence band, the
first and second conduction bands are listed in the table. The value of spin-
splitting of MoSi2N4 at K-point ($\lambda_{K,V}$) of the valence band is 131
meV which is a little smaller than 140 meV from experimental work. However,
130 meV is reported for $\lambda_{K,V}$ in the theoretical papers Li et al.
(2020). Furthermore, the calculated value of $\lambda_{K,V}$ for MoSi2As4 is
0.179 eV and in good agreement with previous reported valuesGuo et al. (2020);
Li et al. (2020). $\lambda_{K,V}$ increases with changing X-atom from N to Sb
and reaches 220 meV for MoSi2Sb4. The spin-splitting at the first conduction
band ($\lambda_{K,C1}$) is not considerable. On the contrary, the second band
demonstrates a significant spin-splitting and for example, is larger than
$\lambda_{K,V}$ in MoSi2N4, but the energy of the second band is much higher
than the first band in this compound. The spin-splitting at K-valley in the
second conduction band ($\lambda_{K,C2}$) decreases from 187 meV in MoSi2N4 to
83 meV in MoSi2P4 and increases by changing X atom from P to Sb. By looking at
the conduction band of MoSi2As4 and MoSi2Sb4 at K-point, one can find that the
first and third bands do not demonstrate remarkable spin-splitting, while the
second and fourth display notable spin-splitting.

Figure 8: Two-dimensional map of the spin-splitting in the first and second conduction bands along with valence band. The first Brillouin zone is highlighted by the dashed line in the first compound. The large spin-splitting at valence and second conduction band is obvious. Table 2: The band gap calculated with PBE functional (E${}^{PBE}_{g}$) and PBE with spin-orbit coupling included (E${}^{SOC}_{g}$), the energies of conduction (EC-EF) and valence band (EV-EF) edges, the compressive strain ($\varepsilon_{trans}$) and pressure (Ptrans) required for semiconductor to metal transition, the spin-splitting at K point of the valence band ($\lambda_{K_{V}}$) and first and second bands of conduction ($\lambda_{K_{C1}}$ and $\lambda_{K_{C2}}$).  |  E${}^{PBE}_{g}$ |  E${}^{SOC}_{g}$ |  VBM |  CBM |  EC-EF |  EV-EF |  $\varepsilon_{trans}$ |  $P_{trans}$ |  $\lambda_{K_{V}}$ |  $\lambda_{K_{C1}}$ |  $\lambda_{K_{C2}}$
---|---|---|---|---|---|---|---|---|---|---|---
|  (eV) |  (eV) |  |  |  (eV) |  (eV) |  ($\%$) |  (GPa) |  (eV) |  (eV) |  (eV)
MoSi2N4 |  1.73 |  1.721 |  $\Gamma$ |  K |  0.582 |  -1.139 |  -22 |  25.3 |  0.131 |  0.003 |  0.187
MoSi2P4 |  0.772 |  0.648 |  K |  K |  0.333 |  -0.315 |  -11 |  9.1 |  0.143 |  0.004 |  0.083
MoSi2As4 |  0.664 |  0.525 |  K |  K |  0.275 |  -0.25 |  -11 |  8.9 |  0.179 |  0.014 |  0.113
MoSi2Sb4 |  0.481 |  0.309 |  K |  K |  0.179 |  -0.131 |  -10 |  7.3 |  0.22 |  0.018 |  0.151
Table 3: The effective masses at K- and $\Gamma$-points of the valence band and K-valley in the conduction band. All reported effective masses are in m0 unit.  |  $m^{v,*}_{K\rightarrow M}$ |  $m^{v,*}_{K\rightarrow\Gamma}$ |  $m^{v,*}_{\Gamma\rightarrow K}$ |  $m^{v,*}_{\Gamma\rightarrow M}$ |  $m^{c,*}_{K\rightarrow M}$ |  $m^{c,*}_{K\rightarrow\Gamma}$
---|---|---|---|---|---|---
MoSi2N4 |  0.794 |  0.595 |  1.375 |  1.37 |  0.544 |  0.485
MoSi2P4 |  0.562 |  0.388 |  1.483 |  1.335 |  0.697 |  0.567
MoSi2As4 |  0.701 |  0.438 |  1.586 |  1.407 |  1.206 |  0.625
MoSi2Sb4 |  0.925 |  0.447 |  2.288 |  1.992 |  1.843 |  0.647

The effective masses of the conduction and valence bands are reported in Table
3. The effective masses of both $\Gamma$\- and K-points of the valence band
are presented in the table. K-point demonstrates a lower effective mass with
respect to $\Gamma$-one. The anisotropic contour around K-valley leads to two
different values at K to M and K to $\Gamma$ paths. The K to M effective mass
($m^{V*}_{K\rightarrow M}$) displays a higher value than K to effective mass
($m^{V*}_{K\rightarrow\Gamma}$). On the other hand, -point owns isotropic
contour, and effective masses at different paths are approximately equal. In
Ref. Hong et al., 2020, the effective mass of the electron is reported to be
0.486 m0, which corresponds to our calculated value 0.485 m0 for K to path.
Furthermore, the reported value of the hole effective mass is 0.683 m0 that is
in the middle of our calculated effective masses at K-valley. However, -point
is the VBM and demonstrates a larger effective masses. Except MoSi2N4, the
effective masses for the valence band increase for heavier compounds and
MoSi2P4 contains the lowest effective mass. On the other hand, only K-valley
contributes to the conduction band. The effective masses at two different
paths for this valley are reported in the table. The effective masses in these
two paths are in the same range for MoSi2N4 and MoSi2P4, whereas their
differences are enhanced for MoSi2As4 and MoSi2Sb4. As mentioned before, the
contour around K-valley changes from circular for MoSi2N4 to triangular in
MoSi2Sb4. In this regard, MoSi2Sb4 demonstrates high anisotropic effective
masses at two different paths. The effective masses of the conduction band
increase for the heavier compounds. MoSi2N4 and MoSi2P4 possess the lowest
effective mass in the conduction and valence bands, respectively. The vertical
strain is known as a powerful tool to modify electrical and spin properties of
two-dimensional compounds Ghobadi and Touski (2021); Shamekhi and Ghobadi
(2020). This strain can be applied by out-of-plane pressure and 2D materials
can be used as a pressure sensor. The vertical strain is applied to the four
structures and their electrical and spin properties are studied. First, the
variation of the band gaps as a function of vertical strain is depicted in
Fig. 5. The band gaps exhibit a maximum value at a specific small tensile
strain and then decrease for larger compressive and tensile strains. The band
gaps vanish at transition strains ($\varepsilon_{trans}$). The value of
compressive $\varepsilon_{trans}$ and its corresponding pressure ($P_{trans}$)
are reported in Table 2. The band gaps of MoSi2P4, MoSi2As4 and MoSi2Sb4
vanish around the strain of -10$\%$ or -11$\%$. On the other hand, MoSi2N4
demonstrates a higher bandgap, and $\varepsilon_{trans}$ is two times larger
at -22$\%$ strain. $P_{trans}$ for MoSi2P4, MoSi2As4 and MoSi2Sb4 is in the
range of [7.3-9.1] GPa, whereas MoSi2N4 demonstrates much larger $P_{trans}$
of 25.3 GPa. The transition pressures for bilayers are reported from 2.18 GPa
in CrSi2N4 to 32.04 GPa in TiSi2N4 Zhong et al. (2021) that are comparable
with MoSi2X4 monolayer. This transition pressure range also proves that
applying out-of-strain to monolayer MoSi2X4 is feasible. The band gaps also
vanish at tensile strain. MoSi2Sb4 demonstrates higher sensitivity to tensile
strain and its band gap vanishes at lower strains. The transition strain at
tensile regime increases as X atom changes from Sb to N. One can conclude from
the figure that tensile transition strain of MoSi2N4 is much larger than
15$\%$.

Figure 9: Projected band structures for the studied materials. The d-orbital
of Mo along with the p-orbital of Si and X atoms have the main contributions
around the band gaps.

The energies of the K-valleys for the first and second conduction bands, and
K- and $\Gamma$-points of the valence band versus vertical strain is plotted
in Fig. 6. The energies of the up- and down-spins are separately shown in the
figure. The strain ranges of the figures are different and the strain ranges
are selected in the semiconducting phase. At large compressive strains, the
energy of $\Gamma$-point of the valence band is higher than K-point and
K-valley determines the VBM. By increasing strain from compressive to the
tensile regime, the energy of $\Gamma$-point decreases, and the energy of
K-point increases. One can find the large spin-splitting at K-point, whereas
$\Gamma$-point does not demonstrate any splitting. On the other hand, two
bands contribute to the conduction band. We have called the bands with low and
large spin-splitting as $\alpha$-band and $\beta$-band, respectively. The
$\alpha$-band contributes as CBM at compressive and small tensile strains. The
larger tensile strains bring down the $\beta$-band and change the location of
these two bands. Therefore, CBM is contributed by $\beta$-band at a large
tensile strain regime. The charge distribution around K- and $\Gamma$-points
of the valence band and K-valleys of the first and second conduction bands in
the case of MoSi2P4 is shown in Fig. 7. A single layer of MoSi2P4 can be
divided into three sub-layers. The charge is significantly localized around
the Mo atom in the K-point of valence and conduction bands. $\Gamma$-point of
the valence band has bonding character between P and Si atoms of two sub-
layers. In addition, the charge is distributed around the most external P
atoms. The out-of-plane strain does not significantly affect charge
distribution at this point. In the K-point of the valence band, the charge is
highly distributed in the internal MoN2 sub-layer and a small charge is around
outer P atoms. At compressive out-of-plane strain, the charge around outer P
atoms is expanded. Furthermore, the first and second conduction bands at
K-valley are compared in the figure. At equilibrium, the charge is localized
in the internal sub-layer for the first band, whereas the charge is
distributed in the whole thickness of the compound for the second band. The
compressive strain does not affect the charge distribution, whereas tensile
strain changes the location and charge density of these two bands.

Figure 10: The spin-splitting of the conduction and valence bands at the
K-point. The valence band along with the first and second conduction bands are
studied.

Two-dimensional maps of the spin-splitting in the first Brillouin zone for all
compounds have been shown in Fig. 8. Both first and second conduction bands
along with the valence band are indicated in the figure. The positive and
negative spin-splitting around K and K’-valleys are obvious in the figures,
respectively. The first conduction bands of all compounds except MoSi2N4
display a low spin-splitting. In the case of MoSi2N4, a large spin-splitting
is around M- and -valley, but these two valleys do not contribute to the CBM.
One can observe that the spin-splitting of the first conduction band is
approximately zero on the boundary of the first Brillouin zone for MoSi2P4 and
MoSi2As4. On the contrary, the second conduction band exhibits a high spin-
splitting which is located at K- and K’- valleys. MoSi2N4 and MoSi2Sb4
demonstrate a higher spin-splitting compared to the others. ̵ُThe valence band
shows a tidy spin-splitting so that the splitting is maximum around K- and
K’-valleys and it vanishes elsewhere. The spin-splitting increases for heavier
compounds and MoSi2Sb4 show the highest value. We have seen from Table 2 that
the valence band of MoSi2Sb4 has the highest spin-splitting as high as 220
meV. The spin-splitting of the valence band is isotropic around the K-valley
of MoSi2P4 and other materials specially MoSi2Sb4 demonstrate anisotropic
spin-splitting. In this regard, the contour around the K-valley of MoSi2P4 is
circular, whereas MoSi2Sb4 has a triangular shape. To clarify the origin of
spin-splitting in the valence and conduction bands, the projected band
structure of four structures is plotted in Fig. 9. As it is obvious, the first
and second conduction bands at K-valley are mainly affected by the d-orbitals
of molybdenum atoms. The p-orbitals of X and Si atoms have a low contribution
on the first and second bands, respectively. The large spin-splitting of the
second band comes from the d-orbitals, whereas the first conduction band with
a high d-orbital contribution shows a small splitting. Maybe, this is related
to the odd and even bands because the third one also displays a negligible
spin-splitting in MoSi2As4 and MoSi2Sb4. In addition, the valence band at K-
and -points is highly formed by the d-orbitals of Mo atoms. The p-orbitals of
X atoms also have a considerable influence at the K-point, whereas p-orbitals
of Si atoms contribute to the -point. Therefore, the d-orbitals of Mo atoms,
the p-orbitals of X and Si atoms have the most impact on the -point of the
valence band, respectively. The large spin-splitting at valence band is
originated from the d-orbitals, whereas the p-orbitals of X atoms enhance the
spin-splitting for heavier compounds. The spin-splitting of the K-point of the
valence band and the first and second bands of the conduction band versus
vertical strain is shown in Fig. 10. $\lambda_{K,V}$ increases slightly when
strain increases from compressive to tensile regime. The increase of the spin-
splitting of the valence band with respect to strain has also observed in the
group-III monochalcogenides Ariapour and Touski (2020). One can find from the
band structures and Table 2, that the first conduction band has a negligible
spin-splitting, while the second band demonstrates a considerable spin-
splitting. $\lambda_{K,C1}$ increases when strain increases from compressive
to tensile, whereas $\lambda_{K,C2}$ decreases. The slope of $\lambda_{K,C2}$
variation is higher in lighter compounds. As one can observe the curves of
$\lambda_{K,C1}$ and $\lambda_{K,C2}$ change with each other at the tensile
strain. As we have seen in Fig. 6, the $\beta$-band moves downward and becomes
lower than $\alpha$-band for tensile strain. So, in the tensile regime,
$\beta$-band with large spin-splitting is the first conduction band. This band
changing is also observed in the figure.

##  IV conclusion

The electrical and spin properties of monolayer MoSi2X4 (X = N, P, As, and Sb)
under vertical strain are investigated. MoSi2N4 demonstrates an indirect
bandgap from $\Gamma$ at valence to K at conduction band, whereas K-valley at
valence band is close to $\Gamma$-point and contributes to the valence band.
On the other hand, three other compounds have direct bandgap at K-point and
$\Gamma$-point at valence band is also close to K-point. Although the first
conduction band demonstrates a small spin-splitting, the second conduction
band has a large spin-splitting and its minimum is near to the first band. The
PDOS demonstrates that the conduction and valence bands are mostly affected by
the d-orbitals of Mo atoms. X atoms have a lower contribution to the valence
band. The projected band structure also demonstrates that the conduction and
valence bands are mainly constructed from the d-orbitals of Mo atoms. The
p-orbitals of X and Si atoms also contribute to the valence and second
conduction bands, respectively. The spin-splitting at K-point
($\lambda_{K,V}$) of valence band increases with changing X-atom from N to Sb
and reaches 220 meV for MoSi2Sb4. The spin-splitting at K-valley in the second
conduction band ($\lambda_{K,C2}$) increases from 83 meV in MoSi2P4 to 151 meV
for MoSi2Sb4. In the following, the effect of vertical strain has been
investigated. The band gaps exhibit a maximum value at small tensile strains,
then they decrease for compressive and larger tensile strains. The band gaps
close at compressive transition strains ($\varepsilon_{trans}$) of around
-10$\%$ for all compounds except MoSi2N4 that its bandgap vanishes at the
larger strain of about -22$\%$. Transition pressures are also distributed from
9.1 to 7.3 GPa for MoSi2P4, MoSi2As4 and MoSi2Sb4, whereas MoSi2N4 displays a
larger $P_{trans}$ of 25.3 GPa. In addition, the charge is mostly localized
around Mo atoms in both valence and conduction bands. The spin-splitting
around K-valley at the valence band is isotropic and circular in the case of
MoSi2P4, whereas the anisotropy increases for heavier compounds and MoSi2Sb4
exhibits a triangular shape contour. The spin-splitting of K-valley of the
valence band increases when strain increases from compressive to tensile
regime. $\lambda_{K,C1}$ increases when strain increases from compressive to
tensile, and at the same time, $\lambda_{K,C2}$ decreases. The slope of
$\lambda_{K,C2}$ variation is higher for the lighter compounds.
$\lambda_{K,C1}$ and $\lambda_{K,C2}$ change with each other at the tensile
strain. In addition to the electrical properties, the vertical strain can also
control the spin properties.

## References

  * Neto et al. (2009) A. C. Neto, F. Guinea, N. M. Peres, K. S. Novoselov, and A. K. Geim, Reviews of modern physics 81, 109 (2009).
  * Radisavljevic et al. (2011) B. Radisavljevic, A. Radenovic, J. Brivio, V. Giacometti, and A. Kis, Nature nanotechnology 6, 147 (2011).
  * Guan et al. (2014) J. Guan, Z. Zhu, and D. Tománek, Physical review letters 113, 046804 (2014).
  * Zhang et al. (2015) S. Zhang, Z. Yan, Y. Li, Z. Chen, and H. Zeng, Angewandte Chemie 127, 3155 (2015).
  * Mudd et al. (2013) G. W. Mudd, S. A. Svatek, T. Ren, A. Patanè, O. Makarovsky, L. Eaves, P. H. Beton, Z. D. Kovalyuk, G. V. Lashkarev, Z. R. Kudrynskyi, et al., Advanced Materials 25, 5714 (2013).
  * Li et al. (2017) Q. Li, L. He, C. Sun, and X. Zhang, The Journal of Physical Chemistry C 121, 27563 (2017).
  * Wang et al. (2016) Y. Wang, S.-S. Wang, Y. Lu, J. Jiang, and S. A. Yang, Nano letters 16, 4576 (2016).
  * Ozdemir (2018) B. Ozdemir, Computational Condensed Matter 17, e00335 (2018).
  * Hong et al. (2020) Y.-L. Hong, Z. Liu, L. Wang, T. Zhou, W. Ma, C. Xu, S. Feng, L. Chen, M.-L. Chen, D.-M. Sun, et al., Science 369, 670 (2020).
  * Wang et al. (2021) L. Wang, Y. Shi, M. Liu, A. Zhang, Y. Hong, R. Li, Q. Gao, M. Chen, W. Ren, H. Cheng, et al., Nature Communications 12 (2021).
  * Zhong et al. (2021) H. Zhong, W. Xiong, P. Lv, J. Yu, and S. Yuan, Physical Review B 103, 085124 (2021).
  * Li et al. (2021) B. Li, J. Geng, H. Ai, H. Bai, K. H. Lo, K. W. Ng, Y. Kawazoe, H. Pan, et al., Nanoscale (2021).
  * Mortazavi et al. (2021) B. Mortazavi, B. Javvaji, F. Shojaei, T. Rabczuk, A. V. Shapeev, and X. Zhuang, Nano Energy 82, 105716 (2021).
  * Yu et al. (2021) J. Yu, J. Zhou, X. Wan, and Q. Li, New Journal of Physics 23, 033005 (2021).
  * Bafekry et al. (2021a) A. Bafekry, M. Faraji, D. Hoat, M. Shahrokhi, M. Fadlallah, F. Shojaei, S. A. H. Feghhi, M. Ghergherehchi, and D. Gogova, Journal of Physics D: Applied Physics 54, 155303 (2021a).
  * Yao et al. (2021) H. Yao, C. Zhang, Q. Wang, J. Li, Y. Yu, F. Xu, B. Wang, and Y. Wei, Nanomaterials 11, 559 (2021).
  * Yang et al. (2021) C. Yang, Z. Song, X. Sun, and J. Lu, Physical Review B 103, 035308 (2021).
  * Ai et al. (2021) H. Ai, D. Liu, J. Geng, S. Wang, K. H. Lo, and H. Pan, Physical Chemistry Chemical Physics 23, 3144 (2021).
  * Nayak et al. (2014) A. P. Nayak, S. Bhattacharyya, J. Zhu, J. Liu, X. Wu, T. Pandey, C. Jin, A. K. Singh, D. Akinwande, and J.-F. Lin, Nature communications 5, 1 (2014).
  * Chi et al. (2014) Z.-H. Chi, X.-M. Zhao, H. Zhang, A. F. Goncharov, S. S. Lobanov, T. Kagayama, M. Sakata, and X.-J. Chen, Physical review letters 113, 036802 (2014).
  * Ghobadi (2019) N. Ghobadi, Physica E 111, 158 (2019).
  * Bafekry et al. (2021b) A. Bafekry, C. Stampfl, M. Naseri, M. M. Fadlallah, M. Faraji, M. Ghergherehchi, D. Gogova, and S. Feghhi, Journal of Applied Physics 129, 155103 (2021b).
  * Wu et al. (2021) Q. Wu, L. Cao, Y. S. Ang, and L. K. Ang, Applied Physics Letters 118, 113102 (2021).
  * Guo et al. (2020) S.-D. Guo, W.-Q. Mu, Y.-T. Zhu, and X.-Q. Chen, Physical Chemistry Chemical Physics 22, 28359 (2020).
  * Soler et al. (2002) J. M. Soler, E. Artacho, J. D. Gale, A. García, J. Junquera, P. Ordejón, and D. Sánchez-Portal, J. Phys.:Condensed Matter 14, 2745 (2002).
  * Perdew and Zunger (1981) J. P. Perdew and A. Zunger, Phys. Rev. B 23, 5048 (1981).
  * Kokalj (2003) A. Kokalj, 28, 155 (2003).
  * Touski and Ghobadi (2020) S. B. Touski and N. Ghobadi, Physica E p. 114407 (2020).
  * Ghobadi and Touski (2020) N. Ghobadi and S. B. Touski, Journal of Physics: Condensed Matter 33, 085502 (2020).
  * Li et al. (2020) S. Li, W. Wu, X. Feng, S. Guan, W. Feng, Y. Yao, and S. A. Yang, Physical Review B 102, 235435 (2020).
  * Ghobadi and Touski (2021) N. Ghobadi and S. B. Touski, Journal of Physics: Condensed Matter (2021).
  * Shamekhi and Ghobadi (2020) M. Shamekhi and N. Ghobadi, Physica B 580, 411923 (2020).
  * Ariapour and Touski (2020) M. Ariapour and S. B. Touski, Journal of Magnetism and Magnetic Materials 510, 166922 (2020).


#  Simple Control Baselines for Evaluating Transfer Learning

Andrei Atanov†∗, Shijian Xu† , Onur Beker†, Andrei Filatov†§, Amir Zamir†
† Swiss Federal Institute of Technology (EPFL) § Moscow Institute of Physics
and Technology (MIPT)
https://transfer-controls.epfl.ch equal contribution

###### Abstract

Transfer learning has witnessed remarkable progress in recent years, for
example, with the introduction of augmentation-based contrastive self-
supervised learning methods. While a number of large-scale empirical studies
on the transfer performance of such models have been conducted, there is not
yet an agreed-upon set of control baselines, evaluation practices, and metrics
to report, which often hinders a nuanced and calibrated understanding of the
real efficacy of the methods. We share an evaluation standard that aims to
quantify and communicate transfer learning performance in an informative and
accessible setup. This is done by baking a number of simple yet critical
control baselines in the evaluation method, particularly the ‘blind-guess’
(quantifying the dataset bias), ‘scratch-model’ (quantifying the architectural
contribution), and ‘maximal-supervision’ (quantifying the upper-bound). To
demonstrate how the evaluation standard can be employed, we provide an example
empirical study investigating a few basic questions about self-supervised
learning. For example, using this standard, the study shows the effectiveness
of existing self-supervised pre-training methods is skewed towards image
classification tasks versus dense pixel-wise predictions. In general, we
encourage using/reporting the suggested control baselines in evaluating
transfer learning in order to gain a more meaningful and informative
understanding.

##  1 Introduction

Creating computer vision models that can support a wide range of downstream
tasks will require the development of representations whose utility extend
beyond the exact same objective they were optimized for. Transfer learning is
a general approach to operationalize this perspective by using representations
of a model trained on a source task to improve the sample complexity of
learning another downstream task. Supervised pre-training approaches to
transfer learning, however, still have the drawback of requiring large
annotated datasets for the source task. Self-supervised learning, in turn,
holds the potential to remove this bottleneck by learning representations
directly from unlabelled data by means of employing a proxy pre-training
objective whose optimization results in transferable representations, but does
not require manual labelling [32, 46, 14]. For example, the recently revisited
contrastive learning approaches [18, 5, 19, 3] constitute a milestone in this
paradigm and were shown to achieve impressive transfer performance competitive
with representations obtained in a fully supervised way for classification
tasks.

Figure 1:  What are the most informative control baselines and transfer
dataset sizes that provide sufficient context for evaluating the effectiveness
of a pre-training method? We employ a calibrated evaluation standard that
incorporates: 1) a scratch control baseline to account for the architectural
inductive biases, 2) a maximal-supervision baseline to approximate the best
achievable performance, and 3) a blind-guess baseline to account for the
dataset bias. We consider a range of data sizes across which the performance
of the scratch baseline varies between the blind-guess level (where there is
large room for improvement; plot’s left) and maximal-supervision level (where
nearly no pre-training is needed; plot’s right). The figure illustrates the
performance of an example model, where the x and y axes show the data size and
calibrated performance, respectively, along with qualitative instances along
the curves.

Given the rapid progress in transfer and self-supervised learning, obtaining a
clear and complete picture of the increasingly large number of methods being
proposed is difficult, in part due to an absence of standard evaluation
practices and the non-uniformity of reported (control) baselines. Ideally,
such an evaluation standard should have a number of desirable properties, such
as:

  1. 1.

providing a simple metric that clearly communicates the “effectiveness”
transfer learning of a given method on a natural and interpretable scale.

  2. 2.

taking into account the effects of irreducible factors that upper-bound the
best possible transfer performance, such as the uncertainty in observations or
the imperfect nature of current neural network training practices.

  3. 3.

normalizing away the benefits due to the statistical regularities of datasets
that lower-bound the worst possible performance [10], to prevent them from
being a confounder in comparisons.

  4. 4.

providing an intuitive way to draw comparisons about the utility of different
self-supervised learning methods across different downstream tasks.

This paper proposes an evaluation standard that satisfies the aforementioned
properties by incorporating a set of key control baselines in the evaluation.
We then proceed with example analyses on questions that would be otherwise
harder to study without such a standard – for example, assessing whether
existing self-supervised learning methods are equally effective across
different types of tasks. The proposed standard is not limited to self-
supervised learning and is applicable to any transfer learning evaluation.

##  2 Related Work

Self-Supervised Representation Learning is a special case of general transfer
learning and one of the most attended ones today. Earlier work on self-
supervised learning largely focused on hand-designed proxy-tasks, such as
solving a jigsaw puzzle [32], image colorization [46], rotation prediction
[14], as well as numerous others [30, 34, 44]. More recently, a number of
augmentation-based contrastive methods have been proposed [5, 19, 3, 22, 6],
and were shown to outperform previous approaches significantly. These methods
employ a contrastive-loss formulation [18], and learn representations that are
similar for augmented views of the same image and dissimilar for different
images. [2, 45, 17] also suggest learning representations that are close for
augmentations of the same image, but do not use negative examples explicitly
and achieve similar performance to contrastive methods.

Benchmarking and Analysis. The literature contains various empirical studies
on the effectiveness of self-supervised pre-training, across a wide range of
methods and downstream tasks. [40, 24, 9] focus on classification tasks and
study how the final transfer performance depends on the image domain of a
dataset, downstream-task complexity, and the available amount of labelled data
for transfer. [13, 31] further include non-classification tasks in their
evaluation setup, and [27] conducts a comprehensive empirical study with a
diverse set of downstream tasks to demonstrate the benefits of contrastive
self-supervised pre-training. [16, 26, 38, 41, 9] examine the effects of
various pre-training parameters such as the model capacity, pretext task
complexity, and the augmentation policy used in the contrastive loss
formulation, to provide a further understanding of the successes of self-
supervised pre-training. [38, 39, 1, 29] build a mathematical framework to
explain how self-supervised pre-training can improve the performance on
downstream tasks, aiming to provide theoretical guarantees. In this work, we
also include non-classification tasks in our showcase analysis, and aim to
study the transfer performance more comprehensively with the proposed control
baselines and evaluations across different data-regimes.

Evaluation Metrics and Baselines. A standard way to evaluate self-supervised
learning methods is through freezing their pre-trained representations and
reporting the performance of a linear classifier trained on top to solve
ImageNet classification. [13] show empirically that this practice is not as
predictive of the downstream performance for non-classification tasks as it is
for classification tasks. [27] adopt a more comprehensive setup that evaluates
transfer performance on a broader range of downstream tasks, and opt for
reporting most of their results in terms of relative improvements over
ImageNet supervised pre-training, but the absolute level of performance of the
ImageNet pre-training baseline remains unclear for their study. [31] and [9]
include comparisons to a model trained from scratch on the same amount of
labelled data, and the former further suggests measuring the transfer efficacy
as the proportion of labelled data saved by self-supervised pre-training
compared to training a model from scratch to achieve the same performance. Our
work also incorporates the scratch performance as a baseline to account for
the difficulty of the downstream task. We additionally put our comparisons
into perspective by attempting to account for the dataset bias and the
performance upper-bound due to irreducible uncertainties or architectural
limitations.

##  3 The Evaluation Standard

This section describes the control baselines and the overall evaluation
standard we propose. We start with a short description of the transfer-
learning setting, and then proceed with presenting the definitions and
motivations behind each of the control baselines. We then incorporate these
baselines into an affine rescaling of the empirical risk, and discuss the
associated visualizations and additional metrics.

###  3.1 Transfer Learning Setting

We consider a standard transfer learning setting that employs an encoder-
decoder architecture. The encoder $\psi:{\mathcal{X}}\to{\mathcal{Z}}$, where
${\mathcal{X}}$ and ${\mathcal{Z}}$ denote the set of input images and their
projected latent representations respectively, is first optimized using a
given pre-training method  (e.g., a self-supervised method, or a source-task
solved in a fully supervised way) on a pre-training dataset
$\mathcal{D_{\mathrm{pre}}}$. The decoder $\phi:{\mathcal{Z}}\to{\mathcal{Y}}$
for the downstream task, with ${\mathcal{Y}}$ denoting the output space, is
then trained in a supervised way using a transfer dataset
$\mathcal{D}_{trans}$ of annotated images to minimize the task-specific loss
${\mathcal{L}}$, and the encoder parameters are also fine-tuned. The number of
images $|\mathcal{D}_{trans}|$ used for this optimization is called the data-
regime for transfer learning. The resulting model is denoted as
$f_{\theta}=\phi\circ\psi$, where $\theta$ includes all the network
parameters. After training, the empirical risk is computed over the test
partition $\mathcal{D}_{\mathrm{test}}$ of the transfer dataset using the loss
${\mathcal{L}}$ for the downstream-task:

$R_{f_{\theta}}=\dfrac{1}{|\mathcal{D}_{\mathrm{test}}|}\sum_{(x,y)\in\mathcal{D}_{\mathrm{test}}}{\mathcal{L}}(y,f_{\theta}(x))).$
(1)

###  3.2 Control Baselines

Motivation. Transfer learning results are commonly reported using the
empirical risk. Therefore, interpreting such results requires knowledge about
the scale of the training loss and its nonlinear relation to the inherent
prediction fidelity/value for the particular downstream task and dataset. To
illustrate with a toy example, when a classification model is reported to
achieve the accuracy of $0.9$, judging whether this model performs well or not
cannot be done solely by means of the reported metric: the model can be
considered proficient if there are 1000 uniformly distributed classes in the
dataset, but not so if $90\%$ of the images belong to the same class – since
the accuracy of merely statistically informed guess is 0.001 (900$\times$
worse than the model’s) for the former and 0.9 (the same as the model’s) for
the latter. Furthermore, the sample complexity of obtaining an _additional
improvement_ will differ depending on the granularity and similarity of the
categories.

Such observations motivate the following questions: What additional context
can we provide to create a more complete picture of the transfer performance?
What are the most sensible, efficient, and simple control baselines that would
allow us to attain a more accurate assessment?

Figure 2:  Right: Exemplary $\bm{\mathrm{cR}_{f}\mathchar 45\relax n}$ and
$\bm{\mathrm{cR}_{f}\mathchar 45\relax\mathrm{cR}_{\mathrm{scratch}}}$ curves
for the SwAV self-supervised pre-training method, transferred to surface
normals estimation. The relative improvement against the scratch control
baseline (i.e. $\Delta/\mathrm{cR}_{\mathrm{scratch}}$) can directly be read
from the plot and provides a visualization for the transfer efficacy of a
given method. Left: corresponding qualitative surface normals predictions for
SwAV and three control baselines: blind-guess, maximal-supervision and
scratch. In this example, the improvement of the SSL pre-training over scratch
is observed to diminishes with more training data, which can also be inferred
from the qualitative examples for 10K/100K. This (rather general) trend
indicates the benefit of transfer learning in high-data regime is less
significant, and therefore, relative improvement comparisons are more
meaningful across different data-regimes (particularly mid and low data) where
the scratch baseline performs relatively poorly.

The Scratch Control Baseline: When training a decoder $\phi$ on a transfer
dataset, choices about training parameters (e.g., the model architecture or
the optimization method used) are an important source of inductive bias.
Similarly, those specifics of a downstream task and dataset that characterize
their amenability for model training have an influence on how the optimization
will proceed. Such considerations are independent of the particular choice of
pre-training method being evaluated, yet they have a critical impact on the
final transfer performance. Therefore, as previous studies also illustrate
[9], it is important to include the scratch control baseline (i.e., a randomly
initialized model trained only using a transfer dataset corresponding to the
data-regime being examined), to disentangle and clarify the benefits of the
pre-training method.

The Maximal-Supervision Control Baseline: Even in the presence of an infinite
amount of data, many factors can still limit the minimum error achievable on a
downstream task. Examples include uncertainties and multi-modality in the
label generation process, finite model capacities and architectural
bottlenecks, or the non-convexity of the optimization problem and the
imperfect nature of the algorithms employed. Having an estimate of such
factors is useful for framing the transfer performance in an interpretable
scale. We, therefore, employ a maximal-supervision control baseline – a
randomly initialized model trained using a large amount of data for the
downstream task intended to provide _an approximation_ to the maximum
achievable performance on the downstream task. This baseline may not always be
attainable, e.g., when sufficiently large training data for the dataset domain
or a theoretical approximation are unavailable. In our experiments, we will
show cases where a good approximation is attainable (e.g., for Tasknomomy
[43]) as well as cases where it is not attainable (e.g., for CIFAR-100 [28]),
and discuss the implications.

The Blind-Guess Control Baseline: Regularities in datasets can allow
statistically informed guesses for the downstream task [10]. Such guesses can
often be unexpectedly performant [43, 12]. The effects of such regularities
can be accounted for by constructing an input-agnostic blind-guess control
baseline, defined as the single constant prediction with the lowest error
$\arg\min_{\hat{y}\in{\mathcal{Y}}}\mathbb{E}_{y}\left[{\mathcal{L}}(\hat{y},y)\right]$,
which corresponds to the mean output for $L_{2}$ loss, the median output for
$L_{1}$ loss, and the most-represented class for the 0-1 loss. This control
baseline serves as a sanity check to clarify whether a pre-training method
provides a tangible benefit, as compared to solely capturing regularities in
dataset statistics in an input-independent prediction.

Importance of Control Baselines: The main goal of incorporating control
baselines is to project the performance of a method onto a comparative scale
to capture how well it works, as well as to provide insight into why it works
through rejecting a set of null-hypotheses. Computer vision literature
contains numerous recent examples of how the inclusion of appropriate
baselines can provide such a clarification for their respective fields and
problems. For example, Tatarchenko et al.[37] employs a set of recognition
baselines for the single-view 3D reconstruction problem, and reveals that the
performance of state of the art models are statistically indistinguishable
from classification and retrieval based methods. Another example is the study
by Zamir et al.[43], which employs a gain metric (i.e., the win rate against a
network trained from scratch) to show that given a target task (e.g.,
classification), multiple source tasks (e.g., colorization) can collapse to a
performance level below the scratch baseline in the lower data regime. Sax et
al.[36] employs a blind intelligent actor baseline (i.e., a policy operating
without visual input) for navigation tasks, and reveals that policies trained
from scratch as well as state of the art representation learning methods
perform at a similar level to this baseline when tested on unseen
environments. Here we use such baselines in a more systematic way.

Transfer learning is a field where empirical studies are prevalent, and this
empirical nature makes rendering evaluations with respect to an appropriate
set of baselines particularly important for measuring the progress in a more
accessible and comparable way, while also systematically revealing and
preventing blind spots. The current common practice is to report and compare
downstream performances between different pre-training methods using the
original loss scale or relative to the ImageNet features performance [27, 13,
24]. While this setup still allows conclusions based on quantitative results,
it is hard to assess whether the difference in the methods’ performance is
significant or not. As an example, comparing the $L_{1}$ loss for the
downstream task of depth estimation as reported in Fig.7 of [27] with the
blind baseline reported by [42] in Tab. 1, one can conclude that all models
perform at the blind guess level, and no meaningful improvement was made.
Employing an appropriate set of common control baselines would make it easier
to identify and prevent miscalibrations across different studies and gain a
more complete and comparable picture about the real effectiveness of methods.

###  3.3 The Proposed Evaluation Setup

Calibrated-Risk: To incorporate all three of the previously proposed control
baselines in a single metric, we calibrate the empirical risk $R_{f}$ of a
transfer model $f$ as follows:

$\mathrm{cR}_{f}=\dfrac{R_{f}-R_{\mathrm{max}}}{R_{\mathrm{blind}}-R_{\mathrm{max}}},$
(2)

where $R_{\mathrm{blind}}$ and $R_{\mathrm{max}}$ correspond to the empirical
risks of the blind-guess and maximal-supervision controls. We refer to
$\mathrm{cR}_{f}$ as the calibrated risk. Similarly, the calibrated risk of
the scratch control is computed as:

$\mathrm{cR}_{\mathrm{scratch}}=\dfrac{R_{\mathrm{scratch}}-R_{\mathrm{max}}}{R_{\mathrm{blind}}-R_{\mathrm{max}}},$
(3)

where $R_{scratch}$ denotes the empirical risk of the scratch control
baseline. It is useful to note that $\mathrm{cR}_{f}$ and
$\mathrm{cR}_{scratch}$ are invariant to affine transformations of the
training loss ${\mathcal{L}}$.

The calibrated risk of a given pre-training method varies depending on the
data-regime, and comparisons across different pre-training methods can
therefore show rank reversals. Accordingly, considering the performance across
a range of different data regimes, in the form of a learning curve [23], can
provide a more revealing picture of the rigidity and practical value of
inductive biases introduced by different pre-training methods. As illustrated
in Fig.2, we suggest two plots to illustrate the resulting curves:

  * •

Plotting $\mathrm{cR}_{f}$ and $\mathrm{cR}_{\mathrm{scratch}}$ on the y-axis
against the transfer dataset size $n$ on the x-axis (i.e.,
$\bm{\mathrm{cR}_{f}\mathchar 45\relax n}$ curves, as shown in Fig.2, middle).

  * •

Plotting $\mathrm{cR}_{f}$ on the y-axis against
$\mathrm{cR}_{\mathrm{scratch}}$ on the x-axis, to emphasize the relative
improvement against scratch performance (i.e., $\bm{\mathrm{cR}_{f}\mathchar
45\relax\mathrm{cR}_{\mathrm{scratch}}}$ curves, as shown in Fig.2, right).

Reading $\bm{\mathrm{cR}_{f}\mathchar 45\relax n}$ Curves: Eq. 2 implies that
calibrated risks for the maximal-supervision and blind-guess control baselines
correspond to horizontal lines $\mathrm{cR}_{f}=0$ and $\mathrm{cR}_{f}=1$. If
the maximal-supervision control baseline is trained using sufficiently many
data samples (i.e., meaning that $R_{\mathrm{max}}$ is a good approximation of
the maximum achievable performance, which may not always be true for small-
scale datasets), then $R_{f}\leq R_{\mathrm{max}}$ holds for all data regimes
and per Eq. 2, $\mathrm{cR}_{f}$ approaches the maximal-supervision level
$\mathrm{cR}_{f}=0$ as $n$ goes to infinity. The $\mathrm{cR}_{f}$ curve of a
good pre-training method should lie below the $\mathrm{cR}_{\mathrm{scratch}}$
curve and as close to the line $\mathrm{cR}_{f}=0$ as possible across all data
regimes.

Reading $\bm{\mathrm{cR}_{f}\mathchar 45\relax\mathrm{cR}_{\mathrm{scratch}}}$
Curves: In most situations, the absolute scalar value $\mathrm{cR}_{f}$ by
itself does not provide a clear indication of how well a pre-training method
performs, and the more informative quantity is instead the relative
improvement  with respect to the scratch performance
$\mathrm{cR}_{\mathrm{scratch}}$. To visually facilitate this comparison, we
propose plotting $\mathrm{cR}_{f}$ against $\mathrm{cR}_{\mathrm{scratch}}$.
This maps the $\mathrm{cR}_{\mathrm{scratch}}\mathchar 45\relax n$ curve onto
the main diagonal $x=y$, and a good pre-training method should therefore lie
below it. The relative-improvement of a pre-training method over scratch can
be read from the ratio of distances $\Delta$ and
$\mathrm{cR}_{\mathrm{scratch}}$ in Fig. 2.

The Choice of Transfer Data-Regimes. A qualitative assessment of the images
given in Fig. 2 and Fig. 4 illustrates how the efficiency of self-supervised
pre-training compared to the scratch baseline depends on the amount of data
available for transfer. Both methods perform similarly in high data-regimes
and approach the performance of the maximal-supervision control, which can
also be seen from qualitative examples for 100k train sizes. It is important,
therefore, to choose training sizes appropriately when evaluating the
efficiency of SSL pre-training. We define low- and high-data regimes as ones
where the scratch performance is close to blind-guess and maximal-supervision
controls respectively, and suggest choosing training-dataset sizes to cover
this range.

Calibrated Cumulative Improvement. In order to facilitate tabular analysis and
allow straightforward comparisons between different pre-training methods, we
introduce a global Calibrated Cumulative Improvement (CCI) metric. We
calculate it as the area between an SSL curve and the main diagonal as
measured on the $\bm{\mathrm{cR}_{f}\mathchar
45\relax\mathrm{cR}_{\mathrm{scratch}}}$ plot, divided by the total area under
the main diagonal, shown in Fig. 2. CCI shows the total improvement of a pre-
training method compared to scratch over multiple data-regimes. The CCI values
of our experiments are available in the legends of each figure.

##  4 Experiment Settings

Figure 3: By how much does contrastive self-supervised learning outperform
training from scratch? We plot $\bm{\mathrm{cR}_{f}\mathchar 45\relax n}$ and
$\bm{\mathrm{cR}_{f}\mathchar 45\relax\mathrm{cR}_{\mathrm{scratch}}}$ curves
for SwAV and MoCov2 across different downstream tasks. For the three
classification tasks, we use encoders pretrained on ImageNet (denoted as IN).
For surface normals and depth estimation, we use the Taskonomy pre-trained
versions of the same backbones. For all the tasks, contrastive SSL performs
better than no pre-training by a relatively large margin, and SwAV outperforms
MoCov2 in most cases.

To illustrate how the proposed control-baselines and evaluation standard can
be employed, we perform an example empirical analysis of transfer learning in
Section 3. The current section describes our experimental setup including pre-
training methods, datasets, and downstream-tasks. All experiments were
conducted using PyTorch [33], and the associated code will be made available
online.

###  4.1 Datasets and Downstream Tasks

The literature on benchmarking transfer performances of different pre-training
methods commonly focuses on image classification tasks [9, 25, 24]. Our
experiments similarly incorporate three examples of such classification
problems, namely ImageNet [11], CIFAR-100 [28], and EuroSAT [21]. In addition
to classification, we also include pixel-wise regression tasks using the
Taskonomy dataset [43]. In total, we use four different datasets, each
representing a distinct image domain with different properties.

Semantic Classification Tasks. ImageNet is a standard large-scale computer
vision dataset of 1.3M natural images, each containing an object from one of
the 1000 classes. We also use the relatively smaller CIFAR-100 and EuroSAT
datasets to evaluate how well self-supervised learned representations transfer
to different domains when only a small amount of labelled data is available.
For ImageNet and CIFAR-100, we use the standard train/test splits, and split
the EuroSAT dataset randomly. We train all models using the cross-entropy loss
and evaluate them on the corresponding test sets using error rate. As the
blind-guess prediction, we use the most common class. We train the maximal-
supervision control baselines for CIFAR-100 and EuroSAT from scratch, using
all available training images and use weights available from the PyTorch
library [33] for ImageNet.

Pixel-Wise Regression Tasks. We use two common pixel-wise regression tasks:
depth and surface normals estimation from the Taskonomy dataset [43].
Taskonomy contains 4M images of natural indoor scenes from about 600 different
buildings. We use images from the official full+ split buildings, and fix a
random subset of 60K images from the set of test buildings to evaluate the
final performances. We use $L_{1}$-norm as the loss for training on both
downstream tasks. The blind-guess control baseline for both tasks is computed
as the pixel-wise median over a set of 60K train images. The maximal-
supervision control baselines are trained from scratch using 1.2M images from
the same full+ split, and reach a level of performance close to the one
reported by [42].

###  4.2 Pre-Training Methods

Contrastive Self-Supervised Learning. In our showcase analysis, we consider
representative samples of contrastive self-supervised learning methods: SwAV
[3], MoCov2 [7], SimCLR [5], and PIRL [30]. These methods have the main
attributes of modern contrastive methods and most of them achieve state-of-
the-art performance on standard benchmarks.

Non-Contrastive Pre-Training. As examples of non-contrastive approaches, we
consider two pretext tasks based on image colorization [46] and jigsaw puzzle
[32]. We also use modern non-contrastive augmentation-based methods SimSiam
[8] and Barlow Twins [45]. We use the pre-trained models from the
corresponding github releases111SwAV:
https://github.com/facebookresearch/swav, MoCov2:
https://github.com/facebookresearch/moco/, SimSiam:
https://github.com/facebookresearch/simsiam, as well as pre-trained models
from the VISSL library 222VISSL: https://github.com/facebookresearch/vissl
[15].

Supervised Pre-Training. We consider two supervised tasks in our study to
measure whether there is a gap between supervised and self-supervised pre-
training approaches. First, we use a standard ImageNet pre-trained encoder, as
commonly employed by many evaluation studies, and generally adopted as an
initialization in the community. Second, for depth and surface normals
estimation, we choose the corresponding pre-training tasks from the Taskonomy
dictionary that were empirically found to result in the best transfer
performance. As reported by [43], this task is reshading for both. We pre-
train the reshading encoder on the Taskonomy dataset using the same number of
images as ImageNet pre-training. For the ImageNet encoder we use the weights
provided by PyTorch.

###  4.3 Architecture and Training Details

All pre-trained encoders share the same ResNet-50 architecture [20]. For
transfers to downstream tasks, we use two types of decoders. For
classification tasks, we use a single fully-connected layer that takes the
output of the final encoder’s layer and outputs the logits for each class. For
pixel-wise regression tasks, we use a UNet-style [35] decoder with six
upsampling blocks and skip-connections from the encoder layers of the same
spatial resolution. We randomly sub-sample $n$ images from the corresponding
training set to form each data-regime and split them into train and validation
sets. We use the validation split for early-stopping, as well as to choose the
hyper-parameters for different downstream tasks. The final performance is
evaluated on the fixed test split.

##  5 Empirical Analysis

The main goal of this section is to showcase an actual application of the
proposed evaluation standard on a set of exemplary scenarios. It presents a
targeted and small-scale empirical study that poses a set of questions about
the transfer performance of self-supervised pre-training methods, and proceeds
with a subsequent analysis that utilizes the suggested controls and
visualizations to address them.

Figure 4:  Do different tasks benefit differently from self-supervised pre-
training? The $\bm{\mathrm{cR}_{f}\mathchar
45\relax\mathrm{cR}_{\mathrm{scratch}}}$ plots above give a comparison across
all pre-training methods for each downstream task. The images given below show
corresponding visualizations of depth and normals predictions for different
methods at two different dataset sizes. It can be observed that I. the
differences among different pre-training methods are more pronounced for
classification tasks compared to pixel-wise regression tasks, and II. the best
pre-training result for classification is notably better than that of dense
regression. Particularly at the high data-regime, such differences become
insignificant. This suggests the development of pre-training methods may be
more curated towards downstream classification tasks.

###  5.1  By how much does contrastive self-supervised learning outperform
training from scratch?

To address this question, we visualize the $\bm{\mathrm{cR}_{f}\mathchar
45\relax\mathrm{cR}_{\mathrm{scratch}}}$ and $\bm{\mathrm{cR}_{f}\mathchar
45\relax n}$ curves for MoCov2 and SwAV pre-training methods. For both
methods, we use Taskonomy pre-trained encoders for pixel-wise regression tasks
and ImageNet pre-trained versions for classification tasks, as we found them
to perform better. The resulting curves are shown in Fig. 3.

We observe that both contrastive self-supervised methods outperform training
from scratch on all downstream tasks in all data-regimes with a relatively
large margin which, as one would expect, diminishes with more labelled data
available for transfer. Between two pre-training methods, SwAV outperforms
MoCov2 in most cases.

We note that the curves for CIFAR-100 eventually fall below the x axis. These
negative calibrated risk values mean that transfer learning eventually starts
outperforming the maximal-supervision control baseline (i.e., training from
scratch using a large amount of available data) due to the small scale of the
CIFAR-100 dataset. The relative improvement and CCI computations with respect
to the scratch performance, as well as comparisons between different methods
for the same task are still valid in this case, and the main difference is
that the curves no longer converge to the maximal-supervision baseline since
it is no longer a good approximation of the best achievable performance on the
downstream task.

###  5.2 Do different tasks benefit differently from self-supervised pre-
training?

The plots in Fig. 4 show how $\bm{\mathrm{cR}_{f}\mathchar
45\relax\mathrm{cR}_{\mathrm{scratch}}}$ curves for different pre-training
methods compare on five different downstream tasks. We observe that the
transfer performance differences between the pre-training methods are larger
on classification tasks and much less pronounced on pixel-wise regression
ones. Additionally, in the high data regime the differences between the
methods become insignificant for pixel-wise regression tasks, which can also
be qualitatively confirmed from the visualizations of depth and normals
predictions at two different dataset sizes. These observations suggest that
the development of pre-training methods may be implicitly biased towards
classification tasks, rather than rather general purpose representations.

###  5.3 By how much do contrastive methods outperform non-contrastive
methods?

Figure 5:  By how much do contrastive methods outperform non-contrastive
methods? We plot the $\bm{\mathrm{cR}_{f}\mathchar
45\relax\mathrm{cR}_{\mathrm{scratch}}}$ curves for contrastive pre-training
methods (SwAV and MoCov2) together with non-contrastive methods (colorization
and jigsaw). Colorization pre-training falls largely behind all other methods.
Jigsaw pre-training, however, performs relatively similar to the two
contrastive methods on pixel-wise regression tasks and EuroSAT classification,
while the gap on CIFAR-100 and ImageNet is much larger (see Sec.5.2).  Figure
6:  Does self-supervised Learning outperform supervised pre-training? We plot
the $\bm{\mathrm{cR}_{f}\mathchar 45\relax\mathrm{cR}_{\mathrm{scratch}}}$
curves for supervised and contrastive pre-training methods. ImageNet pre-
training as a supervised method is included in the plots for all downstream
tasks. For depth and normals estimation, we also include an encoder pre-
trained on the reshading task, which is the optimal supevised transfer domain
as reported by [43]. We observe that supervised pre-training performs
similarly to contrastive methods in most cases, except reshading supervised
pre-training on depth estimation in low-data regimes where it outperforms
self-supervised counterparts (discussion in Sec.5.4).

Recent works demonstrate that for semantic classification tasks, contrastive
pre-training approaches result in better transfer performance compared to non-
contrastive pretext tasks [19, 5]. In this section, we investigate if the same
trend holds for pixel-wise regression tasks. In Fig. 5, we show transfer
performance results for colorization and jigsaw as compared to contrastive
methods. For classification tasks, we replicate the common observation that
contrastive methods provide a significant improvement compared to non-
contrastive methods. However, we again observe that the difference is less
pronounced on pixel-wise regression tasks, echoing the conclusion of Section
5.2.

###  5.4 Does self-supervised pre-training outperform supervised pre-training?

ImageNet Supervised Pre-Training. In Fig. 6, we compare ImageNet pre-training
and contrastive methods. We observe that contrastive methods generally perform
comparable or better than ImageNet pre-training. However, in terms of relative
improvement against the scratch control baseline, this difference was found to
be relatively minor.

Best Informed Pre-Training. For depth and normals estimation, we include
further comparisons against the best pre-training task according to Taskonomy
[43] (see Section 4.2 for more details). The transfer results are presented in
Fig. 6. We did not observe a large difference between contrastive and
supervised pre-training for normals prediction, while the difference is more
pronounced for depth estimation. This seems consistent with [43] where the
best source tasks were found to work well _if_ they it is a close match with
the target.

##  6 Conclusion and Limitations

We put forth an evaluation standard for measuring the effectiveness of pre-
training methods for transfer learning, which incorporates three important
control baselines:

  * •

the scratch control baseline to disentangle the benefits of transfer learning
from the inductive biases introduced by other training choices that are
independent from the particular pre-training method,

  * •

the maximal-supervision control baseline to account for the effects of
irreducible factors inherent to the downstream-task that upper-bound the best
possible transfer performance,

  * •

the blind-guess control baseline to normalize away the benefits of the
statistical regularities of particular datasets that lower-bound the worst
possible transfer performance.

In Section 5, we used these control baselines to define the calibrated risk
metric $\mathrm{cR}_{f}$, with the goal of framing downstream task dependent
empirical risk in an interpretable scale. We further showed two visualizations
to plot calibrated risk curves, with the overall goal of providing an
accessible way to judge the transfer efficacy of a given pre-training method,
as well as to facilitate comparisons across methods. In Section 3, we have
provided a targeted small-scale empirical study that showcases how the
proposed evaluation standard can be applied. Through this study, we mainly
observed that:

  * •

Classification tasks consistently benefit more from contrastive self-
supervised pre-training, compared to non-classification tasks such as pixel-
wise regression (depth and surface normals estimation).

  * •

The differences between self-supervised pre-training methods (and particularly
between contrastive and non-contrastive approaches) are much less pronounced
for non-classification tasks, as compared to classification tasks where there
is a clear improvement from using contrastive methods. These are useful
indications and insights for self-supervised learning research toward
developing truly more general representation.

Our study, however, comes with several limitations:
Cross-Task Comparisons. Comparing methods across different tasks using their
calibrated risk values makes the assumption that the prediction fidelity of
each task can be captured by an affine transformation of its task-specific
loss function, which may not hold true in practice.
Other Benefits of SSL. Besides reducing the need to label data, self-
supervised learning provides other benefits, such as enabling training on
continuous data streams (as opposed to fixed training sets) or reducing the
reliance on rigid category definitions. This paper focused on quantifying the
effectiveness of self-supervised learning in terms of transfer learning and
reducing labeled data demands only.
Pre-training and Downstream Tasks Diversity. In comparison to other works
whose main focus is to provide a comprehensive empirical study [27, 4, 16, 24,
40], this paper mainly aims to propose an evaluation standard, and
consequently we consider a more limited set of pre-training and downstream-
tasks mainly curated to showcase the importance of the proposed control
baselines.
The Effects of Pre-training Dataset Size. We evaluate the transfer efficacy as
a function of the transfer dataset size, but the dependency on the pre-
training dataset size is another important factor that determines the overall
transfer performance, as studied in [16, 19].

##  7 Acknowledgment

We thank Roman Bachmann for sharing the transfer codebase for Taskonomy and
for useful discussion.

## References

  * [1] Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi.  A theoretical analysis of contrastive unsupervised representation learning.  arXiv preprint arXiv:1902.09229, 2019.
  * [2] Adrien Bardes, Jean Ponce, and Yann LeCun.  Vicreg: Variance-invariance-covariance regularization for self-supervised learning.  arXiv preprint arXiv:2105.04906, 2021.
  * [3] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  arXiv preprint arXiv:2006.09882, 2020.
  * [4] Levy Chaves, Alceu Bissoto, Eduardo Valle, and Sandra Avila.  An evaluation of self-supervised pre-training for skin-lesion analysis.  arXiv preprint arXiv:2106.09229, 2021.
  * [5] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [6] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton.  Big self-supervised models are strong semi-supervised learners.  arXiv preprint arXiv:2006.10029, 2020.
  * [7] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  arXiv preprint arXiv:2003.04297, 2020.
  * [8] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750–15758, 2021.
  * [9] Elijah Cole, Xuan Yang, Kimberly Wilber, Oisin Mac Aodha, and Serge Belongie.  When does contrastive visual representation learning work?  arXiv preprint arXiv:2105.05837, 2021.
  * [10] James Coughlan and Alan L Yuille.  The manhattan world assumption: Regularities in scene statistics which enable bayesian inference.  Advances in Neural Information Processing Systems, 13:845–851, 2000\.
  * [11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [12] Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir Zamir.  Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10786–10796, 2021.
  * [13] Linus Ericsson, Henry Gouk, and Timothy M Hospedales.  How well do self-supervised models transfer?  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5414–5423, 2021.
  * [14] Spyros Gidaris, Praveer Singh, and Nikos Komodakis.  Unsupervised representation learning by predicting image rotations.  arXiv preprint arXiv:1803.07728, 2018.
  * [15] Priya Goyal, Quentin Duval, Jeremy Reizenstein, Matthew Leavitt, Min Xu, Benjamin Lefaudeux, Mannat Singh, Vinicius Reis, Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Ishan Misra.  Vissl.  https://github.com/facebookresearch/vissl, 2021.
  * [16] Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra.  Scaling and benchmarking self-supervised visual representation learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6391–6400, 2019.
  * [17] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent: A new approach to self-supervised learning.  arXiv preprint arXiv:2006.07733, 2020.
  * [18] Raia Hadsell, Sumit Chopra, and Yann LeCun.  Dimensionality reduction by learning an invariant mapping.  In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), volume 2, pages 1735–1742. IEEE, 2006.
  * [19] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729–9738, 2020.
  * [20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [21] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth.  Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
  * [22] Olivier Henaff.  Data-efficient image recognition with contrastive predictive coding.  In International Conference on Machine Learning, pages 4182–4192. PMLR, 2020.
  * [23] Derek Hoiem, Tanmay Gupta, Zhizhong Li, and Michal Shlapentokh-Rothman.  Learning curves for analysis of deep networks.  In International Conference on Machine Learning, pages 4287–4296. PMLR, 2021.
  * [24] Ashraful Islam, Chun-Fu Chen, Rameswar Panda, Leonid Karlinsky, Richard Radke, and Rogerio Feris.  A broad study on the transferability of visual representations with contrastive learning.  arXiv preprint arXiv:2103.13517, 2021.
  * [25] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby.  Big transfer (bit): General visual representation learning.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16, pages 491–507. Springer, 2020.
  * [26] Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer.  Revisiting self-supervised visual representation learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1920–1929, 2019.
  * [27] Klemen Kotar, Gabriel Ilharco, Ludwig Schmidt, Kiana Ehsani, and Roozbeh Mottaghi.  Contrasting contrastive self-supervised representation learning pipelines.  In ICCV, 2021.
  * [28] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [29] Jason D Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo.  Predicting what you already know helps: Provable self-supervised learning.  arXiv preprint arXiv:2008.01064, 2020.
  * [30] Ishan Misra and Laurens van der Maaten.  Self-supervised learning of pretext-invariant representations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6707–6717, 2020.
  * [31] Alejandro Newell and Jia Deng.  How useful is self-supervised pretraining for visual tasks?  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7345–7354, 2020.
  * [32] Mehdi Noroozi and Paolo Favaro.  Unsupervised learning of visual representations by solving jigsaw puzzles.  In European conference on computer vision, pages 69–84. Springer, 2016.
  * [33] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  2017\.
  * [34] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context encoders: Feature learning by inpainting.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536–2544, 2016.
  * [35] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [36] Alexander Sax, Jeffrey O. Zhang, Bradley Emi, Amir Zamir, Silvio Savarese, Leonidas Guibas, and Jitendra Malik.  Learning to navigate using mid-level visual priors, 2019.
  * [37] Maxim Tatarchenko, Stephan R Richter, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox.  What do single-view 3d reconstruction networks learn?  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3405–3414, 2019.
  * [38] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola.  What makes for good views for contrastive learning?  arXiv preprint arXiv:2005.10243, 2020.
  * [39] Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency.  Self-supervised learning from a multi-view perspective.  arXiv preprint arXiv:2006.05576, 2020.
  * [40] Grant Van Horn, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha.  Benchmarking representation learning for natural world image collections.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12884–12893, 2021.
  * [41] Tete Xiao, Xiaolong Wang, Alexei A Efros, and Trevor Darrell.  What should not be contrastive in contrastive learning.  arXiv preprint arXiv:2008.05659, 2020.
  * [42] Amir R Zamir, Alexander Sax, Nikhil Cheerla, Rohan Suri, Zhangjie Cao, Jitendra Malik, and Leonidas J Guibas.  Robust learning through cross-task consistency.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11197–11206, 2020.
  * [43] Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese.  Taskonomy: Disentangling task transfer learning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3712–3722, 2018.
  * [44] Amir R Zamir, Tilman Wekel, Pulkit Agrawal, Colin Wei, Jitendra Malik, and Silvio Savarese.  Generic 3d representation via pose estimation and matching.  In European Conference on Computer Vision, pages 535–553. Springer, 2016.
  * [45] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny.  Barlow twins: Self-supervised learning via redundancy reduction.  arXiv preprint arXiv:2103.03230, 2021.
  * [46] Richard Zhang, Phillip Isola, and Alexei A Efros.  Colorful image colorization.  In European conference on computer vision, pages 649–666. Springer, 2016.