PREPRINT 3.5 0.9

11institutetext: Faculty of Electronics, Telecommunications and Informatics
Gdańsk University of Technology, Poland
<EMAIL_ADDRESS><EMAIL_ADDRESS>
# Architecture design of a networked music performance platform for a chamber
choir

Jan Cychnerski  0000-0003-2733-4599 Bartłomiej Mróz  0000-0002-8983-9050

###### Abstract

This paper describes an architecture design process for Networked Music
Performance (NMP) platform for medium-sized conducted music ensembles, based
on remote rehearsals of Academic Choir of Gdańsk University of Technology. The
issues of real-time remote communication, in-person music performance, and NMP
are described. Three iterative steps defining and extending the architecture
of the NMP platform with additional features to enhance its utility in remote
rehearsals are presented. The first iteration uses a regular video
conferencing platform, the second iteration uses dedicated NMP devices and
tools, and the third iteration adds video transmission and utilizes
professional low-latency audio and video workstations. For each iteration, the
platform architecture is defined and deployed with simultaneous usability
tests. Its strengths and weaknesses are identified through qualitative and
quantitative measurements – statistical analysis shows a significant
improvement in rehearsal quality after each iteration. The final optimal
architecture is described and concluded with guidelines for creating NMP
systems for said music ensembles.

###### Keywords:

r

eal-time system, networked music performance, platform design

##  1 Introduction and related work

In recent years, advances in technology have resulted in the increasing use of
real-time remote working devices. Also, the COVID-19 pandemic made many
aspects of everyday life transfer to the virtual world. For some fields of
work this change does not affect the efficiency and quality of work.
Nevertheless, remote creative and artistic work, particularly music
performance, remains a challenge.

###  1.1 Real-time communication

Recently, various real-time conferencing tools (e.g. Zoom, MS Teams, Google
Meet) rapidly expanded and amply gained new users [16]. The success of
conferencing platforms was largely caused by the general features that make
typical Internet communication (Fig. 1) simple and effective, which include
[8, 10]: ease of installation, convenience of use, ability too operate under
varying network conditions, resistance to audio noise (low quality microphone,
background noises), video and screen sharing, chat, file transfer, shared
whiteboard, integration with office/school software.

In order to achieve the aforementioned goals, these applications must exhibit
very high automation and compatibility, which is achieved at a cost of [13]:

  1. 1.

long buffers, high transmission delays reaching several hundred milliseconds

  2. 2.

very aggressive noise reduction, cutting out a large part of the bandwidth,
and even changing the audio signal content using AI

  3. 3.

strong, automated emphasis on the main speaker while muting the others

Figure 1: Typical real-time communication platform architecture

###  1.2 In-person music performance

A different set of requirements is posed by music performance in typical
medium-sized ensembles (e.g. chamber choirs), especially those with a
conductor. During normal stationary in-person rehearsals, communication is
characterized by [14, 18, 20]:

  1. 1.

no audiovisual delays of any kind – the performers are in the same room

  2. 2.

the performers hear each other and adjust their performance in real time to
the close performers and the overall sound of the ensemble (self-feedback)

  3. 3.

a conductor in front of the performers ensures overall cohesion,
interpretation, volume and tempo, conducting visually in real time

  4. 4.

the physical location of the performers matters – people standing closer (so
performing similar musical parts) are heard louder (intra-section feedback)

Figure 2: Typical stationary in-person music performance

###  1.3 Networked music performance

The features of typical conferencing tools are highly inadequate for a music
performance. For this reason, the topic of remote music performance is a
separate branch in research and industry, called Networked Music Performance
[9]. In recent years, advances in technology have resulted in the increasing
use of real-time remote working devices. Nevertheless, remote creative and
artistic work, particularly music performance, remains a challenge. One-way
connections, through which artistic events are transmitted in real-time, have
become available using networks capable of supporting high bandwidth, low-
latency packet routing, and guaranteed quality of service (QoS) [7, 5]. These
can be described as unidirectional, allowing artists to remotely connect only
between the performance venue and the audience, not each other. It is much
more difficult when performers in two or more remote locations attempt to
perform an established composition or improvisation together in real-time.
Weinberg [21] calls this way of performing music the “bridge approach”. In
such cases, the inevitable delay caused by the physical transit time of
network packets is known to affect performance [1, 3, 6, 12]. Therefore,
attempts have been made to account for these delays by design or composition
[2, 4].

A number of NMP experiments and frameworks are described in [19, 15, 17]. Some
platforms, like MusiNet or Diamouses, are large-scale projects run by
universities; other, like LoLa or JackTrip require a subscription or licensing
plan. The Jamulus platform stands out as an open-source, community-driven and
easy to use NMP tool [11]. It allows joint performance via freely available
servers, or via dedicated server configured with Jamulus software on a local
machine. These aspects are especially appealing to smaller, non-professional
ensembles (e.g. students, local communities, etc.) which are not capable of
huge financial investments or involvement in research projects regarding
remote rehearsing.

###  1.4 Contribution

This paper discusses the process of defining and practically realizing a
platform architecture that meets the requirements for networked music
performance of medium-sized conducted amateur music ensembles through the
example of remote rehearsals of a chamber choir. The proposed architecture in
contrast to existing publications and commercial solutions takes into account:
(1) feasibility study in a highly ecologically valid, choral environment (vs
artificial environments in NMP papers [19]), (2) use of available, open-source
platforms for low-latency transmission of both audio and video (vs difficult
to obtain, scientific projects like Internet2, Musinet, Diamouses, GigaPoP,
etc.), (3) consideration for the lack of technical capabilities of performers,
(4) consideration for a low-cost hardware options, (5) proposal for an
architecture for the NMP and a set of good practices.

##  2 Experimental setup

In the experiment, 3 NMP architectures were proposed and investigated for a
$\sim$20-member choral ensemble: Zoom@Home, Jamulus@Home and Jamulus@Univ.
Each architecture was implemented in practice as means for the rehearsals of
the Academic Choir of the Gdansk University of Technology, during the 2021
COVID-19 outbreak in Poland. After deployment, the architectures were used
without major modifications for nearly 2 months each. Architectures were
evaluated qualitatively by the choristers through a survey and by the authors
responsible for the implementation. Quantitative assessment was performed
through a round trip time (RTT) audio and video latency measurements. After
each deployment, various aspects affecting rehearsal quality were determined,
and based on them, decisions were made to implement modifications in the next
iteration. Comparative features of all architectures are shown in Table 1, and
described in detail in the following subsections.

Table 1: Comparison of proposed architecture features Architecture |  Zoom@Home |  Jamulus@Home |  Jamulus@Univ
---|---|---|---
Connection |  any |  broadband |  LAN
Choir audio |  any |  Semi-professional; ASIO drivers |  professional; ASIO interface
Conductor audio |  any |  professional, using ASIO interface
Audio RTT |  300-1000 ms |  63-135 ms |  40-85 ms
Choir video |  any |  – |  –
Conductor video |  any |  – |  low latency camera
Video latency |  500-1000 ms |  – |  25-100 ms
Feedback |  – |  self & intra-section feedback, choir mix
Setup |  automatic, self-supervised |  manual |  manual, semi-supervised
Assistance |  – |  remote |  remote+local
Rehearsal scope |  individual singing with muted mic, solo singing |  singing section parts or tutti with piano |  Jamulus@Home + singing tutti a’cappella

###  2.1 Zoom@Home architecture: a simple conferencing tool

The first proposed architecture for remote choir rehearsals was a standard
web-based communication platform using popular tools. Therefore, Skype, Jitsi,
MS Teams and Zoom platforms were initially tested. Eventually Zoom was
selected due to its popularity, ease of use, compatibility and very low entry
threshold. No changes were made to the typical conferencing architecture, as
shown in Fig. 1. Simultaneous singing was effectively impossible due to
significant delays and the automated speaker highlight feature that amplifies
one person and mutes the others. The choir could rehearse only in the
following capacities: (1) casual conversation as in a typical conference,
where everyone has microphone on and everyone can talk, (2) singing together
with microphones turned off; the only active microphone is the conductor’s,
who leads the rehearsal by playing choral parts on the piano, and (3) solo
singing with occasional help and commentary from the conductor. From a
technical standpoint, the Zoom@Home architecture posed no problems. There were
no requirements for hardware, Internet connection, or location. Installation
and configuration were possible on any device; each chorister configured the
Zoom application without technical assistance; most choristers used a webcam
to enhance the feeling of presence at a rehearsal. RTT latencies ranged from
300-1000 ms for the audio channel and 500-1000 ms for the video channel. Such
high RTTs prevented more than one person from singing at a time, making it
impossible to measure relative latency between choristers.

###  2.2 Jamulus@Home architecture: NMP with choristers at home

The second iteration of the architecture – presented in Fig. 3 – introduced
several improvements. Jamulus software was chosen due to its main features:
open-source, cross-platform, built by the NMP community, focused on low
latency audio connection, no limitations on the number of participants, and no
requirements for specialized hardware. Several inexpensive headsets were
reviewed for latency, compatibility, sound quality, and comfort; then one
optimal model (Logitech PC 960 USB) was selected. Additionally, semi-
professional low-latency compatible audio equipment was allowed. To ensure
minimal delays and high quality of the conductor’s audio connection, he was
provided with professional audio equipment. The video connection got
discarded, further lowering the latency; a dedicated Jamulus server was
deployed at the University, ensuring broadband connection in the nearest area.
A requirement was imposed for a minimum internet connection type (wired
broadband) and a maximum geographical distance from the Jamulus server (100
km). A correctly configured low-latency ASIO driver was required. That
resulted in RTT in the range of 63-135 ms, which allowed for real-time
collaborative singing.

Figure 3: Jamulus@Home NMP platform architecture

Despite attempts to unify the hardware and imposed requirements, the hardware
and software configuration had to be done manually by users, with active
remote support (via Windows Remote Assitance or TeamViewer) provided by two
designated technicians. The real-time NMP solution required a complete
revision of remote workflow. Simultaneous singing became viable – it allowed
the whole ensemble to sing together (tutti) for the first time under pandemic
restrictions. Mechanisms included in the Jamulus software enabled the
choristers to adjust the volume of particular persons, allowing them for
intra-section feedback (strengthening their sections and weakening the rest);
similar to traditional rehearsals. The conductor could communicate with the
choristers on a real-time basis and lead through accompaniment on a piano. It
became possible to rehearse in sections and hear the overall sound of the
pieces, allowing meaningful work on the repertoire

Not all choristers could meet the requirements of NMP (equipment,
connectivity, etc.). A solution to this problem was an additional gateway
server to the Zoom platform. As a countermeasure for high delays in the audio
signal of the Zoom platform, the communication with the gateway server was
one-way. Thus, it allowed choristers not capable of NMP for passive
participation in rehearsals.

###  2.3 Jamulus@University architecture: low-latency audio+video, choristers
and conductor at the University

Figure 4: Jamulus@University NMP platform architecture

Despite the ability to sing together in the previous iteration, persisting
transmission delays caused frequent loss of synchronization between
choristers, requiring the conductor to enforce the tempo through either
accompaniment or counting aloud. Still, maintaining a steady performance was
very difficult and unattainable without a conductor. Moreover, due to the lack
of video input, it was impossible for the conductor to lead visually, making
it impossible for him to control the tempo variability of the pieces
(accelerations, decelerations, fermatas, tempo rubato).

In the third iteration of the architecture design, namely Jamulus@University,
choristers got invited to a dedicated classroom at the university, equipped
with pre-configured professional audio hardware, providing high quality audio
with very low latency. The conductor’s workstation was supplied with low-
latency audio-video equipment and moved to another classroom with a piano. In
order to ensure a low latency video connection, a dedicated Jitsi server was
deployed, providing the conductor’s video with a latency of 25-100 ms. Such a
delay allowed the conductor to feasibly lead the choir in a real-time manner
visually, which was previously inaccessible. All communication was done over a
LAN, reducing the audio latency to 40-85 ms RTT. Controlled environment
allowed creation of semi-automated scripts facilitating deployment of
necessary software for workstations (2-person assistance was still in place,
but in smaller capacity).

The Jamulus@University was the most extensive of the architectures tested, as
presented in Fig 4. It was used for two months of remote rehearsals and was
crucial in sustaining choral activities during the COVID-19 lockdown.

##  3 Results

Each iteration contained a questionnaire assessing NMP architecture’s
usability. Since the remote rehearsals lasted several months, some choristers
participated in several iterations. In total, 23 choir members (9 male, 14
female, aged 18-30) participated in the experiments. The Zoom@Home
architecture assessment consists of 8 responses; the Jamulus@Home and the
Jamulus@University assessments comprise 16 and 13 answers, respectively. All
questions and answer scales are presented in 2.

Table 2: Questions and answer scales in the questionnaire Question | Applies to | Likert scale description
---|---|---
Excersiing difficulty |  Zoom@Home Jamulus@Home Jamulus@University |  | 1-3 - more difficult
---
4 - about the same
5-7 - easier
Rehearsing comfort |  | 1-3 - less comfortably
---
4 - about the same
5-7 - more comfortably
Setup difficulty | Jamulus@Home |  | 1 - very difficult, a lot of help needed
---
5 - very easy, everything highly understandable
Rehearsal safety | Jamulus@University |  | 1 - strong fear of infection
---
5 - no fear of infection

###  3.1 Assessment of the Zoom@Home architecture

(a)

(b)

Figure 5: Exercising difficulty and rehearsing comfort of the Zoom@Home NMP,
respectively. Likert scale: 1-3 = more difficult / less comfortable, 4 = about
the same, 5-7 = easier / more comfortable

The first examined was the Zoom@Home NMP architecture. As it did not differ
much from typical conferencing platforms, only two rehearsal quality aspects
were measured – exercising difficulty and rehearsing comfort, referring to
traditional, non-remote rehearsals – as shown in Fig. 5. Most of the
participants rated the difficulty as 2: more difficult than on traditional
rehearsal. Some participants reported shyness with solo singing; the rating
might reflect that intricacy. Interestingly, the rehearsing comfort was rated
mainly as 3: a bit less comfortably, than on traditional rehearsal, and 4:
about the same, as on traditional rehearsal. Many choristers reported high
comfort in rehearsing at home, especially those with conducive conditions.
However, some participants reported difficulties when singing at home; the
most highlighted were other household members, neighbors getting easily
irritated, or a noisy environment with construction or traffic noise from
nearby places.

###  3.2 Assessment of the Jamulus@Home architecture

(a)

(b)

(c)

Figure 6: Exercising difficulty, rehearsing comfort, and setup difficulty of
Jamulus@Home NMP. Likert scale for (c): 1 = very difficult, a lot of help
needed; 5 = very easy, everything highly understandable.

The Jamulus@Home architecture was evaluated by the following quantitative
factors: exercising difficulty and rehearsing comfort as compared to
traditional, non-remote rehearsals, inter-chorister latency, and setup
difficulty. Most participants rated the difficulty as 2: more difficult, and
3: a bit more difficult, than on traditional rehearsals. This rating appears
slightly higher than for the previous platform. The ratings for rehearsing
comfort are somewhat higher than for the Zoom platform; most choristers rated
the comfort as 4: about the same, as on traditional rehearsal. This
architecture allowed inter-chorister latency measurement: the time difference
of singing the same note was 83 ms ±57 ms. Furthermore, the setup difficulty
rating concentrated around 4 – as shown in Fig. 6(c) – indicating minor setup
difficulties with remote assistance required.

###  3.3 Assessment of the Jamulus@University architecture

(a)

(b)

(c)

Figure 7: Exercising difficulty, rehearsing comfort and safety of
Jamulus@University. Likert scale for (3): 1 = strong fear of infection, 5 = no
fear of infection

The Jamulus@University assessment comprised exercising difficulty and
rehearsing comfort ratings in relation to traditional, non-remote rehearsals
(Fig. 7), as well as inter-chorister latency. Most participants rated the
difficulty as 2: more difficult, and 3: a bit more difficult, than on
traditional rehearsals. The ratings for rehearsing comfort seem to be more
spread but mainly concentrated around score 4: about the same, as on
traditional rehearsal.

Higher comfort resulted also from a reduced sense of shyness in these
conditions. Additionally, the choristers were asked for a rating of perceived
safety, since it involved spending a considerable amount of time in one
classroom. The majority of the choristers rated the safety as 5: no fear of
infection (Fig. 7(c)). Such a result might reflect the convenient placement of
dedicated workstations in separate compartments made with plywood walls. This
setup gave the choristers a level of separation, especially in terms of
loudness and personal distance. It is also worth noting that no COVID-19
outbreak occurred within the choir at that time.

With reduced latency and the addition of video, it became possible to perform
tutti a’cappella pieces without accompaniment, metronome, or other enforced
tempo control. Overall, keeping the tempo became easier, even though some of
the choisters were still using the Jamulus@Home platform. The time difference
between different choristers singing the same note dropped to 47 ±46 ms.

###  3.4 Statistical analysis

All statistical calculations were performed with R software (version 4.1.2).
The most notable packages were MASS (7.3-54), car (3.0-12), emmeans (1.7.1-1),
ordinal (2019.12-10) and RVAideMemoire (0.9-80). The assessment of exercising
difficulty and rehearsing comfort was repeated for each architecture’s
iteration, allowing direct comparison. Some participants answered more than
one question; therefore, a generalized linear mixed model approach was
selected. Since the answers were on a 7-point Likert scale, the Cumulative
Link Mixed Model (CLMM) was chosen for the analysis. The dependent variable,
Platform, being a categorical variable, was encoded using Helmert contrast
coding. Also, the analysis was performed on both unweighted and weighted
response data. The weights were derived from the number of participants and
were assigned as follows: 1-2 rehearsals – 1; 3-6 rehearsals – 2; 7 and more
rehearsals – 3. Results are shown in Fig. 8.

(a)

(b)

Figure 8: Rehearsing comfort and exercising difficulty statistics

An analysis of variance for unweighted data based on ordinal logistic
regression indicated no statistical effect on rehearsing comfort
($\chi^{2}$(2,N=37) = 1.93, n.s.) However, testing on weighted data indicated
a statistical effect ($\chi^{2}$(2,N=86) = 10.84, $p$ < 0.01). Since the
statistical significance was detected on weighted data, post-hoc analysis was
performed for this scenario. Pairwise comparisons using Z-tests, corrected
with Holm’s sequential Bonferroni procedure, indicated that Likert scores for
Jamulus@Home vs. Jamulus@Univ were statistically significantly different (Z =
-3.1, $p$ < 0.01), but Likert scores for Jamulus@Home vs. Zoom@Home and for
Jamulus@Univ vs. Zoom@Home were not significantly different (Z = -1.4, n.s.,
and Z = 1.7, n.s.).

The statistical analysis is consistent with what is shown in Fig. 8a – the
rehearsing comfort is similar for participants in all three platforms.
However, weighted Likert scores induced a statistically significant difference
between Jamulus@Home platform and Jamulus@Univ platform, suggesting higher
comfort of rehearsing for the latter.

An analysis of variance for unweighted data based on ordinal logistic
regression indicated a statistical effect on exercising difficulty
($\chi^{2}$(2, N=37) = 29.5, $p$ < 0.001). For weighted data, this test also
indicated a statistical effect ($\chi^{2}$(2, N=86) = 46.4, $p$ < 0.001).
Thus, post-hoc analysis was performed for both cases. For unweighted Likert
scores, pairwise comparisons using Z-tests, corrected with Holm’s sequential
Bonferroni procedure, indicated that Likert scores for Jamulus@Home vs.
Jamulus@Univ were statistically significantly different (Z = -1910, $p$ <
0.001), also Likert scores for Jamulus@Home vs. Zoom@Home and for Jamulus@Univ
vs. Zoom@Home were statistically significantly different (Z = 24025, $p$ <
0.001, and Z = 29152, $p$ < 0.001). The comparisons for weighted Likert scores
indicated the higher magnitude of statistically significant differences:
Jamulus@Home vs. Jamulus@Univ (Z = -6494, $p$ < 0.001), Jamulus@Home vs.
Zoom@Home Z = 31852, $p$ < 0.001), and Jamulus@Univ vs. Zoom@Home (Z = 42875,
$p$ < 0.001). The statistical analysis is somewhat inconsistent with what is
in Fig. 8b -– the exercising difficulty on the Zoom@Home platform is clearly
higher than on any of the Jamulus platforms, but both Jamulus platforms seem
to have the same exercising difficulty. However, the statistical analysis
showed for both weighted and unweighted Likert data a significant difference
between them, in favor of the Jamulus@Univ one; adding weights increased the
magnitude of this difference. That would be consistent with choristers’
opinions about this platform.

###  3.5 Proposed architecture, recommendations and good practices

The experiments and analysis of the deployed architectures’ assessments
concluded with the construction of the optimal architecture of the NMP
platform for chamber ensembles with a conductor. The general architecture
should fulfill the following requirements and good practices (also shown in
Fig. 9):

  1. 1.

affordable, cross-platform, low-latency (RTT not exceeding 100 ms) NMP
software, along with a dedicated server located near the performers;

  2. 2.

dedicated, low-latency (not exceeding 100 ms) video streaming server;

  3. 3.

conductor’s station with professional equipment, fast network, close to the
servers, allowing unrestricted spoken, accompanied and visual conducting;

  4. 4.

performer stations with professional audio equipment, located close to the
servers, allowing for adjustable self-feedback, the ability to hear the
overall ensemble’s mix, with additional amplification of their own section;

  5. 5.

performers who cannot use (4) should be equipped with dedicated audio
equipment and a broadband Internet connection; unification of hardware reduces
installation problems and allows for the creation of config scripts;

  6. 6.

performers who cannot use (5) may participate in rehearsals passively (watch /
listen only) through a one-way gateway to popular communicators;

  7. 7.

remote assistance must be available for all performers at all times, according
to the users technical skills; it is worthwhile to test the connection in
groups to make sure that the configuration is correct and to fix problems in
advance

Figure 9: Proposed generalized NMP platform architecture

##  4 Summary

This paper outlines three proposed Networked Music Performance architectures
dedicated to chamber musical ensembles with a conductor, which were defined,
deployed and examined in practice. In contrast to other NMP systems and
publications, proposed architectures combine uncommon and conflicting
requirements of medium-sized choirs, like intra-section feedback, conductor’s
real-time video, ease of deploy and more. Subsequent architecture defisn
iterations were assessed against their usefulness. The conclusive architecture
is described along with listed guidelines for its implementation in section
3.5.

The Zoom@Home architecture was easy to use; however, it did not allow for
rehearsals similar in quality to in-person rehearsals. The Jamulus@Home
architecture, implementing software and hardware dedicated to NMP, allowed for
actual remote real-time music performance. However, it required much effort
from performers, the conductor, and extensive technical assistance. The final
Jamulus@University architecture enabled visual input from the conductor and
reduced delays between performers, effectively increasing the overall quality
of rehearsals, and allowing the choir to effectively work on the new
repertoire.

This work proves that it is possible and useful to implement NMP platforms for
said musical ensembles using existing affordable tools and equipment. However,
overall quality of rehearsals needs to be improved. Work needs to be done to
decrease network and audio latencies by incorporating low-latency drivers into
operating systems. A lot of effort must be put to lower the entry threshold
for NMP tools: automatic configuration and volume control, intuitive user
interfaces, integration of video and audio signals, remote assistance helpers.
Enabling technologies like peer-to-peer communication, VST plugins, ambisonics
and virtual reality techniques would improve musicians’ experience even more,
allowing to create better NMP platforms in the future.

## References

  * [1] Bartlette, C., Headlam, D., Bocko, M., Velikic, G.: Effect of network latency on interactive musical performance. Music Perception 24(1), 49–62 (2006)
  * [2] Bouillot, N.: nJam user experiments: Enabling remote musical interaction from milliseconds to seconds. In: Proceedings of the 7th international Conference on New interfaces For Musical Expression. pp. 142–147 (2007)
  * [3] Bouillot, N., Cooperstock, J.R.: Challenges and performance of high-fidelity audio streaming for interactive performances. In: NIME. pp. 135–140. Citeseer (2009)
  * [4] Cáceres, J.P., Hamilton, R., Iyer, D., Chafe, C., Wang, G.: To the edge with China: Explorations in network performance. In: ARTECH 2008: Proceedings of the 4th International Conference on Digital Arts. pp. 61–66 (2008)
  * [5] Chafe, C.: Tapping into the Internet as an acoustical/musical medium. Contemporary Music Review 28(4-5), 413–420 (2009)
  * [6] Chafe, C., Caceres, J.P., Gurevich, M.: Effect of temporal separation on synchronization in rhythmic performance. Perception 39(7), 982–992 (2010)
  * [7] Chafe, C., Wilson, S., Leistikow, R., Chisholm, D., Scavone, G.: A simplified approach to high quality music and sound over IP. In: Proceedings of the COST G-6 Conference on Digital Audio Effects (DAFX-00). pp. 159–164. Citeseer (2000)
  * [8] Correia, A.P., Liu, C., Xu, F.: Evaluating videoconferencing systems for the quality of the educational experience. Distance Education 41(4), 429–452 (2020). https://doi.org/10.1080/01587919.2020.1821607
  * [9] Dessen, M.: Networked music performance: An introduction for musicians and educators. https://ujeb.se/mrEpS5 (2020), [Online; accessed 15-May-2022]
  * [10] Faadhilah, A.F., Elfitri, I.: Comparison of audio quality of teleconferencing applications using subjective test. In: Audio Engineering Society Convention 152\. Audio Engineering Society (2022)
  * [11] Fischer, V.: Case study: Performing band rehearsals on the Internet with Jamulus
  * [12] Gu, X., Dick, M., Kurtisi, Z., Noyer, U., Wolf, L.: Network-centric music performance: Practice and experiments. IEEE Communications 43(6), 86–93 (2005)
  * [13] Hoßfeld, T., Binzenhöfer, A.: Analysis of Skype VoIP traffic in UMTS: End-to-end QoS and QoE measurements. Computer Networks 52(3), 650–666 (2008)
  * [14] King, E.C.: Collaboration and the study of ensemble rehearsal. In: Eighth International Conference on Music Perception and Cognition (ICMPC8) (2004)
  * [15] Mall, P., Kilian, J.: Low-latency online music tool in practice (2021)
  * [16] Matulin, M., Mrvelj, Š., Abramović, B., Šoštarić, T., Čejvan, M.: User quality of experience comparison between Skype, Microsoft Teams and Zoom videoconferencing tools. In: Future Access Enablers for Ubiquitous and Intelligent Infrastructures. pp. 299–307. Springer International Publishing, Cham (2021)
  * [17] Onderdijk, K.E., Acar, F., Van Dyck, E.: Impact of lockdown measures on joint music making: playing online and physically together. Frontiers in psychology 12, 1364 (2021)
  * [18] Pennill, N.: Ensembles working towards performance: Emerging coordination and interactions in self-organised groups. Ph.D. thesis, University of Sheffield (2019)
  * [19] Rottondi, C., Chafe, C., Allocchio, C., Sarti, A.: An overview on networked music performance technologies. IEEE Access 4, 8823–8843 (2016)
  * [20] Volpe, G., D’Ausilio, A., Badino, L., Camurri, A., Fadiga, L.: Measuring social interaction in music ensembles. Philosophical Transactions of the Royal Society B: Biological Sciences 371(1693), 20150377 (2016)
  * [21] Weinberg, G.: Interconnected musical networks: Toward a theoretical framework. Computer Music Journal 29(2), 23–39 (2005)


$L^{p}(\mathbb{T}^{2})$ for $1\leq p<2$ with norm

$\|T^{\mathcal{Q}}\|_{2\to p}\lesssim_{d,p}\|\omega\|\,.$

The operator $T^{\mathcal{Q}}$ is bounded from $L^{2}(\mathbb{T}^{2})$ into
$L^{2,\infty}(\mathbb{T}^{2})$ and the operator $T_{\mathbb{R}}^{\mathcal{Q}}$
defined by (30) is bounded from $L^{2}(\mathbb{R}^{2})$ into
$L^{2,\infty}(\mathbb{R}^{2})$.

######  Remark 4.

Corollary A.3 should be compared to Corollary A.1, which has stronger
continuity requirements, but does not require the kernel to be homogeneous and
odd. The corollary is formulated on $\mathbb{R}^{2}$ with $\alpha=(1,2)$ only
to simplify the notation, it is still true in higher dimension and for all
$\alpha$, with a similar proof. Note also that in the isotropic case this
theorem holds for all integrable
$\phi:\mathbb{S}^{\mathbf{d}-1}\to\mathbb{C}$, without any continuity
assumptions. This follows from the polynomial Carleson theorem, since $K$ can
then be written as a superposition of Hilbert transforms along lines.

######  Remark 5.

We briefly explain where the oddness and homogeneity assumptions are used. In
the proof of Corollary A.3, we will decompose the kernel $K$ into smoother
pieces $K^{\prime}$. To then apply Theorem 1.1, we need good estimates for the
maximal functions $M^{K^{\prime}}$ and $R^{K^{\prime}}$. Since $K$ is odd and
homogeneous, we are able to choose the pieces $K^{\prime}$ to be odd and
homogeneous. Then we can write them as superpositions of Hilbert transforms
along the curves $(t,c\operatorname{sgn}(t)t^{2})$ to obtain bounds for the
maximal functions. The oddness assumption can be replaced by the condition
$K(x,y)=-K(-x,y)$, in that case the same argument works with Hilbert
transforms along parabolas. For general kernels $K$ we are not aware of
methods to decompose them while keeping good control of the maximal functions.

###### Proof of Corollary A.3.

Fix some function $\eta\in C_{c}^{\infty}((-1,1))$ with $\int\eta=1$ and set
$\eta_{j}(x)=2^{-j}\eta(2^{-j}x)$ for $j\leq 0$, and $\eta_{1}(x)=0$. Then we
have $\phi=\sum_{j\leq 0}\phi_{j}$ with $\phi_{j}=\phi*(\eta_{j}-\eta_{j+1})$.
By the Dini continuity of $\phi$ it holds for all $j$ that

$|\phi_{j}(x)|\leq\int|\phi(x-y)-\phi(x)|(|\eta_{j}(y)|+|\eta_{j+1}(y)|)\,\mathrm{d}y\lesssim\omega(2^{j})\,.$

The functions $\phi_{j}$ are differentiable and
$|\phi_{j}^{\prime}(x)|\lesssim\omega(1)2^{-2j}$. We decompose the kernel $K$:
Let

$K^{j}(x)=\frac{1}{\rho(x)^{|\alpha|}}\phi_{j}(\delta_{\rho(x)^{-1}}(x))\,.$

The uniform convergence of $\phi=\sum_{j\leq 0}\phi_{j}$ implies that
$K=\sum_{j\leq 0}K^{j}$ with uniform convergence on compact subsets not
containing $0$. Denoting by $(T^{j})^{\mathcal{Q}}$ the maximal modulation
operator on the torus associated to $K^{j}$, we thus have for all $f\in
L^{2}(\mathbb{T}^{\mathbf{d}})$ that

$T^{\mathcal{Q}}f\leq\sum_{j\leq 0}(T^{j})^{\mathcal{Q}}f$

The estimates for $\phi_{j}$ and $\phi_{j}^{\prime}$ imply that $K^{j}$ is a
Calderón-Zygmund kernel with constant $A(j)\lesssim 2^{-2j}$. Using
boundedness of the maximal functions associated to the curves
$(t,c\operatorname{sgn}(t)t^{2})$, one obtains

$\displaystyle\|M^{K^{j}}\|_{2\to
2}\sim\|\phi_{j}\|_{1}\lesssim\omega(2^{j})\,,\quad\|R^{K^{j}}\|_{2\to
2}\lesssim\|\phi_{j}\|_{1}\lesssim\omega(2^{j})\,.$

Applying Theorem 1.1, it follows that

$\|(T^{j})^{\mathcal{Q}}\|_{2\to
p}\lesssim\omega(2^{j})\log^{2}(e+2^{-2j}\frac{\omega(1)}{\omega(2^{j})})\lesssim
j^{2}\omega(2^{j})\,.$

We conclude that

$\|T^{\mathcal{Q}}\|_{2\to p}\leq\sum_{j\leq 0}\|(T^{j})^{\mathcal{Q}}\|_{2\to
p}\lesssim\sum_{j\leq
0}j^{2}\omega(2^{j})\lesssim\int_{0}^{1}\omega(t)\log^{2}(\frac{1}{t})\,\frac{\mathrm{d}t}{t}\lesssim\|\omega\|\,.$

This completes the proof of the $L^{2}(\mathbb{T}^{2})\to
L^{p}(\mathbb{T}^{2})$ estimate for $T^{\mathcal{Q}}$. The weak type estimates
on the torus and on $\mathbb{R}^{2}$ follow now exactly as in the proof of
Corollary A.1. ∎

## References

  * [Car66] Lennart Carleson  “On convergence and growth of partial sums of Fourier series”  In _Acta Math._ 116, 1966, pp. 135–157  DOI: 10.1007/BF02392815
  * [Chr+99] Michael Christ, Alexander Nagel, Elias M. Stein and Stephen Wainger  “Singular and maximal Radon transforms: analysis and geometry”  In _Ann. of Math. (2)_ 150.2, 1999, pp. 489–577  DOI: 10.2307/121088
  * [Fef73] Charles Fefferman  “Pointwise convergence of Fourier series”  In _Ann. of Math. (2)_ 98, 1973, pp. 551–571  DOI: 10.2307/1970917
  * [Gra14] Loukas Grafakos  “Classical Fourier analysis” 249, Graduate Texts in Mathematics  Springer, New York, 2014, pp. xviii+638  DOI: 10.1007/978-1-4939-1194-3
  * [Guo+17] Shaoming Guo, Lillian B. Pierce, Joris Roos and Po-Lam Yung  “Polynomial Carleson operators along monomial curves in the plane”  In _J. Geom. Anal._ 27.4, 2017, pp. 2977–3012  DOI: 10.1007/s12220-017-9790-7
  * [Lie09] Victor Lie  “The (weak-$L^{2}$) boundedness of the quadratic Carleson operator”  In _Geom. Funct. Anal._ 19.2, 2009, pp. 457–497  DOI: 10.1007/s00039-009-0010-x
  * [Lie20] Victor Lie  “The polynomial Carleson operator”  In _Ann. of Math. (2)_ 192.1, 2020, pp. 47–163  DOI: 10.4007/annals.2020.192.1.2
  * [LM07] Xiaochun Li and Camil Muscalu  “Generalizations of the Carleson-Hunt theorem. I. The classical singularity case”  In _Amer. J. Math._ 129.4, 2007, pp. 983–1018  DOI: 10.1353/ajm.2007.0026
  * [LT00] Michael Lacey and Christoph Thiele  “A proof of boundedness of the Carleson operator”  In _Math. Res. Lett._ 7.4, 2000, pp. 361–370  DOI: 10.4310/MRL.2000.v7.n4.a1
  * [Mna22] Gevorg Mnatsakanyan  “On almost-everywhere convergence of Malmquist-Takenaka series”  In _J. Funct. Anal._ 282.12, 2022, pp. Paper No. 10946133  DOI: 10.1016/j.jfa.2022.109461
  * [Nic72] R.. Nicolaides  “On a class of finite elements generated by Lagrange interpolation”  In _SIAM J. Numer. Anal._ 9, 1972, pp. 435–445  DOI: 10.1137/0709039
  * [PT03] Malabika Pramanik and Erin Terwilleger  “A weak $L^{2}$ estimate for a maximal dyadic sum operator on ${\mathbb{R}}^{n}$”  In _Illinois J. Math._ 47.3, 2003, pp. 775–813  URL: http://projecteuclid.org/euclid.ijm/1258138194
  * [PY19] Lillian B. Pierce and Po-Lam Yung  “A polynomial Carleson operator along the paraboloid”  In _Rev. Mat. Iberoam._ 35.2, 2019, pp. 339–422  DOI: 10.4171/rmi/1057
  * [Ram21] João P.. Ramos  “The Hilbert transform along the parabola, the polynomial Carleson theorem and oscillatory singular integrals”  In _Math. Ann._ 379.1-2, 2021, pp. 159–185  DOI: 10.1007/s00208-020-02075-5
  * [Roo19] Joris Roos  “Bounds for anisotropic Carleson operators”  In _J. Fourier Anal. Appl._ 25.5, 2019, pp. 2324–2355  DOI: 10.1007/s00041-018-09657-7
  * [Sjö71] Per Sjölin  “Convergence almost everywhere of certain singular integrals and multiple Fourier series”  In _Ark. Mat._ 9, 1971, pp. 65–90  DOI: 10.1007/BF02383638
  * [Ste61] E.. Stein  “On limits of sequences of operators”  In _Ann. of Math. (2)_ 74, 1961, pp. 140–170  DOI: 10.2307/1970308
  * [Ste93] Elias M. Stein  “Harmonic analysis: real-variable methods, orthogonality, and oscillatory integrals” With the assistance of Timothy S. Murphy, Monographs in Harmonic Analysis, III 43, Princeton Mathematical Series  Princeton University Press, Princeton, NJ, 1993, pp. xiv+695
  * [Ste95] Elias M. Stein  “Oscillatory integrals related to Radon-like transforms”  In _Proceedings of the Conference in Honor of Jean-Pierre Kahane (Orsay, 1993)_ , 1995, pp. 535–551
  * [SW01] Elias M. Stein and Stephen Wainger  “Oscillatory integrals related to Carleson’s theorem”  In _Math. Res. Lett._ 8.5-6, 2001, pp. 789–800  DOI: 10.4310/MRL.2001.v8.n6.a9
  * [SW78] Elias M. Stein and Stephen Wainger  “Problems in harmonic analysis related to curvature”  In _Bull. Amer. Math. Soc._ 84.6, 1978, pp. 1239–1295  DOI: 10.1090/S0002-9904-1978-14554-6
  * [Zor21] Pavel Zorin-Kranich  “Maximal polynomial modulations of singular integrals”  In _Adv. Math._ 386, 2021, pp. Paper No. 10783240  DOI: 10.1016/j.aim.2021.107832


# Fine-tuning with Very Large Dropout

Jianyu Zhang  Léon Bottou

###### Abstract

It is impossible today to pretend that the practice of machine learning is
compatible with the idea that training and testing data follow the same
distribution. Several authors have recently used ensemble techniques to show
how scenarios involving multiple data distributions are best served by
representations that are both richer than those obtained by regularizing for
the best in-distribution performance, and richer than those obtained under the
influence of the implicit sparsity bias of common stochastic gradient
procedures.

This contribution investigates the use of very high dropout rates instead of
ensembles to obtain such rich representations. Although training a deep
network from scratch using such dropout rates is virtually impossible, fine-
tuning a large pre-trained model under such conditions is not only possible
but also achieves out-of-distribution performances that exceed those of both
ensembles and weight averaging methods such as model soups.

This result has practical significance because the importance of the fine-
tuning scenario has considerably grown in recent years. This result also
provides interesting insights on the nature of rich representations and on the
intrinsically linear nature of fine-tuning a large network using a
comparatively small dataset.

Machine Learning, ICML

1pt 10

##  1 Introduction

The practice of machine learning has been shaped by the assumption that
training and testing examples are independently drawn from the same unknown
probability distribution. This is seldom the case in modern settings, not only
because this i.i.d. assumption breaks down for the problems of interest, but
also because it is often convenient to use multiple datasets that are known to
follow different distributions. For instance, we may pre-train a deep network
on a large dataset designed, fine-tune it on a smaller dataset specific to the
task of interest, and test on a collection of tasks designed to benchmark
various aspects of the system.

Many of the tenets of machine learning should therefore be regarded with
healthy suspicion. For instance, under the i.i.d. assumption, favoring
solutions with sparse representations has well-known benefits on the
generalization performance. Yet, several authors (Zhang et al., 2022; Zhang &
Bottou, 2023; Chen et al., 2023) make the point that scenarios involving
multiple distributions are best served by “ _richer representations_ ”, even
richer than those constructed without sparsity regularization using the usual
stochastic gradient learning procedure.

Although the precise meaning of “richer representation” is often unclear,
Zhang & Bottou (2023) point to the body of evidence suggesting an implicit
sparsity bias in deep networks training with stochastic gradient descent
(Andriushchenko et al., 2023; Blanc et al., 2020). In this setup, a feature
only survives when it brings an incremental training error advantage relative
to what can be achieved using all the other features already present in the
network. Zhang & Bottou (2023) then argue that a feature may not be
incrementally useful under the training distribution but

  * (a)

might be incrementally useful when the data follows a different distributions
of interest, or

  * (b)

might be useful under the training distribution when added to certain subsets
of the other existing features instead of all of them.

Therefore it is tempting to “enrich” the representation with features of type
(b), which can be found using the training data, and hope that some of these
will turn out to also be features of type (a) whose inclusion helps when the
data distribution changes.

The dropout technique (Srivastava et al., 2014) seems well suited to finding
features of type (b) because randomly masking units of a representation layer
during training amounts to forming random subsets of all other available
features. However, in order to form small subsets, one would have to use very
high levels of dropout. Unfortunately, training a sizable deep network from
scratch with such a large dropout is practically impossible.

There is however a practically meaningful scenario in which using a very large
dropout is possible. Both Ramé et al. (2022b) and Wortsman et al. (2022b)
assume that we have access to a model pre-trained using a very large dataset
for a task weakly related to the task of interest. This pre-trained model is
then fine-tuned on datasets that illustrate the task of interest, and then
tested on a dataset for the same task but with a different distribution.
Interestingly the models obtained by fine-tuning are found to be “ _linearly
connected_ ”, making it possible to approximate ensembles of networks by
merely averaging their weights.

In the present work, we adopt this empirical setup. However, instead of
enriching the representations by constructing ensembles (Zhang & Bottou, 2023)
or averaging weights (Ramé et al., 2022b, a; Wortsman et al., 2022b), we
simply _fine-tune using very large dropout levels_ , randomly masking about
90% of the units in the representation layer. We find that this simple
approach _exceeds the performance of either ensembles or weight-averagings_.
This result is not only _practically meaningful_ , but also clarifies the idea
of _richer representation_.

##  2 Related Work

##### Constructing versatile representations

Reusing or transferring features across related tasks has been commonplace for
more than one decade (Collobert et al., 2011; Bottou, 2011; Sharif Razavian et
al., 2014) and plays a fundamental role in the appeal of foundational models
(Bommasani et al., 2021a). However, once the optimization process has
identified a set of features that is sufficient to achieve near-optimal
performance on the training set, additional features are often discarded
because they do not bring an incremental benefit to the training error,
despite the fact that they may independently carry useful information (Zhang &
Bottou, 2023).

Researchers have devised ways to obtain more versatile representations by
engineering a diversity of datasets, architectures, and even hyper-parameters
(Chen et al., 2020; Wang et al., 2022; Dvornik et al., 2020; Bilen & Vedaldi,
2017; Gontijo-Lopes et al., 2021; Li et al., 2021, 2022; Chowdhury et al.,
2021), as an alternative to the most popular approach which consists of simply
using ever larger datasets (Bommasani et al., 2021b).

Interesting results have also been obtained without engineering diversity and
without increasing the dataset sizes. Zhang et al. (2022); Chen et al. (2023)
propose to discover rich representation through multiple training episodes
that adversarially reweight the training dataset to impede the use of
previously learned features. Zhang & Bottou (2023) show that surprisingly good
results can be obtained by concatenating the representations of multiple
networks that are trained in exactly the same way, save for the random seed
used in the stochastic gradient process.

##### Fine-tuning as a near-linear process

Although modern deep residual networks feature highly complex nonconvex cost
functions, several authors have shown that their final training phase remains
mostly confined to a nearly-convex attraction basin (Izmailov et al., 2018; Li
et al., 2018b; Frankle et al., 2020). The same observation holds when fine-
tuning a large pre-trained network using a dataset whose size is considerably
smaller than the dataset size one would need to train such a large network
from scratch. As long as one starts from the same pre-trained model, Wortsman
et al. (2022a); Ramé et al. (2022b, a) have observed that averaging the
weights of diverse fine-tuned models can reproduce the i.i.d. and o.o.d.
performances of the ensemble of these models, implying that fine-tuning is a
near-linear process.

Maddox et al. (2021); Mu et al. (2019) propose instead to approximate the
fine-tuning process with a first-order Taylor expansion, obtaining a linear
system operating on top of the NTK features. (Evci et al., 2022) match the
performance of fine-tuning by merely learning a strongly regularized linear
model that takes all internal layer states as inputs. Meanwhile (Yu et al.,
2023) efficiently fine-tune large foundational language models by essentially
restricting the weight updates to low dimensional manifolds.

##### Fine-tuning with very large dropout

Our contribution takes several aspects of these earlier works. We use very
large dropout to force the learning algorithm to create a redundant
representation without specifically engineering diversity, and we focus on the
fine-tuning scenario to provide a setup that is practically significant and
whose near-linearity makes it possible to use such high dropout rates in the
first place.

##  3 Fine-tuning and dropout

###  3.1 Method

The key results described later in this paper have been obtained with a very
simple method. The base model is a deep learning network with residual
connections trained on data that is related to but substantially larger than
the datasets illustrating the task of interest. Some of these datasets are
used to fine-tune the base model. Performance is reported on both held-out
data from the fine-tuning datasets (i.i.d. performance) and data from the
remaining datasets (o.o.d. performance).

Fine-tuning is carried out with a standard stochastic learning procedure (e.g.
SGD or Adam) after applying a very large dropout to the penultimate layer
representation $\Phi$. More precisely, following the decomposition of residual
networks proposed by Veit et al. (2016),

$\displaystyle\Phi(x)$
$\displaystyle=\underbrace{\vphantom{f_{1}(x)}x}_{\mathclap{\phi_{0}(x)}}+\underbrace{f_{1}(x)}_{\phi_{1}(x)}+\underbrace{f_{2}(x+f_{1}(x))}_{\phi_{2}(x)}+\dots=\sum_{i\in[0,\dots,l]}\phi_{i}(x)\,,$

where $f_{i}$ represents the function implemented by the $i$-th residual
block, and

$\displaystyle\Phi_{\mathrm{dropout}}(x)=\frac{m(\lambda)}{1-\lambda}\odot\Phi(x)\,,$

where $\odot$ represents the component-wise product and $m(\lambda)$ is a
vector of random Bernoulli variables equal to $0$ with probability $\lambda$
and $1$ with probability $1-\lambda$.

This additive decomposition of $\Phi(x)$ makes clear that applying dropout to
$\Phi(x)$ simultaneously blocks the contributions $\phi_{i}(x)$ of all
residual blocks. Applying dropout after each residual block would additionally
affect the inputs of the successive $f_{i}$, but would also greatly increase
the number of dropout parameters and potentially obfuscate the experimental
results. An experiment in appendix (Figure 10) showcases the effect of
dropping inner residual blocks (Huang et al., 2016) in o.o.d. fine-tuning.

###  3.2 The linear perspective

Figure 1: Interpolating pre-trained and fine-tuned weights approximates the
interpolation of the pretrained and fine-tuned model representations. With
mild fine-tuning (red), interpolating the model weights (dashed line) or
interpolating the representation layers (solid line) follow similar
trajectories. This property breaks down when one increases the number of fine-
tuning iterations, often at the expense of performance (blue).

The purpose of this section is to investigate why we can fine-tune with
dropout levels that would irremediably stall the optimization process if we
were training from scratch.

Recall that our initial model has been pre-trained on a dataset substantially
larger than the fine-tuning datasets. Solid empirical evidence (Ramé et al.,
2022b; Wortsman et al., 2022b) shows that one can interpolate the output of a
mixture of the original and fine-tuned models by merely averaging their
weights in the same proportions (Figure 1). This so-called ” _linear
connectivity_ ” property means that the fine-tuning operation is well
described by a first-order approximation.

$\displaystyle\Phi(x;w)=\Phi(x;w_{0})+\left[\frac{\partial\Phi(x;w_{0})}{\partial
w}\right](w-w_{0})$

Such a first-order approximation of fine-tuning can be implemented with a
linear layer that takes both $\Phi(x;w_{0})$ and the gradients
${[\partial\Phi(x;w_{0})}/{\partial w}]$ as inputs (see variants of this idea
in Maddox et al., 2021; Mu et al., 2019; Jaakkola & Haussler, 1998).

Evci et al. (2022) recognize that this linear approximation means that fine-
tuning does not construct new representations but merely exploits the existing
ones, perhaps in a manner similar to neural networks operating in the lazy
regime (Chizat et al., 2020). With substantial sparsity regularization, they
show that training a linear layer that takes all the internal network
representations as inputs matches or exceeds the regular fine-tuning
performance.

Residual networks with ReLU units provide an additional opportunity to
simplify this model. We can express the coefficients of the gradient matrix
$[{\partial\Phi(x;w_{0})}/{\partial w}]$ using the path decomposition of
Neyshabur et al. (2015). Since the weight coefficients must be much smaller
than 1 in order to obtain roughly normalized activations, the gradient
decomposition is dominated by the shorter paths that involve skip-connections.
Skip-connections follow a very regular pattern because they only connect
representation units with the same indices within their layer. Therefore we
propose to approximate fine tuning using a linear layer applied to an adapted
representation $\tilde{\Phi}(x;w_{0})$:

$\begin{split}\tilde{\Phi}(x;w_{0})&=v_{0}\odot\phi_{0}(x;w_{0})+v_{1}\odot\phi_{1}(x;w_{0})+\dots\\\
y&=V\>\tilde{\Phi}(x;w_{0})+b\,.\end{split}$ (1)

We then train the coefficient vectors $v_{i}$ together with the weight matrix
$V,b$ of the linear layer.

This approximation is very aggressive because it only differs from linear
probing in the presence of the few $v_{i}$ coefficients. It is also an
attractive baseline because applying dropout to the adapted representation
$\tilde{\Phi}(x;w_{0})$ is a linear counterpart to applying dropout to
$\Phi(x;w)$ during ordinary fine-tuning. An experimental comparison of these
two approaches is provided later in this paper (Figure 8).

Using dropout in a nonlinear deep network introduces a very complicated noise
which, at high levels, can prevent the formation of internal representation
and stall the optimization process. Dropout in linear systems is a much milder
regularization that does not involve internal representations and is
analytically related to L2 regularization (Srivastava et al., 2014). Therefore
we can reliably use very high dropout levels during fine-tuning as long as it
can be well approximated by linear training, that is, as long as fine-tuning
merely exploits existing representations but does not need to create new ones.

Figure 2: Performance comparison between very large dropout, ensembles, and
weight averaging methods on four DomainBed tasks. The horizontal axis denotes
the i.i.d. performance and the vertical axis the o.o.d. performance. Baseline
results were obtained using plain fine-tuning with different hyperparameters
(1296$\times$⚫). Weight averaging results either average the model weights
collected every 300 iterations along each fine-tuning trajectory
(1296$\times$$\bigstar$) or the final model weights of all fine-tuning
trajectories (1$\times$black$\bigstar$) as in (Ramé et al., 2022b). Ensemble
results average instead the model outputs (1296$\times$▲ and
1$\times$black$\blacktriangle$). Finally, large dropout results were obtained
like the baseline results but using a 90% dropout rate on the penultimate
layer (1296$\times$◆).

##  4 Experiments

We perform most experiments using domain adaptation datasets that are part of
the DomainBed suite111https://github.com/facebookresearch/DomainBed (Gulrajani
& Lopez-Paz, 2020), namely pacs (Li et al., 2017), vlcs (Fang et al., 2013),
office home (Venkateswara et al., 2017), and terra incognita (Beery et al.,
2018). These datasets contain $9,991$ to $24,788$ examples and therefore are
substantially smaller than the pre-training dataset ImageNet with 1.2m
examples. We also use the larger domainnet dataset (Peng et al., 2019), 0.58m
examples, to show that ”linear connectivity” breaks down when the fine-tuning
dataset size is comparable with the pre-training dataset size and justifies
carrying out many more fine-tuning iterations.

Each of these datasets is divided into four sub-datasets that share the same
target label categories but follow a different distribution. For example, one
sub-dataset of pacs contains simple sketch images of ‘dog’ and ‘elephant’,
while another sub-dataset contains real photos of ‘dog’ and ‘elephant’. This
makes it possible to conveniently evaluate o.o.d. performance by fine-tuning
on three sub-datasets and testing on the fourth one.

Unless otherwise stated, all fine-tunings are performed on a ImageNet
pretrained ResNet50222https://pytorch.org/blog/how-to-train-state-of-the-art-
models-using-torchvision-latest-primitives/ neural network. Compared to the
original ResNet50 recipe of He et al. (2016), this ResNet50 is pre-trained
with substantial data augmentation techniques, such as trivialaugment (Müller
& Hutter, 2021), cutmix (Yun et al., 2019), and random erasings (Zhong et al.,
2020). These augmentations mimic the properties of large foundational models
that learn substantial diverse features using very large and diverse pre-
training data.

Figure 3: Effect of diverse dropout rates during fine-tuning. The best o.o.d.
performances are attained using rates around or above 90%.

###  4.1 Very large dropout yields better o.o.d. performance

Using these same datasets, Gulrajani & Lopez-Paz (2020) argue that simple
Empirical Risk Minimization (erm) works almost as well and often better than
carefully designed o.o.d. training methods, such as coral (Sun & Saenko,
2016), mldg (Li et al., 2018a), vrex (Krueger et al., 2021), and irm (Arjovsky
et al., 2019). However, Arpit et al. (2022); Cha et al. (2021); Ramé et al.
(2022b, a) find that ensembling and weight averaging methods consistently
outperform erm in the o.o.d. fine-tuning scenario.

Figure 2 therefore compares the performance of these state-of-the-art methods
with that of simply fine-tuning with a very large dropout333Code is available
at: https://github.com/TjuJianyu/verylarge_dropout.

  * •

Baseline results are obtained by fine-tuning our ResNet50 using sgd with 0.9
momentum for $10,000$ iterations. A 10% learning rate decay is applied at
$5000^{th}$ iterations. For each choice of three training sub-datasets, we
repeat three experiments for each combination of learning rate in
$\\{10^{-3},5.10^{-4}\\}$ and L2 weight decay in
$\\{10^{-4},5.10^{-5},10^{-5}\\}$. We measure the i.i.d. performance using 20%
examples held from the training data, and we measure o.o.d. performance on the
fourth sub-dataset. There are then $(2\times 3)^{4}=1296$ ways to average four
of these results, one for each of the four possible choices of the training
subsets, yielding $1296\times$⚫.

  * •

Dropout results are obtained using the same protocol but using a 90% dropout
rate on the penultimate layer representation, yielding $1296\times$◆.

  * •

Ensemble results are obtained in two ways, either using an ensemble of
checkpoints collected along each fine-tuning trajectory, yielding
$1296\times$▲, or using the ensemble of the final checkpoints collected along
all fine-tuning trajectories with different hyper-parameters, yielding a
single black$\blacktriangle$.

  * •

Weight averaging results approximate the corresponding ensembling results by
averaging the model weights instead of averaging the model outputs, yielding
$1296\times$$\bigstar$ and a single black$\bigstar$.

Figure 4: Comparison of various dropout rates on the larger domainnet dataset
(586K examples), whose size approaches the pretraining dataset size (ImageNet,
1.2M examples). The optimal dropout rate falls to about 50%, a value
comparable to the dropout rates traditionally used when training from scratch.
Figure 5: Comparison of dropout rates when training a ResNet50 network _from
scratch_ on the vlcs dataset. The optimal dropout rate falls to about zero.
Dropout rates greater than 50% negatively impact both the i.i.d. and the
o.o.d. performances.

As expected, both ensemble methods (Ueda & Nakano, 1996; Dietterich, 2000) and
their weight averaging approximation (Ramé et al., 2022b; Wortsman et al.,
2022a) improve both the i.i.d. and o.o.d. baseline performances. However,
fine-tuning with a very large dropout outperforms the o.o.d. performance of
both ensemble and weight averaging methods. There is even a large gap between
the worst dropout results and the best ensemble results for the office home
and pacs datasets. On the other hand, the i.i.d. performance of the large
dropout method lags behind that of ensembles. This shows that the o.o.d.
performance of the large dropout method is not a secondary effect of an i.i.d.
performance improvement.

Figure 6: Same as Figure 2 except for a $10\times$ larger training rate in the
last layer. Comparing these two figures shows that using a larger last layer
training rate yields small or zero incremental improvements over only using a
large dropout rate, but also that using a large dropout rate vastly improves
the o.o.d. performance of merely using the increased learning rate.

Figure 7: Incremental benefits achieved by constructing ensembles or by
averaging the weights of models fine-tuned with very large dropouts. The
baseline (1296$\times$⚫) and dropout (1296$\times$◆) results are the same as
those reported in Figure 2. In contrast, the ensemble
(1296$\times$$\blacktriangle$ and 1$\times$black$\blacktriangle$) and weight
averaging (1296$\times$$\bigstar$ and 1$\times$black$\bigstar$) results are
now obtained by averaging the output or the weights of models fine-tuned _with
large dropouts_. Ensemble and weight averaging techniques provide a marginal
o.o.d. performance improvement on vlcs or office home and a negligible o.o.d.
performance improvement on pacs or terra incognita.

####  4.1.1 Optimal o.o.d. dropout rate

To the best of our knowledge, such large dropout rates (90% and above) are
considered unsuitable for training a network from scratch and have not been
previously used for fine-tuning either. This section illustrates how the
optimal dropout rate can be very high in fine-tuning scenarios and falls to
small values when one gets closer to training the network from scratch.

Figure 3 compares various dropout rates on the four DomainBed tasks. The
optimal dropout rate for o.o.d. performance ranges from 90% to 95% for vlcs
and pacs (10k examples). It seems slightly smaller, about 90%, for the
slighlty larger datassets office home and terra incognita (15k to 25k
examples).

The larger the fine-tuning dataset, the more fine-tuning iterations we can
make without overfitting. When the fine-tuning dataset size approaches the
pre-training dataset size, the difference between fine-tuning and training
from scratch becomes less clear, the linear connectivity property disappears,
the linear approximation perspective on fine-tuning no longer holds, and the
optimal dropout rate falls sharply. Figure 4 illustrates this effect using the
larger domainnet dataset (Peng et al., 2019) that contains 586k examples
(almost half as big as ImageNet) and requires $30,000$ fine-tuning iterations.

Figure 5 shows the effect of various dropout rates when one trains a network
on the vlcs task from scratch, that is starting from a randomly initialized
network instead of a pre-trained one. The optimal dropout rate falls to about
zero. Dropout rates higher than 50% have a negative impact on both the i.i.d.
and the o.o.d. performance of the network. This suggests again that high
dropout rates make it difficult to create new features (a nonlinear
operation), but does not prevent leveraging existing features that were
possibly buried in the network inner layers (a linear operation).

###  4.2 Popular fine-tuning techniques do not substantially improve the
o.o.d. performance of large dropouts

Various fine-tuning techniques have been proposed to improve the fine-tuning
ability to leverage the representations learned by a pre-trained model, such
as using a larger learning rate on the last layer (Caron et al., 2020; Bardes
et al., 2021; Kumar et al., 2022) or, as discussed above, using weight
averaging and ensemble methods (Ramé et al., 2022b, a; Arpit et al., 2022). We
show in this section that using these techniques _in addition to very large
dropout rates_ do not yield much o.o.d. performance improvements over using
large dropout rates alone. Note that we still expect some incremental benefits
because both weight averaging and ensembles reduce the stochastic optimization
noise and accelerate training in general (Polyak & Juditsky, 1992).

####  4.2.1 Large learning rates for the last layer

Several authors routinely use a larger training rate on the last layer on the
intuition that fine-tuning a pre-trained deep network on a different target
task entails training a new last layer from scratch (Caron et al., 2020;
Bardes et al., 2021; Kumar et al., 2022).

Figure 6 is similar to Figure 2 but uses a $10\times$ larger training rate for
the last layer classifier. Comparing these two figures shows that using this
$10\times$ larger last layer training rate yields small or zero incremental
improvements over only using a large dropout rate, but also that using a large
dropout rate vastly improves the o.o.d. performance of merely using the
increased learning rate.

####  4.2.2 Ensemble and weight averaging

Figure 7 similarly explores the incremental benefits achieved by constructing
ensembles or by averaging the weights of models fine-tuned with very large
dropouts. The incremental improvements in o.o.d. performance achieved by these
methods, if any, are much smaller than the improvement achieved by large
dropout rates alone. Comparing Figures 2 and 7 also shows that in contrast,
fine-tuning with large dropout rates before computing ensembles or averaging
model weights brings large o.o.d. performance improvements over fine-tuning
without dropout.

Figure 8: Linear fine-tuning versus fine-tuning on the vlcs dataset. Linear
fine-tuning simplifies fine-tuning by only learning on the last three residual
block features, yields competitive o.o.d. performance as fine-tuning,
outperforms linear probing (which only optimizes the last-linear layer). In
terms of i.i.d. performance, linear fine-tuning legs behind fine-tuning,
perhaps because we follow Gulrajani & Lopez-Paz (2020) and determine the
optimal number of fine-tuning iterations according to the i.i.d. validation
accuracy.

###  4.3 Linear fine-tuning

Section 3.2 proposes a compact linear approximation (1) that is not only
useful to understand what fine-tuning does, but might also be practically
appealing. Whereas the previous sections discussed the use of large dropout
rates in the normal fine-tuning process, this section instead implements the
linear model (1) to fine-tune an ImageNet pre-trained ResNet50 on the
comparatively small vlcs dataset.

Although the ResNet50 architecture is the archetypal residual network, its
residual blocks are sometimes separated by a special layer that halves the
spatial resolution and doubles the number of channels. In order to remain as
close as possible to the formulation of our linear approximation (1), we only
use the three residual blocks that follow the last resolution-changing
operation. These three blocks operate on $7\times 7$ images with $2048$
channels. The final classification layer takes $2048$ inputs after average-
pooling the $7\times 7$ pixels of the last block output. In order to apply our
linear model (1), we average-pool the outputs of each of the last three
residual blocks, yielding three $2048$-dimensional vectors $\phi_{l}$,
$\phi_{l-1}$, and $\phi_{l-2}$. We compute the adapted representation
$\tilde{\Phi}$ by summing these three vectors after multiplying them element-
wise by three parameter vectors $v_{l}$, $v_{l-1}$, and $v_{l-1}$, and feed
into a linear layer,

$\displaystyle\tilde{\Phi}$
$\displaystyle=v_{l}\odot\phi_{l}+v_{l-1}\odot\phi_{l-1}+v_{l-2}\odot\phi_{l-2}$
$\displaystyle y$ $\displaystyle=V\tilde{\Phi}+b$

We train the parameter $v$, $V$, $b$ using the same protocol and hyper-
parameters as section 4.1.

We call this method _linear fine-tuning_ because it computes a linear
transformation of the residual block outputs, albeit one constrained by the
sparse two-layer structure of its parameters. Figure 8 compares the i.i.d. and
o.o.d. performance of regular fine-tuning and linear fine-tuning on the vlcs
dataset using various dropout rates. Linear fine-tuning yields competitive
o.o.d. performance but lags behind in i.i.d. performance, perhaps because we
follow Gulrajani & Lopez-Paz (2020) and determine the optimal number of fine-
tuning iterations according to the i.i.d. validation accuracy.

Figure 9: Comparison of the i.i.d. and o.o.d. performances obtained after
fine-tuning two pre-trained networks: ResNet #1 and ResNet #2. Compared with
ResNet50 #1 (He et al., 2016), ResNet #2 was pre-trained with the vast array
of data augmentation techniques. For each of these two pre-trained networks,
circles represent the results obtained with naive fine-tuning, and diamonds
(◆, ◆) represent the results obtained with advanced fine-tuning techniques,
that is, large dropout (90%), weight averaging, and increased last-layer
learning rate. The results obtained with the richer ResNet #2 dominate those
obtained with ResNet #1. In 3 out of 4 datasets, advanced fine-tuning
techniques on ResNet #1 (◆) barely match the performance of naive fine-tuning
on ResNet #2 (⚫).

###  4.4 Richer pre-training beats sophisticated fine-tuning

We have demonstrated that the very-large dropout method delivers consistently
better o.o.d. performance than computing ensembles or weight-averages of
models fine-tuned without dropout. However we also have argued that fine-
tuning does not create new representations but merely exploits the
representations already present in the pre-trained model. Therefore the final
o.o.d. performance of this fine-tuning process must strongly depend on the
quality and the diversity of the features present in the pre-trained network,
even if these features are not exploited by the pre-trained network but buried
in its hidden layers.

To validate this assertion, we compare the i.i.d. and o.o.d. performance
obtained by various methods applied to ResNet50 networks pre-trained using the
same ImageNet data but using different data augmentation schemes. As explained
in the first paragraphs of section 4, the results reported so far use a
network pre-trained using a broad array of data augmentation techniques,
termed ResNet #2. We now compare its fine-tuning properties with network
termed ResNet #1 pre-trained using the simpler protocol described in He et al.
(2016).

Figure 9 compares the i.i.d. and o.o.d. performances of both networks after
regular fine-tuning and after fine-tuning with all the available tricks, that
is, with dropout, with $10\times$ larger last layer learning rate, and after
averaging the weights of checkpoints collected along the fine-tuning
trajectory. This comparison makes clear that the quality of the pre-trained
representation matters more than the sophistication of the fine-tuning
techniques. This is consistent with the idea that fine-tuning only leverages
the existing features of the pre-trained network and does not create new ones.

##  5 Discussion

The o.o.d. performance of fine-tuning with very large dropout consistently
exceeds that achieved by popular techniques such as ensemble and by more
recent techniques such as weight averaging. Furthermore, ensemble and weight
averaging techniques only bring a small incremental improvement when applied
on top of fine-tuning with large dropout rates. This suggests that very large
dropout implements a key factor that favors o.o.d. performance, which we
believe is related to seeking features of type (a) among features of type (b)
as explained in the introduction.

Ensemble and weight-averaging techniques are well understood in the i.i.d.
scenario. Both techniques can be used for both training a network from scratch
and for fine-tuning. In contrast, it is practically useless to train a network
from scratch with such very large dropout rates. We argue that very large
dropout only works during fine-tuning because fine-tuning can in fact be well
approximated by a linear process that can leverage the existing or buried
features of a pre-trained network but cannot create new ones. In such a linear
process, using large dropout rates is akin to a form of L2 regularization,
expressing a richer set of features even if redundant.

This result also illustrates how the i.i.d. and o.o.d. scenarios can call for
very different techniques. It is well known that sparse representations can be
very helpful in the i.i.d. scenario, and it is increasingly clear that rich
representations are preferable in the o.o.d. scenario (Zhang et al., 2022;
Zhang & Bottou, 2023; Chen et al., 2023). There are no reasons to expect this
to be a lone exception.

For instance, training with stochastic gradient and momentum is not only
justified as a method to accelerate optimization, but also as a method that
can seek a flatter minimum and achieve better i.i.d. generalization (Keskar et
al., 2016; He et al., 2019). However the outcome might be different in an
o.o.d. scenario.

## Impact Statement

This work researches machine learning methods without any particular
application in mind. We do not anticipate any specific societal impact besides
the societal impact of machine learning in general.

## References

  * Andriushchenko et al. (2023) Andriushchenko, M., Varre, A. V., Pillaud-Vivien, L., and Flammarion, N.  Sgd with large step sizes learns sparse features.  In _International Conference on Machine Learning_ , pp. 903–925. PMLR, 2023.
  * Arjovsky et al. (2019) Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez-Paz, D.  Invariant risk minimization.  _arXiv preprint arXiv:1907.02893_ , 2019.
  * Arpit et al. (2022) Arpit, D., Wang, H., Zhou, Y., and Xiong, C.  Ensemble of averages: Improving model selection and boosting performance in domain generalization.  _Advances in Neural Information Processing Systems_ , 35:8265–8277, 2022.
  * Barbu et al. (2019) Barbu, A., Mayo, D., Alverio, J., Luo, W., Wang, C., Gutfreund, D., Tenenbaum, J., and Katz, B.  Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models.  _Advances in neural information processing systems_ , 32, 2019.
  * Bardes et al. (2021) Bardes, A., Ponce, J., and LeCun, Y.  Vicreg: Variance-invariance-covariance regularization for self-supervised learning.  _arXiv preprint arXiv:2105.04906_ , 2021.
  * Beery et al. (2018) Beery, S., Van Horn, G., and Perona, P.  Recognition in terra incognita.  In _Proceedings of the European conference on computer vision (ECCV)_ , pp. 456–473, 2018.
  * Bilen & Vedaldi (2017) Bilen, H. and Vedaldi, A.  Universal representations: The missing link between faces, text, planktons, and cat breeds.  _arXiv preprint arXiv:1701.07275_ , 2017.
  * Blanc et al. (2020) Blanc, G., Gupta, N., Valiant, G., and Valiant, P.  Implicit regularization for deep neural networks driven by an ornstein-uhlenbeck like process.  In _Conference on learning theory_ , pp. 483–513. PMLR, 2020.
  * Bommasani et al. (2021a) Bommasani, R., Hudson, D. A., Adeli, E., al. Russ Altman, Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N. S., Chen, A. S., Creel, K., Davis, J. Q., Demszky, D., Donahue, C., Doumbouya, M., Durmus, E., Ermon, S., Etchemendy, J., Ethayarajh, K., Fei-Fei, L., Finn, C., Gale, T., Gillespie, L., Goel, K., Goodman, N. D., Grossman, S., Guha, N., Hashimoto, T., Henderson, P., Hewitt, J., Ho, D. E., Hong, J., Hsu, K., Huang, J., Icard, T., Jain, S., Jurafsky, D., Kalluri, P., Karamcheti, S., Keeling, G., Khani, F., Khattab, O., Koh, P. W., Krass, M. S., Krishna, R., Kuditipudi, R., and et al.  On the opportunities and risks of foundation models.  _CoRR_ , abs/2108.07258, 2021a.  URL https://arxiv.org/abs/2108.07258.
  * Bommasani et al. (2021b) Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ , 2021b.
  * Bottou (2011) Bottou, L.  From machine learning to machine reasoning.  Technical report, arXiv:1102.1808, February 2011.
  * Caron et al. (2020) Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A.  Unsupervised learning of visual features by contrasting cluster assignments.  _Advances in neural information processing systems_ , 33:9912–9924, 2020.
  * Cha et al. (2021) Cha, J., Chun, S., Lee, K., Cho, H.-C., Park, S., Lee, Y., and Park, S.  Swad: Domain generalization by seeking flat minima.  _Advances in Neural Information Processing Systems_ , 34:22405–22418, 2021.
  * Chen et al. (2020) Chen, T., Kornblith, S., Norouzi, M., and Hinton, G.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pp. 1597–1607. PMLR, 2020.
  * Chen et al. (2023) Chen, Y., Huang, W., Zhou, K., Bian, Y., Han, B., and Cheng, J.  Towards understanding feature learning in out-of-distribution generalization.  _arXiv preprint arXiv:2304.11327_ , 2023.
  * Chizat et al. (2020) Chizat, L., Oyallon, E., and Bach, F.  On lazy training in differentiable programming, 2020.
  * Chowdhury et al. (2021) Chowdhury, A., Jiang, M., Chaudhuri, S., and Jermaine, C.  Few-shot image classification: Just use a library of pre-trained feature extractors and a simple classifier.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 9445–9454, 2021.
  * Collobert et al. (2011) Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., and Kuksa, P.  Natural language processing (almost) from scratch.  _Journal of Machine Learning Research_ , 12:2493–2537, Aug 2011.
  * Dietterich (2000) Dietterich, T. G.  Ensemble methods in machine learning.  In _International workshop on multiple classifier systems_ , pp. 1–15. Springer, 2000.
  * Dvornik et al. (2020) Dvornik, N., Schmid, C., and Mairal, J.  Selecting relevant features from a multi-domain representation for few-shot classification.  In _European Conference on Computer Vision_ , pp. 769–786. Springer, 2020.
  * Evci et al. (2022) Evci, U., Dumoulin, V., Larochelle, H., and Mozer, M. C.  Head2toe: Utilizing intermediate representations for better transfer learning.  In _International Conference on Machine Learning_ , pp. 6009–6033. PMLR, 2022.
  * Fang et al. (2013) Fang, C., Xu, Y., and Rockmore, D. N.  Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pp. 1657–1664, 2013.
  * Frankle et al. (2020) Frankle, J., Dziugaite, G. K., Roy, D., and Carbin, M.  Linear mode connectivity and the lottery ticket hypothesis.  In _International Conference on Machine Learning_ , pp. 3259–3269. PMLR, 2020.
  * Gontijo-Lopes et al. (2021) Gontijo-Lopes, R., Dauphin, Y., and Cubuk, E. D.  No one representation to rule them all: Overlapping features of training methods.  _arXiv preprint arXiv:2110.12899_ , 2021.
  * Gulrajani & Lopez-Paz (2020) Gulrajani, I. and Lopez-Paz, D.  In search of lost domain generalization.  _arXiv preprint arXiv:2007.01434_ , 2020.
  * He et al. (2019) He, F., Liu, T., and Tao, D.  Control batch size and learning rate to generalize well: Theoretical and empirical evidence.  In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R. (eds.), _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.  URL https://proceedings.neurips.cc/paper_files/paper/2019/file/dc6a70712a252123c40d2adba6a11d84-Paper.pdf.
  * He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * Hendrycks et al. (2021a) Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 8340–8349, 2021a.
  * Hendrycks et al. (2021b) Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D.  Natural adversarial examples.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 15262–15271, 2021b.
  * Huang et al. (2016) Huang, G., Sun, Y., Liu, Z., Sedra, D., and Weinberger, K. Q.  Deep networks with stochastic depth.  In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14_ , pp. 646–661. Springer, 2016.
  * Izmailov et al. (2018) Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. G.  Averaging weights leads to wider optima and better generalization.  _arXiv preprint arXiv:1803.05407_ , 2018.
  * Jaakkola & Haussler (1998) Jaakkola, T. and Haussler, D.  Exploiting generative models in discriminative classifiers.  _Advances in neural information processing systems_ , 11, 1998.
  * Keskar et al. (2016) Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P.  On large-batch training for deep learning: Generalization gap and sharp minima.  _arXiv preprint arXiv:1609.04836_ , 2016.
  * Krueger et al. (2021) Krueger, D., Caballero, E., Jacobsen, J.-H., Zhang, A., Binas, J., Zhang, D., Le Priol, R., and Courville, A.  Out-of-distribution generalization via risk extrapolation (rex).  In _International Conference on Machine Learning_ , pp. 5815–5826. PMLR, 2021.
  * Kumar et al. (2022) Kumar, A., Raghunathan, A., Jones, R., Ma, T., and Liang, P.  Fine-tuning can distort pretrained features and underperform out-of-distribution.  _arXiv preprint arXiv:2202.10054_ , 2022.
  * Li et al. (2017) Li, D., Yang, Y., Song, Y.-Z., and Hospedales, T. M.  Deeper, broader and artier domain generalization.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 5542–5550, 2017.
  * Li et al. (2018a) Li, D., Yang, Y., Song, Y.-Z., and Hospedales, T.  Learning to generalize: Meta-learning for domain generalization.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 32, 2018a.
  * Li et al. (2018b) Li, H., Xu, Z., Taylor, G., Studer, C., and Goldstein, T.  Visualizing the loss landscape of neural nets.  _Advances in neural information processing systems_ , 31, 2018b.
  * Li et al. (2021) Li, W.-H., Liu, X., and Bilen, H.  Universal representation learning from multiple domains for few-shot classification.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 9526–9535, 2021.
  * Li et al. (2022) Li, W.-H., Liu, X., and Bilen, H.  Universal representations: A unified look at multiple task and domain learning.  _arXiv preprint arXiv:2204.02744_ , 2022.
  * Maddox et al. (2021) Maddox, W., Tang, S., Moreno, P., Wilson, A. G., and Damianou, A.  Fast adaptation with linearized neural networks.  In _International Conference on Artificial Intelligence and Statistics_ , pp. 2737–2745. PMLR, 2021.
  * Mu et al. (2019) Mu, F., Liang, Y., and Li, Y.  Gradients as features for deep representation learning.  In _International Conference on Learning Representations_ , 2019.
  * Müller & Hutter (2021) Müller, S. G. and Hutter, F.  Trivialaugment: Tuning-free yet state-of-the-art data augmentation.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pp. 774–782, 2021.
  * Neyshabur et al. (2015) Neyshabur, B., Salakhutdinov, R., and Srebro, N.  Path-sgd: Path-normalized optimization in deep neural networks, 2015.
  * Peng et al. (2019) Peng, X., Bai, Q., Xia, X., Huang, Z., Saenko, K., and Wang, B.  Moment matching for multi-source domain adaptation.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pp. 1406–1415, 2019.
  * Polyak & Juditsky (1992) Polyak, B. T. and Juditsky, A. B.  Acceleration of stochastic approximation by averaging.  _SIAM Journal on Control and Optimization_ , 30(4):838–855, 1992.  doi: 10.1137/0330046.
  * Ramé et al. (2022a) Ramé, A., Ahuja, K., Zhang, J., Cord, M., Bottou, L., and Lopez-Paz, D.  Recycling diverse models for out-of-distribution generalization.  _arXiv preprint arXiv:2212.10445_ , 2022a.
  * Ramé et al. (2022b) Ramé, A., Kirchmeyer, M., Rahier, T., Rakotomamonjy, A., Gallinari, P., and Cord, M.  Diverse weight averaging for out-of-distribution generalization.  _Advances in Neural Information Processing Systems_ , 35:10821–10836, 2022b.
  * Recht et al. (2019) Recht, B., Roelofs, R., Schmidt, L., and Shankar, V.  Do imagenet classifiers generalize to imagenet?  In _International conference on machine learning_ , pp. 5389–5400. PMLR, 2019.
  * Sharif Razavian et al. (2014) Sharif Razavian, A., Azizpour, H., Sullivan, J., and Carlsson, S.  Cnn features off-the-shelf: an astounding baseline for recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , pp. 806–813, 2014.
  * Srivastava et al. (2014) Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R.  Dropout: a simple way to prevent neural networks from overfitting.  _The journal of machine learning research_ , 15(1):1929–1958, 2014.
  * Sun & Saenko (2016) Sun, B. and Saenko, K.  Deep coral: Correlation alignment for deep domain adaptation.  In _Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14_ , pp. 443–450. Springer, 2016.
  * Ueda & Nakano (1996) Ueda, N. and Nakano, R.  Generalization error of ensemble estimators.  In _Proceedings of International Conference on Neural Networks (ICNN’96)_ , volume 1, pp. 90–95 vol.1, 1996.  doi: 10.1109/ICNN.1996.548872.
  * Veit et al. (2016) Veit, A., Wilber, M. J., and Belongie, S.  Residual networks behave like ensembles of relatively shallow networks.  _Advances in neural information processing systems_ , 29, 2016.
  * Venkateswara et al. (2017) Venkateswara, H., Eusebio, J., Chakraborty, S., and Panchanathan, S.  Deep hashing network for unsupervised domain adaptation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 5018–5027, 2017.
  * Wang et al. (2019) Wang, H., Ge, S., Lipton, Z., and Xing, E. P.  Learning robust global representations by penalizing local predictive power.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Wang et al. (2022) Wang, H., Frank, E., Pfahringer, B., Mayo, M., and Holmes, G.  Cross-domain few-shot meta-learning using stacking.  _arXiv preprint arXiv:2205.05831_ , 2022.
  * Wortsman et al. (2022a) Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al.  Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.  In _International Conference on Machine Learning_ , pp. 23965–23998. PMLR, 2022a.
  * Wortsman et al. (2022b) Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al.  Robust fine-tuning of zero-shot models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 7959–7971, 2022b.
  * Yu et al. (2023) Yu, Y., Yang, C.-H. H., Kolehmainen, J., Shivakumar, P. G., Gu, Y., Ren, S. R. R., Luo, Q., Gourav, A., Chen, I.-F., Liu, Y.-C., et al.  Low-rank adaptation of large language model rescoring for parameter-efficient speech recognition.  In _2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_ , pp. 1–8. IEEE, 2023.
  * Yun et al. (2019) Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pp. 6023–6032, 2019.
  * Zhang & Bottou (2023) Zhang, J. and Bottou, L.  Learning useful representations for shifting tasks and distributions.  In _International Conference on Machine Learning_ , pp. 40830–40850. PMLR, 2023.
  * Zhang et al. (2022) Zhang, J., Lopez-Paz, D., and Bottou, L.  Rich feature construction for the optimization-generalization dilemma.  In _International Conference on Machine Learning_ , pp. 26397–26411. PMLR, 2022.
  * Zhong et al. (2020) Zhong, Z., Zheng, L., Kang, G., Li, S., and Yang, Y.  Random erasing data augmentation.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pp. 13001–13008, 2020.

Fine-tuning with Very Large Dropout

Supplementary Material

##  Appendix A Experiment details

###  A.1 Representation interpolation in Figure 1

The representation interpolation and weight interpolation experiments are
performed on clip444https://github.com/openai/CLIP pretrained ViT-B/32 model
and ImageNet fine-tuned ViT-B/32 model. Two fine-tuned ViT-B/32
models555https://github.com/mlfoundations/model-soups/ (red and blue in Figure
1) are adapted from Wortsman et al. (2022a). The red model (◆) is fine-tuned
with a higher learning rate ($10^{-4}$) and a longer training iteration (16
epochs), while the blue model (◆) is fine-tuned with a smaller learning rate
($10^{-5}$) and a shorter training iteration (11 epochs).

Denote the feature extractor weights and last-linear layer weights of the pre-
trained and fine-tuned models as: $\theta_{pretrain}$, $\omega_{pretrain}$ and
$\theta_{finetune}$, $\omega_{finetune}$, respectively, the penultimate layer
representations as $\Phi_{pretrain}$ and $\Phi_{finetune}$. For interpolation
coefficients $0\leq\alpha\leq 1$,

  * •

Weight interpolation interpolates all the model weights with

$\displaystyle\alpha\theta_{pretrain}+{(1-\alpha)}\theta_{finetune}$
$\displaystyle\alpha\omega_{pretraine}+{(1-\alpha)}\omega_{finetune}\,.$

  * •

Representation interpolation interpolates the penultimate layer
representations and the last-linear layer weights with

$\displaystyle\alpha\Phi_{pretrain}+{(1-\alpha)}\Phi_{finetune}$
$\displaystyle\alpha\omega_{pretraine}+{(1-\alpha)}\omega_{finetune}$

The o.o.d. accuracy is measured on five shifted distributions:
ImageNetA(Hendrycks et al., 2021b), ImageNet-scratch (Wang et al., 2019),
ImageNetR (Hendrycks et al., 2021a), ImageNetV2 (Recht et al., 2019),
ObjectNet (Barbu et al., 2019).

###  A.2 Fine-tuning DomainNet in Figure 4

The DomainNet o.o.d. fine-tuning experiment in Figure 4 follows the same
pipeline as other o.o.d. fine-tuning experiments on vlcs, pacs, Office Home,
and Terra Incognita datasets. Due to the larger size of DomainNet dataset, we
use larger learning rates $\\{3.10^{-3},5.10^{-3}\\}$ and a longer training
iteration $30,000$ with a 10% learning rate decay at $15,000$ and $25,000$.

###  A.3 Training from scratch in Figure 5

The vlcs scratch training experiment in Figure 5 also follows the same
pipeline as o.o.d. fine-tuning experiments. But it uses larger learning rates
$\\{5.10^{-3},10^{-2}\\}$ on a random initialized ResNet50 network (all
weights are trainable).

##  Appendix B Additional results

##  Appendix C Residual block dropout (stochastic depth) in o.o.d. fine-tuning

Huang et al. (2016) shows dropping a residual block at random with probability
$p$ helps train very deep residual neural networks from scratch. Dropping a
residual block, for example $f_{1}(x)$, in $\Phi(x)$ blinds the feature
$\phi_{1}(x)$ but also changes the input of successive layers:

$\displaystyle\Phi^{\prime}(x)$
$\displaystyle=\underbrace{\vphantom{f_{1}(x)}x}_{\mathclap{\phi_{0}(x)}}+\underbrace{}+\underbrace{f_{2}(x+}_{{\color[rgb]{0.1015625,0.5,0.734375}\definecolor[named]{pgfstrokecolor}{rgb}{0.1015625,0.5,0.734375}\phi^{\prime}_{2}(x)}}+\dots$

So an aggressive residual block dropout heavily disturbs the input of high
residual blocks, hurts the fine-tuning process. 10 showcase the effect of
redisual block dropout in o.o.d. fine-tuning.

Figure 10: Residual block dropout (stochastic depth) and penultimate layer
representation dropout comparison. Residual block dropout random drops a
residual block with probability $\in\\{0.1,0.2,0.3\\}$. Mild residual block
dropouts (0.1) provides some o.o.d. improvements (⚫), but lags behind dropout
(◆) on vlcs, office home, and terra incognita. Aggressive residual block
dropouts hurt fine-tuning (e.g. $\geq 0.3$ on office home and terra
incognita), because dropping a residual block change the input of successive
layers. Likewise the comparison of ensemble, weight averaging, and large last-
layer learning rate in section 4.2, a proper residual block dropout helps
fine-tuning in general, but is still a secondary factor compared with dropout.