# Thermal noise and mechanical loss of SiO2/Ta2O5 optical coatings at
cryogenic temperatures

John M. Robinson  JILA, NIST and University of Colorado, 440 UCB, Boulder,
Colorado 80309, USA  Eric Oelker  JILA, NIST and University of Colorado, 440
UCB, Boulder, Colorado 80309, USA  Current address : School of Physics and
Astronomy, SUPA, University of Glasgow, Glasgow G12 8QQ, UK  William R. Milner
JILA, NIST and University of Colorado, 440 UCB, Boulder, Colorado 80309, USA
Dhruv Kedar  JILA, NIST and University of Colorado, 440 UCB, Boulder, Colorado
80309, USA  Wei Zhang  JILA, NIST and University of Colorado, 440 UCB,
Boulder, Colorado 80309, USA  Current address : Jet Propulsion Laboratory,
4800 Oak Grove Drive Pasadena, CA 91109  Thomas Legero  Physikalisch-
Technische Bundesanstalt, Bundesallee 100, 38116 Braunschweig, Germany  Dan G
Matei  Physikalisch-Technische Bundesanstalt, Bundesallee 100, 38116
Braunschweig, Germany  Current address : Horia Hulubei National Institute of
Physics and Nuclear Engineering, Reactorului 30, 077125 Magurele, Romania
Sebastian Häfner  Physikalisch-Technische Bundesanstalt, Bundesallee 100,
38116 Braunschweig, Germany  Fritz Riehle  Physikalisch-Technische
Bundesanstalt, Bundesallee 100, 38116 Braunschweig, Germany  Uwe Sterr
Physikalisch-Technische Bundesanstalt, Bundesallee 100, 38116 Braunschweig,
Germany  Jun Ye  JILA, NIST and University of Colorado, 440 UCB, Boulder,
Colorado 80309, USA

(September 3, 2024)

###### Abstract

Mechanical loss of dielectric mirror coatings sets fundamental limits for both
gravitational wave detectors and cavity-stabilized optical local oscillators
for atomic clocks. Two approaches are used to determine the mechanical loss:
ringdown measurements of the coating quality factor and direct measurement of
the coating thermal noise. Here we report a systematic study of the mirror
thermal noise from room temperature to 4 K by operating reference cavities at
these temperatures. The directly measured thermal noise is used to extract the
corresponding mechanical loss for SiO2/Ta2O5 coatings, which are compared with
previously reported values.

††journal: optica

Low mechanical loss optical coatings play a critical role in optical reference
cavities [1, 2, 3, 4], gravitational wave detectors [5], and cavity
optomechanics [6]. Mechanical dissipation in the coating often dictates the
amplitude of length fluctuations. These fluctuations are referred to as
Brownian coating thermal noise (CTN), and can limit the precision of
interferometric measurements. Thus, there has been considerable experimental
effort to characterize the level of CTN present in current optical coatings.

The power spectral density (PSD) of thermal noise induced displacement
fluctuations is proportional to temperature. For this reason, several
gravitational wave detectors plan to operate at cryogenic temperatures,
including KAGRA at 20 K [7], LIGO Voyager at 124 K [8, 9], and the Einstein
Telescope at 10 K [10]. It is also clear that cryogenic optical cavities
provide the best laser stability for optical atomic clocks [11, 2]. Improving
the performance of these inferferometers relies upon characterizing and
improving the CTN at cryogenic temperatures.

There are two distinct methods for determining the Brownian CTN of an optical
coating. The first is referred to as the "mechanical ringdown approach". This
involves measuring the mechanical quality factor, Poisson ratio, and Young’s
modulus for each coating material. These mechanical properties for the coating
and substrate are then used to calculate the CTN for a given mirror. This
method of calculating the CTN may not account for all multilayer phenomena in
the coating. The second method, referred to as "direct measurements",
determines the CTN of the coating by measuring the frequency stability of
optical cavities. This approach has the challenge of extracting a coating loss
angle from a single measurement over a broad frequency range, which requires
input on other coating and substrate properties. Due to the challenges in each
approach, it is vital that both are undertaken as independent and
complementary research efforts. In this Letter, we present direct CTN
measurements of SiO2/Ta2O5 using the most stable cavities in operation today
at cryogenic temperatures up to room temperature.

Highly reflective optical coatings are fabricated by stacking alternating
layers of high and low refractive index material. The most common choices are
SiO2 and Ta2O5 for the low and high refractive index, respectively. Mechanical
ringdown measurements have been used to characterize both the coating
materials and multilayer coatings at cryogenic temperatures. Individual thin
films of SiO2 and Ta2O5 have shown a peak in the mechanical loss angle at
cryogenic temperatures [12, 13]. A multilayer coating of SiO2/Ti:Ta2O5 was
measured to have a loss peak, reaching approximately $1.0\times 10^{-3}$ at
$T=20$ K [14]. Ringdown measurements of an undoped SiO2/Ta2O5 coating showed a
largely temperature independent mechanical loss angle of
$4\leavevmode\nobreak\ \times\leavevmode\nobreak\ 10^{-4}$ at cryogenic
temperatures [15]. The possibility of increased mechanical loss at cryogenic
temperatures has motivated research in developing new optical coatings [12,
16, 17]. We are presenting our direct CTN measurements versus temperature to
complement the growing body of CTN research.

We perform direct measurement of CTN of two independent multilayer coatings
composed of SiO2/Ta2O5. By employing these coatings in ultrastable rigid
crystalline silicon (c-Si) Fabry-Perot cavities, we characterize the CTN
around 4 K, 16 K, and 124 K. These particular temperatures are informative for
future gravitational wave detectors. We also show an upper bound for a third
SiO2/Ta2O5 coating at room temperature, measured using an ultra-low expansion
(ULE) cavity.

We study the mechanical loss of the HR coatings by measuring the noise of
three independent Fabry-Perot cavities [2, 4, 18]. All three cavities use Ion-
Beam-Sputtered optical coatings made by Advanced Thin Films. The coatings for
the cryogenic cavities have 21 layers of SiO2 and 20 layers of Ta2O5, while
the room temperature system has 18 SiO2 and 19 Ta2O5 layers respectively. The
coatings were deposited at room temperature, and then annealed at a
temperature of approximately $480^{\circ}$C. The physical details of the
cavities are shown in table 1, where $L$ is the cavity length, $T$ is the
cavity temperature, $d$ is the coating thickness, and ROC1,2 is the radius of
curvature of each mirror, respectively. We calculate individual contributions
to thermal noise for each system according to the analysis in [19]. Here,
noise is quantified in terms of fractional frequency PSD, which is defined as
$S_{y}(f)=\frac{S_{\nu}(f)}{\nu^{2}}$, where $S_{\nu}(f)$ is the single-sided
PSD of frequency fluctuations and $\nu$ is the optical frequency.

Figure 1: Based on representative numbers from existing literature, we illustrate the major contributions to the thermal noise for three optical cavities at various temperatures and with length ranging from 6 to 40 cm. Fractional noise associated with coating and substrate scales inversely with the cavity length. Substrate thermo-elastic and spacer Brownian noise are below $10^{-36}$ Hz-1 for our cryogenic systems. Table 1:  Optical cavity and coating properties L (cm) | T (K) | d ($\mu$m) | Spacer/Sub. | ROC1/ROC2
---|---|---|---|---
6 | 4,16.2 | 9.2 | c-Si/c-Si | 1m/1m
21 | 123.3 | 9.2 | c-Si/c-Si |  $\infty$/1m
40 | 288 | 3.8 | ULE/FS |  $\infty$/1m

The contributions from Brownian motion of the cavity constituents and
substrate thermo-elastic noise are shown in Fig. 1. We use the material
properties in Table 2 and the equations in [19].

Table 2:  Material properties Material | Y (GPa) | $\sigma$ | $\phi$
---|---|---|---
c-Si ($\langle 111\rangle$) | 187.5 [20] | 0.23 [20] |  $10^{-7}$ [21]
SiO2-Ta2O5 | 91(7) [22] | 0.19 [22] | [This work]
Fused silica | 72 [23] | 0.17 [23] |  $1\times 10^{-6}$ [24]
ULE | 68 [25] | 0.17 [25] |  $1.6\times 10^{-5}$ [26]

Substrate thermo-elastic noise scales with the squared power of temperature
$T^{2}$ and the substrate coefficient of thermal expansion (CTE)
$\alpha_{sub}^{2}$, and is thus less than $10^{-36}$ Hz-1 at 1 Hz for our
cryogenic systems [19]. Coating thermo-optic noise is $4\leavevmode\nobreak\
\times 10^{-36}$ Hz-1 at 1 Hz for the room temperature system and is below
$10^{-36}$ Hz-1 at 1 Hz for the cryogenic systems. The relative magnitude of
the spacer and substrate contribution for the cryogenic silicon cavities
highlights the advantage of using these systems for studying the CTN, provided
one operates at a temperature where the silicon CTE is sufficiently small.

We first characterize the 124 K system, which has the lowest thermal noise
floor as seen in Fig. 1. This system is used as the local oscillator for the
JILA Sr1 optical lattice clock [27]. The laser frequency measurement comes
from recording the correction signal applied to keep the laser on resonance
with the clock transition. We use three independent data runs, each lasting at
least 20,000 seconds. The PSD is calculated for Fourier frequencies from
$3\times 10^{-4}$ to $10^{-2}$ Hz and is shown in Fig. 2 as the green data.
These measurements are restricted to low frequencies, limited by the finite
duty cycle of the clock. We also plot an estimate of the PSD for this system
at higher Fourier frequencies based on a cross-spectral density (CSD)
measurement from [11].

The PSD of the 124 K system measured with the optical clock constitutes a
direct measurement of its thermal noise level as the noise contribution from
the clock itself is negligible [11]. We extract the CTN level of the remaining
systems by measuring their heterodyne beat against the 124 K system using a
lambda-type zero dead time frequency counter, and subtracting it’s thermal
noise contribution in quadrature. Since the thermal noise floors are
uncorrelated, we subtract the direct measurement in quadrature from each
heterodyne beat to obtain the 4 K (blue), 16 K (red), and 288 K (orange) noise
PSD. The uncertainty of the fitted 124 K PSD is propagated when we perform the
subtraction.

Figure 2: The measured noise PSD at various temperatures (solid curves). The
dashed lines are the fits to $S_{y}(f)=a/f^{b}$.

Several authors have reported a frequency-dependent loss angle for SiO2/Ta2O5
coatings [28, 29]. We fit the measured spectra to the functional form of
$S_{y}(f)=a/f^{b}$ in order to allow for any potential frequency dependence. A
deviation from $b=1$ for the CTN indicates a frequency-dependent loss angle.
The fits to the measured spectra are shown as dashed lines in Fig. 2. We find
weak dependencies on frequency as summarized in Fig. 4.

To extract a mechanical loss angle from the measured noise spectrum, we
perform a characterization of all other noise sources in the reference cavity
systems. The 124 K system has been thoroughly characterized as described in
[30]. Since that work we have made several improvements to the setup,
including active stabilization of the transmitted optical power, active
temperature control of the outermost vacuum chamber, and improved optical
quality of the optics.

Figure 3: (A) Technical noise contributions for the 4 K system, along with the
sum of the technical noise (pink line) and the measured noise (black line).
(B) Intrinsic thermal noise for the cavity, along with the sum of technical
noise and thermal noise (green) and coating Brownian noise (dashed-red).

The technical noise budget of the 6 cm system is shown in Fig. 3A. We show
fits to the measured noise arising from temperature, Pound-Drever-Hall (PDH)
servo, and the PDH photodiode (PD). The dominant technical noise source is
residual amplitude modulation (RAM), shown as the red data. The sum of the
technical noise is well below the measured noise. Figure 3B shows the
intrinsic thermal noise of the cavity, including Brownian noise from the
spacer (dotted), substrate (dash-dot) and coating (dashed). The measured noise
is shown (black) along with the sum of technical noise and thermal noise
(green). The CTN is the dominant noise source from 5 mHz to 0.8 Hz.

The room temperature ULE cavity was previously used as the clock laser for the
JILA Sr lattice clock [18]. This system has routinely performed at a
fractional frequency stability of $1\times 10^{-16}$. Since we have not
developed a thorough noise budget for this system, we report only an upper
limit on the loss angle.

With the technical noise on the cryogenic cavities being sufficiently low, we
can extract a mechanical loss angle at each respective temperature. We assume
that the mechanical loss in the parallel and perpendicular directions are
identical ($\phi_{\parallel}=\phi_{\perp}$) [31]. For the ith mirror, the
expression for the fractional frequency PSD arising from coating Brownian
noise is then [32]

$\displaystyle S_{y}^{i}(f)=$
$\displaystyle\frac{2k_{B}Td}{\pi^{2}fL^{2}}\frac{1-\sigma_{sub}^{2}}{\omega_{i}Y_{sub}^{2}}\frac{\phi_{c}(f)}{Y_{c}(1-\sigma_{c}^{2})(1-\sigma_{sub}^{2})}\times$
(1)
$\displaystyle\left[Y_{c}^{2}(1+\sigma_{sub})^{2}(1-2\sigma_{sub})^{2}+Y_{sub}^{2}(1+\sigma_{c})^{2}(1-2\sigma_{c})\right]$

Here, "c" labels the coating and "sub" labels the substrate. $\sigma_{c(sub)}$
is the Poisson’s ratio, $Y_{c(sub)}$ is Young’s modulu, $d$ is the coating
thickness, $\omega_{i}$ is the $1/e^{2}$ beam radius at the $i^{th}$ mirror,
$\phi_{c(sub)}$ is the loss angle. Note that we are allowing for the coating
loss angle to have frequency dependence.

With the fits from Fig. 2 and Eq. 1, we derive $\phi_{c}(f)$ and plot the
results in Fig. 4. The results are

$\displaystyle\phi_{4\leavevmode\nobreak\ K}(f)$ $\displaystyle=(5.6\pm
0.9)\times 10^{-4}\leavevmode\nobreak\ \times f^{\leavevmode\nobreak\ -0.05\pm
0.01}$ (2) $\displaystyle\phi_{16\leavevmode\nobreak\ K}(f)$
$\displaystyle=(3.2\pm 0.3)\times 10^{-4}\leavevmode\nobreak\ \times
f^{\leavevmode\nobreak\ -0.11\pm 0.02}$ (3)
$\displaystyle\phi_{124\leavevmode\nobreak\ K}(f)$ $\displaystyle=(2.4\pm
0.3)\times 10^{-4}\leavevmode\nobreak\ \times f^{\leavevmode\nobreak\ 0.06\pm
0.02}$ (4)

Figure 4: Frequency-dependent mechanical loss angle for the SiO2/Ta2O5
coatings at different operating temperature. The shaded bands indicate the
$1\sigma$ uncertainty.

The shaded band indicates the 1$\sigma$ uncertainty. Interestingly, we find
that the frequency dependence, although rather weak, has the opposite sign at
4 K and 16 K compared with the 124 K result. The 4 and 16 K results use
mirrors from a different coating run than the 124 K cavity, and so this
difference could be coating-dependent.

Figure 5: Mechanical loss of the individual coating constituents (SiO2,
Ta2O5), and SiO2/Ta2O5 coatings measured by ringdown (orange diamonds, blue
triangles, green squares, purple triangles). Other direct measurements are
also included (pink square and black triangle). Results from this work from
direct CTN measurements are shown as the red circles.

In order to compare our results with those published in the literature, we
plot the coating loss angle as a function of temperature in Fig. 5. The orange
diamonds [12], green squares [33], and blue triangles [15] are ringdown
measurements that cannot distinguish between $\phi_{\perp}$ and
$\phi_{\parallel}$. The pink square [2], black triangle [34] and purple
diamond [35] are all from direct thermal noise measurements which assume
$\phi_{\perp}=\phi_{\parallel}$. For the sake of comparison, all values are
plotted at 1 Hz. We measured CTN at several temperatures near 4 K, and average
these values together. The x-axis error bar for that data point indicates the
spread in temperature explored, while the y-axis error bar is the standard
deviation of the extracted $\phi_{c}$. We do not see a strong temperature
dependence between 4 to 16 K.

We have presented direct measurements of the thermal noise for SiO2/Ta2O5 HR
coatings. These measurements are complementary to those attained by ringdown
spectroscopy. For the design of future gravitational wave detectors,
independent measurements of coating thermal noise are important. For
ultrastable reference cavities, the use GaAs/AlGaAs crystalline coatings at
cryogenic temperatures is important to explore. Other potentially low thermal
noise optical coatings could also be tested using cryogenic ultrastable
silicon cavities.

We thank T. Bothwell, C. Kennedy and L. Sonderhouse for technical
contributions. We acknowledge funding support from DARPA, NIST, NSF QLCI Award
2016244, PTB, and by the Deutsche Forschungsgemeinschaft (DFG, German Research
Foundation) under Germany’s Excellence Strategy—EXC-2123 Quantum Frontiers.

## References

  * [1] T. Kessler, C. Hagemann, C. Grebing, T. Legero, U. Sterr, F. Riehle, M. Martin, L. Chen, and J. Ye, Nature Photonics 6, 687 (2012).
  * [2] D. G. Matei, T. Legero, S. Häfner, C. Grebing, R. Weyrich, W. Zhang, L. Sonderhouse, J. M. Robinson, J. Ye, F. Riehle, and U. Sterr, Phys. Rev. Lett. 118, 263202 (2017).
  * [3] W. Zhang, J. M. Robinson, L. Sonderhouse, E. Oelker, C. Benko, J. L. Hall, T. Legero, D. G. Matei, F. Riehle, U. Sterr, and J. Ye, Phys. Rev. Lett. 119, 243601 (2017).
  * [4] J. M. Robinson, E. Oelker, W. R. Milner, W. Zhang, T. Legero, D. G. Matei, F. Riehle, U. Sterr, and J. Ye, Optica 6, 240 (2019).
  * [5] G. M. Harry, H. Armandula, E. Black, D. R. M. Crooks, G. Cagnoli, J. Hough, P. Murray, S. Reid, S. Rowan, P. Sneddon, M. M. Fejer, R. Route, and S. D. Penn, Appl. Opt. 45, 1569 (2006).
  * [6] J. Cripe, N. Aggarwal, R. Lanza, A. Libson, R. Singh, P. Heu, D. Follman, G. D. Cole, N. Mavalvala, and T. Corbitt, Nature 568, 364 (2019).
  * [7] K. collaboration, Classical and Quantum Gravity 36, 165008 (2019).
  * [8] B. Shapiro, R. X. Adhikari, O. Aguiar, E. Bonilla, D. Fan, L. Gan, I. Gomez, S. Khandelwal, B. Lantz, T. MacDonald, and D. Madden-Fong, Cryogenics 81, 83 (2017).
  * [9] B. P. Abbott and R. Abbott, Classical and Quantum Gravity 34, 044001 (2017).
  * [10] M. Abernathy, F. Acernese, P. Ajith, B. Allen, P. Amaro Seoane, N. Andersson, S. Aoudia, P. Astone, B. Krishnan, L. Barack _et al._ , ET conceptual design study document (2011).
  * [11] E. Oelker, R. B. Hutson, C. J. Kennedy, L. Sonderhouse, T. Bothwell, A. Goban, D. Kedar, C. Sanner, J. M. Robinson, G. E. Marti _et al._ , Nature Photonics 13, 714 (2019).
  * [12] K. Craig, J. Steinlechner, P. G. Murray, A. S. Bell, R. Birney, K. Haughian, J. Hough, I. MacLaren, S. Penn, S. Reid, R. Robie, S. Rowan, and I. W. Martin, Phys. Rev. Lett. 122, 231102 (2019).
  * [13] I. W. Martin, R. Bassiri, R. Nawrodt, M. M. Fejer, A. Gretarsson, E. Gustafson, G. Harry, J. Hough, I. MacLaren, S. Penn, S. Reid, R. Route, S. Rowan, C. Schwarz, P. Seidel, J. Scott, and A. L. Woodcraft, Classical and Quantum Gravity 27, 225020 (2010).
  * [14] M. Granata, K. Craig, G. Cagnoli, C. Carcy, W. Cunningham, J. Degallaix, R. Flaminio, D. Forest, M. Hart, J.-S. Hennig, J. Hough, I. MacLaren, I. W. Martin, C. Michel, N. Morgado, S. Otmani, L. Pinard, and S. Rowan, Opt. Lett. 38, 5268 (2013).
  * [15] K. Yamamoto, S. Miyoki, T. Uchiyama, H. Ishitsuka, M. Ohashi, K. Kuroda, T. Tomaru, N. Sato, T. Suzuki, T. Haruyama, A. Yamamoto, T. Shintomi, K. Numata, K. Waseda, K. Ito, and K. Watanabe, Phys. Rev. D 74, 022002 (2006).
  * [16] G. D. Cole, W. Zhang, M. J. Martin, J. Ye, and M. Aspelmeyer, Nature Photonics 7, 644 (2013).
  * [17] G. D. Cole, W. Zhang, B. J. Bjork, D. Follman, P. Heu, C. Deutsch, L. Sonderhouse, J. Robinson, C. Franz, A. Alexandrovski, M. Notcutt, O. H. Heckl, J. Ye, and M. Aspelmeyer, Optica 3, 647 (2016).
  * [18] M. D. Swallows, M. J. Martin, M. Bishof, C. Benko, Y. Lin, S. Blatt, A. M. Rey, and J. Ye, IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control 59, 416 (2012).
  * [19] M. J. Martin and J. Ye, “High-precision laser stabilization via optical cavities,” in _Optical Coatings and Thermal Noise in Precision Measurement,_ (Cambridge University Press, 2012).
  * [20] W. A. Brantley, Journal of Applied Physics 44, 534 (1973).
  * [21] R. Nawrodt, A. Zimmer, T. Koettig, C. Schwarz, D. Heinert, M. Hudl, R. Neubert, M. Thürk, S. Nietzsche, W. Vodel, P. Seidel, and A. Tünnermann, Journal of Physics: Conference Series 122, 012008 (2008).
  * [22] D. R. M. Crooks, G. Cagnoli, M. M. Fejer, G. Harry, J. Hough, B. T. Khuri-Yakub, S. Penn, R. Route, S. Rowan, P. H. Sneddon, I. O. Wygant, and G. G. Yaralioglu, Classical and Quantum Gravity 23, 4953 (2006).
  * [23] S. Musikant, _Optical materials: an introduction to selection and application_ (New York : Dekker, 1985).
  * [24] K. Numata, G. B. Bianc, N. Ohishi, A. Sekiya, S. Otsuka, K. Kawabe, M. Ando, and K. Tsubono, Physics Letters A 276, 37 (2000).
  * [25] P. Klocek, _Handbook of infrared optical materials_ (CRC Press, 1991).
  * [26] K. Numata, A. Kemery, and J. Camp, Phys. Rev. Lett. 93, 250602 (2004).
  * [27] T. Bothwell, D. Kedar, E. Oelker, J. M. Robinson, S. L. Bromley, W. L. Tew, J. Ye, and C. J. Kennedy, Metrologia 56, 065004 (2019).
  * [28] A. Amato, G. Cagnoli, M. Canepa, E. Coillet, J. Degallaix, V. Dolique, D. Forest, M. Granata, V. Martinez, C. Michel, L. Pinard, B. Sassolas, and J. Teillon, Journal of Physics: Conference Series 957, 012006 (2018).
  * [29] S. Gras and M. Evans, Phys. Rev. D 98, 122001 (2018).
  * [30] D. Matei, T. Legero, C. Grebing, S. Häfner, C. Lisdat, R. Weyrich, W. Zhang, L. Sonderhouse, J. Robinson, F. Riehle _et al._ , “A second generation of low thermal noise cryogenic silicon resonators,” in _Journal of Physics: Conference Series,_ , vol. 723 (IOP Publishing, 2016), p. 012031.
  * [31] Our measurements presented in this manuscript do not distinguish these two loss components. Without loss of generality we lump them together as a total loss angle. We also note a gradual shift away from the loss angle convention of $\phi_{\parallel}$ and $\phi_{\perp}$ in the LIGO community, and the loss angle can instead be defined in terms of bulk loss ($\phi_{B}$ and shear loss angles $\phi_{S}$ [36]. Approximately, the assumption of $\phi_{B}=\phi_{S}$ is equivalent to that of $\phi_{\parallel}=\phi_{\perp}$, without detailed treatment of light penetration into the coating layers.
  * [32] G. M. Harry, A. M. Gretarsson, P. R. Saulson, S. E. Kittelberger, S. D. Penn, W. J. Startin, S. Rowan, M. M. Fejer, D. R. M. Crooks, G. Cagnoli, J. Hough, and N. Nakagawa, Classical and Quantum Gravity 19, 897 (2002).
  * [33] R. Robie, “Characterisation of the mechanical properties of thin-film mirror coating materials for use in future interferometric gravitational wave detectors,” Ph.D. thesis, University of Glasgow (2018).
  * [34] T. Chalermsongsak, F. Seifert, E. D. Hall, K. Arai, E. K. Gustafson, and R. X. Adhikari, Metrologia 52, 17 (2014).
  * [35] S. Häfner, S. Herbers, S. Vogt, C. Lisdat, and U. Sterr, Optics Express 28, 16407 (2020).
  * [36] T. Hong, H. Yang, E. K. Gustafson, R. X. Adhikari, and Y. Chen, Phys. Rev. D 87, 082001 (2013).


# One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning

Arnav Chavan 1,2, Zhuang Liu3, Deepak Gupta2, Eric Xing1,4, Zhiqiang Shen∗1
1MBZUAI 2Transmute AI Lab 3FAIR, Meta 4CMU
<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Equal contribution. Project page: https://sites.google.com/view/generalized-
lora.

###### Abstract

We present Generalized LoRA (GLoRA), an advanced approach for universal
parameter-efficient fine-tuning tasks. Enhancing Low-Rank Adaptation (LoRA),
GLoRA employs a generalized prompt module to optimize pre-trained model
weights and adjust intermediate activations, providing more flexibility and
capability across diverse tasks and datasets. Moreover, GLoRA facilitates
efficient parameter adaptation by employing a scalable, modular, layer-wise
structure search that learns individual adapter of each layer. Originating
from a unified mathematical formulation, GLoRA exhibits strong transfer
learning, few-shot learning and domain generalization abilities, as it adjusts
to new tasks through additional dimensions on weights and activations.
Comprehensive experiments demonstrate that GLoRA outperforms all previous
methods in natural, specialized, and structured benchmarks, achieving superior
accuracy with fewer parameters and computations on various datasets.
Furthermore, our structural re-parameterization design ensures that GLoRA
incurs no extra inference cost, rendering it a practical solution for
resource-limited applications. Code is publicly available at: GitHub.

##  1 Introduction

The remarkable achievements of large-scale deep neural networks in recent
years have revolutionized the field of artificial intelligence, demonstrating
unprecedented performance across various tasks and domains. These highly
complex models, often with millions or even billions of parameters, have
demonstrated remarkable capabilities in areas such as computer vision [1],
natural language understanding [2], and speech recognition [3]. Typically,
these colossal models are pre-trained on general and large-scale datasets,
such as ImageNet [4], and are subsequently adapted to downstream target
scenarios through fine-tuning or transfer learning. Given the immense
computational resources required by large pre-trained architectures, many
parameter-efficient fine-tuning (PEFT) methods [5, 6, 7, 8, 9] have been
proposed. For instance, Low-Rank Adaptation (LoRA) [5] aims to reduce the
number of trainable parameters by exclusively learning pairs of rank-
decomposition matrices whilst keeping the original model parameter static.
Adapter [10] implements bottleneck adapter modules and incorporates a modest
number of task-specific parameters into a fixed pre-trained model. Similarly,
Visual Prompt Tuning (VPT) [7] introduces a minimal number of learnable
parameters to the input of the Transformer, leaving the entire backbone frozen
during fine-tuning.

However, distinct downstream datasets often possess unique characteristics,
such as natural, specialized, and structured data, which differ significantly
in distribution and composition. A static fine-tuning strategy may not
sufficiently account for these disparities, thereby hindering its capacity to
adapt to diverse datasets. To rectify this, we propose a flexible, parameter-
efficient fine-tuning scheme in this work to manage the variations of multiple
downstream datasets within a consolidated formulation. Our approach presents a
generalized version of LoRA from a unified parameter-efficient fine-tuning
perspective, amplifying LoRA’s capability, scalability, and adaptability by
rescaling and shifting intermediate activations, in conjunction with
implementing a structural re-parameterization design, etc. It is challenging
to devise a unified method that integrates all adjustable dimensions and
possibilities when tuning a pre-trained network, especially in the case of
transformer architectures, while our proposed approach presents a practicable
solution to navigate this complexity.

Our approach exhibits the following advantages: (1) It concurrently takes into
account multiple dimensions to enhance capability and flexibility during fine-
tuning, encompassing weights, features, and input tokens. (2) It confers a
significant edge over prior integration-based NOAH [8] and partial fine-tuning
method PFT [6] in that, subsequent to an evolutionary search procedure for the
ideal subnet or configuration, there is no requirement to retrain the subnet,
in contrast to NOAH and PFT. Consequently, our results are derived directly
from the supernet’s model weights without necessitating additional training.
(3) It conducts an implicit search devoid of any manual hyperparameter tuning,
thus justifying the increased training time. (4) It incurs no additional
inference cost thanks to our structural re-parameterization architecture,
whereby the extra fine-tuning parameters will be fused to the proximate
projection weights post-training.

We conduct comprehensive experiments on VTAB-1K [11], ImageNet [4], and its
variants [12, 13, 14, 15]. The VTAB-1K dataset comprises 19 heterogeneous
vision datasets, enveloping a broad spectrum of visual domains that include
natural objects and scenes, textures and shapes, satellite imagery, among
others. GLoRA surpasses all previous state-of-the-art PEFT methods by a
substantial margin in terms of average accuracy. Additionally, we evaluate the
model’s few-shot learning capacity on five fine-grained visual recognition
datasets, akin to prior works [8, 7], along with its ability for domain
generalization and robustness on ImageNet-V2 [12], ImageNet-Sketch [13],
ImageNet-A [14], and ImageNet-R [15] datasets. GLoRA significantly outperforms
previous methods across all these benchmarks, without incurring any extra
computational overhead during the inference phase.

Our contributions:

  * •

We propose Generalized LoRA (GLoRA), a novel parameter-efficient fine-tuning
framework. GLoRA enhances the low-rank adaptation approach with a more
generalized prompt module design per layer, offering enhanced capability and
flexibility in finetuning.

  * •

We present a unified perspective of formulation for the parameter-efficient
fine-tuning problem through investigating various previous approaches to this
task. Our proposed GLoRA framework is constructed based on this formulation
that can achieve all fine-tuning paradigms from a single formulation, i.e., a
One-for-All 111One-for-All represents that one formulation can be transformed
into various shapes of PEFT paradigms. fine-tuning architecture.

  * •

To evaluate the effectiveness of our proposed approach, we conduct
comprehensive experiments on downstream fine-tuning, few-shot learning, and
domain generalization and robustness using diverse datasets. Our experimental
results demonstrate that GLoRA outperforms all previous methods on these
benchmarks while requiring only a small number of extra parameters and no
additional inference cost.

##  2 GLoRA

In this section, we start from providing a mathematical overview of existing
state-of-the-art PEFT methods and discuss the advantages and disadvantages for
them. Then, we introduce a unified formulation of integrating all existing
SOTA PEFT methods and elaborate our proposed generalized LoRA in detail
following this unified formulation perspective. After that, a structural re-
parameterization design is presented to show the inference efficiency without
additional cost. An evolutionary search for optimal layer-wise configurations
is also introduced to achieve the goal of generalized LoRA. We further give
the theoretical analysis and discussions on the higher capability of the
proposed method.

###  2.1 Previous Solutions with Limitations

Visual Prompt Tuning [7]: VPT introduces a small amount of task-specific
learnable parameters into the input space while freezing the entire pre-
trained Transformer backbone during downstream fine-tuning. It proposes two
strategies: VPT-Shallow, i.e., only input space has the trainable prompt:

$\begin{array}[]{l}{\left[\mathbf{x}_{1},\mathbf{Z}_{1},\mathbf{E}_{1}\right]}=L_{1}\left(\left[\mathbf{x}_{0},\mathbf{P},\mathbf{E}_{0}\right]\right)\\\
{\left[\mathbf{x}_{i},\mathbf{Z}_{i},\mathbf{E}_{i}\right]}=L_{i}\left(\left[\mathbf{x}_{i-1},\mathbf{Z}_{i-1},\mathbf{E}_{i-1}\right]\right)\end{array}$
(1)

where $P$ is prompt, it is trainable and others are frozen. $\mathbf{x}$ is
the [CLS] token, $\mathbf{E}$ are the image patches. Prompts use <1% trainable
parameters as compared to the original model.

VPT-Deep, i.e., every layer has the trainable prompt. The formulation is:

$\begin{array}[]{l}{\left[\mathbf{x}_{i},\ldots,\mathbf{E}_{i}\right]}=L_{i}\left(\left[\mathbf{x}_{i-1},\mathbf{P}_{i-1},\mathbf{E}_{i-1}\right]\right)\end{array}$
(2)

VTP-Deep outperforms full fine-tuning on majority of vision tasks and also has
better accuracy in low data regime. VTP observes that when prompt append is
larger than prompt add, adding prompts to the shallow layers is better than
deep. However, VPT increases cost in the inference stage.

AdaptFormer [16]: AdaptFormer introduces a parallel learnable branch of two
linear layers and ReLU over the MLP block and learns only this path while
freezing other parts.

$\tilde{x}_{\ell}=\mathbf{ReLU}\left(\mathrm{LN}\left(x_{\ell}^{\prime}\right)\cdot\mathbf{W}_{\mathrm{down}}\right)\cdot\mathbf{W}_{\mathrm{up}}$
(3)
$x_{\ell}=\mathbf{MLP}\left(\mathrm{LN}\left(x_{\ell}^{\prime}\right)\right)+s\cdot\tilde{x}_{\ell}+x_{\ell}^{\prime}$
(4)

where $x_{\ell}^{\prime}$ are the tokens after MHSA at the $\ell$-th layer.
$\mathbf{W}_{\mathrm{down}}$ and $\mathbf{W}_{\mathrm{up}}$ are a down-
projection layer and an up-projection layer from the parallel branch,
respectively. $s$ is a scale factor.

LoRA [5]: LoRA proposes to freeze the pre-trained model weights and injects
trainable low-rank decomposition matrices into each layer. It learns only the
residual from pre-trained weight. SOTA performance is achieved if comparing to
prompt learning, adapters, etc., on the GPT-2 model family. Assuming
$\mathbf{W}_{0}$, $\mathbf{b}_{0}$, $x$ are pre-trained weights, bias and
input, let $f$ be a linear layer, thus $f(x)=\mathbf{W}_{0}x+\mathbf{b}_{0}$.
During fine-tuning, $\mathbf{W}_{0}$ and $\mathbf{b}_{0}$ are frozen, the
learning process can be:

$\begin{array}[]{ll}f(x)=\mathbf{W}_{0}x+\Delta\mathbf{W}x+\mathbf{b}_{0}=\mathbf{W}_{\mathrm{LoRA}}x+\mathbf{b}_{0}\end{array}$
(5)

where $\Delta\mathbf{W}$ is the low-rank decomposition weights that are
learnable.

Scaling & Shifting Features (SSF) [17]: SSF modul scales and shifts features
after every MLP, MHSA, Layernorm module during training, and performs re-
parameterization during inference as it is a linear structure.

$\bm{y}=\bm{\gamma}\odot x+\bm{\beta}$ (6)

where $\bm{y}$ is the output features. $\bm{\gamma}$ and $\bm{\beta}$ are the
scale and shift factors, $\odot$ is the dot product. This method has no
increase in inference but the capability is limited to feature adaptation.

FacT [18]: FacT proposes to use a tensorization-decomposition method to store
the additional weight, the weights of the model are tensorized into a single
3D tensor, and their additions are then decomposed into lightweight factors.
In fine-tuning, only the factors will be updated and stored.

$\begin{array}[]{ll}&f(x)=\mathbf{W}_{0}x+\mathbf{b}_{0}+\mathbf{U}\Sigma\mathbf{V}x=\left(\mathbf{W}_{0}+\mathbf{U}\Sigma\mathbf{V}\right)x+\mathbf{b}_{0}\end{array}$
(7)

where $\Delta\mathbf{W}$ in LoRA is decomposed into $\mathbf{U}$, $\mathbf{V}$
and $\bm{\Sigma}$. This is Tensor-Train in FacT.

$\begin{array}[]{ll}f(x)=\mathbf{W}_{0}x+\mathbf{b}_{0}+\mathbf{U}\mathbf{C}\mathbf{P}\mathbf{V}x=\left(\mathbf{W}_{0}+\mathbf{U}\mathbf{C}\mathbf{P}\mathbf{V}\right)x+\mathbf{b}_{0}\end{array}$
(8)

where $\Delta\mathbf{W}$ in LoRA is decomposed into $\mathbf{U}$,
$\mathbf{C}$, $\mathbf{P}$ and $\mathbf{V}$. This is Tucker in FacT.

RepAdapter [9]: RepAdapter inserts lightweight networks into the pre-trained
models, and the additional parameters will be re-parameterized to the nearby
projection weights after training. Adding sequential (not parallel) adapter to
both MHA and MLP, adapter is linear thus can be re-parameterized, and has two
layers: downsampling dense FC layer to downsample inputs; upsampling
downsampled features that are divided into group, and each group has an
upsampling layer. The group of upsampling layers can be merged into a single
sparse upsampling layer and can be re-parameterized directly into the original
MLP/MHSA. The formulation can be:

$\begin{array}[]{ll}f(x)&=\mathbf{W}_{0}\left(x+\mathbf{W}_{u}\left(\mathbf{W}_{d}x+\mathbf{b}_{d}\right)+\mathbf{b}_{u}\right)+\mathbf{b}_{0}\\\
&=\left(\mathbf{W}_{0}+\mathbf{W}_{0}\mathbf{W}_{u}\mathbf{W}_{d}\right)x+\mathbf{W}_{0}\mathbf{W}_{u}\mathbf{b}_{d}+\mathbf{W}_{0}\mathbf{b}_{u}+\mathbf{b}_{0}\end{array}$
(9)

where $\mathbf{W}_{u}$, $\mathbf{W}_{d}$, $\mathbf{b}_{u}$ and
$\mathbf{b}_{b}$ are learnable weights and biases, respectively.

Limitations: In general, many existing PETL methods such as (VPT, Adapter)
increase the inference time since the proposed structure cannot be re-
parameterized. Direct prompt tuning is also hard to design as it brings in
computational burden and requires hyper-parameter tuning i.e., how and where
to place prompts. LoRA can be re-parameterized at inference but it doesn’t
scale up for larger matrices and the adaptation ability is constrained on
weight space. SSF / Repadaptor cannot learn the wieght change i.e.,
$\Delta\mathbf{W}$ in weight space, whereas LoRA / FacT cannot efficiently
learn the scaling and shifting of feature change i.e., $\Delta\mathbf{H}$ in
features space. Both feature and weight space need flexibility while
performing transfer learning from a large model. Our proposed idea in this
work attempts at: $\Delta\mathbf{W}$ tuning, $\Delta\mathbf{H}$ tuning, along
with $\mathbf{W}$ and $\mathbf{H}$ scale and shift learning.

###  2.2 A Unified Formulation of One-for-All

Figure 1: Schematic representation of a linear layer adapted with GLoRA.

For model fine-tuning, we propose a unified formulation that encompasses all
tunable dimensions, including but not limited to the weight space and feature
space. Additionally, we adopt a re-parameterization strategy to incorporate
auxiliary parameters into the adjacent projection weights during the inference
stage. Broadly speaking, our method serves as a superset of all prior
solutions, i.e., one-for-all mechanism. By setting different support tensors
to zero, our GLoRA can be reduced to any of these predecessor methods. Unlike
NOAH [8], our architecture can be succinctly articulated as a unified
mathematical equation. The consolidated formulation to represent all tunable
spaces can be represented as follows:

$f(x)=\left(\mathbf{W}_{0}+\mathbf{W}_{0}\mathbf{A}+\mathbf{B}\right)x+\mathbf{C}\mathbf{W}_{0}+\mathbf{D}\mathbf{b}_{0}+\mathbf{E}+\mathbf{b}_{0}$
(10)

where $\mathbf{A}$, $\mathbf{B}$, $\mathbf{C}$, $\mathbf{D}$, $\mathbf{E}$ are
the trainable support tensors for downstream tasks in our GLoRA,
$\mathbf{W}_{0}$ and $\mathbf{b}_{0}$ are frozen during whole fine-tuning.
$\mathbf{A}$ is utilized to scale the weight. $\mathbf{B}$ has the role to
scale the input and shift the weight. $\mathbf{C}$ is the layer-wise prompt
serving a similar function of VPT-Deep, $\mathbf{D}$ and $\mathbf{E}$ are used
to scale and shift the bias, respectively. A detailed illustration is shown in
Figure 1.

###  2.3 Prompt Modules

In this subsection, we delineate the methodology for designing layer-wise
adaptors or prompt modules for $\mathbf{A}$, $\mathbf{B}$, $\mathbf{C}$,
$\mathbf{D}$, $\mathbf{E}$. In a broad sense, these can take the form of
scalars, vectors, low-rank decompositions, or none. Based on the role of these
trainable support tensors, they can be classified as follows:

$\begin{array}[]{ll}\mathbf{A}=\\{\mathrm{LoRA},\mathrm{vector},\mathrm{scalar},\mathrm{none}\\}\\\
\mathbf{B}=\\{\mathrm{LoRA},\mathrm{vector},\mathrm{scalar},\mathrm{none}\\}\\\
\mathbf{C}=\\{\mathrm{LoRA},\mathrm{vector},\mathrm{none}\\}\\\
\mathbf{D}=\\{\mathrm{vector},\mathrm{scalar},\mathrm{none}\\}\\\
\mathbf{E}=\\{\mathrm{vector},\mathrm{scalar},\mathrm{none}\\}\end{array}$
(11)

where none indicates zero, if all the trainable support tensors are zero, the
model will be degraded to the original formulation and training recipe. In
particular, suppose $\mathbf{W}_{0}\in\mathbb{R}^{d\times d}$, we define
$\mathbf{A}_{d}\in\mathbb{R}^{d\times r}$,
$\mathbf{A}_{u}\in\mathbb{R}^{r\times d}$,
$\mathbf{B}_{d}\in\mathbb{R}^{d\times r}$,
$\mathbf{B}_{u}\in\mathbb{R}^{r\times d}$,
$\mathbf{C}_{d}\in\mathbb{R}^{d\times r}$,
$\mathbf{C}_{u}\in\mathbb{R}^{r\times 1}$, $\mathbf{D}\in\mathbb{R}^{d\times
1}$ and $\mathbf{E}\in\mathbb{R}^{d\times 1}$. Depending upon the current
subnet configuration, in case of LoRA with rank $r_{1}<r$,
$\mathbf{A}_{d}^{r_{1}}\in\mathbb{R}^{d\times r_{1}}$,
$\mathbf{A}_{u}^{r_{1}}\in\mathbb{R}^{r_{1}\times d}$ is indexed from
$\mathbf{A}_{d}$ and $\mathbf{A}_{u}$ respectively; and
$\mathbf{A}=\mathbf{A}_{d}^{r_{1}}\times\mathbf{A}_{u}^{r_{1}}$ is used as the
final tensor, in case of vector $\mathbf{A}\in\mathbb{R}^{d\times 1}$ is
indexed from $\mathbf{A}_{d}$ and in case of scalar
$\mathbf{A}\in\mathbb{R}^{1\times 1}$ is indexed from $\mathbf{A}_{d}$. A
similar strategy is followed for all other support tensors depending upon the
current sampled configuration in the subnet. This weight entanglement strategy
helps to increase the search space without increasing the number of parameters
substantially and also shows faster convergence due to weight sharing in
different subnets.

###  2.4 Structural Re-parameterization Design and Inference Efficiency
Analysis

The fundamental aspect enabling re-parameterization is the elimination of non-
linearity amidst adjacent transformations, thereby permitting the absorption
of supplementary parameters into the preceding ones. As mentioned in
RepAdapter [9], the removal of such non-linear layers does not detrimentally
impact the performance of the networks. The precise concept of GLoRA re-
parameterization is explicated as follows:

$f(x)=\mathbf{W}_{\mathrm{uni}}x+\mathbf{b}_{\mathrm{uni}}$ (12)

where $\mathbf{W}_{\mathrm{uni}}$ and $\mathbf{b}_{\mathrm{uni}}$ are our
final unified trained weight and bias in GLoRA. They are re-parameterized
according to Eq 10:

$\mathbf{W}_{\mathrm{uni}}=\mathbf{W}_{0}+\mathbf{W}_{0}\mathbf{A}+\mathbf{B}$
(13)
$\mathbf{b}_{\mathrm{uni}}=\mathbf{C}\mathbf{W}_{0}+\mathbf{D}\mathbf{b}_{0}+\mathbf{E}+\mathbf{b}_{0}$
(14)

As a result, the re-parameterization strategy we employ, which integrates
learnable parameters into the neighboring projection weights, can be
advantageous as it incurs no additional computational cost during the
inference phase.

###  2.5 Evolutionary Search for Optimal Layer-wise Configurations

Our design for a unified adaptor is implemented on a per-layer basis, thus
allowing for heterogeneity across different layers. To identify the optimal
configuration for each layer, we employ the evolutionary search method [8, 6],
which offers a balance of efficiency and effectiveness. Although the training
time may increase due to this search process, it is important to note that
existing work [8] necessitate an extensive hyperparameter search (such as low-
rank in LoRA and FacT, as well as position and size of adapter modules in
Adapter [10], dimension and structure configuration in RepAdapter [9], among
others). In particular, we make use of a weight sharing strategy where a
single large matrix is defined for each support tensor, and depending upon the
component (LoRA, vector, scalar, or none), a submatrix is indexed and applied
for the current training iteration. This allows better parameter efficiency
since the maximal weight sharing is done in the subnets. Contrarily, our
unified prompt module design conducts an implicit search that eliminates the
need for manual hyperparameter tuning. Therefore, any augmentation in training
time is reasonable and well-justified.

###  2.6 GLoRA with Higher Capacity

Model capacity refers to the capability of a model to approximate a diverse
range of functions. A method for regulating the capacity of a learning
algorithm involves selecting an appropriate hypothesis space, essentially a
set of functions that the learning algorithm is permitted to consider as
potential solutions. The Vapnik-Chervonenkis Dimension (VC Dimension) [19], a
measure of the capacity and complexity of a statistical algorithm, can be
leveraged to provide a formal evidence for this assertion.

###### Theorem 1

Suppose $\mathbf{d}_{\mathrm{vc}}(\mathcal{H})$ is the VC dimension of the
hypothesis $\mathcal{H}$. If
$\mathcal{H}_{\mathrm{i}}\subseteq\mathcal{H}_{\mathrm{uni}}$,

$\mathbf{d}_{\mathrm{vc}}(\mathcal{H}_{\mathrm{uni}})-\mathbf{d}_{\mathrm{vc}}(\mathcal{H}_{\mathrm{i}})\geq\epsilon\
\ \ \ \ s.t.\ \ \ \epsilon\geq 0$

The validity of this theorem stems from the inherent property of our problem
context, where the hypothesis space $\mathcal{H}_{\mathrm{i}}$ is a subset of
$\mathcal{H}_{\mathrm{uni}}$ in our context. $\mathcal{H}_{\mathrm{uni}}$
encompasses all possible shattered scenarios of $\mathcal{H}_{\mathrm{i}}$.
For the extreme case where the VC dimension
$\mathbf{d}_{\mathrm{vc}}(\mathcal{H}_{\mathrm{o}})$
($\mathcal{H}_{\mathrm{o}}$ is the difference set of
$\mathcal{H}_{\mathrm{uni}}$ and $\mathcal{H}_{\mathrm{i}}$) is 0, the error
$\epsilon$ will be zero. As per learning theory, a higher VC dimension implies
greater model flexibility and capability of our approach.

##  3 Experiments

Datasets. We thoroughly evaluate GLoRA on VTAB-1K [11] benchmark at different
parameter counts. VTAB-1K comprises of 19 image classification tasks. Tasks
are clustered into three domains: i) Natural images ii) Specialized tasks
consisting of remote sensing and medical datasets; and iii) Structured tasks
focusing on scene structure understanding, like depth prediction and
orientation prediction, among others. To test the few-shot fine-tuning
performance, we evaluate GLoRA on five fine-grained visual recognition few-
shot datasets: Food101 [20], OxfordFlowers102 [21], StandfordCars [22],
OxfordPets [23], and FGVCAircraft [24]. Following previous work [18], we
evaluate 1, 2, 4, 8, and 16 shot settings. Finally, to show the domain
generalization capabilities of GLoRA, we train GLoRA on ImageNet [25] under a
16-shot setting and test on ImageNetV2 [12], ImageNet-Sketch [13], ImageNet-A
[14], and ImageNet-R [15].

Network Architecture and Implementation Details. For all our experiments, we
employ ViT-B [1] pre-trained on ImageNet-21K as the base model. ViT-B is a
12-block deep transformer model with Multi Head Self-Attention (MHSA) and
Multi-Layer Perceptron (MLP) modules present in each block. MHSA module has
two linear layers; Q-K-V linear transformation layer and Output Projection
layer, similarly, the MLP module has two linear layers. Thus each block has a
total of four linear layers. For optimal flexibility, we implement GLoRA on
all 4 layers across the 12 blocks, accounting for 48 GLoRA-adapted linear
layers in the final model. It is important to note that GLoRA can be added to
any linear layer in any network architecture.

Our method requires minimal hyperparameter tuning (except optimizer
hyperparameters) due to the inherent search mechanism. Based on the previous
works [26, 16, 8], we use AdamW [27] optimizer in all our experiments, weight
decay is fixed at 1e-4 and the learning rate is either 1e-4 for larger
datasets or 5e-4 for smaller datasets. Dataset-specific learning rates are
provided in the supplementary material. We train our supernets for 500 epochs
with a fixed batch size of 64 and a cosine learning rate scheduler. It is
important to note that irrespective of the dataset, this exact policy works
well in all settings. Also, we don’t re-train the weights after the search
process and directly inherit the supernet weights for evaluation. Thus all
supernets of various datasets are only trained for 500 epochs in total. The
comparison of search / training time cost with other search-based methods is
provided in the appendix.

###  3.1 VTAB-1K Dataset

We train three different GLoRA supernet configurations to vary the number of
trainable parameters. The difference in these is only the LoRA dimensions in
the search space which varies from 8 and 4 in the largest model, 4 and 2 in
the intermediate model, and 2 in the smallest model. This added parameter
flexibility in our method allows for user-defined trainable parameter count in
the final models. Results on the VTAB-1k benchmark are shown in Table 1. We
push the state-of-the-art in parameter-efficient transfer learning by up to
2.9%, Even our smallest model outperforms all existing methods by a
substantial margin. It is worth noting that GLoRA performs competitively
across datasets in contrast to all existing works which fail on at least one
dataset; proving GLoRA’s high generalization capabilities. GLoRA pushes the
state of the art in as many as 14 out of 19 datasets in the VTAB-1k benchmark
while performing competitively on the remaining datasets too.

Table 1: Full results on VTAB-1K benchmark. “# params” specifies the number of
trainable parameters in backbones. Average accuracy and # params are averaged
over group-wise mean values.

| Natural | Specialized | Structured |
---|---|---|---|---
|  |  # param (M)
---
|  Inference Cost
---
|  Cifar100
---
|  Caltech101
---
|  DTD
---
|  Flower102
---
|  Pets
---
|  SVHN
---
|  Sun397
---
|  Camelyon
---
|  EuroSAT
---
|  Resisc45
---
|  Retinopathy
---
|  Clevr-Count
---
|  Clevr-Dist
---
|  DMLab
---
|  KITTI-Dist
---
|  dSpr-Loc
---
|  dSpr-Ori
---
|  sNORB-Azim
---
|  sNORB-Ele
---
|  Average
---
_Traditional Finetuning_
Full |  85.8 |  - |  68.9 |  87.7 |  64.3 |  97.2 |  86.9 |  87.4 |  38.8 |  79.7 |  95.7 |  84.2 |  73.9 |  56.3 |  58.6 |  41.7 |  65.5 |  57.5 |  46.7 |  25.7 |  29.1 |  68.9
Linear |  0 |  - |  64.4 |  85.0 |  63.2 |  97.0 |  86.3 |  36.6 |  51.0 |  78.5 |  87.5 |  68.5 |  74.0 |  34.3 |  30.6 |  33.2 |  55.4 |  12.5 |  20.0 |  9.6 |  19.2 |  57.6
_PETL methods_
BitFit [28] |  0.10 |  - |  72.8 |  87.0 |  59.2 |  97.5 |  85.3 |  59.9 |  51.4 |  78.7 |  91.6 |  72.9 |  69.8 |  61.5 |  55.6 |  32.4 |  55.9 |  66.6 |  40.0 |  15.7 |  25.1 |  65.2
VPT-Shall. [7] |  0.06 |  $\uparrow$ |  77.7 |  86.9 |  62.6 |  97.5 |  87.3 |  74.5 |  51.2 |  78.2 |  92.0 |  75.6 |  72.9 |  50.5 |  58.6 |  40.5 |  67.1 |  68.7 |  36.1 |  20.2 |  34.1 |  67.8
VPT-Deep [7] |  0.53 |  $\uparrow$ |  78.8 |  90.8 |  65.8 |  98.0 |  88.3 |  78.1 |  49.6 |  81.8 |  96.1 |  83.4 |  68.4 |  68.5 |  60.0 |  46.5 |  72.8 |  73.6 |  47.9 |  32.9 |  37.8 |  72.0
Adapter [10] |  0.16 |  $\uparrow$ |  69.2 |  90.1 |  68.0 |  98.8 |  89.9 |  82.8 |  54.3 |  84.0 |  94.9 |  81.9 |  75.5 |  80.9 |  65.3 |  48.6 |  78.3 |  74.8 |  48.5 |  29.9 |  41.6 |  73.9
AdaptForm. [16] |  0.16 |  $\uparrow$ |  70.8 |  91.2 |  70.5 |  99.1 |  90.9 |  86.6 |  54.8 |  83.0 |  95.8 |  84.4 |  76.3 |  81.9 |  64.3 |  49.3 |  80.3 |  76.3 |  45.7 |  31.7 |  41.1 |  74.7
LoRA [5] |  0.29 |  - |  67.1 |  91.4 |  69.4 |  98.8 |  90.4 |  85.3 |  54.0 |  84.9 |  95.3 |  84.4 |  73.6 |  82.9 |  69.2 |  49.8 |  78.5 |  75.7 |  47.1 |  31.0 |  44.0 |  74.5
NOAH [8] |  0.36 |  $\uparrow$ |  69.6 |  92.7 |  70.2 |  99.1 |  90.4 |  86.1 |  53.7 |  84.4 |  95.4 |  83.9 |  75.8 |  82.8 |  68.9 |  49.9 |  81.7 |  81.8 |  48.3 |  32.8 |  44.2 |  75.5
FacT [18] |  0.07 |  - |  70.6 |  90.6 |  70.8 |  99.1 |  90.7 |  88.6 |  54.1 |  84.8 |  96.2 |  84.5 |  75.7 |  82.6 |  68.2 |  49.8 |  80.7 |  80.8 |  47.4 |  33.2 |  43.0 |  75.6
SSF [17] |  0.24 |  - |  69.0 |  92.6 |  75.1 |  99.4 |  91.8 |  90.2 |  52.9 |  87.4 |  95.9 |  87.4 |  75.5 |  75.9 |  62.3 |  53.3 |  80.6 |  77.3 |  54.9 |  29.5 |  37.9 |  75.7
RepAdapter [9] |  0.22 |  - |  72.4 |  91.6 |  71.0 |  99.2 |  91.4 |  90.7 |  55.1 |  85.3 |  95.9 |  84.6 |  75.9 |  82.3 |  68.0 |  50.4 |  79.9 |  80.4 |  49.2 |  38.6 |  41.0 |  76.1
GLoRA |  0.86 |  - |  76.4 |  92.9 |  74.6 |  99.6 |  92.5 |  91.5 |  57.8 |  87.3 |  96.8 |  88.0 |  76.0 |  83.1 |  67.3 |  54.5 |  86.2 |  83.8 |  52.9 |  37.0 |  41.4 |  78.0
GLoRA |  0.44 |  - |  76.5 |  92.3 |  75.2 |  99.6 |  92.3 |  91.2 |  57.5 |  87.3 |  96.7 |  88.1 |  76.1 |  80.6 |  67.2 |  53.4 |  84.5 |  83.5 |  52.8 |  35.2 |  40.8 |  77.6
GLoRA |  0.29 |  - |  76.1 |  92.7 |  75.3 |  99.6 |  92.4 |  90.5 |  57.2 |  87.5 |  96.7 |  88.1 |  76.1 |  81.0 |  66.2 |  52.4 |  84.9 |  81.8 |  53.3 |  33.3 |  39.8 |  77.3

###  3.2 Few-shot Learning

Figure 2: Results on few-shot learning datasets. The baseline methods include
Adapter, LoRA, VPT, NOAH. GLoRA consistently performs better across five
datasets and a varying number of training examples per class.

To extend the evaluation of GLoRA under conditions of limited data
availability, we present the performance of GLoRA on fine-grained visual
recognition datasets as the few-show learning, comparing it with LoRA,
Adapter, VPT, and NOAH. The results at 1, 2, 4, 8, and 16 shots are
illustrated in Figure 5. GLoRA demonstrates superior efficacy across the
majority of the few-shot learning datasets, consistently outperforming the
performance of existing methods by a large margin with similar parameter
counts. Interestingly, on the Flowers102 dataset, all methods yield similar
accuracy levels, attributable to the already exceptional overall performance.
On the Food101 dataset, the average accuracy of GLoRA is on par with NOAH.
From the first subgifure we can observe, the average performance boost becomes
more pronounced at higher shot scenarios, nevertheless, even at lower shot
settings, the gains remain significant.

###  3.3 Domain Generalization

Table 2: Results on domain generalization. GLoRA is significantly better than the existing methods. | Source | Target
---|---|---
| ImageNet | -Sketch | -V2 | -A | -R
Adapter [10] | 70.5 | 16.4 | 59.1 | 5.5 | 22.1
VPT [7] | 70.5 | 18.3 | 58.0 | 4.6 | 23.2
LoRA [5] | 70.8 | 20.0 | 59.3 | 6.9 | 23.3
NOAH [8] | 71.5 | 24.8 | 66.1 | 11.9 | 28.5
GLoRA (0.29M) | 78.3 | 30.6 | 67.5 | 13.3 | 31.0

The capacity of out-of-domain generalization holds significant value for
large-scale neural networks [29]. Models fine-tuned via PETL methods should
exhibit enhanced domain generalization aptitude, thereby making them more
applicable in real-world scenarios. We demonstrate the out-of-domain
generalization capabilities of GLoRA in Table 2, where a single ImageNet-1K
[25] fine-tuned GLoRA model is subjected to testing on out-of-domain datasets.
Aligning with preceding research, we limit the number of training examples per
class to 16 for this experiment. It’s noteworthy that the performance for the
fully-scaled ImageNet-1K fine-tuned model stands at 83.97% [1], and our
approach manages to narrow this performance gap, even within a 16-shot setting
(78.3%), thereby exhibiting superior few-shot learning on ImageNet-level
datasets. Furthermore, the out-of-domain performance also witnesses a
substantial boost in comparison to existing methods. When compared with LoRA,
GLoRA enhances out-of-domain performance by as much as 100% (ImageNet-A) and
50% (ImageNet-Sketch).

Figure 3: Distribution of GLoRA (0.86M) parameters across layer types on
VTAB-1K dataset. Q-K-V and Projection are linear layers in MHSA module and FC1
and FC2 are linear layers in MLP module.

###  3.4 Analysis and Discussion

Computational Cost.

Table 3: Inference efficiency comparison of GLoRA with existing methods. Method | $\uparrow$ #Param(M) | $\uparrow$ FLOPs(G) | Latency (imgs/sec)
---|---|---|---
bs = 1 | bs = 4 | bs = 16
Full tuning | 0 | 0 | 91.5 | 375.7 | 539.5
VPT [7] | 0.55 | 5.60 | 86.1 | 283.5 | 381.5
Adapter [10] | 0.16 | 0.03 | 70.9 | 306.6 | 504.7
AdaptFormer [16] | 0.16 | 0.03 | 71.4 | 309.9 | 508.1
NOAH [8] | 0.12 | 0.02 | 72.1 | 312.7 | 492.9
LoRA [5] | 0 | 0 | 91.5 | 375.7 | 539.6
GLoRA

We show the final inference cost of various PEFT methods in Table 3, computed
on an NVIDIA 3090 GPU. It is in this context that GLoRA significantly
outperforms other methods, as GLoRA benefits from zero parameter or FLOPs
overhead during the inference process. An ancillary advantage is the expedited
adaptability in real-world scenarios where previous models are already
deployed. The weights of GLoRA can be directly loaded without necessitating
any manual system modifications. As previously mentioned, GLoRA supports VPT-
Deep level prompts via the support tensor $\mathbf{C}$, however, it does not
impose any computational overhead due to its complete structural re-
parameterization design.

Visualizations of searched fine-tuning strategy for each layer. Figure 3
visually shows the distribution of trainable parameters across the four types
of linear layers embodied in ViT-B. Notably, the projection layer possesses
the minimum quantity of trainable parameters spanning across VTAB-1K
categories. Generally, the MLP module hosts a substantially higher number of
parameters compared to the MHSA. As anticipated, the structured group
necessitates a greater number of parameters for adaptation due to a pronounced
domain drift relative to ImageNet-1K [25]. Figure 4 illustrates the layerwise
configuration of the support tensors as searched by the GLoRA algorithm. Each
support tensor at every layer can potentially undergo 72 distinct adaptations
across datasets. Support tensors $\mathbf{D}$ and $\mathbf{E}$ exhibit
relatively low adaptation due to the prevalence of none adaptations, whereas
$\mathbf{A}$ and $\mathbf{B}$ demonstrate a higher number of adaptations,
though without a discernible pattern regarding the type of adaptation. It’s
important to underscore that even a basic scalar can function effectively as a
support tensor, enabling GLoRA to maintain superior parameter efficiency
despite adapting every linear layer.

Figure 4: Layerwise configuration of support tensors of GLoRA (0.86M) on
VTAB-1K dataset.

Our Limitations. A primary limitation of GLoRA is the evolutionary search
procedure, which increases the total training duration. However, it is fairly
justified since GLoRA requires negligible hyperparameter tuning and no manual
architecture design. Furthermore, owing to its inherent mechanism, GLoRA can
be effectively deployed for downstream tasks in both computer vision and
natural language processing domains, although these applications are not
investigated in this paper. Another facet of GLoRA is its adaptability to
convolutional layers, an aspect not detailed in this paper. However, this
should be straightforward to implement, following the principle of Equation
10.

##  4 Related Work

Due to the exponential growth in model size, the field of NLP proposed many
methods for parameter efficient fine-tuning (PEFT) of large language models
(LLMs) [30, 31, 32, 33, 34, 35, 28, 10]. The effectiveness of parameter-
efficient fine-tuning has been proven in a wide range of natural language
processing tasks [36]. With the advent growth in the size of vision models
[26, 37], methods specifically focused on image models have also been put
forward [18, 17, 16, 9, 8, 7, 38]. Among these methods, LoRA [32] has proven
to transfer well across modalities and tasks. This is partly due to the
simplistic design strategy of LoRA which directly works over weight tensors,
irrespective of model type or configuration. Additionally, unlike Adapters
[10, 16] and Prompt tuning [7], LoRA doesn’t add any additional inference
parameters or latency due to structural re-parameterization (SR) design.
RepAdapter [9] and SSF [17] also propose a SR design for PEFT, however
RepAdapter is specific to model architectures and required manual designing
for different layer configurations; SSF provides a very simple baseline but
suffers from low flexibility and capability due to adaptation limited in the
activation space. FacT [18] further decomposes LoRA matrices for better
parameter efficiency, but we argue that <1M parameter scale is fairly
efficient for fine-tuning on a single GPU. Thus, due to the advantages of LoRA
over other related works, it is of importance to increase the flexibility,
scalabilty and adaptabilty of LoRA.

##  5 Conclusion

We have presented GLoRA, a generalized parameter-efficient fine-tuning
approach that has successfully demonstrated the effectiveness in enhancing the
fine-tuning and transfer learning ability for the large-scale pre-trained
models. By adopting a generalized low-rank adaptation and re-parameterization
framework, GLoRA significantly reduces the number of parameters and
computation required for fine-tuning, making it a more resource-efficient and
practical method for real-world applications. The experiments conducted on a
diverse range of tasks and datasets have substantiated the superiority of
GLoRA over existing PEFT techniques, showcasing its scalability and
adaptability. Moreover, the ablation studies have provided valuable insights
into the inner workings and the relative importance of different GLoRA
components. This work not only contributes to the improvement of the fine-
tuning process for large-scale pre-trained models but also opens up new
avenues for future work, including further exploration of generalized low-rank
adaptation techniques, the development of hybrid approaches, and the
refinement of search and optimization algorithms. These areas of research may
continue to expand the accessibility and efficiency of transfer learning
across a broader range of applications.

## References

  * [1] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2021.
  * [2] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [3] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever.  Robust speech recognition via large-scale weak supervision.  arXiv preprint arXiv:2212.04356, 2022.
  * [4] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009.
  * [5] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  arXiv preprint arXiv:2106.09685, 2021.
  * [6] Zhiqiang Shen, Zechun Liu, Jie Qin, Marios Savvides, and Kwang-Ting Cheng.  Partial is better than all: revisiting fine-tuning strategy for few-shot learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9594–9602, 2021.
  * [7] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIII, pages 709–727. Springer, 2022.
  * [8] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu.  Neural prompt search.  arXiv preprint arXiv:2206.04673, 2022.
  * [9] Gen Luo, Minglang Huang, Yiyi Zhou, Xiaoshuai Sun, Guannan Jiang, Zhiyu Wang, and Rongrong Ji.  Towards efficient visual adaption via structural re-parameterization.  arXiv preprint arXiv:2302.08106, 2023.
  * [10] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for nlp.  In International Conference on Machine Learning, pages 2790–2799. PMLR, 2019.
  * [11] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby.  The visual task adaptation benchmark, 2020.
  * [12] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do imagenet classifiers generalize to imagenet?  In International Conference on Machine Learning (ICML), pages 5389–5400. PMLR, 2019.
  * [13] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing.  Learning robust global representations by penalizing local predictive power.  Advances in Neural Information Processing Systems (NeuIPS), 32, 2019\.
  * [14] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song.  Natural adversarial examples.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15262–15271, 2021.
  * [15] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 8340–8349, 2021.
  * [16] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo.  Adaptformer: Adapting vision transformers for scalable visual recognition.  arXiv preprint arXiv:2205.13535, 2022.
  * [17] Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang.  Scaling & shifting your features: A new baseline for efficient model tuning.  arXiv preprint arXiv:2210.08823, 2022.
  * [18] Shibo Jie and Zhi-Hong Deng.  Fact: Factor-tuning for lightweight adaptation on vision transformer.  arXiv preprint arXiv:2212.03145, 2022.
  * [19] Vladimir N Vapnik and A Ya Chervonenkis.  On the uniform convergence of relative frequencies of events to their probabilities.  Measures of complexity: festschrift for alexey chervonenkis, pages 11–30, 2015.
  * [20] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.  Food-101 – mining discriminative components with random forests.  In European Conference on Computer Vision, 2014.
  * [21] M-E Nilsback and Andrew Zisserman.  A visual vocabulary for flower classification.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), volume 2, pages 1447–1454. IEEE, 2006.
  * [22] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In Proceedings of the IEEE international conference on computer vision workshops, pages 554–561, 2013.
  * [23] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar.  Cats and dogs.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3498–3505. IEEE, 2012.
  * [24] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi.  Fine-grained visual classification of aircraft.  arXiv preprint arXiv:1306.5151, 2013.
  * [25] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 248–255. Ieee, 2009.
  * [26] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al.  Scaling vision transformers to 22 billion parameters.  arXiv preprint arXiv:2302.05442, 2023.
  * [27] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In International Conference on Learning Representations.
  * [28] Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel.  Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1–9, 2022.
  * [29] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy.  Domain generalization in vision: A survey.  arXiv preprint arXiv:2103.02503, 2021.
  * [30] Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang.  P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks.  CoRR, abs/2110.07602, 2021.
  * [31] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao.  Adaptive budget allocation for parameter-efficient fine-tuning.  In The Eleventh International Conference on Learning Representations.
  * [32] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.  Lora: Low-rank adaptation of large language models.  In International Conference on Learning Representations.
  * [33] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang.  Gpt understands, too, 2021.
  * [34] Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–4597, Online, August 2021. Association for Computational Linguistics.
  * [35] Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, 2021.
  * [36] Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Lam, Lidong Bing, and Nigel Collier.  On the effectiveness of parameter-efficient fine-tuning.  arXiv preprint arXiv:2211.15583, 2022.
  * [37] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby.  Big transfer (bit): General visual representation learning.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16, pages 491–507. Springer, 2020.
  * [38] Haoyu He, Jianfei Cai, Jing Zhang, Dacheng Tao, and Bohan Zhuang.  Sensitivity-aware visual parameter-efficient tuning, 2023.
  * [39] Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling.  Autoformer: Searching transformers for visual recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 12270–12280, October 2021.
  * [40] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 9(11), 2008.

## Appendix

##  Appendix A Hyperparameters

For the hyperparameter search, we primarily concentrate on the exploration of
the learning rate for supernet training, limiting our search scope to two
potential alternatives: 1e-4 and 5e-4. For a detailed account of dataset-
specific learning rates, please refer to Table 4. All other training
particulars strictly adhere to the exact training policy delineated in the
works of [18, 9].

Table 4: Dataset specific supernet training learning rate for VTAB-1K
datastet.

Dataset |  Cifar100 |  Caltech101 |  DTD |  Flowers102 |  Pets |  SVHN |  Sun397 |  Camelyon |  EuroSAT |  Resisc45 |  Retinopathy |  Clevr-Count |  Clevr-Dist |  DMLab |  KITTI-Dist |  dSpr-Loc |  dSpr-Ori |  sNORB-Azim |  sNORB-Ele
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
LR | 5e-4 | 5e-4 | 5e-4 | 5e-4 | 5e-4 | 5e-4 | 5e-4 | 5e-4 | 5e-4 | 5e-4 | 1e-4 | 1e-4 | 1e-4 | 5e-4 | 5e-4 | 5e-4 | 5e-4 | 5e-4 | 1e-4

##  Appendix B Hierarchical Transformer

We show the performance of GLoRA on the Swin-B backbone in Table 5. We follow
a dataset-specific learning rate search similar to ViT-B and also add GLoRA to
the reduction linear layer in Swin architecture to maintain uniformity and
avoid architecture-specific tuning. GLoRA can adapt to any layer irrespective
of architecture configuration and perform well across tasks and datasets which
can be clearly seen in Table 5 where GLoRA outperforms all existing works by a
fair margin.

Table 5: Performance on VTAB-1K benchmark with Swin-B pre-trained on ImageNet-21K as the backbone.  Method | Natural | Specialized | Structured | Average
---|---|---|---|---
Full | 79.2 | 86.2 | 59.7 | 75.0
Linear | 73.5 | 80.8 | 33.5 | 62.6
BitFit | 74.2 | 80.1 | 42.4 | 65.6
VPT | 76.8 | 84.5 | 53.4 | 71.6
FacT | 82.7 | 87.5 | 62.0 | 77.4
RepAdapter | 83.1 | 86.9 | 62.1 | 77.4
GLoRA | 83.7 | 88.7 | 61.9 | 78.1

##  Appendix C Training Time

Table 6: Manual design choices in existing approaches. Method | Design Choices/Hyperparameters
---|---
VPT | Prompt Length, Prompt Location, Prompt Depth
AdaptFormer | Adapter Location, Scaling Factor, Hidden dimension, Insertion Form
NOAH | VPT choices, Adapter choices, LoRA rank
RepAdapter | Adapter Location, Number of groups, Hidden dimension, Adapter variants
FacT | Decomposition method, Scaling factor, Decomposition Rank
GLoRA | LoRA ranks in search space

Our GLoRA, being a search-based approach for PEFT, naturally incurs increased
training time due to the requirements of supernet training and evolutionary
search. It is, however, critical to underscore that all current methods
necessitate a manual search for design choices, as evidenced in Table 6. This
necessity significantly inflates the total training time for a specific
dataset, due to the broad search within these design choices. GLoRA
streamlines this process through an automated evolutionary search mechanism,
thus leveraging the benefit of an expansive search space.

##  Appendix D Search Space

In this section, we undertake the computation of the possible number of
subnets within our GLoRA adapted supernet. Each layer offers $4,4,3,3$, and
$3$ options for the support tensor $\mathbf{A}$, $\mathbf{B}$, $\mathbf{C}$,
$\mathbf{D}$, and $\mathbf{E}$, respectively. This results in $432$ possible
configurations for a single linear layer. In our implementation, we
incorporate $48$ such layers within ViT-B, yielding a total of $432\times
48=20,736$ subnets being explored within GLoRA. This figure can escalate if
multiple LoRA ranks coexist within the same search space. For instance, we
allow ranks 8 and 4 in our largest GLoRA models, leading to $82,944$ distinct
subnets. Furthermore, owing to the phenomenon of weight entanglement as per
[39], comparable performance is maintained across all subnets, even if they
aren’t all explored during the training of the supernet.

Figure 5: Visualization of features from SVHN dataset by t-SNE [40].

##  Appendix E Support Tensor

In this section, we justify the choices of support tensors in our framework.
Consider a linear layer that facilitates the transformation of inputs from a
$d_{1}$ dimensional space to a $d_{2}$ dimensional space, with a corresponding
weight matrix $\mathbf{W}_{0}\in\mathbb{R}^{d_{2}\times d_{1}}$. Given that
$\mathbf{A}$ is tasked with scaling $\mathbf{W}_{0}$, $\mathbf{A}$ could
feasibly belong to $\mathbb{R}^{d_{2}\times d_{1}}$, $\mathbb{R}^{d_{2}\times
1}$, or $\mathbb{R}^{1\times 1}$. These matrix dimensions are respectively
indicative of LoRA, vector, and scalar operations. It’s pertinent to note that
in scenarios where $\mathbf{A}\in\mathbb{R}^{d_{2}\times d_{1}}$, LoRA is
realized via corresponding matrices $\mathbf{A}_{d}\in\mathbb{R}^{d_{2}\times
r}$ and $\mathbf{A}_{u}\in\mathbb{R}^{r\times d_{1}}$. A parallel scrutiny of
other support tensors would result in determining the appropriate support
tensor choice, as elaborated in Section 2.3 of the main paper.

##  Appendix F Fine-tuned Embedding Visualization

We present feature visualizations of the ViT-B model adapted via GLoRA and
FacT [18] methods applied to the SVHN dataset, as shown in Figure 5. We
selected FacT as opposed to LoRA [32], given that FacT constitutes a direct
mathematical enhancement over LoRA and presently represents the state-of-the-
art. A clear distinction can be discerned whereby GLoRA exhibits superiorly
segregated clusters in comparison to FacT. Further, the delineations are
broader, and the clusters demonstrate a higher degree of concentration,
signaling the heightened discriminative capacity of the GLoRA-adapted model
features.


CoALP-Ty’16
Pre-proceedings of the Workshop on Coalgebra, Horn Clause Logic Programming
and Types
November 28–29, 2016
Edinburgh, UK Ekaterina Komendantskaya and František Farka (ed.)
https://ff32.host.cs.st-andrews.ac.uk/coalpty16/

###### Contents

  1. Organisation
  2. Foreword
  3. Invited Talks
  4. Contributed Talks

## Organisation

### Organising Committee

#### Workshop Chairs

Ekaterina Komendantskaya,

Heriot-Watt University, UK

John Power,

University of Bath, UK

#### Publicity Chair

František Farka,

University of Dundee, UK and University of St Andrews, UK

### Programme Committee

Ki Yung Ahn,

Nanyang Technological University, Singapore

Davide Ancona,

University of Genoa, Italy

Filippo Bonchi,

CNRS, ENS de Lyon, France

Iavor Diatchki,

Galois, Inc, USA

Peng Fu,

Heriot-Watt University, Edinburgh, UK

Neil Ghani,

University of Strathclyde, UK

Patricia Johann,

Appalachian State University, USA

Ekaterina Komendantskaya,

Heriot-Watt University, Edinburgh, UK

Clemens Kupke,

University of Strathclyde, UK

J. Garrett Morris,

University of Edinburgh, UK

Fredrik Nordvall Forsberg,

University of Strathclyde, UK

John Power,

University of Bath, UK

Claudio Russo,

Microsoft Research Cambridge, UK

Martin Schmidt,

DHBW Stuttgart and Osnabrück University , Germany

Stephan Schulz,

DHBW Stuttgart, Germany

Aaron Stump,

The University of Iowa, USA

Niki Vazou,

University of California, San Diego, USA

Joe Wells,

Heriot-Watt University, Edinburgh, UK

Fabio Zanasi,

Radboud University of Nijmegen, The Netherlands

## Foreword

Welcome to the Workshop on Coalgebra, Horn Clause Logic Programming and Types
and to Edinburgh! The workshop marks the end of the EPSRC Grant _Coalgebraic
Logic Programming for Type Inference_ , by K. Komendantskaya, Heriot-Watt
University and J. Power, University of Bath.

The workshop consist of two parts:

  1. 1.

_Semantics:_ Lawvere theories and Coalgebra in Logic and Functional
Programming

  2. 2.

_Programming languages:_ Horn Clause Logic for Type Inference in Functional
Languages and Beyond

Over recent years, there has been considerable research on the semantics of
operational aspects of logic programming. The underlying mathematics has often
involved coalgebra on a presheaf category or a variant of a presheaf category,
with index given by the Lawvere theory generated by a signature. That has much
in common with many years of research on the semantics of functional
programming.

The combination of coalgebra with Lawvere theories has already led to applied
development, and there seems good reason to hope that that will continue, with
application to logic and functional programming both separately and in
combination with each other. So we would like to spend a few days bringing
researchers in the area together to compare the techniques they are developing
and the problems they are addressing.

Our semantic investigations led to analysis of theorem proving, and that was
reciprocated, with theorem proving further influencing our semantics. Theorem
proving in turn led us to study type inference, leading to the central applied
focus of our work, thus the second topic of the workshop.

Extended abstracts of all accepted contributions appear in this book and on
the workshop webpage. We have six invited and seven regular talks, with topics
ranging from Categorical Logic, Semantics, Type Theory, Proof Theory,
Programming Languages and Type Inference.

We are very fortunate to have six excellent invited speakers in

  * •

John Power from university of Bath with a talk on _Categorical Semantics of
Logic Programming: Laxness and Saturation_ ,

  * •

Steven Ramsay and Luke Ong from Oxford University with a talk on _Refinement
Types and Higher-Order Constrained Horn Clauses_ ,

  * •

Tarmo Uustalu from Tallinn University with a talk on _Comodels and
Interaction_ ,

  * •

Claudio Russo from Microsoft Research Cambridge with a talk on _Type Classes
for the Masses_ ,

  * •

Davide Ancona from University of Genova with a talk on _Abstract Compilation
for Type Analysis of Object-Oriented Languages_ , and

  * •

Ki Yung Ahn from Nanyang Technological University with a talk on _Relational
Specification of Type Systems Using Logic Programming_

We are grateful to them for agreeing to talk at the workshop.

To further facilitate the discussion between the audience and the invited
speakers, the invited talks will be followed by a $45$ minute panel session
“Questions and Answers with Invited Speakers".

Many thanks to all who helped to organise the event! In particular we are
grateful to Morag Jones, June Maxwell, Iain McCrone and Christine McBride for
technical and administrative support. Thanks to International Center for
Mathematical Sciences for providing an excellent venue for the workshop. We
are grateful to EPSRC (grants EP/K031864/1-2, EP/K028243/1) for supporting
this research and this workshop. Finally, we are very grateful to the
Programme Committee, and, in particular, to all those who supported the
workshop by submitting abstracts and attending the meeting. We hope that you
all enjoy the workshop.

Katya Komendantskaya and František Farka,

Edinburgh, 28 November 2016

## Invited Talks

See pages - of abstracts/CoALP-Ty16_paper_8.pdf

See pages - of abstracts/CoALP-Ty16_paper_11.pdf

See pages - of abstracts/CoALP-Ty16_paper_13.pdf

See pages - of abstracts/CoALP-Ty16_paper_10.pdf

See pages - of abstracts/CoALP-Ty16_paper_1.pdf

See pages - of abstracts/CoALP-Ty16_paper_12.pdf

## Contributed Talks

See pages - of abstracts/CoALP-Ty16_paper_4.pdf

See pages - of abstracts/CoALP-Ty16_paper_5.pdf

See pages - of abstracts/CoALP-Ty16_paper_6.pdf

See pages - of abstracts/CoALP-Ty16_paper_2.pdf

See pages - of abstracts/CoALP-Ty16_paper_7.pdf

See pages - of abstracts/CoALP-Ty16_paper_9.pdf

See pages - of abstracts/CoALP-Ty16_paper_3.pdf