# The pseudogap in hole-doped cuprates: possible insights from the Kondo
effect

J. R. Cooper  Cavendish Laboratory, Department of Physics, University of
Cambridge, J.J. Thomson Avenue, CB3 0HE, United Kingdom

###### Abstract

The “states-non-conserving” fermion density of states (DOS), deduced from the
specific heat of hole-doped cuprates, could arise from a Kondo or heavy
fermion-like DOS being suppressed by anti-ferromagnetic spin fluctuations. The
large Fermi surface predicted by band theory and observed experimentally, is
still expected for zero pseudogap, but with an effective mass corresponding to
a Kondo temperature $T_{K}\simeq$800 K. A finite pseudogap could divide it
into Fermi arcs. Theoretical results for the asymmetric Anderson model can
account for the experimental Wilson ratio.

###### pacs:

One of the unsolved questions for cuprate superconductors, in addition to the
pairing mechanism and the linear temperature ($T$) dependence of the
electrical resistivity, is the origin of the pseudogap (PG) in samples with a
certain number of holes ($p$) per CuO2 unit. The PG lowers the superconducting
condensation energy LoramJPCS1998 ; LoramJPCS2001 , the superfluid density,
e.g. Anukool2009 , and an important practical property, the irreversibility
field Cooper1997 where the electrical resistivity first becomes non-zero, even
though the zero-field transition temperature initially stays high. Many
researchers focus on a $T^{*}-p$ “phase diagram” and a $T^{*}(p)$ line,
ascribed to the onset of a PG, that goes to zero as $p$ is increased to
$p_{crit}$ Keimer2015 . Careful experiments reveal anomalies in several
physical properties at, or near $T^{*}$ Keimer2015 , but the values of $T^{*}$
and the size of the anomalies are probably too small to account for the PG.
For example the small Kokanovic2016 charge density wave (CDW) anomalies for
under-doped (UD) samples may be linked Comin2014 to the Fermi arcs, detected
by angle-resolved photoemission (ARPES), that are caused by the pre-existing
PG. Abrupt changes in magnetic anisotropy of YBa2Cu3O6+x (YBCO) crystals
defining a nematic $T^{*}(p)$ line Matsuda2017 , that also extrapolates to
zero at $p_{crit}$, are also small - typically 0.5$\%$ of the average in-plane
spin susceptibility for $\Delta T\sim$50 K. Here we retain the idea, reviewed
along with many experimental facts in TallonLoram2001 , that the PG is best
described as a $T$-independent energy scale $E_{G}$ that varies strongly with
$p$. We argue that it might be caused by low energy anti-ferromagnetic ($af$)
spin fluctuations giving a deep dip in a Kondo or heavy fermion-like density
of states (DOS) at the Fermi energy ($E_{F}$).

Analysis of the specific heat of YBCO Loram1993 , Bi2Sr2CaCu2O8+x (Bi2212)
LoramJPCS2001 , at different oxygen levels ($x$), and La2-ySryCuO4 (LSCO)
LoramJPCS2001 . in terms of a “states-non-conserving” V-shaped fermion DOS
LoramJPCS1998 , centered at $E_{F}$, in an otherwise flat band, as sketched in
Fig. 1(a), gives intriguing insights. It has has some deficiencies because (i)
the large thermoelectric power (TEP) and its scaling with $E_{G}$ Cooper2000 ,
require electron-hole asymmetry, (ii) the detection of quantum oscillations
and Fermi arcs implies that there is a small residual DOS at $E_{F}$ and (iii)
Y89 NMR Knight shift data for over-doped (OD) Y(Ca)BCO Williams1998 show a
fall of up to 20$\%$ in the spin susceptibility, $\chi_{s}(T)$, at higher $T$
rather than the constant value expected for a flat band, there are also small
deviations from flat band behavior for OD Bi2212 LoramJPCS2001 . On the other
hand Cu NMR data for La-doped Bi2201 Kawasaki2010 seem to be consistent with a
flat band. They also show that the PG is not affected by fields up to 44 T,
and that the residual DOS is much larger in Bi2201. The main features of this
first-order phenomenological analysis are as follows.

A. $E_{G}$ falls linearly with $p$, as $J_{af}(1-p/p_{crit})$, disappearing
abruptly at $p_{crit}$ = 0.17-0.19 LoramJPCS1998 , where $J_{af}/k_{B}$ =
1200-1500 K, is similar to the $af$ interaction $J_{af}/k_{B}$= 1700 K
Rossat1992 in the corresponding parent insulator with $p=0$ and $k_{B}$ is
Boltzmann’s constant.

B. Plots of $S_{e}(T,p)$ $vs.$ $T$, where $S_{e}(T,p)$ is the non-phonon,
electronic contribution to the entropy, are approximately linear and parallel
at higher $T$, e.g. Fig. 1(b) of LoramJPCS2001 , and Fig. 5 of Loram1993 , or
Figs. 1(SM) and 2(SM) of Suppmat . $S_{e}(T,p)$ curves for $p<p_{crit}$ have
lower entropy at all $T$ and there is no sign of the entropy “coming back” up
to at least 200 K for Bi-2212 and 300 K for YBCO. This behavior is consistent
with a recent comprehensive NMR paper Haase2020 . For fermions with an
energy($E$)-dependent DOS we expect $S_{e}(T)$ $\alpha$ $T\chi_{s}(T)$,
because both depend on the number of states within the thermal window
$|E-E_{F}|\lesssim 2k_{B}T$. $T\chi_{s}(T)$ curves for Bi2212 are parallel up
to 400 K, see Fig. 13(b) of LoramJPCS2001 or Fig. 3(SM) Suppmat , as are plots
of $T\chi_{s}(T)$ for Y0.8Ca0.2Ba2Cu3O6+x, also up to 400 K Naqib2009 , after
allowing for an $x$-independent Curie or Curie-Weiss term. The difference
between a states-conserving CDW- or superconductor-like DOS and a V-shaped gap
is shown by the results of model calculations of $S_{e}(T)$ in Fig. 4(SM)
Suppmat .

C. Plots of $S_{e}(T_{0})$ $vs.$ $p$ and $k_{B}T_{0}\chi_{s}(T_{0})$ $vs.$ $p$
at temperatures $T_{0}$ show anomalously large increases of $\approx k_{B}$
and 0.3$\mu_{B}^{2}$ per added hole respectively LoramJPCS2001 , where
$\mu_{B}$ is the Bohr magneton. For $p<p_{crit}$ this is caused by the
$p$-dependence of the PG and is consistent with the Wilson ratio discussed
later. For $p>p_{crit}$ they peak near $p=0.25$ before falling again, but this
interesting result has only been established experimentally for LSCO
LoramJPCS2001 .

D. $p_{crit}$ seems to be determined by the product of $J_{af}$ and the DOS at
high energy LoramJPCS2001 . In contrast to points A to C this is not expected
in the present picture because here $J_{af}$ in the parent compound and the
DOS for $E\gg E_{G}$ are not related.

It is widely accepted that the Hubbard model describes the basic physics of
the cuprates, e.g. Georges2018 and references therein, but it has not been
solved rigorously at low $T$ and as far as we know does not account for points
A to D above. Similar underlying physics occurs in the Friedel-Anderson model
Anderson1961 and the Kondo effect Gruner1974 ; Hewson , where metals such as
Cu or Au are alloyed with a small concentration ($c_{imp}$) of magnetic
elements (impurities) such as Fe or Mn. For both Kondo alloys and heavy
fermion compounds, the electronic specific heat coefficient, $\gamma\equiv
dS_{e}/dT$, is strongly enhanced at low $T$, in Wilson’s solution of the Kondo
problem Wilson1975 it is given by

${\gamma_{imp}=c_{imp}\pi^{2}k_{B}w/(6T_{K})}$ (1)

where $T_{K}$ is the Kondo temperature and $w$=0.4128 is the Wilson number.
For impurities with spin 1/2, and $T\ll T_{K}$,
$\chi_{s}^{imp}=c_{imp}(g\mu_{B})^{2}w/(4k_{B}T_{K})$, where $g\simeq 2$ is
the $g-$factor and $\chi_{s}^{imp}/\gamma_{imp}=R_{W}=2R_{0}$, in agreement
with experiments on some Kondo alloys Hewson . Here
$R_{0}=3\mu_{B}^{2}/(\pi^{2}k_{B}^{2})$ is the Wilson ratio for non- or
weakly-interacting fermions.

For Bi2212, $\gamma(200)$ varies from 0.86 to 0.98 mJ/g-at/K2 for 0.095 $<p<$
0.22 LoramJPCS2001 , and taking $c_{imp}$ to be the concentration of Cu atoms,
Eqn. 1 gives a first estimate for $T_{K}\simeq$800 K. We note that such a
single ion Kondo model works well for the heavy fermion compound CePb3 doped
with La CePb . For the same range of $p$, $\chi_{s}(200)$ = 2.18 to 2.38 10-4
emu/mole Bi2212, see footnote footnotechi giving a Wilson ratio
$\chi_{s}/\gamma=$ 1.29 to 1.34$R_{0}$. For YBa2Cu3O7, where there is some
uncertainty from the Cu-O chain contribution, and in the precise value of
$\gamma(300)$, $\chi_{s}/\gamma\simeq 1.2R_{0}$ CooperLoramRev1996 ;
LoramJPCS2001 . So the experimental data is closer to $R_{0}$ rather than the
Wilson value of 2$R_{0}$. This is not a limitation because Fig. 10.9 of Ref.
Hewson shows that for heavy fermion compounds such as CeCu2Si2, CeCu6 and
UPt3, $\chi_{s}/\gamma\simeq R_{0}$, and as explained later, a ratio of 1.3 is
consistent with the present picture. Numerical treatments of the $t-J$ model
Prelovsek2000 , derived from the 2D Hubbard model, give $\chi_{s}/\gamma\simeq
R_{0}$ for $0.1<p<0.2$ and $T<J/k_{B}$. Large values of $T_{K}$ do occur for
dilute alloys, e.g. AlMn where $T_{K}$ = 600 K at low $T$ falling to 470 K
near 300 K because of thermal expansion and the strong volume dependence of
$T_{K}$ Miljak1976 . This reduces the $T$-dependence of $\chi_{s}(T)$ from the
$1/(T+T_{K})$ law usually found for alloys with lower $T_{K}$ and less thermal
expansion.

Here we consider three localized states, with occupancy 0, 1 and 2 described
by the asymmetric single orbital Anderson model. Figs. 1(b) and (c), taken
from Fig. 5 of Costi1994 , show the spectral density for an isolated ion
obtained by numerical renormalization group (NRG) calculations for
$E_{0}=-U/2$, the symmetric case, and various asymmetric cases,
$-E_{0}/\Delta$ = 4, 3 and 2. $|E_{0}|$ and $U-|E_{0}|>0$ are the energies
required to transfer an electron from the singly-occupied state to the Fermi
level of the conduction electrons at $E_{F}$, or from $E_{F}$ to the doubly-
occupied state respectively. $U$, set equal to 4$\pi\Delta$ in Costi1994 , is
the on-site $d-d$ Coulomb repulsion, $\Delta=\pi V^{2}N(E)$ Anderson1961 ,
where $V$ is the hybridization energy between the $d$ state and the conduction
electrons and $N(E)$ is their DOS for a given spin direction at the energy of
the virtual bound state. The familiar triple peak structure in Fig. 1(b) has
two side peaks arising from the localized level and a peak near $E_{F}$ from
the Kondo resonance. The weak $p$ dependence of $\gamma$ and hence $T_{K}$ for
OD Bi2212 and other OD cuprates implies that $E_{0}$ and $\Delta$ are also
only weakly dependent on $p$.

As originally shown by Kondo Kondo1964 the central peak is caused by the $af$
exchange interaction -$2J_{imp}\underline{S}_{imp}.\underline{s}_{e}$, with
$J_{imp}<0$, between the spin on the magnetic ion, $\underline{S}_{imp}$ and
the spin of a conduction electron $\underline{s}_{e}$. The ${S}^{+}_{imp}$ and
${S}^{-}_{imp}$ operators do not commute, and because of this, higher order
scattering processes contain the Fermi function for the conduction electrons.
This leads to the logarithmic Kondo divergence in the scattering rate,
responsible for the resistance minimum observed in dilute magnetic alloys and
for the formation of the Kondo resonance at $E_{F}$. It is a many-body effect
because it depends on the occupancy of many other conduction electron states.

Figure 1: Color online. (a) Fermion DOS LoramJPCS1998 used to rationalize
specific heat data. (b) and (c), taken from Costi1994 with permission, triple
peak structure in the spectral density $\rho_{0}(\omega,T=0)$ calculated using
NRG theory for the single orbital, asymmetric Anderson model with $S_{imp}$
=1/2, for various values of $-E_{0}/\Delta$ = 2$\pi$, 4, 3 and 2. Here we
estimate $-E_{0}/\Delta=2.3$ for all the Cu ions in Bi2212. The horizontal
arrows show $T_{K}/\Delta/2$ for $-E_{0}/\Delta$= 2 and 3 from Costi1994 .
Here the factor 2 ensures consistency between Eqn. 1 Wilson1975 and Costi1994
, details are given in Suppmat .

The Kondo resonance has the same spatial symmetry as the electronic state of
the localized level Gruner1974 ; Blandin1968 . This is why band-structures of
heavy fermion materials, calculated using relatively standard methods, agree
with the extremal orbits observed in quantum oscillation experiments, e.g.
Flouquet2017 , because the band structures depend on this symmetry. But the
effective masses ($m^{*}$) are often very large, because they are determined
by the energy width of the Kondo resonance. For the cuprates this implies that
the Fermi surface is large, with the shape predicted by standard band theory,
but $m^{*}$ is larger.

The effects of an applied magnetic field ($H$) and, more generally, an
exchange field, are important here. In dilute Kondo alloys, $H$ suppresses the
Kondo effect because spin-flip scattering processes become inelastic in a
magnetic field Gruner1974 . Early tunnelling studies where a fraction of a
monolayer of Fe atoms was evaporated on the oxide layer of an Al-Al2O3-Al
junction Bermon1978 showed evidence for this “hole in the DOS” and it is
indeed “non-states-conserving”, see Fig. 9 of Bermon1978 shown in Fig. 5(SM)
Suppmat . Experimentally Monod1967 ; Rohrer1969 and theoretically Horvatic1984
; CostiMR2000 this requires $\mu_{B}H\gtrsim k_{B}T_{K}$. A problem here is
that such a large value of $H$, $\approx k_{B}T_{K}/\mu_{B}$, is incompatible
with the appropriate condition $E_{G}\simeq\mu_{B}H$ when $p\lesssim p_{crit}$
and $E_{G}$ is small.

This difficulty is absent for a molecular field in NRG calculations Costi2007
for the two-band Hubbard model plus a ferromagnetic, predominantly Ising,
exchange interaction $-2J_{ex}S^{z}_{i}.s^{z}$ between the localized magnetic
moments and the conduction electrons. Note that these calculations refer to a
concentrated heavy fermion-like spin system. For $J_{ex}=0$ there are two side
peaks from the narrow band, i.e. the localized magnetic states, and a central
peak from the broad band, equivalent to the conduction electron band here. But
as shown in Costi2007 for large enough $J_{ex}$ there is a quadruple peak in
the DOS curves, i.e. an extra dip at $E_{F}$, whose width is $\simeq 2J_{ex}$.
It is possible that $J_{ex}$ suppresses higher order spin-flip scattering
processes, thereby suppressing the DOS at $E_{F}$. There appears to be no
limitation regarding the ratio of $J_{ex}/k_{B}T_{K}$, in contrast to the
effect of $H$ on Kondo alloys.

Figure 2: Color online. Partial DOS $vs.$ energy $E-E_{F}$; in-plane oxygen
orbitals of YBa2Zn3O7, solid black line, $d$ orbitals for Zn(2) atoms in
YBa2Zn3O7 and Cu(2) atoms in YBa2Cu3O7, short and long dashed lines
respectively. Points show $N(E_{F})$ of YBa2Cu3O7 from LDA and from
$\gamma$(300). When expressed as states/eV/Cu atom (both spins), $\gamma$(300)
for YBa2Cu3O7, gives 3.3$\pm$0.5, the same within errors as 2.9$\pm$0.2
states/eV/Cu atom (both spins) from $\gamma$(200) of Bi2212.

It is difficult to calculate $\Delta$ from first principles, even for noble-
metal hosts, see Hewson , Chapt. 9.7. We attempted this by calculating the
band-structure of the hypothetical compound YBa2Zn3O7, in which all lattice
parameters are kept the same as for YBa2Cu3O7, using the Wien2K code Wien2K .
In a sense this is equivalent to experimentalists using Lu compounds with a
full $f-$shell as a reference for Yb compounds. The Brillouin zone is sketched
in Fig. 6(SM)(a) Suppmat and dispersion curves shown in Fig. 6(SM)(b) Suppmat
. The Zn $d$-levels are well below $E_{F}$ as shown by the the DOS $vs.$
energy plots in Fig. 2 and there are approximately 10 $d$-states per Zn atom,
i.e. a full $d$ shell as expected from chemical arguments. In-plane states
near $E_{F}$ do have a small amount of $d$ character but as shown in Fig. 2
this is only a few $\%$ of the DOS derived from the measured value of
$\gamma$. In contrast, band structure calculations for YBa2Cu3O7 using the
local density approximation (LDA) give substantial contributions to $N(E_{F})$
from the Cu $3d$ orbitals forming the Cu-O2 plane bands. The full LDA DOS for
YBa2Cu3O7 (not shown) agrees well with earlier work Pickett1989 and
$\gamma$(300) is only moderately enhanced over the LDA value, see Fig. 2. But
$N(E_{F})$ from the in-plane oxygen bands and the in-plane Zn atoms in
YBa2Zn3O7 is much smaller, so in the present picture the measured values of
$\gamma$ for YBCO and Bi2212 arise from the Kondo effect - Eqn. 1. Fig.
6(SM)(c) Suppmat shows the Fermi surface of YBa2Zn3O7, with two quasi-
cylindrical sheets of similar size from the bands crossing $E_{F}$ between S
and X and S and Y, and an open surface, from the band crossing $E_{F}$ between
Y and G. From Fig. 6(SM)(b) Suppmat the average value of $E_{F}$ for the two
quasi-cylindrical bands of the Zn compound is 0.54 eV relative to the top of
the band. The average value of $k_{F}$ in the S-X-Y plane is 0.35 $\pi/a$
where $a$ = 0.382 nm is the in-plane lattice spacing, giving $p$ for the two
cylindrical bands which agrees well with the more precise value obtained by
integrating the DOS plots in Fig. 2, for the in-plane O(2), O(3) and Zn(2)
atoms. Namely $p$= 2 x 0.19/unit cell, in surprisingly good agreement with
empirical experimental estimates for YBa2Cu3O7 Presland1991 ; Liang2006 ;
Obertelli1992 . It would be interesting to see whether this procedure works
for the two-chain compound YBa2Cu4O8.

The complex band structure below $\approx$ -1 $eV$ shown in Figs. 2 and
6(SM)(b) Suppmat prevents us estimating $\Delta$ from first principles, so
initially we set $\Delta$ = 1 $eV$, a typical value for 3$d$ impurities in
noble metals Anderson1961 ; Gruner1974 . But, as explained in more detail in
Suppmat , we can account for the experimental value of the Wilson ratio if
$E_{0}/\Delta$ = -2.3, because for this value, linear interpolation of the
data in Table 1 of Costi1994 gives the occupancy of the localized level,
$n_{0}$ = 0.81 and an effective moment of 0.81$\mu_{B}$. This reduces the
susceptibility by 0.812 and the Wilson ratio from the value 2 expected for an
ion with $S$=1/2 to 1.31, in good agreement with experiment. Theoretical
$S_{e}(T)$ curves were obtained by integrating the specific heat data in Fig.
4 of Costi1994 divided by $T$. For $T_{K}$ = 800 K, $\gamma$ increases by a
factor 1.19 between $T$= 200 K and $T$=0, so the initial estimate of $T_{K}$ =
800 K from $\gamma(200)$ of Bi2212 is reduced to 672 K. Theoretical curves for
representative samples of Bi2212, one with no PG, $p$=0.19 and one with a PG,
are linear over the measured range of $T\leq$220 K as shown in Fig. 7(SM)
Suppmat . In contrast those for YBa2Cu3O6.97 with no PG and YBa2Cu3O6.48 with
$E_{G}/k_{B}$ = 320 K, in Fig. 8(SM) Suppmat , both having $T_{K}$=775 K, show
deviations from linearity above 200 K. But these do not rule out the present
picture because they could arise from a modest decrease in $T_{K}$ caused by
thermal expansion, or because the Cu atoms in the CuO chains and the CuO2
planes are inequivalent.

Hence for Bi2212 $T_{K}$=672 K and for $E_{0}/\Delta$ = -2.3,
$k_{B}T_{K}/\Delta$ = 0.073, and $\Delta$= 0.79 $eV$. The observation of
significant $g-$factor anisotropy in YBCO Kokanovic2016 and references
therein, supports the present localized picture. Here the concept of valency
is meaningful, the Cu2+ ($d^{9}$) state will be dominant and for
$-E_{0}=2.3\Delta$ there is a certain admixture of Cu3+ ($d^{8}$). The
$d^{10}$ and $d^{9}$ states are analogous to those in heavy fermion Yb
compounds with 0 or 1 hole in the 4$f$ shell, which implies that all the Cu
$d$ electrons are contributing to $U$. As shown in Fig. 1(c) the half-width-
half-maximum (HWHM) values of the spectral density at $\omega<0$ for
$-E_{0}/\Delta$ = 3 and 2 are $\simeq$0.076$\Delta$ and 0.135$\Delta$
respectively giving 0.117$\Delta$ for $-E_{0}/\Delta$ = 2.3. The carrier
density in the resonance $n_{K}$, obtained by assuming that it has a
Lorentzian shape for $\omega<0$ and normalizing to the area of the lowest
broad peak in Fig. 1(b) that contains $\simeq$ 1 electron, agrees well with
(1-$n_{0}$)/2 = 0.095. For $\Delta$ = 0.79 $eV$ the HWHM is $T_{L}$=1072 K.
When transformed in the usual way into $\underline{k}-$ and then by Fourier
transform into $\underline{r}-$ space, _and in the absence of a PG_ , the
carrier density decays as $n_{K}\exp(-r/R)$ with a range $R=\frac{\hbar
v_{F}}{k_{B}T_{L}}$. The resonance is centered at the Fermi level and because
of the sharpness of the Fermi surface, the Fourier transform could also give
rise to Friedel or Rudermann-Kittel-Kasuya-Yosida type oscillations in
$n_{k}(r)$, that could also be important but are not considered here. The
Fermi velocity $v_{F}$= 1.6 107 cm/sec, estimated for a cylindrical surface
containing 1+$p$ holes with $m^{*}=5.2m_{e}$ Bangura2010 , agrees well with
nodal ARPES data for UD and optimally-doped (OP) crystals of the single layer
cuprate Hg1201 shown in Fig. 2(b) of Vishik2020 , 1.7, 2.2 and 2.0 107 cm/sec
for UD70, UD80 and OP98. This agreement supports our statement that $T_{K}$
and hence $m^{*}$ vary little with $p$.

The above values of $v_{F}$ and $T_{L}$ give $R=2.83a$. From the previous
spatial symmetry argument, $n_{K}$ will have $|d_{x^{2}-y^{2}}|$ symmetry and
be larger in the (0,1) and (1,0) directions, where $af$ spin fluctuations
cause neighboring spins to be anti-parallel over a certain spatial range,
$\xi$. A typical value for $p$ = 0.1 is $\xi$ = 0.9 nm, i.e. 2-3$a$ Rossat1992
. If $\xi=2a$ then only the 4 nearest neighbors have correlated spins. Their
$af$ Kondo screening “clouds” will give a ferromagnetic conduction electron
spin polarization at the central site. Including the effect of the PG on both
$n_{k}$ and $R$ (by self-consistently representing the PG by a narrower
negative Lorentzian) reduces $n_{k}$ at the central site by a factor $\simeq
2$, giving a magnetic moment from the 4 overlapping Kondo “clouds” of 2 x
0.095 x $\exp(-1/2.83)\mu_{B}$ = 0.133$\mu_{B}$. It is difficult to prove that
this spin polarization is equivalent to $J_{ex}$ in Costi2007 , but, for
$\chi_{s}$ = 2.3 10-4 emu/mole-Bi2212, a spin polarization of 0.133
$\mu_{B}$/Cu is produced by a field $H$ where $\mu_{B}H/k_{B}$ = 433 K, so
$J_{ex}/k_{B}$ = 866 K. From Fig. 2 of Costi2007 the half energy gap,
equivalent to $E_{G}$ here, is $\simeq J_{ex}$, in reasonable agreement with
the experimental value $E_{G}/k_{B}\simeq$600 K for $p$ = 0.1. Furthermore,
the imaginary part of the self energy is unusual and typical of a “bad metal”
Costi2007 in agreement with experiments on the cuprates. However, more work is
needed to see whether $J_{ex}$ falls linearly with $p$ and whether
calculations allowing for asymmetry in the DOS about $E_{F}$ on a scale of
$J_{ex}$, i.e. $E_{G}$, would account for the strong $p$-dependence of the TEP
and its scaling with $k_{B}T/E_{G}$ CooperLoramRev1996 ; Cooper2000 , shown in
Fig. 9(SM) Suppmat .

Alternatively, in a related picture, the effect of the $af$ fluctuations, i.e.
the interactions between neighboring Cu spins, could be considered directly,
without appealing to the work of Ref. Costi2007 . These will reduce the
entropy of the Cu spins and hence that available for the Kondo resonance. Key
theoretical questions are (a) can they cause a deep PG even when the
interaction energy ($W$) is much smaller than $k_{B}T_{K}$ and (b) does $W$
fall linearly with $p$ and become very small for $p\geq p_{crit}$?
Experimental evidence for (b) is given by ratio of the 63Cu and 17O NMR
relaxation rates $T_{1}^{-1}$ TallonLoram2001 which is a measure of the
strength of low frequency spin fluctuations for which the effect of the PG on
$\chi_{s}$ cancels out. It does indeed fall linearly with $p$, approaching
zero at $p_{crit}$. Analysis Tallon2008 of neutron scattering data shows the
same decrease for $af$ spin fluctuations of energy below 50 $meV$, so on the
time scale of the Kondo resonance, with $k_{B}T_{K}\approx$ 60 $meV$, they
will provide a molecular field that is nearly static.

Within the usual localized picture $W$ arises from superexchange via
completely occupied or completely empty oxygen $2p$ states. For a band with
Fermi energy $E_{F}$, we should consider Wannier functions that have a
lifetime of order $\hbar/E_{F}$. $E_{F}$ will increase with $p$, thereby
reducing $W$, which could account for point (b) above. As sketched for example
in Fig. 13 of CooperLoramRev1996 , only those parts of the large Fermi surface
spanned by the $af$ wave vector
$\underline{Q}\approx(\frac{\pi}{a},\frac{\pi}{a})$, will be affected by $af$
fluctuations. This accounts for the Fermi arcs, where $E_{G}$ =0, seen by
ARPES Chen2019 and scanning tunnelling microscopy (STM) Davis2014 . It also
gives a strong energy dependence of the scattering rate responsible for the
TEP CooperLoramRev1996 , see also Hildebrand1997 , and its scaling with
$k_{B}T/E_{G}$ shown in CooperLoramRev1996 and Fig. 9(SM) Cooper2000 ; Suppmat
.

Evidence for an energy scale $\approx$60$~{}meV$ in the many spectroscopic
studies of cuprates above $T_{c}$ would be a good experimental test of the
present picture. ARPES data do show structure near 50 $meV$, e.g. Vishik2020
that is presently ascribed to other factors. However, we are dealing with a
many-body effect, so comparison of ARPES, STM Davis2014 and optical data
Basov_Timusk2005 ; Tajima2016 for cuprates with those for heavy fermion
compounds as well as with theory could be informative. Another test would the
apparent correlation between the $T^{1}$ term in the electrical resistivity
and the superfluid density Culo2021 in OD compoumds. One prediction of the
present approach is that $m^{*}$ and related properties of OD Tl2201 crystals
could be strongly pressure dependent since for classical Kondo alloys the
volume ($V$) dependence of $T_{K}$ is large, -d$\ln T_{K}$/d$\ln V$ = 16-18
Miljak1976 ; Schilling1973 .

In summary, guided by the unusual behavior of the electronic entropy revealed
by specific heat measurements LoramJPCS1998 ; LoramJPCS2001 ; Loram1993 , we
propose that the pseudogap is the energy scale over which a Kondo-like
enhanced DOS at the Fermi energy is suppressed by $af$ spin interactions.

The author would like to thank A. Carrington, who also calculated the band
structure of YBa2Zn3O7, J. L. Tallon and V. Zlatić for helpful suggestions and
discussions and to acknowledge a long and happy collaboration with the late
Dr. J. W. Loram.

## References

  * (1) J. W. Loram, K. A. Mirza, J. R. Cooper and J. L. Tallon, Specific heat evidence on the normal state pseudogap, J. Phys. Chem. Solids 59, 2091-94 (1998).
  * (2) J. W. Loram, J. Luo, J. R. Cooper, W. Y. Liang and J. L. Tallon, Evidence on the pseudogap and condensate from the electronic specific heat, J. Phys. Chem. Solids 62, 59-64 (2001).
  * (3) W. Anukool, S. Barakat, C. Panagopoulos and J. R. Cooper, Effect of hole doping on the London penetration depth in Bi2.15Sr1.85CaCu2O8+δ and Bi2.1Sr1.9Ca0.85Y0.15Cu2O8+δ, Phys. Rev. B 80 024516 (2009).
  * (4) J. R. Cooper, J. W. Loram, J. D. Johnson, J. W. Hodby, and Chen Changkang, 3D XY scaling of the irreversibility line of YBa2Cu3O7 crystals, Phys. Rev. Lett. 79, 1730 (1997).
  * (5) B. Keimer, S. A. Kivelson, M. R. Norman, S. Uchida and J. Zaanen, From quantum matter to high-temperature superconductivity in copper oxides, Nature 519 179-186 (2015).
  * (6) I. Kokanović and J. R. Cooper, Magnetic susceptibility of YBa2Cu3O6+x crystals: Unusual Curie behavior and small contributions from charge density waves, Phys. Rev. B 94 075155-11 (2016).
  * (7) R. Comin, A. Frano, M. M. Yee, Y. Yoshida, H. Eisaki, E. Schierle, E.Weschke, R. Sutarto, F. He, A. Soumyanarayanan, Y. He, M. Le Tacon, I. S. Elfimov, J. E. Hoffman, G. A. Sawatzky, B. Keimer, and A. Damascelli, Charge order driven by Fermi-arc instability in Bi2Sr2-xLaxCuO6+δ, Science 343, 390-392 (2014).
  * (8) Y. Sato, S. Kasahara1, H. Murayama, Y. Kasahara, E.-G. Moon, T. Nishizaki, T. Loew, J. Porras, B. Keimer, T. Shibauchi and Y. Matsuda, Thermodynamic evidence for a nematic phase transition at the onset of the pseudogap in YBa2Cu3Oy, Nature Physics, 13 1074 (2017).
  * (9) J. L. Tallon and J. W. Loram, The doping dependence of T* - what is the real high-T-c phase diagram?, Physica C- Superconductivity and its Applications 349 53-68 (2001).
  * (10) J. W. Loram, K. A. Mirza, J. R. Cooper and W. Y. Liang, Electronic specific heat of YBa2Cu3O6+x from 1.8 to 300 K, Phys. Rev. Lett. 71 1740-1743 (1993).
  * (11) J. R. Cooper, H. Minami, V. W. Wittorff, D. Babić and J. W. Loram, Effect of the normal state gap on the thermoelectric power, irreversibility line and c-axis resistivity of YBa2Cu3O7-δ, Physica C 341 855-858 (2000).
  * (12) G. V. M. Williams, J. L. Tallon, R. Michalak and R. Dupree, NMR studies of overdoped Y1-xCaxBa2Cu3O7-δ, Phys. Rev. B 57 8696-8701 (1998).
  * (13) S. Kawasaki, C. T. Lin, P. L.Kuhns, A. P. Reyes and G. Zheng, Carrier-concentration dependence of the pseudogap ground state of superconducting Bi2Sr2-xLaxCuO6+δ revealed by Cu-63,Cu-65-Nuclear Magnetic Resonance in very high magnetic fields, Phys. Rev. Lett. 105 137002 (2010).
  * (14) J. Rossat-Mignod, L. P. Regnault, C. Vettier, P. Bourges, P. Burlet, J. Bossy, J. Y. Henry and G. Lapertot, Spin dynamics in the high Tc system YBa2Cu3O6+x, Physica B 180-181 383-388 (1992).
  * (15) See Supplemental Material(SM) for additional Figures and explanations.
  * (16) J. Nachtigal, M. Avramovska, A. Erb, D. Pavi$\breve{c}$evi$\acute{c}$, R. Guehne and J. Haase, Temperature-independent cuprate pseudogap from planar oxygen NMR, Condens. Matter 5, 66-84 (2020).
  * (17) S. H. Naqib, J. R. Cooper, and J. W. Loram, Effects of Ca substitution and the pseudogap on the magnetic properties of Y1-xCaxBa2Cu3O7-δ, Phys. Rev. B 79, 104519-8 (2009).
  * (18) W. Wu, M. S. Scheurer, S. Chatterjee, S. Sachdev, A. Georges and M. Ferrero, Pseudogap and Fermi-surface topology in the two dimensional Hubbard model, Phys. Rev. X 8 021048 (2018).
  * (19) P. W. Anderson, Localized magnetic states in metals, Phys. Rev. 124, 41 (1961).
  * (20) G. Grüner Experimental evidence for many-body effects in dilute alloys, Adv. Phys. 23 941-1023 (1974).
  * (21) A. C. Hewson, “The Kondo problem to heavy fermions”, Cambridge University Press, Cambridge, U.K. (1993).
  * (22) K. G. Wilson, Renormalization group- critical phenomena and Kondo problem, Rev. Mod. Phys. 47 773-840 (1975).
  * (23) C. L. Lin, A. Wallash, J. E. Crow, T. Mihalisin and P. Schlottmann, Heavy-fermion behavior and the single-ion Kondo model, Phys. Rev. Lett. 58 1232-1235 (1987).
  * (24) The measured susceptibility of Bi2212 at 300 K, is 1.1 to 1.3 10-4 emu/mole-Bi2212 LoramJPCS2001 for $0.095<p<0.22$, giving $\chi_{s}$ = 2.18 to 2.38 10-4 emu/mol-Bi2212 after corrections for core diamagnetism and Van Vleck orbital paramagnetism. These are respectively -1.95 10-4 emu/mol-Bi2212 Pascalbook and 0.866 10-4 emu/mol-Bi2212, using the Cu-O2 plane value for YBCO MilaRice1989 .
  * (25) J. R. Cooper and J. W. Loram, Some correlations between the thermodynamic and transport properties of high $T_{c}$ oxides in the normal state, J. Phys. I France 6 2237-2263 (1996).
  * (26) J. Jaklić and P. Prelovšek, Finite-temperature properties of doped antiferromagnets, Adv. Phys. 49 1-92 (2000).
  * (27) J. R. Cooper and M. Miljak, Single impurity behaviour and interaction effects in the magnetic susceptibility of AlMn and AlCr alloys, J. Phys. F: Met. Phys. 6 21 51-2164 (1976).
  * (28) T. A. Costi, A. C. Hewson and V. Zlatić, Transport coefficients of the Anderson model via the numerical renormalization group, J. Phys.: Condens. Matter 6 2519-2558 (1994).
  * (29) J. Kondo, Resistance minimum in dilute magnetic alloys, Prog. Theor. Phys. 32 37-49 (1964).
  * (30) A. Blandin, Magnetic impurities in metals, J. Appl. Phys. 39 1285-1294 (1968).
  * (31) A. Pourret, M-T. Suzuki, A. P. Morales, G. Seyfarth, G. Knebel, D. Aoki, and J. Flouquet, Fermi surfaces in the antiferromagnetic, paramagnetic and polarized paramagnetic states of CeRh2Si2 compared with quantum oscillation experiments, J. Phys. Soc. Jpn. 86, 084702-7 (2017).
  * (32) S. Bermon, D. E. Paraskevopoulos and P. M. Tedrow, Ultra-high magnetic field study of the Kondo-type zero-bias conductance peak in magnetically doped metal-insulator-metal tunnel junctions, Phys. Rev. B 17 2110-2123 (1978).
  * (33) P. Blaha, K. Schwarz, G. K. H. Madsen, D. Kvasnicka and J. Luitz,“WIEN2K, An augmented plane wave + local orbitals program for calculating crystal properties”, Karlheinz Schwarz, Techn. Universität Wien, Austria, ISBN 3-9501031-1-2 (2001).
  * (34) W. E. Pickett, Electronic structure of the high-temperature oxide superconductors, Rev. Mod. Phys. 61 434-511 (1989).
  * (35) M. R. Presland, J. L. Tallon, R. G. Buckley, R. S. Liu and N. E. Flower, General trends in oxygen stoichiometry on $T_{c}$ in Bi and Tl superconductors, Physica C 176 95-105 (1991).
  * (36) R. Liang, D. A. Bonn and W. N. Hardy, Evaluation of CuO2 plane hole doping in YBa2Cu3O6+x single crystals, Phys. Rev. B 73 180505-180508(R) (2006).
  * (37) S. D. Obertelli, J. R. Cooper and J. L. Tallon, Systematics in the thermoelectric power of high T-c oxides, Phys. Rev. B 46 14928(R) (1992).
  * (38) P. Monod, Magnetic field dependence of the Kondo resistivity minimum in CuFe and CuMn alloys, Phys. Rev. Lett. 19 1113-1156 (1967).
  * (39) H. Rohrer, High-field magnetoresistance of dilute Cu-Mn and Cu-Fe alloys, Journ. Applied Phys. 40, 1472-1473 (1969).
  * (40) B. Horvatić and V. Zlatić, Magnetic field effects for the asymmetric Anderson Hamiltonian, Phys. Rev. B 30 6717-6731 (1984).
  * (41) T. A. Costi, Kondo effect in a magnetic field and the magnetoresistivity of Kondo alloys, Phys. Rev. Lett. 85 1504-1507 (2000).
  * (42) T. A. Costi and A. Liebsch, Quantum phase transition in the two-band Hubbard model, Phys. Rev. Lett. 99 236404-4 (2007).
  * (43) A. F. Bangura, P. M. C. Rourke, T. M. Benseman, M. Matusiak, J. R. Cooper, N. E. Hussey, and A. Carrington, Fermi surface and electronic homogeneity of the overdoped cuprate superconductor Tl2Ba2CuO6+δ as revealed by quantum oscillations, Phys. Rev. B 82 140501(R) (2010).
  * (44) S. A. Sreedhar, A. Rossi, J. Nayak, Z. W. Anderson, Y. Tang, B. Gregory, M. Hashimoto, D.-H. Lu, E. Rotenberg, R. J. Birgeneau, M. Greven, M. Yi and I. M. Vishik, Three interaction energy scales in the single-layer high-Tc cuprate HgBa2CuO4+δ, Phys. Rev. B 102, 205109 (2020).
  * (45) J. G. Storey, J. L. Tallon and G. V. M. Williams, Pseudogap ground state in high temperature superconductors, Phys. Rev. B 78 140506(R) (2008).
  * (46) S. Chen, M. Hashimoto, Y. He, D. Song, K-J. Xu, J.-F. He, T. P. Devereaux, H. Eisaki, D-H. Lu, J. Zaanen and Z-X. Shen, Incoherent strange metal sharply bounded by a critical doping in Bi2212, Science 366, 1099-1102 (2019).
  * (47) K. Fujita, C. K. Kim, I. Lee, J. Lee, M. H. Hamidian, I. A. Firmo, S. Mukhopadhyay, H. Eisaki, S. Uchida, M. J. Lawler, E-A. Kim and J. C. Davis, Simultaneous transitions in cuprate momentum-space topology and electronic symmetry breaking, Science 344 612-616 (2014).
  * (48) G. Hildebrand, T. J. Hagenaars, W. Hanke, S. Grabowski and J. Schmalian, Effects of electronic correlations on the thermoelectric power of the cuprates, Phys. Rev. B 56 R4317-4320 (1997).
  * (49) D. N. Basov and T. Timusk, Electrodynamics of high-Tc superconductors, Rev. Mod. Phys. 77 721 (2005).
  * (50) S. Tajima, Optical studies of high-temperature superconducting cuprates, Rep. Prog. Phys. 79 094001 (2016).
  * (51) M. Čulo, C. Duffy, J. Ayres, M. Berben, Y.-T. Hsu, R. D. H. Hinlopen, B. Bernáth and N. E. Hussey, Possible superconductivity from incoherent carriers in overdoped cuprates, SciPost Phys. 11, 012 (2021).
  * (52) J. S. Schilling and W. B. Holzapfel, Effect of pressure on the Kondo temperature of Cu:Fe-existence of a universal resistivity curve, Phys. Rev. B 8 1216-1237 (1973).
  * (53) P. W. Selwood, “Magnetochemistry”, Interscience Publishers (1964).
  * (54) F. Mila and T. M. Rice, Analysis of magnetic resonance experiments in YBa2Cu3O7, Physica C 157 561-570 (1989).
  * (55) H.-U. Desgranges and K. D. Schotte, Specific heat of the Kondo model, Phys. Lett. 91A 240 (1982).
  * (56) V. T. Rajan, Magnetic susceptibility and specific heat of the Coqblin-Schrieffer model, Phys. Rev. Lett. 51 308-311 (1983).
  * (57) J. W. Loram, K. A. Mirza and J. R. Cooper, “IRC Research Review”, pp 75-97, Ed. W. Y.Liang, University of Cambridge, Cambridge, U.K. (1998).


# From Mimicking to Integrating:
Knowledge Integration for Pre-Trained Language Models

Lei Li1, Yankai Lin2,3, Xuancheng Ren1, Guangxiang Zhao1, Peng Li4, Jie Zhou5,
Xu Sun1
1MOE Key Lab of Computational Linguistics, School of Computer Science, Peking
University
2Gaoling School of Artificial Intelligence, Renmin University of China,
Beijing, China
3Beijing Key Laboratory of Big Data Management and Analysis Methods , Beijing,
China
4Institute for AI Industry Research (AIR), Tsinghua University, China
5Pattern Recognition Center, WeChat AI, Tencent Inc., China
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Investigating better ways to reuse the released pre-trained language models
(PLMs) can significantly reduce the computational cost and the potential
environmental side-effects. This paper explores a novel PLM reuse paradigm,
Knowledge Integration (KI). Without human annotations available, KI aims to
merge the knowledge from different teacher-PLMs, each of which specializes in
a different classification problem, into a versatile student model. To achieve
this, we first derive the correlation between virtual golden supervision and
teacher predictions. We then design a Model Uncertainty–aware Knowledge
Integration (MUKI) framework to recover the golden supervision for the
student. Specifically, MUKI adopts Monte-Carlo Dropout to estimate model
uncertainty for the supervision integration. An instance-wise re-weighting
mechanism based on the margin of uncertainty scores is further incorporated,
to deal with the potential conflicting supervision from teachers. Experimental
results demonstrate that MUKI achieves substantial improvements over baselines
on benchmark datasets. Further analysis shows that MUKI can generalize well
for merging teacher models with heterogeneous architectures, and even teachers
major in cross-lingual datasets.111Our code is available at
https://github.com/lancopku/MUKI. Part of the work was done while Yankai Lin
and Peng Li were working at Tencent.

##  1 Introduction

Large-scale pre-trained language models (PLMs), such as BERT (Devlin et al.,
2019), RoBERTa (Liu et al., 2019) and T5 (Raffel et al., 2020) have recently
achieved promising results after fine-tuning on various natural language
processing (NLP) tasks. Many fine-tuned PLMs are generously released for
facilitating researches and deployments. Reusing these PLMs can greatly reduce
the computational cost of retraining the PLM from scratch and alleviate the
potential environmental side-effects like carbon footprints (Strubell et al.,
2019), thus making NLP systems greener (Schwartz et al., 2020). A commonly
adopted model reuse paradigm is knowledge distillation (Hinton et al., 2015;
Romero et al., 2015), where a student model learns to mimic a teacher model by
aligning its outputs to that of the teacher. In this way, though achieving
promising results with PLMs (Sun et al., 2019; Jiao et al., 2020), the student
is restricted to perform the same task as the teacher model, thus restricting
re-utilization of abundant available PLMs fine-tuned on different tasks, e.g.,
models fine-tuned on various label sets or even different datasets.

Figure 1: Comparison of knowledge distillation (KD) and knowledge integration
(KI). KD assumes that the student performs predictions on the identical label
set with the teacher, while KI trains a student model that is capable of
performing classification over the union label set of teacher models.

In this paper, we generalize the idea of KD from mimicking teachers to
integrating knowledge from teachers, and propose _Knowledge Integration_ (KI)
for PLMs. Given multiple fine-tuned teacher-PLMs, each of which is capable of
performing classification over a unique label set, KI aims to train a
versatile student that can make predictions over the union of teacher label
sets. As the labeled data for training the teachers may not be publicly
released due to data privacy issues, we assume no human annotations are
available during KI. The benefits of KI are two-fold. First, compared to KD,
KI can make full use of the released PLMs specializing different tasks.
Besides, the ability of the versatile student, i.e., the label set coverage,
can be improved over time by integrating newly released teacher models. Figure
1 illustrates the main difference between KD and KI.

As no annotations are available, the core challenge of KI lies in the
integration of outputs from teachers to form golden supervision, i.e., the
class probability distribution over the union label set, for guiding the
student. Through theoretical derivation, we first build the bridge between the
teacher predictions and the golden supervision, which indicates that the key
to recovering such supervision is to identify the adequate teacher for each
instance. However, due to the over-confident problem of PLMs (Desai and
Durrett, 2020), selecting qualified teachers for unlabeled instances is non-
trivial, and our exploration shows that prediction entropy is misleading.
Inspired by Monte-Carlo Dropout (Gal and Ghahramani, 2016), we inject
parameter perturbations to the teacher models during inference and then
estimate the model uncertainties over averaged predictions for indicating the
possible correct teacher model. Our Model Uncertainty–aware Knowledge
Integration (MUKI) framework is then proposed based on the estimated model
uncertainty. Specifically, the golden supervision is approximated by either
taking the outputs of the most confident teacher, or softly integrating
different teacher predictions according to the relative importance of each
teacher. Furthermore, for instances on which teachers achieve close
uncertainty scores, we introduce a re-weighting mechanism based on the margin
of uncertainty scores, to down-weight the contribution of instances with
potential conflicting supervision signals.

Experimental results show that MUKI can successfully achieve the goal of
knowledge integration, significantly outperforming baseline methods, and even
obtaining comparable results with models trained with labeled data. Further
analysis shows that MUKI can produce supervision close to the golden one and
generalize well for merging knowledge from heterogeneous teachers with
different architectures, or even cross-lingual teacher models.

The main contributions of this work can be summarized as follows: (1) We
explore knowledge integration for PLMs, which is capable of making full use of
released PLMs with different label sets and has great extendability. (2) We
present MUKI, a generalizable KI framework, which integrates the knowledge
from teachers according to model uncertainty estimated via Monte-Carlo Dropout
and re-weights the instance contribution based on the uncertainty margin. (3)
Experimental results demonstrate that MUKI is effective and generalizable,
significantly outperforming baselines.

##  2 Knowledge Integration for PLMs

In this section, we first give the task formulation for knowledge integration,
followed by the elaboration on the proposed MUKI framework.

###  2.1 Problem Formulation

Given $N$ teacher PLM models $TS=\left\\{T_{1},\dots,T_{N}\right\\}$, where
each teacher $T_{i}$ specializes in a specific classification problem, i.e., a
set of classes $Y_{i}$, knowledge integration aims to train student model $S$
for performing predictions over the comprehensive class set
$Y=\bigcup_{i=1}^{N}Y_{i}$, with an unlabeled dataset $\mathcal{D}$. We assume
that for each instance in $\mathcal{D}$ there is at least one teacher capable
of handling it and we focus on a practical setting where the teacher
specialties are totally disjoint, i.e., $Y_{i}\cap Y_{j}=\emptyset,\forall
i\neq j$, as merging teachers with overlapping classes can be easily converted
in to the disjoint situation.

###  2.2 Model Uncertainty–Aware Knowledge Integration

As there are no annotated data available due to the data privacy issue, we
need to construct supervision for guiding the student. Given a golden label
distribution $\mathcal{T}(x)$ for each instance $x$ over $Y$, we can train the
student by minimizing the KL-divergence:

$\mathcal{L}=\sum_{x\in\mathcal{D}}\text{KL}\left(S\left(x\right)||\mathcal{T}\left(x\right)\right),$
(1)

where $S(x)$ denotes the output distribution of the student for input $x$. As
we only operate on the output distribution level, thus this framework is
generalizable for PLMs that potentially differ in the model architectures and
training data distribution. To estimate the golden supervision
$\mathcal{T}(x)$, we first derive the correlation between $\mathcal{T}(x)$ and
the prediction $T_{i}(x)$ of teacher model $T_{i}$. Specifically, as teacher
$T_{i}$ specializes in label set $Y_{i}$, it can only predict $T_{i}(y\mid x)$
for instance $x$ when $y\in Y_{i}$. Therefore, the correlation between
$T_{i}(y\mid x)$ and global probability $\mathcal{T}(y\mid x)$ over the full
class set can be derived as:

$\displaystyle T_{i}(y\mid x)$ $\displaystyle=\mathcal{T}\left(y\mid x,y\in
Y_{i}\right)$ (2) $\displaystyle=\frac{\mathcal{T}\left(y,y\in Y_{i}\mid
x\right)}{\mathcal{T}\left(y\in Y_{i}\mid x\right)}.$ (3)

The above derivation indicates that we can recover the golden probability
distribution by (1) getting the teacher predictions, and (2) estimating the
denominator, which means how likely the instance $x$ lies in the teacher
$T_{i}$ specialty $Y_{i}$. As instances associated with classes not in $Y_{i}$
can be treated as the out-of-distribution data for the teacher $T_{i}$, the
teacher predictions would be more uncertain about these instances than that of
in-distribution instances (Hendrycks and Gimpel, 2017). We thus propose to
approximate the denominator in an opposite direction, i.e., estimating how
likely the instance is not belong to teacher $T_{i}$ via model uncertainty.
Followingly, we first explore different uncertainty estimations for recovering
the golden supervision, and then introduce how we incorporates teacher
predictions according to the estimated uncertainty scores.

####  2.2.1 Uncertainty Estimation

Figure 2: Model uncertainty (normalized) distributions evaluated with $1000$
instances randomly sampled from the AG News dataset. The vanilla prediction
entropy distributions of two teacher models overlaps greatly (left), while
Monte-Carlo Dropout produces a more accurate uncertainty approximation for
distinguishing the correct teacher model (right). Best viewed in color.

A naïve estimation is directly taking the statics like prediction entropy of
predicted class distribution. However, due to the over-confident issues of
over-parameterized models like PLMs (Guo et al., 2017; Desai and Durrett,
2020), this simple estimation can be unreliable. We investigate this by first
splitting the instances of the AG News dataset (Zhang et al., 2015) into two
sets with disjoint labels, and then fine-tuning teacher models on each set
separately. For each instance, there is a correct teacher that is capable of
handling it and a wrong teacher that is not qualified for processing it. We
plot the prediction entropy distributions of the correct teacher and the wrong
teacher in the left part of Figure 2. It can be found that the wrong teacher
also produces confident predictions even for instances that are not in its
speciality with nearly zero uncertainty scores, exhibiting a great overlap
with the correct teacher model. This indicates that utilizing the simple
metric will mislead the identification of the adequate teacher. To remedy
this, inspired by recent progress in Bayesian neural networks (Blundell et
al., 2015; Gal and Ghahramani, 2016), we propose to add small perturbations to
the model weights during inference to find out the correct teacher model. The
intuition behind is that, as the instance is well fitted by the parameter of
the qualified teacher model, the teacher can produce confident results
consistently in the multiple predictions even with small perturbed parameters.
On the contrary, small perturbations on the model weights of the wrong teacher
will lead to a drastic change in the output probabilities, resulting in more
uncertain predictions on average. Therefore, we can estimate the model
uncertainty more accurately according to the average predictions under
parameter perturbations. Specifically, we adopt Monte-Carlo Dropout (Gal and
Ghahramani, 2016), where the output distribution of an instance $x$ with
$T_{i}$ is calculated as:

$\displaystyle p_{i}\left(y\mid x,\mathcal{D}\right)$
$\displaystyle\approx\frac{1}{K}\sum_{k=1}^{K}p_{i}\left(y\mid\mathbf{W}_{k}^{i},x\right)$
(4)
$\displaystyle=\frac{1}{K}\sum_{k=1}^{K}T_{i}\left(x,\mathbf{W}_{k}^{i}\right),$
(5)

where $\mathbf{W}_{k}^{i}$ is the $k$-th masked weights of $T_{i}$ sampled
from the Dropout distribution (Srivastava et al., 2014), and $K$ is the
sampling number. The model uncertainty of teacher model $T_{i}$ thus can be
summarized as the entropy of the averaged probability distribution $p_{i}$:

$u_{i}=H\left(p_{i}\right)=-\sum_{y=1}^{|Y_{i}|}p_{i}^{y}\log p_{i}^{y}.$ (6)

As shown in the right part of Figure 2, the uncertainty distributions of the
correct teacher and the wrong teacher model estimated via Monte-Carlo Dropout
exhibit a clearer difference than vanilla prediction entropy, indicating its
great potential for guiding the probability combination.

Figure 3: Overview of the proposed MUKI framework, which consists of (a) Model
uncertainty scores estimation with Monte-Carlo Dropout. (b) Knowledge
integration for estimating the golden supervision according to uncertainty
scores with (c) instance-wise re-weighting mechanism. Best viewed in color.

####  2.2.2 Knowledge Integration

With the accurately estimated teacher uncertainties
$U=\\{u_{1},\dots,u_{N}\\}$ for each instance at hand, we design two methods
for approximating the golden supervision to guide the student model:

MUKI-Hard which directly takes the supervision provided by the teacher with
the lowest uncertainty as the golden distribution:

$\displaystyle i^{*}$ $\displaystyle=\mathop{\arg\min}_{i}u_{i}/\log|Y_{i}|$
(7) $\displaystyle\mathcal{T}(x)$
$\displaystyle\approx\text{Pad}\left(T_{i^{*}}\left(x\right)\right)$ (8)
$\displaystyle\text{Pad}\left(T_{i^{*}}\left(y\mid x\right)\right)$
$\displaystyle=\left\\{\begin{array}[]{cl}T_{i^{*}}(y\mid x)&y\in Y_{i^{*}}\\\
0&y\in Y-Y_{i^{*}}\end{array}\right.$ (11)

where $\log|Y_{i}|$ is a normalizing factor. As $T_{i^{*}}$ only provides the
label relation over the class set $Y_{i^{*}}$, the probabilities of classes
not in $Y_{i^{*}}$ are set to zeros, denoted by the Pad operation. In this
way, we are actually set $\mathcal{T}(y\in Y_{i^{*}}\mid x)=1$, and thus the
student can learn from the teacher model that is most confident about $x$.

MUKI-Soft which estimates the golden supervision as a weighted sum of teacher
model predictions by taking the relative uncertain level into consideration:

$\displaystyle c_{i}$ $\displaystyle=1-u_{i}/\log|Y_{i}|$ (12) $\displaystyle
w_{i}$
$\displaystyle=\frac{\exp\left(c_{i}/\tau\right)}{\sum_{j=1}^{N}\exp(c_{j}/\tau)}$
(13) $\displaystyle\mathcal{T}(x)$
$\displaystyle\approx\sum_{i=1}^{N}w_{i}\text{Pad}\left(T_{i}\left(x\right)\right)$
(14)

where $c_{i}$ denotes the confidence score which indicates how likely $x$
belongs to $C_{i}$, and $\tau$ is a hyper-parameter for controlling the
smoothness of the weights. In this way, the teacher with a higher confidence
score contributes more to the estimated golden supervision signal. Besides,
the difference between the confidence scores reflects the inner correlation
between the classes in different label groups, thus providing extra
information for the classes in disjoint label sets.

####  2.2.3 Instance Re-weighting

Furthermore, the uncertainty distribution overlapping in the right part of
Figure 2 indicates that there is still a small portion of instances on which
teacher models achieve similar confidence levels with Monte-Carlo Dropout. For
these instances, MUKI-Hard may wrongly select the supervision source, and
MUKI-Soft would assign close weights to all teacher predictions, thus
providing a vague even conflicting supervision signal. To remedy this, we
devise an instance re-weighting mechanism by modifying the objective in Eq.
(1):

$\displaystyle\mathcal{L}_{\text{MUKI}}$
$\displaystyle=\sum_{x\in\mathcal{D}}v\left(x\right)\text{KL}\left(S\left(x\right)||\mathcal{T}(x)\right)$
(15) $\displaystyle v\left(x\right)$
$\displaystyle=c_{\text{max}}-c_{\text{sec}},$ (16)

where $c_{\text{max}}$ and $c_{\text{sec}}$ denotes the largest and the second
large teacher confidence score for instance $x$, respectively. By minimizing
the instance-level weighted objective, the student is encouraged to focus more
on the pivotal instances with clearer supervision signals, thus reducing the
effect of potential confusing instances.

In summary, MUKI consists of supervision estimation and instance re-weighting
mechanism based on model uncertainty for integrating the knowledge from
different teachers. Figure 3 gives an overview of MUKI framework.

##  3 Experiments

Method | Model Size | AG News | THUCNews | Google Snippets | 5Abstracts Group | Average
---|---|---|---|---|---|---
Supervised | 110M | 94.6 $\pm$ 0.00 | 97.8 $\pm$ 0.00 | 89.3 $\pm$ 0.00 | 90.7 $\pm$ 0.00 | 93.10
Teacher 1 | 110M | 49.9 $\pm$ 0.00 | 48.8 $\pm$ 0.00 | 50.2 $\pm$ 0.00 | 42.0 $\pm$ 0.00 | 47.73
Teacher 2 | 110M | 47.5 $\pm$ 0.00 | 49.8 $\pm$ 0.00 | 43.5 $\pm$ 0.00 | 51.5 $\pm$ 0.00 | 48.08
Ensemble | 220M | 59.8 $\pm$ 0.00 | 93.1 $\pm$ 0.00 | 80.4 $\pm$ 0.00 | 62.3 $\pm$ 0.00 | 73.90
Vanilla KD | 110M | 63.1 $\pm$ 0.81 | 94.9 $\pm$ 0.18 | 83.7 $\pm$ 1.30 | 67.0 $\pm$ 1.14 | 77.18
DFA | 110M | 66.4 $\pm$ 2.33 | 94.4 $\pm$ 0.22 | 82.6 $\pm$ 0.18 | 57.7 $\pm$ 4.41 | 72.78
CFL | 110M | 61.4 $\pm$ 1.18 | 95.1 $\pm$ 0.21 | 84.5 $\pm$ 0.45 | 61.6 $\pm$ 0.12 | 75.65
UHC | 110M | 78.8 $\pm$ 1.42 | 92.1 $\pm$ 0.63 | 86.3 $\pm$ 0.39 | 71.4 $\pm$ 0.67 | 82.15
MUKI-Hard (Ours) | 110M | 87.0 $\pm$ 0.40 |  97.2 $\pm$ 0.12 |  88.4 $\pm$ 0.32 | 79.0 $\pm$ 0.82 | 87.90
MUKI-Soft (Ours) | 110M |  87.1 $\pm$ 0.19 |  97.2 $\pm$ 0.08 | 87.9 $\pm$ 0.32 |  79.3 $\pm$ 0.85 | 87.88
Table 1: Comparisons on the benchmark datasets. The results are classification
accuracy averaged by three seeds, and standard deviations are reported. Both
MUKI variants achieve statistically significant improvements over the best-
performing baselines ($p<0.01$). Best results are shown in bold.

###  3.1 Experimental Settings

##### Datasets

We conduct evaluations on four classic text classification benchmarks,
including three English datasets: AG News (Zhang et al., 2015), Google
Snippets (Phan et al., 2008), 5Abstracts Group (Liu et al., 2018), and a
Chinese dataset THUCNews (Sun et al., 2016). We randomly split 5% of data from
the training set for datasets without a validation set to form a validation
set for model selection. The statistics of datasets can be found in Appendix
A.

##### Compared Methods

We implement various baselines to evaluate our proposal, as follows:

_Simple Baselines_ , which require no additional training, including: (1)
Original Teacher: The teacher models are used independently for prediction. We
set the probabilities of classes out of the teacher speciality to zeros. (2)
Ensemble: The output logits of teachers are directly concatenated for
predictions over the union label set.

_Distillation Methods_ , which assume internal states of the teacher model are
available and the student is trained via aligning the states of teacher models
on $\mathcal{D}$, including: (1) Vanilla KD (Hinton et al., 2015): The student
is trained to mimic the soft targets produced by logits combination of all
teacher models, via minimizing the vanilla KL-divergence objective. (2) DFA
(Shen et al., 2019): DFA designs a layer-wise feature adaptation mechanism for
providing extra guidance based on Vanilla KD. The student aligns its features
to the merged features of multiple teachers layer by layer. (3) CFL (Luo et
al., 2019): CFL first maps the hidden representations of the student and the
teachers into a common space. The student is trained by aligning the mapped
features to that of the teachers, with supplemental supervision from the
logits combination. (4) UHC (Vongkulbhisal et al., 2019): UHC splits the
student logits into subsets corresponding to the class sets of teacher models.
Each subset is trained to mimic the corresponding output of the teacher model.

A supervised learning method with labeled data is also included, to serve as a
performance upper-bound for better understanding of the results.

##### Implementation Details

We implement our framework using the HuggingFace transformers library (Wolf et
al., 2020). In our main setting, we set the teacher number $N$ to $2$ and we
explore integrating multiple teachers in Section 3.4. For each dataset, the
classes are randomly split into two non-overlapping parts, and two teachers
are fine-tuned on each set separately to imitate the actual applications.
Detailed class split can be found in Appendix A. The teacher and student
models for English datasets and THUCNews are BERT-base-uncased (Devlin et al.,
2019) and BERT-wwm-ext (Cui et al., 2020), respectively. We first fine-tune
the teacher models with the split labeled data for $3$ epochs with a learning
rate $2\times 10^{-5}$. The trained teacher model weights are frozen during
the student training process. We set the forward number $K$ of Monte-Carlo
Dropout uncertainty estimation to $16$ and the dropout rate is set to $0.1$.
Temperature $\tau$ in Eq. (13) is set to $0.2$ according to our hyper-
parameter analysis results in Appendix B. The student model then is learned by
optimizing the KL-divergence objective for $3$ epochs, with a $2\times
10^{-5}$ learning rate and $32$ batch size. The student is evaluated on the
validation set every $100$ step. We select the best performing checkpoints for
final evaluation. The results are replicated with $3$ random seeds and we
report the averaged accuracy.

###  3.2 Main Results

The model performance comparison on the four datasets and the corresponding
model size are listed in Table 1. Our findings are: (1) Simple baselines fall
far behind, showing that it is necessary to design a proper integration
strategy for amalgamating the knowledge from different teachers. (2) While
extra feature alignment objectives are adopted, DFA and CFL cannot achieve
consistent improvements over Vanilla KD. We speculate the reason is that the
supervision based on feature alignments is unstable, as teacher features are
fine-tuned for specializing in different semantic classes. (3) UHC achieves
better average results than Vanilla KD, while performing relatively worse on
the THUCNews dataset. It indicates there exists potential supervision conflict
as UHC matches the student output independently to that of teachers, thus
limiting its generalizability on different datasets. (4) Two variants of MUKI
both significantly outperform the previous baseline models on all the
datasets, and the average accuracy of MUKI-Hard is achieves a $5.75$ points
gain over the best performing baseline model. On the THUCNews dataset, while
no label information is included during the knowledge amalgamation, MUKI can
obtain a $97.2$ accuracy, which is very close to $97.8$ of the supervised
learning method. We attribute the success to that MUKI provides the student
with the accurately estimated golden probability distribution over the union
label set according to model uncertainty, which can effectively transfer the
knowledge and alleviate potential supervision conflicting. These promising
results indicate that our MUKI framework can produce better supervision for
training the student model, thus has great potentials for reusing PLMs with
different label sets.

###  3.3 Ablation Studies

Method | AG News | THUCNews
---|---|---
MUKI-Hard | 87.0 $\pm$ 0.40 | 97.2 $\pm$ 0.12
w/o Monte-Carlo Dropout | 65.1 $\pm$ 1.67 | 97.0 $\pm$ 0.13
w/o Instance Re-weighting | 85.5 $\pm$ 0.51 | 96.9 $\pm$ 0.06
MUKI-Soft | 87.1 $\pm$ 0.19 | 97.2 $\pm$ 0.08
w/o Monte-Carlo Dropout | 74.2 $\pm$ 0.28 | 95.4 $\pm$ 0.20
w/o Instance Re-weighting | 86.7 $\pm$ 0.30 | 96.8 $\pm$ 0.03
Table 2: Ablation analysis of MUKI. The removed modules both lead to
deteriorated performance.

We conduct ablation experiments on two large datasets, i.e., AG News and
THUCNews for stable results, to explore the following two questions.

How Monte-Carlo Dropout benefits knowledge source identification? We replace
the Monte-Carlo Dropout estimation of model estimation with a single forward
estimation, i.e., setting $K=1$ in Eq. (5). As shown in Table 2, we find that
the performance is degraded on both datasets. Interestingly, we find that the
accuracy drop is much clearer on the AG News than that on the THUCNews. To
explore this, we compute the average ECE score (Guo et al., 2017) of two
teacher models on the out-of-distribution samples, where higher ECE scores
indicate more severe over-confident predictions. The teacher models of AG News
achieve an average $45.42$ ECE score, while that of THUCNews is $19.52$.
Therefore, the teacher models of AG News exhibit a much more serious over-
confident issue than that of THUCNews. This result verifies that our adoption
of the Monte-Carlo Dropout technique is effective for accurately identifying
the adequate teacher model, especially when teachers tend to make over-
confident predictions.

Figure 4: Supervision quality measured by the KL-divergence to the golden
supervision of different methods. For MUKI-Soft, we divide instances into two
groups according to the uncertainty margin $v(x)$.

How instance-wise re-weighting benefits supervision integration? In Table 2,
we find that removing the instance re-weighting mechanism leads to
deteriorated results for both MUKI variants. We further probe whether the re-
weighting mechanism is capable of resolving the vague supervision issue when
teachers achieve similar uncertainty scores. Specifically, we train a PLM with
all labeled data as the proxy of an oracle model, which thus can provide
golden supervision over the union label set. We then calculate the KL-
divergence between the golden probability distribution and the approximated
one of different combination methods. Lower KL-divergence indicates the
combined predictions are more correct. We discard results of MUKI-Hard as KL-
divergence is not defined for distributions with zeros, and divide MUKI-Soft
into two groups according to the uncertainty score margin $v(x)$, i.e.
instances with $v(x)\geq 0.5$ and that with $v(x)<0.5$.

As shown in Figure 4, we find that the predictions of instances with $v(x)\geq
0.5$, are much closer to the golden distributions than the Vanilla KD. This
indicates that the estimated supervision of instances with a clearer
confidence margin is of higher quality, thus paying more attention to these
instances is effective for knowledge integration.

Method | 3 Teachers | 4 Teachers | 5 Teachers
---|---|---|---
{3,3,4} | {2,2,2,4} | {2,2,2,2,2}
Teacher 1 | 29.9 $\pm$ 0.00 | 20.0 $\pm$ 0.00 | 20.0 $\pm$ 0.00
Teacher 2 | 29.8 $\pm$ 0.00 | 19.1 $\pm$ 0.00 | 19.1 $\pm$ 0.00
Teacher 3 | 39.8 $\pm$ 0.00 | 19.9 $\pm$ 0.00 | 19.9 $\pm$ 0.00
Teacher 4 | N / A | 39.8 $\pm$ 0.00 | 20.0 $\pm$ 0.00
Teacher 5 | N / A | N / A | 20.0 $\pm$ 0.00
Ensemble | 91.0 $\pm$ 0.00 | 74.0 $\pm$ 0.00 | 80.6 $\pm$ 0.00
Vanilla KD | 92.5 $\pm$ 1.13 | 76.5 $\pm$ 0.57 | 82.6 $\pm$ 1.64
DFA | 91.1 $\pm$ 1.25 | 79.6 $\pm$ 1.34 | 82.0 $\pm$ 2.04
CFL | 93.9 $\pm$ 1.07 | 77.4 $\pm$ 1.34 | 84.0 $\pm$ 0.24
UHC | 84.4 $\pm$ 0.91 | 71.6 $\pm$ 2.86 | 69.4 $\pm$ 2.73
MUKI-Hard |  94.7 $\pm$ 0.20 | 93.4∗ $\pm$ 0.58 |  90.5∗ $\pm$ 0.32
MUKI-Soft |  94.7 $\pm$ 0.14 |  93.6∗ $\pm$ 0.30 |  90.5∗ $\pm$ 1.13
Table 3: Results of merging multiple teacher models on the THUCNews dataset. ∗
denotes the improvement over the best performing baseline is significant
($p<0.05$). N / A means that the teacher model does not exist in the
corresponding setting.

###  3.4 Results in Challenging Settings

##### KI with Multiple Teachers

As MUKI is agnostic to the number of teacher models, we explore its
adaptability with more teacher models. We conduct experiments on the THUCNews
dataset as it has $10$ classes, allowing us to train up to $5$ teacher models
specialized in different class sets. The $10$ classes are split into
$\\{3,3,4\\}$, $\\{2,2,2,4\\}$ and $\\{2,2,2,2,2\\}$ for $3$, $4$ and $5$
teachers, respectively. As shown in Table 3, the proposed MUKI framework
generalizes well to this setting, outperforming previous baselines with a
clear margin. Besides, we find that all baselines based on logits alignment
perform poorly under the $4$-teacher scenario. We attribute it to that when
some teacher models have more classes than others, they usually produce a
larger range of logits to make the prediction more distinguishable. Directly
combing the teacher logits thus leads to a biased probability distribution.
Our MUKI instead estimates the golden supervision according to model
uncertainty scores, thus producing better supervision even teacher models
exhibit different logit scales.

##### KI with Heterogeneous Teachers

As MUKI only operates on the output distribution level, it is generalizable
for heterogeneous teachers. We verify this by merging teachers with different
model structures. Specifically, we adopt BERT-base ($12$ layers and $768$
hidden units) and BERT-large ($24$ layers and $1024$ hidden units) as the
teachers, respectively. As shown in Table 4, we find that while a larger
teacher tends to perform better, the student model performs worse on the
THUCNews dataset than learning from two BERT-base teachers, indicating it is
challenging to integrate knowledge in this setting. Our MUKI achieves the best
results on these two datasets, showing its effectiveness for heterogeneous
teachers.

Method | _$\text{T}_{1}$ : BERT-base_ | _$\text{T}_{2}$ : BERT-large_
---|---|---
AG News | THUCNews
Teacher 1 | 49.7 $\pm$ 0.00 | 48.8 $\pm$ 0.00
Teacher 2 | 47.2 $\pm$ 0.00 | 49.8 $\pm$ 0.00
Ensemble | 76.6 $\pm$ 0.00 | 79.3 $\pm$ 0.00
Vanilla KD | 79.6 $\pm$ 0.22 | 82.9 $\pm$ 1.36
DFA | 78.2 $\pm$ 0.30 | 84.7 $\pm$ 1.74
CFL | 75.9 $\pm$ 0.63 | 81.9 $\pm$ 1.37
UHC | 78.3 $\pm$ 2.65 | 92.3 $\pm$ 1.08
MUKI-Hard | 78.3 $\pm$ 1.58 |  95.4∗ $\pm$ 0.45
MUKI-Soft |  80.6 $\pm$ 0.17 |  95.4∗ $\pm$ 0.29
Table 4: Results of merging BERT-base and BERT-large. ∗ denotes results are
statistically significant ($p<0.05$).

##### KI with Cross-Dataset Teachers

Specifically, we fine-tune teacher models on different datasets separately and
then train a student to perform classification over the union label set of
both datasets. The multilingual BERT-base is adopted for the teachers and the
student in the cross-lingual setting. The results of merging knowledge from
two English datasets, AG News and Google-Snippets and even cross-lingual
datasets, AG News (in English) and THUCNews (in Chinese) are listed in Table
5. We find that MUKI still outperforms previous baseline models in both
settings. Interestingly, we find that the MUKI-Hard is consistently better
than MUKI-Soft in this setting. We speculate the reason is that the
correlations between classes of different datasets are weak, thus modeling the
label relation in these disjoint groups is unnecessary.

Method | _$\text{T}_{1}$ : AG News (en)_ | _$\text{T}_{1}$ : AG News (en)_
---|---|---
_$\text{T}_{2}$ : GS (en)_ | _$\text{T}_{2}$ : THUCNews (zh)_
Teacher 1 | 26.8 $\pm$ 0.00 | 0.50 $\pm$ 0.00
Teacher 2 | 63.0 $\pm$ 0.00 | 54.6 $\pm$ 0.00
Ensemble | 74.3 $\pm$ 0.00 | 54.5 $\pm$ 0.00
Vanilla KD | 75.1 $\pm$ 0.58 | 54.6 $\pm$ 0.05
DFA | 74.7 $\pm$ 0.30 | 54.1 $\pm$ 0.20
CFL | 74.0 $\pm$ 0.50 | 54.3 $\pm$ 0.65
UHC | 72.0 $\pm$ 0.73 | 53.5 $\pm$ 0.17
MUKI-Hard |  76.3 $\pm$ 0.54 |  68.1∗ $\pm$ 0.20
MUKI-Soft | 75.9 $\pm$ 0.47 | 65.6∗ $\pm$ 0.40
Table 5: Cross-dataset results. GS is short for Google Snippets. (Left Column)
Integrating teachers major in AG News and GS, respectively. (Right Column)
Merging teachers major in AG News (in English) and THUCNews (in Chinese),
respectively. ∗ denotes results are statistically significant with $p<0.05$.

###  3.5 Results for Structured Prediction

Method | CoNLL 2003 | OntoNotes 5.0
---|---|---
Teacher 1 | 55.4 $\pm$ 0.00 | 46.20 $\pm$ 0.00
Teacher 2 | 38.6 $\pm$ 0.00 | 25.34 $\pm$ 0.00
Ensemble | 79.5 $\pm$ 0.00 | 49.85 $\pm$ 0.00
Vanilla KD | 83.6 $\pm$ 0.66 | 53.33 $\pm$ 0.00
DFA | 83.9 $\pm$ 0.45 | 52.66 $\pm$ 0.13
CFL | 83.1 $\pm$ 0.79 | 53.39 $\pm$ 0.63
UHC | 81.4 $\pm$ 0.24 | 55.17 $\pm$ 1.20
MUKI-Hard |  85.3∗ $\pm$ 0.26 |  59.83∗∗ $\pm$ 0.31
MUKI-Soft | 84.5 $\pm$ 0.30 | 59.33 $\pm$ 0.80
Table 6: F1-score results on two NER datasets. The results are statistically
significant (∗ for $p<0.05$ and ∗∗ for $p<0.01$). MUKI outperforms all the
baselines significantly, verifying its generalizability for complicated tasks
like structured prediction.

We extend the KI framework into a classic structured prediction task, i.e.,
named entity recognition (NER). The problem is modeled as a tagging problem
following Devlin et al. (2019), and we conduct evaluations on CoNLL 2003 (Sang
and De Meulder, 2003) and OntoNotes 5.0 (Pradhan et al., 2013). Specifically,
we split the entity types of the dataset into two groups and train two
teachers responsible for identifying the entities in each group, respectively.
We refer readers to Appendix A for the detailed dataset statistics and the
division of entity types. We adapt MUKI to the NER task by estimating the
model uncertainty and integrating the predictions at the token level. Besides,
we notice that the teacher will predict the non-entity tag with high
confidence for tokens of entities it cannot handle. Therefore, we adjust the
uncertainty estimation procedure by calculating the entropy of probability
distribution over the tags of entity types. The knowledge integration remains
the same with the classification problem. As shown in the in Table 6, our MUKI
still performs the best among all methods, validating that the proposed
framework is generalizable for structured predictions tasks like NER.

##  4 Related Work

Our work is mainly related to knowledge distillation (KD), which aims to
transfer the knowledge from a teacher model to a student model. Hinton et al.
(2015) utilize the soft labels of the teacher model for the student to learn,
and Romero et al. (2015) align the internal representations between the
student and the teacher. Recent studies apply KD for PLMs successfully by
matching the intermediate states (Sun et al., 2019; Wang et al., 2020) and
enriching the training data with data augmentation (Jiao et al., 2020; Liang
et al., 2021), learning from multiple teachers (Wu et al., 2021), and
dynamically adjusting the learning objectives (Li et al., 2021). Nevertheless,
all these KD studies assume that the student has an identical label set with
the target teacher model(s). Instead, knowledge integration removes this
restriction by merging knowledge from multiple teacher with various label sets
to train a versatile student model.

Recently, the idea of integrating knowledge from models with different skills
has been explored in computer vision (Shen et al., 2019; Ye et al., 2019; Luo
et al., 2019; Vongkulbhisal et al., 2019) and graph neural networks (Jing et
al., 2021), or extended to a semi-supervised setting (Thadajarassiri et al.,
2021). To the best of our knowledge, we are the first to explore knowledge
integration for PLMs, which is of great practical value as there are abundant
released PLMs. Besides, different from previous methods relying on supervision
from feature alignments (Shen et al., 2019; Luo et al., 2019) or independent
logits matching (Vongkulbhisal et al., 2019), our MUKI framework operates on
the distribution level by utilizing model uncertainty to approximate the
golden supervision. Therefore, MUKI is more effective and generalizable for
integrating knowledge from heterogeneous PLMs.

##  5 Error Analysis

The MUKI is built on the assumption that the model uncertainty estimation can
faithfully reflect the ability of teacher model. We perform an error analysis
to investigate when this assumption will fail, i.e, the estimated model
uncertainty misguides the teacher selection. Specifically, we probe the label
distribution of instances on which MUKI assigns a higher uncertainty score to
the correct teacher on THUCNews. As shown in the left part of Figure 5, the
uncertainty-based teacher selection only fails on a small portion of training
examples, i.e., $2.8$% of total training instances. Interestingly, the labels
of these instances are not uniformly distributed, e.g., _Estate_ and _Tech._
have higher error rates than other classes. We further plot the label
confusion matrix of an oracle model that is fine-tuned with labeled data with
all categories, in the right part of Figure 5. We find that there are classes
that can even confuse the oracle model, e.g., instances of _Estate_ are
tending to be classified into _Politics_ and _Finance_ , indicating that the
mis-identification is partially due to the inherent class similarity. These
findings suggest that the performance MUKI can be limited when the integrating
teacher models whose classes are highly correlated. As a remedy, the proposed
instance re-weighting mechanism is an effective ad-hoc strategy, as the
average teacher uncertainty margin $v(x)$ is $0.04$, which can greatly reduce
the negative impact of these instances. Developing better model uncertainty
estimation techniques like incorporating more class information into the
estimation process is also promising. Besides, the uncertainty estimation can
also be influenced by the model capacity. As deeper models tend to produce
more confident predictions, the model selection based on uncertainty scores
will favor stronger teacher models when there exist particularly weak teacher
models. In such a case, applying model calibration techniques like temperature
scaling according to the teacher model size (Guo et al., 2017; Desai and
Durrett, 2020) before estimating the uncertainty can be beneficial.

##  6 Conclusion

In this paper, we explore knowledge integration for PLMs to promote better
model reuse. We present MUKI, which integrates teacher predictions according
to the model uncertainty estimated via Monte-Carlo Dropout, and dynamically
adjusts the instance contribution according to the uncertainty margin.
Extensive results on benchmark datasets demonstrate that MUKI can
substantially outperform strong baselines, and perform well in challenging
settings such as merging heterogeneous teachers. Further investigation shows
that MUKI can be extended to sequence labeling. In the future, we are
interested in developing better integration frameworks for more complex tasks.

##  7 Ethical Considerations

Our work faces several ethical challenges. As the released PLMs may exhibit
potential biases against specific groups, e.g., gender or ethnic minorities
(Kurita et al., 2019; Kennedy et al., 2020), these social biases can be
propagated to the merged student model. Besides, users may collect unlabeled
data from the web for conducting knowledge integration, which possibly
contains offensive content and thus introduces new biases into the merged
student model as well.

Figure 5: (Left) Label distribution of instances with wrongly selected
teachers. Labels in the same color indicate they are in the same teacher
specialty. (Right) The confusion matrix of an oracle model.

We offer possible remedies to reduce the concerns. For the biases exhibited in
the teacher PLMs, de-biasing techniques (Zmigrod et al., 2019; Liang et al.,
2020; Schick et al., 2021) can be applied to eliminate the potential biases in
the teachers before integration. For the offensive unlabeled data collected
from the internet, simple template-based or human-in-the-loop data cleaning
strategies can be adopted, to identify and filter potential biased data.
Except for these techniques, developing a bias-aware knowledge transfer
framework that can de-bias the supervision for the student model while
maintaining task performance is also promising (Gupta et al., 2022).

## Acknowledgements

We thank all the anonymous reviewers for their constructive comments, and
Shuhuai Ren for his valuable suggestions in preparing the manuscript. This
work was supported by a Tencent Research Grant. Xu Sun is the corresponding
author.

## References

  * Blundell et al. (2015) Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. 2015.  Weight uncertainty in neural network.  In _Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015_ , volume 37 of _JMLR Workshop and Conference Proceedings_ , pages 1613–1622. JMLR.org.
  * Cui et al. (2020) Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2020\.  Revisiting pre-trained models for Chinese natural language processing.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 657–668, Online. Association for Computational Linguistics.
  * Desai and Durrett (2020) Shrey Desai and Greg Durrett. 2020.  Calibration of pre-trained transformers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 295–302, Online. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani. 2016.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016_ , volume 48 of _JMLR Workshop and Conference Proceedings_ , pages 1050–1059. JMLR.org.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017.  On calibration of modern neural networks.  In _Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 1321–1330. PMLR.
  * Gupta et al. (2022) Umang Gupta, J. Dhamala, Varun Kumar, Apurv Verma, Yada Pruksachatkun, Satyapriya Krishna, Rahul Gupta, Kai-Wei Chang, Greg Ver Steeg, and A. G. Galstyan. 2022.  Mitigating gender bias in distilled language models via counterfactual role reversal.  _ArXiv_ , abs/2203.12574.
  * Hendrycks and Gimpel (2017) Dan Hendrycks and Kevin Gimpel. 2017.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015.  Distilling the knowledge in a neural network.  _ArXiv preprint_ , abs/1503.02531.
  * Jiao et al. (2020) Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020.  TinyBERT: Distilling BERT for natural language understanding.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4163–4174, Online. Association for Computational Linguistics.
  * Jing et al. (2021) Yongcheng Jing, Yiding Yang, Xinchao Wang, Mingli Song, and Dacheng Tao. 2021.  Amalgamating knowledge from heterogeneous graph neural networks.  In _CVPR_ , pages 15709–15718.
  * Kennedy et al. (2020) Brendan Kennedy, Xisen Jin, Aida Mostafazadeh Davani, Morteza Dehghani, and Xiang Ren. 2020.  Contextualizing hate speech classifiers with post-hoc explanation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5435–5442, Online. Association for Computational Linguistics.
  * Kurita et al. (2019) Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019.  Measuring bias in contextualized word representations.  In _Proceedings of the First Workshop on Gender Bias in Natural Language Processing_ , pages 166–172, Florence, Italy. Association for Computational Linguistics.
  * Li et al. (2021) Lei Li, Yankai Lin, Shuhuai Ren, Peng Li, Jie Zhou, and Xu Sun. 2021.  Dynamic knowledge distillation for pre-trained language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 379–389, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Liang et al. (2021) Kevin J. Liang, Weituo Hao, Dinghan Shen, Yufan Zhou, Weizhu Chen, Changyou Chen, and Lawrence Carin. 2021.  Mixkd: Towards efficient distillation of large-scale language models.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Liang et al. (2020) Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2020.  Towards debiasing sentence representations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5502–5515, Online. Association for Computational Linguistics.
  * Liu et al. (2018) Qian Liu, Heyan Huang, Yang Gao, Xiaochi Wei, Yuxin Tian, and Luyang Liu. 2018.  Task-oriented word embedding for text classification.  In _Proceedings of the 27th International Conference on Computational Linguistics (COLING)_ , Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized BERT pretraining approach.  _ArXiv preprint_ , abs/1907.11692.
  * Luo et al. (2019) Sihui Luo, Xinchao Wang, Gongfan Fang, Yao Hu, Dapeng Tao, and Mingli Song. 2019\.  Knowledge amalgamation from heterogeneous networks by common feature learning.  In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019_ , pages 3087–3093. ijcai.org.
  * Phan et al. (2008) Xuan Hieu Phan, Minh Le Nguyen, and Susumu Horiguchi. 2008.  Learning to classify short and sparse text & web with hidden topics from large-scale data collections.  In _Proceedings of the 17th International Conference on World Wide Web, WWW 2008, Beijing, China, April 21-25, 2008_ , pages 91–100. ACM.
  * Pradhan et al. (2013) Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013.  Towards robust linguistic analysis using ontonotes.  In _Proceedings of the Seventeenth Conference on Computational Natural Language Learning_ , pages 143–152.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _JMLR_ , 21:140:1–140:67.
  * Romero et al. (2015) Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. 2015.  Fitnets: Hints for thin deep nets.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Sang and De Meulder (2003) Erik F Sang and Fien De Meulder. 2003.  Introduction to the conll-2003 shared task: Language-independent named entity recognition.  _arXiv preprint cs/0306050_.
  * Schick et al. (2021) Timo Schick, Sahana Udupa, and Hinrich Schütze. 2021.  Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in nlp.  _ArXiv preprint_ , abs/2103.00453.
  * Schwartz et al. (2020) Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. 2020.  Green ai.  _Communications of the ACM_ , 63(12):54–63.
  * Shen et al. (2019) Chengchao Shen, Xinchao Wang, Jie Song, Li Sun, and Mingli Song. 2019.  Amalgamating knowledge towards comprehensive classification.  In _The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019_ , pages 3068–3075. AAAI Press.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: a simple way to prevent neural networks from overfitting.  _JMLR_ , 15(1):1929–1958.
  * Strubell et al. (2019) Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019.  Energy and policy considerations for deep learning in NLP.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3645–3650, Florence, Italy. Association for Computational Linguistics.
  * Sun et al. (2016) Maosong Sun, Jingyang Li, Zhipeng Guo, Z Yu, Y Zheng, X Si, and Z Liu. 2016.  Thuctc: an efficient chinese text classifier.  _GitHub Repository_.
  * Sun et al. (2019) Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019.  Patient knowledge distillation for BERT model compression.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4323–4332, Hong Kong, China. Association for Computational Linguistics.
  * Thadajarassiri et al. (2021) Jidapa Thadajarassiri, Thomas Hartvigsen, Xiangnan Kong, and Elke A. Rundensteiner. 2021.  Semi-supervised knowledge amalgamation for sequence classification.  In _AAAI_ , pages 9859–9867.
  * Vongkulbhisal et al. (2019) Jayakorn Vongkulbhisal, Phongtharin Vinayavekhin, and Marco Visentini Scarzanella. 2019.  Unifying heterogeneous classifiers with distillation.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_ , pages 3175–3184. Computer Vision Foundation / IEEE.
  * Wang et al. (2020) Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020.  Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Wu et al. (2021) Chuhan Wu, Fangzhao Wu, and Yongfeng Huang. 2021.  One teacher is enough? pre-trained language model distillation from multiple teachers.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4408–4413, Online. Association for Computational Linguistics.
  * Ye et al. (2019) Jingwen Ye, Yixin Ji, Xinchao Wang, Kairi Ou, Dapeng Tao, and Mingli Song. 2019\.  Student becoming the master: Knowledge amalgamation for joint scene parsing, depth estimation, and more.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_ , pages 2829–2838. Computer Vision Foundation / IEEE.
  * Zhang et al. (2015) Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  In _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_ , pages 649–657.
  * Zmigrod et al. (2019) Ran Zmigrod, Sabrina J. Mielke, Hanna Wallach, and Ryan Cotterell. 2019.  Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1651–1661, Florence, Italy. Association for Computational Linguistics.

##  Appendix A Datasets Details

The label sets of datasets we used in the main paper are first sorted
according to the name and will be evenly divided into subsets according to the
number of teacher models. Table 7 gives the dataset statistics and the class
number for two teacher models experiments. Table 8 gives the label (or entity
types for NER datasets) list on each dataset. For teacher number that cannot
evenly dividing the label sets, the final label set will include the left
labels. For example, when there are $4$ teacher models are needed on THUCNews
dataset, the label set will be split into {_Sports_ , _Enter._}, {_Furniture_
, _Estate_}, {_Education_ , _Fashion_} and {_Politics_ , _Game_ ,_Tech._ ,
_Finance_}.

Dataset | #Class (Ent.) | #Train | #Test | $\\{|Y|\\}$
---|---|---|---|---
AG News | 4 | 120k | 7.6k | {2, 2}
5Abstracts Group | 5 | 53k | 1k | {2, 3}
Google Snippets | 8 | 10k | 2.2k | {4, 4}
THUCNews | 10 | 50k | 10k | {5, 5}
CoNLL 2003 | 4 | 14k | 3.5k | {2, 2}
Ontonotes v5 | 18 | 115.8k | 12.2k | {9, 9}
Table 7: Statistics of datasets used in our paper. Ent. denotes the entity types and $\\{|Y|\\}$ is the number of classes each teacher model specializes. Dataset | Label Order
---|---
AG News | World, Sports, Business, Sci/Tech
THUCNews | Sports, Entertainment, Furniture,
Estate, Education, Fashion,
Politics, Game, Technology, Finance
Google Snippets | Business, Computers,
Culture-Arts-Entertainment,
Education-Science, Engineering,
Health, Politics-Society, Sports
5Abstracts Group | Business, CSAI, Law, Sociology, Trans
CoNLL 2003 | PER, MISC, ORG, LOC
OntoNotes 5.0 | LAW, TIME, DATE, LOC,
ORG, GPE, NORP, LANGUAGE,
EVENT, CARDINAL, MONEY,
PRODUCT, PERCENT, FAC,
PERSON, QUANTITY,
WORK OF ART, ORDINAL
Table 8: Sorted label names (entity types) of datasets. Label names of
THUCNews are translated into English.

##  Appendix B Hyper-parameter Search for $\tau$

We perform a hyper-parameter search experiment for the optimal $\tau$ in MUKI-
soft. We conduct experiments on AG News and THUCNews for stable results. The
values of $\tau$ are picked from $\\{0.01,0.1,0.2,0.5,1.0,2.0,5.0,10.0\\}$,
and the results are shown in Figure 6. We observe that the accuracy drops
significantly when $\tau$ is set to high values, where the teacher weights
distribution is becoming a uniform distribution, while it reaches a peak when
$\tau$ is set to a small value between $0.2$ and $0.5$. It indicates that
slightly sharpening the teacher weights distribution is helpful for KI.
Therefore, we adopt $\tau=0.2$ in all the experiments.

Figure 6: Varying temperature $\tau$ for MUKI-Soft. The average accuracy of
three seeds are plotted with standard deviation in shade.