# Localized excitons in 1D–dimensional half–filled paramagnetic Hubbard model.

N. I. Chashchin
Ural State Forestry University
Ekaterinburg, Sibirskii trakt 37, 620100 Russia E–mail:
<EMAIL_ADDRESS>
###### Abstract

By the example of the Hubbard model we analytically and numerically examine
the formating and coexisting of localized electron–electron pairs (doublons)
and localized electron–hole pairs (Frenkel–type excitons) . Here we
demonstrate that at a variation of the on-site Coulomb repulsion U there
occurs a quantum transition from the doublon to the exciton regime that is
conditioned by the low-energy effective hybridization of the valence and
conductivity subbands. We calculate momentum distribution functions and
electronic spectrum functions for different U, which reveal topologically
nontrivial behaviour at the Fermi level.

Keywords: Hubbard model, electronic spectrum function, exciton, doubly
occupied sites, momentum distribution function, nontrivial topology.

##  1 . Introduction

Interplay of band theory and strong electron correlation effects has been a
basic phenomenon for understanding physics of many body models and materials.
Electron repulsion due to local Coulomb interactions tends to localize
electrons, on the other hand, kinetic effects are favourable to electron
itineracy. The energetic competition between them lead to different phases of
comparable magnitude.

The single band Hubbard model (HM) [1] with one electron per site
(half–filled) is the simplest model and prototypical example for investigation
of correlated electrons. The initial band of noninteracting electrons is
splitted into two subbands in correlated state: the lower – valence band and
the upper – conductivity band. The electronic spectrum of a crystal can be
organized in the form of energy bands as a function of the crystal momentum k,
which is guaranteed to be a good quantum number due to the translational
symmetry of the lattice. When an electron is excited from the valence into the
conductivity band it leaves behind a vacancy – a hole. In a divalent crystal
[2] electrons are concentrated near minimums of the conductivity band and
holes near maximums of the valence band. Nearby these extremums band energy
spectra in simple form are $E(k)=\hbar^{2}k^{2}/2m^{*}$, where $m^{*}$ is an
effective mass of the electrons or the holes correspondingly. The vacant place
(hole) in the filled valence band has the negative mass in definition, though
it is convenient to consider this hole as a real particle with the positive
mass and charge.

The double occupancy is a correlation function of a local pair of electrons
with antiparallel spins $\langle n_{\uparrow}n_{\downarrow}\rangle$. The
parameter plays an important role in the study of correlated systems, the Mott
metal–insulator transition [3], the semiconductor - semimetal mixing [4],
quantum phase transition [5], local moment formation, transport properties
[6]. The local Coulomb repulsion disfavors doubly-occupied sites, and
decreases the double occupancy [7, 8].

It is appropriate to assume that there also has to be a local bound state of
an electron in the conductivity band and a hole in the valence band which are
attracted to each other by the Coulomb force. The localazed on a site
Frenkel–type exciton $\langle(1-n_{\sigma})n_{\bar{\sigma}}\rangle$ is an
electrically neutral quasiparticle that exists in various electronic phases of
the model [9, 10, 11]. The exciton is regarded, as an elementary excitation of
condensed matter, which can transport energy without transporting electric
charge. The correlation effects caused by these bosonic–type single quantum
states are of an interest in the study of electronic systems.

##  2 . Theoretical Model and Method

Consider the Hubbard model in the representation we got in our previous works
[12, 13, 14, 15] by using the method of generating functional of Green
functions with the subsequent Legendre transformation. The Hamiltonian of the
half-filled and symmetrical Hubbard model is

$\mathcal{H}=-t\sum\limits_{\langle
i,j\rangle\sigma}c_{i\sigma}^{{\dagger}}c_{j\sigma}+\sum\limits_{i\sigma}\varepsilon_{\sigma}n_{i\sigma}+U\sum\limits_{i}n_{i\uparrow}n_{j\downarrow}\,,$
(2.1)

where U is the parameter of the Coulomb interaction at a site;
$c_{i\sigma}(c_{i\sigma}^{\dagger})$ are fermion operators that describe the
annihilation (generation) of electrons with spins up and down
($\sigma=\uparrow,\downarrow$); $n_{i\sigma}$ indicates the operators of the
number of particles; $t$ is the parameter of hopping of electrons from site to
site; and in the designation $\langle i,j\rangle$ crystall sites are the
nearest, $\varepsilon_{\sigma}=-\displaystyle{\sigma\frac{h}{2}-\mu}$, where
$h=g\mu_{B}H$, $g$ is the electron $g$–factor, $\mu_{B}$ is the Bohr magneton,
$H$ is the external magnetic field, and $\mu$ is the chemical potential.

The general equations derived in [15] are valid for different solutions —
paramagnetic or magnetic. Restricting ourselves to the paramagnetic case, we
get connected systems of equations for two types of Green functions.

The fermionic charge propagator ($N=2\,G$)

$\displaystyle N({\bf
k},i\omega_{n})=\frac{2}{i\omega_{n}-\varepsilon_{k}-\Sigma({\bf
k},i\omega_{n})}\,,$ (2.2)

where $\varepsilon_{k}=-\cos(k)$ is the 1D electronic free spectrum,
$\omega_{n}=(2n+1)\,\pi T$ ($n=0,\pm 1,\pm 2,\dots$) are fermionic Matsubara
frequencies, and $\Sigma({\bf k},i\omega_{)}$ is the fermionic particle self
energy.

The bosonic propagator

$\displaystyle Q({\bf
q},i\Omega_{\nu})=-\frac{1}{1+\frac{U}{2}\Pi(q,i\Omega_{\nu})}\,,$ (2.3)

where $\Omega_{\nu}=2\,\nu\,\pi T$ ($\nu=0,\pm 1,\pm 2,\dots$) are bosonic
Matsubara frequencies, and $\Pi({\bf q},i\Omega_{\nu})$ is the bosonic
particle self energy.

The Green function of electrons (2.2) lets the representation of the charge
propagator as $N=G_{-}+\,G_{+}$, and the band-to-band propagator
$N^{\,{}^{\prime}}=G_{-}-\,G_{+}$, where the indexes $\pm$ correspond to
separation of the initial electron band into the valence and conductivity
subbands.

$\begin{array}[]{l}\displaystyle
G_{-}(k,i\omega_{n})=\frac{1}{i\,\omega_{n}-\varepsilon_{k}-\displaystyle\Sigma_{-}(k,i\omega_{n})}\,,\\\
{}\\\ \displaystyle
G_{+}(k,i\omega_{n})=\frac{1}{i\,\omega_{n}+\varepsilon_{k}-\displaystyle\Sigma_{+}(k,i\omega_{n})}\,;\end{array}$
(2.4)

The electronic self energy part is evaluated by the next set of equations

$\left\\{\begin{array}[]{l}\displaystyle\Im\Sigma_{+}(k,\omega)=\frac{U}{4}\sum\limits_{q^{\prime}}\displaystyle\left[\left(1-\tanh(\frac{\varepsilon_{q^{\prime}}}{2T})\displaystyle\tanh(\frac{\varepsilon_{q^{\prime}}-\omega}{2T})\right)\right.\displaystyle\Im
Q_{-}(q^{\prime}-k;\varepsilon_{q^{\prime}}-\omega)+\\\ \hskip
113.81102pt\displaystyle\left.\left(1-\tanh(\frac{\varepsilon_{q^{\prime}}}{2T})\displaystyle\tanh(\frac{\varepsilon_{k^{\prime}}+\omega}{2T})\right)\displaystyle\Im
Q_{+}(q^{\prime}-k;\,-\varepsilon_{k^{\prime}}-\omega)\right]\,,\\\ {}\\\
\displaystyle\Re\Sigma_{+}(k,\omega)=\frac{1}{\pi}\int_{-\infty}^{\infty}\displaystyle\frac{\Im\Sigma_{+}(k,\omega^{\prime})}{\omega^{\prime}-\omega}\,d\omega^{\prime}\,;\\\
{}\\\
\displaystyle\Im\Sigma_{+}(k+\pi,\omega)=\Im\Sigma_{-}(k,\omega),\quad\Re\Sigma_{+}(k+\pi,\omega)=\Re\Sigma_{-}(k,\omega)\,;\\\
{}\\\
\displaystyle\Im\Sigma_{+}(k,-\omega)=\Im\Sigma_{-}(k,\omega),\quad\Re\Sigma_{+}(k,-\omega)=-\,\Re\Sigma_{-}(k,\omega)\,.\end{array}\right.$
(2.5)

The bosonic self energy part is evaluated by the equations

$\left\\{\begin{array}[]{l}\displaystyle\Im\Pi_{+}(q,\Omega)=\frac{U}{4}\sum\limits_{k^{\prime}}\displaystyle\left[\left(1-\tanh(\frac{\varepsilon_{k^{\prime}}}{2T})\displaystyle\tanh(\frac{\varepsilon_{k^{\prime}}-\Omega}{2T})\right)\right.\Im
G_{-}(k^{\prime}-q;\varepsilon_{k^{\prime}}-\Omega)+\\\ \hskip
93.89418pt\displaystyle\left.\left(1-\tanh(\frac{\varepsilon_{k^{\prime}}}{2T})\displaystyle\tanh(\frac{\varepsilon_{k^{\prime}}+\Omega}{2T})\right)\Im
G_{+}(k^{\prime}-q;-\varepsilon_{k^{\prime}}-\Omega)\right],\\\ {}\\\
\displaystyle\Re\Pi_{+}(q,\Omega)=\frac{1}{\pi}\int_{-\infty}^{\infty}\displaystyle\frac{\tanh(\frac{\Omega^{\prime}}{2T})\Im\Pi_{+}(q,\Omega^{\prime})}{\Omega^{\prime}-\Omega}\,d\Omega^{\prime}\,;\\\
{}\\\ \Im
Q_{+}(q;\Omega)=\displaystyle\frac{\,\Im\Pi_{+}(q;\Omega)}{\left[1+\Re\Pi_{+}(q;\Omega)\right]^{2}+\left[\Im\Pi_{+}(q;\Omega)\tanh(\frac{\Omega}{2T})\right]^{2}}\,;\\\
{}\\\
\displaystyle\Im\Pi_{+}(k+\pi,\omega)=\Im\Pi_{-}(k,\omega),\quad\Re\Pi_{+}(k+\pi,\omega)=\Re\Pi_{-}(k,\omega)\,;\\\
{}\\\
\displaystyle\Im\Pi_{+}(k,-\omega)=-\,\Im\Pi_{-}(k,\omega),\quad\Re\Pi(k,-\omega)=\Re\Pi_{-}(k,\omega)\,.\end{array}\right.$
(2.6)

Thus, we get two coupled sets of the integral equations (2.5, 2.6) which can
be numerically calculated and investigated. As we see, they are structurally
identical over the mutual substitutions: ${\Im N}\leftrightarrow{\Im Q}$,
${\Im\Sigma}\leftrightarrow{\Im\Pi}$, and
${\Re\Sigma}\leftrightarrow{\Re\Pi}$.

The imaginary and real parts of the Green functions obey to the next symmetry
properties:

$\begin{array}[]{l}\displaystyle\Im G_{+}(k\pm\pi,\omega)=\Im
G_{-}(k,\omega),\quad\Re G_{+}(k\pm\pi,\omega)=\Re G_{-}(k,\omega);\\\
\displaystyle\Im G_{+}(k,-\omega)=\Im G_{-}(k,\omega),\quad\Re
G_{+}(k,-\omega)=-\>\Re G_{-}(k,\omega)\,.\end{array}$ (2.7)

##  3 . Results and Discussion

A. Double occupancy states and local excitons

Write down the two–particle Green function in the explicit form

$\displaystyle\langle\hat{T}\,n_{1\uparrow}n_{1\downarrow}\rangle=\displaystyle\langle(n_{1\uparrow}-\frac{1}{2})(n_{1\downarrow}-\frac{1}{2})\rangle=\langle
n_{1\uparrow}n_{1\downarrow}\rangle-\frac{\langle
n_{1}\rangle}{2}+\frac{1}{4}\,,$ (3.1)

as in our symmetrical representation
$\displaystyle\hat{T}n_{1\sigma}=n_{1\sigma}-\frac{1}{2}$.

From the other hand [12],

$\displaystyle\langle\hat{T}\,n_{1\uparrow}n_{1\downarrow}\rangle=\frac{1}{4}\left[\frac{\delta^{2}\Phi}{\delta\rho(11)\,{\delta\rho(11)}}-\frac{\delta^{2}\Phi}{\delta\eta(11)\,{\delta\eta(11)}}\right]+\frac{1}{4}\left[\left(\frac{\delta\Phi}{\delta\rho(11)}\right)^{2}-\left(\frac{\delta\Phi}{\delta\eta(11)}\right)^{2}\right]\,,$
(3.2)

where $\displaystyle\frac{\delta\Phi}{\delta\rho(11)}=\langle n_{1}\rangle-1$,
$\displaystyle\frac{\delta\Phi}{\delta\eta(11)}=\langle
n_{1\uparrow}-n_{1\downarrow}\rangle$; $\Phi=\ln Z$ is the generated
functional of the connected Green functions; $1\equiv(i,\tau)$, i–a crystal
site, $\tau$–an imaginary time; $\eta(12)=h\,\delta_{12}$ is the external
magnetic field; $M_{1}=\langle n_{1\uparrow}-n_{1\downarrow}\rangle$ — the
local magnetic moment.

As we see from (3.2), determinal factors for the propagator of double occupied
sites are two–particle Green functions:

$\displaystyle\langle\hat{T}\,M_{1}^{2}\rangle=\frac{\delta^{2}\Phi}{\delta\eta(11)\,{\delta\eta(11)}},\;\;\displaystyle\langle\hat{T}\,n_{1}^{2}\rangle=\frac{\delta^{2}\Phi}{\delta\rho(11)\,{\delta\rho(11)}}\,.$
(3.3)

Pauli exclusion principle in the functional form [12] is

$\displaystyle\frac{\delta^{2}\Phi}{\delta\rho(11)\,{\delta\rho(11)}}+\frac{\delta^{2}\Phi}{\delta\eta(11)\,{\delta\eta(11)}}+\left(\frac{\delta\Phi}{\delta\rho(11)}\right)^{2}+\left(\frac{\delta\Phi}{\delta\eta(11)}\right)^{2}=0,$

which allow to write (3.2) in the form

$\displaystyle\langle\hat{T}\,n_{1\uparrow}n_{1\downarrow}\rangle=-\frac{1}{2}\,\frac{\delta^{2}\Phi}{\delta\eta(11)\,{\delta\eta(11)}}-\frac{1}{2}(\langle
M_{1}\rangle)^{2}\,.$ (3.4)

Comparing two expressions (3.1, 3.4) we get general formula for the number of
double–occupied sites

$\displaystyle\langle n_{1\uparrow}n_{1\downarrow}\rangle=\frac{\langle
n_{1}\rangle}{2}-\frac{1}{4}-\frac{(\langle
M_{1}\rangle)^{2}}{2}-\frac{\langle\hat{T}\,M^{2}_{1}\rangle}{2}\,.$ (3.5)

The mathematical expression for $\langle\hat{T}\,M^{2}\rangle$ was earlier
derived in [15]:

$\begin{array}[]{l}\displaystyle\langle\hat{T}\,M^{2}\rangle=-\frac{1}{\pi
U}\sum\limits_{k}\int_{-\infty}^{\infty}\tanh\left(\frac{\omega}{2T}\right)\left(\omega-\varepsilon(k)\right)\,\Im
G_{-}(k,\omega)\,d\omega\,=\\\ \\\
\displaystyle-\frac{1}{\pi^{2}U}\int_{-1}^{1}\int_{-\infty}^{\infty}\frac{d\varepsilon}{\sqrt{1-\varepsilon^{2}}}\tanh\left(\frac{\omega}{2T}\right)\left(\omega-\varepsilon\right)\,\Im
G_{-}(\varepsilon,\omega)\,d\omega\,.\end{array}$ (3.6)

Finally for paramagnetic $\langle M\rangle=0$ and half–filled $\langle
n\rangle=1$ solution the average number of double–occupied sites (3.5) is

$\displaystyle\langle
n^{e}_{\uparrow}n^{e}_{\downarrow}\rangle=\frac{1}{4}-\frac{\langle\hat{T}\,M^{2}\rangle}{2}\equiv\mathcal{X}(U)\,,$
(3.7)

where we denote now $n^{e}_{\sigma}$ as the operator for electrons.

Put the next graphic designations:  is just a vacant place uncorrelated with
other fermions, and unbound to a host–site;  is a negative charged electron;
is a positive charged hole.

In that case all electronic configurations on a single site can be represented
in one of particular form ,,,,,.

Using the operator expression for a hole’s number $n^{h}_{\sigma}$=
$1-n^{e}_{\sigma}$ we can write the obvious identity for the average number of
electrons with a certain spin direction in the system as $\displaystyle\langle
n^{e}_{\sigma}\rangle=\langle
n^{e}_{\sigma}n^{e}_{\bar{\sigma}}\rangle+\langle
n^{e}_{\sigma}n^{h}_{\bar{\sigma}}\rangle$; from which we get

$\begin{array}[]{l}\displaystyle\langle
n^{e}_{\sigma}n^{e}_{\bar{\sigma}}\rangle=\mathcal{X}(U)\\\
\displaystyle\langle n^{e}_{\sigma}n^{h}_{\bar{\sigma}}\rangle-\langle
n^{e}_{\sigma}\rangle=-\mathcal{X}(U)\,.\end{array}$ (3.8)

A numerical value of $\langle n^{e}_{\uparrow}n^{h}_{\downarrow}\rangle$ can
be expressed in the graphic view as $\displaystyle\langle
n^{e}_{\uparrow}n^{h}_{\downarrow}\rangle$=
$\langle\,\begin{picture}(7.0,7.0)\put(2.0,1.0){\circle{3.0}}
\put(5.2,1.0){\circle{3.0}} \put(2.0,2.5){\vector(0,1){3.0}}
\put(5.0,-0.5){\vector(0,-1){3.0}} \put(0.5,1.2){\line(1,0){3.0}}
\put(3.5,1.2){\line(1,0){3.0}} \put(5.0,-0.5){\line(0,1){3.0}}
\end{picture}\,\rangle$ \+
$\langle\,\begin{picture}(7.0,7.0)\put(2.0,1.0){\circle{3.0}}
\put(5.2,1.0){\circle{3.0}} \put(2.0,2.5){\vector(0,1){3.0}}
\put(0.5,1.2){\line(1,0){3.0}} \end{picture}\,\rangle.$ At that
$\langle\,\begin{picture}(7.0,7.0)\put(2.0,1.0){\circle{3.0}}
\put(5.2,1.0){\circle{3.0}} \put(2.0,2.5){\vector(0,1){3.0}}
\put(0.5,1.2){\line(1,0){3.0}} \end{picture}\,\rangle$ = $\langle
n^{e}_{\uparrow}\rangle$, and
$\langle\,\begin{picture}(7.0,7.0)\put(2.0,1.0){\circle{3.0}}
\put(5.2,1.0){\circle{3.0}} \put(2.0,2.5){\vector(0,1){3.0}}
\put(5.0,-0.5){\vector(0,-1){3.0}} \put(0.5,1.2){\line(1,0){3.0}}
\put(3.5,1.2){\line(1,0){3.0}} \put(5.0,-0.5){\line(0,1){3.0}}
\end{picture}\,\rangle$ = $\langle\mathcal{N}^{\text{exc}}_{\uparrow}\rangle$
– an excitonic pair for the single ($\uparrow$) spin direction. From the
second equation of (3.8) we finally find the average number of localized
excitons for both spin directions as

$\displaystyle\langle\mathcal{N}^{\text{exc}}\rangle=-2\,\mathcal{X}(U)\,.$
(3.9)

Positive or negative value of the parameter $\mathcal{X}(U)$ for different U
specifies the electronic state of the system, its quantum phase. The
interaction drives the bands from a semimetallic configuration to an excitonic
insulating one. The excitonic nature of the new state is revealed itself in
the deformed and mixed valence and conductivity subbands, which are indicating
the hybridization of the original band and the correlated coexistence of
electrons with holes (see Fig. 1).

ł

Figure 1: Double–occupied and localized excitonic sites. Here $\langle
n^{e}_{1\uparrow}n^{e}_{1\downarrow}\rangle$ is an average number of sites
with the double occupancy by electrons, $\langle
n^{e}_{1\uparrow}n^{h}_{1\downarrow}\rangle$ is an average number of sites,
which occupied by the localized excitons. The excitons are appeared and
disappeared quite steeply in region $1.7<U<4.3$, but small amount of the
double–occupancy also appear in the region. For another values of the Coulomb
repulsion we observe only quite smoothly varying parameter of $\langle
n^{e}_{1\uparrow}n^{e}_{1\downarrow}\rangle$. This situation can be considered
as a sort of quantum phase transition.

B. Electronic spectra and momentum distribution function

Correlated electronic energy spectra of the conductivity $E_{-}(k)$ and the
valence $E_{+}(k)$ bands as functions of the momentum k are obtained as
solutions of the following electronic dispersion equations ($\Re
G_{\pm}^{-1}(k,\omega+i0)=0$):

$\begin{array}[]{l}\displaystyle
E_{-}(k)-\varepsilon(k)-\Re\Sigma_{-}(k,E_{-}(k))=0\,,\\\ \displaystyle
E_{+}(k)+\varepsilon(k)-\Re\Sigma_{+}(k,E_{+}(k))=0\end{array}$ (3.10)

that have the following symmetry properties

$\displaystyle E_{\pm}(k)=E_{\pm}(k\pm\pi),\quad
E_{\pm}(k)=E_{\mp}(k\pm\pi)\,.$ (3.11)

We calculate the behaviour of the momentum distribution function per one spin
direction for the ground state of the 1D Hubbard model as

$\displaystyle
n(k)=\frac{1}{2}+\frac{1}{\pi}\displaystyle\int_{-\infty}^{\infty}\tanh\left(\frac{\omega}{2T}\right)\,\Im
G_{-}(k,\omega)\,d\omega\,.$ (3.12)

The momentum distribution function $n(k)$ gives an average occupation number
of electronic states of the momentum $k$ [18].

ł  and $U=2.5$. The first picture in every row represents electronic spectra
$E_{\mp}(k)$, the second one is the same but near $k_{F}=\pi/2$; the third
picture depicts the momentum distribution functions with insets for $k\approx
k_{F}$. A flat behaviour of $n_{k}$-curve around the Fermi level demonstrates
an equality of electron ($k>k_{F}$) and hole ($k<k_{F}$) numbers at the Fermi
level. Note the nontrivial topology of $E_{\mp}(k)$ in the area under
consideration.

Figure 2: The top and the bottom panels are corresponding to $U=0.5$

A presence or absence of a finite gap at the Fermi momentum $k_{F}$ are
generally considered for discerning among different sorts of an electronic
behavior. The finite jump would indicate that the qusiparticle excitations are
of the Fermi–liquid type without any gap in the spectrum and therefore, the
system is of a metallic–type. For other electronic phases: insulators,
semiconductors, semimetls, and excitonic insulators the function’s view near
the $k_{F}$ considerably differs from the metallic–type behavior.

The relevance of function (3.12) refers to its analitical properties. Formally
one of the determinal factors distingwishing electronic phases of the system
is an energy gap between the conductivity and the valence bands at the Fermi
momentum: $\Delta=E_{-}(k_{F})-E_{+}(k_{F})$ which distinguish the correlated
electrons from the free ones. The finite jump when $\Delta=0$ would indicate
that the qusiparticle excitations are of the Fermi–liquid type without any gap
in the spectrum and therefore, the system is of a metallic–type. At $\Delta>0$
we have an insulator, when $\Delta\simeq 0$ we get a gapless semiconductor
[19] and a semimetal or the excitonic insulator [10, 11] for $\Delta<0$.

In the weak-coupling limit (small U), near the semimetal to semiconductor
transition the system can become unstable against the formation of multiple
excitons and this can lead to a condensation state called the excitonic
insulator. Different values of band gap for U = 0.5 and U = 2.5 permit to
refer the first case to the SM - phase, and the second one to EI - phase.

## References

  * [1] J. Hubbard Proc. R. Soc. A 281, 401 (1964)
  * [2] A.N.Kozlov and L.A.Maksimov Soviet Physycs JETP, vol.21 4 (1965)
  * [3] N.F.Mott Metal-Insulator Transitions Taylor and Francis London (1990)
  * [4] Franz X. Bronold and Holger Fehske arXiv:cond-mat/0605415v3 [cond-mat.str-el] (2006)
  * [5] V.Yu.Irkhin and Yu.N.Skryabin arXiv:1909.06248v1 [cond-mat.str-el] (2019)
  * [6] M.Rontani and L.J.Sham Phys. Rev. Lett. 94 186404 (2005)
  * [7] Tudor D. Stanescu and Philip Phillips arXiv:cond–mat/0104478v2 (2001)
  * [8] Erik G.C.P. van Loon, Friedrich Krien, Hartmut Hafermann, Evgeny A.Stepanov, Alexander I. Lichtenstein, and Mikhail I. Katsnelson arXiv:1602.09129v1 (2016)
  * [9] N.M.Plakida Phys.Rev.Lett. V.92 P.256401 (2011)
  * [10] B.Zenker, arXiv:1409.2230v1 [cond-mat.str-el], (2014)
  * [11] LeeAnn M.Sager, Shiva Safaei, and David A.Mazziotti arXiv:2002.08445v1 [cond-mat.str-el] (2020)
  * [12] Chaschin N.I. The Physics of Metals and Metallography, vol.111 No.3 p.221 (2011)
  * [13] N.I.Chaschin The Physics of Metals and Metallography, vol.111 No.4 p.329 (2011)
  * [14] Chaschin N.I. The Physics of Metals and Metallography, vol.112 No.6 p.533 (2012)
  * [15] N.I.Chaschin The Physics of Metals and Metallography, vol.117 No.7 p.663 (2016)
  * [16] Izyumov Yu.A., Chaschin N.I., Alexeev D.S., Mancini F. Eur.Phys.J. B V.45 P.69–86 (2005)
  * [17] Chaschin N.I., arXiv:[707.01798v] [cond-mat.str-el], (2017)
  * [18] M.Brech, J.Voit and H. B $\ddot{u}$ttner Europhys. Lett., 12 (4)) p. 289 (1990)
  * [19] I.M. Tsidilkovski (Akademie,Berlin 1988)


#  InstructIR: A Benchmark for Instruction Following of Information Retrieval
Models

Hanseok Oh1 Hyunji Lee1 Seonghyeon Ye1 Haebin Shin1
Hansol Jang2 Changwook Jun2 Minjoon Seo1
1KAIST AI 2LG AI Research
{hanseok, hyunji.amy.lee, vano1205, haebin.shin<EMAIL_ADDRESS>
{hansol.jang<EMAIL_ADDRESS>
 Work performed during internship at LG AI.

###### Abstract

Despite the critical need to align search targets with users’ intentions,
retrievers often only prioritize query information without delving into the
users’ intended search context. Enhancing the capability of retrievers to
understand the intentions and preferences of users, akin to language model
instructions, has the potential to yield more aligned search targets. Prior
studies restrict the application of instructions in information retrieval to a
task description format, neglecting the broader context of diverse and
evolving search scenarios. Furthermore, the prevailing benchmarks utilized for
evaluation lack explicit tailoring to assess instruction-following ability,
thereby hindering progress in this field. In response to these limitations, we
propose a novel benchmark, InstructIR, specifically designed to evaluate
instruction-following ability in information retrieval tasks. Our approach
focuses on user-aligned instructions tailored to each query instance,
reflecting the diverse characteristics inherent in real-world search
scenarios. Through experimental analysis, we observe that some retrievers
fine-tuned to follow task-style instructions, such as INSTRUCTOR (Su et al.,
2022), can underperform compared to their non-instruction-tuned counterparts.
This underscores potential overfitting issues inherent in constructing
retrievers trained on existing instruction-aware retrieval datasets 111Code
and dataset are available at https://github.com/kaistAI/InstructIR.

##  1 Introduction

Large Language Models (LLMs) are often further trained to align user
instructions and preferences with instruction tuning for diverse generative
tasks (Ouyang et al., 2022; Wang et al., 2022a; Zhang et al., 2023b). This
kind of alignment to user preferences is also important for information
retrievers to reflect diverse users’ search intentions and preferences for the
search targets. For example, when a user writes a blog post for children about
the current climate change issue, it may be better to search for articles that
are easy to understand rather than complex scientific articles. However,
current retrievers often do not take this into account, focusing on utilizing
only ambiguous queries even simplifying the details for users through
reformulation (Ma et al., 2023a). Moreover, lack of benchmarks to evaluate
retrievers on user-aligned scenarios prevents the mature discussions of
instruction following in retrieval task.

Figure 1: InstructIR benchmark is designed to evaluate instruction following
ability in information retrieval tasks. As unique user-aligned instructions
change, different search targets should be retrieved to reflect real-world
search scenarios. Figure 2: Overview of data creation pipeline for building
InstructIR benchmark. To build datasets that demand diverse user-aligned
instructions for each query, we begin by selecting seed examples from the
MSMARCO datasets. Subsequently, we generate a variety of instructions suitable
for each query, revise the target text to align with these instructions, and
systematically filter the generated content. The resulting dataset is used for
InstructIR benchmark. GPT-4 is employed in this generation pipeline.

In order to effectively reflect the various intentions and situations that
real-world users actually ask, employing instance-wise instructions for
queries is more appropriate than relying on coarse-grained instructions that
share the same task-specific guidance for various queries (Wang et al., 2022a;
Ye et al., 2023). Several studies explore the integration of instructions into
retrievers, but they primarily concentrate on building general purpose
retrievers, which often limit the examination to task-specific instructions
(Asai et al., 2023; Su et al., 2022; Wei et al., 2023). For instance, these
studies uniformly apply the same instructions to all instances, presenting
them in the form of task descriptions, such as "Search for the Wikipedia
paragraph that answers this question". Furthermore, the evaluation of these
instruction-tuned models relies on benchmarks that inherently do not mandate
instructions for task resolution (Thakur et al., 2021; Santhanam et al., 2021;
Muennighoff et al., 2022). Due to these limitations, the extent to which
retrievers can effectively follow instructions has not been thoroughly
evaluated.

In this work, we introduce a novel benchmark, InstructIR, specifically
designed to evaluate instruction-following ability of retrieval models with
diverse user-aligned instructions for each query, mirroring real-world search
scenarios. We collect a total of 9,906 of instance-wise instructions that
involve details about the search users, such as their job, background,
situation, location, hobbies, additional interests, and search goals, and
preferred sources. Notably, InstructIR stands out from other benchmarks that
evaluate task-aware instructions due to the distinctiveness in instruction
types and diversity, as delineated in Table 1. Instructions and corresponding
search targets are acquired through our multi-stage data creation pipeline and
filtering process, leveraging GPT-4 (OpenAI, 2023), as illustrated in Figure
2. The quality of the datasets is verified through a combination of human
evaluation and machine filtering, resulting in a high-quality dataset.
Additionally, we introduce the Robustness score as an evaluation metric,
quantifying the ability of retrievers to robustly follow instructions. These
metrics offer a holistic perspective on how effectively retrievers adapt to
changes while conveying the same query with varying instructions.

We evaluate over 12 retriever baselines on InstructIR including both naïve
retrievers (not explicitly instruction-tuned) and instruction-tuned
retrievers. With our experiments, we find that task-style instruction-tuned
retrievers, such as INSTRUCTOR (Su et al., 2022), consistently underperform
compared to their non-tuned counterparts, which cannot be found with the
previous benchmarks. Notably, utilizing an instruction-tuned language model
and larger model as the backbone demonstrates the most potent performance
improvement. Through InstructIR, we gain valuable insights into the diverse
characteristics of existing retrieval systems. We anticipate that this
benchmark will contribute to accelerating progress in the development of more
sophisticated, controllable, and instruction-aware information access systems.

##  2 Related Works

#### Evaluation for Instruction Following.

Insturction tuning is a crucial technique to enhance the capabilities and
controllability of large language models (LLMs). Instruction tuning refers to
the process of further training LLMs on a dataset consisting of (instruction,
output) pairs in a supervised fashion, which bridges the gap between the next-
word prediction objective of LLMs and the users’ objective of having LLMs
adhere to human instructions (Ouyang et al., 2022; Zhang et al., 2023a). For
the approaches to evaluate instruction following capabilities in generative
tasks can be categorized as follows: Instructions for cross-task
generalization, User Aligned instructions, and Verifiable instructions.
Instructions for cross-task generalization focus on evaluating cross-task
generalization under instructions training models to follow instructions on a
subset of tasks and evaluating them on the remaining unseen ones (Wang et al.,
2022b; Liang et al., 2022; Wang et al., 2022b). User Aligned instructions
focus on evaluating how instruction-based models handle diverse and unfamiliar
instructions (Ye et al., 2023; Wang et al., 2022a; Dubois et al., 2023).
Unlike coarse-grained evaluation as for cross-task generalization
instructions, they are different per instance. Verifiable instructions are
focusing on straightforward and easy-to reproduce evaluation benchmark that
focuses on a set of “verifiable instructions” (Efrat et al., 2022; Zhou et
al., 2023; Jiang et al., 2023).

#### Instruction Following in Information Retrieval.

Building instruction-tuned models on text embedding tasks is mostly focused on
building general purpose model that can solve multiple tasks with task
description as an instructions (Asai et al., 2023; Su et al., 2022; Wei et
al., 2023). Models are trained with multiple source of train dataset with task
descriptions as instructions, such as BERRI and MEDI, and evaluated on the
held out tasks which haven’t seen during the train time. However, for solving
these tasks it is not essential to follow instructions. The relationship
between query and targets are already in one to one relationship, where hard
to evaluate effect of instructions. Moreover, the benchmarks used for
evaluating instruction following ability are BEIR (Thakur et al., 2021), LoTTE
(Santhanam et al., 2021), X2 (Asai et al., 2023), which are not suitable for
evaluating the retrieval model’s ability to follow instructions because of
coarse-grained evaluation per task not fine-grained instance-wise evaluation.
MTEB (Muennighoff et al., 2022) is a similar type of benchmark that is widely
used for evaluating instruction-tuned text embedding models, but it is not
focused on retrieval tasks, so we do not compare it deeply. Additionally,
M-BEIR (Wei et al., 2023) is introduced for assessing multimodal retrieval
tasks but is also constrained by a task-description style.

##  3 The InstructIR Benchmark

###  3.1 Data Creation Pipeline

Constructing a framework to evaluate instruction-following capabilities in
information retrieval models necessitates correlating multiple instructions
with the same query and adjusting their targets accordingly (i.e.,
instruction, query, target text). Therefore, in contrast to previous
approaches that evaluate coarse-grained task description-style instructions on
information retrieval datasets with up to 15 instructions, we focus on
creating per-query, instance-specific instructions as Table 1. We employ GPT-4
222We use gpt-4-1106-preview for our work. (OpenAI, 2023) to facilitate the
creation of such a setting. The development of our InstructIR evaluation
datasets adheres to a systematic set of steps as Figure 2, outlined as
follows:

#### Step 1. Select Seed Examples.

In tackling the challenge of generating all components from scratch, we opt to
levarage the MSMARCO dataset (Nguyen et al., 2016) for passage ranking into
our seed examples 333We utilize the validation split from MSMARCO, consisting
of 6,980 queries.. This dataset is renowned for its comprehensive coverage of
diverse topics collected from the real web. We carefully select a total of
1,743 queries $Q$ and corresponding target texts $T$ as seed examples,
adhering to the following criteria: 1) The seed query should demonstrate the
potential to match various targets as instructions evolve. 2) The target
should offer substantial content, allowing for modifications that align with
provided instructions. 3) Ensuring ease in controlling false negatives is
crucial. To meet the first criterion, we focus on queries with a length
ranging from 25% to 75% (i.e., 24 - 40). This approach prevents the seed
query, pivotal for formulating subsequent instructions, from being overly
vague or verbose. Similarly, for the second criterion, aligning with the
rationale for query selection, we chose target texts within the text length
range of 255 - 371 to avoid ambiguity. Lastly, for the third criterion, to
facilitate the control of potential false negatives, we exclusively extract
instances with only one positive target.

#### Step 2. Generate Instructions.

Following the careful selection of seed examples in the previous step, we
harness the power of GPT-4 to produce a set of instructions $I_{i}$
corresponding to each query $q_{i}$, where $i$ ranges from 1 to $n$, denoting
the number of queries. To evaluate the adherence of retrievers to
instructions, we emphasize the importance of deploying the same query with
different instructions as illustrated in Figure 1. This approach enables us to
measure how effectively models dynamically retrieve relevant targets. We also
insist to move away from defining instructions in the form of rigid task
descriptions, instead to embrace a more realistic approach that captures real-
world scenarios in which retrieval systems are employed. This involves
incorporating diverse information about users, such as their occupation,
search context, location, search objectives, and preferred sources. To achieve
this, we adopt the prompt outlined in Figure 4. We produce a set of
instructions $I_{i}=\\{I_{i,1},...,I_{i,k}\\}$ (where $k$ is set to 10) for
each query $q_{i}$ with a particular focus on aligning these instructions with
a precise reflection of scenarios that real users may encounter.

#### Step 3. Revise Target Text.

During this phase, GPT-4 refines the original target text $T$ by integrating
the instructions $I_{i}$ generated in step 2. The model takes in a query
$q_{i}$, an instruction $I_{i,k}$, and the original target $t_{i}$ derived
from seed examples. Subsequently, GPT-4 adjusts the target to better align
with the provided instructions, resulting in $t_{i,k}^{{}^{\prime}}$. This
adjustment involves revising the target to accurately represent the given
scenario, taking into account factors like the user’s background, situation,
location, occupation, hobbies, interests, or goals for the search.
Additionally, it is crucial to incorporate information related to the user’s
preferences. To ensure the generation of diverse targets in response to
evolving instructions, we employ a prompt, illustrated in Figure 5.

#### Step 4. Filtering Process.

To assess the quality of machine-generated datasets during steps 2 and 3, we
proceed by filtering out datasets in this stage. The selection of high-quality
instances is based on the evaluation of two key criteria: Q1. Does the revised
target align with the original query? and Q2. Does the revised target align
with the given query and instructions, while other targets do not?.
Additionally, we leverage the capabilities of GPT-4 as a quality evaluator.
For Q1, we employ the prompt presented in Figure 7 to retain instances with a
high score between the original query $q_{i}$ and the revised target text
$t_{i,k}^{{}^{\prime}}$ 444Empirically, we observe that a score exceeding 3
out of 5 indicates good quality.. To address Q2, the prompt in Figure 8 is
utilized to identify instances where GPT-4 accurately predicts the gold target
$t_{i,k}^{{}^{\prime}}$ among distractors $t_{i,m}^{{}^{\prime}}$ (where
$m\neq k$), generated from the same query with different instructions. This is
based on the scores between the original query $q_{i}$ and an instruction
$I_{i,k}$ pairs, and the set of revised target text $T_{i}^{\prime}$.
Following the filtering stage, we select instances that possess more than 6
instructions with the same query, facilitating an effective evaluation of
retrievers in dynamically changing scenarios. This results in a total of 9,906
instances, as detailed in Table 2. Further statistics regarding the filtering
stage and examples can be found in the Appendix A.

###  3.2 Dataset Analysis

| User-aligned | Type of inst. | # of inst. | Metrics
---|---|---|---|---
InstructIR | ✓ | instance-wise | 9,906 | Robustness, nDCG
BEIR | ✗ | task-wise | 15 | nDCG
LoTTE | ✗ | task-wise | 5 | nDCG
X2 | ✗ | task-wise | 6 | nDCG

Table 1: Table comparing InstructIR with other IR benchmarks used to measure
instruction following ability. Since the other benchmarks are not designed to
evaluate instruction-following ability of information retrieval task, they are
based on reformulated versions from previous studies (Asai et al., 2023; Su et
al., 2022).

#### Comparison Table.

We characterize our InstructIR dataset as shown in Table 1. Note that all
other benchmarks (BEIR, LoTTE, and X2) are not proposed for evaluating
instruction following ability of retrieval systems originally, it is
reformulated with author proposed instructions in the previous studies (Asai
et al., 2023; Su et al., 2022; Wang et al., 2023). Other benchmarks are not
focused on user aligned scenario, leading to structured formats that are far
from search scenario of real users. Also, others utilize task-specific
instructions, where up to 15 instructions are used for the evaluation, which
is too small to evaluate whether a retrieval model can follow the
instructions.

#### Dataset Quality.

To ensure the quality and authenticity of our generated and filtered datasets,
a human verification stage is implemented for validation purposes. For about
8% of the randomly sampled groups, a total of 10 annotators evaluate two
instances per group. For each instance, we pose three questions, and three
annotators are assigned to assess them, aiming to measure the inter-agreement
between annotators (Cohen, 1960; Randolph, 2005). We consider the final human
decision by majority voting. Further detail settings and results with inter-
agreement are available in Appendix B. The first question (Q1: Is the
instruction valid for the search user?) asks whether the provided instruction
is suitable for the search user scenario. For this question, all instances
were answered suitable for the search scenario. The second question (Q2: Is
the instruction natural for the given query?) asks whether the instruction is
naturally aligned with the query. About 97% instances were answered that the
instruction and query have well-aligned relations. The third question (Q3:
Which passage is the most natural for the given instruction and query?) asks
to choose the most relevant passage. This question can estimate the difficulty
of this benchmark, and evaluate the alignment between the results of Q2 from
the filtering process (step 4) and human judgment. As a result, we got kappa
coefficient (Cohen, 1960) of 0.6468 which indicates substantial agreement
between human judgement and dataset (Landis and Koch, 1977). However,
annotators demonstrate an approximately 76.5% top-1 accuracy, emphasizing the
need for careful consideration when selecting the top-1 passage that follows
the instructions. User interface and instruction for annotators are described
in Figure 9.

| Number
---|---
Avg. instructions per query | 7.81
number of queries | 1,267
Number of query with instructions | 9,906
number of corpus | 16,072
relevancy | binary

Table 2: Statitistics for InstrucIR dataset

#### Dataset Diversity and Statistics.

Table 2 presents the data statistics for datasets within InstructIR. Through
our data creation pipeline, we acquire an average of 7.81 instructions per
query. Furthermore, to simulate real-world noisy search scenarios, we
incorporate a subset of targets from the seed datasets. This results in an
augmentation of approximately 6k additional targets, supplementing our revised
targets. Consequently, our benchmark comprises a total of 10k instances,
featuring a rich variety of instructions and a corpus with 16k entries,
collectively constituting the InstructIR benchmark. Recognizing the
significance of encompassing diverse instructions for the same query to
reflect various user search scenarios, we evaluate the diversity of our
dataset using the ROUGE score distribution inspired by Wang et al. (2022a).
Specifically, we calculate the average ROUGE-L score for instructions
associated with the same query. As illustrated in Figure 10, the instructions
within InstructIR encapsulate highly diverse scenarios with low similarity to
each other, achieving an average ROUGE-L score of 0.238 within each group.

###  3.3 Evaluation Metric

We employ the Normalized Cumulative Discount Gain (nDCG@k) following Thakur et
al. (2021), as it is a widely accepted evaluation metric for assessing
retrieval models. However, we emphasize the need for specialized metrics to
assess the ability to follow instructions when retrieving information,
especially when measuring dynamic changes in retrieval targets as instructions
for a given query change. In instances where diverse search scenarios and user
preferences serve as instructions alongside a given query, retrieval systems
should adapt to these instructions to identify appropriate targets
effectively. Hence, inspired by Zhong et al. (2022) and Oh et al. (2023), we
introduce a Robustness score to assess how consistently the model predict
targets over evolving instructions using the same query. To quantify
robustness, we group instances with identical queries, calculate the minimum
nDCG@k score within each group, and subsequently average these per-group
scores to derive the final Robustness@k score.

##  4 Experiments

Instruction-tuned | Type | Models | Size | Robustness@10 | nDCG@10
---|---|---|---|---|---
No | Lexical | BM25 | - | 26.92 | 76.01
Late-Interaction | ColBERT-v2.0 | 110M | 14.15 | 68.47
Bi-Enc. | Contriever-msmarco | 110M | 47.40 | 84.85
GTR-base | 110M | 34.06 | 73.35
GTR-large | 335M | 37.56 | 75.95
GTR-XL | 1.5B | 38.34 | 75.20
| RepLLaMa | 7B | 52.58 | 87.62
Yes | Bi-Enc. | TART-dual | 110M | 47.46 | 84.81
INSTRUCTOR-base | 110M | 23.73 | 50.44
INSTRUCTOR-large | 335M | 22.08 | 48.80
INSTRUCTOR-XL | 1.5B | 21.53 | 48.63
|  | E5-mistral-7b-instruct | 7B | 55.42 | 86.33

Table 3: Zero-shot performances on InstructIR benchmark. We group models based
on whether instruction-tuned or not, type of scoring methods, and the size of
the models. Bi-Encoder is abbreviated as Bi-Enc.

###  4.1 Baselines

We utilize InstructIR to compare various retriever systems in a zero-shot
manner. InstructIR exclusively offers test datasets designed to evaluate the
proficiency of existing retrievers in addressing information retrieval tasks
that require adherence to specific instructions. In our experimentation, we
use pre-trained checkpoints accessible online for all models. We categorize
the models based on the following criteria: non-instruction-tuned retrievers
and instruction-tuned retrievers.

#### Non-instruction-tuned Retrievers.

For non-instruction-tuned retrievers, we select BM25 (Robertson et al., 2009),
Contriever-MSMARCO (Izacard et al., 2021), GTR (Ni et al., 2022), and RepLLaMa
(Ma et al., 2023b). BM25 represents a lexical matching retriever. Contriever-
MSMARCO, a bert-base sized model, is fine-tuned on the MSMARCO (Nguyen et al.,
2016) passage ranking dataset following an unsupervised contrastive pre-
training stage. GTR comprises variants of a t5-encoder based bi-encoder,
trained on both MSMARCO and NQ datasets. RepLLaMa is a bi-encoder model based
on the LLaMa-2-7b decoder, extracting token embeddings from the end-of-
sequence token to generate text embeddings.

#### Instruction-tuned Retrievers.

For instruction-tuned retrievers, we utilize TART-dual (Asai et al., 2023),
INSTRUCTOR (Su et al., 2022), and E5-mistral-7b-instruct (Wang et al., 2023).
TART-dual is a retriever with a bi-encoder architecture based on the
Contriever model. It is additionally trained on BERRI, an instruction-aware
information retrieval dataset comprising approximately 40 diverse retrieval
tasks, each accompanied by a task description serving as instructions.
INSTRUCTOR offers various size versions, including base, large, and xl, all
finetuned on the GTR retriever as a backbone 555As INSTRUCTOR also leverages
corpus-side instruction, we adhere to its approach by using the corpus
instruction: ’Represent the document for retrieval:’ utilized for the MSMARCO
dataset.. It is further trained on the MEDI dataset, encompassing 330 distinct
text embedding tasks, each with a human-written task instruction, including
multiple retrieval datasets. Lastly, E5-mistral-7b-instruct is a bi-encoder
architecture retriever trained on a proprietary Language Model(LLM), the
mistral-7b decoder. It undergoes training solely with synthetic data for text
embedding tasks, with task definitions serving as instructions.

###  4.2 Results

We evaluate various retrieval systems on InstructIR benchmark to evaluate
their capability to follow instructions in a zero-shot setting as Table 3. The
baselines are categorized into instruction-tuned and non-instruction-tuned
models across different sizes and architectures.

#### Non-Instruction-Tuned Models.

Considering overall high score of nDCG over 50, due to the characteristics of
distinct instructions in InstructIR datasets, lexical bias exists. However,
the lexical matching model BM25 and late interaction model ColBERT-v2.0 show
Robustness@10 of 26.92 and 14.15 respectively. And it means that focusing
individual keywords can’t truly understand evolving instructions, which leads
to huge drop in Robustness score. Representative retrievers that are trained
with contrastive training objectives, such as Contriever-msmarco and GTR
variants (base, large, and xl), are considered, with sizes ranging from 110M
to 1.5B parameters. These models exhibited a significant improvement in
Robustness over BM25, which Contriever-msmarco shows most powerful and robust
performance compared to the same size models in non-instruction tuned
baselines even superior to way more larger size GTR-xl. The largest model,
RepLLaMa with 7B parameters, achieve the highest nDCG@10 of 87.62 and
Robustness@10 of 52.58, indicating a strong correlation between model size and
performance metrics in non-instruction-tuned settings.

#### Instruction-Tuned Models.

TART-dual undergoes additional training steps with BERRI, a comprehensive set
of over 40 retrieval datasets accompanied by task-specific instructions
generated by human experts, based on Contriever-msmarco. INSTRUCTOR variants
are subsequently trained, aligning with the respective sizes of GTR
retrievers, using MEDI datasets. These datasets are reformulated training sets
covering 330 datasets, incorporating instructions spanning diverse text
embedding task categories and domains, including retrieval tasks. However,
both series of instruction-tuned baselines do not show superior performance
than their non-instruction-tuned counterparts. Notably, INSTRUCTOR variants
exhibit a huge performance drop compared to the backbone model GTR variants.
This can be interpreted that finetuning retrievers with only task-style
instructions doesn’t guarantee good performance in various free creative user-
aligned style instructions. Conversely, E5-mistral-7b-instruct, shows
Robustness@10 with 55.42 outperforming all other models. This highlights the
importance of using large instruction-tuned models for search tasks to follow
instructions as well.

##  5 Analysis

#### Scaling Up Model Size Leads Better Instruction Following.

It is not surprising that larger models derive greater benefits from
instruction tuning, indicating that their enhanced capacity enables a more
effective integration of instruction-following abilities. In the case of non-
instruction-tuned baselines, GTR demonstrates superior performance as model
sizes increase, particularly in terms of Robustness scores. Remarkably,
Contriever-msmarco exhibits competitive performance even with smaller sizes.
However, INSTRUCTOR variants display reverse trends, resulting in lower
Robustness scores as model sizes increase. This hints at an overfitting issue
to diverse task description style instructions, leading to diminished
performance across varying instruction styles and longer, unseen user-aligned
scenarios. Nonetheless, the outstanding performances of 7B size models,
RepLLaMA, and E5-mitral-7b-instruct underscore the significance of both model
size and instruction tuning in developing competent retrieval systems for
complex, instruction-based queries.

#### Instruction Order Sensitivity Exists For Instruction Tuned Retrievers.

To analyze the importance of the order of instructions and queries, we conduct
additional experiments by changing their sequence (query-> instruction).
Remarkably, INSTRUCTOR exhibits a significant performance gain when query
precede the instructions as Table 4. Considering the average length of
instructions in the training dataset used for INSTRUCTOR, it is approximately
12.16 tokens. In contrast, the average number of tokens for instructions
utilized in our user-aligned instructions is about 64.47. This discrepancy
highlights the challenge of generalizing models trained solely on task-
description style instructions, which exhibit limited creativity and variety,
to more user-aligned cases. For a detailed comparison between coarse-grained
task description instructions and user-aligned instance-specific instructions,
please refer to the examples provided in the Appendix 8.

Models | Robustness Gap | nDCG Gap
---|---|---
BM25 | -17.62 | -14.69
Contriever-msmarco | -2.31 | -1.78
ColBERT-v2.0 | +22.20 | +10.45
GTR-base | -2.62 | -1.53
GTR-large | -1.97 | -1.34
GTR-XL | -2.53 | -1.22
RepLLaMa | -4.11 | -2.94
TART-dual | -0.58 | -0.82
INSTRUCTOR-base | \+ 17.06 | \+ 32.82
INSTRUCTOR-large | \+ 34.70 | \+ 40.74
INSTRUCTOR-XL | +23.26 | \+ 37.27
E5-mistral-7b-instruct | +2.89 | +1.61

Table 4: Performance gap for changing instruction order.

#### Weighting Individual Terms leads Instruction Sensitivity for the
Paraphrased Instructions.

To analyze the sensitivity of instructions when paraphrased, we randomly
select one instance from each group in InstructIR, resulting in 1,267 subsets
of instances. Subsequently, we generate five paraphrased versions for each
instance using GPT-4. This process yields a total of 1,267 groups of
paraphrased instructions. The evaluation is based on the smallest score among
the five paraphrased instructions. As illustrated in Figure 3, retrievers
specialized in emphasizing specific terms, such as BM25 and ColBERT-v2.0,
exhibit a substantial performance drop of 20.53 and 35.17, respectively. In
contrast, most bi-encoder based models demonstrate less fluctuation compared
to these two models. Notably, the E5-mistral-7b-instruct model displays the
most robust performance in adapting to changing instructions.

Figure 3: Prompt sensitivity per models. Blue bar and orange bar denote
performance of original instructions and smallest score of paraphrased
versions respectively.

#### Relying on Lexical Redundancy Reduces Robustness.

Models | Robustness Gap | nDCG Gap
---|---|---
Contriever-msmarco | +4.10 | -25.31
ColBERT-v2.0 | +10.91 | -36.43
GTR-base | +3.09 | -27.95
GTR-large | +5.95 | -24.48
GTR-XL | +4.65 | -24.55
RepLLaMa | +1.35 | -24.96
TART-dual | +6.57 | -22.93
INSTRUCTOR-base | +6.65 | -14.73
INSTRUCTOR-large | +9.62 | -11.80
INSTRUCTOR-XL | +10.74 | -11.36
E5-mistral-7b-instruct | +6.90 | -17.14

Table 5: Performance gap when removing high lexical overlap. BM25 is removed
as used for filtering.

Considering existence of distinct information per the instruction, lexical
hint can exist to find proper target. Therefore, we eliminate instances with
high lexical overlap by filtering out cases where BM25 predicts the ground
truth answer in the top 10, and evaluate how other semantic matching models
are affected by this filtering process. Overall, average Robustness score for
the less lexical overlap cases increases in all model, and nDCG score drops.
And it means some of datasets in InstructIR can be solved by lexical matching,
but relying heavily on lexical cues can lead to wrong targets, which leads to
lower Robustness score. Among all, ColBERT-v2.0 shows the largest changes
showing average plus 10.91 Robustness score, and minus 36.43 nDCG score.
RepLLaMA shows least Robustness performance change.

##  6 Conclusion and Future Work

In this study we introduce a novel benchmark, InstructIR, designed for
evaluating the instruction following capabilities of information retrieval
models. Despite the critical importance of aligning models with user
instructions and reflecting user preferences in information retrieval tasks,
existing evaluations often fall short in comprehensively assessing these
aspects. Therefore, our benchmark focuses on evaluating user-aligned
instructions tailored to each query instance, reflecting the diverse
characteristics inherent in real-world search scenarios. Our experimental
investigation sheds light on the instruction following capabilities of
information retrieval models, presenting valuable insights that contribute to
the current understanding of this domain. In particular, we highlight the gaps
in current instruction-tuned retrievers in the limited style of training
datasets organized around task description style instructions.

One promising direction for future research is the exploration of
methodologies, such as Reinforcement Learning from Human Feedback (RLHF), to
enhance the alignment of retrieval models with users’ search intentions as
proposed by Ouyang et al. (2022). Investigating the integration of RLHF
techniques can potentially lead to more effective and adaptive information
retrieval systems that better understand and respond to user instructions.
Additionally, future studies could delve into the development of more diverse
instruction-aware retrieval training datasets that capture the nuances of user
preferences and instructions in a more intricate manner. By addressing these
challenges, we anticipate significant advancements in the field, ultimately
improving the overall user experience in information retrieval scenarios.

##  7 Limitations

While our proposed benchmark, InstructIR, leverages a representative retrieval
dataset that mirrors real user behavior patterns and has undergone systematic
filtering processes and human validation steps, the challenge lies in
accurately simulating real user instructions within an automated data
generation setting. Our study incorporates user-aligned information related to
job, background, situation, location, occupation, hobbies, interests, goals of
the search, and document preferences. However, there is room for improvement
by enhancing the comprehensiveness of instructions through the integration of
diverse seed datasets across various domains, thereby offering a more
comprehensive evaluation. Additionally, we conduct experiments in a zero-shot
setting, where the retrievers are not explicitly tuned based on user-aligned
instructions, aiming to assess the adaptability of current instruction-tuned
retrieval systems to different styles of instructions. Further analysis to
explore the potential for performance improvement through user-aligned
instruction-aware training would deepen our understanding of the models’
adaptability to user instructions in the future.

## References

  * Asai et al. (2023) Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, and Wen-tau Yih. 2023.  Task-aware retrieval with instructions.  In _Findings of the ACL_.
  * Cohen (1960) Jacob Cohen. 1960.  A coefficient of agreement for nominal scales.  _Educational and Psychological Measurement_.
  * Dubois et al. (2023) Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023.  Alpacafarm: A simulation framework for methods that learn from human feedback.  _arXiv_.
  * Efrat et al. (2022) Avia Efrat, Or Honovich, and Omer Levy. 2022.  Lmentry: A language model benchmark of elementary language tasks.  _arXiv_.
  * Izacard et al. (2021) Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021.  Unsupervised dense information retrieval with contrastive learning.  _Trans. Mach. Learn. Res._
  * Jiang et al. (2023) Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. 2023.  Followbench: A multi-level fine-grained constraints following benchmark for large language models.  _arXiv_.
  * Landis and Koch (1977) J Richard Landis and Gary G. Koch. 1977.  The measurement of observer agreement for categorical data.  _Biometrics_.
  * Liang et al. (2022) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022.  Holistic evaluation of language models.  _arXiv_.
  * Ma et al. (2023a) Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023a.  Query rewriting for retrieval-augmented large language models.  _arXiv_.
  * Ma et al. (2023b) Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. 2023b.  Fine-tuning llama for multi-stage text retrieval.  _arXiv_.
  * Muennighoff et al. (2022) Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2022.  Mteb: Massive text embedding benchmark.  In _EACL_.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  Ms marco: A human-generated machine reading comprehension dataset.
  * Ni et al. (2022) Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, et al. 2022.  Large dual encoders are generalizable retrievers.  In _EMNLP_.
  * Oh et al. (2023) Hanseok Oh, Haebin Shin, Miyoung Ko, Hyunji Lee, and Minjoon Seo. 2023.  Ktrl+ f: Knowledge-augmented in-document search.  _arXiv_.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. 2022.  Training language models to follow instructions with human feedback.  _ArXiv_.
  * Randolph (2005) Justus J. Randolph. 2005.  Free-marginal multirater kappa (multirater k[free]): An alternative to fleiss’ fixed-marginal multirater kappa.
  * Robertson et al. (2009) Stephen Robertson, Hugo Zaragoza, et al. 2009.  The probabilistic relevance framework: Bm25 and beyond.  _Foundations and Trends® in Information Retrieval_ , 3(4):333–389.
  * Santhanam et al. (2021) Keshav Santhanam, O. Khattab, Jon Saad-Falcon, Christopher Potts, and Matei A. Zaharia. 2021.  Colbertv2: Effective and efficient retrieval via lightweight late interaction.  In _NAACL_.
  * Su et al. (2022) Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022.  One embedder, any task: Instruction-finetuned text embeddings.  _ArXiv_.
  * Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Ruckl’e, Abhishek Srivastava, and Iryna Gurevych. 2021.  Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models.  _ArXiv_.
  * Wang et al. (2023) Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023.  Improving text embeddings with large language models.  _arXiv_.
  * Wang et al. (2022a) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022a.  Self-instruct: Aligning language model with self generated instructions.  _arXiv_.
  * Wang et al. (2022b) Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. 2022b.  Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks.  _arXiv_.
  * Wei et al. (2023) Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. 2023.  Uniir: Training and benchmarking universal multimodal information retrievers.  _ArXiv_.
  * Ye et al. (2023) Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. 2023.  Flask: Fine-grained language model evaluation based on alignment skill sets.  _arXiv_.
  * Zhang et al. (2023a) Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, and Guoyin Wang. 2023a.  Instruction tuning for large language models: A survey.  _ArXiv_.
  * Zhang et al. (2023b) Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. 2023b.  Instruction tuning for large language models: A survey.  _arXiv_.
  * Zhong et al. (2022) Victor Zhong, Weijia Shi, Wen-tau Yih, and Luke Zettlemoyer. 2022.  Romqa: A benchmark for robust, multi-evidence, multi-answer question answering.  _arXiv_.
  * Zhou et al. (2023) Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023.  Instruction-following evaluation for large language models.  _arXiv_.

## Appendix

##  Appendix A Details for Dataset Construction Pipeline

⬇

>> SYSTEM PROMPT

You are a helpful, respectful and creative assistant.

>> USER INSTRUCTION

Your task is to generate a set of scenarios for the provided search query.

Here is the specification for the scenario generation task:

- The scenario should reflect a very specific scenario where a user is interacting with an AI search engine.

- Within the scenario, the user could write about his/her job, background, situation, location, occupation, hobbies, interests, or goals of doing the search. Also, the user could explicitly reflect about his/her preference regarding the document to be searched.

- The scenario SHOULD be written from a first person’s view point. For example, it should start with phrases like "I am a {job}, …", "I am in a situation …", "During my {situation}".

- While the provided query is about "what" is being searched for, the scenario you will generate should be about "how" the search should be approached and what values or criteria should be prioritized in that search. This distinction makes the role of the scenario clear as a guiding framework for the search process, as opposed to the query which is more about the specific target of the search.

- However, the scenario should be RELATED with the provided query. In other words, it shouldn’t be applicable to other queries in general.

- You should generate based on the following format (note that there is a phrase

"[END]" after each scenario being generated):

Scenario 1: {generate the first scenario} [END]

Scenario 2: {generate the second scenario} [END]

Scenario 10: {generate the last scenario} [END]

- Please do not generate any other opening, closing, and explanations. Just generate the set of scenarios!

Figure 4: Prompt for generating instructions (step 2)

⬇

>> System Prompt

You are a helpful, respectful and creative assistant.

>> USER INSTRUCTION

Your task is to generate a REVISED DOCUMENT for the provided search QUERY and
SCENARIO pair.

Here is the specification for the DOCUMENT revising task:

- The REVISED DOCUMENT should reflect the user’s unique SCENARIO where a user is interacting with an AI search engine.

- Within the REVISED DOCUMENT, revise details reflecting the user’s background, situation, location, occupation, hobbies, interests, or goals of doing the search. Also, containing information related to the user’s preference is important.

- Directly revise given DOCUMENT that has good quality that can be found by an AI search engine. Don’t just suggest it!

- Do NOT include the same keywords from the given SCENARIO in REVISED DOCUMENT. Paraphrase it.

- However, the REVISED DOCUMENT should be RELATED with the provided query. In other words, it should be applicable to query in general.

- You should generate based on the following format (note that there is a phrase "[END]" after each elements being generated):

PLAN: {generate the plan for the strategy for revision} [END]

REVISED DOCUMENT: {revise the document} [END]

- Please do not generate any other opening, closing, and explanations. Just generate the PLAN and REVISED TARGET !

Figure 5: Prompt for revising target text (step 3)

#### Filtering Process.

During the filtering process in step 4 of data creation pipeline, we perform
two filtering criteria. For the first criterion, Q1. Does the revised target
align with the original query?, we utilize a prompt in Figure 7. To select
high relevant revised target for the given query, we measure scores for the
given score rubric from 1 to 5 and corresponding explanation. Average score
distribution for this step is in Figure 6. When we randomly sample the
outputs, score exceeding 3 out of 5 shows good quality. After this process,
15,669 instances are survived out of 16,157 instances. Detailed examples with
both high and low scores are available in the Table 6. Next, for the second
criterion, Q2. Does the revised target align with the given query and
instructions, while other targets do not?., we use a prompt in Figure 8. In
this step, we only select correct case that GPT-4 predicts annotated target
among other distractors, and 11,992 instances are selected out of 16,157.
After merging these two steps, and select where more than 6 instances exist
per group, we get total 1,267 groups with 9,906 instances left.

Figure 6: Relevancy score distribution between revised target after data
generation step3 and query.

⬇

>>> SYSTEM PROMPT

You are a helpful, respectful and creative assistant.

>>> USER INSTRUCTION

You are a similarity evaluator! You’re tasked with calculating the similarity
between QUERY, and DOCUMENT displayed below based on their relevancy. In the
evaluation, I want you to rate the relevancy of the pair according to the
following score rubric:

Score 1: The DOCUMENT, QUERY have very little or no relevance to each other.
The elements compared share almost no common attributes or context.

Score 2: The DOCUMENT, QUERY have some relevance but are quite distinct. They
share a few attributes or contextual details, but there are significant
differences in the majority of aspects.

Score 3: The DOCUMENT, QUERY are moderately relevant to each other. They share
a fair amount of attributes or context, but there are still some notable
differences that prevent a high similarity score.

Score 4: The DOCUMENT, QUERY have high relevance to each other. They share
many attributes or contextual details, with only a few differences that do not
majorly impact the overall similarity.

Score 5: The DOCUMENT, QUERY are very highly relevant or almost identical to
each other. They share nearly all attributes or the context is almost exactly
the same, with very minor or negligible differences.

You will be given QUERY and DOCUMENT pair.

[QUERY]

{{query}}

[DOCUMENT]

{{candidate document}}

You should generate based on the following format:

<Explanation>

{explanation for the score}

</Explanation>

<Score>

{score}

</Score>

Please give feedback on the DOCUMENT with respect to each QUERY, and provide a
score on a scale of 1 to 5 whether it satisfies the requirements, where a
higher score indicates better performance.

Figure 7: Prompt for filtering stage Q1 (step 4). Type |  Example
---|---
Good |  [Query]
|  Heteromorphic definition.
|  [Target]
|  In the realm of genetic research and study, "heteromorphic" specifically
denotes the occurrence of distinct forms, or morphs, within an organism’s life
cycle, or among different organisms, as a result of genetic mutation. This
term is often utilized in scholarly articles to describe variations that are
not merely different in size but also morphologically distinct, such as
varying chromosome shapes in the gametes of different sexes of the same
species. In an educational setting, one would explore the significance of
heteromorphism in understanding evolutionary processes and the resulting
diversity among living forms. It’s applied in studies that examine the genetic
basis of these morphological variations and their implications for the
survival and reproduction of species.
|  [Score]
|  5
|  [Explanation]
|  The document provides a detailed explanation of the term "heteromorphic" in
the context of genetic research and study, explaining that it refers to the
occurrence of distinct forms within an organism’s life cycle or among
different organisms as a result of genetic mutation. It also discusses the
relevance of heteromorphism in evolutionary processes and the diversity of
living forms. This explanation aligns well with the query, which asks for the
definition of "heteromorphic." The document offers a clear, in-depth
definition and contextualizes the term within its scientific application,
demonstrating a high relevance to the query.
Bad |  [Query]
|  is catecholamine a steroid
|  [Target]
|  For those actively engaged in physical conditioning, it’s crucial to
understand the role of various hormones. In the body’s natural arsenal, there
are hormones derived from the amino acid tyrosine that have significant roles,
such as the thyroid hormone, which is pivotal for organ development and
metabolic rate—an aspect vital for anyone with a focus on fitness.
|  [Score]
|  1
|  [Explanation]
|  the query is asking whether catecholamine is a steroid, which is a question
about categorization of a substance based on its chemical nature. the document
discusses hormones derived from the amino acid tyrosine, mentioning thyroid
hormone and its role in metabolism and organ development, which is related to
the broader topic of hormones. however, the document does not specifically
address catecholamines or steroids, nor does it provide information to
determine whether catecholamines are steroids. therefore, while there is a
connection in the general subject matter (hormones), the specific question
posed by the query is not answered by the document.
Table 6: Dataset example for filtering stage Q1 (step 4).

⬇

>>> SYSTEM PROMPT

You are a helpful, respectful and creative assistant.

>>> USER INSTRUCTION

You are a ranker agent! Each potential DOCUMENT has a corresponding DOCUMENT
id and you’re tasked with ranking the answers based on their relevancy to the
pair of QUERY, SCENARIO pair. In the evaluation, I want you to rate the
relevancy of the pair according to the following score rubric:

Score 1: The DOCUMENT lacks relevance to the user’s SCENARIO, providing little
to no connection to the user’s job, background, situation, location,
occupation, hobbies, interests, or goals. It fails to consider preferences and
context, resulting in an overall inadequate fit.

Score 2: The DOCUMENT has limited relevance, with only a few elements aligning
with the user’s SCENARIO and QUERY. While some contextual understanding and
preference consideration may be present, it falls short of providing a
comprehensive and well-fitted response.

Score 3: The DOCUMENT demonstrates moderate relevance, capturing some aspects
of the user’s SCENARIO. It shows an adequate contextual fit and considers a
majority of the user’s stated preferences. However, there is room for
improvement in terms of depth and clarity.

Score 4: The DOCUMENT exhibits high relevance, aligning well with the user’s
SCENARIO and covering most relevant aspects. It demonstrates a strong
contextual fit, addresses the user’s preferences effectively, and maintains
high clarity and conciseness. However, there may be minor areas for
improvement.

Score 5: The DOCUMENT is perfectly relevant, precisely addressing all aspects
of the user’s SCENARIO, QUERY, and preferences. It seamlessly integrates with
the user’s context, demonstrating a profound understanding. The DOCUMENT is
exceptionally clear, concise, and exhaustive in providing information,
offering a flawless fit.

You SHOULD ONLY generate the top ranked id from the given search DOCUMENT (id
: 1~10) and no additional comments as [id]. This is VERY IMPORTANT!

You will be given list of DOCUMENT and a pair of QUERY,SCENARIO.

[DOCUMENT LIST]

[1] {DOCUMENT_1}

[2] {DOCUMENT_2}

[3] {DOCUMENT_3}

…

[10] {DOCUMENT_10}

[SCENARIO]

{scenario}

[QUERY]

{query}

You should generate based on the following format:

<Explanation>

{explanation for the ranking}

</Explanation>

<Ranking>

{top ranked DOCUMENT id}

</Ranking>

Please give a top ranked DOCUMENT id with respect to each SCENARIO and QUERY
pair, and provide a score on a scale of 1 to 5 whether it satisfies the
requirements, where a higher score indicates better performance.

Figure 8: Prompt for filtering stage Q2 (step 4).

##  Appendix B Details for Human Verification

#### User Interface.

Ten annotators are instructed to answer three questions as illusted in Figure
9. For the first question (Q1: Is the instruction valid for the search user?)
and second question (Q2: Is the instruction natural for the given query?),
annotators are required to choose either Yes or No. If the answer is No,
annotator are also required to provide a short reason. In the third question
(Q3: Which passage is the most natural for the given instruction and query?),
annotators are provided three passages from the same group, including the
correct answer.

#### Evaluation Settings.

For each instance, the responses from three annotators are reported and regard
the final human judgement by majority voting. In the first and second
questions, we assess the reliability of the annotator’s responses through
kappa coefficient (Randolph, 2005) with the proportion of Yes responses. In
the third question, we compare the final human judgement and InstructIR
dataset through kappa coefficient (Cohen, 1960) and report the top-1 accuracy.

#### Inter-agreement.

Following the Landis and Koch (1977), the kappa coefficient for each question
is interpreted as follow: poor agreement (< 0); slight agreement (0.01-0.20);
fair agreement (0.21–0.40); moderate agreement (0.41–0.60); substantial
agreement (0.61–0.80); almost perfect agreement (0.81–1.00). The first
question shows almost perfect agreement, and the second question shows
moderate agreement among the three annotators for each instance. And the third
question shows substantial agreement between the human judgement and
InstructIR dataset.

Q1. Is the instruction valid for the search user? |
---|---
kappa coefficient (Randolph, 2005) | 0.9133
Proportion of Yes response | 100%
Q2. Is the instruction natural for the given query? |
kappa coefficient (Randolph, 2005) | 0.5267
Proportion of Yes response | 97%
Q3. Which passage is the most natural for the given instruction and query? |
kappa coefficient (Cohen, 1960) | 0.6468
Top-1 Accuracy | 76.5%

Table 7: Human verification results for InstructIR.

(a) Instructions for Annotators.

(b) Questions for each instance.

Figure 9: User Interface for Human Verification. Figure 10: ROUGE-L score
distribution for the instructions.

[Query] what is the annual fee for spirit
---
[Instruction]
I am a budget-conscious traveler looking to cut costs on my frequent trips. I
prefer to invest more in experiences than in transport, so I need accurate
information on the annual fee for Spirit Airlines’ membership or credit card
programs that offer benefits like waived bag fees or discounts.
[Target]
Thrifty explorers who habitually travel can find value in Spirit’s credit
card, especially since the $59 fee is waived for the initial 12 months,
allowing for savings early on. Although to offset the cost from the second
year onwards, cardholders are expected to spend at least $5,900 annually. Yet,
the card can be lucrative for those who accumulate enough expenses, as it
offers benefits like complimentary bag exemptions and fare reductions—perks
that align with the preference to allot more for enriching experiences over
transport expenses.
[Instruction]
As a travel agent, I’m preparing a cost-analysis presentation for a client
who’s interested in low-cost carriers. I am searching for the most up-to-date
annual fee for Spirit’s exclusive membership to include in my comparison chart
alongside other budget airline fees.
[Target]
The exclusive membership offered by Spirit, recognized for its affordability,
incurs an annual fee of $59. This figure is essential for clients evaluating
the comparative costs of low-cost airlines and determining the best value for
their frequent travels. Notably, this fee is typically waived during the
initial year of membership, enhancing the cost-effectiveness for new members.
The fee is a crucial datapoint in the broader context of budget travel
expenses and should be accounted for when mapping out annual travel budgets
for cost-conscious clients.
[Instruction]
I am a college student on a tight budget with a penchant for exploring new
places. I am researching various airlines’ membership fees, particularly
Spirit’s annual fee, to figure out if the cost-saving benefits align with my
limited resources and travel frequency.
[Target]
Balancing a wallet-friendly lifestyle while nurturing your wanderlust can be
challenging, especially when it comes to airline memberships. Taking a closer
look at Spirit’s offerings, there’s a $59 yearly charge for their credit card,
fortunately waived for the initial twelve months. To offset this expense, an
expenditure of $5,900 on the card annually is needed. One should consider
their usual spending habits and the practicality of reaching such a sum in
regular yearly outlays to determine if the membership aligns with prudent
financial management as a diligent academic explorer.
[Instruction]
I am a Human Resources professional planning the company’s travel budget for
the upcoming fiscal year. I need to find the annual fee for Spirit to
calculate whether bulk memberships for our staff travel would be advantageous
compared to pay-per-use options.
[Target]
The annual membership fee for Spirit’s corporate clients is designed to
accommodate businesses looking to manage their travel expenses more
effectively. While individual memberships or pay-per-use arrangements can
potentially add to travel costs over time, opting for an annual corporate
membership can offer substantial savings. This approach ensures that a
company’s travel budget is maximized, especially when considering the volume
of staff travel. Keep in mind that corporate packages may offer additional
perks, such as discounts on group bookings and other travel-related services,
which further enhance the value proposition when compared to transaction-based
fees. Evaluating these membership fees and aligning them with the company’s
projected travel volume will determine the most cost-efficient strategy for
the fiscal year’s travel budget.

Table 8: Dataset example in InstructIR benchmark.


Sunandan Gangopadhyay, Rituparna Mandal and Amitabha Lahiri 11institutetext:
Department of Theoretical Sciences, S.N. Bose National Centre for Basic
Sciences, Block JD, Sector III, Salt Lake, Kolkata 700106, India
11email<EMAIL_ADDRESS><EMAIL_ADDRESS>
# Bianchi-I cosmology in quantum gravity

Sunandan Gangopadhyay 11 Rituparna Mandal  Amitabha Lahiri

###### Abstract

The exact renormalization group flow equations for gravity lead to quantum
corrections of Newton’s constant and cosmological constant. Using this we
investigate the Bianchi-I cosmological model at late times. In particular, we
obtain the scale factors in different directions, and observe that they
eventually evolve into Friedmann-Lemaître-Robertson-Walker (FLRW) universe for
radiation. However, for stiff matter the universe shows a Kasner like
behaviour. The presentation is based on our work published in [9].

###### keywords:

Exact renormalization group equations, Bianchi-I model

##  1 Introduction

In the last couple of decades, exact renormalization group (RG) equations [1]
have become a powerful tool for nonperturbative investigation of both
renormalizable and effective quantum field theories. The framework for exact
RG of Euclidean quantum gravity [2],[3] has opened up the possibility of
investigating cosmological models [4] in a systematic manner. An essential
ingredient of this framework is the effective average action
$\Gamma_{k}[g_{\mu\nu}]$ [5, 6, 7, 8] describing all gravitational phenomena,
including the effect of all loops, at a momentum scale $k$ . Assuming that
there is an ultra-violet fixed point much higher than $k$, all quantum
fluctuations with momenta $p^{2}>k^{2}$ are included, while the momenta
$p^{2}<k^{2}$ are suppressed by an infrared regulator. The form of this
effective average action is

$\displaystyle\Gamma_{k}[g,\bar{g}]=\left(16\pi G(k)\right)^{-1}\int
d^{d}x\sqrt{g}\left\\{-R(g)+2{\Lambda}(k)\right\\}+S_{gf}[g,\bar{g}],$ (1)

where $\bar{g}_{\mu\nu}$ is a background metric and $S_{gf}[g,\bar{g}]$ is a
classical background gauge fixing term. The scale dependent Newton’s constant
$G(k)$ and the cosmological constant $\Lambda(k)$ can be obtained from the RG
equation [9]

$\displaystyle G(k)$ $\displaystyle=G_{0}\left[1-\omega
G_{0}k^{2}+\omega_{1}G^{2}_{0}k^{4}+\mathcal{O}(G^{3}_{0}k^{6})\right]\,$ (2)
$\displaystyle\Lambda(k)$
$\displaystyle=\Lambda_{0}+G_{0}k^{4}\left[\nu+\nu_{1}G_{0}k^{2}+\mathcal{O}(G^{2}_{0}k^{4})\right]\,.$
(3)

##  2 Bianchi I universe with running $G$ and $\Lambda$

We now proceed to investigate the Bianchi type-I cosmology which reads

$\displaystyle ds^{2}=-dt^{2}+a^{2}(t)dx^{2}+b^{2}(t)dy^{2}+c^{2}(t)dz^{2}\,.$
(4)

The improved Einstein field equations with time varying Newton’s gravitational
constant and cosmological constant read

$\displaystyle R_{\mu\nu}-\frac{1}{2}Rg_{\mu\nu}=-8\pi
G(t)T_{\mu\nu}+\Lambda(t)g_{\mu\nu}$ (5)

with $T_{\mu\nu}=(p+\rho)v_{\mu}v_{\nu}+pg_{\mu\nu}$ being the energy-momentum
tensor of a perfect fluid. The covariant conservation of the energy-momentum
tensor yields the consistency equation $8\pi\rho\dot{G}+\dot{\Lambda}=0.$

One possibility is to relate the scale $k$ to $t$ as $k\sim t^{-1}$ in an FLRW
universe [4]. For a Bianchi-I universe one needs to include higher order terms
in $t^{-1}$. We have taken
$k=\frac{\xi}{t}+\frac{\sigma}{t^{2}}+\frac{\delta}{t^{3}}\,.$

Then we can write from eq.(s) (2, 3)

$\displaystyle G(t)$
$\displaystyle=G_{0}\left[1-\frac{\tilde{\omega}G_{0}}{t^{2}}\left(1+\frac{2\tilde{\sigma}}{t}+\frac{2\tilde{\delta}}{t^{2}}+\frac{\tilde{\sigma}^{2}}{t^{2}}\right)+\frac{\tilde{\omega_{1}}G_{0}^{2}}{t^{4}}+\mathcal{O}\left(\frac{t_{Pl}^{6}}{t^{6}}\right)\right]\,,$
(6) $\displaystyle\Lambda(t)$
$\displaystyle=\Lambda_{0}+\frac{G_{0}}{t^{4}}\left[\tilde{\nu}\left(1+\frac{4\tilde{\sigma}}{t}+\frac{4\tilde{\delta}}{t^{2}}+\frac{6\tilde{\sigma}^{2}}{t^{2}}\right)+\frac{\tilde{\nu_{1}}G_{0}}{t^{2}}+\mathcal{O}\left(\frac{t_{Pl}^{4}}{t^{4}}\right)\right]\,,$
(7)

where we have defined
$\tilde{\omega}\equiv\omega\xi^{2},~{}\tilde{\omega_{1}}\equiv\omega_{1}\xi^{4},~{}\tilde{\nu}\equiv\nu\xi^{4}$,
$\tilde{\nu_{1}}\equiv\nu_{1}\xi^{6}$,
$\tilde{\sigma}\equiv\frac{\sigma}{\xi}$,
$\tilde{\delta}\equiv\frac{\delta}{\xi}$, and $t_{Pl}=\sqrt{G_{0}}$ is the
Planck time in natural units.

For an equation of state $p(t)=\Omega\rho(t)$, the conservation of energy-
momentum together with the consistency equation fixes the energy density
$\rho$ and the average scale factor ${\mathcal{R}}=\sqrt[3]{abc}$ to be

$\displaystyle\rho(t)$
$\displaystyle=\frac{1}{4\pi}\left(\frac{\tilde{\nu}}{\tilde{\omega}}\right)\frac{1}{G_{0}t^{2}}\left\\{1+\frac{2\tilde{\sigma}}{t}+\frac{2\tilde{\delta}}{t^{2}}+\frac{\tilde{\sigma}^{2}}{t^{2}}+\left(\frac{2\tilde{\omega_{1}}}{\tilde{\omega}}+\frac{3\tilde{\nu_{1}}}{2\tilde{\nu}}\right)\frac{G_{0}}{t^{2}}+\mathcal{O}\left(\frac{t_{Pl}^{4}}{t^{4}}\right)\right\\}\,,$
(8) $\displaystyle\mathcal{R}(t)$
$\displaystyle=\left[\frac{\mathcal{M}G_{0}}{2}\left(\frac{\tilde{\omega}}{\tilde{\nu}}\right)\right]^{\frac{1}{(3+3\Omega)}}t^{\frac{2}{(3+3\Omega)}}\left\\{1-\frac{1}{(3+3\Omega)}\left(\frac{2\tilde{\sigma}}{t}+\frac{2\tilde{\delta}}{t^{2}}-\frac{(3\Omega+5)}{(3+3\Omega)}\frac{\tilde{\sigma}^{2}}{t^{2}}+\right.\right.$
$\displaystyle\left.\left.\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\left(\frac{2\tilde{\omega_{1}}}{\tilde{\omega}}+\frac{3\tilde{\nu_{1}}}{2\tilde{\nu}}\right)\frac{G_{0}}{t^{2}}\right)+\mathcal{O}\left(\frac{t_{Pl}^{4}}{t^{4}}\right)\right\\}.$
(9)

We now discuss two special cases of cosmic matter separately, namely,
radiation ($\Omega=\frac{1}{3}$) and stiff fluid ($\Omega=1$).

For $\Omega=1/3$, the directional Hubble parameters read,

$\displaystyle\frac{\dot{a}}{a}$
$\displaystyle=H(t)+\frac{l(2+\beta)\alpha}{3}H_{1}(t)\,,$
$\displaystyle\frac{\dot{b}}{b}$
$\displaystyle=H(t)+\frac{l(\beta-1)\alpha}{3}H_{1}(t)\,,$
$\displaystyle\frac{\dot{c}}{c}$
$\displaystyle=H(t)-\frac{l(1+2\beta)\alpha}{3}H_{1}(t)\,.$ (10)

where
$\alpha=\left[\frac{\mathcal{M}G_{0}}{2}\left(\frac{\tilde{\omega}}{\tilde{\nu}}\right)\right]^{-\frac{1}{1+\Omega}}$
and $\beta,l$ are constants of integration. Here the “average Hubble
parameter” $H(t)=\dot{\mathcal{R}}/{\mathcal{R}}$ includes isotropic quantum
corrections,

$H(t)=\frac{1}{2t}\left[1+\frac{\tilde{\sigma}}{t}+\left(2\tilde{\delta}-\tilde{\sigma}^{2}+\left(\frac{2\tilde{\omega_{1}}}{\tilde{\omega}}+\frac{3\tilde{\nu_{1}}}{2\tilde{\nu}}\right)G_{0}\right)\frac{1}{t^{2}}+\mathcal{O}\left(\frac{t_{Pl}^{3}}{t^{3}}\right)\right].$
(11)

The function $H_{1}(t)$ also includes quantum corrections but it will become
irrelevant as we shall see now. Keeping terms up to
$\mathcal{O}\left(\frac{t_{Pl}}{t}\right)^{4}$ in the $tt$-component of
Einstein equation and comparing inverse powers of $t$, we find
$\Lambda_{0}=0$, $\frac{\tilde{\omega}}{\tilde{\nu}}=8/3$, and
$4\left(\frac{\tilde{\nu}}{\tilde{\omega}}\right)\tilde{\sigma}=3\tilde{\sigma}/2-l^{2}\alpha^{2}\left(\beta^{2}+\beta+1\right)/3$,
leading to

$l^{2}\alpha^{2}\left(\beta^{2}+\beta+1\right)=0\,.$ (12)

If $l\neq 0$, we must have $\beta^{2}+\beta+1=0$ which implies that the two
roots of $\beta$ are complex. Since the scale factors must be real, it follows
that $l=0$ . Hence we observe that all the directional Hubble parameters must
be equal, that is, the universe must be FLRW in the presence of radiation.
Thus we conclude that the scale factors of anisotropic Bianchi-I metric flow
to the isotropic FLRW cosmology due to RG flow of the Newton’s gravitational
constant and the cosmological constant.

For $\Omega=1$, which corresponds to stiff matter, the directional Hubble
parameters have the same form as in eq. (10). The expression for the
(isotropic) average Hubble parameter reads

$\displaystyle\frac{\dot{\mathcal{R}}}{\mathcal{R}}=\frac{1}{3t}\left[1+\frac{\tilde{\sigma}}{t}+\left(2\tilde{\delta}-\tilde{\sigma}^{2}+\left(\frac{2\tilde{\omega_{1}}}{\tilde{\omega}}+\frac{3\tilde{\nu_{1}}}{2\tilde{\nu}}\right)G_{0}\right)\frac{1}{t^{2}}+\mathcal{O}\left(\frac{t_{Pl}^{3}}{t^{3}}\right)\right].$
(13)

Proceeding as before, we find $\Lambda_{0}=0\,$ and

$\displaystyle
2\left(\frac{\tilde{\nu}}{\tilde{\omega}}\right)=\frac{1}{3}\left(1-\alpha^{2}l^{2}(\beta^{2}+\beta+1)\right).$
(14)

Writing $l$ in terms of the constant $\beta$, we get

$\displaystyle l$ $\displaystyle=$
$\displaystyle\frac{1}{\alpha}\frac{\sqrt{1-6(\frac{\tilde{\nu}}{\tilde{\omega}})}}{\sqrt{\left(\beta^{2}+\beta+1\right)}}~{}\,.$
(15)

From the reality of the directional Hubble parameters, we get the following
condition from the above equation

$1-6\left(\frac{\tilde{\nu}}{\tilde{\omega}}\right)\geq
0\qquad\Rightarrow\qquad\xi^{2}\leq\frac{1}{6}\frac{\omega}{\nu}~{}.$ (16)

When eq. (16) is an equality, we see from eq. (14) that
$l^{2}(\beta^{2}+\beta+1)=0$ , which implies that $l=0$ since $\beta$ must be
real. Therefore, in that case we again have an FLRW universe.

When eq. (16) is not an equality, that is, $l\neq 0$, we consider the
directional Hubble parameters for large $\beta$. This is a Kasner type
universe, which means that there are expanding and contracting directions.

##  3 Conclusions

The presentation is based on the study of the Bianchi-I cosmological model at
late times taking quantum gravitational effects into account [9]. We used the
RG improved cosmological equations following from the scale dependence of
Newton’s constant and the cosmological constant. For radiation, the scale
factors take the same form and expand at the same rate in all directions. For
stiff matter there are solutions which do not flow to the FLRW universe and
show a Kasner like behaviour. The observation of the quantum gravity induced
time dependence of the Hubble parameter or scale dependence of $G$, $\Lambda$
would determine the validity of this approach.

## References

  * [1] J. Berges, N. Tetradis, and C. Wetterich, “Non-Perturbative Renormalization Flow in Quantum Field Theory and Statistical Physics,” Physics Reports,363, 223-386 (2002).
  * [2] M. Reuter, “Nonperturbative evolution equation for quantum gravity,” Phys. Rev. D 57, no. 10, 971 (1998).
  * [3] M. Reuter and F. Saueressig, “Quantum Gravity and the Functional Renormalization Group: The Road towards Asymptotic Safety,” Cambridge monographs on mathematical physics; doi:10.1017/9781316227596.
  * [4] A. Bonanno, M. Reuter, “Cosmology of the Planck era from a renormalization group for quantum gravity,” Phys. Rev. D 65, 043508 (2002).
  * [5] M. Niedermaier and M. Reuter, “The Asymptotic Safety Scenario in Quantum Gravity”, Living Rev. Rel. 9, 5 (2006).
  * [6] A. Codello, R. Percacci and C. Rahmede, “Investigating the Ultraviolet Properties of Gravity with a Wilsonian Renormalization Group Equation”, Annals Phys. 324 (2009), 414-469.
  * [7] M. Reuter and F. Saueressig, “Renormalization group flow of quantum gravity in the Einstein-Hilbert truncation,” Phys. Rev. D 65, 065016 (2002).
  * [8] K. Falls, D. F. Litim, K. Nikolakopoulos and C. Rahmede, “Further evidence for asymptotic safety of quantum gravity,” Phys. Rev. D 93, no. 10, 104022 (2016).
  * [9] R. Mandal, S. Gangopadhyay and A. Lahiri, “Cosmology of Bianchi type-I metric using renormalization group approach for quantum gravity”, Class. Quant. Grav. 37, 065012 (2020).