# Effective Magnetic Hamiltonian at Finite Temperatures for Rare Earth
Chalcogenides

Zheng Zhang1,2 Jianshu Li1,2 Weiwei Liu1,2 Zhitao Zhang3 Jianting Ji2 Feng
Jin2 Rui Chen4 Junfeng Wang4 Xiaoqun Wang5 Jie Ma5 Qingming Zhang6,2
<EMAIL_ADDRESS>1Department of Physics, Renmin University of China, Beijing
100872, China  2Beijing National Laboratory for Condensed Matter Physics,
Institute of Physics, Chinese Academy of Sciences, Beijing 100190, China
3Anhui Province Key Laboratory of Condensed Matter Physics at Extreme
Conditions, High Magnetic Field Laboratory, Chinese Academy of Sciences, Hefei
230031 China  4Wuhan National High Magnetic Field Center and School of
Physics, Huazhong University of Science and Technology, Wuhan 430074, China
5Department of Physics and Astronomy, Shanghai Jiao Tong University, Shanghai
200240, China  6School of Physical Science and Technology, Lanzhou University,
Lanzhou 730000, China

###### Abstract

Alkali metal rare-earth chalcogenide ARECh2 (A=alkali or monovalent metal,
RE=rare earth, Ch=O, S, Se, Te), is a large family of quantum spin liquid
(QSL) candidates we discovered recently. Unlike YbMgGaO4, most members in the
family except for the oxide ones, have relatively small crystalline electric-
field (CEF) excitation levels, particularly the first ones. This makes the
conventional Curie-Weiss analysis at finite temperatures inapplicable and CEF
excitations may play an essential role in understanding the low-energy spin
physics. Here we considered an effective magnetic Hamiltonian incorporating
CEF excitations and spin-spin interactions, to accurately describe
thermodynamics in such a system. By taking NaYbSe2 as an example, we were able
to analyze magnetic susceptibility, magnetization under pulsed high fields and
heat capacity in a systematic and comprehensive way. The analysis allows us to
produce accurate anisotropic exchange coupling energies and unambiguously
determine a crossover temperature ($\sim$25 K in the case of NaYbSe2), below
which CEF effects fade away and pure spin-spin interactions stand out. We
further validated the effective picture by successfully explaining the
anomalous temperature dependence of electron spin resonance (ESR) spectral
width. The effective scenario in principle can be generalized to other rare-
earth spin systems with small CEF excitations.

_Introduction_ —Recently rare-earth chalcogenide compoundsLiu  _et al._ (2018)
were reported as QSL candidate materialsAnderson (1973, 1987) and have
attracted increasing attention. Extensive studies are focusing on the
representative members of NaYbO2Bordelon  _et al._ (2019); Ding  _et al._
(2019), NaYbS2Baenitz  _et al._ (2018); Sarkar  _et al._ (2019), and
NaYbSe2Ranjith  _et al._ (2019a); Zhang  _et al._ (2020a). Compared to the
previous QSL candidate materials like ZnCu3(OH)6Cl2Helton  _et al._ (2007),
EtMe3Sb[Pd(dmit)2]2Itou  _et al._ (2008), and YbMgGaO4Li  _et al._ (2015a),
the rare earth family has a perfect triangular spin-lattice. The diversity of
the family allows us to control and tune physical properties by various
material engineering ways like element replacement or doping. This provides an
ideal playground for the study of strongly correlated electron systems with
complex magnetic interactions, particularly QSL physics. NaYbSe2 is unique
among the family members. Compared to NaYbO2 and NaYbS2, NaYbSe2 has a Yb-Se-
Yb bond angle closer to $None$et al. . And high-quality and large single
crystal of NaYbSe2 (5 $\sim$ 15 mm) are already availableGray  _et al._
(2003); Ranjith  _et al._ (2019a); Zhang  _et al._ (2020a). We have pointed
out that the smaller energy gap of NaYbSe2 ($\sim$1.92 eV) suggests a higher
possibility of metallization even superconductivityLiu  _et al._ (2018). The
pressure-induced metallization and superconductivity have been reported very
recentlyZhang  _et al._ (2020b); Jia  _et al._ (2020). Particularly, NaYbSe2
shows a relatively small CEF excitationZhang  _et al._ (2020a); Dai  _et al._
(2020) compared to other members like NaYbO2Ding  _et al._ (2019) and
NaYbS2Baenitz  _et al._ (2018). It means that there exists a delicate
competition between CEF excitations and spin-spin interactions at finite
temperatures (T$\sim$J) in the system. In this case, one needs to carefully
figure out a crossover temperature or energy scale which serves as a boundary
wall separating the two parts, and below which we can talk about the low-
energy spin Hamiltonian safely and strictly. More generally, the issue more or
less exists in most compounds of the family except for the oxide ones. The
conventional Curie-Weiss analysis on thermodynamic data has been extensively
adopted for rare earth chalcogenides including NaYbSe2 Liu  _et al._ (2018);
Ranjith  _et al._ (2019a); Ding  _et al._ (2019); Baenitz  _et al._ (2018);
Xing  _et al._ (2019a); Bastien  _et al._ ; Xing  _et al._ (2019b); Ranjith
_et al._ (2019b). The analysis has output valuable basic parameters of the
family members. On the other hand, due to the reason discussed above, most of
the analysis was established on a non-verified or non-guaranteed basis, if we
strictly look into it. We need to investigate the issue accurately and
comprehensively.

In this paper, we considered an effective and practical magnetic Hamiltonian
which contains both CEF effects and spin-spin interactions. We discussed its
physical basis and then applied it to thermodynamic measurements like magnetic
susceptibility, magnetization under pulsed high fields, and heat capacity. The
thermodynamic data are well simulated and fitted in a mean-field (MF) level,
and exchange coupling and other parameters are accurately extracted. The CEF
contribution to magnetic susceptibility was quantitatively analyzed with the
CEF parameters obtained by inelastic neutron scattering (INS) and other
measurements in NaYbSe2. We further applied the effective scenario to our ESR
measurements and successfully explained the anomalous temperature dependence
of spectral width. The above simulations self-consistently output a crossover
energy scale ($\sim$25 K) below which the system can be strictly described by
an anisotropic spin-1/2 picture. This gives us confidence that the feature
near 0.8 K can be identified as the strong short-range correlation and quantum
fluctuations.

_Samples and Experimental Techniques_ —The high-quality single crystals of
NaYbSe2 ($\sim$ 5 mm) were grown by the NaCl-flux methodLiu  _et al._ (2018);
Zhang  _et al._ (2020a). The single crystals were used for magnetic
susceptibility, magnetization, pulsed high magnetic field (PHMF), and ESR
experiments. The polycrystalline samples of NaYbSe2, as well as the
nonmagnetic isostructural polycrystalline samples of NaLuSe2, were synthesized
by the Se-flux methodLiu  _et al._ (2018); Zhang  _et al._ (2020a). The
polycrystalline samples were used for the heat capacity experiments.

$\sim$ 4 mg of NaYbSe2 single crystals were prepared to make magnetic
susceptibility and magnetization measurements. The anisotropic measurements
along the c-axis and in the ab-plane were performed using a Quantum Design
Physical Property Measurement System (PPMS) from 1.8 to 300 K under a magnetic
field of 0 to14 T.

The high-field magnetization data were obtained from the PHMF, where the
magnetic fields were applied parallel to the c-axis and ab-plane of NaYbSe2 at
4.8 K, 0.8 K and, 0.76 K under 0$\sim$25 T, and $\sim$ 10 mg of NaYbSe2
single-crystal crystals were used in the measurements.

$\sim$ 1 mg of NaYbSe2 single crystals were used in the ESR measurements which
were performed with a Bruker EMX plus 10/12 continuous-wave spectrometer at
X-band frequencies ($f$ $\sim$ 9.39 GHz), where the magnetic fields were
applied to the different crystallographic orientations of NaYbSe2 at 2$\sim$55
K.

The zero-field heat capacity data of polycrystalline samples of NaYbSe2
($\sim$ 11 mg) and NaLuSe2 ($\sim$ 10 mg) were performed using PPMS at
1.8$\sim$100 K and He3 - He4 Dilution Refrigerator(DR) at 50 mK$\sim$4 K.

_Spin-Orbit Coupling, Magnetic Ion Environment and Anisotropic Spin
Hamiltonian_ —The electron configuration of the Yb^3+ ion is $4f^{13}$Li  _et
al._ (2015a, b). For rare-earth ions, $4f$ electrons have strong spin-orbit
coupling(SOC)Li  _et al._ (2016, 2017a, 2018). The spectral terms of Yb^3+ ion
with SOC are ${}^{2}F_{7/2}$ and ${}^{2}F_{5/2}$, respectively. Even for free
Yb^3+, the energy level difference between the two states is as high as 1 eV
($\sim$ 10000 K)Zangeneh  _et al._ (2019). This means that it is enough to
focus on the lower ${}^{2}F_{7/2}$ state at our measurement temperatures. The
CEF energy scale in NaYbSe2 is much smaller than that of SOC and the highest
excitation level is about 30 meV($\sim$ 350 K)Zhang  _et al._ (2020a). That is
why we took NaYbSe2 as a representative to investigate the CEF impact on
thermodynamics.

The energy scale of spin-spin exchange coupling in NaYbSe2 is even lower than
10 K. The CEF excitations will be frozen in the spin ground state at very low
temperatures ($T\ll J$). Even in this case, one must also be careful since
some external conditions like pulsed high magnetic fields can still activate
the CEF effect through the van Vleck mechanism and produce a significant
paramagnetic contribution. The solution is to include all the effects in a
comprehensive Hamiltonian. A complete Hamiltonian for our case is as follows:

$\hat{H}=\hat{H}_{0}+\hat{H}_{e-e}+\hat{H}_{SOC}+\hat{H}_{CEF}+\hat{H}_{spin-
spin}+\hat{H}_{zeeman}$ (1)

where $\hat{H}_{0}$ is the kinetic energy of the electrons and the
electron–nucleus interactions, $\hat{H}_{e-e}$ is the interaction between
electrons, $\hat{H}_{SOC}$ is SOC term, $\hat{H}_{CEF}$ is the CEF
contribution, $\hat{H}_{spin-spin}$ is the exchange interaction between spins,
and $\hat{H}_{zeeman}$ is Zeeman splitting energy produced by the external
magnetic field. The first three terms determine the basic shell structure of
an isolated $4f$ ion or atom and their energy scales are far away from our
measurement conditions (about 60 mK $\sim$ 300 K). The other three terms offer
us an effective magnetic Hamiltonian describing thermodynamics at temperatures
of interest. Based on the fact that NaYbSe2 and YbMgGaO4Li  _et al._ (2015b)
share the same triangular spin-lattice and octahedral configuration, the spin-
exchange interaction in NaYbSe2 can also be described by the anisotropic spin
exchange Hamiltonian. The effective magnetic Hamiltonian can be written
asZhang  _et al._ (2020a); Li  _et al._ (2015b):

$\begin{split}\hat{H}_{eff}&=\hat{H}_{CEF}+\hat{H}_{spin-
spin}+\hat{H}_{zeeman}\\\ &=\sum_{i}\sum_{m,n}B_{m}^{n}\hat{O}_{m}^{n}\\\
&\quad+\sum_{\left\langle
ij\right\rangle}[J_{zz}S_{i}^{z}S_{j}^{z}+J_{\pm}(S_{i}^{+}S_{j}^{-}+S_{i}^{-}S_{j}^{+})\\\
&\quad+J_{\pm\pm}(\gamma_{ij}S_{i}^{+}S_{j}^{+}+\gamma_{ij}^{*}S_{i}^{-}S_{j}^{-})\\\
&\quad-\frac{iJ_{z\pm}}{2}(\gamma_{ij}S_{i}^{+}S_{j}^{z}-\gamma_{ij}^{*}S_{i}^{-}S_{j}^{z}+\left\langle
i\longleftrightarrow j\right\rangle)]\\\
&\quad-\mu_{0}\mu_{B}\sum_{i}[g_{ab}(h_{x}S_{i}^{x}+h_{y}S_{i}^{y})+g_{c}h_{c}S_{i}^{z}]\end{split}$
(2)

where $B_{m}^{n}$ is CEF parameters, $J_{zz}$, $J_{\pm}$, $J_{\pm\pm}$, and
$J_{z\pm}$ are anisotropic spin-exchange parameters, the phase factor
$\gamma_{ij}=1,e^{i2\pi/3},e^{-i2\pi/3}$ for the nearest neighbor (NN)
interactions, and $g_{ab}$ and $g_{c}$ ($\sim$ 3.1 and 0.97)Zhang  _et al._
(2020a) represent the Lande factors in the directions of the ab-plane and the
c-axis, respectively. The effective Hamiltonian allows us to quantitatively
estimate the contributions from CEF, spin-exchange interaction, and the
external magnetic field, and accurately fit the fundamental parameters in the
Hamiltonian.

Figure 1:  Magnetic measurements under pulsed high magnetic fields (PHMF).
Open and closed circles are experimental data and solid lines except for the
cyan ones in (d) and (f) are mean-field (MF) simulations. (a) Temperature-
dependent magnetic susceptibility. (b) Magnetization under PHMF at 4.8 K. The
data are calibrated with PPMS up to 14 T. (c) and (e) Magnetization under PHMF
in the ab-plane and along the c-axis at 0.8 K, respectively. Their
corresponding derivatives are shown in (d) and (f). The insets in (d) and (f)
show the possible spin configurations under different magnetic fields. Figure
2:  Magnetic susceptibility and anisotropic exchange parameters. (a), (c) and
(e) Open circles and solid lines are experimental data and mean-field (MF) or
high-temperature series expansion (HTSE) simulations, respectively. (b) and
(d) The deviation $R_{p}$ of the experimental susceptibility from the MF
calculation results in the ab-plane and the c-axis. The white crosses indicate
optimal values. (f) The deviation $R_{p}$ of the experimental susceptibility
from the HTSE calculation results in the c-axis. The white dashed line marks
the possible optimal values. (b), (d) and (f) share the same color scale in
the lower right corner.

_Susceptibility, Magnetization and Pulsed High Magnetic Fields_ —The
experimentally determined CEF excitation levels in NaYbSe2 are located at
15.79 meV, 24.33 meV and, 30.53 meV, respectivelyZhang  _et al._ (2020a); Dai
_et al._ (2020). It is expected that the CEF excitations will significantly
contribute to thermodynamics at finite temperatures. We calculated the CEF
contributions to magnetic susceptibility in the ab-plane and along the c-axis
(shown in Fig. 1(a))SI . A characteristic temperature of $\sim$25 K emerges
from the comparison between experimental and calculation results. The
calculated curves well match the experimental data above the temperature,
while the calculations significantly deviate from the experimental ones below
the temperature. The temperature is an energy scale below which the CEF
contributions are negligible and spin-spin interactions become dominant. It
should be noted that the general orbital diamagnetism in any material is
responsible for the slight difference between the calculated and measured
results above 25 K.

If the effective scenario works, one can anticipate that the CEF effect frozen
below the characteristic temperature can be reactivated by high magnetic
fields through the van Vleck mechanism. That is what one can see in Fig. 1(b),
in which magnetization under PHMF at 4.8 K is measured and the self-consistent
MF simulations including both CEF excitations and spin-spin interactions, are
in agreement with the experiments. More specifically, the agreement along the
c-axis seems good while it shows a deviation in ab-plane under higher magnetic
fields ($\textgreater$ 14 T). There are several possibilities for the
deviation. One reason is the increasing instrumental vibration under higher
fields. Another reason may be that the measurement temperature of 4.8 K is
already close to the scale of spin exchange coupling, and the strong
fluctuations may reduce the reliability of MF approximation. It should be
noted that the PHMF data have been calibrated using PPMS measurements.

The magnetization under PHMF at a lower temperature of 0.8 K is presented in
Fig. 1(c) and 1(e), and the MF self-consistent calculations have also been
carried out. The calculated results follow the experimental data. When further
checking the derivatives of magnetization (Fig. 1(d) and Fig. 1(f)), one may
catch some subtle signs of spin transitions between the spin states induced by
magnetic fields, which have been revealed by neutron scatteringMa  _et al._
(2020); et al. ; Bordelon  _et al._ (2019); Dai  _et al._ (2020). With
increasing temperatures, the spin system sequentially undergoes the phases of
the spin liquid, up-up-down, spin-flop, and spin-flip, which corresponds to
three spin transitions at 3.1 T, 6.8 T and 12 T, respectively. The system
seems to be fully polarized above $\sim$12 T.

The effective magnetic Hamiltonian allows extracting spin exchange and other
fundamental parameters in a more accurate way. We were able to do the fitting
by using a self-consistent equation based on MF approximationSI  (Fig. 2). The
temperature range for fitting is safely selected to be 15$\sim$24 K, according
to the above analysis. The fitting gives the spin exchange parameter $J_{\pm}$
of 3.27 K and $J_{zz}$ of -0.95 K. And we have also obtained the paramagnetic
susceptibility which related to the van Vleck mechanism contributed by the CEF
ground state, and the values in the ab-plane and along the c-axis are 0.0101
${\rm emu\cdot Oe^{-1}\cdot mol^{-1}}$ and 0.0232 ${\rm emu\cdot Oe^{-1}\cdot
mol^{-1}}$, respectively. It should be noted that the self-consistent
calculations begin to deviate from the experimental data below 10 K (Fig. 2(a)
and 2(c)). The deviation can be explained by the fact that the contributions
from $J_{\pm\pm}$ and $J_{z\pm}$ terms are smeared out in the MF
approximation.

The other two anisotropic spin exchange interactions $J_{\pm\pm}$ and
$J_{z\pm}$ are extracted from high-temperature series expansion (HTSE), which
gives the following expression of magnetic susceptibility along c-axisLi  _et
al._ (2015b):

$\chi_{c}=\frac{{{\mu_{0}}g_{c}^{2}\mu_{B}^{2}}}{{4{k_{B}}T}}\left({1-\frac{{3{J_{zz}}}}{{2{k_{B}}T}}-\frac{{3J_{\pm}^{2}+J_{\pm\pm}^{2}+J_{z\pm}^{2}}}{{2k_{B}^{2}{T^{2}}}}+\frac{{15J_{zz}^{2}}}{{8k_{B}^{2}{T^{2}}}}}\right)$
(3)

We made the HTSE fitting for magnetic susceptibility along the c-axis from 5
to 15 K (Fig. 2(e)). The HTSE simulations effectively suppress the divergent
trend in the MF calculations below 10 K, as anisotropic spin exchange
interactions $J_{\pm\pm}$ and $J_{z\pm}$ are taken into account in the HTSE
method. Fig. 2(f) further presents the parameters space for HTSE and the while
line marks the optimal values for $J_{\pm\pm}$ and $J_{z\pm}$, i.e.
$|J_{\pm\pm}|^{2}+|J_{z\pm}|^{2}=0.965$ $K^{2}$. The square form stems from
the expression (3) itself. In Fig. 2, we plot the normalized deviation of the
experimental results from the MF or HTSE calculation one

$R_{p}=Norm\left[\frac{|\chi_{exp}-\chi_{the}|}{\chi_{exp}}\right]$ (4)

Figure 3: (a)Heat capacity of NaYbSe2 and NaLuSe2. (b) Magnetic heat capacity
of NaYbSe2. The red closed circles are experimental data and the purple solid
line is given by CEF calculations. (c) Temperature dependence of magnetic heat
capacity of NaYbSe2 at ultra-low temperatures. (d) Magnetic entropy of
NaYbSe2.

_Zero-Field Heat Capacity_ —In the effective picture, it can be expected that
the weaker CEF in NaYbSe2 will also contribute to magnetic heat capacity. That
motivates us to examine heat capacity of NaYbSe2 under zero field. We measured
the heat capacity of the non-magnetic control sample NaLuSe2 to experimentally
determine the phonon contribution to the heat capacity. As shown in Fig. 3(a),
the heat capacity of NaLuSe2 follows the Debye model wellSI  and is less than
0.01 $\rm J\cdot K^{-1}mol^{-1}$ below 2 K. We obtained magnetic heat capacity
by extracting the phonon part given by NaLuSe2. It should be emphasized here
that the extracted magnetic heat capacity is relatively accurate since the
contribution from phonons is experimentally rather than theoretically
determinedLi  _et al._ (2015a).

Then we calculated the CEF contribution to magnetic heat capacity in NaYbSe2SI
(Fig. 3(b)). There is a reasonable agreement between experiments and
simulations. We can further identify that the magnetic heat capacity at higher
temperatures (>50 K) is dominated by CEF excitations. More importantly, the
CEF contribution to heat capacity goes down to zero below 25 K, consistent
with what we have seen in magnetic susceptibility in Fig. 1. Again it suggests
a characteristic energy scale of 25 K, below which the CEF influence fades
away and the system can be safely described by an effective spin-1/2 picture.

The magnetic heat capacity at ultra-low temperatures is shown in Fig .3 (c)Liu
_et al._ (2018). A broad peak appears in the temperature range of 0.5-2 K. The
similar broad peaks in magnetic heat capacity have been observed in
NaYbO2Bordelon  _et al._ (2019), NaYbS2Baenitz  _et al._ (2018), and many
other quantum spin liquid candidatesLi  _et al._ (2015a). Considering that the
family is almost disorder or impurity free and the CEF contribution has been
ruled out, it is reasonable to attribute the peak to strong spin fluctuations.
And the anisotropic spin-exchange parameters $J_{\pm\pm}$ and $J_{z\pm}$
($\sim$1 K) coincide with the peak position. The rapid increase below 0.1 K is
related to the contribution from nuclei.

By integrating magnetic heat capacity, we obtained magnetic entropy (Fig.
3(d)) which verifies the validity of the effective spin-1/2 picture below 25
K. Compared to YbMgGaO4 ($\sim$50 K)Li  _et al._ (2015b); SI , NaYbSe2 has a
lower characteristic temperature below which the system enters into an
effective spin-1/2 state, due to the smaller first CEF excitation.

Figure 4:  (a) and (b) Differential ESR spectra of NaYbSe2 in the ab-plane and
along the c-axis. (c) Rotation-angle dependence of g factors at 20 K. (d) and
(e) Temperature-dependent of g factors in the ab-plane and along the c-axis.
The red closed circles are experimental data and the blue solid lines are CEF
calculations. (f) ESR full width at half maximum (FWHM) of NaYbSe2 vs. the
temperature. The closed circles are experimental data and the solid lines are
obtained by different fitting schemes.

_Electron Spin Resonance_ —The CEF effect not only affects thermodynamics but
also has an impact on spectral excitations. Here, we studied ESR spectra of
NaYbSe2, which are essentially different from those of YbMgGaO4Li  _et al._
(2015b) and ZnCu3(OH)6Cl2Zorko  _et al._ (2008). The differential ESR spectra
in NaYbSe2 are significantly broadened above 50 K. This is similar to the case
of NaYbS2Sichelschmidt  _et al._ (2019). The rotation-angle dependence of the
obtained g factors at 20 K shows a strong anisotropy. The angle dependence can
be described by the following formula:

$g(\theta)=\sqrt{g_{ab-
plane}^{2}cos^{2}(\theta)+g_{c-axis}^{2}sin^{2}(\theta)}$ (5)

where $\theta$ is the angle between the magnetic field and the c-axis. Then we
obtained $g_{ab}=3.10$ and $g_{c}=0.96$. The values are consistent with those
extracted from inelastic neutron scattering experimentsZhang  _et al._
(2020a). Generally, the g factors given by ESR experiments are more direct and
accurate. The temperature dependence of $g_{ab-plane}$ and $g_{c-axis}$ is
shown in Fig. 4(d) and 4(e) and is compared to the CEF calculations. Again we
meet the characteristic temperature of $\sim$25 K, below which the g-factors
both in the ab-plane and along the c-axis exhibit little temperature
dependence. But the g factors rapidly increase with the temperature above 25
K. We can develop a quantitative understanding using the following formula
involving CEF excitations:

$\displaystyle\ket{\psi_{+}}=\frac{1}{Z}\sum_{i=0,1,2,3}exp(-E_{i}/k_{B}T)\ket{\psi_{i,+}}$
(6)
$\displaystyle\ket{\psi_{-}}=\frac{1}{Z}\sum_{i=0,1,2,3}exp(-E_{i}/k_{B}T)\ket{\psi_{i,-}}$
(7) $\displaystyle g_{ab-
plane}=g_{j}|\bra{\psi_{\pm}}m_{J_{\pm}}\ket{\psi_{\mp}}|$ (8) $\displaystyle
g_{c-axis}=2g_{j}|\bra{\psi_{\pm}}m_{J_{z}}\ket{\psi_{\pm}}|$ (9)

where $Z$ is the partition function, $g_{j}=8/7$, $E_{i}$ and
$\ket{\psi_{\pm}}$ are Lande factor, the CEF energy level, and the CEF
eigenstateZhang  _et al._ (2020a); SI . The simulation curves coincide with
the experimental one (Fig. 4(d) and 4(e))). The difference between experiments
and simulations may be explained by the fact that the CEF eigenstates adopted
here come from INS experiments while the g factors here are given by ESR
measurements. It is worth noting that the g factors slightly increase with
temperature, especially above 25 K. As the temperature increases, more and
more magnetic ions are excited to higher CEF states. And this eventually
causes a subtle change in the g factor.

The temperature dependence of FWHM of ESR spectra in NaYbSe2, is shown in Fig.
4(f). The FWHM data in the ab-plane and along the c-axis share a similar
temperature evolution, but with distinct amplitudes. This is simply an
indication of strong anisotropy. We can discuss FWHM in the two temperature
regimes separated by the characteristic temperature. For $T>$ 25 K, the rapid
increase of FWHM with temperature is related to the CEF excitations. Using the
following formula, we can estimate the energy gap ($\Delta E$) between the CEF
ground state and the first excited state in NaYbSe2Sichelschmidt  _et al._
(2019, 2020).

$\mu_{0}\Delta H\simeq\frac{1}{exp(\Delta E/T)-1}$ (10)

This gives the CEF excitation gap of 133 K (ab-plane) and 169 K (c-axis). The
result is well consistent with that measured by INS experiments ($\sim$ 180
K)Zhang  _et al._ (2020a). Below 25 K, FWHM slowly increases with decreasing
temperatures. This is due to anisotropic spin-exchange interactions.
$\mu_{0}\Delta H$ can be fitted to the following empirical formulaSeehra and
Castner (1970); Reynaud  _et al._ (2001); Sichelschmidt  _et al._ (2019):

$\mu_{0}\Delta H\simeq\frac{A}{(T-T_{\theta})^{p}}$ (11)

where $A$ is the proportionality coefficient, $p$ is the power coefficient,
and $T_{\theta}$ is the Weiss temperature, and $T_{\theta}^{ab}=-3J_{\pm}$ and
$T_{\theta}^{c}=-3J_{zz}/2$Li  _et al._ (2015b). $J_{\pm}$ and $J_{zz}$ have
been obtained by making MF fitting for magnetic susceptibility discussed
above. The fitting gives $A=0.4626$, $p=0.7458$ in the ab-plane and
$A=0.4126$, $p=0.3024$ along the c-axis. Since the FWHM is proportional to the
local magnetic susceptibility, the magnetic susceptibility in the ab-plane and
along the c-axis is very similar to ESR FWHM in both the trend and the
intensity. This is reflected in the similarity between the empirical formula
(11) and Curie-Weiss law, regardless of the power coefficient $p$. The power
coefficient $p$ of less than 1 stems from spin-spin correlation below 25 K and
strong magnetic anisotropy. Both factors push the magnetic behavior at low
temperatures away from the picture of simply isolated ions (Curie-Weiss law).
The strong anisotropy is again demonstrated by the distinct power coefficients
$p$ in the ab-plane and along the c-axis.

The broadening of ESR linewidth at low temperatures has also been observed in
other spin-frustrated materials, such as ZnCu3(OH)6Cl2Zorko  _et al._ (2008)
and LiNiO2Reynaud  _et al._ (2001). It should be pointed out that the
temperature evolution of FWHM in NaYbSe2 is substantially different from that
of YbMgGaO4Li  _et al._ (2015b). No obvious broadening of ESR FWHM in YbMgGaO4
can be seen when varying temperatures. This is explained by the higher CEF
levels of YbMgGaO4 ($\textgreater$300 K)Li  _et al._ (2017b). In addition, the
spin-exchange parameters of YbMgGaO4 is smaller (less than 1 K)Li  _et al._
(2015b). Therefore, spin-exchange interactions have little effect on ESR
linewidth at relatively high temperatures.

_Summary_ —We conducted a comprehensive analysis of thermodynamic data and the
ESR spectra of NaYbSe2. We considered an effective magnetic Hamiltonian
involving strong SOC and CEF effects. At the MF level, we can well understand
the thermodynamic data of NaYbSe2. The analysis determines a characteristic
temperature of $\sim$25 K. The CEF effect makes a major contribution to
thermodynamics above the temperature. On the other hand, the spin-spin
correlation begins to play a major role below the temperature. The
consideration allows us to separate the CEF contribution from thermodynamic
data and accurately determine the fundamental parameters of the spin system.
Furthermore, we applied the effective scenario to ESR measurements and
successfully explained the unusual temperature evolution of FWHM. The present
study provides a realistic scheme to include the CEF influence in a
comprehensive way and may be generalized to other rare-earth systems.

_Acknowledgments_ —This work was supported by the National Key Research and
Development Program of China (2017YFA0302904 & 2016YFA0300500) and the NSF of
China (11774419 & U1932215). A portion of this work was performed on the
Steady High Magnetic Field Facilities, High Magnetic Field Laboratory, CAS.
The CEF calculations of magnetic susceptibility is based on our python program
package MagLabSI , which is designed for the fitting, calculations, and
analysis of the CEF magnetism, and will be continuously updated. The MF
analysis and fitting of magnetic susceptibility, magnetization and ESR data at
low temperatures are based on MathWorks® MATLAB® software (Academic License
for Renmin University of China).

## References

  * Liu  _et al._ (2018) Weiwei Liu, Zheng Zhang, Jianting Ji, Yixuan Liu, Jianshu Li, Xiaoqun Wang, Hechang Lei, Gang Chen, and Qingming Zhang, “Rare-earth chalcogenides: A large family of triangular lattice spin liquid candidates,” Chin. Phys. Lett. 35, 117501 (2018).
  * Anderson (1973) P.W. Anderson, “Resonating valence bonds: A new kind of insulator?” Mater. Res. Bull. 8, 153–160 (1973).
  * Anderson (1987) P.W. Anderson, “The resonating valence bond state in La2CuO4 and superconductivity,” Science 235, 1196–1198 (1987).
  * Bordelon  _et al._ (2019) Mitchell M. Bordelon, Eric Kenney, Chunxiao Liu, Tom Hogan, Lorenzo Posthuma, Marzieh Kavand, Yuanqi Lyu, Mark Sherwin, N. P. Butch, Craig Brown, M. J. Graf, Leon Balents, and Stephen D. Wilson, “Field-tunable quantum disordered ground state in the triangular-lattice antiferromagnet NaYbO2,” Nat. Phys. 15, 1058–1064 (2019).
  * Ding  _et al._ (2019) Lei Ding, Pascal Manuel, Sebastian Bachus, Franziska Grußler, Philipp Gegenwart, John Singleton, Roger D. Johnson, Helen C. Walker, Devashibhai T. Adroja, Adrian D. Hillier, and Alexander A. Tsirlin, “Gapless spin-liquid state in the structurally disorder-free triangular antiferromagnet NaYbO2,” Phys. Rev. B. 100, 144432 (2019).
  * Baenitz  _et al._ (2018) M. Baenitz, Ph. Schlender, J. Sichelschmidt, Y. A. Onykiienko, Z. Zangeneh, K. M. Ranjith, R. Sarkar, L. Hozoi, H. C. Walker, J.-C. Orain, H. Yasuoka, J. van den Brink, H. H. Klauss, D. S. Inosov, and Th. Doert, “NaYbS2: A planar spin-1/2 triangular-lattice magnet and putative spin liquid,” Phys. Rev. B. 98, 220409(R) (2018).
  * Sarkar  _et al._ (2019) R. Sarkar, Ph. Schlender, V. Grinenko, E. Haeussler, Peter J. Baker, Th. Doert, and H.-H. Klauss, “Quantum spin liquid ground state in the disorder free triangular lattice NaYbS2,” Phys. Rev. B. 100, 241116(R) (2019).
  * Ranjith  _et al._ (2019a) K. M. Ranjith, S. Luther, T. Reimann, B. Schmidt, Ph. Schlender, J. Sichelschmidt, H. Yasuoka, A. M. Strydom, Y. Skourski, J. Wosnitza, H. Kühne, Th. Doert, and M. Baenitz, “Anisotropic field-induced ordering in the triangular-lattice quantum spin liquid ${\mathrm{NaYbSe}}_{2}$,” Phys. Rev. B 100, 224417 (2019a).
  * Zhang  _et al._ (2020a) Zheng Zhang, Xiaoli Ma, Jianshu Li, Guohua Wang, D. T. Adroja, T. G. Perring, Weiwei Liu, Feng Jin, Jianting Ji, Yimeng Wang, Xiaoqun Wang, Jie Ma, and Qingming Zhang, “Crystalline electric-field excitations in quantum spin liquids candidate NaYbSe2,” (2020a), arXiv:2002.04772 [cond-mat.str-el] .
  * Helton  _et al._ (2007) J. S. Helton, K. Matan, M. P. Shores, E. A. Nytko, B. M. Bartlett, Y. Yoshida, Y. Takano, A. Suslov, Y. Qiu, J.-H. Chung, D. G. Nocera, and Y. S. Lee, “Spin dynamics of the spin-1/2 kagome lattice antiferromagnet ZnCu3(OH)6Cl2,” Phys. Rev. Lett. 98, 107204 (2007).
  * Itou  _et al._ (2008) T. Itou, A. Oyamada, S. Maegawa, M. Tamura, and R. Kato, “Quantum spin liquid in the spin-1/ 2 triangular antiferromagnet EtMe3Sb[Pd(dmit)2]2,” Phys. Rev. B. 77, 104413 (2008).
  * Li  _et al._ (2015a) Yuesheng Li, Haijun Liao, Zhen Zhang, Shiyan Li, Feng Jin, Langsheng Ling, Lei Zhang, Youming Zou, Li Pi, Zhaorong Yang, Junfeng Wang, Zhonghua Wu, and Qingming Zhang, “Gapless quantum spin liquid ground state in the two-dimensional spin-1/2 triangular antiferromagnet YbMgGaO4,” Sci. Rep. 5, 1–8 (2015a).
  * (13) Jie Ma et al., Unpublished .
  * Gray  _et al._ (2003) A. K. Gray, B. R. Martin, and P. K. Dorhout, “Crystal structure of sodium ytterbium (iii) selenide, NaYbSe2,” Z. Kristallogr. NCS 218, 19–19 (2003).
  * Zhang  _et al._ (2020b) Zheng Zhang, Yunyu Yin, Xiaoli Ma, Weiwei Liu, Jianshu Li, Feng Jin, Jianting Ji, Yimeng Wang, Xiaoqun Wang, Xiaohui Yu, and Qingming Zhang, “Pressure induced metallization and possible unconventional superconductivity in spin liquid NaYbSe2,” (2020b), arXiv:2003.11479 [cond-mat.supr-con] .
  * Jia  _et al._ (2020) Ya-Ting Jia, Chun-Sheng Gong, Yi-Xuan Liu, Jian-Fa Zhao, Cheng Dong, Guang-Yang Dai, Xiao-Dong Li, He-Chang Lei, Run-Ze Yu, Guang-Ming Zhang, and Chang-Qing Jin, “Mott transition and superconductivity in quantum spin liquid candidate NaYbSe2,” Chin. Phys. Lett. 37, 097404 (2020).
  * Dai  _et al._ (2020) Peng-Ling Dai, Gaoning Zhang, Yaofeng Xie, Chunruo Duan, Yonghao Gao, Zihao Zhu, Erxi Feng, Chien-Lung Huang, Huibo Cao, Andrey Podlesnyak, Garrett E. Granroth, David Voneshen, Shun Wang, Guotai Tan, Emilia Morosan, Xia Wang, Lei Shu, Gang Chen, Yanfeng Guo, Xingye Lu, and Pengcheng Dai, “Spinon fermi surface spin liquid in a triangular lattice antiferromagnet NaYbSe2,” (2020), arXiv:2004.06867 [cond-mat.str-el] .
  * Xing  _et al._ (2019a) Jie Xing, Liurukara D. Sanjeewa, Jungsoo Kim, G. R. Stewart, Andrey Podlesnyak, and Athena S. Sefat, “Field-induced magnetic transition and spin fluctuations in the quantum spin-liquid candidate CsYbSe2,” Phys. Rev. B. 100, 220407(R) (2019a).
  * (19) G. Bastien, B. Rubrecht, E. Haeussler, P. Schlender, Z. Zangeneh, S. Avdoshenko, R. Sarkar, A. Alfonsov, S. Luther, Y. A. Onykiienko, H. C. Walker, H. Kühne, V. Grinenko, Z. Guguchia, V. Kataev, H. H. Klauss, L. Hozoi, J. van den Brink, D. S. Inosov, B. Büchner, A. U. B. Wolter, and T. Doert, “Long-range magnetic order in the S=1/2 triangular lattice antiferromagnet KCeS2,” arXiv:2005.06300v3 [cond-mat.str-el] .
  * Xing  _et al._ (2019b) Jie Xing, Liurukara D. Sanjeewa, Jungsoo Kim, William R. Meier, Andrew F. May, Qiang Zheng, Radu Custelcean, G. R. Stewart, and Athena S. Sefat, “Synthesis, magnetization, and heat capacity of triangular lattice materials NaErSe2 and KErSe2,” Phys. Rev. Mater. 3, 114413 (2019b).
  * Ranjith  _et al._ (2019b) K. M. Ranjith, D. Dmytriieva, S. Khim, J. Sichelschmidt, S. Luther, D. Ehlers, H. Yasuoka, J. Wosnitza, A. A. Tsirlin, H. Kühne, and M. Baenitz, “Field-induced instability of the quantum spin liquid ground state in the ${J}_{\mathrm{eff}}=\frac{1}{2}$ triangular-lattice compound ${\mathrm{NaYbO}}_{2}$,” Phys. Rev. B 99, 180401(R) (2019b).
  * Li  _et al._ (2015b) Yuesheng Li, Gang Chen, Wei Tong, Li Pi, Juanjuan Liu, Zhaorong Yang, Xiaoqun Wang, and Qingming Zhang, “Rare-earth triangular lattice spin liquid: a single-crystal study of YbMgGaO4,” Phys. Rev. Lett. 115, 167203 (2015b).
  * Li  _et al._ (2016) Yao-Dong Li, Xiaoqun Wang, and Gang Chen, “Anisotropic spin model of strong spin-orbit-coupled triangular antiferromagnets,” Phys. Rev. B 94, 035107 (2016).
  * Li  _et al._ (2017a) Yao-Dong Li, Yuan-Ming Lu, and Gang Chen, “Spinon fermi surface $U(1)$ spin liquid in the spin-orbit-coupled triangular-lattice mott insulator ${\mathrm{YbMgGaO}}_{4}$,” Phys. Rev. B 96, 054445 (2017a).
  * Li  _et al._ (2018) Yao-Dong Li, Yao Shen, Yuesheng Li, Jun Zhao, and Gang Chen, “Effect of spin-orbit coupling on the effective-spin correlation in ${\mathrm{YbMgGaO}}_{4}$,” Phys. Rev. B 97, 125105 (2018).
  * Zangeneh  _et al._ (2019) Ziba Zangeneh, Stanislav Avdoshenko, Jeroen van den Brink, and Liviu Hozoi, “Single-site magnetic anisotropy governed by interlayer cation charge imbalance in triangular-lattice AYbX2,” Phys. Rev. B. 100, 174436 (2019).
  * (27) Supplementary Information .
  * Ma  _et al._ (2020) Jie Ma, Jianshu Li, Yong Hao Gao, Changle Liu, Qingyong Ren, Zheng Zhang, Zhe Wang, Rui Chen, Jan Embs, Erxi Feng, Fengfeng Zhu, Qing Huang, Ziji Xiang, Lu Chen, E. S. Choi, Zhe Qu, Lu Li, Junfeng Wang, Haidong Zhou, Yixi Su, Xiaoqun Wang, Qingming Zhang, and Gang Chen, “Spin-orbit-coupled triangular-lattice spin liquid in rare-earth chalcogenides,” (2020), arXiv:2002.09224 [cond-mat.supr-con] .
  * Zorko  _et al._ (2008) A. Zorko, S. Nellutla, J. van Tol, L. C. Brunel, F. Bert, F. Duc, J.-C. Trombe, M. A. de Vries, A. Harrison, and P. Mendels, “Dzyaloshinsky-moriya anisotropy in the spin-1/2 kagome compound ZnCu3(OH)6Cl2,” Phys. Rev. Lett. 101, 026405 (2008).
  * Sichelschmidt  _et al._ (2019) Jörg Sichelschmidt, Philipp Schlender, Burkhard Schmidt, Michael Baenitz, and Thomas Doert, “Electron spin resonance on the spin-1/2 triangular magnet NaYbS2,” J. Phys. Condens. Matter 31, 205601 (2019).
  * Sichelschmidt  _et al._ (2020) Jörg Sichelschmidt, Burkhard Schmidt, Philipp Schlender, Seunghyun Khim, Thomas Doert, and Michael Baenitz, “Effective spin-1/2 moments on a Yb^3+ triangular lattice: An ESR study,” in  _Proceedings of the International Conference on Strongly Correlated Electron Systems (SCES2019)_ (Journal of the Physical Society of Japan, 2020).
  * Seehra and Castner (1970) Mohindar S. Seehra and T.G. Castner, “Critical broadening of the epr line width in MnF2,” Solid State Commun. 8, 787–790 (1970).
  * Reynaud  _et al._ (2001) F. Reynaud, D. Mertz, F. Celestini, J.-M. Debierre, A. M. Ghorayeb, P. Simon, A. Stepanov, J. Voiron, and C. Delmas, “Orbital frustration at the origin of the magnetic behavior in LiNiO2,” Phys. Rev. Lett. 86, 3638 (2001).
  * Li  _et al._ (2017b) Yuesheng Li, Devashibhai Adroja, Robert I Bewley, David Voneshen, Alexander A Tsirlin, Philipp Gegenwart, and Qingming Zhang, “Crystalline electric-field randomness in the triangular lattice spin-liquid YbMgGaO4,” Phys. Rev. Lett. 118, 107202 (2017b).


# Pegasus-1 Technical Report

###### Abstract

This technical report introduces Pegasus-1, a multimodal language model
specialized in video content understanding and interaction through natural
language.111Please cite this paper as (Twelve Labs, 2024). Please see
Authorship Section at the end of this report for the full list of
contributors. Pegasus-1 is designed to address the unique challenges posed by
video data, such as interpreting spatiotemporal information, to offer nuanced
video content comprehension across various lengths. This technical report
overviews Pegasus-1’s architecture, training strategies, and its performance
in benchmarks on video conversation, zero-shot video question answering, and
video summarization. We also explore qualitative characteristics of Pegasus-1
, demonstrating its capabilities as well as its limitations, in order to
provide readers a balanced view of its current state and its future
direction.222Pegasus-1 blog post and API are available at
https://www.twelvelabs.io/blog/upgrading-pegasus-1

##  1 Introduction

In the evolving landscape of large language models (LLMs) [1, 18, 2, 4, 3],
cultivating the video understanding capabilities of LLMs [8, 21, 22, 7, 12,
10, 9, 15, 16] has emerged as a frontier of innovation and practical utility.
This technical report introduces and analyzes Pegasus-1, a state-of-the-art
multimodal model that offers versatile capabilities in interpreting,
generating, and interacting with video content through natural language.

The major goal of developing Pegasus-1 is to overcome inherent challenges in
video data that contain multiple modalities within a single format. Critical
to this understanding is to interpret the temporal sequence of visual data,
capturing the essence of movement and change over time while providing a
spatially detailed analysis within each frame of the video. Concurrently,
audio information should be incorporated to enhance the interpretation of
visual elements and ensure a nuanced understanding of video content. Moreover,
handling a wide range of video lengths is essential to cover various video
types from short clips to extended footage.

In this technical report, we discuss our approach to dealing with these
challenges to enable Pegasus-1 to understand video content comprehensively.
The discussion will include a brief description of its model architecture,
training data, and training strategies, providing a broad perspective on the
factors contributing to Pegasus-1’s advanced video understanding capabilities.

Pegasus-1 achieves new state-of-the-art results in video conversation
benchmark [7], zero-shot video question answering [19, 20], and video
summarization [5, 17]. By outperforming both open-source and proprietary
models, Pegasus-1 demonstrates its generalization capabilities in
understanding complex video data.

This report also presents a wide range of qualitative results to offer
insights into Pegasus-1’s capabilities and to explore potential emerging use
cases. Our objective is to unveil and preview the range of functionalities
Pegasus-1 can potentially deliver, acknowledging that while these capabilities
showcase new possibilities, they may not yet be fully reliable or consistent.
Through this exploration, we aim to highlight Pegasus-1’s potential to
catalyze new applications and advancements in the field.

Despite its capabilities, Pegasus-1 has inherent limitations that users should
be aware of, as we aim for transparency while continuing to refine and enhance
its functionalities. These limitations are briefly discussed within this
technical report. Our objective is to provide users with a comprehensive
understanding of Pegasus-1’s current strengths, weaknesses and areas for
growth, ensuring informed usage and setting accurate expectations for its
performance and applicability.

##  2 Model Architecture and Training

###  2.1 Model Architecture

Figure 1: Architectural Overview of Pegasus-1: Pegasus-1 is comprised of three
main components: 1) the Video Encoder model for generating multi-modal
embeddings from visual and audio inputs, the Video-Language Alignment Model
for synchronizing video and text representations, and the Large Language Model
for generating contextually relevant textual output.

To efficiently comprehend video data, it is crucial to seamlessly integrate
both auditory and visual information while effectively managing longer video
lengths. In our architectural strategy, we have developed an architecture
capable of jointly processing audio and visual data, and it is distinguished
by its efficient computation tailored for longer videos. As depicted in Fig.
1, Pegasus-1 is structured into a comprehensive tripartite framework designed
to encode, align, and decode:

  * •

Video Encoder Model: Powered by our Marengo 2.6 [6], the video encoder model
processes the video input to produce rich embeddings from both video frames
and audio speech recognition (ASR) data. These embeddings are dense
representations that capture the visual and auditory essence of the video
content.

  * •

Video-language Alignment Model: The model maps video embeddings to
corresponding language embeddings, establishing a shared space where video and
text representations are coherently aligned. Such an alignment is pivotal,
enabling the model to discern and correlate the visual content of the video
with its descriptive language effectively. Additionally, to enhance the
processing of longer video lengths, we employ token reduction techniques aimed
at minimizing the input token sizes within the large language model, thereby
optimizing computational efficiency and maintaining context integrity over
longer durations.

  * •

Large Language Model (Decoder Model): The large language model decoder takes
the aligned embeddings and user prompts to generate coherent and contextually
relevant text output. This output can range from descriptive summaries to
answers to specific questions about the video content. This large language
model decoder leverages efficient attention mechanisms for efficiently
handling the long context of video data.

###  2.2 Training

The training of Pegasus-1, a state-of-the-art multimodal foundation model,
focuses on the harmonious understanding of various modalities in video data,
including visual and audio modes. To achieve this, we capitalized on the
unique strengths of each modality: employing spatial understanding from image
data and deriving insights from audio data for robust audio understanding.
Notably, despite the video having a richer information source compared to
standalone images or audio, there is a marked scarcity of high-quality video
data. We address this challenge by amassing an extensive collection of
proprietary text annotations for training. Specifically, we annotate 10M+
diverse videos with highly detailed descriptions. These descriptions capture
most of the events that appear in each video, incorporating detailed real-
world knowledge about activities and objects, including specific names of
objects and technical terms for various sports.333Note that the number of
tokens per video depends on its length and spatiotemporal resolution. For
instance, if one assumes 1 frame per second, 256 patches per frame, and
3-minute (180s) per video for simplicity, each video would be equivalent to
around 46k tokens and 10M videos would be equivalent to 0.46T tokens.

Our training process for Pegasus-1 consists of two phases: pretraining phase
and instruction tuning phase. During the pretraining phase, we initialize both
the video encoder and large language model with pre-trained weights,
subsequently training with our expansive multi-modal datasets, which include
the rich detailed video data mentioned previously. In the second phase, we
conduct supervised instruction finetuning, utilizing proprietary multimodal
instruction datasets to refine the model’s responsiveness to user
instructions. However, a significant challenge in such multi-stage training is
the risk of catastrophic forgetting [13], where the pretrained model loses
previously acquired knowledge upon assimilating new information.

To mitigate this, our training strategy incorporates two meticulously planned
stages, aiming to harmonize the integration of new data while preserving
existing knowledge. The cornerstone of this strategy is the selective
unfreezing of model parameters combined with precise adjustments to learning
rates during the training progression. This approach ensures that Pegasus-1
not only acquires new capabilities efficiently but also retains and refines
its previously learned skills, establishing a robust foundation for advanced
video-language understanding.

##  3 Results on Benchmarks

We evaluate Pegasus-1’s performance across well-established video large
language model benchmarks, including video conversation [12], zero-shot video
question answering [20, 19], and video summarization [5]. This section
presents the performance of Pegasus-1 in comparison to both proprietary and
open-source models.

###  3.1 Comparison Models

We compare our model against the following baselines:

  * •

Gemini models (proprietary) [16, 15]: Gemini is Google’s proprietary
multimodal model, also known for its state-of-the-art performance in video-
language tasks. To ensure the accuracy and relevance of our comparative
analysis, we excerpt the performance of Gemini 1.0 and Gemini 1.5 from
Google’s technical report for the Video Question Answering benchmark. For
other benchmarks that are not included in their reports, such as video
conversation and summarization, we run the February 17 2024 version of the
publicly available Gemini API.444Gemini API documentation:
https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-
started/intro_gemini_pro_vision_python.ipynb Note that Gemini 1.5 Pro API is
not available as of March 30th, 2024 and is only accessible through its
playground-like preview access.

  * •

Open-source models: We compare Pegasus-1 with VideoChat [7], Video-ChatGPT
[12], Video LLAMA [21], BT-adapter [10], LLaMA-VID [9], and VideoChat2 [8]. We
use the benchmark results reported in their papers if they are available.

###  3.2 Results

#### Video conversation benchmark.

Model | Correctness | Detail | Context | Temporal | Consistency
---|---|---|---|---|---
VideoChat [7] | 2.23 | 2.5 | 2.53 | 1.94 | 2.24
Video-ChatGPT [12] | 2.4 | 2.52 | 2.62 | 1.98 | 2.37
Video LLAMA [21] | 1.96 | 2.18 | 2.16 | 1.82 | 1.79
BT-adapter [10] | 2.68 | 2.69 | 3.27 | 2.34 | 2.46
LLaMA-VID [9] | 3.07 | 3.05 | 3.6 | 2.58 | 2.63
VideoChat2 [8] | 3.02 | 2.88 | 3.51 | 2.66 | 2.81
Gemini Pro [16] | 2.98 | 2.99 | 3.44 | 2.32 | 2.32
Pegasus-1 | 3.79 | 3.76 | 4.29 | 3.34 | 4.03
Table 1: In the comparison conducted using the video conversation benchmark
[12], the open-ended generation outputs are evaluated by a language model
(LLM) through a comparison with the ground truth answers. This evaluation
specifically focuses on five distinct aspects. Notably, Pegasus-1 outperforms
both open-source models and a leading proprietary model. These results were
compiled in February 2024. At that time, the Gemini 1.5 API was not yet
generally available, hence its exclusion from the comparative table.

| Action | Direction | Speed |  | Event
---
Order
| Attribute
---
Change
Avg.
| Fine | Coarse | Object | Camera | Absolute | Relative | Order | Color | Size | Combined | Other |
Random | 39.7 | 40.1 | 39.8 | 39.0 | 40.8 | 42.0 | 41.5 | 40.4 | 39.9 | 38.9 | 39.4 | 40.5
mPLUG-Owl | 48.8 | 66.1 | 38.7 | 36.8 | 42.2 | 38.4 | 42.0 | 41.7 | 44.7 | 41.9 | 39.9 | 44.4
V-LLaVA | 55.0 | 83.1 | 41.9 | 42.1 | 44.7 | 41.6 | 46.2 | 45.7 | 45.7 | 43.5 | 49.8 | 49.6
VideoChat2 | 60.4 | 77.2 | 45.0 | 40.8 | 52.6 | 41.5 | 44.8 | 50.8 | 47.9 | 49.9 | 44.3 | 50.8
Pegasus-1 | 79.8 | 92.6 | 46.1 | 44.4 | 48.1 | 44.2 | 56.3 | 51.7 | 49.6 | 46.7 | 45.7 | 57.1

Table 2: TempCompass [11] assesses the temporal understanding capabilities of
video language models across five dimensions. Pegasus-1 clearly outperforms
competing open-source models.

In Table 1, we present the evaluation results of Pegasus-1 on a video-based
conversation benchmark [12]. The results indicate that Pegasus-1 performs
robustly in the context of video conversations, achieving notable performance
in various evaluative dimensions. Specifically, Pegasus-1 attains a score of
3.79 in Correctness and 4.29 in Context, which demonstrates its effective
processing and understanding of video conversation content and its relevant
contexts. These scores are indicative of Pegasus-1’s performance in critical
areas such as Correctness, Detail, Contextual Awareness, Temporal
Comprehension, and Consistency, emphasizing its capability to interpret and
engage with video-based dialogue.

#### Zero-shot video question answering.

Model | ActivityNet-QA | NExT-QA
---|---|---
| Test Split (%) | Test Split (%)
Video-ChatGPT [12] | 35.2 | -
VideoChat2 [8] | 49.1 | 61.7
Gemini 1.0 Pro [16] | 49.8 | 28.0
Gemini 1.0 Ultra [16] | 52.2 | 29.9
Gemini 1.5 Pro [15] | 56.7 | -
Pegasus-1 | 59.9 | 71.1
Table 3: Zero-shot video QA results on ActivityNet-QA [20] and NExT-QA dataset
[19]. Note that Gemini 1.5 Pro result is not available for NExT-QA as it is
not reported in its technical report.

Additionally, we detail Pegasus-1’s performance on two popular zero-shot video
question-answering benchmarks—ActivityNet-QA [20] and NExT-QA [19]—in Table 3.
In video question-answering tasks, Pegasus-1 showcases significant enhancement
in zero-shot capabilities when evaluated on the ActivityNet-QA and NExT-QA
datasets compared to open-source models and Gemini series. The generated
responses are provided to GPT-3.5 Turbo, which is utilized to determine if the
predictions align with the ground truth answers..

#### Video summarization.

Model | Correctness | Detail | Context | Average
---|---|---|---|---
Video-ChatGPT [12] | 1.19 | 1.33 | 1.42 | 1.31
VideoChat2 [8] | 1.78 | 1.52 | 1.98 | 1.76
Gemini 1.0 Pro [16] | 1.65 | 1.69 | 1.94 | 1.76
Pegasus-1 | 2.30 | 2.58 | 2.75 | 2.54
Table 4: Utilizing the video conversation benchmark [12], which assesses
generation output across five dimensions, our evaluation focuses on three
specific aspects relevant to the video summarization task. Within this
framework, Pegasus-1 demonstrates superior performance compared to other
baseline models. ActivityNet with detailed caption proposed in [12] was
utilized as a source dataset.

To evaluate the performance of video summarization, we follow the video
conversation benchmarks [12] and evaluate the generated summary in three
areas: Correctness of Information, Detailed Orientation, and Contextual
Understanding. GPT-4 measures the score for each metric by setting the
reference summary as ground truth. Here, we use the ActivityNet detailed
caption dataset proposed in [12], and Table 4 shows these results. As shown
above, Pegasus-1 outperforms the baseline models in all metrics by a
significant margin.

#### Temporal understanding.

As a video language model, Pegasus-1 is expected to have the capability to
grasp temporal information. This capability is put to the test with
TempCompass [11], a dataset designed specifically for evaluating temporal
understanding. TempCompass focuses on assessing five aspects of temporal
information: action, direction, speed, event order, and attribute change. To
evaluate temporal understanding, the benchmark includes synthetic videos
created from a single original video. These manipulations involve reversing
the footage, playing it forward at varying speeds, or slowing it down, thus
creating conflicting scenarios where the visual content remains constant while
the temporal dynamics are altered. According to the findings presented in
Table 2, Pegasus-1 outperforms other open-source benchmarks, notably
surpassing VideoChat2 [8], which is also designed with a focus on temporal
information. These results are detailed in TempCompass [11]. Given that
TempCompass is formatted as a multiple-choice QA dataset requiring parsable
responses, an intermediate output from Pegasus-1 is employed. Although this
intermediate output conveys the same information as the final response, it is
more readily adjusted to align with the dataset’s parsing requirements.

##  4 Capabilities of Pegasus-1

Here, we present qualitative results to offer insight into Pegasus-1’s
capabilities to explore potential emerging use cases. Also, to show the
performance gap between Pegasus-1 and other state-of-the-art models, we
provide a comparative analysis with qualitative results from Gemini 1.0 Pro
(as of March 15th, 2024, using its publicly available API) and Gemini 1.5 Pro
(as of March 30th, 2024 using its playground interface). In our analysis, we
exclusively include Gemini 1.0 Pro and Gemini 1.5 Pro as our comparison
baseline. Our selection criteria are twofold: the baseline must support native
video input, and it must be widely acknowledged for its robust performance
across an extensive array of benchmarks.

###  4.1 Comparison Protocol for Qualitative Results

The presented prompts are carefully crafted, ensuring clarity and coherence to
accurately assess the model’s performance. While crafting the prompts, we are
meticulous in avoiding excessive obscurity or unnecessary complexity, aiming
to ensure that the prompts remain straightforward and facilitate an
unambiguous evaluation of the model’s capabilities.

Additionally, we focus on comparing the abilities in visually accurate
responses, a capability not achievable through language model runoff. Language
model runoff refers to the tendency of language models to rely on their strong
linguistic priors for generating contextually relevant information. For
example, once a concept, such as "beach" is recognized by a Large Language
Model (LLM), it can utilize its inherent knowledge to produce responses
related to that concept, like "sunny weather" or "people swimming," without
direct evidence from pixel information [14]. This process, while generating
coherent responses, may not accurately reflect the visual content.

###  4.2 Characteristics of Pegasus-1

#### Verbose output

Pegasus-1 is designed to follow instructions closely, providing extensive
details about a video’s audio and visual elements. As a result, the outputs
from Pegasus-1 can be verbose, offering an abundance of information in
response to queries. However, these features are deliberate for two key
reasons.

First, Pegasus-1 is not equipped with chat interface capabilities; therefore,
it is designed to provide comprehensive responses in a single interaction.
This approach aligns well with video understanding tasks, given that videos
are rich in information. Second, our research indicates that users tend to
prefer detailed explanations over shorter ones.

Despite these benefits, reinforcing Pegasus-1 to generate elongated and
detailed responses may lead to hallucinations. To mitigate this, we plan to
implement a feature allowing users to adjust the level of detail in
Pegasus-1’s responses in the future.

#### Categories of capabilities

Our observations of Pegasus-1 reveal various capabilities in video
understanding. These capabilities encompass a wide range of functionalities,
demonstrating the model’s proficiency across different dimensions of video
analysis, including, but not limited to:

  * •

Real-world knowledge: Pegasus-1 integrates extensive real-world knowledge,
enabling it to contextualize and interpret video content with remarkable
accuracy and depth.

  * •

Video-based reasoning: The model exhibits sophisticated reasoning abilities,
drawing inferences from the video data to construct coherent understandings
and insights.

  * •

3D spatial understanding: Pegasus-1’s capability to comprehend 3D spatial
relationships within video frames allows it to interpret complex scenes and
object interactions, enhancing its analysis of video content that requires an
understanding of depth and space.

  * •

Temporal reasoning: Pegasus-1 maintains an awareness of the chronological
sequence of events in a video, a critical feature for understanding
narratives.

  * •

Visual referring prompts: Building upon traditional text-based prompting
methods, Pegasus-1 adopts visual referring prompts, enabling modifications
directly within the pixel space. This advanced approach allows users to use
arrows, boxes, or other visual markers to pinpoint specific regions, directing
the model’s focus to these emphasized areas.

###  4.3 Real World Knowledge

In this section, we assess the real-world knowledge capabilities of Pegasus-1,
a video language model proficient in recognizing and identifying real-world
entities within video content. We evaluated Pegasus-1 by challenging it to
name specific real-world objects shown across different video scenarios.

The qualitative results, illustrated in Fig. 2 to 5, substantiate Pegasus-1’s
capacity for such identification. In Fig. 2, Pegasus-1 demonstrates its
precision by correctly identifying the Tuileries Garden from merely its visual
cues, distinguishing specific landscape features like gravel pathways and
manicured hedges, which are indicative of its advanced visual recognition
capabilities. Conversely, Fig. 3 depicts Pegasus-1’s ability to discern the
city of Kyoto from a compilation of distinct sceneries, showcasing its
adeptness at integrating multiple visual inputs to form a coherent
identification. In this instance, Pegasus-1 not only recognizes individual
elements but also understands their collective representation of Kyoto,
evidencing a sophisticated level of contextual comprehension.

Further reinforcing its object recognition proficiency, Pegasus-1 accurately
identifies a Bugatti Chiron in Fig. 4, contrasting with a Gemini Pro 1.0’s
less precise identification. Moreover, as seen in Fig. 5, Pegasus-1 accurately
discerns the video game title "The Legend of Zelda: Breath of the Wild,"
attesting to its robust capability in accurately categorizing even within
specific entertainment domains.

These instances collectively affirm Pegasus-1’s adeptness at extracting and
interpreting nuanced visual details, affirming its advanced real-world
knowledge capabilities and its potential application in varied contexts where
accurate visual recognition is paramount.

###  4.4 Video-based Reasoning

We investigate the visual reasoning capabilities of Pegasus-1, an endeavor
that necessitates a synergy of visual interpretation and logical reasoning. To
adeptly navigate visual reasoning tasks, the model must harness these
competencies in tandem to formulate precise responses to inquiries. Such tasks
encompass a broad spectrum of challenges, and to demonstrate the model’s
proficiency, we showcase three distinct scenarios, each highlighting a
different aspect of Pegasus-1’s visual reasoning ability.

The first scenario, depicted in Fig. 6 and 7, illustrates Pegasus-1’s ability
to comprehend ongoing events within videos and anticipate future developments
based on the present context. In Fig. 6, Pegasus-1 accurately predicts the
immediate reactions required in a post-tidal surge scenario, contrasting with
Gemini 1.0 Pro’s less precise anticipation of events. Similarly, Figure 7
demonstrates Pegasus-1’s adeptness at inferring the likely cause and
subsequent unfolding of events encountered during a night drive, indicative of
its nuanced understanding of context and sequence.

The second scenario, represented by Fig. 8, demands advanced reasoning, such
as estimating property values in New York. Here, Pegasus-1 showcases its
capacity for intricate reasoning by synthesizing visual cues into coherent,
logical conclusions, outperforming Gemini 1.0 Pro’s more superficial analysis
as highlighted in the respective figures.

The third scenario focuses on commonsense reasoning, where Pegasus-1 discerns
the unusualness of a scenario involving a cat “driving” a car, as shown in
Fig. 9. This case emphasizes Pegasus-1’s ability to differentiate between the
ordinary and the extraordinary, leveraging visual details to arrive at
plausible interpretations that align with real-world logic.

These examples collectively attest to Pegasus-1’s advanced capability in
interpreting and reasoning from visual content.

###  4.5 3D spatial Understanding

In assessing the 3D spatial understanding capabilities of Pegasus-1, we focus
on its ability to interpret and navigate complex spatial relationships as
depicted in video content. The task involves extracting and synthesizing
spatial information from visual cues within a video to provide accurate
navigational directions within a three-dimensional space.

Illustrated in Fig. 10, the video presents an exploration of various spaces
within a residential setting, challenging the model to interpret and
articulate the spatial layout effectively. The specific task, “Suppose I am in
front of the porch. Tell me the way to get to the fridge”, requires Pegasus to
deduce a path through the house’s structure, demonstrating an understanding
that transcends mere object recognition to encompass spatial orientation and
logical pathway deduction.

Pegasus-1’s response to this task is indicative of its advanced spatial
reasoning. It successfully deciphers the sequence of spaces and their
interconnections presented in the video: from the porch, through the entrance,
up the stairs, and finally to the kitchen where the fridge is located. This
performance highlights Pegasus-1’s ability to process, integrate, and
articulate complex 3D spatial data, affirming its competency in navigating and
explaining real-world spatial scenarios based on video content.

###  4.6 Temporal Reasoning

In the realm of video-language modeling, grasping the sequential nature of
audio-visual content is imperative. A model must discern and maintain the
chronological order of events within a video to understand and interpret the
content accurately. Pegasus-1 exemplifies this capability, as illustrated in
Fig. 11 and 12, by effectively tracking and preserving the temporal sequence
of events and demonstrating an acute awareness of their order.

In Fig. 11, the video details the meticulous process of preparing biryani, a
dish requiring sequential steps. Pegasus-1 articulates each stage
methodically: beginning with the preparation of chicken, followed by the
spices, then layering with onions and rice, and culminating in the combination
of all ingredients. This delineation underscores Pegasus-1’s adeptness at
temporal reasoning within a culinary context. Conversely, Gemini 1.0 Pro
exhibits a discrepancy by misinterpreting the absence of narration, an element
often expected in instructional cooking videos, demonstrating a gap in its
temporal analysis.

Figure 12 presents a video montage of iconic London landmarks. Here, Pegasus-1
distinguishes itself by accurately capturing and relating the sequence of the
presented landmarks, illustrating its proficiency in processing and
understanding the temporal flow between distinct scenes. On the other hand,
Gemini 1.0 Pro’s response includes inaccuracies and unwarranted assumptions,
such as the misidentification of geographical features and the incorrect
attribution of narration, highlighting its challenges with temporal and
contextual accuracy in video content interpretation.

These instances affirm Pegasus-1’s temporal reasoning capabilities, showcasing
its ability to interpret and convey the flow of events in video narratives
accurately, a critical skill for comprehensive video understanding.

###  4.7 Visual Referring Prompts

Pegasus-1 demonstrates a notable capacity to direct its attention to specific
segments within a video, as indicated by visual markers. This proficiency is
pivotal when the model is required to interpret and respond to cues that
emphasize particular areas or actions within the video frame. Figure 14, 15,
and 13 exemplify this ability, where discerning and focusing on highlighted
details are essential for accurate analysis.

In Fig. 13, a soccer match clip singles out a player with a red circle,
drawing attention to their role in the game. Pegasus-1 correctly focuses on
this highlighted player, mentioning that the player is participating in
offensive play to move the ball forward to the opposing teams goal.
Conversely, Gemini 1.0 Pro inaccurately identifies the player’s team position,
demonstrating a lapse in contextual understanding when analyzing the specified
visual cue.

Figure 14 and 15 delve into a technical examination of Gerrit Cole’s pitching
mechanics, with certain movements accentuated by white circles and green
arrows. Pegasus-1 aptly acknowledges the importance of these marked regions,
showing a clear comprehension of the content delineated by the white circles,
in stark contrast to Gemini 1.0 Pro’s inability to interpret these annotations
correctly.

These scenarios underline Pegasus-1’s adeptness in responding to visual
referring prompts, showcasing its advanced capability to integrate and analyze
specific visual information within broader video content.

###  4.8 Other Specific Use Cases

In this section, we highlight some domain-specific use cases, such as medical
video analysis and anomaly detection in dashcam videos that could help
autonomous driving. Figure 16 and 17 feature two instances of anomaly
detection captured by a dashcam. The former presents a more urgent and
significant scenario (a car catching on fire), while the latter depicts a less
critical situation (failing to signal when turning left). In both instances,
Pegasus-1 accurately identifies the events unfolding in the videos. Figure 18
shows that Pegasus-1 effectively comprehends the complexities of a surgery
video, which contains highly specialized and rare content.

Figure 2: The video, captured in the Tuileries Garden, lacks any verbal
narration, allowing Pegasus-1 to deduce the location purely through visual
cues. Impressively, it identifies key visual elements such as manicured
hedges, ornamental trees, and the distinctive facade of a grand
building—critical factors in pinpointing the exact setting. This capability
showcases Pegasus-1’s proficiency in analyzing and interpreting essential
visual information to arrive at accurate conclusions. Figure 3: The input
video showcases various landscapes across Kyoto. Pegasus’s response
meticulously orders scenes to reflect their chronological appearance in the
video, detailing Kyoto’s landmarks in sync with their sequence. It
intelligently infers the filming season as autumn, deduced from the presence
of autumn foliage. Adhering precisely to the inquiry, Pegasus concludes by
accurately providing Kyoto’s coordinates, demonstrating its adeptness in both
recognizing visual patterns and extracting relevant contextual information
from the imagery presented. Figure 4: The video showcases a Bugatti Chiron
performing high-skill maneuvers, including producing smoke from its tires.
Pegasus accurately identifies the exact car model, showcasing its broad
knowledge of the real world. This highlights Pegasus’s ability to recognize
and interpret specific details from visual inputs. Figure 5: The video
features a brief gameplay moment from "Legend of Zelda: Breath of the Wild."
Unlike the Gemini models, Pegasus provides a detailed visual analysis, such as
noting the enemy’s health bar, pinpointing the scene’s location on a wooden
platform, and mentioning the electric blue light during an attack.
Additionally, Pegasus accurately identifies the game’s title, showcasing its
ability to interpret and convey comprehensive visual details along with
correct contextual understanding. Figure 6: The video captures a tidal surge,
setting a complex scenario for Pegasus-1 to analyze. Tasked with predicting
the aftermath, Pegasus-1 demonstrates both an understanding of the current
situation and the ability to foresee post-disaster actions, such as securing
boats and assessing damage. This highlights Pegasus-1’s comprehensive
reasoning capabilities and awareness of real-world phenomena and their
consequences. Figure 7: The video depicts an individual driving on a highway
at night, encountering traffic cones along the way. Pegasus-1 accurately
deduces that the scene likely represents a construction site, a reasonable
explanation for the cones’ presence on the highway during nighttime, despite
the absence of explicit context. Pegasus-1 predicts the driver will continue
to follow the road, demonstrating its ability to process and interpret not
only the visual but also the temporal dynamics of the scenario. This is
evident from Pegasus-1’s responses highlighted in green, showcasing its
nuanced understanding. Conversely, Gemini 1.0 Pro’s interpretation
significantly deviates from the video’s content, offering an unrelated
response that fails to acknowledge the situational elements depicted. Figure
8: The video features a compilation of real estate photos. Both Pegasus-1 and
the Gemini families offer logical evaluations, suggesting a price range that
encompasses the actual value of $695,000. Pegasus-1 notably excels by
discerning specific visual details, such as signs of a child’s presence in the
nursery and the quality of hardwood flooring, showcasing its adeptness at
detailed visual analysis. Figure 9: The video depicts a cat humorously
positioned as if "driving" a Tesla, clad in a floral shirt and interacting
with the steering wheel. Pegasus-1 effectively notes the comedic illusion of
the cat driving and observes the car’s motion through the rapidly changing
scenery, highlighting its capability to capture and interpret key visual
elements within the content. Figure 10: Pegasus-1 adeptly reconstructs spatial
information from the video, guiding seamlessly from the porch into the living
room while accurately noting the adjacency of the kitchen. However, within the
Gemini families, there’s a slight misunderstanding of the house’s 3D layout.
Often, they misdirect after ascending the stairs or misplace the location of
the fridge, necessitating correction. Figure 11: The video showcases the step-
by-step process of making biryani. Pegasus explains the sequence of actions in
detail: starting with cooking the chicken, followed by preparing the spices,
then adding onions, cooking the rice, and finally combining all the components
to create the final dish, biryani. Also, an intriguing error in Gemini 1.0
Pro’s response is the false detection of narration—a feature commonly expected
in such videos—when, in fact, there is no narration present. Figure 12: The
video comprises a compilation of short shots capturing landmarks of London.
Pegasus meticulously describes each shot, adhering to the chronological order
in which they appear. Conversely, Gemini families occasionally omit certain
shots, resulting in a relatively low recall rate. One notable error in Gemini
1.0 Pro is its erroneous suggestion of the presence of "narration by a British
man with a posh accent," despite the absence of any narration in the video.
This incorrect detection by Gemini 1.0 Pro is particularly noteworthy since
such videos commonly feature narration, yet in this instance, no narration is
present. Figure 13: The video features a brief segment of a soccer match, with
a player highlighted in red engaged in an offensive play. Pegasus-1 adeptly
focuses on this player within the red circle, accurately noting their
involvement in offensive maneuvers aimed at advancing the ball closer to the
opposing team’s goal. Figure 14: The video centers on an analysis of Gerrit
Cole’s pitching mechanics, utilizing white circles and green arrows to
visually underscore pivotal aspects. When asked about the areas designated by
the white circles, Pegasus-1 showcases an understanding of their role as
visual markers pertaining to pitching form. In contrast, Gemini 1.0 Pro
entirely disregards the instructions. However, Gemini 1.5 Pro shifts its focus
to the white circle following the query, demonstrating robustness compared to
its previous version. Figure 15:  The green arrow signifies the intended
direction of the ball’s delivery by the pitcher. Both Pegasus-1 and the Gemini
families comprehend the green arrow as a visual indicator of the pitching
direction. Figure 16: This dashcam video captures a car on fire. Initially,
the vehicle seems small and distant, but as the video continues, it comes
closer. Pegasus-1 accurately detects the hazardous condition, highlighting the
smoke rising from the roadside. Additionally, Pegasus-1 notes the dashcam-
equipped car’s approach toward an exit, accurately reflecting the events in
the video. On the other hand, Gemini 1.0 Pro fails to identify the danger
presented in the dashcam video. However Gemini 1.5 Pro detects the car on
fire, demonstrating superior performance compared to Gemini 1.0 Pro.  Figure
17: The dashcam video captures casual scenes of driving around a town. When
queried about any potentially dangerous moments requiring law enforcement
attention, Pegasus-1 identifies two incidents present in the video. One
involves a man riding a bike on the road without a proper helmet, and the
other pertains to a vehicle turning left without signaling. Gemini 1.0 Pro
falls short of providing the correct answer, continuously printing responses
that is not present in the video. (For convenience, it was truncated).
Conversely, Gemini 1.5 Pro offers a relatively convincing response, noting
that the cyclist is riding in the middle of the road. Figure 18: This video is
part of a medical seminar demonstrating surgical management. While endoscopic
videos are relatively rare, Pegasus-1 captures essential information. It
recognizes that it is an educational video for the neurosurgery field,
implying the potential usage of Pegasus-1 for domain-specific cases.

##  5 Limitations

Despite the initial success of Pegasus in various domains, there are
limitations that have been identified and are targeted for improvement in
future developments.

#### Maximum Video Duration

The current iteration of Pegasus is optimized for videos up to 15 minutes in
length. A notable limitation is its diminishing efficiency in capturing the
relationships between timestamps that are significantly apart as the video
length increases. Enhancing the model’s capability to process longer videos
and maintain accuracy in connecting distant frames stands as a principal
objective in our road map.

#### Hallucinations

Pegasus, like other vision-language models, is prone to hallucinations, which
can manifest as inaccurately identifying nonexistent objects, actions, or the
sequential order of events. These inaccuracies underscore the need for further
refinement in the model’s interpretive accuracy.

#### Safety and Biases

Safety and biases represent a critical area of concern. Pegasus can
unintentionally perpetuate biases present in its training data, leading to the
reinforcement of stereotypes or incorrect perceptions. Additionally, the model
may validate or produce content that is inappropriate or harmful, posing risks
of spreading misinformation. The reliance on visual cues for interpretation
further complicates this issue, as ambiguities in these cues can result in
interpretative errors. Moreover, the current capabilities of Pegasus do not
fully grasp the complexity of social contexts and dynamics, potentially
leading to outputs that are contextually inappropriate. Efforts to mitigate
these challenges will focus on developing strategies for creating more
equitable and ethically aware models and improving content moderation
mechanisms.

#### Chat Capability

The absence of chat functionality in the current version of Pegasus is also
noted as a limitation. Plans to incorporate chat capabilities in future
versions are underway, aiming to enhance the model’s interactivity and user
engagement.

Overall, addressing these limitations requires a dedicated approach to
research and development, with the aim of advancing the model’s performance,
ethical standards, and user experience.

## Authorship

This is a joint team effort across multiple functional groups including model
and data, engineering, product, and business development. (“core” indicates
Core Contributor; first-name alphabetical order)

#### Model

Raehyuk Jung (core), Hyojun Go (core), Jaehyuk Yi (core), Jiho Jang (core),
Aiden Lee, Cooper Han, Jae Lee, Jeff Kim, Jin-Young Kim, Junwan Kim, Kyle
Park, Lucas Lee, Mars Ha, Minjoon Seo,

#### Data

Daniel Kim (core), Jay Suh (core)

#### Deployment

Abraham Jo, Ed Park, Hassan Kianinejad, SJ Kim, Tony Moon, Wade Jeong

#### Product

Andrei Popescu, Esther Kim, EK Yoon, Genie Heo, Henry Choi, Jenna Kang, Kevin
Han, Noah Seo, Sunny Nguyen, Ryan Won, Yeonhoo Park

#### Business & Operations

Anthony Giuliani, Dave Chung, Hans Yoon, James Le, Jenny Ahn, June Lee,
Maninder Saini, Meredith Sanders, Soyoung Lee, Sue Kim, Travis Couture

## References

  * [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * [2] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.  See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6, 2023.
  * [3] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  Journal of Machine Learning Research, 24(240):1–113, 2023.
  * [4] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.  Mistral 7b.  arXiv preprint arXiv:2310.06825, 2023.
  * [5] Twelve Labs.  Introducing Pegasus-1, 2023.
  * [6] Twelve Labs.  Introducing marengo-2.6-medium, 2024.
  * [7] KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao.  Videochat: Chat-centric video understanding.  arXiv preprint arXiv:2305.06355, 2023.
  * [8] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al.  Mvbench: A comprehensive multi-modal video understanding benchmark.  arXiv preprint arXiv:2311.17005, 2023.
  * [9] Yanwei Li, Chengyao Wang, and Jiaya Jia.  Llama-vid: An image is worth 2 tokens in large language models.  arXiv preprint arXiv:2311.17043, 2023.
  * [10] Ruyang Liu, Chen Li, Yixiao Ge, Ying Shan, Thomas H Li, and Ge Li.  One for all: Video conversation is feasible without video instruction tuning.  arXiv preprint arXiv:2309.15785, 2023.
  * [11] Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou.  Tempcompass: Do video llms really understand videos?  arXiv preprint arXiv:2403.00476, 2024.
  * [12] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan.  Video-chatgpt: Towards detailed video understanding via large vision and language models.  arXiv preprint arXiv:2306.05424, 2023.
  * [13] Michael McCloskey and Neal J Cohen.  Catastrophic interference in connectionist networks: The sequential learning problem.  In Psychology of learning and motivation, volume 24, pages 109–165. Elsevier, 1989.
  * [14] Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick.  Linearly mapping from image to text space.  In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
  * [15] Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al.  Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.  arXiv preprint arXiv:2403.05530, 2024.
  * [16] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  arXiv preprint arXiv:2312.11805, 2023.
  * [17] Twelve Labs Team.  Pegasus-1 open beta: Setting new standards in video-language modeling, 2024.
  * [18] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [19] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua.  Next-qa: Next phase of question-answering to explaining temporal actions.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777–9786, 2021.
  * [20] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao.  Activitynet-qa: A dataset for understanding complex web videos via question answering.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 9127–9134, 2019.
  * [21] Hang Zhang, Xin Li, and Lidong Bing.  Video-llama: An instruction-tuned audio-visual language model for video understanding.  arXiv preprint arXiv:2306.02858, 2023.
  * [22] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao.  Llama-adapter: Efficient fine-tuning of language models with zero-init attention.  arXiv preprint arXiv:2303.16199, 2023.