# Implications of the electron-phonon coupling in CuPb9(PO4)6O for high-
temperature superconductivity: an ab initio study

Hari Paudyal<EMAIL_ADDRESS>Department of Physics and Astronomy,
University of Iowa, Iowa City, Iowa 52242, USA  Ames National Laboratory of
the US DOE, Iowa State University, Ames, IA 50011, USA  Michael E. Flatté
<EMAIL_ADDRESS>Department of Physics and Astronomy, University
of Iowa, Iowa City, Iowa 52242, USA  Department of Applied Physics, Eindhoven
University of Technology, Eindhoven, The Netherlands  Durga Paudyal
<EMAIL_ADDRESS>Ames National Laboratory of the US DOE, Iowa State
University, Ames, IA 50011, USA  Department of Electrical and Computer
Engineering, Iowa State University, Ames, IA 50011, USA

(August 27, 2023)

###### Abstract

We report $ab~{}initio$ calculations of the electronic and vibrational
properties in CuPb9(PO4)6O, including the electron-phonon coupling strength
via strong-coupling Migdal-Eliashberg theory. We verify the presence of
appealing flat electronic bands near the Fermi level, a strong hybridization
between the Cu $3d$ and O $2p$ states, and soft low-energy phonons, which can
suggest high-temperature superconducting behavior. However, the electron-
phonon coupling strength appears insufficient to overcome the Coulomb
repulsion between an electron pair and thus does does not support high-
temperature superconductivity in CuPb9(PO4)6O via the conventional electron-
phonon Migdal-Eliashberg mechanism. Even neglecting Coulomb repulsion of the
electron pair we find this electron-phonon coupling suggests a superconducting
transition temperature less than 2 K.

Achieving ambient pressure superconductivity at room temperature remains a
challenging and elusive goal Flores_PhysRep20perspective ; Lilia_JPCM22 . The
advent of MgB2 Nagamatsu_Nat01MgB2  has proved useful for superconducting
technologies despite a critical temperatureBraccini_PhysC07MgB2 ;
Penco_IEEE07MgB2  far from the maximum achieved in a material due to the
ability to form and process MgB2 into useful forms. Compacted superhydrides
have exhibited conventional superconducting behavior at elevated temperatures
under extreme pressure Drozdov_Nat15H3S ; Somayazulu_PRL19 ; Kong_NatCommun21
, however, employing such pressure limits these materials’ potential utility.
Therefore, the quest for room temperature ambient pressure superconductors
represents a groundbreaking pursuit and its achievement would revolutionize
applications including energy transmission and storage, transportation, and
healthcare Yao2021superconducting_applications . Recently CuxPb10-x(PO4)6O
with $0.9<x<1.1$ has exhibited a sudden drop in resistivity, along with a
negative diamagnetic susceptibility, and distinct voltage spikes below 400 K,
attributed to the strain due to the Cu doping Lee_arxiv23First ;
Lee_arxiv23Supercond , leading to suggestions it might exhibit high-
temperature superconductivity. A solid-state reaction recipe was presented for
the synthesis of this material, however, the experimental characterization
only measures an average crystal structure and cannot detect possible changes
in local geometries with respect to the partly filled O sites. The absence to
date of independent reproduction of these results Liu_arxiv23_expt ;
Kumar_arxiv23 ; Kumar_Super23  requires further experimental validation, and
suggests that theoretical studies may be helpful. Some electronic structure
calculations Kumar_arxiv23 ; Si-Held_arxiv23 ; Si-Simone_arxiv23 ; Sun-
Vlamidir_arxiv23 ; Lai_JMST23  provide a few encouraging but limited results.
The appearance of flat CuO bands, with a relatively high density of states
(DOS) at the Fermi level ($E_{\rm F}$), could support high-temperature
superconductivity. However, those calculations do not rule out magnetic
configurations which could suppress the superconducting behavior Sun-
Vlamidir_arxiv23 ; Zhang_arxiv23 . A few studies suggest substantial effective
electron-phonon coupling due to this elevated DOS in the proximity to $E_{\rm
F}$ in this material Griffin_arxiv23 ; Cabezas_arxiv23 , but did not directly
simulate the electron-phonon coupling.

In this letter, we report a systematic study of the electronic, vibrational,
and electron-phonon coupling properties of CuPb9(PO4)6O through ab initio
calculations within strong-coupling electron-phonon superconductivity (Migdal-
Eliashberg) theory. The calculated lattice parameters are found in good
agreement with previously reported results, as well as confirming that lattice
compression occurs with Cu substitution. The insulating to metallic transition
is attributed to the presence of flat Cu $3d$ states hybridized with O $2p$
states at $E_{\rm F}$. As a result, our electronic structure calculations
exhibit a relatively large DOS at $E_{\rm F}$, and multiple Van Hove
singularities Sun-Vlamidir_arxiv23 . Phonon calculations show that
CuPb9(PO4)6O is dynamically stable with many low frequency vibration modes
below 5 meV. Despite having a high DOS at $E_{\rm F}$ and low energy phonons,
we found a weak electron-phonon coupling strength that would only provide
pairing below 2K for conventional superconductivity in this material.

Our ab initio calculations have been performed using Quantum ESPRESSO
giannozzi2017o  with relativistic norm-conserving pseudopotentials
PseudoDojo2018  in conjunction with the Perdew-Burke-Ernzerhof perdew1996o
exchange-correlation functional in the generalized gradient approximation. A
plane wave kinetic-energy (charge density) cutoff value of 80 Ry (320 Ry), a
$\Gamma$-centered $8\times 8\times 8$ Monkhorst-Pack k-mesh Monkhors1976s ,
and a Methfessel and Paxton smearing Methfessel1989s  width of 0.01 Ry have
been used for the Brillouin-zone sampling. The atomic positions and lattice
parameters are optimized until the self-consistent energy is converged within
$2.7\times 10^{-5}$ eV and the maximum force on each atom is less than 0.005
eV/Å. For the DOS and Fermi surface calculations, denser k-meshes of $16\times
16\times 16$ and $32\times 32\times 32$ are used. The dynamical matrices and
the linear variation of the self-consistent potential are calculated within
density-functional perturbation theory Baroni2001s  on an irreducible $4\times
4\times 4$ q mesh.

Figure 1: Crystal structure of Pb10(PO4)6O in a (a) single unit cell and (b)
supercell. The extra O with 25% occupancy lies outside of the PO4 tetrahedra.
There are two distinct non-equivalent Pb($6h$) and Pb($4f$) sites. The
Pb($6h$) atoms form two triangular layers (with 3.71 Å separation and 60o
rotation), whereas the Pb($4f$) atoms adopt a hexagonal arrangement. For Cu
substitution we considered the Pb($4f$) site marked with a green cross.

Migdal-Eliashberg theory Allen1975s ; Margine2013s , as implemented in the
Electron-Phonon Wannier (EPW) code ponce2016o ; EPW2023 , is used to
investigate the electron-phonon properties. Wannier interpolation Pizzi2019s
on a uniform $8\times 8\times 8$ $\Gamma$-centered k-grid is used to calculate
the Cu $3d$ and O $2p$ orbital Wannier functions used in the electron-phonon
calculations. Denser (uniform) $20\times 20\times 20$ k\- and q-point grids
are used for electron-phonon matrix element calculations. Effective Coulomb
repulsion values of $\mu^{*}=0,0.1$, $0.15$, and $0.2$ are used to estimate
the superconducting critical temperature by self-consistently solving the
isotropic Migdal-Eliashberg equations. The Matsubara frequency cutoff is set
to 1.5 eV and the Dirac deltas are replaced by Lorentzians of width 50 meV for
electrons and 0.1 meV for phonons.

The apatite crystal structure of CuxPb10-x(PO4)6O with 0.9 $<$ x $<$ 1.1 is
obtained by replacing a specific Pb sublattice with a Cu atom from the parent
Pb10(PO4)6O compound. The parent structure [Fig. 1(a)], which exhibits
semiconducting behavior Cabezas_arxiv23 ; Si-Held_arxiv23 ; Liu_arxiv23phonon
, contains six Pb($6h$) and four Pb($4f$) sites as well as six PO4 tetrahedra
each containing an O(1), O(2), and O(3) with $6h$, $12i$, and $6h$ sites,
respectively. The Pb($6h$) atoms form two triangular layers separated by 3.71
Å with a 60o rotation, whereas the Pb($4f$) atoms adopt a hexagonal
arrangement [Fig. 1(b)]. In addition, four O(4) atoms occupy $4e$ sites
constituting a quarter of the occupancy [Fig. 1(a)]. To model this partial
occupancy we considered the structure with the lowest energy, simultaneously
maintaining full occupation of one out of the four O(4) sites. Structural
optimization slightly shifts the position of this O(4) atom, breaking its
original symmetry, with imperfect long-range order in the crystal structure.
This extra O atom lies outside the PO4 tetrahedra, inducing a potential local
distortion.

Figure 2: Electronic properties of CuPb9(PO4)6O: band structure in wide (a)
and narrow (b) energy windows, density of states (c), and Fermi surface
(inset).

To model the CuPb9(PO4)6O structure we replaced one of the Pb atoms by Cu.
However, the exact site within the Pb sublattice where the Cu atom is
substituted remains uncertain. Therefore we examined Cu substitutions in one
of the (identical) Pb($6h$) and one of the (identical) Pb($4f$) sites
individually to identify the position of lowest energy. These calculations
confirm that a Cu at the Pb($6h$) site is energetically favorable forming a
bond of length 1.93 Å with Cu-O(4). Instead of transforming the system to
metallic, the structure remains semiconducting with a band gap of $\sim$1 eV,
consistent with previous results Zhang_arxiv23 ; Liu_arxiv23phonon . The
structure with Cu substited at the Pb($4f$) site is $\sim$0.3 eV/f.u (per
formula unit) higher in energy with respect to the one substituted at the
Pb($6h$) site, in agreement with previous work Sun-Vlamidir_arxiv23 , and
shows a metallic electronic band structure (discussed later). As Migdal-
Eliashberg theory begins from a metallic state, we assumed the Pb($4f$)
location was preferred over the Pb($6h$) site, as shown by the green cross in
Fig. 1(a). The optimized lattice parameters of the Cu-substituted system ($a$
= 9.957 Å, $c$ = 7.430 Å, and $V$ = 637.94 Å3) decrease relative to the parent
structure ($a$ = 10.032 Å, $c$ = 7.492 Å, and $V$ = 652.99 Å3). This shrinkage
of lattice parameters and reduction in volume agrees with experimental and
theoretical reports Lee_arxiv23First ; Si-Held_arxiv23 .

Figure 3: Vibrational and electron-phonon properties of CuPb9(PO4)6O. Phonon
dispersions are plotted in wide (a) and narrow (b) energy windows. (c) Phonon
density of states (PHDOS) along with the atomic projection of the PHDOS. (d)
PHDOS, Eliashberg spectral function $\alpha^{2}F(\omega)$, and total electron-
phonon coupling strength $\lambda$. (e) Momentum-resolved anisotropic
electron-phonon coupling $\lambda_{n\textbf{k}}$ on the Fermi surface plotted
in a primitive Brillouin zone. (f) Distribution of the electron-phonon
coupling strength $\lambda_{\textbf{k}}$. (g) Variation of the DOS and
$\lambda$ with respect to energy relative to $E_{\rm F}$.

Figure 2(a) and 2(b) show the electronic band structure of CuPb9(PO4)6O. The
Cu $3d$ and O $2p$ orbital contributions are indicated by the blue and red
colors. Fig. 2(b) shows the metallic band structure within a small energy
window close to the $E_{\rm F}$, in which a flat band, predominantly of Cu
$3d$ and O $2p$ character, crosses $E_{\rm F}$. This band has a camel shape
dispersion with the two humps at the K and H points of the Brillouin zone
along the M-K-$\Gamma$ and L-H-A directions. These crossings produce a waisted
cylindrical shaped Fermi surface with a K (or K′)-centered hole pocket [Fig.
2(a) inset]. The DOS and orbital projections are shown in Fig. 2(c). The Cu
$3d$ and O $2p$ orbitals contribute equally at $E_{\rm F}$. As the electronic
band structure and DOS remain relatively unchanged across a broad energy
range, spin-orbit coupling is not considered. We note that the electronic
structure of CuPb9(PO4)6O is not consistent among the recent theoretical
studies Si-Held_arxiv23 ; Si-Simone_arxiv23 ; Kumar_arxiv23 , especially
regarding the relative position of $E_{\rm F}$. The primary reason for this
inconsistency appears to be sensitivity to the convergence parameters (such as
k-mesh, energy cut-off, and the value of smearing) and the pseudopotential
types used. We have used norm-conserving pseudopotentials and found that the
relative position of $E_{\rm F}$ lies $\sim$30 meV above the one calculated
with projector augmented wave pseudopotentials, especially in the $\Gamma$-A
Brillouin zone direction. Therefore, we must examine the sensitivity of our
results (discussed in the electron-phonon coupling) to a small shift on either
side of $E_{\rm F}$.

Figure 3(a), 3(b), and 3(c) show the calculated phonon dispersions of
CuPb9(PO4)6O along with the atom-projected phonon DOS (PHDOS). The optical
phonons exhibit a distinct flat dispersion relation across the entire
vibrational spectrum, whereas the acoustic branches display conventional
linear behavior. As a result of the flat phonon dispersion, sharp peaks in the
PHDOS are observed. The complete spectrum is divided into four parts: low
frequency vibrations below 35 meV, intermediate frequency vibrations 40-55 meV
and 60-70 meV, and high frequency vibrations above 105 meV. The lower
frequency modes predominantly correspond to vibrations of the Pb atoms,
whereas the modes within the intermediate range (60-70 meV) and the higher
frequency modes are associated with vibrations of the P atoms. The vibrations
of the O atoms extend across the entire spectrum; they are an integral part of
the PO4 tetrahedra, giving rise to collective oscillations involving mixed
vibrational modes with other atoms. This indicates a complex interplay of
chemical bonding and structural flexibility of the tetrahedral structure in
CuPb9(PO4)6O. Cu substitution primarily impacts the low-frequency modes below
15 meV [Fig. 3(c)], as Cu is considerably heavier than the O and P atoms. Cu
vibrations, mixing with the lighter atom modes, induce substantial
anharmonicity and mode coupling, highlighting the pivotal role of lattice
distortions on the electronic and vibrational properties.

To further assess whether CuPb9(PO4)6O would superconduct via the strong-
coupling electron-phonon mechanism, we examine the isotropic Eliashberg
spectral function $\alpha^{2}F(\omega)$ and the cumulative electron-phonon
coupling strength $\lambda$ [Fig. 3(d)]. A sharp increase in the
$\alpha^{2}F(\omega)$ distribution is found up to 20 meV, indicating strong
electron-phonon coupling in the low frequency region. This is unsurprising
because of the large value of the DOS at $E_{\rm F}$. Further, near the
intermediate frequency 40 meV a sharp peak is noticed in
$\alpha^{2}F(\omega)$. A continuous isotropic electron-phonon coupling
strength $\lambda_{n\textbf{k}}$, ranging from 0.3 to 0.5, develops on the
Fermi surface [Fig. 3(e) and Fig. 3(f)] with a peak at the K-point of the
Brillouin zone. Consequently, a weak ($\sim$0.4) average electron-phonon
coupling strength is found, insufficient to support conventional electron-
phonon strong-coupling high-temperature superconductivity. It is concerning
that the obtained value of $\lambda$ lies within the upper range of electron-
electron repulsion strengths that have been reported for most conventional
superconductors Kostrzewa_SciRep18_Coulomb ; thus the electron-phonon pairing
is unlikely to be robust in the presence of repulsion.

Even neglecting the Coulomb repulsion of the electron pair we find that the
value we obtain for the electron-phonon coupling suggests a superconducting
transition temperature less than 2 K. We estimate the superconducting
transition temperature by solving temperature dependent coupled nonlinear
Eliashberg equations to find the superconducting gap, incorporating the
effective Coulomb repulsion $\mu^{*}$, in the isotropic approximation Allen-
Mitro1983 ; Margine2013s . The temperature at which the gap vanishes is the
superconducting transition temperature. Calculated superconducting transition
temperatures of 250 mK, 26 mK, and $\sim$0 mK correspond to more realistic
effective Coulomb repulsion values of $\mu^{*}=0.1$, $0.15$, and $0.2$. Hence
we do not see any signature of high-temperature superconductivity in
CuPb9(PO4)6O that can be described by conventional strong-coupling Migdal-
Eliashberg theory.

To assess the accuracy of these electron-phonon calculations for CuPb9(PO4)6O
we examined their sensitivity to the Fermi energy. The variation of the DOS
and the cumultative electron-phonon coupling strength are shown in Fig. 3(g)
for a rigid shift in the Fermi energy on either side of $E_{\rm F}$ between
-50 meV and 20 meV in steps of 10 meV. As expected the value of electron-
phonon coupling strength increases as the DOS increases (and conversely), and
reaches its largest value, 0.5, for a -50 meV shift in $E_{\rm F}$. The
overall shape of $\alpha^{2}F(\omega)$ for a shifted Fermi energy is similar
to the one calculated at the actual $E_{\rm F}$, but with stronger coupling
observed in the low-frequency modes. However, the value of the electron-phonon
coupling strength remains insufficient to overcome electron-electron repulsion
and produce high-temperature superconductivity.

We employed ab initio Migdal-Eliashberg theory to identify whether or not the
electron-phonon coupling strength found in CuPb9(PO4)6O is sufficient to
describe its high-temperature superconducting behavior via a strong-coupling
electron-phonon mechanism. Electronic structure calculations indicated flat
electronic bands, described by strongly hybridized Cu $3d$ and O $2p$ states,
close to the Fermi level along with relatively low non-dispersive phonon
frequencies, predominantly associated with vibrations of Pb. These features
could suggest strong-coupling high-temperature superconductivity via the
electron-phonon mechanism. Unfortunately we calculate a low cumulative
electron-phonon coupling strength. Our obtained value may not be enough even
to overcome the Coulomb repulsion between two electrons forming a Cooper pair
and produce superconductivity at any temperature. However, even if the system
can become superconducting, our calculated value of the electron-phonon
coupling strength does not support high-temperature superconductivity in
CuPb9(PO4)6O via a strong-coupling electron-phonon mechanism within Migdal-
Eliashberg theory.

This work is supported by the U.S. Department of Energy, Office of Science,
Office of Basic Energy Sciences under Award Number DE-SC0016379. The Ames
National Laboratory is operated for the U.S. Department of Energy by Iowa
State University of Science and Technology under Contract No. DE-
AC02-07CH11358. We acknowledge use of the computational facilities at the
University of Iowa (Argon cluster).

## References

  * (1) J. A. Flores-Livas, L. Boeri, A. Sanna, G. Profeta, R. Arita, and M. Eremets, Phys. Rep. 856, 1 (2020).
  * (2) B. Lilia, R. Hennig, P. Hirschfeld, G. Profeta, A. Sanna, E. Zurek, W. E.Pickett, M. Amsler, R. Dias, M. I. Eremets, C. Heil, R. J. Hemley, L. Hanyu, Y. Ma, C. Pierleoni, A. N. Kolmogorov, N. Rybin, D. Novoselov, V. Anisimov, A. R. Oganov, C. J. Pickard, T. Bi, R. Arita, I. Errea, C. Pellegrini, R. Requist, E. K. U. Gross, E. R. Margine, S. R. Xie, Y. Quan, A. Hire, L. Fanfarillo, G. R. Stewart, J. J. Hamlin, V. Stanev, R. S. Gonnelli, E. Piatti, D. Romanin, D. Daghero, and R. Valenti, J. Condens. Matter Phys. 34, 183002 (2022).
  * (3) J. Nagamatsu, N. Nakagawa, T. Muranaka, Y. Zenitani, and J. Akimitsu, Nature 410, 63 (2001).
  * (4) V. Braccini, D. Nardelli, R. Penco, and G. Grasso, Phys. C: Supercond 456, 209 (2007).
  * (5) R. Penco and G. Grasso, IEEE Trans. Appl. Supercond. 17, 2291 (2007).
  * (6) A. Drozdov, M. Eremets, I. Troyan, V. Ksenofontov, and S. I. Shylin, Nature 525, 73 (2015).
  * (7) M. Somayazulu, M. Ahart, A. K. Mishra, Z. M. Geballe, M. Baldini, Y. Meng, V. V. Struzhkin, and R. J. Hemley, Phys. Rev. Lett. 122, 027001 (2019).
  * (8) P. Kong, V. S. Minkov, M. A. Kuzovnikov, A. P. Drozdov, S. P. Besedin, S. Mozaffari, L. Balicas, F. F. Balakirev, V. B. Prakapenka, S. Chariton, D. A. Knyazev, E. Greenberg, and M. I. Eremets, Nat. Commun. 12, 5075 (2021).
  * (9) C. Yao and Y. Ma, iScience 24 (2021).
  * (10) S. Lee, J.-H. Kim, and Y.-W. Kwon, arXiv preprint arXiv:2307.12008 (2023).
  * (11) S. Lee, J. Kim, H.-T. Kim, S. Im, S. An, and K. H. Auh, arXiv preprint arXiv:2307.12037 (2023).
  * (12) L. Liu, Z. Meng, X. Wang, H. Chen, Z. Duan, X. Zhou, H. Yan, P. Qin, and Z. Liu, arXiv preprint arXiv:2307.16802 (2023).
  * (13) K. Kumar, N. Karn, Y. Kumar, and V. Awana, arXiv preprint arXiv:2308.03544 (2023).
  * (14) K. Kumar, N. K. Karn, and V. S. Awana, Supercond. Sci. Technol. (2023).
  * (15) L. Si and K. Held, arXiv preprint arXiv:2308.00676 (2023).
  * (16) L. Si, M. Wallerberger, A. Smolyanyuk, S. di Cataldo, J. M. Tomczak, and K. Held, arXiv preprint arXiv:2308.04427 (2023).
  * (17) Y. Sun, K.-M. Ho, and V. Antropov, arXiv preprint arXiv:2308.03454 (2023).
  * (18) J. Lai, J. Li, P. Liu, Y. Sun, and X.-Q. Chen, J. Mater. Sci. Technol. 171, 66 (2024).
  * (19) J. Zhang, H. Li, M. Yan, M. Gao, F. Ma, X.-W. Yan, and Z. Xie, arXiv preprint arXiv:2308.04344 (2023).
  * (20) S. M. Griffin, arXiv preprint arXiv:2307.16892 (2023).
  * (21) J. Cabezas-Escares, N. Barrera, C. Cardenas, and F. Munoz, arXiv preprint arXiv:2308.01135 (2023).
  * (22) P. Giannozzi, O. Andreussi, T. Brumme, O. Bunau, M. B. Nardelli, M. Calandra, R. Car, C. Cavazzoni, D. Ceresoli, M. Cococcioni, N. Colonna, I. Carnimeo, A. D. Corso, S. de Gironcoli, P. Delugas, R. DiStasio, A. Ferretti, A. Floris, G. Fratesi, G. Fugallo, R. Gebauer, U. Gerstmann, F. Giustino, T. Gorni, J. Jia, M. Kawamura, H.-Y. Ko, A. Kokalj, E. Küçükbenli, M. Lazzeri, M. Marsili, N. Marzari, F. Mauri, N. L. Nguyen, H.V. Nguyen, A. O. de-la Roza, L. Paulatto, S. Poncé, D. Rocca, R. Sabatini, B. Santra, M. Schlipf, A. P. Seitsonen, A. Smogunov, I. Timrov, T. Thonhauser, P. Umari, N. Vast, X. Wu, and S. Baroni, J. Phys.: Condens. Matter 29, 465901 (2017).
  * (23) M. J. van Setten, M. Giantomassi, E. Bousquet, M. J. Verstraete, D. R. Hamann, X. Gonze, and G.-M. Rignanese, Comput. Phys. Commun. 226, 39 (2018).
  * (24) J. P. Perdew, K. Burke, and M. Ernzerhof, Phys. Rev. Lett. 77, 3865 (1996).
  * (25) H. J. Monkhorst and J. D. Pack, Phys. Rev. B 13, 5188 (1976).
  * (26) M. Methfessel and A. T. Paxton, Phys. Rev. B 40, 3616 (1989).
  * (27) S. Baroni, S. de Gironcoli, A. D. Corso, and P. Giannozzi, Rev. Mod. Phys. 73, 515 (2001).
  * (28) P. B. Allen and R. C. Dynes, Phys. Rev. B 12, 905 (1975).
  * (29) E. R. Margine and F. Giustino, Phys. Rev. B 87, 024505 (2013).
  * (30) S. Poncé, E. Margine, C. Verdi, and F. Giustino, Comput. Phys. Commun. 209, 116 (2016).
  * (31) H. Lee, S. Poncé, K. Bushick, S. Hajinazar, J. Lafuente-Bartolome, J. Leveillee, C. Lian, J.-M. Lihm, F. Macheda, H. Mori, H. Paudyal, W. H. Sio, S. Tiwari, M. Zacharias, X. Zhang, N. Bonini, E. Kioupakis, E. R. Margine, and F. Giustino, npj Comput. Mater. 9, 156 (2023).
  * (32) G. Pizzi, V. Vitale, R. Arita, S. Blügel, F. Freimuth, G. Géranton, M. Gibertini, D. Gresch, C. Johnson, T. Koretsune, J. Ibañez-Azpiroz, H. Lee, J.-M. Lihm, D. Marchand, A. Marrazzo, Y. Mokrousov, J. I. Mustafa, Y. Nohara, Y. Nomura, L. Paulatto, S. Poncé, T. Ponweiser, J. Qiao, F. Thöle, S. S. Tsirkin, M. Wierzbowska, N. Marzari, D. Vanderbilt, I. Souza, A. A. Mostofi, and J. R. Yates, J. Phys.: Condens. Matter. 32, 165902 (2019).
  * (33) R. Liu, T. Guo, J. Lu, J. Ren, and T. Ma, arXiv preprint arXiv:2308.08454 (2023).
  * (34) M. Kostrzewa, R. Szcześniak, J. K. Kalaga, and I. A. Wrona, Sci. Rep. 8, 11957 (2018).
  * (35) P. B. Allen and B. Mitrović, Solid State Phys. 37, 1 (1983).


# Towards a Generalist and Blind RGB-X Tracker

Yuedong Tan1 Zongwei Wu1 Yuqian Fu2 Zhuyun Zhou1 Guolei Sun2
Chao Ma3 Danda Dani Paudel4 Luc Van Gool2,4 Radu Timofte1
1University of Würzburg 2CVL, ETH Zürich
3AI Lab, Shanghai Jiao Tong University 4 INSAIT, Sofia University

###### Abstract

With the emergence of a single large model capable of successfully solving a
multitude of tasks in NLP, there has been growing research interest in
achieving similar goals in computer vision. On the one hand, most of these
generic models, referred to as generalist vision models, aim at producing
unified outputs serving different tasks. On the other hand, some existing
models aim to combine different input types (aka data modalities), which are
then processed by a single large model. Yet, this step of combination remains
specialized, which falls short of serving the initial ambition. In this paper,
we showcase that such specialization (during unification) is unnecessary, in
the context of RGB-X video object tracking. Our single model tracker, termed
XTrack, can remain blind to any modality X during inference time. Our tracker
employs a mixture of modal experts comprising those dedicated to shared
commonality and others capable of flexibly performing reasoning conditioned on
input modality. Such a design ensures the unification of input modalities
towards a common latent space, without weakening the modality-specific
information representation. With this idea, our training process is extremely
simple, integrating multi-label classification loss with a routing function,
thereby effectively aligning and unifying all modalities together, even from
only paired data. Thus, during inference, we can adopt any modality without
relying on the inductive bias of the modal prior and achieve generalist
performance. Without any bells and whistles, our generalist and blind tracker
can achieve competitive performance compared to well-established modal-
specific models on 5 benchmarks across 3 auxiliary modalities, covering
commonly used depth, thermal, and event data. The code and models will be
available at https://github.com/supertyd/XTrack.

##  1 Introduction

Artificial General Intelligence (AGI), which aims at achieving a generalist
model with the ability to understand, learn, and apply knowledge across a wide
range of tasks, has recently gained significant research attention [6, 44, 33,
46]. In the realm of Natural Language Processing (NLP), a bunch of flagship
generalist models [12, 42, 55, 41] capable of handling various tasks have been
proposed. However, AGI’s progress in computer vision remains comparatively
less advanced [46, 64, 5]. Recent advancements in multi-modal vision models,
which integrate and process data from multiple sensor modalities, have shown
promising potential in improving the generalization ability of vision models
[36, 15]. Particularly, the RGB-X video object tracking [66, 53, 47] which
naturally involves multi-modal inputs and requires the model to predict a
unified output is a very typical exemplar for building such multi-modal vision
models. Thus, in this paper, we aim to build a generalist model for the RGB-X
video object tracking task further pushing the research boundary.

RGB-X video object tracking involves the continuous tracking of objects in
video sequences using multi-modal data, based on the input RGB images combined
with additional modalities (X) such as depth [51], event [45], and thermal
data [32]. RGB, which carries the majority of the visual information, is the
most basic and important modality, while other modalities serve as
supplementary aids. For example, depth information can help address occlusion
issues [18, 52], thermal images can provide valuable cues in low-light
conditions [60, 63], and event data can enhance motion sensitivity [58, 69].
This combination of these various modalities enhances the robustness and
accuracy of object tracking in various challenging scenarios. The inherent
complexity of handling multiple modalities and the practical importance of
this task make it an ideal candidate for testing and advancing AGI
capabilities in multi-modal visions.

Figure 1: Motivation: There have been some more-or-less foundation trackers
for RGB VOT. However, current works on RGB-X trackers are still far away from
the ultimate goal, since they leverage modality-specific architecture and/or
parameters. Differently, we aim for a generalist RGB-X tracker, with one
single parameter set and without the need to know the modal prior.

Several efforts have been made to promote RGB-X video object tracking,
including works such as [53, 66, 47, 67, 51, 57, 14]. Among them, early works
often adopt task-specific designs, where the learning modules are tailored for
one particular modaity [51, 14], as shown in Fig.1(a). Recently, some works
have achieved architecture unification [53, 66], while still requiring
multiple sets of parameters to handle different modalities and need prior
knowledge of the input modality as in Fig.1(b). From our perspective, such a
reliance on prior modality knowledge fails to align with the principal
expectation of AGI. We believe a good generalist model should be able to
autonomously recognize and process different input modalities without any
prior indication. Such an AGI tracker has more-or-less achieved in RGB VOT
domain. As shown in Fig.1(c), current RGB trackers only require one single
parameter but can deal with any input from any dataset and any setting. This
motivates us to propose our single and unified model tracker, termed XTrack.
As illustrated in Fig.1(d), our XTrack uses a unified architecture with one
single parameter set for different modalities. Most importantly, our XTrack
could remain blind to any modality X during the inference stage, which we
believe can be the right path towards AGI for RGB-X tracking.

Essentially, to achieve the generalist and blind tracker, our key ideas behind
XTrack lie in: 1) reformulating all the input modalities as a single-blind
input X; 2) disentangling the modality-shared features and modality-specific
features. In other words, regardless of whether its actual modality is depth,
thermal, or event, we treat it equally as X and input X into our unified
model. To fully explore the X, two different types of features are extracted.
One is the modality-shared features, which capture common information across
various modalities; The other is the modality-specific features, which
represent unique cues specific to each modality, such as the geometry from
depth, temporal information from thermal, and motion from the event data.
Technically, a novel mixture of model experts (MeME) together with a dynamic
routing system is proposed to achieve these two objectives, resulting in our
XTrack method.

Concretely, our MeME contains a shared expert and several domain-specific
experts. The shared expert is responsible for extracting modality-shared
features, while the domain-specific experts obtain modality-specific features.
Additionally, a router is included in MeME to determine which expert should be
used for the current modality X, automatically recognizing the modality type
for X. Based on the router’s prediction, we can forward X to its corresponding
expert to obtain its modality-specific features. The final feature is a
combination of the modality-shared features and the modality-specific
features. In this way, we achieve cross-modal unification through joint
embedding, while preserving modality-specific clues. Notably, the training of
our model is extremely simple: we directly employ a multi-label classification
loss to supervise the selection of the most appropriate experts. Finally, the
learned generalist knowledge is used to prompt the RGB foundation tracker.

Extensive studies are performed to show the effectiveness of our XTrack, which
outperforms established modal-specific models by leveraging a unified set of
parameters, viewing the multi-modal system holistically. Our generalist XTrack
maintains efficiency with less than 10% modality-aware tuning parameters over
our RGB baseline while achieving a tolerable performance degradation of less
than 1% compared to the absolute state-of-the-art, i.e., modality-specific
trackers.

##  2 Related Work

There are extensive surveys on the generalist vision models [16, 2, 56, 37].
In this section, we focus exclusively on related RGB-X tracking works, aiming
to narrow the gap towards AGI.

Multi-Modal Tracking: In the evolution of object tracking [62, 10, 8, 65], the
model architecture has undergone a continuous transformation, transitioning
from correlation filters [24, 19] to Siamese networks [8, 17], and then to the
current transformer architecture [7, 35] that integrates unified feature
extraction and interaction. However, the performance and stability of object
tracking are still constrained by the insufficiency of RGB information. Multi-
modal information, including depth [18, 52], event [58, 69], and thermal data
[60, 63], can compensate for this deficiency and enhance the robustness of
object tracking networks. BAT [3] and TBSI [22] have designed a dual-
Transformer architecture specifically for RGBT object tracking, while
DepthTrack [51] is tailored for RGBD object tracking. Existing multi-modal
object tracking algorithms predominantly follow the design paradigm of prompt
learning, aiming to establish a unified object tracking paradigm [66, 21, 20,
47, 53] that integrates the model architecture for multi-modal object
tracking. However, learning modality-specific parameters is not the right path
toward AGI. This drives us to consider downstream modalities as auxiliary
means to holistically improve tracking performance.

Multi-Task Learning: Multi-task learning [4] enables different tasks to be
processed through a single network. Current research primarily focuses on
integrating multiple tasks, with each task having its dedicated tokenizer and
task-specific parameters. The Meta-Transformer [61] aligns various modalities
into a unified feature space, while ImageBind [15] uses images to align data
from different modalities. PolyViT [34] has pioneered the unified training of
a Vision Transformer (ViT) backbone using multiple modal data. Although these
works have achieved partial unification, they still require different
parameters for input and output due to the distinct nature of the tasks being
processed. In multi-modal object tracking, the different input modalities is
all image-based, and the tasks being addressed are consistent. Therefore, in
this paper, we explore a novel approach that models images from different
modalities, allowing for the maintenance of uniform parameters across
different modal inputs without the need for any task-specific parameters,
thereby accomplishing a single task.

Mixture-of-Experts: Mixture of Experts (MoE) has emerged as a powerful
technique, enhancing model capability and efficiency by delegating tasks to
multiple expert networks. This model architecture is particularly suited for
handling complex and diverse data, as it dynamically selects the most
appropriate expert based on the characteristics of the input data. Models such
as VMoE [39] and DAMeX [23] have significantly improved image classification
and object detection tasks by extending the feed-forward network (FFN) layers
within the vision transformer architecture. These models dynamically adjust
the combination of their internal expert networks based on the features of the
input images, thereby capturing complex patterns more effectively. DeepSeekMoE
[9] exemplifies the application of MoE in large language models, significantly
enhancing language understanding and generation through the expert mixture
mechanism. This model selects the most suitable expert network based on the
semantics and context of the input text, thus improving the accuracy and
efficiency of language processing. Models like IMP [1] and LiMoE [38] leverage
MoE to unify tasks across different modalities, such as visual, auditory, and
textual data processing. These models share and transfer knowledge across
different modalities, thereby enhancing the performance of multi-modal tasks.
However, existing MoE algorithms still face challenges in achieving effective
division of labor among experts. In this paper, we propose a novel approach
that unifies not only the feature extraction part across different modalities
but also the input part. This means that regardless of the modality of the
input data, the model can effectively process it through the MoE expert
mechanism. Additionally, we introduce a constraint mechanism to ensure a clear
division of labor among experts.

Figure 2: Architecture: Our XTrack leverages a Mixture of Modal Experts (MeME,
see Sec.3.1 & Fig. 3) to bind all the modalities into a shared latent space,
while without losing the modal-specific clues. Such a design naturally creates
emergent alignment and enables joint benefit across modalities, while only
paired RGB-X data are available during training.

##  3 Method

The core idea of our framework is to reformulate all input sensory modalities,
such as depth, thermal, and event data, as a single blind input X, while
allowing the network to dynamically learn the modal nature, conditioned on
which the most appropriate reasoning is accordingly assigned to improve the
tracker’s robustness. This dynamic distribution is achieved through a mixture
of experts design combined with a classification loss. Similar to the current
mainstream target tracking paradigm, we input both the template and the search
region, along with their associated multi-modal patches, into the network
simultaneously to locate the target object.

###  3.1 Mixture of Modal Experts (MeME)

Basic Architecture: To align all the modalities, we employ a Mixture of Modal
Experts (MeME). MeME operates on two key principles: (a) assigning the most
appropriate and specialized experts based on the input modality, and (b)
isolating the shared experts that are common across all modal inputs to
facilitate cross-modal alignment and reasoning. By jointly incorporating both
specialized and shared experts, MeME allows for the exploration of modal-
specific clues with finer granularity thanks to the reduced redundancy within
each specialized expert, and ensures emergent alignment across modalities,
even when trained solely on paired data. An overview can be found in Fig. 2.

####  3.1.1 Routing among Specialized Experts with Multi-Label Classification
Loss

In designing routing experts, we segment them into finer granularity to
achieve higher specialization. While existing MoE networks primarily balance
the input information for each expert, they often lack clearly defined roles
for individual experts. This is particularly important in our modal setting,
where the distinct modal features necessitate experts from different domains
for appropriate reasoning.

To achieve this, our MeME starts by taking a pair of downstream modal features
as input, compromising template modal input $T_{x}\in\mathbb{R}^{3\times
H_{t}\times W_{t}}$ and search region in the modal input
$S_{x}\in\mathbb{R}^{3\times H_{s}\times W_{s}}$. These patches are initially
segmented and subsequently flattened into sequences of patch vectors,
specifically $T_{p}\in\mathbb{R}^{N_{t}\times(3\cdot P^{2})}$ and
$S_{p}\in\mathbb{R}^{N_{s}\times(3\cdot P^{2})}$, where $P\times P$ defines
the spatial extent of each individual patch. The quantities $N_{t}$ and
$N_{s}$ , calculated as $N_{t}=\frac{H_{t}W_{t}}{P^{2}}$ and
$N_{s}=\frac{H_{s}W_{s}}{P^{2}}$, respectively, signify the count of patches
pertaining to the template and search regions from the modal input. Following
this transformation, a learnable linear projection layer $proj$ is utilized to
map the patch sequences $T_{p}$ and $S_{p}$ into a $D$-dimensional latent
space. These sequences are then concatenated, to meet the tracking
specificity, and form $O_{x}$ for subsequent processing and analysis:

$O_{x}=concat(proj(T_{p}),proj(S_{p})).$ (1)

Therefore, the routed output is as follows:

$y=\sum_{i\in\text{t}op\\_k}p_{i}(O_{x})\epsilon_{i}(O_{x}),$ (2)

where we adopt $top\\_k$ experts for routing the images, $p_{i}$ is the
probability, and $\epsilon_{i}$ is the expert.

Classification Loss: For the modality-aware specialization, during training,
we employ a multi-class loss to constrain the data that each expert can
process. Given a collection $M$ of modalities, indexed by $n$, denoted as
${m_{n}}_{n=1}^{|M|}$. We introduce a mapping function
$h:M\rightarrow\epsilon$ such that each modality $m_{n}$ is associated with
several particular experts $e_{i}\in\epsilon$, which ideally are not shared
from one modality to another. The auxiliary loss $\mathcal{L}_{moe}$ is
computed using a binary cross-entropy loss, comparing the logits $p_{i}$
(representing the probability of selecting experts) with the labels $h(m_{n})$
(indicating the target expert for tokens from modality $m_{n}$). The
expression for $\mathcal{L}_{moe}$ is formulated as follows:

$\mathcal{L}_{moe}=-\sum_{n=1}^{|M|}\sum_{i=1}^{|\epsilon|}1{\left\\{h\left(m_{n}\right)=i\right\\}}\log
p_{i}(O_{x}),$ (3)

where $p_{i}(O_{x})$ is the probablity, $1_{{h(m_{n})=i}}$ is the indicator
function that equals to 1 if $h(m_{n})=i$ and 0 otherwise, and $|\epsilon|$ is
the cardinality of the set of experts $\epsilon$.

Figure 3: Mixture of Modal Experts (MeME): MoE has demonstrated significant
advancements recently. Recent work [9] has further leveraged shared experts to
reduce redundancy. However, it remains unclear which expert learns what, due
to the implicit learning setting. In contrast, we make the learning protocol
explicit by assigning specific experts to particular modal inputs.
Additionally, we enhance the shared expert model with inductive edge bias,
increasing efficiency when dealing with relatively limited downstream data.

Expert Balance Loss: To ensure a better distribution of modalities, we also
use a balance loss compromising two parts: the importance loss
$\mathcal{L}_{ExpImp}$ and the loading loss $\mathcal{L}_{ExpLoad}$. We
consider a norm distribution denoted as $\mathcal{N}(0,\sigma^{2}\textbf{I})$,
where the standard deviation $\sigma$ is defined as the ratio of gate noise to
the number of experts $|\epsilon|$. The cumulative distribution function (CDF)
of this normal distribution is denoted by $\Phi(\cdot)$.

For the load $Load_{i}$ of expert $e_{i}$, we evaluate the CDF at the
probabilities $p_{i}(O_{x})$, represented as $\Phi(p_{i}(O_{x}))$, and sum
these values over all inputs $O_{x}$ in a single mini-batch. The load loss
$\mathcal{L}_{ExpLoad}$ is then defined as:

$\mathcal{L}_{ExpLoad}=\frac{\operatorname{Var}(Load)}{\operatorname{Mean}(Load)^{2}};\
\;Load_{i}=\sum_{O_{x}\in batch}\Phi(p_{i}(O_{x})).$ (4)

Here, $Load_{i}$ represents the number of assignments allocated to expert
$e_{i}$. Similarly, we obtain the importance loss $\mathcal{L}_{ExpImp}$
without considering the CDF by:

$\mathcal{L}_{ExpImp}=\frac{\operatorname{Var}(Imp)}{\operatorname{Mean}(Imp)^{2}};\
\;Imp_{i}=\sum_{O_{x}\in batch}p_{i}(O_{x}).$ (5)

Hence, our balance loss $\mathcal{L}_{balance}$ is caluculated by:

$\mathcal{L}_{balance}=\mathcal{L}_{ExpImp}+\mathcal{L}_{ExpLoad}.$ (6)

Finally, during training and on top of conventional tracking, our model
additionally leverage both classification loss and balance loss to achieve our
targeted generalist setting:

$\mathcal{L}_{generalist}=\mathcal{L}_{moe}+\lambda\cdot\mathcal{L}_{balance},$
(7)

where $\lambda$ is the proportion hyperparameter.

###  3.2 Efficient Expertized Reasoning and Gathering

Experts with Low-Rank Reasoning: The goal of each specialized expert is to
project the input feature into a dedicated space for further processing.
Inspired by recent successes in tuning large language models (LLMs), we aim to
enhance efficiency by mitigating low-rank feature decomposition and
reconstruction. Specifically, we project each input token with channel $c$ to
the significantly lower dimensional latent space $k$ ($k<<c$). This approach
preserves the principal features while reducing computational cost, making the
learning feasible on a single 24GB GPU.

Assume that our router decomposes the mixed modal input $M$ into subset
features $m_{n}$. Their respective $k_{th}$ low-rank matrices ${m_{n}}_{k}$,
are approximated by:

${m_{n}}_{k}=\epsilon_{n}(m_{n}),$ (8)

where $\epsilon_{n}$ mimicks the low-rank projection for feature
transformation. Note that all the $\epsilon_{n}$ structures are identical but
without weight sharing, allowing for distinct reasoning.

Figure 4:  Details on experts and prompts.

Edge Guided Shared Expert: Simultaneously, we compute the shared low-rank
matrix $m_{s}$ from the shared expert $\epsilon_{s}$. However, we believe that
for the shared expert, a single projection through implicit learning might not
be sufficient, especially when the training data size is relatively small
compared to that used for training large foundation models. Therefore, we
think it is necessary to incorporate additional human prior knowledge to guide
the shared learning process. Since the objective is to find commonalities, we
believe that high frequency is a naturally shared feature among these
modalities, typically manifesting in the form of edges.

In fact, event cameras naturally capture high-frequency temporal changes,
recording events whenever there is a change in the scene. This is particularly
useful for detecting rapid movements and transitions, leading to a denser
distribution of event spikes around object boundaries. Similarly, spatial
frequency in depth images refers to geometrical changes, often correlating
with sharp edges. In thermal images, frequency can relate to the spatial
variation of temperature. Areas with high spatial frequency changes in thermal
readings often indicate edges where materials with different thermal
properties meet. Therefore, we incorporate the additional edge clues as an
inductive bias for shared expert reasoning.

Technically, we incorporate the shared low-rank projection with an additional
gating module. Let ${m_{s}}_{k}$ be the shared low-rank feature, the meta-
architecture is formulated as:

$\begin{split}&X=Norm({m_{s}}_{k}),\\\ &Out=(\sigma(EdgeMixer(XW_{1}))\cdot
XW_{2})W_{3}+{m_{s}}_{k},\end{split}$ (9)

where $Norm()$ denotes the conventional normalization; $EdgeMixer()$ refers to
the module responsible for conducting token mixing. Technically, we begin by
transferring the token to a 2D feature form and then apply convolution to
enable interaction between neighboring tokens. Unlike previous approaches, in
our case, we additionally initialize the convolution with a Laplacian Filter,
incorporating edge priors while remaining open to discovering more meaningful
and learnable shared features. $W_{1},W_{2},W_{3}$ represent learnable
parameters within the low-rank latent space. A graphical illustration of this
process can be found in Fig. 4.

Fusion of Expertized Tokens: The next step aims for fusing previously
separately reasoned tokends together into a global low-rank matrix $M_{k}$. It
should contain both modality-specific and modality-shared clues. Technically,
we develop a shringkage fusion design, where we first concatenate
${m_{n}}_{k}$ together batch-wise. Then, the we learn a joint low-rank
approximation from the whole batch, which is further incorporating with the
shared matrix $m_{s}$. This pipeline can be expressed as follows:

$M_{k}=([{m_{n}}_{k}]W_{4}+{m_{s}}_{k})W_{5},$ (10)

where $[.]$ is the batch combination and $W_{4},W_{5}$ are low-rank learnable
parameters.

###  3.3 Modal Prompting

Finally, we aim to prompt the RGB token obtained from the frozen RGB
foundation tracker and transform them to be modality-aware, making them more
suitable for downstreaming and challenging cases. Following the same idea, we
first project the RGB token into the low-rank space $I_{k}$. Then, we use
appromiated modal low-rank matrix $M_{k}$ from Eq. 10 to prompt $I_{k}$. Our
motivation is use the modality clues as a gating module to improve the RGB
feature modeling. Specifically, we have:

$\begin{split}&X_{i}=Norm({I}_{k}),\,\;X_{m}=Norm({M}_{k}),\\\
&Out=((X_{i}W_{5}\cdot\sigma(X_{m}W_{6}))W_{7}+{I}_{k})W_{8},\end{split}$ (11)

where $TokenMixer(·)$ refers to learnable 2D convolution and
$W_{5},W_{6},W_{7}$ are the learning parameters in the low-rank space, where
$W_{8}$ projects the prompted token back to the initial embedding space.

Table 1: State-of-the-art comparison on DepthTrack[51], LasHeR[32], and
VisEvent[45]. The best three results are shown in red, blue, and green fonts.

Method | Type |  DepthTrack [51] |  |  LasHeR [32] |  |  VisEvent [45]
---|---|---|---|---|---|---
F-score($\uparrow$) |  Re($\uparrow$) |  Pr($\uparrow$) |  |  PR($\uparrow$) |  SR($\uparrow$) |  |  Precision($\uparrow$) |  Success($\uparrow$)
XTrack | Generalist | 59.7 | 59.7 | 59.8 |  | 65.5 | 52.5 |  | 75.6 | 59.1
UnTrack [47] | Unified but with prior knowledge | 61.0 | 61.0 | 61.0 |  | 64.6 | 51.3 |  | 75.5 | 58.9
ViPT [66] | Depth-Specific Param. | 59.4 | 59.6 | 59.2 |  | - | - |  | - | -
ViPT [66] | Thermal-Specific Param. | - | - | - |  | 65.1 | 52.5 |  | - | -
ViPT [66] | Event-Specific Param. | - | - | - |  | - | - |  | 75.8 | 59.2
ProTrack [53] | Depth-Specific Param. | 57.8 | 57.3 | 58.3 |  | - | - |  | - | -
ProTrack [53] | Thermal-Specific Param. | - | - | - |  | 53.8 | 42.0 |  | - | -
ProTrack [53] | Event-Specific Param. | - | - | - |  | - | - |  | 63.2 | 47.1
SPT [67] | Depth-Specific Model | 53.8 | 54.9 | 52.7 |  | - | - |  | - | -
DeT [51] | Depth-Specific Model | 53.2 | 50.6 | 56.0 |  | - | - |  | - | -
DDiMP [26] | Depth-Specific Model | 48.5 | 46.9 | 50.3 |  | - | - |  | - | -
ATCAIS [26] | Depth-Specific Model | 47.6 | 45.5 | 50.0 |  | - | - |  | - | -
MaCNet [57] | Thermal-Specific Model | - | - | - |  | 48.2 | 35.0 |  | - | -
DAFNet [14] | Thermal-Specific Model | - | - | - |  | 44.8 | 31.1 |  | - | -
mfDiMP [59] | Thermal-Specific Model | - | - | - |  | 44.7 | 34.3 |  | - | -
FANet [68] | Thermal-Specific Model | - | - | - |  | 44.1 | 30.9 |  | - | -
SGT [30] | Thermal-Specific Model | - | - | - |  | 32.7 | 23.2 |  | - | -
TransT_E [7] | Event-Specific Model | - | - | - |  | - | - |  | 65.0 | 47.4
PrDiMP_E [11] | Event-Specific Model | - | - | - |  | - | - |  | 64.4 | 45.3
VITAL_E [40] | Event-Specific Model | - | - | - |  | - | - |  | 64.9 | 41.5
Stark_E [50] | Event-Specific Model | - | - | - |  | - | - |  | 61.2 | 44.6
SiamCAR [17] | Event-Specific Model | - | - | - |  | - | - |  | 59.9 | 42.0

##  4 Experiments

###  4.1 Implementation Details

We adopt [54] as RGB foundation by freezing the parameters. To train our
XTrack, we only train our shallow embedding and deep prompts, as shown in
Fig.2. We set the batch size to 64, with a learning rate of 0.0004, training
for 60 epochs. The learning rate is decreased by a factor of 10 after 48
epochs. In addition to the object tracking loss shared with the RGB Tracker,
we incorporate a loss that guides MoE expert balance and specialization of
modality-specific experts, as detailed in Sec 3.1.1.

Regarding dataset selection, we utilize DepthTrack for RGB-D sequences, LasHeR
for RGB-T sequences, and VisEvent for RGB-E sequences. Previous well-
established modality-tailored works [66, 53, 20, 21], we train our model by
inputting data from all three datasets simultaneously.

We want to highlight that only one RGB-X pair (i.e. RGB-depth, or RGB-thermal,
or RGB-event) is available at a time, which makes our problem setting
challenging.

###  4.2 Within Distribution Evaluation

We first analyze the within distribution performances, while our models are
jointly trained on DepthTrack [51], LasHeR [32], and VisEvent [45], and tested
on the dedicated testing set from each modality. Notably, we compare our
XTrack with three groups of competitors: 1) specific models only for tackling
one dataset, e.g., SPT [67], DeT [51], DDiMP [26], ATCAIS [25], MaCNet [57],
DAFNet [14], and so on; 2) single model but has different sets of parameters
for different datasets: ViPT [66] and ProTrack [29]; 3) Unified model but
requires modality prior: UnTrack [47]. The comparison results are summarized
in Tab. 1. Note that our model uses a single parameter set and treats all
modalities equally as a blind X input, while all other methods require prior
knowledge of the modal nature and most of them adopt a parameter- and/or
model-specific design. The detailed analysis of each dataset is as follows.

DepthTrack: DepthTrack [51] is a comprehensive long-distance dataset
comprising 50 sets of test video sequences. We evaluate the accuracy of
various trackers using F-score, Recall (Re), and Precision (Pr) metrics. From
the results, we observe that: 1) our XTrack outperforms all the specific
models proposed for the DepthTrack dataset, as well as the unified models with
different sets of parameters. This demonstrates the effectiveness of our
method; 2) UnTrack performs slightly better than our method. However, given
that UnTrack uses more information, specifically the modality types, this
slight disadvantage is acceptable.

LasHeR The LasHeR [32] dataset is a large-scale, high-quality dataset
comprising 245 test sequences. We evaluate the performance of trackers using
precision (Pr) and success plots (SR). Notably, XTrack achieves SOTA on this
dataset clearly outperforming all the competitors even including the most
competitive UnTrack[47] and ViPT[66]. This enhancement well validates the
advantages of our method: take advantage of both the modality-shared and
modality-specific features.

VisEvent VisEvent[45] is a large-scale dataset containing RGB images and event
data for object tracking, comprising 320 test sequences. We use precision (PR)
and success plots (SR) to evaluate the performance of different trackers.
Results show that our XTrack outperforms the UnTrack[47] and obtains very
comparable to the ViPT[66] again indicating our effectiveness. In addition, we
highlight that our inference stage is quite efficient due to the MeME design,
only activating the selected experts.

Table 2:  Overall performance on RGBT234 dataset [31]. Our Generalist model
sets new SOTA records.

| Thermal-Specific Parameters | Generalist
---|---|---
|  | mfDiMP
---
[59]
| SGT
---
[30]
| DAFNet
---
[14]
| FANet
---
[68]
| MaCNet
---
[57]
| CMPP
---
[43]
| APFNet
---
[48]
| ProTrack
---
[53]
| ViPT
---
[66]
| XTrack
---
(ours)
MPR($\uparrow$) | 64.6 | 72.0 | 79.6 | 78.7 | 79.0 | 82.3 | 82.7 | 79.5 | 83.5 | 84.8
MSR($\uparrow$) | 42.8 | 47.2 | 54.4 | 55.3 | 55.4 | 57.5 | 57.9 | 59.9 | 61.7 | 62.2

Table 3:  Overall performance on VOT-RGBD2022 dataset [28].

| Depth-Specific Parameters | Generalist
---|---|---
|  | DRefine
---
[27]
| Stark_d
---
[50]
| DMTracker
---
[13]
| DeT
---
[51]
| SBT_D
---
[49]
| SPT
---
[67]
| ProTrack
---
[53]
| ViPT
---
[66]
| XTrack
---
(ours)
EAO($\uparrow$) | 59.2 | 64.7 | 65.8 | 65.7 | 70.8 | 65.1 | 65.1 | 72.1 | 71.4
Accuracy($\uparrow$) | 77.5 | 80.3 | 75.8 | 76.0 | 80.9 | 79.8 | 80.1 | 81.6 | 81.2
Robustness($\uparrow$) | 76.0 | 79.8 | 85.1 | 84.5 | 86.4 | 85.1 | 80.2 | 87.1 | 86.5

###  4.3 Out-of-Distribution Generalization

To further assess the generalization performance of our model, we also
conducted experiments on two out-of-distribution datasets including RGBT234
and RGBD2022.

RGBT234: The RGBT234[31] dataset is a comprehensive video dataset, including
234 sequences videos for test, encompassing a variety of environmental
challenges, including rainy conditions, nighttime scenes, and extreme weather
scenarios ranging from cold to hot. Results in Tab. 2 show that our Xtrack
gains clear improvements compared to other methods even in unseen sequences.

VOT-RGBD2022: The VOT-RGBD2022[28] dataset, comprising 127 video sequences, is
the latest RGB-D object tracking benchmark. We evaluate the performance of our
tracking model using accuracy, robustness, and expected average overlap (EAO).
Results are given in Tab. 3. Although our model’s performance is not as strong
as desired, primarily due to the limited RGB-D training data (150 sequences)
compared to RGB-E with 500 sequences and RGB-T datasets with 979 sequences.
However, as a generalist model, XTrack still demonstrates commendable overall
performance.

###  4.4 Ablation Study

We conduct ablation studies on the DepthTrack dataset.

Table 4: Ablation studies on the MoME by gradually removing the key components. The performance deteriorates as expected. Model |  Shared Experts |  Modality-Specific Experts | DepthTrack
---|---|---|---
F-score | Re | Pr
| ✓ | ✓ | 59.7 | 59.7 | 59.8
XTrack |  | ✓ | 59.0 | 59.2 | 58.8
| ✓ |  | 57.8 | 57.9 | 57.8

Table 5:  Ablation study of number of modality-specific experts. Nbr. | F-score | Re | Pr
---|---|---|---
1 | 56.6 | 56.6 | 56.6
2 | 59.7 | 59.7 | 59.8
3 | 57.5 | 57.9 | 57.1

##### Shared Experts

In the design of MeME, we use a shared expert to capture modality-shared
features across different modalities. To evaluate the impact of this shared
expert, we removed it from MeME, making the output of each MeME consist solely
of modality-specific experts. As shown in Tab. 5, we found that removing the
shared experts resulted in a 0.7% decrease in F-score. This indicates that the
shared experts successfully capture commonalities across modalities. In
addition, removing the shared expert may lead to redundancy in the modality-
specialized experts, reducing their effectiveness and hence leading to a
performance drop. Such an observation joins the conclusion from the previous
work [9]. It is worthy noting that our whole model only requires around 100M
parameters, with only +8M parameters compared to our RGB baseline. The 8M
parameters are used by all the experts, in total 8 experts in our final
settings. Hence, the shared experts only represent around 1M parameters, while
enabling significant performance improvement.

##### Modality Specific Experts

In addition to the shared expert, our MeME also includes modality-specific
experts to capture unique information from each modality. As shown in Tab. 5,
we removed the modality-specific experts and processed all modalities using a
single expert. We observed a performance degradation, with a 1.9% drop in the
F-score. The performance decline produced by removing the modality-specific
experts indicates the effectiveness of our decomposition of modality-shared
and modality-specific features. When there is no modality-specific experts,
all the experts are treated equally, losing the identification such as
geometry from depth, motion from event, or temperature from Thermal. Our
modality-specific design enables us to effectively model each modality’s
characteristics and make the different modalities benefit from each other.

##### Number of Experts

We also explored the impact of the number of modality-specific experts used
for each modality on the results. We set the number of experts specific to
each modality to 1, where each modality uses a single expert. We used cross-
entropy loss to force each expert to learn modality-specific information.
Additionally, we set the number of experts corresponding to each modality to 2
and 3, as shown in Tab. 5, transforming the classification loss to a multi-
label one. The experimental results indicate that using two modality-specific
experts yields the best performance. If each modality has only one dedicated
expert, the feature representation is constrained. If each modality has three
experts, internal conflicts among the modality-specific experts arise,
negatively impacting the performance of the model.

We believe that the number of expert is also correlated to the expert design.
Since we leverage a low-rank architecture, while being efficient, this limits
the network’s capability of deeply explore the more meaningful clues. On the
other hand, this can also be impacted by the choice on the baseline tracker,
which somehow limits the performance upbound. Finally, the training data size
can also have an influence here. We believe that with more data, we shall be
able to have better performance with more experts.

##  5 Conclusion

In conclusion, we have made significant strides in the field of RGB-X video
object tracking, aiming to develop a generalist tracker that better aligns
with AGI expectations. Our proposed tracker, termed as XTrack, which
incorporates a novel mixture of model experts (MeME) and a dynamic router, can
handle any modality as input without requiring prior knowledge during
inference. Extensive experimental results have validated the effectiveness of
our method, demonstrating its robustness and efficiency in various tracking
scenarios including in-distribution and out-of-distribution cases. These
contributions pave the way for future advancements in creating more robust and
generalizable tracking models. Our experiments are reported in the well tested
benchmark datasets, where an exhaustive comparison on different fronts can
also be found. Furthermore, the well utilized RGB-only dataset followed by the
inclusion of the additional modalities of the RGB-X shows a way of adding
other modalities in the similar fashion, without requiring large modality
specific datasets anymore.

##### Limitations and social impact.

The current performance of the proposed generalist tracker is not always
competitive against the specialist models. Furthermore, while being blind
during inference, the current method still requires the modalities to be known
during the training. Like other deep learning algorithms, our method needs to
be trained on large-scale datasets to perform well. The object tracking has
widespread use, and we support its use only in the socially beneficial cases.

## References

  * Akbari et al. [2024] H. Akbari, D. Kondratyuk, Y. Cui, R. Hornung, H. Wang, and H. Adam.  Alternating gradient descent and mixture-of-experts for integrated multimodal perception.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Awais et al. [2023] M. Awais, M. Naseer, S. Khan, R. M. Anwer, H. Cholakkal, M. Shah, M.-H. Yang, and F. S. Khan.  Foundational models defining a new era in vision: A survey and outlook.  _arXiv preprint arXiv:2307.13721_ , 2023.
  * Cao et al. [2024] B. Cao, J. Guo, P. Zhu, and Q. Hu.  Bi-directional adapter for multimodal tracking.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 38, pages 927–935, 2024.
  * Caruana [1997] R. Caruana.  Multitask learning.  _Machine learning_ , 28:41–75, 1997.
  * Chen et al. [2023a] S. Chen, P. Sun, Y. Song, and P. Luo.  Diffusiondet: Diffusion model for object detection.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 19830–19843, 2023a.
  * Chen et al. [2023b] T. Chen, L. Li, S. Saxena, G. Hinton, and D. J. Fleet.  A generalist framework for panoptic segmentation of images and videos.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 909–919, 2023b.
  * Chen et al. [2021] X. Chen, B. Yan, J. Zhu, D. Wang, X. Yang, and H. Lu.  Transformer tracking.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 8126–8135, 2021.
  * Chen et al. [2022] Z. Chen, B. Zhong, G. Li, S. Zhang, R. Ji, Z. Tang, and X. Li.  Siamban: Target-aware tracking with siamese box adaptive network.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(4):5158–5173, 2022.
  * Dai et al. [2024] D. Dai, C. Deng, C. Zhao, R. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, et al.  Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models.  _arXiv preprint arXiv:2401.06066_ , 2024.
  * Danelljan et al. [2019] M. Danelljan, G. Bhat, F. S. Khan, and M. Felsberg.  Atom: Accurate tracking by overlap maximization.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4660–4669, 2019.
  * Danelljan et al. [2020] M. Danelljan, L. V. Gool, and R. Timofte.  Probabilistic regression for visual tracking.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 7183–7192, 2020.
  * Devlin et al. [2018] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Gao et al. [2022] S. Gao, J. Yang, Z. Li, F. Zheng, A. Leonardis, and J. Song.  Learning dual-fused modality-aware representations for rgbd tracking.  In _European Conference on Computer Vision_ , pages 478–494. Springer, 2022.
  * Gao et al. [2019] Y. Gao, C. Li, Y. Zhu, J. Tang, T. He, and F. Wang.  Deep adaptive fusion network for high performance rgbt tracking.  In _2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)_ , pages 91–99, 2019.  doi: 10.1109/ICCVW.2019.00017.
  * Girdhar et al. [2023] R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and I. Misra.  Imagebind: One embedding space to bind them all.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15180–15190, 2023.
  * Gu et al. [2023] J. Gu, Z. Han, S. Chen, A. Beirami, B. He, G. Zhang, R. Liao, Y. Qin, V. Tresp, and P. Torr.  A systematic survey of prompt engineering on vision-language foundation models.  _arXiv preprint arXiv:2307.12980_ , 2023.
  * Guo et al. [2020] D. Guo, J. Wang, Y. Cui, Z. Wang, and S. Chen.  Siamcar: Siamese fully convolutional classification and regression for visual tracking.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 6269–6277, 2020.
  * He et al. [2021] B. He, H. Li, S. Wu, D. Wang, Z. Zhang, Q. Dong, C. Xu, and F. Gao.  Fast-dynamic-vision: Detection and tracking dynamic objects with event and depth sensing.  In _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 3071–3078. IEEE, 2021.
  * Henriques et al. [2014] J. F. Henriques, R. Caseiro, P. Martins, and J. Batista.  High-speed tracking with kernelized correlation filters.  _IEEE transactions on pattern analysis and machine intelligence_ , 37(3):583–596, 2014.
  * Hong et al. [2024] L. Hong, S. Yan, R. Zhang, W. Li, X. Zhou, P. Guo, K. Jiang, Y. Chen, J. Li, Z. Chen, et al.  Onetracker: Unifying visual object tracking with foundation models and efficient tuning.  _arXiv preprint arXiv:2403.09634_ , 2024.
  * Hou et al. [2024] X. Hou, J. Xing, Y. Qian, Y. Guo, S. Xin, J. Chen, K. Tang, M. Wang, Z. Jiang, L. Liu, et al.  Sdstrack: Self-distillation symmetric adapter learning for multi-modal visual object tracking.  _arXiv preprint arXiv:2403.16002_ , 2024.
  * Hui et al. [2023] T. Hui, Z. Xun, F. Peng, J. Huang, X. Wei, X. Wei, J. Dai, J. Han, and S. Liu.  Bridging search region interaction with template for rgb-t tracking.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13630–13639, 2023.
  * Jain et al. [2024] Y. Jain, H. Behl, Z. Kira, and V. Vineet.  Damex: Dataset-aware mixture-of-experts for visual understanding of mixture-of-datasets.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Kiani Galoogahi et al. [2017] H. Kiani Galoogahi, A. Fagg, and S. Lucey.  Learning background-aware correlation filters for visual tracking.  In _Proceedings of the IEEE international conference on computer vision_ , pages 1135–1143, 2017.
  * Kristan et al. [2020a] M. Kristan, A. Leonardis, J. Matas, M. Felsberg, R. Pflugfelder, J.-K. Kämäräinen, M. Danelljan, L. Č. Zajc, A. Lukežič, O. Drbohlav, et al.  The eighth visual object tracking vot2020 challenge results.  In _Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16_ , pages 547–601. Springer, 2020a.
  * Kristan et al. [2020b] M. Kristan, A. Leonardis, J. Matas, M. Felsberg, R. Pflugfelder, J.-K. Kämäräinen, M. Danelljan, L. Č. Zajc, A. Lukežič, O. Drbohlav, et al.  The eighth visual object tracking vot2020 challenge results.  In _Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16_ , pages 547–601. Springer, 2020b.
  * Kristan et al. [2021] M. Kristan, J. Matas, A. Leonardis, M. Felsberg, R. Pflugfelder, J.-K. Kämäräinen, H. J. Chang, M. Danelljan, L. Cehovin, A. Lukežič, et al.  The ninth visual object tracking vot2021 challenge results.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 2711–2738, 2021.
  * Kristan et al. [2022] M. Kristan, A. Leonardis, J. Matas, M. Felsberg, R. Pflugfelder, J.-K. Kämäräinen, H. J. Chang, M. Danelljan, L. Č. Zajc, A. Lukežič, et al.  The tenth visual object tracking vot2022 challenge results.  In _European Conference on Computer Vision_ , pages 431–460. Springer, 2022.
  * Lester et al. [2021] B. Lester, R. Al-Rfou, and N. Constant.  The power of scale for parameter-efficient prompt tuning.  _arXiv preprint arXiv:2104.08691_ , 2021.
  * Li et al. [2017] C. Li, N. Zhao, Y. Lu, C. Zhu, and J. Tang.  Weighted sparse representation regularized graph learning for rgb-t object tracking.  In _Proceedings of the 25th ACM international conference on Multimedia_ , pages 1856–1864, 2017.
  * Li et al. [2019] C. Li, X. Liang, Y. Lu, N. Zhao, and J. Tang.  Rgb-t object tracking: Benchmark and baseline.  _Pattern Recognition_ , 96:106977, 2019.
  * Li et al. [2021] C. Li, W. Xue, Y. Jia, Z. Qu, B. Luo, J. Tang, and D. Sun.  Lasher: A large-scale high-diversity benchmark for rgbt tracking.  _IEEE Transactions on Image Processing_ , 31:392–404, 2021.
  * Li et al. [2023] H. Li, J. Zhu, X. Jiang, X. Zhu, H. Li, C. Yuan, X. Wang, Y. Qiao, X. Wang, W. Wang, et al.  Uni-perceiver v2: A generalist model for large-scale vision and vision-language tasks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2691–2700, 2023.
  * Likhosherstov et al. [2021] V. Likhosherstov, A. Arnab, K. Choromanski, M. Lucic, Y. Tay, A. Weller, and M. Dehghani.  Polyvit: Co-training vision transformers on images, videos and audio.  _arXiv preprint arXiv:2111.12993_ , 2021.
  * Liu et al. [2021] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 10012–10022, 2021.
  * Lu et al. [2022] J. Lu, C. Clark, R. Zellers, R. Mottaghi, and A. Kembhavi.  Unified-io: A unified model for vision, language, and multi-modal tasks.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Moor et al. [2023] M. Moor, O. Banerjee, Z. S. H. Abad, H. M. Krumholz, J. Leskovec, E. J. Topol, and P. Rajpurkar.  Foundation models for generalist medical artificial intelligence.  _Nature_ , 616(7956):259–265, 2023.
  * Mustafa et al. [2022] B. Mustafa, C. Riquelme, J. Puigcerver, R. Jenatton, and N. Houlsby.  Multimodal contrastive learning with limoe: the language-image mixture of experts.  _Advances in Neural Information Processing Systems_ , 35:9564–9576, 2022.
  * Riquelme et al. [2021] C. Riquelme, J. Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. Susano Pinto, D. Keysers, and N. Houlsby.  Scaling vision with sparse mixture of experts.  _Advances in Neural Information Processing Systems_ , 34:8583–8595, 2021.
  * Song et al. [2018] Y. Song, C. Ma, X. Wu, L. Gong, L. Bao, W. Zuo, C. Shen, R. W. Lau, and M.-H. Yang.  Vital: Visual tracking via adversarial learning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 8990–8999, 2018.
  * Subramanian et al. [2024] S. Subramanian, P. Harrington, K. Keutzer, W. Bhimji, D. Morozov, M. W. Mahoney, and A. Gholami.  Towards foundation models for scientific machine learning: Characterizing scaling and transfer behavior.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Touvron et al. [2023] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Wang et al. [2020] C. Wang, C. Xu, Z. Cui, L. Zhou, T. Zhang, X. Zhang, and J. Yang.  Cross-modal pattern-propagation for rgb-t tracking.  In _Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition_ , pages 7064–7073, 2020.
  * Wang et al. [2024] W. Wang, Z. Chen, X. Chen, J. Wu, X. Zhu, G. Zeng, P. Luo, T. Lu, J. Zhou, Y. Qiao, et al.  Visionllm: Large language model is also an open-ended decoder for vision-centric tasks.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Wang et al. [2023a] X. Wang, J. Li, L. Zhu, Z. Zhang, Z. Chen, X. Li, Y. Wang, Y. Tian, and F. Wu.  Visevent: Reliable object tracking via collaboration of frame and event flows.  _IEEE Transactions on Cybernetics_ , pages 1–14, 2023a.
  * Wang et al. [2023b] X. Wang, W. Wang, Y. Cao, C. Shen, and T. Huang.  Images speak in images: A generalist painter for in-context visual learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6830–6839, 2023b.
  * Wu et al. [2023] Z. Wu, J. Zheng, X. Ren, F.-A. Vasluianu, C. Ma, D. P. Paudel, L. Van Gool, and R. Timofte.  Single-model and any-modality for video object tracking.  _arXiv preprint arXiv:2311.15851_ , 2023.
  * Xiao et al. [2022] Y. Xiao, M. Yang, C. Li, L. Liu, and J. Tang.  Attribute-based progressive fusion network for rgbt tracking.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 2831–2838, 2022.
  * Xie et al. [2022] F. Xie, C. Wang, G. Wang, Y. Cao, W. Yang, and W. Zeng.  Correlation-aware deep tracking.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8751–8760, 2022.
  * Yan et al. [2021a] B. Yan, H. Peng, J. Fu, D. Wang, and H. Lu.  Learning spatio-temporal transformer for visual tracking.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 10448–10457, 2021a.
  * Yan et al. [2021b] S. Yan, J. Yang, J. Käpylä, F. Zheng, A. Leonardis, and J.-K. Kämäräinen.  Depthtrack: Unveiling the power of rgbd tracking.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10725–10733, 2021b.
  * Yang et al. [2022a] J. Yang, Z. Li, S. Yan, F. Zheng, A. Leonardis, J.-K. Kämäräinen, and L. Shao.  Rgbd object tracking: An in-depth review.  _arXiv preprint arXiv:2203.14134_ , 2022a.
  * Yang et al. [2022b] J. Yang, Z. Li, F. Zheng, A. Leonardis, and J. Song.  Prompting for multi-modal tracking.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , pages 3492–3500, 2022b.
  * Ye et al. [2022] B. Ye, H. Chang, B. Ma, S. Shan, and X. Chen.  Joint feature learning and relation modeling for tracking: A one-stream framework.  In _European Conference on Computer Vision_ , pages 341–357. Springer, 2022.
  * Yuan [2023] Y. Yuan.  On the power of foundation models.  In _International Conference on Machine Learning_ , pages 40519–40530. PMLR, 2023.
  * Zhang et al. [2023a] C. Zhang, L. Liu, Y. Cui, G. Huang, W. Lin, Y. Yang, and Y. Hu.  A comprehensive survey on segment anything model for vision and beyond.  _arXiv preprint arXiv:2305.08196_ , 2023a.
  * Zhang et al. [2020] H. Zhang, L. Zhang, L. Zhuo, and J. Zhang.  Object tracking in rgb-t videos using modal-aware attention network and competitive learning.  _Sensors_ , 20(2):393, 2020.
  * Zhang et al. [2022a] J. Zhang, B. Dong, H. Zhang, J. Ding, F. Heide, B. Yin, and X. Yang.  Spiking transformers for event-based single object tracking.  In _CVPR_ , 2022a.
  * Zhang et al. [2019] L. Zhang, M. Danelljan, A. Gonzalez-Garcia, J. Van De Weijer, and F. Shahbaz Khan.  Multi-modal fusion for end-to-end rgb-t tracking.  In _Proceedings of the IEEE/CVF International conference on computer vision workshops_ , pages 2252–2261, 2019.
  * Zhang et al. [2022b] P. Zhang, J. Zhao, D. Wang, H. Lu, and X. Ruan.  Visible-thermal uav tracking: A large-scale benchmark and new baseline.  In _CVPR_ , 2022b.
  * Zhang et al. [2023b] Y. Zhang, K. Gong, K. Zhang, H. Li, Y. Qiao, W. Ouyang, and X. Yue.  Meta-transformer: A unified framework for multimodal learning.  _arXiv preprint arXiv:2307.10802_ , 2023b.
  * Zhao et al. [2023] H. Zhao, D. Wang, and H. Lu.  Representation learning for visual object tracking by masked appearance transfer.  In _CVPR_ , pages 18696–18705, 2023.
  * Zhao et al. [2021] J. Zhao, X. Zhang, and P. Zhang.  A unified approach for tracking uavs in infrared.  In _ICCV_ , pages 1213–1222, 2021.
  * Zhao et al. [2024] X. Zhao, Y. Pang, W. Ji, B. Sheng, J. Zuo, L. Zhang, and H. Lu.  Spider: A unified framework for context-dependent concept understanding.  _arXiv preprint arXiv:2405.01002_ , 2024.
  * Zhu et al. [2023a] J. Zhu, Z. Chen, Z. Hao, S. Chang, L. Zhang, D. Wang, H. Lu, B. Luo, J.-Y. He, J.-P. Lan, et al.  Tracking anything in high quality.  _arXiv preprint arXiv:2307.13974_ , 2023a.
  * Zhu et al. [2023b] J. Zhu, S. Lai, X. Chen, D. Wang, and H. Lu.  Visual prompt multi-modal tracking.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9516–9526, 2023b.
  * Zhu et al. [2023c] X.-F. Zhu, T. Xu, Z. Tang, Z. Wu, H. Liu, X. Yang, X.-J. Wu, and J. Kittler.  Rgbd1k: A large-scale dataset and benchmark for rgb-d object tracking.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 37, pages 3870–3878, 2023c.
  * Zhu et al. [2020] Y. Zhu, C. Li, J. Tang, and B. Luo.  Quality-aware feature aggregation network for robust rgbt tracking.  _IEEE Transactions on Intelligent Vehicles_ , 6(1):121–130, 2020.
  * Zhu et al. [2023d] Z. Zhu, J. Hou, and D. O. Wu.  Cross-modal orthogonal high-rank augmentation for rgb-event transformer-trackers.  In _ICCV_ , pages 21988–21998, 2023d.

## Appendix

### S.1 Selection of Experts

For each modality, we employ a multi-class loss on top of the router to ensure
that each modality has its own dedicated experts. To validate the design, we
visualize the distribution of different experts, as shown in Fig. 5. For ease
of review, we take event dataset VisEvent [45] as an example. It can be seen
that our router perfectly assign the event experts to analyze the input event
data, with only a negligible small amount of misclassification in the shallow
embedding space. Such a visualization validates our modality-aware multi-label
classification loss and corresponds well to our initial motivation of
assigning right experts to input.

Figure 5: The selection of RGB-Event data, "embed" representing the selection
of experts in the patch embedding layer.

### S.2 Visualization results

In our main manuscript, we have shown that our XTrack has achieved overall
best performance on the challenging RGBT234 [31] dataset. Here, we aim for a
more in-depth analysis of the performances vis-a-vis sequences with different
attributes. As shown in Fig. 6, the results show that our generalist and blind
XTrack outperforms SOTA counterparts with thermal-specialized weight in
various challenging scenarios, such as scale variation and camera moving.
These sequences are typically the challenges that can be dealt by depth camera
and event camera. Hence, our superior performance shows that our tracker can
effectively leverage the cross-modal joint benefit with emergent alignments,
even though depth and event are not naturally available in the RGB-T dataset.

Figure 6: Performance of models under different attributes in RGB234 dataset.