# Some remarks on compositeness of $T^{+}_{cc}$

###### Abstract

Recently LHCb experimental group find an exotic state $T^{+}_{cc}$ from the
process $p\bar{p}\to D^{0}D^{0}\pi^{+}+X$. A key question is if it is just a
molecule or may have confined tetraquark ingredient. To investigate this,
different methods are taken, including two channel ($D^{*+}D^{0}$ and
$D^{*0}D^{+}$) $K$-matrix unitarization and single channel Flatté-like
parametrization method analysed by pole counting rule and spectral density
function sum rule. It demonstrates that $T^{+}_{cc}$ is a molecular state,
though the possibility that there may exist elementary ingredient can not be
excluded, by rough analysis on its production rate.

Chang Chen${}^{\dagger\,}$, Ce Meng†, Zhi-Guang Xiao♡,111Corresponding
author., Han-Qing Zheng♡

† Department of Physics, Peking University, Beijing 100871, P. R. China
♡ College of Physics, Sichuan University, Chengdu, Sichuan 610065, P. R. China

##  1 Introduction

The LHCb collaboration has found a very narrow peak structure named
$T^{+}_{cc}$ in the $D^{0}D^{0}\pi^{+}$ invariant mass spectrum, in $pp\to
X+D^{0}D^{0}\pi^{+}$ process[1]. The mass parameters obtained from a generic
constant-width Breit-Wigner fit are obtained as

$\delta m_{BW}=-273\pm 61\pm 5^{+11}_{-14}~{}\mathrm{keV}\
,\,\,\,\Gamma_{BW}=410\pm 165\pm 43^{+18}_{-38}~{}\mathrm{keV}$

where $\delta m_{BW}$ defines the mass shift with respect to the $D^{*+}D^{0}$
threshold. Later it is suggested that $T^{+}_{cc}$ is more possibly an
isoscalar state with spin-parity quantum numbers $J^{P}=1^{+}$[2]. The
constituent of $T^{+}_{cc}$ is $cc\bar{u}\bar{d}$ and there is no annihilated
quark pair, similar as $X_{1}(2900)$ ($ud\bar{s}\bar{c}$)[3, 4].

The experimental observation has stimulated a lot of theoretical discussions.
First of all, there are some dynamical lattice QCD simulations about double
charmed tetraquarks, though not providing a definite conclusion on the
existence of the $T^{+}_{cc}$ state[5, 6]. Recently, based on (2 + 1)-flavor
lattice QCD simulations, $D^{*}D$ system is studied more carefully. It is
verified that there is a loosely bound state near threshold (-10 keV) [7].
Many phenomenological studies have also been made. A theoretical prediction is
that there may exist a $cc\bar{u}\bar{d}$ tetraquark with $J^{P}=1^{+}$ near
$D^{*+}D^{0}$ threshold [8]. Besides, the heavy meson chiral effective field
theory (HMChEFT) is used, which considers contact and one pion exchange (OPE)
interaction. The analysis prefers that $T^{+}_{cc}$ state is a molecule of
$D^{*+}D^{0}$ and $D^{*0}D^{+}$[9, 10]. The effect of triangle diagram
singularity is also evaluated with $D^{*}D\pi$ interactions. It is found that
the contribution is too weak than that of the tree diagram, which suggests
that $T^{+}_{cc}$ is not from triangle singularity [11]. The pole parameters
of $T^{+}_{cc}$ extracted from a simple $K$-matrix amplitude are also studied
and it is found that $T^{+}_{cc}$ may originate from a $D^{*+}D^{0}$ virtual
state [12]. The extended chiral lagrangian with $K$-matrix unitarity approach
is also applied, and it is suggested that vector meson exchanges play a
crucial role in forming $T^{+}_{cc}$ bound state of $D^{*}D$ [13].

In this work, to estimate that whether $T^{+}_{cc}$ is just a loosely-bound
s-wave molecule of $D^{*}D$ or it contains $cc\bar{u}\bar{d}$ ingredient,
different approaches are used. Firstly we take the approach similar to that of
Ref. [13], using two channel ($D^{*+}D^{0}$ and $D^{*0}D^{+}$) $K$-matrix
unitarization with hidden gauge chiral lagrangian. In reference [13], the
authors only consider the vector exchanging diagram contributions and there is
no experimental data fitting. In this paper, more complete interactions
including pseudoscalar, vector exchanges and $D^{*}D$ contact terms are
introduced and a combined fit of $DD\pi$, $D\pi$ and $DD$ channels is made. It
indicates that the vector meson $\rho$ coupling exchanges really make non-
negligible contributions in generating $T^{+}_{cc}$ resonance comparing with
other two interactions. In this scheme, there exists a bound state near the
$D^{*+}D^{0}$ threshold which suggests that $T^{+}_{cc}$ may be a $D^{*}D$
molecule. Furthermore, the Flatté-like parametrization is also used. Through a
combined fit on three-body and two-body invariant mass spectrum, we find that
the result is same based on PCR and spectral density function sum rule
calculation[14, 15, 16, 17, 18, 19]: there is only one pole near $D^{*}D$
threshold and the corresponding $\mathcal{Z}\simeq 1$. We also try to judge
the compositeness of $T^{+}_{cc}$ by comparing its productivity ($pp\to
T^{+}_{cc}+X$) with different theoretical estimations. The $D^{*}D$ molecule
productivity should be significantly less than a confined tetraquark
productivity[20]. However, according to a rough comparison with $\Xi_{cc}$
data, the order of magnitude of $T^{+}_{cc}$ production rate is in between,
hence it is also hard to make a judgement.

This paper is as follows: Sec. I is the introduction, in Sec. II, the
traditional K-matrix unitarization approach with hidden gauge chiral
lagrangian in $s$-wave approximation is introduced and its numerical fit is
shown. In Sec. III, other frameworks are employed to analyse the compositeness
of $T^{+}_{cc}$. Finally, in Sec. IV, a brief conclusion on the structure of
$T^{+}_{cc}$ is made.

##  2 K-matrix unitarity approach

A chiral lagrangian with hidden gauge symmetry is often used to describe
vector and pseudoscalar meson interactions[21, 22, 23, 24]. Here we list
relevant coupling terms

$\displaystyle\mathcal{L}=\mathcal{L}_{0}-ig\mathrm{Tr}([P,\partial_{\mu}P]V^{\mu})+ig\mathrm{Tr}([V^{\nu},\partial_{\mu}V_{\nu}]V^{\mu})$
(1)
$\displaystyle-\frac{g^{2}}{2}\mathrm{Tr}([P,V_{\mu}]^{2})+\frac{g^{2}}{4}\mathrm{Tr}([V_{\mu},V_{\nu}]^{2})\
,$

where $\mathcal{L}_{0}$ is the free lagrangian for pseudoscalar and vector
mesons. P and V denote, respectively, properly normalized pseudoscalar and
vector meson matrices

$P=\begin{pmatrix}\frac{\eta}{\sqrt{3}}+\frac{\eta^{\prime}}{\sqrt{6}}+\frac{\pi^{0}}{\sqrt{2}}&\pi^{+}&K^{+}&\bar{D}^{0}\\\
\pi^{-}&\frac{\eta}{\sqrt{3}}+\frac{\eta^{\prime}}{\sqrt{6}}-\frac{\pi^{0}}{\sqrt{2}}&K^{0}&D^{-}\\\
K^{-}&\bar{K}^{0}&-\frac{\eta}{\sqrt{3}}+\sqrt{\frac{2}{3}}\eta^{\prime}&D^{-}_{s}\\\
D^{0}&D^{+}&D^{+}_{s}&\eta_{c}\end{pmatrix}\ ,$ (2)
$V=\begin{pmatrix}\frac{\omega}{\sqrt{2}}+\frac{\rho^{0}}{\sqrt{2}}&\rho^{+}&K^{*+}&\bar{D}^{*0}\\\
\rho^{-}&\frac{\omega}{\sqrt{2}}-\frac{\rho^{0}}{\sqrt{2}}&K^{*0}&D^{*-}\\\
K^{*-}&\bar{K}^{*0}&\phi&D^{*-}_{s}\\\
D^{*0}&D^{*+}&D^{*+}_{s}&J/\psi\end{pmatrix}\ .$ (3)

It needs to point out that, only $SU(2)$ symmetry really holds. That is all
couplings constants, appearing in vertices of PPV, VVV and PPVV types, are
invariable in isospin space. Different vertices (including $s$ and $c$) should
be considered case by case. We also adopt previous theoretical work to
estimate these coupling constants.

From Eq. (1), we can get the contact, $t$ and $u$ channel diagrams about
$D^{*+}D^{0}\to D^{*0}D^{+}$ process. We list their amplitudes successively.
First is contact diagrams,

Figure 1: Contact diagrams.

$iM^{c}_{ij}=i~{}g_{D^{*}DD^{*}D}^{2},$ (4)

where $i,j=1,2$ refer to $D^{*+}D^{0}$ and $D^{*0}D^{+}$ channel,
respectively. The coupling $g_{4D}$ has been estimated when studying $X(3872)$
[17], that $g(=g_{D^{*}DD^{*}D})\simeq 16$.

The $t$ channel diagrams include vector meson ($J/\psi$ or $\rho$, $\omega$)
exchanges [13].222The $\omega$ and $\rho^{0}$ exchange diagram have the same
coupling constant with opposite signs. So they almost cancel each other. We
also neglect the momentum dependence in the denominator of the propagator near
the threshold. Here an estimation about coupling constants from the PPV and
VVV vertices are given. That is when $i,j=1,1$ or $2,2$, the coupling constant
$g(=g_{J/\psi D^{(*)}D^{(*)}})\simeq 7.7$, and when $i,j=1,2$ or $2,1$,
$g(=g_{\rho D^{(*)}D^{(*)}})\simeq 3.9$, Which is obtained from the vector
meson dominance (VMD) assumption [22]. The $t$ channel amplitudes are hence
written as follows (Fig. 2),

Figure 2: The $t$ channel diagrams.

$iM^{t}_{ij}=iD_{ij}(p_{1}+p_{3})\cdot(p_{2}+p_{4})~{}\epsilon(p_{1})\cdot\epsilon^{*}(p_{3}),$
(5)

where

$D_{ij}=\begin{pmatrix}\frac{g^{2}_{J/\psi
D^{(*)}D^{(*)}}}{M^{2}_{J/\psi}}&\frac{g^{2}_{\rho
D^{(*)}D^{(*)}}}{m^{2}_{\rho}}\\\ \frac{g^{2}_{\rho
D^{(*)}D^{(*)}}}{m^{2}_{\rho}}&\frac{g^{2}_{J/\psi
D^{(*)}D^{(*)}}}{M^{2}_{J/\psi}}\\\ \end{pmatrix}_{ij}.$ (6)

The third type is $u$ channel diagrams with $\pi$ exchanges as in Fig. 3,

Figure 3: The $u$ channel diagrams.

and the amplitudes are

$iM^{u}_{ij}=iE_{ij}g_{\pi
DD^{*}}^{2}\frac{\epsilon(p_{1})\cdot(p_{1}-2p_{4})~{}\epsilon^{*}(p_{3})\cdot(p_{3}-2p_{2})}{(p_{1}-p_{4})^{2}-m_{\pi}^{2}},$
(7)

where

$E_{ij}=\begin{pmatrix}-1&\frac{1}{2}\\\ \frac{1}{2}&-1\\\
\end{pmatrix}_{ij}.$ (8)

The coupling strength $g_{\pi DD^{*}}$ can be restricted by the decay process
$D^{*+}\to D^{0}\pi^{+}$, and we take the value $g(=g_{\pi DD^{*}})\simeq 8.4$
[25]. There also exist $u$ channel diagrams with pseudo-scalar meson
$\eta_{c}$ exchanges. They are not important at this energy range as tested
numerically, so we neglect these diagrams. As for the amplitudes corresponding
to Fig. 3, the $u$ channel exchanging $\pi$ is somewhat special because it is
possible to exchange one on-shell $\pi$ meson. After partial wave projection,
there exists, in tree level amplitudes, an additional cut in the energy region
above $D^{*+}D^{0}$ threshold. Here this singularity will disturb unitarity.
To remedy this, we keep the relation $m_{D^{*}}=m_{D}+m_{\pi}$ to keep away
from the singularity, similar to Refs. [9, 10]. At last, we get the total
couple channel amplitudes

$M_{ij}=M^{c}_{ij}+M^{t}_{ij}+M^{u}_{ij}\ .$ (9)

Furthermore, it can be unitized by the relation

$\mathbf{T}^{-1}=\mathbf{K}^{-1}-\mathbf{g}(s),$ (10)

where $\mathbf{T}$ is the unitarized scattering $T$ matrix, $\mathbf{K}$ is a
two channel scattering amplitude matrix in s-wave approximation [26]. And
$\mathbf{g}(s)\equiv\mathrm{diag}\\{g_{i}(s)\\}$. In our normalization

$g_{i}(s;M_{i},m_{i})=-16\pi
i\int\frac{d^{4}q}{(2\pi)^{4}}\frac{1}{(q^{2}-M_{i}^{2}+i\epsilon)((P-q)^{2}-m_{i}^{2}+i\epsilon)},\,\,(s=P^{2})$
(11)

where $M_{i}$ is the vector meson mass and $m_{i}$ the pseudoscalar meson mass
in the $i$-th channel. The expression of $g_{i}(s)$ in Eq. (11) is
renormalized using standard $\overline{\mathrm{MS}}$ scheme, which introduces
an explicit renormalization scale ($\mu$) dependence. In our fit we choose the
same $\mu$ parameter in two channels.

To get a finite width for the $T_{cc}^{+}$ state below $D^{*}D$ threshold we
need to consider the finite width of the $D^{*}$ state. This is accomplished
by performing a convolution of the $g_{i}(s)$ functions with the mass
distribution of the $D^{*}$ states [27]:

$S\left(s_{V};M_{V},\Gamma_{V}\right)\equiv-\frac{1}{\pi}\operatorname{Im}\left\\{\frac{1}{s_{V}-M_{V}^{2}+iM_{V}\Gamma_{V}}\right\\}$
(12)

such that

$\tilde{g}_{i}(s;M_{i},m_{i})=\mathcal{C}\int_{s_{Vmin}}^{s_{Vmax}}ds_{V}g_{i}(s;\sqrt{s_{V}},m_{i})S\left(s_{V};M_{i},\Gamma_{i}\right)\
,$ (13)

where $\mathcal{C}$ is a normalization factor. The main contribution to this
integration is from the region around the unstable mass $s_{V}\sim M_{V}^{2}$,
so we can introduce a cutoff $s_{Vmin}$ and $s_{Vmax}$. For example, for
$\tilde{g}_{1}$ it is integrated from $(m_{D_{0}}+m_{\pi^{+}})^{2}$ to
$(m_{D^{*+}}+2\Gamma_{D^{*+}})^{2}$ and for $\tilde{g}_{2}$ it is integrated
from $(m_{D_{0}}+m_{\pi^{0}})^{2}$ to $(m_{D^{*0}}+2\Gamma_{D^{*0}})^{2}$.
Here we take the decay widths as constants because we only focus on the region
near the $D^{*+}D^{0}$ threshold and it makes little difference neglecting the
$s$ dependence in numerical calculations. The constant decay widths suggested
by PDG [28] read

$\Gamma_{D^{*+}}=83.4~{}\mathrm{keV},~{}~{}\Gamma_{D^{*0}}=55.3~{}\mathrm{keV}.$
(14)

In order to fit the final state three-body invariant mass spectrum of
$D^{0}D^{0}\pi^{+}$, the final-state interaction (FSI) [19] between
$D^{*+}D^{0}$ and/or $D^{*0}D^{+}$ needs to be considered, before considering
the $D^{*+}\to D^{0}+\pi^{+}$ decay. The FSI amplitude reads

$\mathcal{F}_{D^{*+}D^{0}}(s)=\alpha_{1}~{}T_{11}+\alpha_{2}~{}T_{21}\ ,$ (15)

where $\alpha_{1}$, $\alpha_{2}$ are smooth real polynomials, and since the
energy region of interest is very small, we treat them as constant parameters
near the thresholds of $D^{*+}D^{0}$ and $D^{*0}D^{+}$. Finally, the decay of
$T^{+}_{cc}\to D^{0}D^{0}\pi^{+}$ can therefore be expressed as in Fig. 4. And
the final scattering amplitude is written as333Analogous equation is used in
[13] earlier. The difference is that here the propagator of $D^{*}$ is written
in unitary gauge rather than feynman gauge. Tough these two gauges make little
numerical difference near and below $D^{*+}D^{0}$ threshold region.

Figure 4: The process of $T^{+}_{cc}\to D^{0}D^{0}\pi^{+}$.

$\displaystyle t=$
$\displaystyle\mathcal{F}_{D^{*+}D^{0}}\left[\frac{{\epsilon}\cdot[({p}_{1}-{p}_{2})+\frac{m_{\pi^{+}}^{2}-m_{D^{0}}^{2}}{m_{D^{*+}}^{2}}({p}_{1}+{p}_{2})]}{M_{12}^{2}-m_{D^{*+}}^{2}+iM_{12}\Gamma_{D^{*+}}(M_{12})}\right.$
(16)
$\displaystyle\left.+\frac{{\epsilon}\cdot[({p}_{3}-{p}_{2})+\frac{m_{\pi^{+}}^{2}-m_{D^{0}}^{2}}{m_{D^{*+}}^{2}}({p}_{3}+{p}_{2})]}{M_{23}^{2}-m_{D^{*+}}^{2}+iM_{23}\Gamma_{D^{*+}}(M_{23})}\right].$

where $M_{12}$ and $M_{23}$ are Dalitz kinematic variables of the final three-
body state. The corresponding definition is $M_{ij}^{2}=(p_{i}+p_{j})^{2}$,
and ${\epsilon}={\epsilon}(P)$ corresponds to the polarization vector of
$T^{+}_{cc}$, $P=\left(p_{1}+p_{2}+p_{3}\right)$, $P^{2}=s$. These invariants
have the relation that
$M^{2}_{12}+M^{2}_{13}+M^{2}_{23}=P^{2}+p_{1}^{2}+p_{2}^{2}+p_{3}^{2}$.
Finally the decay width of $T_{cc}^{+}$ is given by

$d\Gamma(\sqrt{s})=\frac{\mathcal{N}}{2}\frac{32}{\pi}\frac{1}{s^{3/2}}|t|^{2}ds_{12}ds_{23}.$
(17)

The factor $\frac{1}{2}$ in the above equation comes from averaging the two
integrals of $D^{0}$ in the final state. In order to fit the experimental
data, the normalization factor $\mathcal{N}$ should be introduced. As for the
two FSI parameters, $\alpha_{1}$ can be absorbed in the coefficients
$\mathcal{N}$. So $\alpha_{1}=1$ is fixed and there remains one free parameter
$\alpha_{2}$. Besides, when we get the yields for the $D^{0}D^{0}\pi^{+}$
invariant mass spectrum, the resolution function is convoluted

$\mathrm{Yields}(l)=\int_{l-2\sigma}^{l+2\sigma}dl^{\prime}\frac{1}{\sqrt{2\pi}\sigma}\Gamma\left(l^{\prime}\right)\mathrm{e}^{-\frac{\left(l^{\prime}-l\right)^{2}}{2\sigma^{2}}},$
(18)

where $\sigma=1.05\times 263\mathrm{keV}$ [1]. At last, invariant mass
distributions for the selected two body (particles 2 and 3 for example) can
also be derived as the the following function

$\frac{dBr}{dm_{23}}=\mathcal{N}^{\prime}\int_{m^{2}_{D^{0}D^{0}\pi^{+}}}^{m_{max}^{2}}ds\int
ds_{12}|t(s,s_{12},s_{23})|^{2}$ (19)

where $\mathcal{N}^{\prime}$ is another normalization constant, $m_{23}$ is
the invariant mass of particles 2 and 3. The $T^{+}_{cc}$ energy is integrated
from the initial energy $m_{D^{0}D^{0}\pi^{+}}$ to a cutoff $m_{max}$.444Since
$T_{cc}^{+}$ lies just below the threshold of $D^{*}D$ with a sharp peak, we
can make a rough cutoff about one or two its Breit–Wigner widths above the
threshold. The subsequent results are not sensitive to this uncertainty.

Data obtained from LHCb collaboration about three body final states
$D^{0}D^{0}\pi^{+}$[1] and two body invariant mass distributions
$D^{0}\pi^{+}$, $D^{0}D^{0}$ and $D^{+}D^{0}$[2] are used to make a combined
fit. The normalization $\mathcal{N}$, $\mathcal{N^{\prime}}$, FSI parameter
$\alpha_{2}$ and renormalization scale $\mu$ are regarded as fit parameters
and all coupling constants found in the literature are regarded as fixed
parameters (Scheme I).

Figure 5: $D^{0}D^{0}\pi^{+}$ final state invariant mass spectrum. The
vertical purple dash line indicates the $D^{*+}D^{0}$ threshold and the green
one corresponds to the $D^{*0}D^{+}$ threshold. Data come from Ref. [1].

See Fig. 5 for the fit (Scheme I). It is shown that the fit result is very
sensitive to the parameter $\mu$. That is because the peak ($T^{+}_{cc}$
state) is too narrow, considering that unit of $\mu$ is GeV but the signal
range is in MeV. The discussion above seems to suggest that the fit result
prefers a particular choice of parameter $\mu$. This looks annoying but is not
actually a physical problem. That is because in the fit we have fixed all
coupling strength constants previously determined. To avoid the problem, we
can for example take scheme II that all coupling parameters are regarded as
fit parameter, and $\mu$ is removed by replacing $g_{i}(s;M_{i},m_{i})$ as
$i\rho_{i}(s;M_{i},m_{i})$ (i.e., on shell approximation). The expression
$\rho_{i}(s;M_{i},m_{i})$ is the two-body phase space factor

$\rho_{i}(s;M_{i},m_{i})=\frac{\sqrt{s-(M_{i}+m_{i})^{2}}\sqrt{s-(M_{i}-m_{i})^{2}}}{s}.$
(20)

The result is that it can still fit well, just that the coupling $g_{\rho DD}$
becomes larger, see Table 1.

The pole location on the $s$-plane is also studied. If $D^{*}$ is taken as a
stable particle, Then $T_{cc}^{+}$ appear as a bound state pole located at
$\sqrt{s}=3.8746$, i.e., about $500$keV below $D^{*+}D^{0}$ threshold
($\sqrt{s}=3.8751$). Since there is no nearby accompanying virtual pole, we
conclude that, according to the pole counting rule (PCR), $T^{+}_{cc}$ is a
pure moleclue composed of $DD^{*}$. However, due to the instability of
$D^{*}$, the $D^{*}D$ channel opens at the energy a little bit smaller than
$m_{D^{*}}+m_{D}$ and the decay of $T_{cc}^{+}$ take place [13].

Figure 6: The $D^{0}\pi^{+}$ invariant mass spectrum from three body final
state $D^{0}D^{0}+X$ (Data from Ref. [2]).

Furthermore, invariant mass distributions for any two of three final state
particles are also taken into consideration. As for $D^{0}\pi^{+}$ state,
which comes from $D^{*+}D^{0}$, we take $m_{max}=3.8751\mathrm{GeV}$ and it
implies only $T^{+}_{cc}$ below $D^{*+}D^{0}$ threshold make sense to decay
into $D^{0}\pi^{+}$ (normalization constant
$\mathcal{N}^{\prime}=\mathcal{N_{D\pi}}$).

As for $D^{0}D^{0}$ and $D^{+}D^{0}$ two-body final states, we take the same
$m_{max}=3.8751\mathrm{GeV}$ ($\mathcal{N}^{\prime}=\mathcal{N_{DD}}$ here).
On the other side, the $D^{+}D^{0}$ final state, which comes from the
$D^{+}D^{0}\pi^{0}$ final state, is different. Since $D^{+}D^{0}$ state may
come from two channels, $D^{*+}D^{0}$ and $D^{*0}D^{+}$, they need to be
considered altogether aided by isospin symmetry. Since the threshold of second
channel is higher, we take $m_{max}=3.8766\mathrm{GeV}$, and on account of a
symmetry factor $\frac{1}{2}$ in channel including $D^{0}D^{0}$, the
normalization constant here is doubled
($\mathcal{N}^{\prime}=2\mathcal{N_{DD}}$). The fitting results are plotted in
Fig. 7. Both invariant mass spectrums ($D^{0}D^{0}$and $D^{+}D^{0}$) are with
an incoherent background component, parameterised as a product of two-body
phase space function $\Phi_{DD}$ and a linear function. For $D^{+}D^{0}$ from
channel $T^{+}_{cc}\to D^{*0}D^{+}\to D^{+}D^{0}\pi^{0}/D^{+}D^{0}\gamma$,
because the decay channel $D^{*0}\to D^{0}\gamma$ accounts for 35% of total
$D^{*0}$ decay width, this incoherent background contribution is non-
negligible and needs to be counted specially. Here we take this estimation
from Ref. [2] directly.

Figure 7: $D^{0}D^{0}$ ($D^{+}D^{0}$) invariant mass spectrum from three body
final state $D^{0}D^{0}+X$ ($D^{+}D^{0}+X$) [2], where the vertical line of
dashes show $D^{0}D^{0}$ ($D^{+}D^{0}$) threshold.

Finally, the fit parameters are listed in Table 1.

Table 1: Parameters. Scheme | I | II
---|---|---
$\chi^{2}/d.o.f$ | 1.16 | 1.06
$\alpha_{2}$ | $-0.43\pm 0.10$ | $-0.48\pm 0.13$
$\mu/\mathrm{GeV}$ | $1.122\pm 0.001$ | –
$g_{D^{*}DD^{*}D}$ | fixed=$16$ [17] | $16.0\pm 7.36$
$g_{\pi DD^{*}}$ | fixed=$8.4$ [25] | $8.4\pm 6.51$
$g_{\rho D^{(*)}D^{(*)}}$ | fixed=$3.9$ [22] | $7.78\pm 1.46$
$g_{J/\psi D^{(*)}D^{(*)}}$ | fixed=$7.7$ [22] | $13.8\pm 9.16$

##  3 Other insights on $T^{+}_{cc}$

In this section, the production of $T^{+}_{cc}$ in some other methods are also
analysed to figure out its compositeness. First of all, a single channel
Flatté-like parametrization is used. Like the previous calculation in Sec. II,
this process is regarded as a cascade decay. The propagator of $T^{+}_{cc}$ is
approximated by Flatté form. The later propagator of $D^{*}$ here can only
take an simple Breit–Wigner amplitude form, because the energy of this process
is near the threshold of $D^{*}D$ and its range is small enough. Besides, the
momentum dependent polynomial in the numerator is also normalized by a
constant factor $\mathcal{N}$ for convenience. Numerical calculations indicate
that these approximation can make little difference.

Figure 8: Cascade decay

The total s-wave approximation amplitude about process $T^{+}_{cc}\to
D^{0}D^{0}\pi^{+}$ can be written as

$\displaystyle t=$
$\displaystyle\frac{1}{s-M^{2}+iM\left(g\rho(s)\right)}\times$ (21)
$\displaystyle\left(\frac{1}{M_{12}^{2}-m_{D^{*+}}^{2}+iM_{12}\Gamma_{D^{*+}}}+\frac{1}{M_{23}^{2}-m_{D^{*+}}^{2}+iM_{23}\Gamma_{D^{*+}}}\right),$

where $g$ presents the coupling strength with only channel $D^{*}D$. The
doubling of the kinetic variables of the $D^{*}$ propagator is due to the
indistinguishability of $D\pi$ in three-body final state, and the symmetry
factor $\frac{1}{2}$ is absorbed by the total normalization $\mathcal{N}$. By
this parametrization, we make the energy resolution convolution as before and
fit the three-body decay width and two-body invarint mass spectrum at the same
time using previous Eqs. (17, 19). It is worth pointing out that under normal
conditions it will form a divergent peak because of the zero partial decay
width. But if we regard $D^{*}$ as an unstable particle, in other words the
amplitude can emerge imaginary part when the energy does not reach the
$D^{*}D$ threshold yet, the peak is not divergent anymore. We can use the same
trick as Eq. (13) to treat $\rho$ in Eq. (21), or more simply take the value
$m_{D^{*}}$ in $\rho$ with a imaginary part $\Gamma_{D^{*}}$. The choice of
them does not effect the result except for the goodness of fit. Here we take
the latter scenario. The united fits result is following.

Figure 9: The three-body and two-body invarint mass spectrum

We list the corresponding parameters in Table 2, and the pole structure of the
Flatté amplitude is drawn in Fig. 10.

Table 2: Fit parameter $\chi^{2}/d.o.f$ | $g$ | $M$
---|---|---
$0.81$ keV | $0.075\pm 0.015$ |  $3874.1\pm 0.2$MeV
Figure 10: The pole structure of flatté amplitude

Furthermore, according to the Flatté-like parametrization, it is nature to
calculate the probability of finding an ‘elementary’ state in the continuous
spectrum by the spectral density function[29]

$\omega(E)=\frac{1}{2\pi}\frac{\tilde{g}\sqrt{2\tilde{M}E}\theta(E)+\tilde{\Gamma}_{0}}{\left|E-E_{f}+\frac{\mathrm{i}}{2}\tilde{g}\sqrt{2\tilde{M}E}+\frac{\mathrm{i}}{2}\tilde{\Gamma}_{0}\right|^{2}},$
(22)

where $E=\sqrt{s}-m_{D^{*}D}$, $E_{f}=M-m_{D^{*}D}$, $\tilde{M}$ is the
reduced mass of $D^{*}D$, $\theta$ is the step function at threshold and
$\tilde{\Gamma}_{0}$ is the constant partial width for the remaining
couplings. By integrating it with a cut off (usually comparable to the total
decay width$\sim\Gamma$), the possibility of finding an ‘elementary’ state in
the final state is

$\mathcal{Z}=\int_{E_{\min}}^{E_{\max}}\omega(E)\mathrm{d}E.$ (23)

Considering that there is no other channels coupling with $T^{+}_{cc}$ under
the $D^{*}D$ threshold, the $\tilde{\Gamma}_{0}$ here should be set zero. In
this case, the integrating results in different sections are as follows.

Table 3: Spectral density function integrating $\mathcal{Z}$ $[M-\Gamma,M+\Gamma]$ | $[M-2\Gamma,M+2\Gamma]$ | $[M-3\Gamma,M+3\Gamma]$
---|---|---
$\sim$ 0 |  $\sim$ 0 | 0.01

The result suggests that in a simple single channel Flatté-like
parametrization framework, $T^{+}_{cc}$ is a pure molecular state. This is in
agreement with the result of Ref. [30], obtained using effective range
expansion approximation. Our result is much more definite than that obtained
in Ref. [2].

Furthermore, there are also discussions on the compositeness from the
production rate of a particle. There is a cross section relation between
confined state $\Xi_{cc}$($ccu/ccd$)[31] and $T^{+}_{cc}$. These two
experimental data are all collected after 2016, and they are from the same
experimental condition, like transverse momentum truncation $p_{T}$ and
luminance $9fb^{-1}$. After taking the detection efficiency and branching
fractions difference[32], there is a rough relation that

$\frac{\sigma(pp\to
T^{+}_{cc})}{\sigma(pp\to\Xi_{cc})}\sim\frac{1}{3}\times\frac{1}{10}.$ (24)

If we agree that there exists a universal relation between $(Q/QQ)q$ and
$(Q/QQ)qq$ productivity in high energy collision[33], where $Q$ represents
heavy quark and $q$ is light quark, we can get a factor $1/3$, which means
catching two light quarks are always more difficult, i.e.,
$\sigma(pp\to\Xi_{cc})\simeq 3\sigma(pp\to(cc\bar{u}\bar{d}))$. So the ratio
between observed $T^{+}_{cc}$ and hypothetical tetraquark cross section is
derived

$\frac{\sigma(pp\to
T^{+}_{cc})}{\sigma(pp\to(cc\bar{u}\bar{d}))}\sim\frac{1}{10}.$ (25)

On the other hand, it is natural to estimate the different theoretical cross
section orders between ‘elementary’ and ‘molecular’ picture of $T^{+}_{cc}$.
Thanks to that $X(3872)$ resonance has analogous characteristics [34,
35](e.g., binding energy and double c quark), there have been some comparisons
about these orders of magnitude on $X(3872)$[36, 20]. One can borrow the
discussions here and it can be estimated that roughly for $T^{+}_{cc}$

$\frac{\sigma(pp\to(c\bar{u})(c\bar{d}))}{\sigma(pp\to(cc\bar{u}\bar{d}))}\sim\mathcal{O}(10^{-2})-\mathcal{O}(10^{-3}).$
(26)

By comparing Eq. (25) and Eq. (26), the productivity of $T^{+}_{cc}$ just
falls in between two different cases. So using the productivity argurement
does not provide a clear conclusion on the nature of $T^{+}_{cc}$. On the
contrary, the analysis provided in this paper, e.g., Table 3 clearly indicates
the molecular nature of $T^{+}_{cc}$.

##  4 Summary

In this work, we study the nature of $T^{+}_{cc}$ by different methods. We use
the extended hidden local chiral lagrangian with two channel ($D^{*+}D^{0}$
and $D^{*0}D^{+}$) $K$-matrix approach to describe the process $T^{+}_{cc}\to
D^{0}D^{0}\pi^{+}$. The three-body and two-body invariant mass spectrum can be
fitted well at the same time. Also the numerical fit results reveal that the
vector meson exchanges is more important comparing with $\pi$ exchanges and
contact interactions. We also use Flattè formula to study the problem. We
conclude that $T^{+}_{cc}$ is definitely a pure molecular state composed of
$D^{*}D$, in agreement with many of the results found in the literature, but
on a much more confident level.

$Acknowledgements:$ We would like to thank Hao Chen for a careful reading of
the manuscript and very helpful discussions. At last, This work is supported
in part by National Nature Science Foundations of China under Contract Numbers
11975028.

## Reference

  * [1] LHCb, R. Aaij et al.,  Nature Phys. 18, 751 (2022).
  * [2] LHCb, R. Aaij et al.,  Nature Commun. 13, 3351 (2022).
  * [3] LHCb, R. Aaij et al.,  Phys. Rev. D 102, 112003 (2020).
  * [4] H. Chen, H.-R. Qi, and H.-Q. Zheng,  Eur. Phys. J. C 81, 812 (2021).
  * [5] Y. Ikeda et al.,  Phys. Lett. B 729, 85 (2014).
  * [6] P. Junnarkar, N. Mathur, and M. Padmanath,  Phys. Rev. D 99, 034507 (2019).
  * [7] Y. Lyu et al.,  (2023), arXiv: 2302.04505.
  * [8] M. Karliner and J. L. Rosner,  Phys. Rev. Lett. 119, 202001 (2017).
  * [9] M.-L. Du et al.,  Phys. Rev. D 105, 014024 (2022).
  * [10] Z.-Y. Lin, J.-B. Cheng, and S.-L. Zhu,  (2022), arXiv: 2205.14628.
  * [11] N. N. Achasov and G. N. Shestakov,  Phys. Rev. D 105, 096038 (2022).
  * [12] L.-Y. Dai, X. Sun, X.-W. Kang, A. P. Szczepaniak, and J.-S. Yu,  Phys. Rev. D 105, L051507 (2022).
  * [13] A. Feijoo, W. H. Liang, and E. Oset,  Phys. Rev. D 104, 114015 (2021).
  * [14] Y. Lu, C. Chen, K.-G. Kang, G.-y. Qin, and H.-Q. Zheng,  (2023), arXiv: 2302.04150.
  * [15] C. Chen, H. Chen, W.-Q. Niu, and H.-Q. Zheng,  Eur. Phys. J. C 83, 52 (2023).
  * [16] Q.-R. Gong et al.,  Phys. Rev. D 94, 114019 (2016).
  * [17] C. Meng, J. J. Sanz-Cillero, M. Shi, D.-L. Yao, and H.-Q. Zheng,  Phys. Rev. D 92, 034020 (2015).
  * [18] O. Zhang, C. Meng, and H. Q. Zheng,  Phys. Lett. B 680, 453 (2009).
  * [19] D.-L. Yao, L.-Y. Dai, H.-Q. Zheng, and Z.-Y. Zhou,  Rept. Prog. Phys. 84, 076201 (2021).
  * [20] C. Bignamini, B. Grinstein, F. Piccinini, A. D. Polosa, and C. Sabelli,  Phys. Rev. Lett. 103, 162001 (2009).
  * [21] M. Bando, T. Kugo, S. Uehara, K. Yamawaki, and T. Yanagida,  Phys. Rev. Lett. 54, 1215 (1985).
  * [22] Z.-w. Lin and C. M. Ko,  Phys. Rev. C 62, 034903 (2000).
  * [23] M. Harada and K. Yamawaki,  Phys. Rept. 381, 1 (2003).
  * [24] D. Djukanovic, M. R. Schindler, J. Gegelia, G. Japaridze, and S. Scherer,  Phys. Rev. Lett. 93, 122002 (2004).
  * [25] D.-L. Yao, M.-L. Du, F.-K. Guo, and U.-G. Meißner,  JHEP 11, 058 (2015).
  * [26] S. U. Chung,  Spin formalisms, CERN Academic Training Lecture (CERN, Geneva, 1971),  CERN, Geneva, 1969 - 1970.
  * [27] G. Y. Wang, L. Roca, and E. Oset,  Phys. Rev. D 100, 074018 (2019).
  * [28] Particle Data Group, P. A. Zyla et al.,  PTEP 2020, 083C01 (2020).
  * [29] V. Baru, J. Haidenbauer, C. Hanhart, Y. Kalashnikova, and A. Kudryavtsev,  Physics Letters B 586, 53 (2004).
  * [30] L. R. Dai, L. M. Abreu, A. Feijoo, and E. Oset,  (2023), arXiv: 2304.01870.
  * [31] LHCb, R. Aaij et al.,  Phys. Rev. Lett. 119, 112001 (2017).
  * [32] F.-S. Yu et al.,  Chin. Phys. C 42, 051001 (2018).
  * [33] CMS, S. Chatrchyan et al.,  Phys. Lett. B 714, 136 (2012).
  * [34] BaBar, B. Aubert et al.,  Phys. Rev. D 77, 011102 (2008).
  * [35] Belle, T. Aushev et al.,  Phys. Rev. D 81, 031103 (2010).
  * [36] E. Braaten and M. Kusunoki,  Phys. Rev. D 71, 074005 (2005).


11institutetext:  Google Research, San Francisco CA 94105, USA
22institutetext: Google Research, Cambridge MA 02142, USA
22email<EMAIL_ADDRESS>

# Multimodal LLMs for health grounded in individual-specific data

Anastasiya Belyaeva Equal contribution; authors listed alphabetically.11
Justin Cosentino∗ 11 Farhad Hormozdiari 22 Krish Eswaran 11 Shravya Shetty 11
Greg Corrado 11 Andrew Carroll 11
Cory Y. McLean Equal supervision.22 Nicholas A. Furlotte † 11

###### Abstract

Foundation large language models (LLMs) have shown an impressive ability to
solve tasks across a wide range of fields including health. To effectively
solve personalized health tasks, LLMs need the ability to ingest a diversity
of data modalities that are relevant to an individual’s health status. In this
paper, we take a step towards creating multimodal LLMs for health that are
grounded in individual-specific data by developing a framework (HeLM: Health
Large Language Model for Multimodal Understanding) that enables LLMs to use
high-dimensional clinical modalities to estimate underlying disease risk. HeLM
encodes complex data modalities by learning an encoder that maps them into the
LLM’s token embedding space and for simple modalities like tabular data by
serializing the data into text. Using data from the UK Biobank, we show that
HeLM can effectively use demographic and clinical features in addition to
high-dimensional time-series data to estimate disease risk. For example, HeLM
achieves an AUROC of 0.75 for asthma prediction when combining tabular and
spirogram data modalities compared with 0.49 when only using tabular data.
Overall, we find that HeLM outperforms or performs at parity with classical
machine learning approaches across a selection of eight binary traits.
Furthermore, we investigate the downstream uses of this model such as its
generalizability to out-of-distribution traits and its ability to power
conversations around individual health and wellness.

###### Keywords:

Multimodal Large Language Models Health UK Biobank.

##  1 Introduction

Foundation large language models (LLMs) have been shown to solve a range of
natural language processing (NLP) tasks without having been explicitly trained
to do so [4, 36]. As a result, researchers are adapting LLMs to solve a
variety of non-traditional NLP problems across domains. A recent perspective
[23] outlined a variety of health-related use cases that could benefit from
foundation LLMs that have not only generalist medical knowledge but that are
also infused with individual-specific information such as lab values (e.g.,
cholesterol and triglycerides), imaging, time-series data, health tracker
metrics (e.g., daily step count and heart rate), genome sequences, genetic
risk scores, and other omics data modalities. These use cases range from AI
clinician assistants to AI-powered early warning systems to user-facing health
and wellness chatbots.

While the potential applications for foundation LLMs in health are wide-
ranging, at the core of each there is a fundamental need for the model to
ingest complex multimodal individual-specific data and use it to gain an
understanding of the individual’s underlying health risks. The model can then
condition responses to queries on the derived risk profile of an individual.
Though there has been promising recent work in developing generalist medical
LLMs [38, 30, 31, 32], the problem of using multimodal individual-specific
information as context for health-related tasks remains understudied. More
broadly, this capability represents one aspect of the general movement towards
personalization of LLMs [18, 28], which encompasses not only the technical
challenges of data integration, but also the complex ethical questions around
how the model can and should be used.

Figure 1: Overview of HeLM, a multimodal LLM for health. Text features
(orange) are tokenized and embedded into the token embedding space via a
standard embedding matrix. Non-text modalities such as clinical data (blue) or
high-dimensional lung function measures (green) are encoded into the same
token embedding space via modality-specific encoders. The LLM is tuned to
quantify disease risk given complex multimodal inputs.

Providing relevant health information to an LLM could be as simple as
including important disease risk factors, such as lab values, in the prompt by
representing these factors as text [16, 10]. Furthermore, in-context learning
techniques such as few-shot prompting [4] can be employed by giving the model
examples that help it to connect risk factors to underlying disease. However,
this solution isn’t likely to work for the many complex data modalities found
in the health space [1]. For example, it isn’t clear how to incorporate
images, time-series data, or even rich structured tabular data into the LLM
prompt. Furthermore, health factors may not be clearly captured by a single
modality but rather may be best predicted by simultaneously incorporating
features drawn from multiple modalities.

Recently, a variety of methods have been introduced to extend LLMs to the
multimodal setting. The majority of these methods have focused on images and
text [25, 21, 2, 17, 39, 24] with some models adding the ability to
incorporate diverse modalities, such as the movements of robots, video, and
audio [11, 12, 22]. However, these nascent methods have not yet been applied
to the health domain. On the other hand, there are a variety of classical
machine learning methods for integrating multiple data modalities that are
routinely applied in the health domain [1]. Examples include logistic
regression classifiers that take multiple modalities as input, various fusion
models [19], autoencoder-based models [37], and cross-supervision models [40].
However, these traditional approaches lack several potential advantages when
compared to LLM-based methods.

First, foundation LLMs may have encoded extensive prior knowledge about health
related traits. For example, an LLM likely understands that hypertension is
the same as high blood pressure and may even know something about what number
ranges correspond to—normal or high. This prior knowledge can be useful when
dealing with heterogeneous data or with data that only has fuzzy labels.
Secondly, LLMs can incorporate additional prior knowledge through prompt
engineering, whereas in traditional ML methods including priors can be
cumbersome. Thirdly, LLMs may have a high degree of flexibility in working
with missing data. Whereas traditional methods require imputation of missing
values or dropping samples, LLMs can be prompted to ignore missing values or
they can be omitted completely without architectural changes. Finally, many
foundation LLMs are conversational by design, so they can more naturally be
used for applications such as those mentioned previously: user-facing health
and wellness chatbots.

In this paper, we take a step towards multimodal LLMs for health that are
grounded in individual-specific data by developing a framework called HeLM
that enables LLMs to use health-related features to estimate underlying
disease risk. The idea is that an LLM with a representation of background
risk, can use this as context in answering health-related queries. We
formulate a disease classification task and then use the LLM, Flan-PaLMChilla
62b [7], to score potential outcomes. We compare the LLM score with classic
supervised machine learning methods such as logistic regression and gradient-
boosted decision trees by evaluating their ability to distinguish between
individuals with and without the disease.

Using data from the UK Biobank [5], we evaluate different ways of prompting
and encoding data and passing it to the LLM to classify disease status. First,
we serialize health-related features into text similar to previously proposed
approaches [16, 10]. We evaluate zero-shot, few-shot, and a parameter-
efficient soft-prompt tuning approach [20]. Generally, we find that the LLM
performs better than random in the zero-shot and few-shot cases, and has
comparable and often equivalent performance to standard approaches such as
logistic regression and a gradient-boosted decision trees (implemented in
XGBoost [6]) after soft-prompt tuning. Additionally, for some diseases (e.g.,
diabetes), the zero-shot and few-shot approaches perform surprisingly well
when compared with logistic regression and XGBoost, giving evidence to the
model’s use of prior knowledge related to disease risk. However, we observe
that performance degrades with an increase in the number of input features
indicating that the model does not always fully capture signal from serialized
text.

To better capture signal from quantitative data, we propose HeLM, a multimodal
approach to disease risk estimation based on the PaLM-E framework [11]. In
HeLM, non-text modalities are mapped via encoders (trained over input
examples) into the same token embedding space as text, where a separate
encoder is learned for each modality (see Figure 1). We show that HeLM matches
the performance of classical machine learning methods (logistic regression and
XGBoost) and, in some cases, outperforms both the logistic regression and
XGBoost models. We show results for encoding tabular data and spirogram
curves, a time series representation of lung function used for respiratory
disease diagnosis [33, 8]. Finally, we explore how HeLM that has been tuned to
quantify disease risk can be used in downstream conversational tasks. Here, we
find that conversational ability degrades in the tuned LLM, which is
consistent with what others have reported [35]. We discuss future directions
such as fine-tuning schemes that may mitigate degradation.

##  2 Methods

###  2.1 LLMs with Tabular Data

An individual’s health status is often represented by a set of values stored
in tabular format. We explore whether serializing tabular data into natural
language and passing the resulting text to the LLM (Flan-PaLMChilla 62b [7])
is sufficient to achieve strong predictive performance. We construct
serialized inputs by mapping table column names and corresponding values into
JSON-like format, which is combined with the base prompt. For example, for
diabetes prediction, we formulate the following sentence using the data:
“Predict if a patient has the condition or not. bmi: {28.9}. age: {67.0}. sex:
{male}. diabetes: {”. We then compute the log-likelihood of the sentence being
completed with “yes}” or “no}”. This log-likelihood serves as a risk score and
can be evaluated using metrics such as AUROC and AUPRC to assess
discriminatory power.

We evaluate the LLM’s performance in the zero-shot, few-shot and soft-prompt
tuning settings. In the zero-shot setting, the serialized text input is given
directly to the model. In this case, the LLM heavily leverages prior
knowledge. In the few-shot scenario, we prefix 10 examples randomly sampled
from the training dataset to the model’s prompt. For soft-prompt tuning,
following [20], we learn a soft prompt to condition the frozen language model
to perform well on the diseases of interest.

Briefly, soft-prompt tuning is a parameter efficient tuning technique that’s
commonly used to steer the frozen LLM to perform well on a downstream task
given labeled data. Instead of fine-tuning the weights of the LLM or forming a
hard prompt out of tokens as in the few-shot scenario, soft-prompt tuning
learns a matrix $P\subset\mathbb{R}^{p\times k}$ in the token embedding space,
where $p$ is the length of the prompt and $k$ is the size of the language
embedding space. The soft-prompt $P$ is then concatenated with the embedded
tokens and passed to the decoder. We train the soft-prompt using pairs of
examples $(X,Y)$ and backpropagation to update the soft-prompt with the goal
of maximizing the probability of $Y$. We train for 20,000 steps and use 1000
training examples.

###  2.2 Multimodal LLMs for health: HeLM

To enable the LLM to reason over complex high-dimensional inputs, we embed
non-text data modalities, including time-series data like spirograms and
tabular data, into the same latent space as the language tokens (see Figure
1). We use separate encoders for each non-text data modality, where each
encoder learns a mapping to the token embedding space. This approach is based
on the PaLM-E framework [11]. All of the inputs are then processed together by
a pre-trained LLM. More precisely, LLMs typically tokenize the words in a
sentence and map the resulting tokens $w_{i}$ into a language embedding space
$\mathcal{X}\subset\mathbb{R}^{k}$ via a large embedding matrix
$x_{i}=\gamma(w_{i})$ where $\gamma:\mathcal{W}\to\mathcal{X}$. In HeLM, non-
text modalities are mapped to a sequence of vectors in $\mathcal{X}$ via
encoders. For example, for mapping the spirogram time-series we trained
$\phi_{s}:\mathcal{S}\to\mathcal{X}^{q_{s}}$ encoder and for mapping tabular
data we trained $\phi_{t}:\mathcal{T}\to\mathcal{X}^{q_{t}}$ encoder, where
$q_{s}$ and $q_{t}$ correspond to the number of vectors in $\mathcal{X}$ space
or in other words how many “multimodal” tokens each modality is mapped to. We
set $q_{s}=6$ and $q_{t}=6$ for all experiments, however in general these can
be treated as tunable hyper-parameters. In summary, each vector $x_{i}$ is
formed from either the token embedder $\gamma$ or a modality-specific encoder:

$x_{i}=\begin{cases}\gamma(w_{i})&\text{if }i\text{ is a text token}\\\
\phi_{s}(s_{i})&\text{if }i\text{ corresponds to the spirogram modality}\\\
\phi_{t}(t_{i})&\text{if }i\text{ corresponds to the tabular
modality}\end{cases}$

Non-text and text modalities can be injected in any order. We train the
modality-specific encoders keeping the pre-trained LLM weights frozen. This is
similar to soft-prompt tuning but conditioned on data from a specific
modality. For experiments considering multiple diseases we train a single HeLM
model on a mixture of all diseases as opposed to one model per disease.

###  2.3 UK Biobank Dataset Preparation

We obtain clinical features, spirometry, and disease labels from the UK
Biobank (UKB) [5]. Similar to previous work, we focus on the European genetic
ancestry subpopulation [3]. Limiting to European ancestry within the UK
Biobank is a standard heuristic for reducing phenotypic heterogeneity, due to
the correlation between population structure and phenotypic variation [9].
Differences in phenotypes across ancestries are multifactorial—socio-economic,
cultural, etc.—but are often highly correlated with genetic background and
thus population structure. Therefore, selecting study individuals based on a
single genetic background is a convenient way to reduce heterogeneity in
underlying disease risk, at the expense of creating bias in the dataset and
subsequent analyses that utilize this data.

We defined binary phenotype labels using medical records comprising ICD-9
hospital inpatient (HESIN) billing codes, ICD-10 primary care and general
practitioner (GP) read codes, and self-report data. Diseases include asthma,
diabetes, hypertension, stroke, myocardial infarction, major depression,
migraine, all cause mortality, cataracts, gastroesophageal reflux disease
(GERD), hay fever eczema (atopic eczema), osteoarthritis, and pneumonia. The
following clinical features, lab values, and self-reported statuses sourced
from questionnaires were used as model inputs: age, sex, body mass index
(BMI), high-density lipoprotein (HDL) cholesterol, low-density lipoprotein
(LDL) cholesterol, total cholesterol, triglycerides, diastolic blood pressure,
smoking status, snoring, insomnia, daytime napping, average nightly sleeping,
and chronotype. See Table S1 for details describing feature definitions.

Spirometry data was prepared following the preprocessing procedures outlined
in [8]. In short, we used raw volumetric flow curves containing exhalation
volume sampled at 10-ms intervals to liters and then computed the
corresponding flow curve by approximating the first derivative with respect to
time by taking a finite difference. The volume-time and flow-time curves were
then normalized to length 1,000 and combined to generate a one-dimensional
flow-volume spirogram. Following well-accepted spirometry quality control
standards, we filter the dataset to individuals with at least one acceptable
blow from the first visit [27, 29, 8].

We randomly partitioned patients with valid data entries for all phenotype
labels and clinical features into distinct training and validation datasets.

##  3 Experimental Results

We present LLM risk prediction performance across eight disease classification
tasks under varied experimental settings. We begin by assessing the
effectiveness of zero-shot and few-shot prompting as well as soft-prompt
tuning on a pre-trained foundation model using health data serialized into
text. We then demonstrate that directly mapping quantitative features into the
LLM’s latent token space significantly improves model performance. Finally,
using asthma as a case study, we show that this mapping procedure generalizes
to high-dimensional lung function data.

###  3.1 Quantifying disease risk using zero-shot, few-shot, and soft-prompt
tuning

We first establish a baseline for LLM disease risk prediction using zero-shot,
few-shot, and soft-prompt tuning with a frozen Flan-PaLMChilla 62b model [7].
We define classification tasks using a diverse set of binary phenotype targets
(see Section 2.3). For each task and prompting method, we evaluate a
“baseline” set of model inputs consisting of age, sex, and BMI as well as an
“expanded” set that includes eleven additional clinical and wellness features:
HDL cholesterol, LDL cholesterol, total cholesterol, total triglycerides,
diastolic blood pressure, smoking status, average sleep duration, insomnia,
snoring, daytime napping, and chronotype (i.e., whether a patient is a morning
or evening person). These predictors were chosen based on prior knowledge that
they should be informative about the selected targets.

We score a validation dataset ($n=3,000$) using the methodology outlined in
Section 2. For each validation sample, we generate a disease risk score by
computing the log probability of a positive disease label. Similarly, to
obtain the logistic regression and XGBoost scores, we compute the probability
of a positive disease label given the respective model fit on a separate
training set ($n=10,000$).

Table 1 shows performance for each prompting method and input set compared to
the baseline models. We observe that at least one prompting technique is
competitive with the baselines for most tasks. In some cases (e.g.,
hypertension and diabetes), zero-shot and few-shot models perform surprisingly
well despite seeing little to no training data compared to the baselines, an
observation also made by [16]. This suggests that the LLM uses prior knowledge
about the relationships between age, sex, BMI and disease likelihood.

Table 1: Comparison of AUC and AUPRC between LLM-based classifiers and
classical machine learning approaches on the validation set. The models with
“baseline” input features use age, sex and BMI as model features, while the
models with the “expanded” set also include 11 additional clinical and
wellness features. The mean AUC/AUPRC and the corresponding 95% confidence
intervals were calculated across 1,000 bootstrapping iterations. Bold cells
denote the best models for a given phenotype and input feature set, where
statistical significance is determined via paired bootstrapping. Logistic
regression and XGBoost models were trained on 10,000 samples, few-shot on 10
samples and soft-prompt tuning on 1,000 samples.

Phenotype Model Baseline AUC Expanded AUC Baseline AUPRC Expanded AUPRC All
Cause Mortality Zero-shot 0.61 (0.57–0.64) 0.61 (0.57–0.65) 0.10 (0.08–0.12)
0.11 (0.09–0.14) (prevalence = 6.83%) Few-shot 0.65 (0.61–0.69) 0.65
(0.61–0.69) 0.12 (0.10–0.15) 0.12 (0.09–0.15) Soft-prompt 0.69 (0.66–0.73)
0.69 (0.65–0.72) 0.13 (0.10–0.16) 0.14 (0.11–0.18) LogReg 0.71 (0.67–0.74)
0.68 (0.64–0.72) 0.15 (0.12–0.19) 0.16 (0.13–0.21) XGBoost 0.67 (0.62–0.70)
0.67 (0.62–0.70) 0.13 (0.10–0.17) 0.13 (0.10–0.17) Diabetes Zero-shot 0.70
(0.67–0.73) 0.61 (0.57–0.65) 0.18 (0.14–0.23) 0.12 (0.10–0.15) (prevalence =
7.60%) Few-shot 0.72 (0.69–0.76) 0.67 (0.64–0.70) 0.19 (0.15–0.24) 0.14
(0.11–0.17) Soft-prompt 0.72 (0.69–0.76) 0.68 (0.64–0.72) 0.23 (0.18–0.28)
0.17 (0.14–0.22) LogReg 0.74 (0.70–0.77) 0.75 (0.72–0.79) 0.23 (0.18–0.28)
0.26 (0.21–0.32) XGBoost 0.73 (0.69–0.76) 0.73 (0.69–0.76) 0.22 (0.17–0.26)
0.22 (0.17–0.26) Hypertension Zero-shot 0.70 (0.68–0.72) 0.68 (0.66–0.70) 0.60
(0.56–0.62) 0.57 (0.54–0.60) (prevalence = 40.03%) Few-shot 0.73 (0.71–0.75)
0.72 (0.70–0.73) 0.62 (0.59–0.64) 0.59 (0.56–0.62) Soft-prompt 0.72
(0.70–0.74) 0.72 (0.70–0.74) 0.60 (0.57–0.63) 0.59 (0.56–0.62) LogReg 0.74
(0.72–0.76) 0.74 (0.72–0.76) 0.63 (0.60–0.66) 0.65 (0.62–0.68) XGBoost 0.77
(0.75–0.78) 0.77 (0.75–0.78) 0.66 (0.63–0.69) 0.66 (0.63–0.69) Major
Depression Zero-shot 0.54 (0.51–0.57) 0.58 (0.55–0.61) 0.15 (0.13–0.17) 0.17
(0.14–0.19) (prevalence = 13.03%) Few-shot 0.50 (0.47–0.53) 0.55 (0.52–0.58)
0.14 (0.12–0.15) 0.15 (0.13–0.17) Soft-prompt 0.60 (0.57–0.63) 0.47
(0.44–0.50) 0.20 (0.17–0.23) 0.12 (0.11–0.14) LogReg 0.60 (0.57–0.63) 0.62
(0.59–0.65) 0.19 (0.16–0.22) 0.19 (0.17–0.22) XGBoost 0.54 (0.52–0.57) 0.54
(0.52–0.57) 0.16 (0.14–0.18) 0.16 (0.14–0.18) Migraine Zero-shot 0.51
(0.45–0.56) 0.49 (0.45–0.55) 0.05 (0.03–0.07) 0.04 (0.03–0.05) (prevalence =
3.77%) Few-shot 0.45 (0.40–0.50) 0.47 (0.42–0.53) 0.03 (0.03–0.04) 0.03
(0.03–0.04) Soft-prompt 0.64 (0.59–0.69) 0.51 (0.46–0.56) 0.07 (0.05–0.09)
0.04 (0.03–0.06) LogReg 0.62 (0.56–0.68) 0.62 (0.56–0.67) 0.07 (0.04–0.11)
0.06 (0.04–0.08) XGBoost 0.60 (0.54–0.66) 0.60 (0.54–0.66) 0.06 (0.04–0.07)
0.06 (0.04–0.07) Myocardial Infarction Zero-shot 0.61 (0.57–0.65) 0.61
(0.56–0.65) 0.08 (0.06–0.10) 0.09 (0.07–0.12) (prevalence = 5.93%) Few-shot
0.67 (0.63–0.71) 0.65 (0.61–0.68) 0.11 (0.09–0.14) 0.10 (0.08–0.12) Soft-
prompt 0.71 (0.67–0.74) 0.68 (0.64–0.71) 0.12 (0.10–0.15) 0.11 (0.09–0.14)
LogReg 0.71 (0.67–0.74) 0.70 (0.65–0.74) 0.12 (0.09–0.15) 0.16 (0.12–0.21)
XGBoost 0.69 (0.65–0.73) 0.69 (0.65–0.73) 0.14 (0.11–0.19) 0.14 (0.11–0.19)
Stroke Zero-shot 0.61 (0.56–0.66) 0.55 (0.49–0.60) 0.06 (0.05–0.09) 0.05
(0.04–0.06) (prevalence = 4.00%) Few-shot 0.65 (0.60–0.69) 0.60 (0.56–0.65)
0.07 (0.05–0.09) 0.05 (0.04–0.07) Soft-prompt 0.63 (0.58–0.68) 0.50
(0.45–0.55) 0.09 (0.06–0.12) 0.05 (0.03–0.06) LogReg 0.69 (0.64–0.74) 0.73
(0.68–0.77) 0.10 (0.07–0.14) 0.14 (0.09–0.19) XGBoost 0.68 (0.63–0.72) 0.68
(0.63–0.72) 0.09 (0.06–0.12) 0.09 (0.06–0.12)

Focusing on the baseline feature set (age, sex, and BMI), we aimed to
understand how the LLM derives scores. We estimated the importance of each
input feature by regressing the features against the scores output by the
model. Coefficients from the linear regression model are used to measure
feature importance and concordance across methods. Figure S1 shows the result
of this analysis for four traits. For diabetes, hypertension, and stroke, we
see concordance between logistic regression, XGBoost and the LLM models in
terms of direction and relative magnitude of effects. Additionally, we find a
strong correlation between logistic regression and LLM scores (Spearman =
0.46–0.93 across prompting methods), while the correlation between LLM and
XGBoost scores is weaker (Spearman = 0.39–0.65). On the other hand, for
migraine, we see little concordance in direction and relative magnitude of
effects for zero-shot and few-shot LLMs, indicating that the LLM doesn’t have
sufficient prior knowledge to relate migraine with the input features.
However, this is corrected in the soft-prompt case, where we see concordance
and high Spearman correlation between soft-prompt tuned LLM and logistic
regression (0.85). On the other hand, the soft-prompt tuned LLM has low
Spearman correlation with XGBoost (0.31), which is a non-linear model. Given
this, we hypothesize that the LLM is effectively scoring outcomes using what
translates to a simple linear function of the input features and that this
linear mapping is effectively learned via soft-prompting.

In the expanded feature set, we often find that the LLM with zero-shot
prompting has poorer performance (e.g., on diabetes) when compared with the
LLM with zero-shot prompting using the smaller feature set. This may indicate
that the LLM has insufficient prior knowledge about the features contained in
the expanded set. However, unlike the baseline features, the model is not able
to match baseline performance even when using soft-prompt tuning. This
indicates that for a complex set of input features, text serialization does
not yield a representation that fully captures the available signal. Together,
these results motivate the use of a multimodal approach to directly embed
quantitative data into the prompt.

###  3.2 Encoding quantitative data using HeLM

To assess the benefit of directly embedding quantitative data into the LLM’s
latent token space, we repeat the experiments from Section 3.1 but encode the
“extended” inputs using the HeLM framework. We learn an encoder $\phi_{t}$, an
MLP (two hidden layers of size 1024 and 4096, respectively), that takes as
input quantitative features and maps them into the embedding space. We train
this model over a mixture of all seven binary traits ($n=10,000$ for each
trait).

Table 2 shows performance metrics for HeLM compared with logistic regression
and XGBoost. We see that by directly encoding the tabular data into token
space, HeLM performs at parity with the best baseline methods and for all
cause mortality, hypertension and myocardial infarction HeLM outperforms the
best baseline. In comparing the scoring methods, we again find that HeLM
scores are highly correlated with logistic regression scores (Spearman =
0.70–0.87; Figure 4(a)-Figure 5(c)). In Figure S2, we repeat the feature
importance analysis, this time comparing HeLM with logistic regression and
XGBoost. Similar to Section 3.1, we see strong concordance between feature
weights, particularly for features that have the most importance. Taken
together, these results are consistent with our previous hypothesis that the
LLM scores outcomes in a way that is highly consistent with logistic
regression and that the LLM is arriving at these scores by similarly linearly
weighting input features.

In the case of hypertension, we see that HeLM significantly outperforms
logistic regression and XGBoost, while XGBoost appears to have a slight
advantage over logistic regression. In addition, the HeLM score is more
correlated with the XGBoost score when compared with logistic regression
(Spearman 0.87 vs. 0.76), which is not the case for any of the other traits.
Although, in no way conclusive, this may indicate that by mapping the tabular
data into the token embedding space, the LLM takes advantage of non-linear
relationships between input features and hypertension. This phenomenon may
account for some of the cases where HeLM outperforms the baseline methods.

We hypothesized that HeLM has an advantage over logistic regression due to
being trained over a mixture of traits and thus benefiting from transfer
learning across traits. To evaluate this, we selected three traits
(hypertension, all cause mortality and myocardial infarction) where HeLM
performed better than logistic regression in terms of AUROC. For each, we
trained a single task HeLM (a separate model for each trait) and compared with
the mixture. Overall, we see that the mixture trained HeLM does not have an
advantage over the single task model (Table S2). Training on a mixture of
diseases is still advantageous since it yields a single model that can be used
for a variety of diseases as opposed to separate models for each disease (as
is the case for logistic regression and XGBoost).

Table 2: Comparison of AUC and AUPRC between a HeLM, logistic regression and
XGBoost models on the validation set. The binary phenotypes are predicted from
the 14 “expanded set” input features from Table 1. The features are both
encoded as text as in Table 1 and also included as a secondary quantitative
data modality.

Phenotype Model AUC AUPRC All Cause HeLM 0.71 (0.68–0.75) 0.18 (0.14–0.23)
Mortality LogReg 0.68 (0.64–0.72) 0.16 (0.13–0.21) XGBoost 0.67 (0.62–0.70)
0.13 (0.10–0.17) Diabetes HeLM 0.77 (0.74–0.80) 0.27 (0.22–0.33) LogReg 0.75
(0.72–0.79) 0.26 (0.21–0.32) XGBoost 0.73 (0.69–0.76) 0.22 (0.17–0.26)
Hypertension HeLM 0.79 (0.77–0.81) 0.69 (0.66–0.72) LogReg 0.74 (0.72–0.76)
0.65 (0.62–0.68) XGBoost 0.77 (0.75–0.78) 0.66 (0.63–0.69) Major HeLM 0.63
(0.60–0.65) 0.19 (0.17–0.22) Depression LogReg 0.62 (0.59–0.65) 0.19
(0.17–0.22) XGBoost 0.54 (0.52–0.57) 0.16 (0.14–0.18) Migraine HeLM 0.61
(0.55–0.66) 0.06 (0.04–0.07) LogReg 0.62 (0.56–0.67) 0.06 (0.04–0.08) XGBoost
0.60 (0.54–0.66) 0.06 (0.04–0.07) Myocardial HeLM 0.73 (0.69–0.77) 0.15
(0.12–0.19) Infarction LogReg 0.70 (0.65–0.74) 0.16 (0.12–0.21) XGBoost 0.69
(0.65–0.73) 0.14 (0.11–0.19) Stroke HeLM 0.73 (0.68–0.77) 0.14 (0.09–0.20)
LogReg 0.73 (0.68–0.77) 0.14 (0.09–0.19) XGBoost 0.68 (0.63–0.72) 0.09
(0.06–0.12)

###  3.3 Estimating asthma risk using multiple modalities

Using HeLM to leverage tabular data clearly showed that this is a promising
direction for quantifying disease risk. Next, we evaluate whether HeLM can
incorporate more complex data modalities such as a spirogram, a time-series
curve which measures the amount of air an individual can breath in and out of
their lungs. Spirometry is commonly used to assess pulmonary function and the
presence of respiratory diseases. Thus, we focus on the task of quantifying
asthma risk.

To this end, we trained a HeLM model with three modalities as input: tabular
data (14 “expanded set” input features) as well as spirometry, along with
tabular data serialized to text. We do not include a textual description of
the spirogram since it’s unclear how to summarize this data in text. We encode
spirometry data into the token embedding space via a one-dimensional variant
of the ResNet18 architecture [14, 15], followed by an MLP (two hidden layers
of size 1024 and 4096, respectively). We use a pre-trained model from [8] for
the ResNet18 part of the encoder and only update the weights of the MLP. The
ResNet18 model was trained to predict asthma and COPD. We take as input the
128-dimensional embedding, corresponding to the penultimate layer. Similar to
Section 3.2, we use an MLP to encode tabular data into the token embedding
space.

The model is instructed to predict asthma using supervised labels (“yes” or
“no”) as targets in training. We trained on $n=16,724$ samples from the UK
Biobank, which were obtained by subsampling the dataset to achieve a one-to-
one case-control class distribution to ensure a balanced representation of
categories. As a direct baseline, we also trained a model that linearly
combined the 128-dimensional embedding from the ResNet18 model and tabular
data, which we term ResNet18 1D (tabular data $+$ spirogram).

In order to assess whether the multimodal HeLM model is leveraging the
additional spirogram modality for asthma prediction, we trained several models
on tabular data only, without the spirogram. Following Section 3.2 we trained
a logistic regression model, XGBoost, an LLM with soft-prompt tuning and HeLM
using tabular data only.

Figure 2: Inclusion of additional input modalities improves HeLM asthma
detection. (a) ROC and (b) Precision-recall (PR) for asthma phenotype
prediction using tabular data only versus tabular data and spirometry. We
compare HeLM, ResNet18 1D, logistic regression, XGBoost, and a soft-prompt
tuned LLM trained on either a single modality (tabular) or two modalities
(tabular and spirometry).

Figure 2 shows the model performance on a held-out set of individuals with
valid spirograms ($n=2,289$). We observe that the HeLM model trained on two
non-text modalities (tabular data and spirogram) is successfully leveraging
the additional spirogram modality to boost the performance on asthma
prediction. For example, the AUROC (AUPRC) increases from $0.49\pm 0.02$
($0.14\pm 0.01$) for the LLM trained on tabular data only via soft-prompt
tuning to $0.75\pm 0.01$ ($0.38\pm 0.02$) for HeLM trained on tabular data and
a spirogram. The comparison with soft-prompt tuned LLM on tabular data is
particularly important since it is the only other method that can also
generate natural language text and thus provide recommendations, answer
questions, and summarize given an individual’s health context. We also observe
that HeLM trained on tabular and spirogram data performs on par with the
linear combination of the ResNet18 1D model on spirogram and tabular data.

###  3.4 Using HeLM for out-of-distribution traits

Deep learning methods have shown significant performance degradation when
applied to out-of-distribution (OOD) data [26]. We investigated HeLM OOD
performance by applying it to a set of traits not in the training set.
Considering cataract, gastroesophageal reflux disease (GERD), hay fever eczema
(atopic eczema), osteoarthritis, and pneumonia, we compared HeLM with trait-
specific logistic regression models (Table 3). We observed that HeLM (not
trained on the trait) performs on par with logistic regression (trained on the
trait) in four out of five of the tested traits. Taken at face-value this
implies that HeLM is leveraging prior information about the impact of risk
factors on disease learned from related traits in the training data. However,
to truly understand how the model arrives at the OOD scores, will take
additional research.

Table 3: Comparison of AUC and AUPRC between zero-shot, few-shot LLM-based
classifiers, HeLM and a logistic regression model on the validation set in OOD
setting. While logistic regression models were trained on the data for the
trait, HeLM was not trained for the trait. The binary phenotypes are predicted
from the 14 “expanded set” input features from Table 1. The features are both
encoded as text as in Table 1 and also included as a secondary quantitative
data modality. “prev” denotes prevalence.

Phenotype Model AUC AUPRC Cataract Zero-shot 0.58 (0.55–0.61) 0.17 (0.15–0.20)
(prev=13.51%) Few-shot 0.63 (0.60–0.65) 0.18 (0.16–0.20) HeLM 0.71 (0.69–0.73)
0.24 (0.21–0.27) LogReg 0.72 (0.69–0.74) 0.27 (0.23–0.31) GERD Zero-shot 0.58
(0.55–0.60) 0.22 (0.19–0.25) (prev=17.33%) Few-shot 0.57 (0.54–0.60) 0.21
(0.18–0.23) HeLM 0.61 (0.59–0.64) 0.25 (0.22–0.28) LogReg 0.61 (0.59–0.64)
0.26 (0.23–0.29) Hay Fever Zero-shot 0.48 (0.46–0.51) 0.23 (0.21–0.24) Eczema
Few-shot 0.46 (0.44–0.49) 0.23 (0.21–0.25) (prev=23.84%) HeLM 0.47 (0.44–0.49)
0.22 (0.20–0.24) LogReg 0.57 (0.55–0.60) 0.28 (0.26–0.31) Osteoarthritis Zero-
shot 0.60 (0.58–0.62) 0.36 (0.33–0.39) (prev=28.13%) Few-shot 0.63 (0.61–0.65)
0.38 (0.35–0.40) HeLM 0.65 (0.63–0.67) 0.41 (0.38–0.44) LogReg 0.67
(0.65–0.69) 0.44 (0.40–0.47) Pneumonia Zero-shot 0.56 (0.52–0.60) 0.08
(0.06–0.10) (prev=6.50%) Few-shot 0.59 (0.55–0.63) 0.08 (0.07–0.10) HeLM 0.68
(0.65–0.72) 0.16 (0.12–0.21) LogReg 0.67 (0.63–0.71) 0.16 (0.12–0.20)

###  3.5 Natural language generation

Finally, we assess whether an LLM can incorporate multimodal data to tailor
conversational tasks. Approximating a two-stage approach in which a model is
trained to quantify risk and then trained to use these risk estimates in
conversational tasks, we first use HeLM to compute asthma risk from a
spirogram embedding and encode the predicted risk as text into the PaLM-2
model [13] for exercise recommendations: “I have a p% chance of having asthma.
What are some exercises you would suggest, given this information? Please
tailor your recommendations to my health status.” Over 110 examples that span
the asthma risk spectrum we observe differential recommendations based on
predicted asthma risk (Figure S3).

This approach is inefficient as it requires transforming risk predictions back
to the textual domain. Ideally, the LLM should condition recommendations
directly upon the embedded multimodal data. To this end, we qualitatively
explore the natural language generation capability of a HeLM model trained for
the asthma task: “Let’s think step by step. Given the following
spirogram:$\langle$spirogram$\rangle$, do they have asthma? Based on that what
is the most recommended exercise?”. A sample answer for an individual with
asthma is “low intensity aerobic exercise. The answer: yes.” and for an
individual without asthma “Swimming. The answer: no.”. We observe that while
the model gives sample exercise recommendations that are reasonable (e.g., a
low intensity exercise for someone with asthma), it also learned to include
yes/no in its answer. This is likely because it has associated our particular
question/answer format with the presence of multimodal tokens. The risk
prediction results demonstrated that the model can learn to operate on
spirograms in the token space, so we expect that including more diverse input
and output pairs and generative tasks may improve conversational ability and
plan to explore this in future work.

##  4 Discussion

Grounding LLMs in individual-specific information is required to create
personalized experiences across a large set of applications. Effective
application in health presents unique challenges owing to the high-dimensional
and multimodal nature of relevant input data that do not clearly map into
text. In this paper, we defined a framework (HeLM) for mapping non-text data
modalities into token embedding space and providing this information as
context for a foundation LLM to perform disease risk prediction. Using data
from the UK Biobank, we showed that HeLM is competitive with classic ML
approaches and that—in some cases—the LLM outperforms these methods. The
results highlight the promise of enabling LLMs to quantify underlying health
risks by leveraging complex multimodal health data.

While these results demonstrate the effectiveness of a multimodal approach,
there are many extensions to explore in future work. Though we have focused
solely on scalar lab values and high-dimensional lung function data, biobanks
contain individual-level data spanning a wide array of clinical modalities. An
immediate next step is exploring the effectiveness of simultaneously embedding
additional imaging data, such as fundus images or cardiac MRI, in a single
input prompt to better understand how jointly modeling such inputs impacts
predictive power. Additionally, this experimental setting motivates the study
of how an LLM handles missing data and whether the model could “impute”
predictive signal from various modality combinations. Furthermore, we have
only explored whether an LLM can learn to understand non-text data, but have
not assessed whether a similar approach can generate non-text-based outputs.

A major question that arises from this work is whether enabling LLMs to use
multimodal data for health risk prediction can improve their ability to
provide relevant personalized suggestions. To this end, we experimented with
using HeLM to offer health-related recommendations that are conditioned on an
understanding of risk. However, we found that conversational ability degraded
after model tuning. This is consistent with previous observations [35], though
there is some evidence that larger models are more robust to degradation [11].

In addition, it will be important for conversational agents to explain why
they placed individuals into high or low risk categories, and to quantify
their level of uncertainty. Alignment issues are also keenly important, as
some health-related topics are sensitive and complex to explain, lack expert
consensus for best practices, or are sensitive to the culture and preferences
of the user.

Finally, LLM-based models that have been trained on available health-related
data to quantify disease may show differential performance across demographic
groups due to computational, systemic, and human biases [34]. Evaluation and
mitigation of this is key to avoid perpetuating and increasing existing health
disparities. These are complex issues that we have not attempted to address in
this work, but will be extremely important to address before deploying
multimodal LLMs for health.

####  4.0.1 Acknowledgements

The authors would like to thank Katrin Tomanek for providing software,
inspiration, and know-how that influenced the direction of this work. We also
thank Ted Yun for helpful discussions and feedback.

## References

  * [1] Acosta, J.N., Falcone, G.J., Rajpurkar, P., Topol, E.J.: Multimodal biomedical AI. Nature Medicine 28(9), 1773–1784 (2022)
  * [2] Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.: Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems 35, 23716–23736 (2022)
  * [3] Alipanahi, B., Hormozdiari, F., Behsaz, B., Cosentino, J., McCaw, Z.R., Schorsch, E., Sculley, D., Dorfman, E.H., Foster, P.J., Peng, L.H., et al.: Large-scale machine-learning-based phenotyping significantly improves genomic discovery for optic nerve head morphology. The American Journal of Human Genetics 108(7), 1217–1230 (2021)
  * [4] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in Neural Information Processing Systems 33, 1877–1901 (2020)
  * [5] Bycroft, C., Freeman, C., Petkova, D., Band, G., Elliott, L.T., Sharp, K., Motyer, A., Vukcevic, D., Delaneau, O., O’Connell, J., et al.: The UK Biobank resource with deep phenotyping and genomic data. Nature 562(7726), 203–209 (2018)
  * [6] Chen, T., Guestrin, C.: XGBoost: A scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pp. 785–794. KDD ’16, ACM, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939785, http://doi.acm.org/10.1145/2939672.2939785
  * [7] Chung, H.W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., et al.: Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 (2022)
  * [8] Cosentino, J., Behsaz, B., Alipanahi, B., McCaw, Z.R., Hill, D., Schwantes-An, T.H., Lai, D., Carroll, A., Hobbs, B.D., Cho, M.H., et al.: Inference of chronic obstructive pulmonary disease with deep learning on raw spirograms identifies new genetic loci and improves risk models. Nature Genetics 55, 787–795 (2023)
  * [9] Diaz-Papkovich, A., Anderson-Trocmé, L., Ben-Eghan, C., Gravel, S.: UMAP reveals cryptic population structure and phenotype heterogeneity in large genomic cohorts. PLOS Genetics 15(11), e1008432 (2019)
  * [10] Dinh, T., Zeng, Y., Zhang, R., Lin, Z., Gira, M., Rajput, S., Sohn, J.y., Papailiopoulos, D., Lee, K.: Lift: Language-interfaced fine-tuning for non-language machine learning tasks. Advances in Neural Information Processing Systems 35, 11763–11784 (2022)
  * [11] Driess, D., Xia, F., Sajjadi, M.S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al.: Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 (2023)
  * [12] Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K.V., Joulin, A., Misra, I.: Imagebind: One embedding space to bind them all. arXiv preprint arXiv:2305.05665 (2023)
  * [13] Google: PaLM 2 technical report. arXiv preprint arXiv:2305.10403 (2023)
  * [14] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 770–778 (2016)
  * [15] He, T., Zhang, Z., Zhang, H., Zhang, Z., Xie, J., Li, M.: Bag of tricks for image classification with convolutional neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 558–567 (2019)
  * [16] Hegselmann, S., Buendia, A., Lang, H., Agrawal, M., Jiang, X., Sontag, D.: TabLLM: few-shot classification of tabular data with large language models. In: International Conference on Artificial Intelligence and Statistics. pp. 5549–5581. PMLR (2023)
  * [17] Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q., Sung, Y.H., Li, Z., Duerig, T.: Scaling up visual and vision-language representation learning with noisy text supervision. In: International Conference on Machine Learning. pp. 4904–4916. PMLR (2021)
  * [18] Kirk, H.R., Vidgen, B., Röttger, P., Hale, S.A.: Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback. arXiv preprint arXiv:2303.05453 (2023)
  * [19] Kline, A., Wang, H., Li, Y., Dennis, S., Hutch, M., Xu, Z., Wang, F., Cheng, F., Luo, Y.: Multimodal machine learning in precision health: A scoping review. npj Digital Medicine 5(1), 171 (2022)
  * [20] Lester, B., Al-Rfou, R., Constant, N.: The power of scale for parameter-efficient prompt tuning. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pp. 3045–3059 (2021). https://doi.org/10.18653/v1/2021.emnlp-main.243
  * [21] Li, J., Li, D., Savarese, S., Hoi, S.: BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 (2023)
  * [22] Lu, J., Clark, C., Zellers, R., Mottaghi, R., Kembhavi, A.: Unified-IO: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.08916 (2022)
  * [23] Moor, M., Banerjee, O., Abad, Z.S.H., Krumholz, H.M., Leskovec, J., Topol, E.J., Rajpurkar, P.: Foundation models for generalist medical artificial intelligence. Nature 616(7956), 259–265 (2023)
  * [24] OpenAI: GPT-4 technical report. arXiv preprint arXiv:2303.08774 (2023)
  * [25] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning. pp. 8748–8763. PMLR (2021)
  * [26] Recht, B., Roelofs, R., Schmidt, L., Shankar, V.: Do CIFAR-10 classifiers generalize to CIFAR-10? arXiv preprint arXiv:1806.00451 (2018)
  * [27] Sakornsakolpat, P., Prokopenko, D., Lamontagne, M., Reeve, N.F., Guyatt, A.L., Jackson, V.E., Shrine, N., Qiao, D., Bartz, T.M., Kim, D.K., Lee, M.K., Latourelle, J.C., Li, X., Morrow, J.D., Obeidat, M., Wyss, A.B., Bakke, P., Barr, R.G., Beaty, T.H., Belinsky, S.A., Brusselle, G.G., Crapo, J.D., de Jong, K., DeMeo, D.L., Fingerlin, T.E., Gharib, S.A., Gulsvik, A., Hall, I.P., Hokanson, J.E., Kim, W.J., Lomas, D.A., London, S.J., Meyers, D.A., O’Connor, G.T., Rennard, S.I., Schwartz, D.A., Sliwinski, P., Sparrow, D., Strachan, D.P., Tal-Singer, R., Tesfaigzi, Y., Vestbo, J., Vonk, J.M., Yim, J.J., Zhou, X., Bossé, Y., Manichaikul, A., Lahousse, L., Silverman, E.K., Boezen, H.M., Wain, L.V., Tobin, M.D., Hobbs, B.D., and, M.H.C.: Genetic landscape of chronic obstructive pulmonary disease identifies heterogeneous cell-type and phenotype associations. Nature Genetics 51(3), 494–505 (2019)
  * [28] Salemi, A., Mysore, S., Bendersky, M., Zamani, H.: Lamp: When large language models meet personalization. arXiv preprint arXiv:2304.11406 (2023)
  * [29] Shrine, N., Guyatt, A.L., Erzurumluoglu, A.M., Jackson, V.E., Hobbs, B.D., Melbourne, C.A., Batini, C., Fawcett, K.A., Song, K., Sakornsakolpat, P., Li, X., Boxall, R., Reeve, N.F., Obeidat, M., Zhao, J.H., Wielscher, M., Weiss, S., Kentistou, K.A., Cook, J.P., Sun, B.B., Zhou, J., Hui, J., Karrasch, S., Imboden, M., Harris, S.E., Marten, J., Enroth, S., Kerr, S.M., Surakka, I., Vitart, V., Lehtimäki, T., Allen, R.J., Bakke, P.S., Beaty, T.H., Bleecker, E.R., Bossé, Y., Brandsma, C.A., Chen, Z., Crapo, J.D., Danesh, J., DeMeo, D.L., Dudbridge, F., Ewert, R., Gieger, C., Gulsvik, A., Hansell, A.L., Hao, K., Hoffman, J.D., Hokanson, J.E., Homuth, G., Joshi, P.K., Joubert, P., Langenberg, C., Li, X., Li, L., Lin, K., Lind, L., Locantore, N., Luan, J., Mahajan, A., Maranville, J.C., Murray, A., Nickle, D.C., Packer, R., Parker, M.M., Paynton, M.L., Porteous, D.J., Prokopenko, D., Qiao, D., Rawal, R., Runz, H., Sayers, I., Sin, D.D., Smith, B.H., Artigas, M.S., Sparrow, D., Tal-Singer, R., Timmers, P.R.H.J., den Berge, M.V., Whittaker, J.C., Woodruff, P.G., Yerges-Armstrong, L.M., Troyanskaya, O.G., Raitakari, O.T., Kähönen, M., Polašek, O., Gyllensten, U., Rudan, I., Deary, I.J., Probst-Hensch, N.M., Schulz, H., James, A.L., Wilson, J.F., Stubbe, B., Zeggini, E., Jarvelin, M.R., Wareham, N., Silverman, E.K., Hayward, C., Morris, A.P., Butterworth, A.S., Scott, R.A., Walters, R.G., Meyers, D.A., Cho, M.H., Strachan, D.P., Hall, I.P., Tobin, M.D., Wain, L.V.: New genetic signals for lung function highlight pathways and chronic obstructive pulmonary disease associations across multiple ancestries. Nature Genetics 51(3), 481–493 (2019)
  * [30] Singhal, K., Azizi, S., Tu, T., Mahdavi, S.S., Wei, J., Chung, H.W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al.: Large language models encode clinical knowledge. arXiv preprint arXiv:2212.13138 (2022)
  * [31] Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Hou, L., Clark, K., Pfohl, S., Cole-Lewis, H., Neal, D., Schaekermann, M., Wang, A., Amin, M., Lachgar, S., Mansfield, P., Prakash, S., Green, B., Dominowska, E., Aguera y Arcas, B., et al.: Towards expert-level medical question answering with large language models. arXiv preprint arXiv:2212.13138 (2022)
  * [32] Steinberg, E., Jung, K., Fries, J.A., Corbin, C.K., Pfohl, S.R., Shah, N.H.: Language models are an effective representation learning technique for electronic health record data. Journal of Biomedical Informatics 113, 103637 (2021)
  * [33] Vestbo, J., Hurd, S.S., Agustí, A.G., Jones, P.W., Vogelmeier, C., Anzueto, A., Barnes, P.J., Fabbri, L.M., Martinez, F.J., Nishimura, M., Stockley, R.A., Sin, D.D., Rodriguez-Roisin, R.: Global strategy for the diagnosis, management, and prevention of chronic obstructive pulmonary disease. American Journal of Respiratory and Critical Care Medicine 187(4), 347–365 (2013)
  * [34] Vokinger, K.N., Feuerriegel, S., Kesselheim, A.S.: Mitigating bias in machine learning for medicine. Communications Medicine 1(1), 25 (2021)
  * [35] Wang, Y., Si, S., Li, D., Lukasik, M., Yu, F., Hsieh, C.J., Dhillon, I.S., Kumar, S.: Preserving in-context learning ability in large language model fine-tuning. arXiv preprint arXiv:2211.00635 (2022)
  * [36] Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al.: Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 (2022)
  * [37] Yang, K.D., Belyaeva, A., Venkatachalapathy, S., Damodaran, K., Katcoff, A., Radhakrishnan, A., Shivashankar, G., Uhler, C.: Multi-domain translation between single-cell imaging and sequencing data using autoencoders. Nature Communications 12(1), 31 (2021)
  * [38] Yang, X., Chen, A., PourNejatian, N., Shin, H.C., Smith, K.E., Parisien, C., Compas, C., Martin, C., Costa, A.B., Flores, M.G., et al.: A large language model for electronic health records. npj Digital Medicine 5(1), 194 (2022)
  * [39] Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., Wu, Y.: CoCa: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917 (2022)
  * [40] Zhou, H.Y., Chen, X., Zhang, Y., Luo, R., Wang, L., Yu, Y.: Generalized radiograph representation learning via cross-supervision between images and free-text radiology reports. Nature Machine Intelligence 4(1), 32–40 (2022)

##  Appendix 0.A Data Availability

Phenotypes and genotypes are available for approved projects through the UK
Biobank study (https://www.ukbiobank.ac.uk). This research has been conducted
under Application Number 65275.

##  Appendix 0.B Definitions of UK Biobank phenotypes

We restricted analyses to European ancestry individuals within the UK Biobank
to reduce phenotypic heterogeneity and limit the impact of population
structure. To define European ancestry, we first filtered to individuals with
self-reported “British” ancestry according to UKB field 21000. We then
computed the medoid of the British ancestry set in the 15-dimensional genetic
principal component (PC) space and calculated the distance of each individual
in the UK Biobank to this medoid. Finally, we constructed the “European” set
by selecting all individuals with a British-medoid distance of less than 40.
This cutoff is based on the 99th percentile of distances of individuals who
self-identified as British or Irish.

Table S1: The set of UK Biobank phenotypes and their Data-Field IDs.
International Classification of Disease version 9 (ICD-9) hospital inpatient
(HESIN) codes are taken from UKB field 41271 while International
Classification of Disease version 10 (ICD-10) general practitioner note codes
are taken from UKB field 42040. We define self-reported statuses according to
the target disease coding using UKB field 20002. If multiple label sources are
provided for a given phenotype (e.g., hypertension lists ICD-9, ICD-10, and
self-report codes), we perform a binary OR across label sources to determine
case-control status.

Phenotype UKB Data-Fields Age 21003 BMI 21001 Chronotype 1180 (“morning
people” responded with “definitely a” or “more of a” morning person) Daytime
napping 1190 (cases responded with “sometimes” or “usually”) Diastolic blood
pressure 4079 HDL cholesterol 30760 Insomnia 1200 (cases responded with
“sometimes” or “usually”) LDL cholesterol 30780 Sex 31 Sleep duration field
1160 Smoking status 20160 Snoring 1210 (cases responded with “yes”) Total
cholesterol 30690 Triglycerides 30870 All cause mortality 40000 Asthma ICD-9
code 493, ICD-10 codes J45 and J46, or self-report code 1111 Cataract ICD-10
codes H25 and H26 and fields 4700 and 131164–131167 Diabetes ICD-9 code 250,
ICD-10 codes E10–E14, self-report codes 1220, 1222, 1223, and fields 2443,
6153, and 6177 Gastro-oesophageal reflux (GERD) ICD-10 codes K20 and K21 and
field 131584 Hay fever and eczema ICD-10 codes L20-L30 and field 3761 Hyperte
nsion ICD-9 codes 401 and 405, ICD-10 codes I10 and I15, or self-report code
1065 Major depression ICD-10 codes F32 and F33 and field 20126 Migraine ICD-9
code 346, ICD-10 code G43, and self-report code 1265 Myocardial infarction
ICD-9 codes 410, 412, 4109, and 4129, ICD-10 codes I21, I252, and Z034, self-
report code 1075, and fields 6150, 131298, and 131299 Osteoarthritis ICD-10
codes M15–M19 and fields 131868, 131869, 131876, 131877, and 131870–131873
Pneumonia ICD-9 codes 480–484 and 486 and ICD-10 codes J12-J18 Stroke ICD-9
field 434.91, ICD-10 fields I63 and I64, and fields 6150, 131368, and 131369
Spirometry Field 3066 (preprocessed following the procedure outlined in [8])

##  Appendix 0.C LLM-based vs. logistic regression and XGBoost feature
importances

Figure S1: Comparison of feature weights between different scoring methods
across four phenotypes. For each disease risk score (i.e., Zero-shot, Few-
shot, Soft-prompt and logistic regression), we fit a linear regression to
predict the score given the original input features (i.e., age, sex, and BMI).
By comparing the regression coefficients we can get a sense of how much weight
is given to each input feature in the scoring method.

Figure S2: Comparison of feature weights between logistic regression, XGBoost
and HeLM.

##  Appendix 0.D Single task vs. mixture of tasks comparison

Table S2: Comparison of validation AUC and AUPRC across all cause mortality,
hypertension, and myocardial infarction between a HeLM model trained on a
mixture of all seven diseases and a HeLM model trained on only the target
disease. Models were trained on the extended feature set comprising 14
clinical and wellness features. The mean AUC/AUPRC and 95% confidence
intervals were calculated across 1,000 bootstrapping iterations. Bold cells
denote the best models for a given phenotype and input feature set, where
statistical significance is determined via paired bootstrapping.

Phenotype Model AUC AUPRC All Cause HeLM (Mixture) 0.71 (0.68–0.75) 0.18
(0.14–0.23) Mortality HeLM (Single Task) 0.71 (0.67–0.74) 0.17 (0.13–0.22)
Hypertension HeLM (Mixture) 0.79 (0.77–0.81) 0.69 (0.66–0.72) HeLM (Single
Task) 0.79 (0.78–0.81) 0.69 (0.66–0.73) Myocardial HeLM (Mixture) 0.73
(0.69–0.77) 0.15 (0.12–0.19) Infarction HeLM (Single Task) 0.72 (0.68–0.76)
0.16 (0.13–0.21)

##  Appendix 0.E Differential LLM recommendations based on predicted asthma
risk

Figure S3: Exercise recommendation frequency as a function of predicted asthma
risk.

##  Appendix 0.F Spearman’s rank correlation between different methods

(a) All cause mortality

(b) Diabetes

(c) Hypertension

(d) Major Depression

Figure S4: Spearman’s rank correlation between different methods’ scores
across (a) all cause mortality, (b) diabetes, (c) hypertension, and (d) major
depression.

(a) Migraine

(b) Myocardial infarction

(c) Stroke

Figure S5: Spearman’s rank correlation between different methods’ scores
across (a) migraine, (b) myocardial infarction, and (c) stroke.