# Rapidity distribution as a probe for elliptical flow at intermediate
energies

Sanjeev Kumar  Varinderjit Kaur  Suneel Kumar<EMAIL_ADDRESS>School
of Physics and Materials Science, Thapar University, Patiala-147004, Punjab
(India)

###### Abstract

Interplay between the spectator and participant matter in heavy-ion collisions
is investigated within isospin dependent quantum molecular dynamics (IQMD)
model in term of rapidity distribution of light charged particles. The effect
of different type and size rapidity distributions is studied in elliptical
flow. The elliptical flow patterns show important role of the nearby spectator
matter on the participant zone. This role is further explained on the basis of
passing time of the spectator and expansion time of the participant zone. The
transition from the in-plane to out-of-plane is observed only when the mid-
rapidity region is included in the rapidity bin, otherwise no transition
occurs. The transition energy is found to be highly sensitive towards the size
of the rapidity bin, while weakly on the type of the rapidity distribution.
The theoretical results are also compared with the experimental findings and
are found in good agreement.

###### pacs:

25.70.-z, 25.75.Ld

††preprint: APS/123-QED

##  I Introduction

Since last many years, investigation about the nuclear equation of state
(NEOS) at the extreme conditions of density and temperature has been one of
the primary driving forces in heavy ion studies at intermediate energies. The
interest in low energies, however, are for isospin effects in fusion process
Aich91 ; Kuma105 . These investigations has been performed with the help of
rare phenomena such as multifragmentation, collective flow, particle
production as well as nuclear stopping Kuma105 ; Stoc86 ; West935 ; Luka055 ;
Andr055 ; Luka045 ; Sood065 ; Chen065 ; Sing005 ; Saks105 ; Zhan065 ; Huan93 ;
Gyul835 . The relation between the nuclear EOS and flow phenomena has been
explored extensively in the simulations.

Recently the analysis of transverse-momentum dependence of elliptical flow has
also been put forwarded Luka055 ; Andr055 ; Dani00 . The elliptical flow is
shaped by the interplay between the geometry and mean field and, when gated by
the transverse momentum, reveals the momentum dependence of the mean field at
supra-normal densities. The parameter of the elliptic flow is quantified by
the second-order Fourier coefficient Volo965

$v_{2}~{}=~{}<cos2\phi>~{}=~{}\langle\frac{p_{x}^{2}-p_{y}^{2}}{p_{x}^{2}+p_{y}^{2}}\rangle,$
(1)

from the azimuthal distribution of detected particles at mid rapidity as

$\frac{dN}{d\phi}=p_{0}(1+2v_{1}cos\phi+2v_{2}cos2\phi+.....),$ (2)

where $p_{x}$ and $p_{y}$ are the ${\it x}$ and ${\it y}$ components of
momentum. The $p_{x}$ is in the reaction plane, while, $p_{y}$ is
perpendicular to the reaction plane and $\phi$ is the azimuthal angle of
emitted particles momentum relative to the x-axis. The positive values of
$<cos2\phi>$ reflect preferential in-plane emission, while negative values
reflect preferential out-of-plane emission. The pulsating of sign observed
recently at intermediate energies has received particular attention as it
reflects the increased pressure buildup in the non isotropic collision zone
Adle03 .
After the pioneering measurements at Saturne Demo90  and Bevalac Gutb89 , a
wealth of experimental results have been obtained at Bevalac and SIS Luka055 ;
Andr055 ; Luka045 ; Wang96  as well as at AGS Pink995 , SPS Adle03  and RHIC
Acke01 . In recent years, the FOPI, INDRA, and PLASTIC BALL Collaborations
Luka055 ; Andr055 ; Luka045  are actively involved in measuring the excitation
function of elliptical flow from Fermi energies to relativistic one. In most
of these studies, collisions of ${}_{79}Au^{197}~{}+~{}_{79}Au^{197}$ is
undertaken. Interestingly, elliptical flow was reported to change from
positive (in-plane) taken. negative (out-of-plane) values around 100
MeV/nucleon and maximum squeeze-out is observed around 400 MeV/nucleon. These
observations were reported recently by us and others theoretically Luka055 ;
Andr055 ; Zhan065 ; Dani00 . A careful analysis reveals that elliptic flow is
very sensitive towards the choice of rapidity cut which makes the difference
between spectator and participant matter.

The elliptical flow pattern of participant matter is affected by the presence
of cold spectators Dani00 . When nucleons are decelerated in the participant
region, the longitudinal kinetic energy associated with the initial colliding
nuclei is converted into the thermal and potential compression energy. In a
subsequent rapid expansion(or explosion), the collective transverse energy
develops Dani00  and many particles from the participant region get emitted
into the transverse directions. The particles emitted towards the reaction
plane can encounter the cold spectator pieces and, hence, get redirected. In
contrast, the particles emitted essentially perpendicular to the reaction
plane are largely unhindered by the spectators. Thus, for the beam energies
leading to rapid expansion in the vicinity of the spectators, elliptic flow
directed out of the reaction plane (squeeze-out) is expected. This squeeze-out
is related with the pace at which expansion develops, and is, therefore,
related to the EOS. This contribution of the participant and spectator matter
Dani00  in the intermediate energy heavy-ion collisions motivated us to
perform a detailed analysis of the excitation function of elliptical flow over
different regions of participant and spectator matter. If excitation function
is found to be affected by the different contributions it will definitely,
also affect the in-plane to out-of-plane emission i.e. transition energy.
The rapidity distribution is an important parameter to study the participant-
spectator contribution in the intermediate energy heavy-ion collisions Sood09
. In this paper, we will study the effect of participant and spectator matters
in term of different rapidity distributions on the excitation function of
elliptical flow. Attempts shall also be made to parameterize the transition
energy in term of different rapidity bins.
The entire work is carried out in the framework of isospin-dependent quantum
molecular dynamics (IQMD) Hart98 . The IQMD model is discussed in detail in
the following section.

##  II ISOSPIN-dependent QUANTUM MOLECULAR DYNAMICS (IQMD) MODEL

The IQMD model Saks105 ; Hart98 , which is an improved version of QMD model
Aich91  developed by J. Aichelin and coworkers, then has been used
successfully to various phenomena such as collective flow, disappearance of
flow, fragmentation & elliptical flow

Stoc86 ; Sood065 ; Sood09 ; Acke01 . The isospin degree of freedom enters into
the calculations via symmetry potential, cross-sections and Coulomb
interaction Saks105 ; Hart98 . The details about the elastic and inelastic
cross-sections for proton-proton and neutron-neutron collisions can be found
in Ref. Hart98 .
In IQMD model, the nucleons of target and projectile interact via two and
three-body Skyrme forces, Yukawa potential and Coulomb interactions. In
addition to the use of explicit charge states of all baryons and mesons, a
symmetry potential between protons and neutrons corresponding to the Bethe-
Weizsacker mass formula has been included.
The hadrons propagate using classical Hamilton equations of motion:

$\frac{d\vec{r_{i}}}{dt}~{}=~{}\frac{d\it{\langle~{}H~{}\rangle}}{d\vec{p_{i}}}~{}~{};~{}~{}\frac{d\vec{p_{i}}}{dt}~{}=~{}-\frac{d\it{\langle~{}H~{}\rangle}}{d\vec{r_{i}}},$
(3)

with

$\displaystyle\langle~{}H~{}\rangle$ $\displaystyle=$
$\displaystyle\langle~{}T~{}\rangle+\langle~{}V~{}\rangle$ (4)
$\displaystyle=$
$\displaystyle\sum_{i}\frac{p_{i}^{2}}{2m_{i}}+\sum_{i}\sum_{j>i}\int
f_{i}(\vec{r},\vec{p},t)V^{\it ij}({\vec{r}^{\prime},\vec{r}})$
$\displaystyle\times
f_{j}(\vec{r}^{\prime},\vec{p}^{\prime},t)d\vec{r}d\vec{r}^{\prime}d\vec{p}d\vec{p}^{\prime}.$

The baryon-baryon potential $V^{ij}$, in the above relation, reads as:

$\displaystyle V^{ij}(\vec{r}^{\prime}-\vec{r})$ $\displaystyle=$
$\displaystyle V^{ij}_{Skyrme}+V^{ij}_{Yukawa}+V^{ij}_{Coul}+V^{ij}_{sym}$ (5)
$\displaystyle=$
$\displaystyle\left(t_{1}\delta(\vec{r}^{\prime}-\vec{r})+t_{2}\delta(\vec{r}^{\prime}-\vec{r})\rho^{\gamma-1}\left(\frac{\vec{r}^{\prime}+\vec{r}}{2}\right)\right)$
$\displaystyle+~{}t_{3}\frac{exp(|\vec{r}^{\prime}-\vec{r}|/\mu)}{(|\vec{r}^{\prime}-\vec{r}|/\mu)}~{}+~{}\frac{Z_{i}Z_{j}e^{2}}{|\vec{r}^{\prime}-\vec{r}|}$
$\displaystyle+t_{6}\frac{1}{\varrho_{0}}T_{3}^{i}T_{3}^{j}\delta(\vec{r_{i}}^{\prime}-\vec{r_{j}}).$

Here $Z_{i}$ and $Z_{j}$ denote the charges of $i^{th}$ and $j^{th}$ baryon,
and $T_{3}^{i}$, $T_{3}^{j}$ are their respective $T_{3}$ components (i.e. 1/2
for protons and -1/2 for neutrons). The parameters $\mu$ and
$t_{1},.....,t_{6}$ are adjusted to the real part of the nucleonic optical
potential. For the density dependence of nucleon optical potential, standard
Skyrme-type parameterization is employed. The potential part resulting from
the convolution of the distribution function with the Skyrme interactions
$V_{\it Skyrme}$ reads as :

${\it
V}_{Skyrme}~{}=~{}\alpha\left(\frac{\rho_{int}}{\rho_{0}}\right)+\beta\left(\frac{\rho_{int}}{\rho_{0}}\right)^{\gamma}~{}~{}\cdot$
(6)

The two of the three parameters of equation of state are determined by
demanding that at normal nuclear matter density, the binding energy should be
equal to 16 MeV. The third parameter $\gamma$ is usually treated as a free
parameter. Its value is given in term of the compressibility:

$\kappa~{}=~{}9\rho^{2}\frac{\partial^{2}}{\partial\rho^{2}}\left(\frac{E}{A}\right)~{}~{}\cdot$
(7)

The different values of compressibility give rise to Soft and Hard equations
of state. It is worth mentioning that Skyrme forces are very successful in
analysis of low energy phenomena like fusion, fission and cluster-
radioactivity.
As noted Zhan02 , elliptical flow is weakly affected by the choice of equation
of state. On the other hand, in the refs. Sood065 ; Sood09 ; Mage00 , the hard
equation of state is used to study the directed as well as elliptical flow.
For the present analysis, a hard (H) equation of state has been employed along
with standard energy dependent cross-section.

##  III Results and Discussion

For the present analysis, simulations are carried out for thousand of events
for the reaction of ${}_{79}Au^{197}~{}+~{}_{79}Au^{197}$ at semi-central
geometry using a hard equation of state. The whole of the analysis is
performed for light charged particles (LCP’s)[1$\leq$ A $\leq$ 4]. The
reaction conditions and fragments are chosen on the basis of availability of
experimental data Luka055 ; Andr055 ; Luka045 . As noted in Ref. Lehm93 , the
relativistic effects do not play role at these incident energies and the
intensity of sub-threshold particle production is very small. The phase space
generated by the IQMD model has been analyzed using the minimum spanning tree
(MST) Sood065  method. The MST method binds two nucleons in a fragment if
their distance is less than 4 fm. This is the one of the widely used method in
the intermediate energy heavy-ion collisions. However, some improvements like
momentum and binding energy are also discussed in the literature Sood065 . In
recent years, more sophisticated and complicated algorithms are also available
in the literature saca . The entire calculations are performed at $t=200$
fm/c. This time is chosen by keeping in view the saturation of the collective
flow Sood065 .
As our purpose of present study is to understand the effect of participant-
spectator matter on the excitation function of elliptical flow, one will
concentrate on the rapidity distribution only Sood09 . The important concept
of spectators and participants in collisions was first introduced by Bowman et
al. Bowm73  and later employed for the description of a wide-angle energetic
particle emission by Westfall et al. West76 . The two nuclei slamming against
one other can be viewed as producing cylindrical cuts through each other. The
swept-out nucleons or participants (from projectile and target) undergo a
violent collision process. In the meantime, the remnants of the projectile and
target continue with largely undisturbed velocities, and are much less
affected by the collision process than the participant nucleons. On one hand,
this picture is supported by features of the data Andr055  and, on the other,
by dynamic simulations Gait00 . During the violent stage of a reaction, the
spectators can influence the behavior of participant matter. Following the
same, we are dividing the rapidity distribution in the different cuts in term
of $Y_{c.m.}/Y_{beam}$ parameter, which is given as:

$Y(i)=\frac{1}{2}~{}ln\frac{E(i)+P_{z}(i)}{E(i)-P_{z}(i)}$ (8)

where E(i) and $P_{z}(i)$ are respectively, the total energy and longitudinal
momentum of $i^{th}$ particle.
As we are interested in studying the effect of rapidity distribution on the
incident energy dependence of elliptical flow, one has to understand the
$dN/dY$ as a function of $Y_{c.m.}/Y_{beam}~{}=~{}Y^{red}$ at different
energies. For this, in Fig.1, we display the rapidity distribution for the
light charged fragments (LCP’s) at different incident energies. The rapidity
distribution is found to vary drastically throughout the range from
$|Y^{red}|~{}\leq~{}1.75$.

This is indicating the compressed or participant zone around $0$ value and
decay into spectator zone towards both side of the $0$ value. However around
$0$ value, the region between $-0.1$ to $0.1$ is specified as mid-rapidity
region in the Literature Andr055 ; Zhan065 . The decay from $-0.1$ towards
negative side approach to target like spectator, while, other side is known as
projectile like spectator. Interestingly, these regions are found to affect
drastically with incident energy. With increase in incident energy, number of
particles are increasing in the region from $-1.25$ to $1.25$, while
decreasing away from this region on both side. The inset in the figure is
showing the clear view of change in behavior around $-1.25$, which is also
true for other side also. The rapidity distribution is also used as
thermalization source in the literature many times Kuma105 . On the basis of
this, we have divided the rapidity distribution in five different zones out of
which three are with including $|Y^{red}|~{}<~{}0.1$(participant zone), while
other two are with excluding this region (target and projectile
spectator).They are as: (i) From -0.1 $\leq$ $Y^{red}$ $\leq$ 0.1 with
increment of 0.2 on both side upto -1.5 $\leq$ $Y^{red}$ $\leq$ 1.5. (ii) From
0 $\leq$ $Y^{red}$ $\leq$ 0.1 with increment of 0.2 on latter side upto 0
$\leq$ $Y^{red}$ $\leq$ 1.5. (iii) From -0.1 $\leq$ $Y^{red}$ $\leq$ 0 with
decrement of 0.2 the on former side upto -1.5 $\leq$ $Y^{red}$ $\leq$ 0\. (iv)
From $Y^{red}$ $\geq$ 0.1 with increment of 0.2 upto $Y^{red}$ $\geq$ 1.5. and
(v)From $Y^{red}$ $<$ -0.1 with decrement of 0.2 upto $Y^{red}$ $<$ -1.5.

Figure 1: The rapidity distribution dN/dY as a function of $Y_{c.m.}/Y_{beam}$
at different incident energies ranging from 50 to 1000 MeV/nucleon for light
charged particles (LCP’s)[1$\leq$ A $\leq$ 4]. The inset is showing only one
side of the rapidity distribution. Figure 2: The incident energy dependence of
elliptical flow for LCP’s collectively for projectile as well as target matter
including mid-rapidity region. The different lines are at different size of
the rapidity bin, which includes the participant as well as spectator matter.

Let us now understand the effect of these rapidity cuts on the excitation
function of elliptical flow. In Fig.2, we present the excitation function of
elliptical flow for the first condition. This condition is the mixture of the
participant as well as spectator zone from projectile as well as target
matter. There are two points to discuss here. One is the change in the
elliptical flow with incident energy and other is to see the effect of bin
size on elliptical flow. The elliptical flow evolves from a positive value,
rotational-like, emission to an negative value, collective expansion, with
increase in the incident energy. In other words, transition from the in-plane
to out-of-plane takes place. The energy at which this transition takes place
is known as transition energy. This transition is due to the competition
between the mean field at low incident energy and NN collisions at high
incident energies. The incident energy dependence of directed flow also show
such type of transitions from negative to positive value, which is know as
balance energy Sood065 . After this transition, the strength of collective
expansion overcomes the rotational like motion Wang96 . This leads to increase
of out-of-plane emission towards a maximum around 400 MeV/nucleon. This maxima
is further supported by the nuclear stopping at 400 MeV/nucleon Reis04 .

Beyond this energy, elliptical flow decreases indicating a transition to in-
plane preferentially emission Pink995 . This rise and fall behavior of the
elliptical flow in the expansion region is due to the variation in the passing
time $t_{pass}$ of the spectator and expansion time of the participant zone
Andr055 . In a simple participant spectator model,
$t_{pass}~{}=~{}2R/(\gamma_{s}v_{s})$, where R is the radius of the nucleus at
rest, $v_{s}$ is the spectator velocity in c.m. and $\gamma_{s}$ the
corresponding Lorentz factor. Due to the comparable value of the passing time
and expansion time in the collective expansion region up to 400 MeV/nucleon,
the elliptical flow results an interplay between fireball expansion and
spectator shadowing. In other words, due to the comparable size of two
times(the fireball expansion and spectator shadowing), the participant
particles which will come in the way of spectator shadowing are deflected
towards the out-of-plane and hence more squeeze out. However, in the energy
range from 400 MeV/nucleon to 1.49 GeV/nucleon, $t_{pass}$ decreases from 30
to 16 fm/c (not shown here), implying that overall the expansion gets about
two times faster in this energy region Andr055 . This is supported by the
average expansion velocities extracted from the particle spectra Wang96 . In
this case, corresponding to much shorter passing time compared to the
expansion time, the participant zone is affected by the shadowing of the
spectator at very early times, however no shadowing effect is observed at
later time where expansion of the participant matter is still happening. In
other words, the participant particles at later times are not blocked by the
shadowing of spectator and hence decrease is observed in the out-of-plane
emission after 400 MeV/nucleon.
On the other hand, with increase in the rapidity region, the transition energy
is found to affect to a great extent. In literature, the balance energy using
directed flow is also calculated, but was over the entire rapidity
Distribution Sood065 . The rapidity distribution affects many phenomena in
intermediate energy region like particle production, nuclear stopping Kuma105
and now the elliptical flow. The transition energy increases with rapidity
region between $|Y^{red}|~{}\leq~{}0.1$ and $|Y^{red}|~{}\leq~{}1.5$. With the
increase in the rapidity region, dominance of the spectator matter from
projectile as well as target takes place that will further result in the
dominance of the mean field up to higher energies. After the transition
energy, the collective expansion is found to have less squeeze out with an
increase in the rapidity region. In this region, the spectator zone
contribution along with participant zone starts to come in play. As we know,
the passing time for the spectator is very less compared to expansion time of
the participant zone, leading to the decreasing effect of the spectator
shadowing on the participant zone. The chances of the participant to move in-
plane increases with increase in the rapidity bin and hence less squeeze out
is observed with increase in the size of the rapidity bin. If one see
carefully, no transition is observed after $|Y^{red}|~{}\leq~{}1.1$. This is
due to negligible effect of the participant zone compared to the spectator
zone. The inset in the figure shows interesting results: the intersection of
all the rapidity bins takes place at a particular incident energy. Below and
above this energy, incident energy dependence of elliptical flow is changing
the behavior with rapidity distribution. One can have a good study of
elliptical flow below and above this particular incident energy with variation
in the rapidity distribution.

Figure 3:  Same as in fig.2, but the contribution for fragments is from the
projectile like matter only. All the bins includes mid-rapidity region. Figure
4: Same as in fig.2, but the contribution for fragments is from the target
like matter only. All the bins includes mid-rapidity region.

In Fig.2, we had displayed the effect on the participant as well as spectator
matter from the projectile as well as target collectively including the
distribution at $|Y^{red}|~{}<~{}0.1$. It becomes quite interesting to study
the participant and spectator matter contribution on the incident energy
dependence of elliptical flow for projectile as well as target matter
separately. For this, in Figs.3 and 4, the incident energy dependence of the
elliptical flow are displayed from the mid-rapidity to spectator zone for
projectile and target matter. Both of the figures are following the universal
behavior of the in-plane to out-of-plane emission with increase in incident
energy, as is displayed in Fig.2. On the other hand, the effect of rapidity
bins is also quite similar as seen in of Fig.2.

If one observes the maximum squeeze out values in Figs. 2-4 around 400
MeV/nucleon, it is found that less squeeze out is observed for projectile
matter and spectator matter, separately as compared to projectile and target
matter, collectively. However, it is almost same for the projectile or target
matter, separately. This is obvious as when rapidity bin is from 0 to 0.1 or
-0.1 to 0, then participant zone is less compared to the region -0.1 to 0.1.
This can be further clarify from the Fig.1, where rapidity distribution for
different bins is displayed. This is true for all the bins under
investigation. The effect of rapidity distribution on the transition energy
will be discussed later under different conditions.

Figure 5: Same as in fig.2, but the contribution for fragments is from the
projectile like matter only. All the bins excludes the mid-rapidity region. In
this case, we move from participant+spectator contribution towards spectator
contribution. Figure 6:  Same as in fig.2, but the contribution for fragments
is from the target like matter only. All the bins excludes the mid-rapidity
region. In this case, we move from participant+spectator contribution towards
spectator contribution.

One notices from the above figures that major contribution for elliptical flow
comes from the mid rapidity region. One is further interested to know the fall
of elliptical flow if the mid-rapidity region is excluded. For this, by
excluding the $|Y^{red}|~{}<~{}0.1$ region, we have displayed the incident
energy dependence for projectile and target matter from participant to purely
projectile or target spectator matter in Fig.5 and Fig.6, respectively. The
noted behavior is entirely different compared to previous three figures. The
differences are: (a) The behavior of rise and fall in the elliptical flow at a
particular incident energy is obtained as were in the previous figures, but,
no transition is observed from in-plane to out-of-plane emission. Over all the
incident energies, the value of elliptical flow remains positive, however,
some competition is observed at low incident energies around 150 MeV/nucleon.
This is due to the dominance of attractive mean field from the spectator
matter, which restricts the effects of participant zone.
(b) A higher value of in-plane flow is observed at high incident energies
compared to low incident energies upto $Y^{red}~{}\geq~{}0.7$ in Fig.5 for
projectile matter and $Y^{red}~{}\leq~{}-0.7$ for target matter, which is in
contrast to the previous figures. This is due to the effect that at low
incident energies, the passing time of spectator and expansion time of the
participant and comparable, while, with increase in the incident energy, the
passing time is found to be decrease as compared to expansion time. This will
reduce the shadowing effect at higher incident energies on the particles of
the participant zone, results in the enhanced in plane flow. Moreover, more
higher value of in-plane flow at higher incident energies as compared to lower
one is due to the absence of most compressible region from the region
$|Y^{red}|~{}\leq~{}-0.1$, which competes with the mean field of the spectator
to a great extent.
(c) With increase in the rapidity region from participant and spectator
($Y^{red}~{}\geq~{}0.1$ )to purely spectator matter ($Y^{red}~{}\geq~{}1.5$ ),
the dominance of in-plane flow takes place upto $Y^{red}~{}\geq~{}0.7$ and
after that suddenly fall in the in-plane flow takes place upto
$Y^{red}~{}\geq~{}1.5$ in Fig.5, which is also true for the Fig.6. The fall in
the in-plane flow after $Y^{red}~{}\geq~{}0.7$ can be explained with the help
of the Fig.1. After this region, the contribution of the participant is almost
negligible and size of the spectator also decreases. Due to the decreasing
size of the spectator matter, the violence of the incident energy as well as
short passing time of the spectator, they start to fly out-of-plane, which
were enjoying in-plane earlier due to their heavier size. Hence, minimum in-
plane flow is observed for $Y^{red}~{}\geq~{}1.5$ where only 5-7 particles
exist as is clear from the Fig.1. The same behavior is observed in the Fig.6.

Figure 7:  (Color online) Comparison of the results of incident energy
dependence of elliptical flow including different rapidity bins with the
experimental results of different collaborations Luka055 ; Andr055 . The left
hand side is for Z $\leq$ 2 particles, while, right hand side is for the
protons. Figure 8: The dependence of rapidity distributions on the transition
energy. The transition energies are extracted from Fig.2, 3and 4 for top,
middle and bottom panels, respectively. The numbering from 1to 7 is
representing the different rapidity bin from the respective figures mentioned
above. All the panels are parameterized with the straight line interpolation
$Y=mX~{}+~{}C$, where m is the slope, displayed in respective panels.

Going through all the aspects of rapidity distribution, it is important to
include the mid-rapidity region $|Y^{red}|~{}<~{}0.1$ in the study of
elliptical flow in heavy-ion collisions. In order to compare the findings with
the experimental one, one must have the transition from in-plane to out-of-
plane, which are observed in Figs.2-4. Moreover, more negative values are
observed in Fig.2 compared to other two one. From this discussion, it is
fruitful to compare the findings of Fig.2 with experimental findings of
different collaborations. This is displayed in Fig.7 for LCP’s (Z $\leq$ 2)
and protons. The curves have the similar trend as displayed in Fig.2.
Interestingly, it is observed that there is a competition in the rapidity bin
$|Y^{red}|\leq 0.1$ and $|Y^{red}|\leq 0.3$. After that systematic deviation
is observed in the elliptical flow values from the data values with increase
in the rapidity bin. There are some deviation in the middle region in case of
LCP’s between theory and data, while data is well explained for the protons.
From here, it is concluded that one can vary the mid rapidity region from the
$|Y^{red}|\leq 0.1$ to $|Y^{red}|\leq 0.3$ to get the better agreement with
the experimental one. However, the discrepancy with the data for LCP’s can be
reduced by varying the isospin-dependent cross sections.
Last, but not least, the rapidity distribution dependence of the transition
energy is displayed in the Fig.8. The top, middle and bottom panels are
representing the transition energies extracted from Fig.2-4, respectively. The
numbering from 1 to 7 in the figure is representing the bin size from the
respective figures. All the curves are fitted with straight line equation
$Y~{}=~{}mX~{}+~{}C$, where m is slope of line and C is a constant. The
transition energy is found to be sensitive towards the different bins of
rapidity distributions. It is observed that transition energy is found to
increase with the size of the rapidity bin for light charged particles. The
necessary condition for the transition energy is that the mid-rapidity region
must be included in the rapidity distribution bin. It is also observed that no
transition energy is obtained when the rapidity distribution region extends
away from $|Y_{red}|~{}\leq~{}1.1$. This is due to the dominance of the
spectator matter, which enjoys in-plane as compared to out-of-plane. The
transition energy is found to be weakly sensitive towards the choice of the
different type of rapidity distributions (projectile-target or projectile or
target) while we have included the mid-rapidity region. However, transition
energy is found to disappear when mid-rapidity region is excluded. In other
studies, system size dependence of the transition energy is also studied by us
and others for different kind of Fragments Zhan065 . Recently, we have studied
the fragment size dependence of transition energy under the influence of
different equations of state, nucleon-nucleon cross sections etc Sood065 .
These studies were made at a fixed rapidity bin. The detailed analysis of the
transition energy with rapidity distributions has revealed many interesting
aspects for the first time.

For more interest, in the near future, we are performing the comparative study
of different QMD and IQMD simulations. From the preliminary results, it is
observed that isospin content of the colliding partners $(N/Z~{}=~{}1.49)$ is
supposed to playing the appreciable role in the analysis of elliptical flow in
intermediate energy heavy-ion collisions in term of (i) closeness of the
results with experimental findings of different collaborations and (ii) effect
on the elliptical flow with change in the rapidity bins. The detailed analysis
of these findings will be presented in the near future.

##  IV Conclusion

Within the semi classical transport simulations of energetic semi central
collisions of ${}_{79}Au^{197}~{}+~{}_{79}Au^{197}$ reaction, we have carried
out a new investigation of the interplay between the participant and spectator
regions in term of rapidity distributions. The maxima and minima in the
incident energy dependence of elliptical flow is produced due to the different
contributions of passing time of the spectator and expansion time of the
participant. The shadowing of the spectator matter plays important role up to
later times due to the comparable magnitude of the passing and expansion time
up to energy 400 MeV/nucleon, however at high energies, shadowing effect is
dominant only at earlier times due to the shorter passing time as compared to
expansion time. The transition from in-plane to out-of-plane is observed only
when the mid-rapidity region is included in the rapidity bin, otherwise, no
transition is observed. The transition energy is found to be strongly
dependent on the size of the rapidity bin, while, weakly dependent on the type
of the rapidity distributions. The transition energy is parameterized with a
straight line interpolation. Comparison with experimental bin, reveals the
competition is observed between the rapidity bin of $|Y^{red}|~{}\leq~{}0.1$
and $|Y^{red}|~{}\leq~{}0.3$. To remove this discrepancy in the middle region
for LCP’s, one has to reduce the strength of nucleon-nucleon cross-section.

###### Acknowledgements.

This work has been supported by the Grant no. 03(1062)06/ EMR-II, from the
Council of Scientific and Industrial Research (CSIR) New Delhi, Govt. of
India.

## References

  * (1) J. Aichelin, Phys. Rep. 202, 233(1991);G. Peilert, H. Stocker, W. Greiner, A. Rosenhauer, A. Bohnet and J. Aichelen, Phys. Rev. C 39, 1402(1989).
  * (2) S. Kumar, S. Kumar and R. K. Puri, Phys. Rev. C 81, 014601 (2010); G. Lehaut et al., Phys. Rev. Lett. 104, 232701 (2010); S. Kumar and S. Kumar, Chin. Phys. Lett. 27, 062504 (2010). R. K. Puri, S. S. Malik and R. K. Gupta, Eur. Phys. Lett. 9, 767(1989); R. K. Puri and R. K. Gupta, J. Phys. G: Nucl. Part. Phys. 18, 903(1992); R. K. Gupta, M. Balasubramanium, R. K. Puri and W. Schied, J. Phys. G: Nucl. Part. Phys. 26, L23(2000). R. K. Puri and R. K. Gupta, Phys. Rev. C 45, 1837(1992); R. K. Puri, P. Chattopadhyay and R.K. Gupta, Phys. Rev. C 43, 315(1991).
  * (3) A. Le Fevre and J. Aichelin, Phys. Rev. Lett. 100, 042701 (2008); A. Le Fevre et al,Phys. Rev. C 80, 044615(2009); Y. K. Vermani, J. K. Dhawan, S. Goyal, R. K. Puri and J. Aichelin, J. Phys. G: Nucl. Part. Phys. 37, 015105(2010).
  * (4) G. D. Westfall et al., Phys. Rev. Lett. 71, 1986 (1993); M. B. Tsang et al., Phys. Rev. C 53, 1959 (1996); Y. M. Zheng, C. M. Ko, B. A. Li, and B. Zhang, Phys. Rev. Lett. 83, 2534 (1999); A. B. Larionov, W. Cassing, C. Greiner, and U. Mosel, Phys. Rev. C 62, 064611 (2000); B. A. Li, A. T. Sustich, and B. Zhang, ibid. 64, 054604 (2001); C. Alt et al., ibid.68, 034903 (2003).
  * (5) J. Lukasik, G. Auger, and M. L. Begemann-Blaich et al., Phys. Lett. B 608, 223 (2005).
  * (6) A. Andronic et al., Nucl. Phys. A 679, 765 (2001); Phys. Lett. B 612, 173 (2005).
  * (7) J. Lukasik, et.al., INDRA Collaborations, Int. Workshop on Multifragmentation and related topics (IWM 2003) Caen, France (2003).
  * (8) A. D. Sood and R. K. Puri, Phys. Rev. C 69, 054612 (2004); Eur. Phys. A 30, 571 (2006); A. D. Sood and R. K. Puri, Phys. Rev. C 70, 034611 (2004); S. Kumar, M. K. Sharma, R. K. Puri, K. P. Singh, and I. M. Govil, ibid. 58, 3494 (1998), A. D. Sood, R. K. Puri, and J. Aichelin, Phys. Lett. B 594, 260 (2004).S. Kumar and R. K. Puri, Phys. Rev. C 60, 054607 (1999); S. Kumar, S. Kumar, and R. K. Puri, Phys. Rev. C 78, 064602 (2008); Y. K. Vermani, S. Goyal, and R. K. Puri, ibid.79, 064613 (2009); R. Chugh and R. K. Puri, Phys. Rev. C 82, 014603 (2010).
  * (9) L. W. Chen and C. M. Ko, Phys. Lett. B 634, 205 (2006); Phys. Rev. C 73, 014906 (2006).
  * (10) J. Singh, S. Kumar, and R. K. Puri, Phys. Rev. C 62, 044617 (2000); S. Kumar and R. K. Puri, ibid.58, 1618 (1998).
  * (11) S. Gautam et al., J. of Phys. G 37, 085102 (2010).
  * (12) Y. Zhang and Z. Li, Phys. Rev. C 74, 014602 (2006).
  * (13) S. W. Huang, A. Faesseler, G. Q. Li, R. K. Puri, E. Lehmann, D. T. Khoa, and M. A. Matin, Phys. Lett. B 298, 41 (1993); G. Batko et al., J. Phys. G: Nucl. Part. Phys. 20, 461(1994).
  * (14) M. Gyulassy, K. A. Frankel, and H. Stöcker, Phys. Lett. B 110, 185 (1982).
  * (15) P. Danielewicz, Nucl. Phys. A673, 375 (2000). T. Z. Yan et al., Chin. Phys. lett. 26, 112501 (2009).
  * (16) S. Voloshin and Y. Zhang, Z. Phys. C 70, 665 (1996).
  * (17) C. Adler et al., Phys. Rev. Lett. 90, 032301 (2003).
  * (18) M. Demoulins et al., Phys. Lett. B 241, 476 (1990).
  * (19) H. H. Gutbrod et al., Phys. Lett. B 216, 267 (1989).
  * (20) S. Wang et al., Phys. Rev. Lett. 76, 3911 (1996); D. Brill et al., Z. Phys. A 355, 61 (1996); M.B. Tsang et al., Phys. Rev. C 53, 1959 (1996); N. Bastid et al., Nucl. Phys. A 622, 573 (1997); P. Crochet et al., Nucl. Phys. A 624, 755 (1997); G. Stoicea et al., Phys. Rev. Lett. 92, 072303 (2004).
  * (21) C. Pinkenburg et al., Phys. Rev. Lett. 83, 1295 (1999); A. Andronic et al., Eur. Phys. J A 30, 31 (2006).
  * (22) K.H. Ackermann et al., Phys. Rev. Lett. 86, 402 (2001).
  * (23) A. D. Sood and R. K. Puri, Phys. Rev. C 79, 064618 (2009).
  * (24) C. Hartnack et al., Eur. Phys. J. A1, 151 (1998).
  * (25) H. Y. Zhang et al., J. Phys. G: Nucl. and Part. 28, 2397 (2002).
  * (26) D. J. Magestro, W. Bauer, and G. D. Westfall, Phys. Rev. C 62, 041603(R) (2000); J. Singh and R. K. Puri, Phys. Rev. C 62, 054602 (2000); A. D. Sood and R. K. Puri, Phys. Rev. C 73, 067602 (2006).
  * (27) E. Lehmann et.al., Prog. Part. Nucl. Phys. 30, 219 (1993); Phys. Rev. C 51, 2113 (1995).
  * (28) R. K. Puri, Ch. Hartnack and J. Aichelin, Phys. Rev. C 54, R28 (1996). P. B. Gossiaux, R. K. Puri, Ch. Hartnack and J. Aichelin, Nucl. Phys. A 619, 379(1997). R. K. Puri and J. Aichelin, J. Comp. Phys. 162, 245(2000).
  * (29) J. D. Bowman, W. J. Swiatecki and C. F. Tsang, Lawrence Berkeley Laboratory Report No. LBL-2908, 1973 (unpublished).
  * (30) G. Westfall et al., Phys. Rev. Lett. 18, 1202 (1976).
  * (31) T. Gaitanos, H. H. Wolter, C. Fuchs, Phys. Lett. B 478, 79 (2000); J. Gosset, H. H. Gotbrod, W. G. Meyer, A. M. Poskanzer, A. Sandoval, R. Stock, and G. D. Westfall, Phys. Rev. C 16, 629 (1977).
  * (32) W. Reisdorf et al., Phys. Rev. Lett. 92, 232301 (2004).


# Explaining the Effectiveness of Multi-Task Learning for Efficient Knowledge
Extraction from Spine MRI Reports

Arijit Sehanobish, McCullen Sandora, Nabila Abraham,
Jayashri Pawar, Danielle Torres, Anasuya Das, Murray Becker,
Richard Herzog, Benjamin Odry, Ron Vianu
Covera Health, New York City, New York
{arijit.sehanobish, mccullen.sandora, nabila.abraham,
jayashri.pawar, danielle.torres, anasuya.das, rherzog,
murray.becker, benjamin.odry<EMAIL_ADDRESS>
Corresponding Author

###### Abstract

Pretrained Transformer based models finetuned on domain specific corpora have
changed the landscape of NLP. However, training or fine-tuning these models
for individual tasks can be time consuming and resource intensive. Thus, a lot
of current research is focused on using transformers for multi-task learning
(Raffel et al., 2020) and how to group the tasks to help a multi-task model to
learn effective representations that can be shared across tasks (Standley et
al., 2020; Fifty et al., 2021). In this work, we show that a single multi-
tasking model can match the performance of task specific models when the task
specific models show similar representations across all of their hidden layers
and their gradients are aligned, i.e. their gradients follow the same
direction. We hypothesize that the above observations explain the
effectiveness of multi-task learning. We validate our observations on our
internal radiologist-annotated datasets on the cervical and lumbar spine. Our
method is simple and intuitive, and can be used in a wide range of NLP
problems.

Figure 1: Figure showing how a report looks as it goes through our pipeline.

##  1 Introduction

Since the seminal work by (Vaswani et al., 2017), Transformers have become the
main architecture for almost all Natural Language Processing (NLP) tasks.
Self-supervised pretraining of massive language models like BERT (Devlin et
al., 2019) and GPT (Brown et al., 2020) has allowed practitioners to use these
large language models with little or no finetuning to various downstream
tasks. Multi-task learning (MTL) in NLP has been a very promising approach and
has shown to lead to performance gains even over task specific fine-tuned
models (Worsham and Kalita, 2020; Raffel et al., 2020; Aribandi et al., 2021).
However, applying these large pre-trained Transformer models to downstream
medical NLP tasks is quite difficult. Medical NLP has its unique challenges
ranging from domain specific corpora, noisy annotation labels and scarcity of
high quality labeled data. Despite these challenges, a number of researchers
and practitioners have successfully finetuned these large language models for
various medical NLP tasks. However, there is not much literature that uses
multi-task learning in medical NLP to classify and extract diagnoses from
clinical text (Peng et al., 2020; Crichton et al., 2017). Moreover, there is
almost no work in predicting spine pathologies from the radiologists’ notes
Azimi et al. (2020).

In this article, we are interested in extracting information from
radiologists’ notes on the cervical and the lumbar spine. In a given note, the
radiologist discusses the specific, and often multiple pathologies, present in
the medical images and grade their severity. Extracting relevant pathologies
from these reports can facilitate the creation of structured databases that
can be used for a number of downstream use-cases, such as cohort creation,
quality assessment and outcome tracking. Single-task learning for information
extraction in medical NLP has enjoyed much success in deep learning
(Kanakarajan et al., 2021).

However, an ultimate NLP system for a complete understanding of the medical
report must be able to perform many diverse information extraction and
classification tasks simultaneously and efficiently. Such a system can be
enabled by MTL, where one model shares weights across multiple tasks and makes
multiple inferences in one forward pass. Such networks can not only be trained
with limited resources, but are more scalable and deployable when compared to
several single-task models. Moreover, the shared features within these MTL
networks can induce more robust regularization and boost performance. Thus
there is a lot of interest in the academic and industry research communities
to understand when multi-task learning improves performance over single-
tasking models (Crawshaw, 2020), and how to group a diverse set of tasks to
encourage the model to learn a representation that can be shared across tasks
(Standley et al., 2020; Fifty et al., 2021; Bingel and Søgaard, 2017; Zamir et
al., 2020). Some of the aforementioned works, most notably in (Shui et al.,
2019), define a notion of task similarity via the Wasserstein distance and
show that a small Wasserstein distance between tasks aids in MTL.

This work is an extension of our earlier work (Sehanobish et al., 2022) where
we used parameter efficient MTL models to extract information from cervical
spine. In that work, we defined tasks as a conditional distribution over the
classes, and we attributed our success of MTL to smaller Wasserstein distance
between tasks. However, computing Wasserstein distance is expensive and
suffers from the curse of dimensionality (Cuturi, 2013), which requires the
number of samples to be significantly larger than the dimension of the
representation ($768$ for many transformer models) in order for the distance
to be accurately estimated. This prevents us from being able to estimate
Wasserstein distance for some of our minority classes, which have about $200$
examples. Even for majority classes where we have about 5k samples, our work
suffers from large error rates. Thus, to alleviate the above drawbacks, in
this work, we sought to use methods that are applicable to small data regimes
that lie in high dimensional space.

Inspired by the work of (Yu et al., 2020; Chen et al., 2020) and (Kornblith et
al., 2019), we hypothesize if the single-task models show similar
representations across their hidden layers and the task specific gradients are
aligned (see Definition 1 in Section $4.2$), the multi-task model can match or
outperform the task-specific, single-task models. We validate this hypothesis
on two multi-task settings on our internal datasets: (a) Four of the most
common pathologies in the cervical spine - central canal and foraminal
stenosis, disc herniation and cord compression, and (b) Three pathologies in
the lumbar spine - central canal stenosis, disc herniation and nerve root
impingement.

In this work, we (a) extend our novel pipeline to extract and predict the
severity of various pathologies in the lumbar and cervical spine at each
motion segment, (b) compute Central Kernel Alignment (CKA) and show similarity
between the transformer layers trained for individual tasks on a given
dataset, (c) compute dot products between the gradients of the task specific
loss functions with respect to various parameters and show that most of the
gradients flow along a similar trajectory and (d) show how to leverage that
information into a simple MTL framework allowing us to achieve significant
model compression during deployment and also speed up our inference without
sacrificing the accuracy of our predictions.

##  2 Datasets

We use an internal dataset consisting of radiologists’ MRI reports on the
cervical and the lumbar spine. Our dataset is heterogeneous and is diversely
sampled from a large number of different radiology practices and medical
institutions; the cervical MRI data consists of $1578$ reports from $97$
different radiology practices detailing various pathologies of the cervical
spine and our lumbar MRI data contains $2004$ reports from $170$ different
practices.

We annotate the cervical reports with the $4$ following pathologies: spinal
stenosis, disc herniation, cord compression, and neural foraminal stenosis,
and the lumbar reports with the $3$ pathologies: disc herniation, spinal
stenosis, and nerve impingement. Each of these pathologies is accompanied by
an indication of severity. In the cervical reports, the three categories for
the central canal stenosis are based on gradation; none/mild are not
clinically significant, moderate and severe definitions involve cord
compression or flattening. The moderate versus severe gradation refers to the
varying degrees of cord involvement. For disc herniation and central canal
stenosis, the categories are based on a continuous spectrum and it is a
standard practice in radiology for any continuous spectrum to be bucketed in
mild, moderate and severe discrete categories. Cord compression severity is
binary: compression/signal change versus none. This is because both cord
compression and signal change can cause symptoms, and are therefore clinically
relevant. Foraminal stenosis is treated as a binary task as well: severe
versus non-severe, as severe foraminal stenosis may indicate nerve
impingement, which is clinically significant. Similar considerations are taken
into account when annotating the lumbar reports. The splits and the details of
each category can be found in Table 1. The data distribution is highly
imbalanced, and about $25\%$ of these reports are OCR-ed, which leads to
additional challenges stemming from bad OCR errors.

Dataset | Pathology | Training Label Distribution | Test Label Distribution
---|---|---|---
Lumbar | Disc |  | None/Mild : 1885
---
Moderate : 1998
Severe : 456
| None/Mild : 1068
---
Moderate :1588
Severe :332
Stenosis |  | None/Mild : 3787
---
Moderate : 350
Severe : 202
| None/Mild : 2411
---
Moderate : 304
Severe : 273
Nerve |  | Normal : 3790
---
Abnormal : 549
| Normal : 2376
---
Abnormal : 612
Cervical | Disc |  | None/Mild : 2731
---
Moderate : 2699
Severe : 797
| None/Mild : 401
---
Moderate : 378
Severe : 101
Stenosis |  | None/Mild : 5488
---
Moderate : 561
Severe : 178
| None/Mild : 793
---
Moderate : 68
Severe : 19
Cord Compression |  | Normal : 5702
---
Abnormal : 525
| Normal : 806
---
Abnormal : 74
Neural Foraminal Stenosis |  | Normal : 5262
---
Abnormal : 965
| Normal : 789
---
Abnormal : 91

Table 1: Table showing statistics of our datasets

For a given report, each task is to predict the severity of a pathology for
each motion segment - the smallest physiological motion unit of the spinal
cord Swartz et al. (2005). Breaking information down at the motion segment
level in this way enables pathological findings to be correlated with clinical
exam findings, and can inform future treatment interventions.

Every report is tagged by annotators with labels for relevant pathologies and
severities, along with span information indicating which part(s) of the report
mentions each pathology. For example, in a report for the lumbar spine, the
sentence “L1-L2: There is no disc herniation. No spinal canal or foraminal
narrowing" would be given normal or $0$ class for each of the $3$ pathologies
(central canal stenosis, disc herniation and nerve root impingement).
Similarly in a cervical spine report, the sentence “ C2-3: Normal; no disc
herniation or bulge. No central canal stenosis or neuroforaminal narrowing"
would be given a normal or $0$ class for all the $4$ pathologies. An example
of a full radiology report can be found in Appendix A.

##  3 Workflow

In this section, we will briefly describe our pipeline. The reports are first
de-identified according to HIPAA regulations. Next, a Spacy (Honnibal et al.,
2020) parser is used to break the report into sentences.

A BERT based NER model which we call the _report segmenter_ is then used to
identify the motion segment(s) referenced in each sentence, and all the
sentences containing a particular motion segment are concatenated together.
This report segmenter has been shown to achieve an F1 score of $.9$ on our
internal datasets, and the same model is common across both the lumbar and the
cervical datasets. More details about the NER model and the hyperparameters
used to train it can be found in Appendix B and C. All pathologies are
predicted using the concatenated text for a particular motion segment.
Finally, the severities for each pathology are modeled as multi-label
classification problem, and a pre-trained transformer is finetuned using the
text for each motion segment.

For more details about our pipeline and data processing, please see Appendix
B. Figure 1 breaks down how a report looks as it is processed through our
spine pipeline.

Figure 2: CKA between activation matrices between different finetuned single-
task models. The top 2 rows are single-task models trained to predict specific
pathologies from cervical dataset and the bottom row for the lumbar dataset.
The y-axis is chosen to be between the min and the max values, i.e. in the
interval (.86, 1.0)

##  4 Similarity of Representations between Task Specific Models

In this section we will describe our methodology to understand the similarity
between the representations of various single task models. For all the
experiments in this section, we use the PubMedBERT Gu et al. (2020) as the
backbone.

###  4.1 Central Kernel Alignment

We use the linear Central Kernel Alignment (CKA), introduced in (Kornblith et
al., 2019). CKA is a scalar similarity index that can be used to compare
representations within and across neural networks. (Linear) CKA can be defined
by the following: Given $N$ examples and two activation outputs on these
examples, $R_{1}\in\mathbb{R}^{N\times d_{1}}$ and
$R_{2}\in\mathbb{R}^{N\times d_{2}}$,

$\text{CKA}(R_{1},R_{2})=\frac{||R^{\top}_{1}R_{2}||_{F}}{||R^{\top}_{1}R_{1}||_{F}||R^{\top}_{2}R_{2}||_{F}}$
(1)

where $||\cdot||_{F}$ is the Frobenius norm.

It is widely believed that similar representations lead to similar
performances on downstream tasks (Nguyen et al., 2021). In this work, we
compare the representations learned by various single tasking models. For two
single task models trained on a specific part of a spine, the CKA between the
matrix of activations for each layer of the corresponding models is computed.
For illustration purposes, we collect all the CKA values for various
activation matrices in a given layer and plot them in a box plot, as shown in
figure 2. We observe that for various tasks on both cervical and lumbar spine,
all layers of the task specific models learn similar representations.

Additional results on comparing models from the tasks from the lumbar dataset
and the cervical dataset can be found in Appendix D.

However, the high value of CKA may also be attributed to the following factors
: (i) larger and deeper networks converge to similar solutions (Morcos et al.,
2018) and (ii) CKA values do not change drastically when models start from
pretrained weights and are only trained for a few epochs (Mirzadeh et al.,
2021).

Thus in addition to the above analysis of the activations with the CKA, in the
next subsection we look at the gradient level information to understand the
trajectory of the task specific learned activations.

###  4.2 Gradient Alignment

There has been a lot of work in understanding the task specific gradients in
the context of MTL. Given tasks $T_{1},\cdots T_{n}$ (for example, they can be
classification tasks), one can define $n$ loss functions $\mathcal{L}_{T_{j}}$
for each task $T_{j}$. In our work, all loss functions are cross-entropy
losses. Then the task specific gradients are defined to be
$\nabla_{\theta_{j}}\mathcal{L}_{T_{j}}$ where $\theta_{j}$ are the parameters
of the task specific model. More specifically, it is shown in (Chen et al.,
2018), that MTL is competitive with single task learners when the norms of the
task specific gradients have similar magnitudes. However in (Yu et al., 2020;
Javaloy and Valera, 2021), the authors show that the direction of the gradient
flow is more important than the magnitude for the success of MTL. More
precisely, Theorem 1 in (Yu et al., 2020), shows that the multitask objective
converges to the optimum of one of the tasks or a sub-optimal minima in the
presence of conflicting gradients. Furthermore, authors in (Javaloy and
Valera, 2021) use a synthetic toy example to show the difficulties of
optimizing a multi-task loss in the presence of conflicting gradients.

Inspired by the above works, we define the following:

###### Definition 1

Two gradient vectors $g_{i}$ and $g_{j}$ are aligned if $g_{i}\cdot g_{j}>0$,
i.e. the vectors are pointing in the same direction.

To show that the gradients get more aligned as models are trained, we store
the gradients for all the parameters for all the mini-batches after every
epoch. We then compute the dot products between the corresponding gradients
for two tasks. We observe that as the task specific models gets trained, an
overwhelming proportion of these gradients are aligned (see Table 2). To
illustrate our findings, we take the proportion of these aligned parameters in
a given layer and plot them using a box plot in Figure 3. Finally, we compute
the proportion of weights across all layers for which the gradients are
aligned which we call the Average Proportion of Aligned Gradients (APAG).

$\text{APAG}=\frac{1}{N}_{\text{layers}}\frac{1}{N}_{\text{heads}}\sum_{\text{\it{layers}}}\sum_{\text{\it{heads}}}\theta\left(g_{i}\cdot
g_{j}\right)$ (2)

where $\theta(x)$ is the Heaviside step function. This is a scalar value that
summarizes the box plot and we show the progression of alignment of the
gradients as training progress and the end of the training (Table 2 and Table
6 in Appendix D respectively). Note that, in the above formula, the token
embedding layer is included in the computation and it is assumed to have $1$
head.

Dataset | Task Comparisons | Epoch 1 | Epoch 2 | Epoch 3 | Epoch 4
---|---|---|---|---|---
Cervical | Cord-Stenosis | .46 | .67 | .75 | .81
Cord-Disc | .37 | .52 | .69 | .74
Cord-Foraminal | .49 | .61 | .77 | .83
Disc-Stenosis | .51 | .62 | .69 | .78
Disc-Foraminal | .47 | .59 | .65 | .73
Foraminal-Stenosis | .54 | .66 | .72 | .79
Lumbar | Disc-Stenosis | .44 | .53 | .59 | .68
| Nerve-Stenosis | .51 | .57 | .66 | .73
| Disc-Nerve | .48 | .55 | .63 | .71

Table 2: Results showing the Average Proportion of Aligned Gradients between
various task specific models at various epochs.

To summarize: The task specific models not only show similar representations
but they arrive at these representations by moving in a similar direction
after starting from the pretrained weights. We would also like to point that
we observe similar behavior when we run our experiments with the BERT (Devlin
et al., 2019) and the Clinical BERT (Alsentzer et al., 2019) models.

Figure 3: Box Plot showing the proportion of aligned gradients between various
task specific models, after training. The top 2 rows are single tasking models
trained to predict specific pathologies from the cervical dataset and the
bottom row for the lumbar dataset. The y-axis is chosen to be between the min
and the max values, i.e. in the interval (.38, .725).

##  5 Results on Multi-Task Models

In this section, we give empirical evidence on the success of MTL for our
datasets. The results shown in this section are from our test set.

For our classification task, the PubMedBERT model is used as the backbone.
This BERT model is finetuned on the the cervical tasks resulting in $4$ task-
specific BERT sequence classifier models which provides our baseline results.
For the lumbar dataset, the PubMedBERT model is finetuned on the $3$
classification tasks resulting in $3$ task-specific BERT sequence classifier
models.

Now, instead of finetuning the task specific models for extracting various
pathology information from the cervical spine dataset, $4$ classifier heads
(i.e. $4$ linear layers) are added to a single PubMedBERT model to create an
output layer of shape $[3,3,2,2]$, where the first $3$ outputs correspond to
the logits for the stenosis severity prediction, the next $3$ for the disc
severity, the next $2$ for the cord severity and the final $2$ logits for the
foraminal severity. For the lumbar dataset, $3$ classifier heads are added to
the PubMedBERT model to create an output layer of shape $[3,3,2]$, where the
first $3$ outputs correspond to the logits for the stenosis severity
prediction, the next $3$ for the disc severity, and the final $2$ logits for
the nerve severity.

For the experiments, with both the datasets, a dropout of $.5$ is added to the
BERT vectors before passing them to the classifier layers. Each of these
classifier heads is trained with a cross entropy loss with the predicted
logits and the ground truth targets. All the losses are added up which allows
the gradients to backpropagate through the whole model and train these
classifier heads jointly.

The results for our experiments are shown in Table 3 for the lumbar dataset
and Table 4 for the cervical dataset.

Backbone | Model | Disc | Stenosis | Nerve
---|---|---|---|---
| BERT
---
BASE
| Baseline
---
(single tasker)
$.78\pm.03$ | $.79\pm.02$ | $.8\pm.03$
| Multi-Tasking | $.77\pm.02$ | $.78\pm.01$ | $.79\pm.02$
| CLINICAL
---
BERT
| Baseline
---
(single tasker)
$.81\pm.03$ | $.83\pm.02$ | $.82\pm.03$
| Multi-Tasking | $.83\pm.02$ | $.8\pm.04$ | $.81\pm.02$
| MSR PubMedBERT
---
| Baseline
---
(single tasker)
$.82\pm.03$ | $.83\pm.03$ | $.81\pm.04$
| Multi-Tasking | $\mathbf{.84\pm.01}$ | $\mathbf{.84\pm.03}$ | $\mathbf{.86\pm.04}$

Table 3: Table showing the macro F1 scores over 5 trials of our Baseline and
Multi-Tasking Models on the Lumbar Dataset.

For fair comparisons, we also conduct experiments with the BERT base and the
Clinical BERT models as well. We notice that the PubMedBERT produces slightly
better results than both the Clinical BERT and the BERT base. We believe this
is due to the fact that the vocabulary for PubMedBERT is tailored for clinical
text, unlike that of Clinical BERT, which uses the same vocabulary as that of
BERT.

Backbone | Model | Stenosis | Disc | Cord | Foraminal
---|---|---|---|---|---
| BERT
---
BASE
| Baseline
---
(single tasker)
$.62\pm.03$ | $.64\pm.03$ | $.70\pm.03$ | $.79\pm.03$
| Multi-Tasking | $.62\pm.02$ | $.65\pm.03$ | $.72\pm.02$ | $.78\pm.01$
| CLINICAL
---
BERT
| Baseline
---
(single tasker)
$.64\pm.05$ | $.66\pm.02$ | $.71\pm.02$ | $.82\pm.01$
| Multi-Tasking | $.63\pm.02$ | $.67\pm.01$ | $.75\pm.01$ | $.79\pm.03$
| MSR PubMedBERT
---
| Baseline
---
(single tasker)
$.66\pm.03$ | $.68\pm.04$ | $\mathbf{.73\pm.05}$ | $\mathbf{.84\pm.01}$
| Multi-Tasking | $\mathbf{.67\pm.01}$ | $\mathbf{.69\pm.01}$ | $.72\pm.04$ | $.83\pm.03$

Table 4: Table showing the macro F1 scores over 5 trials of our Baseline and
Multi-Tasking Models on the Cervical Dataset.

The hyperparameters and other training and implementation details can be found
in Appendix C.

##  6 Deployment

We deploy our spine pipeline system on an AWS p3.2x machine with a single
NVIDIA V100 GPU. Reports are passed through the pipeline daily and first go
through the report segmenter which tags sentences belonging to our set of
motion segments. Post-processing is done per report to aggregate sentences
belonging to each motion segment group and to filter out any reports that do
not contain motion segments. Each grouping of motion segments is individually
classified through our MTL model to predict a severity class per pathology.
Both the report segmenter and the multi-tasking model are processed in batch
mode with latencies of 31ms/report and 56ms/report, respectively. Compared to
single pathology models, we observe a 3x improvement in latency per study when
using the MTL pathology model. The spine pipeline is routinely evaluated in an
offline setting for studies that do not produce any motion segment groupings
or fail to capture any sentences for a given motion segment, per report. Our
current deployment only supports the lumbar reports and we are in the process
of extending our deployment to also support the cervical pathologies.

##  7 Conclusion and Future Work

In this work, a simple multi-tasking model is presented that is competitive
with task specific models. Instead of training and deploying task specific
models, only one model is trained and deployed. This allows us to save
significant costs during training and faster inference during deployment while
achieving significant model compression, without any loss in the quality of
performance. Our work opens the possibility of using multi-tasking models to
extract information over various different body parts, allowing users to
leverage large transformer models using limited compute resources.

Our novel pipeline is one of the very few works that attempts to extract
pathologies and their severities from a heterogeneous source of radiologists’
notes on lumbar and cervical spine MRIs at the level of motion segments. These
findings suggest that our approach may not only be more widely generalizable
and applicable, but also more clinically actionable.

We believe our analysis with CKA and gradient alignment sheds more light on
the success of MTL. This insight has led to our process change from single-
task BERT based models to a more cost-effective MTL system. Our analysis is
widely applicable for other datasets and tasks.

It is tempting to ask if one can use one multi-task model for both the lumbar
and the cervical datasets. This is a work in progress and we have found strong
similarity between single task models in the two datasets (most notably
between the lumbar disc and the cervical disc models and the lumbar stenosis
and the cervical stenosis models). However, unlike in the above analysis, we
see low CKA scores between various other task specific models which may make
MTL difficult (see Appendix D). We are in the process of using our analysis,
along with insights borrowed from (Standley et al., 2020; Yu et al., 2020) to
either group tasks from the two datasets or align different task-specific
gradients to create an efficient learner.

The biggest drawback of our work is the limited amount of data on which our
observations are verified. We are actively addressing this issue as we
annotate more reports concerning various pathologies in different body parts.

## Ethical Considerations

Because of legal and institutional concerns arising from the sensitivity of
clinical data, it is difficult for the NLP community to gain access to
relevant data except for MIMIC (Johnson et al., 2016). Despite its large size
(covering over $58k$ hospital admissions), it is only representative of
patients from a particular clinical domain (the intensive care unit) and
geographic location (a single hospital in the United States). Such a sample is
not representative of either larger population of patient admissions or other
geographical regions/hospital systems. We have tried to address the second
issue by collecting data across multiple practices in the US. However, it is
impossible to predict whether our models will generalize to the entire patient
population without actually evaluating on all the different radiology
practices. Thus we have to be extra careful about out-of-distribution data
since the actionable insights we generate from our models can be potentially
faulty and can lead to severe consequences.

Finally, we recognize the need to minimize ethical risks of AI implementation
which can include threats to privacy and confidentiality, informed consent,
and patient autonomy. We strongly believe that stakeholders should be
encouraged to be flexible in incorporating AI technology, most likely as a
complementary tool and not a replacement for a physician. Thus, we develop our
workflows, annotation guidelines and generate actionable insights by working
in conjunction with a varied group of radiologists and medical professionals.

## References

  * Alsentzer et al. (2019) Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019.  Publicly available clinical BERT embeddings.  In _Proceedings of the 2nd Clinical Natural Language Processing Workshop_ , pages 72–78, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
  * Aribandi et al. (2021) Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jai Gupta, Kai Hui, Sebastian Ruder, and Donald Metzler. 2021.  Ext5: Towards extreme multi-task scaling for transfer learning.
  * Azimi et al. (2020) Parisa Azimi, Taravat Yazdanian, Edward C. Benzel, Hossein Nayeb Aghaei, Shirzad Azhari, Sohrab Sadeghi, and Ali Montazeri. 2020.  A Review on the Use of Artificial Intelligence in Spinal Diseases.  _Asian Spine J_ , 14(4):543–571.
  * Bingel and Søgaard (2017) Joachim Bingel and Anders Søgaard. 2017.  Identifying beneficial task relations for multi-task learning in deep neural networks.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 164–169, Valencia, Spain. Association for Computational Linguistics.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Chen et al. (2018) Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. 2018.  Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks.  In _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018_ , volume 80 of _Proceedings of Machine Learning Research_ , pages 793–802. PMLR.
  * Chen et al. (2020) Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. 2020.  Just pick a sign: Optimizing deep multitask models with gradient sign dropout.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Crawshaw (2020) Michael Crawshaw. 2020.  Multi-task learning with deep neural networks: A survey.
  * Crichton et al. (2017) Gamal Crichton, Sampo Pyysalo, Billy Chiu, and Anna Korhonen. 2017.  A neural network multi-task learning approach to biomedical named entity recognition.  _BMC Bioinformatics_ , 18(1):368.
  * Cuturi (2013) Marco Cuturi. 2013.  Sinkhorn distances: Lightspeed computation of optimal transport.  In _Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States_ , pages 2292–2300.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Fifty et al. (2021) Christopher Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. 2021.  Efficiently identifying task groupings for multi-task learning.
  * Gu et al. (2020) Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2020.  Domain-specific language model pretraining for biomedical natural language processing.  ArXiv.
  * Honnibal et al. (2020) Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020.  spaCy: Industrial-strength Natural Language Processing in Python.
  * Javaloy and Valera (2021) Adrián Javaloy and Isabel Valera. 2021.  Rotograd: Gradient homogenization in multitask learning.
  * Johnson et al. (2016) Alistair E.W. Johnson, Tom J. Pollard, Lu Shen, Li-wei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. 2016.  Mimic-iii, a freely accessible critical care database.  _Scientific Data_ , 3(1):160035.
  * Kanakarajan et al. (2021) Kamal raj Kanakarajan, Bhuvana Kundumani, and Malaikannan Sankarasubbu. 2021.  BioELECTRA:pretrained biomedical text encoder using discriminators.  In _Proceedings of the 20th Workshop on Biomedical Language Processing_ , pages 143–154, Online. Association for Computational Linguistics.
  * Kornblith et al. (2019) Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton. 2019.  Similarity of neural network representations revisited.  In _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 3519–3529. PMLR.
  * Mirzadeh et al. (2021) Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Dilan Görür, Razvan Pascanu, and Hassan Ghasemzadeh. 2021.  Linear mode connectivity in multitask and continual learning.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Morcos et al. (2018) Ari S. Morcos, Maithra Raghu, and Samy Bengio. 2018.  Insights on representational similarity in neural networks with canonical correlation.  In _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_ , pages 5732–5741.
  * Nguyen et al. (2021) Thao Nguyen, Maithra Raghu, and Simon Kornblith. 2021.  Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  Pytorch: An imperative style, high-performance deep learning library.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 8024–8035.
  * Peng et al. (2020) Yifan Peng, Qingyu Chen, and Zhiyong Lu. 2020.  An empirical study of multi-task learning on BERT for biomedical text mining.  In _Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing_ , pages 205–214, Online. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.
  * Sehanobish et al. (2022) Arijit Sehanobish, Nathaniel Brown, Ishita Daga, Jayashri Pawar, Danielle Torres, Anasuya Das, Murray Becker, Richard Herzog, Benjamin Odry, and Ron Vianu. 2022.  Efficient extraction of pathologies from C-spine radiology reports using multi-task learning.
  * Shui et al. (2019) Changjian Shui, Mahdieh Abbasi, Louis-Émile Robitaille, Boyu Wang, and Christian Gagné. 2019.  A principled approach for learning task similarity in multitask learning.  In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019_ , pages 3446–3452. ijcai.org.
  * Standley et al. (2020) Trevor Standley, Amir Roshan Zamir, Dawn Chen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. 2020.  Which tasks should be learned together in multi-task learning? In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 9120–9132. PMLR.
  * Swartz et al. (2005) Erik E. Swartz, R. T. Floyd, and Mike Cendoma. 2005.  Cervical Spine Functional Anatomy and the Biomechanics of Injury due to Compressive Loading.  _Journal of athletic training_ , 40(3):155–161.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 5998–6008.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Worsham and Kalita (2020) Joseph Worsham and Jugal Kalita. 2020.  Multi-Task Learning for Natural Language Processing in the 2020s: Where are we going? _Pattern Recognition Letters_ , 136:120–126.
  * Yu et al. (2020) Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020.  Gradient surgery for multi-task learning.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Zamir et al. (2020) Amir Roshan Zamir, Alexander Sax, Nikhil Cheerla, Rohan Suri, Zhangjie Cao, Jitendra Malik, and Leonidas J. Guibas. 2020.  Robust learning through cross-task consistency.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020_ , pages 11194–11203. IEEE.

##  Appendix A Example of our Dataset

Figure 4: An example of a report from our Lumbar Dataset.

In this section, we will show some examples of lumbar and cervical reports
from our dataset.

Figure 5: An example of a report from our Cervical Dataset.

Hyperparameter Type |  | Single Tasking Models
---
on Cervical Dataset
| Multi-Tasking Models
---
on Cervical Dataset
| Single Tasking Models
---
on Lumbar Dataset
| Multi-Tasking Models
---
on Cervical Dataset
NER
Epochs | 5 | 12 | 6 | 11 | 5
Batch Size | 16 | 16 | 16 | 16 | 16
Sequence Length | 512 | 512 | 512 | 512 | 256
Optimizer | AdamW | AdamW | AdamW | AdamW | AdamW
Learning Rate | 2e-5 | 3e-5 | 2e-5 | 3e-5 | 1e-5
Weight Decay | 1e-4 | 1e-4 | 1e-4 | 1e-4 | 1e-3
Gradient Clip | 2 | 5 | 2 | 5 | 2
Early Stopping | Yes | Yes | Yes | Yes | Yes
Learning Rate Scheduler | Linear | Linear | Linear | Linear | Linear

Table 5: Hyperparameters used for all our experiments

##  Appendix B More Details about our Workflow

In this section, we give a more detailed description of our novel workflow.
Our main goal is to detect pathologies at the motion segment level from
radiologists’ MRI reports. The motion segments in the cervical reports that we
are interested in are C2-C3, C3-C4, C4-C5, C5-C6, C6-C7 and C7-T1 and the
motion segments of interest in the lumbar reports are L1-L2, L2-L3, L3-L4,
L4-L5 and L5-S1. We first make sure that the reports are de-identified and
then use a Spacy Honnibal et al. (2020) parser to break the report into
sentences. Then each sentence is tagged by annotators and they are given
labels of various pathologies and their severities if the sentence mentions
that pathology. To detect pathologies at a motion segment level, we use our
BERT based NER system to tag the locations present in each sentence. Our BERT
based NER model is a binary classifier model (Location Tag vs the Other Tag).
It is is trained on both lumbar and cervical MRI reports that can predict the
location tags in those reports. Our NER model achieves an F1 score of $.9$.

We then use an appropriate body part specific rule based system to group all
sentences to the correct motion segment. If a sentence does not explicitly
have a motion segment mentioned in it, we use a rule based method to assign
the sentence to one of the above mentioned motion segments or to a generic
category ”No motion segments found". Given the disparate source of our data
and due to typos and OCR errors, for example, L23, L2L3, L@L3, $L2\\_L3$ all
may refer to the motion segment L2-L3 and thus our systems are mindful of this
diversity of the clinical notes. Finally to use our BERT based models for
pathology detection on the level of motion segments for a given report, we
concatenate all sentences for a given motion segment and use the [CLS] token
for the segment that is used for the downstream classification task.

Since we are interested in predictions at the motion segment level, we do not
use the sentences that are grouped under ”No motion segments found" to train
the classifier models, nor do we evaluate our classifier models on those
sentences.

##  Appendix C Hyperparameters and Other Training Details

We create a validation set using $20\%$ of the samples of the training set
where the samples are drawn via stratified samples so the data distribution is
maintained across splits. The hyperparameters used for training the NER model
and various classification models can be found in Table 5.

PyTorch (Paszke et al., 2019) and the HuggingFace library (Wolf et al., 2020)
is used to conduct our experiments which are run on 1 NVIDIA V100 16GB GPU.

Figure 6: Box plot showing CKA scores between models trained on tasks on the
lumbar and the cervical dataset. The y-axis is chosen to be (0,.85). The
figure shows the low CKA scores between the cord and the nerve models and high
scores between the stenosis models and the disc models.

##  Appendix D Additional CKA Results and Gradient Alignment Results

In this section, we present some additional results on comparing
representations between our various models.

We present the average proportion of aligned gradients (APAG) at the end of
training in Table 6. We also compute the cosine similarity between the
gradients. We then take the average of them for a given layer, thus yielding a
scalar value per layer. This yields cosine similarity values which are over 90
% positive. For simplicity, we average those numbers to produce a scalar value
that measures the cosine similarity between the gradients of two models. Model
level statistics can be found in Table 6.

Dataset | Task Comparisons | Cosine Similarity |  | Average Proportion
---
of Aligned Gradients
Cervical | Cord-Stenosis | .013 | .89
Cord-Disc | .005 | .87
Cord-Foraminal | .011 | .91
Disc-Stenosis | .012 | .88
Disc-Foraminal | .007 | .87
Foraminal-Stenosis | .005 | .84
Lumbar | Disc-Stenosis | .008 | .75
| Nerve-Stenosis | .01 | .86
| Disc-Nerve | .002 | .86

Table 6: Results showing the Cosine Similarity and the Average Proportion of
Aligned Gradients between various task specific models after the end of
training.

Given some similarities between certain label spaces in the lumbar and the
cervical dataset (particularly for the disc herniation and the central canal
stenosis labels), we believe that some task specific models between tasks
across datasets may show similar representations. To validate this hypothesis,
we computed the CKA between lumbar stenosis and the cervical stenosis models
and the lumbar disc and the cervical disc models. The natural question is :
what happens to the single tasking models that are trained on label spaces
that are semantically different? Fig 6 shows low CKA scores between the cord
and the nerve models. This is an active work in progress to be able to group
similar tasks (Standley et al., 2020) to create a MTL framework that works for
both the cervical and the lumbar spine. Another future direction is to use
realign gradients using the techniques in (Yu et al., 2020). However to
realign the gradients, one has to save the entire computation graph after the
backward pass via loss.backward(retain_graph=True) which becomes a bottleneck
for large transformer models. To mitigate this issue, one can use parameter
efficient methods like adapters which we have shown to work in these MTL
settings in our previous work (Sehanobish et al., 2022).

##  Appendix E Annotation Process

All data are annotated by our team of inhouse annotators with clinical
expertise. All annotators are trained for the given task and provided clear
guidelines on the task and performance is measured periodically on a benchmark
set and feedback is provided.