Gindraft=false ]}

# Gold Standard for Expert Ranking:
A Survey on the XWiki Dataset

Matthieu Vergne  Center for Information and Communication Technology, FBK-ICT
Via Sommarive, 18 I-38123 Povo, Trento, Italy
<EMAIL_ADDRESS>Doctoral School in Information and Communication Technology
Via Sommarive, 5 I-38123 Povo, Trento, Italy
<EMAIL_ADDRESS>
(2016-03-11)

###### Abstract

We are designing an automated technique to find and recommend experts for
helping in Requirements Engineering tasks, which can be done by ranking the
available people by level of expertise. For evaluating the correctness of the
rankings produced by the automated technique, we want to compare them to a
gold standard. In this work, we ask external people to look at a set of
discussions and to rank their participants, before to evaluate the reliability
of these rankings to serve as a gold standard. We describe the setting and
running of this survey, the method used to build the gold standard from the
rankings of the subjects, and the analysis of the results to obtain and
validate this gold standard. Through the analysis of the results, we conclude
that we obtained a reasonable gold standard although we lack evidences to
support its total correctness. We also made the interesting observation that
the most reliable subjects build the least ordered rankings (i.e. has few
ranks with several people per rank), which goes against the usual assumptions
of Information Retrieval measures.

Keywords— Expert Finding, Requirements Engineering, Gold Standard, Survey  To
the extent possible under law, Matthieu Vergne has waived all copyright and
related or neighboring rights to this technical report. For a more detailed
description of this waiving, visit:
https://creativecommons.org/publicdomain/zero/1.0/

##  1 Introduction

In Requirements Engineering (RE), we aim at managing the requirements of a
project (i.e. the formalized needs of its stakeholders) in an effective and
efficient way. One task for this is to _elicit_ the requirements, which means
to go to the stakeholders and identify their needs before to formalize them
into specifications. Another important task is to ensure that these
requirements _evolve_ with the needs of the stakeholders, who can discover new
constraints, face an evolving environment, or simply change their minds after
receiving additional information. In order to help in this requirements
building and refinement, a high level of expertise is generally required in
order to consider the multiple perspectives and their interdependencies.
Consequently, we focus on finding experts within a community of stakeholders,
which is particularly relevant in Open Source projects having huge communities
of diverse participants. In particular, we are designing an automated
technique to help finding the most expert participants, which can be done by
ranking them by level of expertise [Vergne and Susi, 2014].

To ensure that this automated technique works properly, we plan to compare it
to a Gold Standard (GS), which should allow us to know, given a topic of
expertise, how to rank the participants by decreasing expertise. The issue
here is that, for each community, the participants are different and the
topics change, leading to the inability to provide a general GS applicable to
any community. This means that we need to have community-specific GSs, which
build on the data available in this community to rank its participants.

In this work, we target the XWiki community, which is described in more
details in Section 2 and is a large community composed of professional
programmers, volunteer contributors, and simple users of the XWiki platform.
In order to build a GS for XWiki, we organised a survey involving people out
of this community, and we asked them to look at our XWiki dataset to evaluate
and rank their participants, as described in Section 3. We organised the
survey in order to give enough flexibility to build partial rankings, and
designed a method in Section 4 to infer the final GS based on the multiple
rankings provided by the subjects of the survey. The running of the survey is
described in Section 5 and its results are described and analysed in Section
6. We enriched the survey with additional questions to assess the reliability
of the subjects’ rankings, which is of particular importance to validate the
GS built from them. All the data of this survey can be accessed freely
online111Experiment access: http://selab.fbk.eu/vergne/Experiment-2014-02-19/.

##  2 XWiki Dataset

XWiki222XWiki platform: http://dev.xwiki.org is an Open Source Software (OSS)
which takes the form of a platform for managing wikis. It has a community of
contributors, including a company managing the development of the OSS and
selling support and training on it. This community interact through different
media, in particular a mailing list for support and discussions about the
software.

We have used the archives of the XWiki mailing list, which are freely
available online333XWiki archives: http://lists.xwiki.org/pipermail/users/, to
retrieve the e-mails exchanged and re-build the discussion threads. We have
restricted to e-mails of the year 2012, and we removed the discussions started
before 2012 to ensure having consistent threads. Consequently, we retrieved
2728 e-mails organized in 713 threads, having each between 1 and 37 e-mails.
All of them have been organized and formatted in order to present them to
human subjects444Survey threads:
http://selab.fbk.eu/vergne/Experiment-2014-02-19/dataset/.

To build our GS, we needed to identify the topics on which people should be
ranked, what we did by searching for topics having a reasonable amount of
information. By reasonable, we mean:

  * _E-mail min_

having enough e-mails about the topic to ensure that the subjects have a high
chance to obtain relevant information to evaluate the expertise of each
participant,

  * _E-mail max_

having small enough data to ensure that an average human can deal with it,

  * _Thread average_

avoiding short discussions (i.e. 1-2 messages) which tend to remain
superficial on the topic.

The exact limits cannot be decided a priori because (i) it should be balanced
with the number of topics our subjects will have to work on, otherwise they
could be overwhelmed by the amount of information to consider, and (ii) we did
not know in advance how many topics could satisfy these requirements. To
satisfy them, we extracted the terms used in the e-mails to identify the
available topics, and for each of them we counted how many threads are about
them and how many e-mails they represent. We did so automatically to have an
approximative idea and finalized the selection manually, which lead us to
select two topics (the lists of numbers provide the thread IDs in the
dataset):

  * _Debian_

34 e-mails in 6 threads: 71, 251, 546, 560, 562, 667

  * _Hibernate_

37 e-mails in 8 threads: 147, 153, 154, 172, 185, 444, 576, 687

Because we wanted all our subjects to deal with both topics in an hour, we
evaluated that 30 to 40 e-mails per topic was a reasonable amount, and an
average of 5 e-mails per thread was enough to have informative discussions.

##  3 Survey Procedure and Material

Following the terminology of [Wohlin et al., 2012], the procedure described
here is something between a survey and a quasi-experiment. This is a survey in
the sense that we aim at obtaining opinions from a population of subjects
rather than checking some pre-defined hypothesis, but an experiment aspect is
provided by the control variables we use to help analysing and validating the
results (the GSs built). The classification as a quasi-experiment rather than
an experiment is due to the selection of the topics, which is not random.
Additionally, the fact that we do not use random subjects but volunteers from
a specific community (mainly PhD students in RE) means that we make a
_convenience sampling_ , which significantly reduces the randomness too.

The survey was organized in several phases:

  1. 1.

Present the survey to the subjects

  2. 2.

fill the pre-questionnaire

  3. 3.

Execute the main task on one of the two topics and fill a questionnaire

  4. 4.

Execute the main task on the other topic and fill another questionnaire

  5. 5.

fill the post-questionnaire

The presentation provides a common perspective to all the subjects to work on
their tasks and describe the survey process. The pre-questionnaire focuses on
the subject’s profile and the post-questionnaire on the feedback about the
tasks executed and the survey in general. More details are given in the
following subsections.

###  3.1 Presentation: Common Subject Perspective

In order to minimize the interpretation misalignments of the subjects, we gave
them a common perspective by introducing a synthetic context555Survey
presentation:
http://selab.fbk.eu/vergne/Experiment-2014-02-19/presentation.pdf. This
context was designed based on the expected profile of the subjects, mainly PhD
students in RE. Consequently, the context presented was to take the role of a
requirement analyst in a small company in Information Technologies, aiming for
refining a set of existing requirements. For their imaginary job, they were
asked to find people to help them obtain the relevant information about some
topics related to the requirements to refine. This is with this goal in mind
that we asked them to rank XWiki participants by level of expertise, based on
the intervention of these participants.

###  3.2 Pre-Questionnaire: Subject Profiles

The pre-questionnaire666Pre-questionnaire:
http://selab.fbk.eu/vergne/Experiment-2014-02-19/pre-questionnaire.pdf focuses
on the profile of the human subject. In particular, we asked their current
position (e.g. undergraduate, PhD, professional) and how familiar they are
with OSS in general and XWiki in particular. Additionally, because we ask them
to rank people by expertise, we also asked the subjects how familiar they are
with the expert finding task. We also asked how familiar they are with
requirement analysis because it is the perspective they were asked to take
(i.e. searching for experts to help them refining requirements). We did not
informed the subjects about the topics (i.e. Debian and Hibernate) before to
give them the corresponding questionnaire, which is described below.

###  3.3 Main Questionnaire: Expert Rankings

The main task aims at searching for experts on a given topic, Debian or
Hibernate, by looking at the e-mails of the XWiki participants. Subjects are
given one of the two topics777Debian questionnaire:
http://selab.fbk.eu/vergne/Experiment-2014-02-19/questionnaire-
debian.pdf888Hibernate questionnaire:
http://selab.fbk.eu/vergne/Experiment-2014-02-19/questionnaire-hibernate.pdf
and are asked to search for relevant discussion threads on this topic and rank
their participants by decreasing expertise. Consequently, we asked the
subjects to list the discussions they looked at and to rank their
participants. We also asked them about factors which could hurt the
reliability of the subject’s ranking: the expertise of the subject himself on
the topic, the confidence he has in his ranking, and how difficult it was to
build it. Each subject has executed this task on both topics, half starting
with Debian, the other half with Hibernate, so we can have enough data for
each topic. Swapping the starting topic between subjects can help to identify
a learning effect, for instance by seeing if participants highly ranked on the
first topic tend to be ranked higher in the second.

In order to produce the rankings, a lot of flexibility was provided: a large
blank area was available, with an arrow to show that the most experts should
be placed at the top and the least experts at the bottom of this area. This
way, the subjects can place several people at the same level, allowing us to
know when the subject has not enough information to tell which one is better,
or to draw more complex structures if needed, as shown in Figure 1. During the
presentation of the survey, the subjects have been explicitly requested to
exploit the area in such a way if required.

Figure 1: Example of use of the ranking space: the right side of the scale is
used to place and revise the position of each participant, while the left side
summarizes the final ranking.

###  3.4 Post-Questionnaire: Feedback

To ensure that the survey runs properly, it is important to know if something
went wrong or if the subjects had any difficulty to execute the requested
tasks. Many issues can be managed on the fly by the survey manager, like
answering questions about the survey or helping to access the online
resources, but some issues might remain unnoticed and need to be requested
explicitly to the subjects. The post-questionnaire999Post-questionnaire:
http://selab.fbk.eu/vergne/Experiment-2014-02-19/post-questionnaire.pdf allows
to trace such issues, making us able to consider them when evaluating the
results of the survey. We asked in particular about the perceived ability of
the subject to achieve the requested tasks in the available time, the clarity
of the requests, and the ability to use the provided resources properly. A
free comment area was also available for any feedback that the subject would
like to share which was not part of the questionnaires.

We also used the post-questionnaire as an opportunity to obtain additional
feedback to know how the subjects built their rankings. In particular, we
asked which messages were helpful or not, and to describe the methods used to
rank the participants. While the subjects rankings allow us to build a GS for
evaluating our automated approach, the answers of these additional questions
can be of interest for fixing or improving it.

##  4 Gold Standard Building

###  4.1 Retrieval of Ordered Pairs

In this survey, several subjects provide rankings for each topic, leading to a
set of rankings $R=\\{r_{1},...,r_{n}\\}$ for each topic which need to be
translated into a single ranking $\hat{r}$ acting as a GS. This single ranking
is built by considering, for each pair of participants $(a,b)$, the most
probable order ($a>b$ or $a<b$) depending on the different rankings in $R$. In
such a way, we build a _centroid_ for $R$, meaning a ranking which is “in the
center” of $R$. To compute the ordered pair of a given pair $(a,b)$, a 2D
vector representation is used with Euclidian coordinates $(x,y)$, such that
$x,y\in[0;1]$. In particular, as illustrated in Figure 2, we associate a
specific vector to each case of ordered pair:

  * •

$a>b\Rightarrow(1,0)$

  * •

$a<b\Rightarrow(0,1)$

  * •

$\text{no order}\Rightarrow(0,0)$

The last case occurs when $a$ or $b$ (or both) are not present in the ranking,
so no order can be considered. To identify the centroid ordered pair for
$(a,b)$, we compute a weighted average of these vectors, with the weights
corresponding to the number of times they appear in $R$. More formally, for a
set $R$ of $n$ rankings, $n_{s}$ rankings return $a>b$, $n_{i}$ return $a<b$,
and $n_{u}$ return no order, with $n_{s}+n_{i}+n_{u}=n$. We compute the
average vector $(x,y)$ such that $x=\frac{n_{s}}{n}$ and $y=\frac{n_{i}}{n}$,
which makes it falls between the three cases, as illustrated in Figure 2, and
use the closest order ($>$ or $<$) to obtain the centroid ordered pair. In the
case of perfect balance ($x=y$), no order is considered for the centroid.

$+$$+$$+$ $(0,0)$
---
No order
$(1,0)$
---
$a>b$
$a<b$
---
$(0,1)$
$(a)$$(b)$$y$$x$ Figure 2: Distribution of the three cases of ordered pairs in
a 2D space. Falling in the area (a) leads to use a $a>b$ for the centroid, (b)
leads to $a<b$, and the dashed line leads to no order. An example of vector
$(x,y)$ falls in the area (a), thus being interpreted as $a>b$.

The interpretation behind this model is that, if a majority of rankings agree
on having a given ordered pair, this ordered pair will be the one used for the
centroid. We might argue that the “no order” case is not well represented and
should share the 2D area with the two other cases but, as we show in the next
section, this leads to a loss of information translated into arbitrary ordered
pairs. Consequently, we prefer to reduce the “no order” case to the minimum
and favour the two other cases to preserve as much original ordered pairs as
possible.

It is worth noting that, because we consider the ordered pairs independently,
the transitivity of the rankings is not necessarily preserved in the centroid
pairs. Indeed, by having the rankings $r_{1}=a>b>c$, $r_{2}=c>a>b$, and
$r_{3}=b>c>a$, we obtain the centroid pairs $a>b$, $b>c$, and $c>a$, so a
loop. To obtain a proper ranking, we need to restore the transitivity property
of these pairs, process that we describe in the next section.

###  4.2 Restoring the Transitivity Property

By retrieving the ordered pairs separately, we do not consider their
dependencies, leading to a set of ordered pairs which do not correspond to a
proper ranking (i.e. the transitivity property is not satisfied). In order to
fix it, we use Algorithm 1 which can be summarized in 4 steps:

  1. (1-9)

retrieve all the ordered pairs $a>b$,

  2. (11-15)

add the transitive ordered pairs ($a>b\wedge b>c\Rightarrow a>c$),

  3. (17-21)

remove the loops ($a>b\wedge b>a\Rightarrow\text{no order for $a$ and $b$}$),

  4. (23-32)

build a ranking by looking iteratively for dominant participants.

0: $pairs$: ordered pairs for the centroid

0: $\hat{r}$: ranking built

1: $SUP\leftarrow\emptyset$

2: $E\leftarrow elementsOf(pairs)$

3: for each $(a,b)\in E\times E$ do

4: if $a>b\in pairs$ then

5: $SUP\leftarrow SUP\cup\\{(a,b)\\}$

6: else if $a<b\in pairs$ then

7: $SUP\leftarrow SUP\cup\\{(b,a)\\}$

8: end if

9: end for

10:

11: for each $(a,b,c)\in E\times E\times E$ do

12: if $\\{(a,b),(b,c)\\}\subset SUP$ then

13: $SUP\leftarrow SUP\cup\\{(a,c)\\}$

14: end if

15: end for

16:

17: for each $(a,b)\in E\times E$ do

18: if $\\{(a,b),(b,a)\\}\subset SUP$ then

19: $SUP\leftarrow SUP\backslash\\{(a,b),(b,a)\\}$

20: end if

21: end for

22:

23: $\hat{r}\leftarrow\emptyset$

24: $rank\leftarrow 0$

25: while $||SUP||>0$ do

26: $top\leftarrow\\{e\in E|\exists x\in E,(e,x)\in SUP\wedge\nexists y\in
E,(y,e)\in SUP\\}$

27: for each $e\in top$ do

28: $\hat{r}(e)\leftarrow rank$

29: end for

30: $SUP\leftarrow SUP\backslash\\{(e,x)\in SUP|e\in top\\}$

31: $rank\leftarrow rank+1$

32: end while

Algorithm 1 Procedure used to build a ranking from a set of ordered pairs.

The first phase retrieves the explicit data, the second phase infers the
implicit one, the third phase resolves the over-constrained pairs, and the
last phase resolves the under-constrained ones (add arbitrary ordered pairs to
produce a proper ranking). In particular, during this last phase, if the
information inferred so far shows that $a>b>c$ and $d>e>f>g$, without having
any information between the elements of the two subsets, then the final
ranking arbitrarily merges them into ${r}=[a,d]>[b,e]>[c,f]>g$. Even if some
relations occur, like for $a>x>b>c$ and $d>e>x>f$, the final ranking
arbitrarily merges them into ${r}=[a,d]>e>x>[b,f]>c$, while it could have been
${r}=d>[a,e]>x>b>[f,c]$ as well as many others. These arbitrary choices having
an impact on how the stakeholders are ranked (so who we consider as more
expert), it is important to obtain sufficient information to be able to build
a proper ranking (at least for the top stakeholders). This is why we reduce,
in the previous section, the “no order” case to a single line rather than a 2D
area.

##  5 Survey Execution

For running the survey, we have invited by e-mail people from a RE research
group to participate as volunteers (no incentive was provided). 10 people have
accepted to participate in the survey. The plan of the survey has been
followed rigorously, starting from the presentation of the survey to the
subjects (no detail have been given in the initial invitation to avoid any
preparation). The common perspective has been described and the dataset and
questionnaires have been presented, showing how the task can be executed (a
different topic have been used for the presentation to not bias the subjects).
Once the pre-questionnaires have been filled, all the subjects have received
their first main questionnaire at the same time. The task has last 20 minutes
and we notified them 5 minutes before the end so they could properly finish
their task. Once done, the questionnaires of the first task have been
exchanged with new questionnaires for the second task (i.e. for the other
topic) and the same process of 20 minutes occurred. Once finished, the
questionnaires have been exchanged with the post-questionnaires, and the
subjects were free to leave once their post-questionnaire was filled.

No significant issue was noticed during the execution of the survey, but the
feedback of the post-questionnaires highlights some issues which may have
significant impacts on the reliability of the rankings produced. The most
important issue seems to be the lack of time, which made it hard to read twice
the e-mails, refine the rankings, or even consider all the relevant
discussions. A subject also checked first on Wikipedia (the survey was run
with the online dataset, so they had access to Internet) to know better about
the two topics before to work on the task, which means that even less time was
available for this subject. Another issue was the doubt the subjects could
have in using the right criteria to rank properly the participants, or the
lack of specificity of the topics, which decreases the confidence that
subjects have in their rankings. More superficial issues were mentioned, like
using the names of the participants rather than short IDs makes it harder to
rank them, or the fact that a subject was bored by the presentation. In short,
the main issues seem to be (i) some rankings are based on less information
than others, and (ii) subjects may lack in confidence in their rankings.
Putting aside these free comments, the whole analysis of the survey provide
additional insights which are given in the next section.

##  6 Survey Results

The complete analysis of the questionnaires is done in this section. We first
analyse the subjects who participated through their answers to the pre-
questionnaire in Section 6.1. Then, we introduce a common ground to the two
main tasks in Section 6.2 before to go in a deep analysis of each task in the
sections 6.3 and 6.4, where we identify the GSs for each topic and evaluate
their reliability. We conclude this section by analysing the feedback given in
the post-questionnaires in Section 6.5 and list the main threats to validity.

###  6.1 Subjects’ Profiles

10 people have participated as subjects for the survey: 1 undergraduate and
professional, 8 PhD students, and 1 researcher. As expected, all of them are
familiar with requirement analysis methods (4 have worked with some, 6 are
used to apply them), so none of them should have difficulties to act based on
the common context given during the presentation. However, none of them is
familiar with the expert finding task (6 never did it, 4 did it without
applying any method), which means that not only it could be difficult for them
to rank the people, but they could also use wrong ranking criteria. None of
them are familiar with the XWiki dataset used (7 did not know about XWiki, 3
only heard about it) which means that the rankings produced should have no
bias due to initial knowledge of the subjects on this project. Finally,
although they don’t know about XWiki, some of them are familiar with Open
Source Software (5 have produced OSS code or participated in OSS communities),
still half of them are not familiar and so could have additional difficulties
to understand the discussions.

###  6.2 Main Tasks: Common Grounds

By summing both topics (Debian and Hibernate), the rankings have been produced
based on 14 discussion threads containing 71 e-mails written by 18
participants. For facilitating the redaction of this report, we assign each
participant an ID:

  1. 1.

Adrian Fita

  2. 2.

Arnaud Bourree

  3. 3.

Nicolas Cheneau-Grehalle

  4. 4.

Denis Gervalle

  5. 5.

Eugene Colesnicov

  6. 6.

Guillaume Fenollar

  7. 7.

Jeremie Bousquet

  8. 8.

Marius Dumitru Florea

  9. 9.

Markus Kalkbrenner

  10. 10.

Matt Hammond

  11. 11.

Paul Libbrecht

  12. 12.

Philippe Marzouk

  13. 13.

Richard Hierlmeier

  14. 14.

Richard Rafalski

  15. 15.

Sergiu Dumitriu

  16. 16.

Thomas Mortagne

  17. 17.

Ricardo Rodriguez

  18. 18.

Ryszard Lach

###  6.3 Debian Rankings

For the topic Debian, 6 discussion threads were concerned (71, 251, 546, 560,
562, 667), with a total of 34 e-mails written by 13 participants (1, 2, 4, 5,
6, 8, 9, 10, 12, 13, 14, 15, 16). Among the 10 subjects, 4 have considered all
the threads, 2 have missed 1 thread (4-5 e-mails), and 4 have missed 2 threads
(9-10 e-mails). Over the 6 who missed threads, 4 were the first task, which
might support a _warm up effect_ : the people who deal with the topic in
second might have “warmed up” while treating the other topic first, leading to
a better performance with the second topic.

The rankings produced by subjects working on Debian for their first task are
the following:

  * Subject 4

$[8]>[6,13]>[16]>[14]>[10]$

  * Subject 5

$[8]>[15]>[6]>[4]>[16]>[2]>[9]>[13,14]>[10]>[12]>[5]$

  * Subject 7

$[8,13]>[6]>[16]>[1,2]>[14]>[9]$

  * Subject 9

$[8]>[4,6]>[16,13,15]>[1]>[14]>[10]>[12]$

  * Subject 11

$[1,16,4]>[6]>[9,13]>[10,15]>[14]$

and these ones for the second task:

  * Subject 1

$[16]>[4,8,13]>[6,15]>[2,12]>[1]>[5,9,10,14]$

  * Subject 3

$[1,14]>[4,8,9,10,13]>[16,2,6]>[5,12,15]$

  * Subject 6

$[16]>[4,6,8,13,14,15]>[1,2,9,10,12]>[5]$

  * Subject 8

$[16]>[4,8]>[2,12,13,15]>[6,9,14]>[1,5,10]$

  * Subject 10

$[16]>[6,13,14]>[4]>[1,8,10,12,15]$

An obvious difference appears between the two cases here. With the first task,
subjects mainly consider that the best expert for Debian is 8, and the fact
that the only ranking not putting it first does not even consider it allows to
say that it is a broad agreement. At the opposite, when Debian is the second
task, almost all the rankings agree that 16 should be considered as the top
expert, although for the first task it was prone to be ranked in the middle.

We could consider several explanations for this significant change of the rank
of 16. The first explanation is that people having worked on Hibernate first
(detailed in the Hibernate section) could have been influenced by the
information learned about Hibernate. Indeed, for both first and second task,
Hibernate rankings put at unanimity 16 as top expert, leading to think that
some strong evidences makes everyone agreeing on this perception. Such a
strong evidence might have influenced the subjects to put 16 as top expert
also for Debian, especially if we interpret its middle location in the first
task as poor evidence of high nor low expertise. Another explanation is that,
rather than a matter of influence, 16 could have provided additional
information on his broad experience, including Debian, in discussions about
Hibernate. This explanation might be supported by the fact that the only
ranking putting 16 on top for the first Debian task is made by the only
professional subjects, so we might wonder whether the professional experience
was helpful to analyse the discussions more efficiently, while others might
have needed additional information to assess the expertise of 16. The fact
that some subjects mentioned not having enough time to revise their judgements
could also be related to this case. Unfortunately, the small data we have does
not allow us to favour one explanation over the other.

For building the GS, we use the procedure described in Section 4 based on the
rankings provided by the subjects. In our case, we can build 3 GSs for Debian:

  * First task

$[8]>[4,6]>[13,15,16]>[1,2]>[9]>[14]>[10]>[12]>[5]$

  * Second task

$[16]>[4,13]>[8,14]>[6]>[15]>[2]>[12]>[1]>[9]>[10]>[5]$

  * Both tasks

$[8]>[16]>[4]>[13]>[6]>[15]>[2]>[1]>[14]>[9]>[10]>[12]>[5]$

We can see, through the ranks of the participants 8 and 16, how the GS based
on the rankings of the first task differs from the GS based on the rankings of
the second task. The last GS, based on both, merges these perspectives by
having both participants on top.

These GSs are built from a set of rankings with the aim of reducing all these
rankings to a single one, meaning that the GS should represent at best these
rankings. In order to assess this representativeness, we can measure the
amount of agreement between the rankings of the set and the GS built from
them. More formally, we can decompose a ranking of the set and the
corresponding GS into sets of ordered pairs and count how many pairs are in
the same order ($a>b$ for both or $a<b$ for both), in the opposite order
($a>b$ for one and $a<b$ for the other), or in an unspecified state (at most
one of them gives an order).

If we compare the rankings of the first task with their GS (Table 1), we can
see that the disagreement is always close to 0%, showing that everyone is well
represented. We can see that the amount of unspecified is generally high, but
this is due to the incompleteness of the rankings (e.g. the subject 4 ranks
only 6 participants over 13) and their partial ordering (e.g. the subject 11
ranks 9 participants in only 5 ranks). It is worth noting that no subject
ranking is complete: with 13 participants in total, the subject rankings have
between 6 and 12 participants, with an average of 9 participants per ranking.

For the second task, only the subject 10 is incomplete (10 participants), but
the partial ordering is still relevant: although 13 participants are ranked,
they are distributed in 4 to 6 ranks only. The completeness of the rankings
explains why the unspecified value is significantly lower than for the first
task, while the partial ordering explains why it remains still far from 0%.
Regarding the disagreement, we also have low values excepted for 1 ranking,
which is the only ranking not having 16 at the top.

If we do not make the difference between the first and second task, and
compare all the rankings to the global GS, we do not see significant changes.
The different values vary generally with a small amount (around 3 units), so
the observations made by separating the tasks remain the same in the global
case.

Subject | Agreement | Disagreement | Unspecified
---|---|---|---
Comparison with first task GS
4 | 13 (17%) | 0 (0%) | 65 (83%)
5 | 57 (73%) | 4 (5%) | 17 (22%)
7 | 23 (29%) | 2 (3%) | 53 (68%)
9 | 41 (53%) | 0 (0%) | 37 (47%)
11 | 21 (27%) | 6 (8%) | 51 (65%)
Comparison with second task GS
1 | 61 (78%) | 5 (6%) | 12 (15%)
3 | 35 (45%) | 25 (32%) | 18 (23%)
6 | 53 (68%) | 0 (0%) | 25 (32%)
8 | 55 (71%) | 8 (10%) | 15 (19%)
10 | 27 (35%) | 3 (4%) | 48 (62%)
Comparison with both tasks GS
1 | 62 (79%) | 5 (6%) | 11 (14%)
3 | 36 (46%) | 25 (32%) | 17 (22%)
4 | 12 (15%) | 2 (3%) | 64 (82%)
5 | 54 (69%) | 11 (14%) | 13 (17%)
6 | 50 (64%) | 3 (4%) | 25 (32%)
7 | 24 (31%) | 2 (3%) | 52 (67%)
8 | 55 (71%) | 10 (13%) | 13 (17%)
9 | 38 (49%) | 3 (4%) | 37 (47%)
10 | 22 (28%) | 10 (13%) | 46 (59%)
11 | 24 (31%) | 7 (9%) | 47 (60%)
Table 1: Amount of agreements between the subjects’ rankings and the GSs based
on them for the topic Debian.

Additionally, we can compare the GSs between each other (Table 2). The small
differences between the task-specific comparison and the global comparison let
imagine that the task-specific GSs are quite close to the global one, which is
confirmed by their low disagreements (5% and 13%). Naturally, the global GS
being a trade-off between the two tasks, by summing their disagreements with
the global GS we retrieve the disagreement between both (18%). The biggest
difference between the GSs and their rankings is that, because each GS brings
as much information as possible from its rankings, they tend to be complete
(all participants are ranked) and totally ordered (as many ranks than
participants), although it is not guaranteed. This is why the amount of
unspecified is close to 0% when comparing GSs.

GSs | Agreement | Disagreement | Unspecified
---|---|---|---
Both vs. first | 69 (88%) | 4 (5%) | 5 (6%)
Both vs. second | 66 (85%) | 10 (13%) | 2 (3%)
First vs. second | 57 (73%) | 14 (18%) | 7 (9%)
Table 2: Amount of agreements between the Debian GSs.

Finally, our aim being of producing a reliable GS for Debian, we need to
evaluate the reliability of our 3 GSs. For this, we can look at the perception
of the subjects (Table 3), from who we asked to evaluate their own level of
expertise on the topic, their confidence in the ranking they have produced,
and how difficult it was to produce it. Basically, someone having a high level
of expertise, a high level of confidence, and a low level of difficulty should
be particularly well represented by our final GS.

From the first task, the subjects 4 and 5 are the highest experts (4/5), and 4
in particular also has a high confidence (4/5) and a low difficulty (2/5). By
looking at the agreement between 4 and the GSs (Table 1), we see that it is
indeed perfectly represented by the first GS (0 disagreement) and has the
lowest disagreement with the global one. The main issue with 4 is that his
ranking ranks only 6 participants over 13, so it does not provide a lot of
information.

By considering the second task, only the subject 6 seems to stand out through
his expertise, reinforced by a perfect confidence (5/5) and no difficulty
(1/5). Once again, Table 1 shows that this subject is perfectly represented
with its task-specific GS (0 disagreement) and is among the best at the global
level (only 3 disagreements). However, if he does not suffer from the
incompleteness issue of the previous expert, he suffers from the partial
ordering: only 4 ranks to order 13 participants, which means that it also
lacks a lot of information.

As additional evidences, we might consider other subjects having high
confidence (7 for first task, 3 for second), but their lack of expertise (1 or
2) decreases their reliability, and we can see from Table 1 that 7 is indeed
among the closest to the GS but 3 is the farthest, with two or three times
more disagreement than the second farthest. Consequently, we cannot rely much
on these evidences to strengthen our first perception, so we can only say that
all the Debian GSs seem to be globally correct with some reserves on the
details. In such a situation, we might say that the global GS, which builds on
a trade-off, might be the most reliable GS.

Subject | Participants | Ranks | Expertise | Confidence | Difficulty
---|---|---|---|---|---
ranked | used | (1-5) | (1-5) | (1-5)
First task
4 | 6 | 5 (83%) | 4 | 4 | 2
5 | 12 | 11 (92%) | 4 | 3 | 4
7 | 8 | 6 (75%) | 1 | 4 | 3
9 | 10 | 7 (70%) | 2 | 3 | 4
11 | 9 | 5 (56%) | 2 | 3 | 3
Second task
1 | 13 | 6 (46%) | 2 | 3 | 3
3 | 13 | 4 (31%) | 2 | 4 | 2
6 | 13 | 4 (31%) | 4 | 5 | 1
8 | 13 | 5 (38%) | 2 | 3 | 4
10 | 10 | 4 (40%) | 2 | 3 | 3
Table 3: Ranking properties and subjects’ perception for Debian.

###  6.4 Hibernate Rankings

For the topic Hibernate, 8 discussion threads were concerned (147, 153, 154,
172, 185, 444, 576, 687), with a total of 37 e-mails written by 10
participants (2, 3, 5, 7, 11, 14, 15, 16, 17, 18). Among the 10 subjects, 7
have considered all the threads, 1 have missed 2 threads (3 e-mails), and 2
have missed 3 threads (4-6 e-mails). Over the 3 who missed threads, 2 were the
first task and 1 the second, which seems to support a warm up effect although
it is rather small (at least, it does not contradict it). Additionally, among
the 3 who missed threads for Hibernate, 2 of them also missed threads for
Debian, but the number of missed threads for Debian is doubled (6), so if we
might have some subjects who tend to be slower than others, the warm up effect
still seems to be a relevant explanation (one of the subjects even confirmed
that he re-used his rankings from the first task for the second). It is also
interesting to notice that, although Hibernate has more discussions and more
e-mails, the subjects have been generally more efficient in this task, even
when it was the first task. The fact that less participants were involved
could be an explanation, because it reduces the ranking effort, but it could
also be that the discussions were easier to understand.

The rankings produced by subjects working on Hibernate for their first task
are the following:

  * Subject 1

$[15,16]>[5,7,11,18]>[2,3,17]>[14]$

  * Subject 3

$[14,16]>[7,18]>[15]>[2,3,5,11,17]$

  * Subject 6

$[16]>[14]>[15,18]>[2,5,11]>[3,7,17]$

  * Subject 8

$[16]>[2,14,17,18]>[5]$

  * Subject 10

$[16]>[2,5,11,14,17,18]$

and these ones for the second task:

  * Subject 4

$[16]>[14,15,18]>[5,7]>[2]>[3,11,17]$

  * Subject 5

$[16]>[15]>[14]>[2]>[18]>[7]>[3,11]>[5,17]$

  * Subject 7

$[16]>[11,14,15]>[2,3,5,7,17,18]$

  * Subject 9

$[16]>[15]>[14]>[11]>[18]>[17]>[2]>[3,7]>[5]$

  * Subject 11

$[16]>[11]>[14]>[5,18]>[2]$

At the opposite of Debian, no clear difference appear in the number of
participants nor their order between the rankings of the first task and the
rankings of the second task. However, we observe a significant difference on
the informativeness of these rankings: for the first task, the 10 participants
are ranked on 2 to 5 ranks (3.6 in average), while the second task has between
3 and 9 ranks (6 in average). Like for Debian, this difference can support a
warm up effect, making the subjects more efficient on their second task.

For building the GS, we use the procedure described in Section 4 based on the
rankings provided by the subjects. Like for Debian, we can build 3 GSs for
Hibernate:

  * First task

$[16]>[14]>[15,18]>[5,7,11]>[2]>[3,17]$

  * Second task

$[16]>[15]>[14]>[11]>[18]>[2]>[7]>[3,17]>[5]$

  * Both tasks

$[16]>[15]>[14]>[18]>[7,11]>[2,5]>[3,17]$

Like for the subjects’ rankings, there is few differences between the GSs,
mainly the lack of information for the first task leading to have less ranks
for its GS (6 ranks) compared to the second (9 ranks). The global GS,
naturally, makes a trade-off between the two.

If we compare the rankings of the first task to their GS (Table 4), we can see
that the disagreement is, as expected, close to 0%, excepted for the subject
1. The high disagreement of this subject comes mainly from the participant 14,
which is generally ranked high excepted for subject 1 who ranks her last. We
can also see that the subjects 8 and 10 have a high amount of unspecified,
which is due in part to their incompleteness (6 or 7 participants over 10) but
mainly to their reduced ordering (2 or 3 ranks, which is really poor in
information).

If we look at the rankings of the second task, the subjects 7 and 11 also have
a high amount of unspecified, 7 because of its partial ordering (only 3 ranks)
and 11 mainly because of its incompleteness (6 participants over 10). Still,
the disagreement remains low, so the rankings are well represented by their
GS. Only the subject 4 reaches a high disagreement (18%) because of the
participants 5 and 11 which are swapped compared to the GS.

Although we observe a warm up effect, the main difference between the first
and second task is the informativeness of their rankings. Consequently,
building a global GS makes more sense than for Debian, for which we saw
significant differences in the orders. By comparing all the rankings to this
global GS, like for Debian, the values are not significantly impacted (around
2 units of difference in general). This is coherent with the consistency we
could observe between the rankings, consistency which is reflected in their
GSs: if we compare the GSs between each other (Table 5), the disagreement is
even closer to 0% than for Debian.

Subject | Agreement | Disagreement | Unspecified
---|---|---|---
Comparison with first task GS
1 | 26 (58%) | 8 (18%) | 11 (24%)
3 | 29 (64%) | 1 (2%) | 15 (33%)
6 | 35 (78%) | 1 (2%) | 9 (20%)
8 | 7 (16%) | 2 (4%) | 36 (80%)
10 | 6 (13%) | 0 (0%) | 39 (87%)
Comparison with second task GS
4 | 30 (67%) | 8 (18%) | 7 (16%)
5 | 38 (84%) | 4 (9%) | 3 (7%)
7 | 27 (60%) | 0 (0%) | 18 (40%)
9 | 41 (91%) | 2 (4%) | 2 (4%)
11 | 12 (27%) | 2 (4%) | 31 (69%)
Comparison with both tasks GS
1 | 27 (60%) | 7 (16%) | 11 (24%)
3 | 29 (64%) | 3 (7%) | 13 (29%)
4 | 34 (76%) | 2 (4%) | 9 (20%)
5 | 36 (80%) | 4 (9%) | 5 (11%)
6 | 34 (76%) | 3 (7%) | 8 (18%)
7 | 25 (56%) | 1 (2%) | 19 (42%)
8 | 7 (16%) | 1 (2%) | 37 (82%)
9 | 35 (78%) | 6 (13%) | 4 (9%)
10 | 6 (13%) | 0 (0%) | 39 (87%)
11 | 11 (24%) | 2 (4%) | 32 (71%)
Table 4: Amount of agreements between the subjects’ rankings and the GSs based on them for the topic Hibernate. GSs | Agreement | Disagreement | Unspecified
---|---|---|---
Global vs. 1st | 38 (84%) | 1 (2%) | 6 (13%)
Global vs. 2nd | 38 (84%) | 4 (9%) | 3 (7%)
1st vs. 2nd | 34 (76%) | 6 (13%) | 5 (11%)
Table 5: Amount of agreements between the Hibernate GSs.

Finally, our aim being of producing a reliable GS for Hibernate, we need to
evaluate the reliability of our 3 GSs. For this, like for Debian, we can look
at the perception of the subjects (Table 6) and rely on subjects having a high
level of expertise, a high level of confidence, and a low level of difficulty.

For the first task, the most confident subject (subject 6) is also the least
expert (1/5), so we might see (Table 4) that it is well represented by both
its task-specific and global GS (1 disagreement each) but the lack of
expertise make it unreliable to call it an evidence. At the opposite, the most
expert (subject 1) has a mitigated confidence (3/5) and difficulty (3/5), and
seeing that it is the farthest from both its task-specific and global GS makes
it even more questionable.

The second task is more interesting, because the highest expert (subject 7) is
also the most confident (4/5) and among the ones having the least difficulty
(2/5). Like for Debian, we can assess that this subject’s ranking is very well
represented by both its task-specific GS (0 disagreement) and the global one
(1 disagreement). Still, like for Debian, this subject is from far the least
informative with only 3 ranks to order 10 participants.

In short, we have less evidences than for Debian to confirm the reliability of
our GSs, but the high similarity of all the rankings makes it less
problematic. Yet, we still see that the most expert and confident subjects
have a tendency to have extremely partial rankings.

Subject | Participants | Ranks | Expertise | Confidence | Difficulty
---|---|---|---|---|---
ranked | used | (1-5) | (1-5) | (1-5)
First task
1 | 10 | 4 (40%) | 5 | 3 | 3
3 | 10 | 4 (40%) | 3 | 3 | 2
6 | 10 | 5 (50%) | 1 | 4 | 3
8 | 6 | 3 (50%) | 1 | 3 | 4
10 | 7 | 2 (29%) | 2 | 3 | 2
Second task
4 | 10 | 5 (50%) | 2 | 4 | 2
5 | 10 | 8 (80%) | 2 | 3 | 2
7 | 10 | 3 (30%) | 4 | 4 | 2
9 | 10 | 9 (90%) | 2 | 3 | 4
11 | 6 | 5 (83%) | 3 | 3 | 3
Table 6: Ranking properties and subjects’ perception for Hibernate.

###  6.5 Feedback and Discussion

From Table 7, we can see that the survey ran more or less smoothly. In
particular, the objectives, the main notions and the description of the tasks
were clear, and the dataset was easy to use. However, as mentioned earlier,
the time was not sufficient for everyone to achieve the requested tasks and,
although the dataset itself was easy to use, the relevant discussions were not
easy to select nor understand. These observations can provide an explanation
to the difficulty values of 3 and more provided in the tables 3 and 6.

Question | No Yes | Avg
---|---|---
1 | 2 | 3 | 4 | 5
The time to perform the lab tasks was sufficient. | 1 | 1 | 3 | 3 | 2 | 3.4
The objectives of the lab were clear. | 0 | 0 | 0 | 5 | 5 | 4.5
The notion of “requirement analyst” was clear. | 0 | 0 | 1 | 6 | 3 | 4.2
The notion of “expert finding” was clear. | 0 | 1 | 0 | 3 | 6 | 4.4
The tasks were clear. | 0 | 0 | 1 | 2 | 7 | 4.6
Using the dataset was easy. | 0 | 0 | 3 | 3 | 4 | 4.1
The relevant discussions were easy to select. | 0 | 2 | 4 | 1 | 3 | 3.5
The discussions I have read were easy to understand. | 0 | 2 | 4 | 2 | 2 | 3.4
Table 7: Feedback questions to evaluate how well the survey has run.

Additional questions of the post-questionnaire are also helpful to identify
the properties which make a discussion relevant for building rankings. In
particular, the subjects highlighted the usefulness of discussions about
problem resolutions and question-answers, but these types of discussions are
the most common in our dataset, so other types of discussions might also be
useful. However, they also mentioned that clarification requests and messages
with long logs are not helpful, probably because the content specific to the
participant is minimal. Naturally, long discussions were the most informative
for the subjects, probably because they allow to work deeper on the
problem/question, giving the opportunity to the participants to show better
their expertise. The explicit assessments of expertise, like self-assessment
and recommendations of other people, were also considered by the subjects.

Once the relevant discussions are identified, the subjects built their
rankings with their own strategies, several of them relying on the types of
messages to evaluate who is a higher expert. In particular, people providing
answers were ranked higher than people asking, and people providing detailed
messages were ranked higher than people writing short messages. The number of
messages was also a criteria, with more participation leading to a higher
expertise. More subjective criteria were also used to rank the participants,
in particular the self-assessments and recommendations, but also the apparent
confidence of the participant and the apparent broadness and depth of his
knowledge.

A specific issue occurred from our side, in the analysis of the post-
questionnaires. Two questions were asked to the subjects, to know if they
recognized some of the discussions and participants in the dataset. These
questions were asked with the intention to get more details from subjects
already knowing about XWiki. The issue was that, from the pre-questionnaire, 3
subjects only heard about XWiki, but in the post-questionnaire 2 subjects said
that they recognized some of the discussions, which let think that they were
more involved in the XWiki community than what they said in their pre-
questionnaire. Additionally, 5 said that they recognized some of the
participants: although it might be that they know about some participants from
contexts different to XWiki, it is still half of the subjects, and this
possibility seems to us low enough to rise the flag. Consequently, it might be
that some misunderstandings occurred on these questions, for instance it might
be that subjects mixed the recognition of participants in the _discussions_
with the recognition of participants in the _survey_. With such an
interpretation, the results can be easily explained: most if not all the
subjects know each other. Due to this apparent contradictions, we let these
answers out of our analysis.

Finally, based on all the results presented, we can discuss the validity of
the survey in building reliable GSs. Several observations might support a
threat to the proper conduct of this survey (internal validity): the low
confidence and high difficulty for some subjects to build their rankings,
which can be due to the lack of time and the difficulty to select and
understand the discussions, but also to the lack of expertise of the subjects
in the topics. The threats to the generalizability of our results (external
validity) are obvious and numerous: few subjects, few topics, specific
dataset, specific discussions, ranking based only on the participants of these
discussions, etc. make our GSs highly specific to the context in which they
have been built. However, this is not an issue for us, because this survey was
precisely intended to build a GS based on specific data, such that we can use
this very same data in our automated technique and see how close it is from
the GS. In particular, by having subjects from the XWiki community, they would
already have some knowledge to help them build their rankings: we preferred to
have subjects out of the community to be closer to the situation of the
automated technique, which does not have this initial knowledge. We did not
have an initial hypothesis to check, so we consider no threat to the construct
validity, but the conclusion validity, that we link to the reliability of our
GS, have some threats which deserve to be considered. In particular, we saw
that the most reliable subjects (high expertise, high confidence, low
difficulty) were among the least informative (most incomplete or most
partially ordered), thus giving only superficial support to confirm the
reliability of our GSs. Only the broad agreement of these GSs (how close they
are to the rankings they are based on) supports their reliability, although it
is only an evidence of agreement, not of correctness.

##  7 Conclusion

We are designing an automated technique to find and recommend experts for
helping in Requirements Engineering tasks, which can be done by ranking the
available people by level of expertise. For evaluating the correctness of the
rankings produced by the automated technique, we want to compare them to a
gold standard. In this work, we ask external people to look at a set of
discussions and to rank their participants, before to evaluate the reliability
of these rankings to serve as a gold standard. We describe the setting and
running of this survey, the method used to build the gold standard from the
rankings of the subjects, and the analysis of the results to obtain and
validate this gold standard.

Through this survey, we tried to build a gold standard to know how to rank
people by decreasing expertise for a specific dataset (XWiki). Through the
analysis of the survey, we obtained a reasonable gold standard although we
lack evidences to support fully its correctness. We also made the interesting
observation that the most reliable subjects build the least ordered rankings
(i.e. has few ranks with several people per rank), which goes against the
usual expectations for Information Retrieval measures. This observation
appears to us as an important one, because expert finding systems are mainly
inspired from Information Retrieval systems [Balog, 2012], where the ranking
validation procedures are designed for complete and totally ordered gold
standards [Manning et al., 2008].

Additionally, it might be interesting to investigate further the agreement
between the rankings with more usual measures, like Cohen’s and Fleiss’ kappa,
or the correlation coefficients of Kendall and Spearman. A particular care
should however be given on the impact on these values of the unspecified
agreements, produced by the presence of unordered pairs.

#### Acknowledgement

The authors is grateful to Itzel Morales-Ramirez for her help in executing the
survey and her comments. This work is a result of the RISCOSS project, funded
by the EC 7th Framework Programme FP7/2007-2013, agreement number 318249.

## References

  * [Balog, 2012] Balog, K. (2012).  Expertise Retrieval.  Foundations and Trends® in Information Retrieval _6_ , 127–256.
  * [Manning et al., 2008] Manning, C. D., Raghavan, P. and Schütze, H. (2008).  Introduction to information retrieval.  Cambridge University Press, New York.
  * [Vergne and Susi, 2014] Vergne, M. and Susi, A. (2014).  Expert Finding Using Markov Networks in Open Source Communities.  In Advanced Information Systems Engineering, (Jarke, M., Mylopoulos, J., Quix, C., Rolland, C., Manolopoulos, Y., Mouratidis, H. and Horkoff, J., eds), number 8484 in Lecture Notes in Computer Science pp. 196–210. Springer International Publishing.
  * [Wohlin et al., 2012] Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B. and Wesslén, A. (2012).  Experimentation in Software Engineering.  Springer Berlin Heidelberg, Berlin, Heidelberg.


# ClinicalMamba: A Generative Clinical Language Model on Longitudinal Clinical
Notes

Zhichao Yang1, Avijit Mitra1, Sunjae Kwon1, Hong Yu1,2
1 College of Information and Computer Sciences, University of Massachusetts
Amherst
2 Department of Computer Science, University of Massachusetts Lowell
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

The advancement of natural language processing (NLP) systems in healthcare
hinges on language models’ ability to interpret the intricate information
contained within clinical notes. This process often requires integrating
information from various time points in a patient’s medical history. However,
most earlier clinical language models were pretrained with a context length
limited to roughly one clinical document. In this study, We introduce
ClinicalMamba, a specialized version of the Mamba language model, pretrained
on a vast corpus of longitudinal clinical notes to address the unique
linguistic characteristics and information processing needs of the medical
domain. ClinicalMamba models, with 130 million and 2.8 billion parameters,
demonstrate superior performance in modeling clinical language across extended
text lengths compared to Mamba and clinical Llama. With few-shot learning,
ClinicalMamba achieves notable benchmarks in speed and performance,
outperforming existing clinical language models and large language models like
GPT-4 in longitudinal clinical tasks.

##  1 Introduction

Clinical narratives, such as patient histories, consultation notes, and
discharge summaries, contain detailed and complex information that extends
over long text sequences (Wu et al., 2019). To fully understand a patient’s
condition, treatments, and outcomes, NLP systems need to integrate information
from various parts of these narratives, which often requires understanding the
context provided in those long form text (Blumenthal, 2010).

Understanding the sequence of health events is crucial for diagnoses,
treatment plans, and patient monitoring (Wang et al., 2024; Yang et al., 2023;
Eva, 2005). This often involves putting together information from different
time points within a patient’s health history (Gao et al., 2024). Long context
enables NLP systems to perform temporal reasoning by tracking events over time
longitudinally, which is essential for tasks like predicting disease
progression or extracting medical relation (Chen et al., 2023; Jia et al.,
2019; Wiegreffe et al., 2019).

It becomes imperative to design models for the need for processing longer
texts (Parmar et al., 2023; Tay et al., 2020). Prior studies have introduced
Mamba (Gu and Dao, 2023), a selective state space model, that selects and
compresses all necessary information into latent space from context, and
achieves linear-time efficiency with context length. While these advancements
have been primarily directed towards processing general domain text, the
unique linguistic features of clinical narratives differ significantly from
general domain (Lehman et al., 2023), motivating us to develop specialized
Mamba models in the clinical domain.

In this work, we build and publicly release ClinicalMamba - a Mamba model
pretrained on longitudinal clinical notes. Furthermore, we demonstrate that
ClinicalMamba outperforms multiple language models on longitudinal clinical
NLP tasks. In particular, our contributions are as follows:

  * •

We publicly release ClinicalMamba with 130m and 2.8b parameters trained on
MIMIC-III (Johnson et al.,
2016).111https://github.com/whaleloops/ClinicalMamba

  * •

Through distributed training, ClinicalMamba-2.8b model was pretrained in under
60 hours on 4 A100 GPUs and it is the first clinical autoregressive language
model with a 16k maximum token length.

  * •

Through few-shot prompt-based finetuning, we demonstrate both ClinicalMamba
outperforms original Mamba, GPT4, and other existing clinical long context
language models on well-established long context clinical information
extractions tasks: cohort selection for clinical trial and international
classification of diseases (ICD) coding.

Figure 1: Perplexity of different generative language models on MIMIC-III when
evaluated at various preceding context lengths (1k, 4k, and 16k tokens). The
X-axis is in the log scale. The subfigure is a zoom-out plot with perplexity
ranges 0-100. Experiment settings and detailed results are in section 4.

##  2 Related Work

###  2.1 Pretraining clinical narratives

The rapid expansion of the utilization of electronic health records (EHRs)
into the healthcare landscape underscores an urgent need for a clinical
language model (Kang et al., 2019). Previous work, such as (Alsentzer et al.,
2019) on Clinical BERT embeddings and (Huang et al., 2019) with ClinicalBERT,
adapted general-purpose language models to the clinical domain to enhance
performance on clinical tasks. These models have been pivotal in demonstrating
the effectiveness of adapting general-purpose NLP tools to the intricacies of
clinical text. Similarly, the creation of GatorTron (Yang et al., 2022a)
scales up clinical language models to billions of parameters, while NYUTron
(Jiang et al., 2023) harness billions of unstructured data found in EHRs. Both
underscores the potential of domain-adapted language models to advance
clinical NLP by improving performance across various tasks such as concept
extraction and outcome prediction (Yang et al., 2022a; Jiang et al., 2023).

To handle complex and nuanced tasks, recent studies investigated training
generative models with prompt (Kweon et al., 2023; Peng et al., 2023; Wang et
al., 2023; Lu et al., 2022; Wang and Sun, 2022). These models not only excel
in classification but also in generating clinically relevant text that can be
indistinguishable from human-written notes. Most previous methods focus on
pretraining transformer models with a context window less than 2k tokens.
However, we pretrained a selective state space model with a context window of
16k tokens, which includes more than 98% of the visits in MIMIC-III.

###  2.2 Clinical information extraction on long document

Handling long texts in clinical NLP has always been challenging. Traditional
methods of information extraction tackle this by marking specific locations
within the sentence, but such labeling is not always available, and hiring
annotators can be costly (Fu et al., 2020; Kwon et al., 2022; Deshpande et
al., 2024). Recent advancements in document information extraction involve
pairing labels with documents. However, BERT struggles with processing these
lengthy documents directly.

To address this, prior research introduced Hierarchical-ClinicalRoberta, which
involves breaking down long documents into shorter segments of 512 tokens,
applying ClinicalRoberta to each segment to obtain embeddings, and then using
additional layers to leverage these embeddings for label classification (Huang
et al., 2022; Zhang and Jankowski, 2022). However, this method combines
information from each segment only at the final layer, which can hinder
performance when training data is limited.

To mitigate this issue, ClinicalLongformer is designed to efficiently process
longer context length by employing a self-attention mechanism across all
layers, which is key to its proficiency in managing dense information
exchanges within a specified contextual range (Li et al., 2022; Ji et al.,
2023). This mechanism, while powerful, is limited by its focus on a
predetermined window of text, restricting its scope to what falls within this
window.

To overcome these limitations, the Mamba model emerges as a revolutionary
approach. Mamba employs a selective state space model strategy to meticulously
choose critical data for incorporation into its state (Gu and Dao, 2023),
thereby, enhancing its capability to manage information beyond the
conventional self-attention window. In general domain language modeling, Mamba
surpasses Transformers of equivalent size in task performance and speed.

##  3 Methods

###  3.1 Pretraining

We gather 82,178 hospital visits along with their deidentified free-text
clinical notes (2,083,180) from 46,520 patients in MIMIC-III (Johnson et al.,
2016). Rather than breaking down the notes into chunks of 512 tokens to act as
individual data instances, we aggregate all notes related to a visit
longitudinally. The distribution of token counts per data instance is detailed
in Table A.1. For information on our text pre-processing methods, please refer
to section A.1.

Following previous works (Li et al., 2019; Alsentzer et al., 2019), we
continue to pretrain Mamba using MIMIC-III clinical notes with the causal
language modeling objective. This pretraining process utilizes 4 Nvidia
A100-80GB GPUs. It’s important to note that some of our downstream evaluation
tasks utilize a small subset (6,049) of hospital visits from MIMIC-III, so we
exclude them from the pretaining data. A comprehensive training recipe is
available in section A.2.

###  3.2 Prompt based fine-tuning

We leverage the inherent capabilities of pre-trained language models by
introducing a novel fine-tuning strategy that aligns with the specific demands
of few-shot learning in clinical NLP. Recognizing the limitations of
traditional fine-tuning methods when applied to clinical NLP tasks with
limited labeled data, we adapted a prompt-based fine-tuning mechanism
following previous works (Gao et al., 2021; Yang et al., 2022b; Taylor et al.,
2023). Specifically, we first identify a set of representative prompts that
encapsulate key aspects of the clinical tasks, such as the patient’s alcohol
consumption. These prompts are then appended after each input clinical note
and incorporated into the fine-tuning phase, where the language model learns
to associate them with label tokens (Yes/No) based on a limited dataset. The
generated label tokens are then mapped to label space.

Figure 2: Illustration of Prompt-based fine-tuning.

As shown in Figure 2, we transfer the downstream information extraction task
into a pretraining-like task - label token generation.

###  3.3 Fine-tuning tasks

Cohort selection for clinical trial addresses the challenge of interpreting
unstructured clinical narratives to streamline the patients selection process.
It aims to classify patients based on whether they meet 13 specific
eligibility criteria, such as the usage of aspirin to prevent myocardial
infarction, excessive alcohol consumption, and HbA1c values between 6.5 and
9.5%, among others. The input contains multiple clinical notes with a total
length of 4924 tokens on average. This dataset was released as part of n2c2
challenge (track 1) in 2018.

ICD coding interprets complex clinical narratives, translating them into
standardized codes that facilitate accurate billing, statistical analysis, and
healthcare management. It aims to extract patient’s disease and procedure
codes from clinical text. We followed general instructions from Mullenbach et
al. (2018) in building this task, but instead of using a single discharge
summary as input, we used all previous discharge summaries and assigned ICD
code descriptions from previous visits. We further filtered 50 infrequent
codes as Code-rare and 50 frequent codes as Code-common following Yang et al.
(2022b). The average length is 4,223 and 7,062 tokens respectively. Detailed
dataset statistics are shown in Table A.1.

For each task, we report the micro precision, recall, F1 scores, and the
receiver operating characteristic/area under the curve (ROCAUC) on the test
dataset.

##  4 Results & Discussions

Model | Prec | Recall | F1 | AUC
---|---|---|---|---
CLlama2 | 70.0 | 79.1 | 77.7 | 84.3
Hi-CRoberta | 72.4 | 82.6 | 79.2 | 88.1
CLongformer | 69.7 | 78.6 | 76.1 | 83.5
GPT-4 | 88.1 | 79.9 | 84.8 | -
Mamba-130m | 75.4 | 80.2 | 77.7 | 85.7
CMamba-130m | 79.0 | 86.2 | 82.2 | 91.8
CMamba-2.8b | 88.6 | 89.5 | 88.8 | 95.7
Table 1: Results on cohort selection task, where C is model pretrained in clinical domain. Model | Code-rare | Code-common
---|---|---
Prec | Recall | F1 | AUC | Prec | Recall | F1 | AUC
MultiResCNN | 20.34 | 2.07 | 5.19 | 47.2 | 70.5 | 60.78 | 66.24 | 92.04
Hi-CRoberta | 46.19 | 10.96 | 16.74 | 77.11 | 73.76 | 65.01 | 69.23 | 93.14
CLongformer | 50.27 | 17.81 | 28.69 | 80.52 | 78.42 | 64.97 | 71.14 | 94.24
GPT-4 | 30.91 | 36.12 | 33.29 | - | 72.48 | 62.28 | 68.19 | -
Mamba-130m | 57.75 | 28.08 | 37.79 | 84.8 | 73.71 | 62.87 | 68.94 | 92.75
CMamba-130m | 70.97 | 30.14 | 42.31 | 91.08 | 76.82 | 68.03 | 74.34 | 94.23
CMamba-2.8b | 75.28 | 45.89 | 56.51 | 92.75 | 75.53 | 72.12 | 73.64 | 94.54
Table 2: Results on ICD coding task, where C indicates model pretrained in
clinical domain.

In this section, we will first compare the model’s language modeling ability
on MIMIC-III clinical notes. We will then describe the evaluation on different
clinical information extraction tasks. Finally, we will describe trade-offs
between language modeling abilities (perplexity) and inference speed
(throughput) for several generative models on MIMIC-III notes. Baseline models
used are detailed in section A.3.

ClinicalMamba stands as the sole model capable of handling clinical notes of
up to 16k tokens. As demonstrated in Figure 1, the perplexity for
ClinicalMamba-2.8b decreased from 3.11 to 2.61 as the context length expanded
from 1k to 16k tokens during inference. This is in contrast to the performance
of prior clinical autoregressive language models, where perplexity levels rose
with increased context lengths. For instance, with ClinicalLlama-7b,
perplexity escalated from 2.82 to 94.02 as the context length grew from 4k to
16k. This limitation arises because these models were trained on contexts not
exceeding 4k, impairing their accuracy for next token prediction when given
previous contexts beyond 4k.

In the domain of extracting information from longitudinal clinical records,
ClinicalMamba demonstrates superior performance compared to Mamba.
ClinicalMamba achieved ROCAUC scores of 91.8, 42.3, and 94.2 on Cohort
selection, Code-rare, and Code-common, while Mamba obtained ROCAUC scores of
85.7, 37.8, and 92.8 respectively. ClinicalMamba also outperformed previous
long-range clinical language models with similar number of parameters.
ClinicalMamba significantly outperformed Hierachical-ClinicalRoberta and
ClinicalLongformer by relatively 52.7% and 19.1% on ROCAUC respectively. This
is particularly notable in the Code-rare task with limited training data (5
shots), where ClinicalMamba attained an AUC of 91.1, compared to 77.1 of
Hierarchical-ClinicalRoberta and 80.5 of ClinicalLongformer. Surprisingly,
ClinicalMamba-2.8b also outperformed zero-shot GPT-4, achieving F1 scores of
88.8, 56.6, and 73.6 on Cohort selection, Code-rare, and Code-common tasks,
whereas GPT-4 obtained a F1 score of 84.8, 33.3, and 68.2 respectively.

ClinicalMamba also offers a great tradeoff between language modeling abilities
and inference speed. On one side, small language models’ perplexity is
limited. On the other side, the remarkable perplexity reduction delivered by
large language models comes at a steep increase in computational cost. As
shown in Table A.4, the perplexity of ClinicalMamba-2.8b (2.61) is comparable
to that of ClinicalLlama-7b (2.82) trained with the same computation budget.
Moreover, the inference speeds of ClinicalMamba-2.8b and ClinicalMamba-130m
are 3 to 30 times faster than that of ClinicalLlama7b.

##  5 Conclusion

In this study, we developed and released Mamba models pretrained on a large
collection of clinical notes. Our findings demonstrate the superior
performance of our ClinicalMamba in extracting information from long text
documents compared to other models. We strongly believe that clinical NLP
researchers can benefit from such long-context generative language models that
alleviates the need of a substantial computational power, without any
performance trade-off. Building on the groundwork laid by this study, future
endeavors can further refine and expand the capabilities of ClinicalMamba,
thereby enhancing the effectiveness of clinical data processing across diverse
medical fields.

## Limitations

This work has several notable limitations. First, we do not experiment with
more recent parameter-efficient fine-tuning strategies such as soft prompting
(Lester et al., 2021) and Low-Rank Adaptation (LoRa) (Hu et al., 2021). This
potentially undermined ClinicalMamba on downstream tasks. Second, our
adaptation of the Mamba framework was restricted solely to textual data
documented during visits. EHRs are rich with multifaceted information,
including but not limited to radiology images taken at different times and
Electrocardiogram waveforms that span various periods. Future research could
develop a multimodal Mamba framework to leverage all other modalities. Third,
the MIMIC-III dataset, which serves as the foundation of our study, only
includes notes from the intensive care unit of a single hospital within the
United States. This limits the generalizability of our findings, as care
practices vary significantly across different institutions and countries. We
did not pretrain on MIMIC-IV because it only has a limited number of notes
(and also limited type: discharge summary and radiology report) per visit.
Lastly, the linguistic scope of the MIMIC dataset is limited to English, which
presents a barrier to understanding and applying our findings in non-English
speaking contexts. Addressing these limitations could substantially broaden
the applicability and relevance of our work in future endeavors.

## Ethics Statement

In this research, we gained authorized access to the MIMIC and N2C2 dataset
and used de-identified clinical notes following their license agreement and
HIPAA regulations. When language models are trained on extensive clinical
text, they can inherit biases within the data. For instance, they might prefer
inquiries concerning smoking habits or link specific medical conditions to
certain demographic groups. These biases could be mitigated by enhancing model
alignment with each patient’s background.

## References

  * Alsentzer et al. (2019) Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019.  Publicly available clinical BERT embeddings.  In _Proceedings of the 2nd Clinical Natural Language Processing Workshop_ , pages 72–78, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
  * Blumenthal (2010) David Blumenthal. 2010.  Launching hitech. _The New England journal of medicine_ , 362 5:382–5.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Chen et al. (2023) Zekai Chen, Mariann Micsinai Balan, and Kevin Brown. 2023.  Language models are few-shot learners for prognostic prediction.  _ArXiv_ , abs/2302.12692.
  * Deshpande et al. (2024) Vijeta Deshpande, Minhwa Lee, Zonghai Yao, Zihao Zhang, Jason Brian Gibbons, and Hong Yu. 2024.  Localtweets to localfealth: A mental health surveillance framework based on twitter data.
  * Eva (2005) K. Eva. 2005.  What every teacher needs to know about clinical reasoning.  _Medical Education_ , 39.
  * Fu et al. (2020) Sunyang Fu, David C. Chen, Huan He, Sijia Liu, Sungrim Moon, Kevin J. Peterson, Feichen Shen, Liwei Wang, Yanshan Wang, Andrew Wen, Yiqing Zhao, Sunghwan Sohn, and Hongfang Liu. 2020.  Clinical concept extraction: A methodology review.  _Journal of biomedical informatics_ , page 103526.
  * Gao et al. (2024) Shanghua Gao, Teddy Koker, Owen Queen, Thomas Hartvigsen, Theodoros Tsiligkaridis, and Marinka Zitnik. 2024.  Units: Building a unified time series model.
  * Gao et al. (2021) Tianyu Gao, Adam Fisch, and Danqi Chen. 2021.  Making pre-trained language models better few-shot learners.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3816–3830, Online. Association for Computational Linguistics.
  * Gu and Dao (2023) Albert Gu and Tri Dao. 2023.  Mamba: Linear-time sequence modeling with selective state spaces.  _ArXiv_ , abs/2312.00752.
  * Hu et al. (2021) J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.  _ArXiv_ , abs/2106.09685.
  * Huang et al. (2022) Chao-Wei Huang, Shang-Chi Tsai, and Yun-Nung Chen. 2022.  PLM-ICD: Automatic ICD coding with pretrained language models.  In _Proceedings of the 4th Clinical Natural Language Processing Workshop_ , pages 10–20, Seattle, WA. Association for Computational Linguistics.
  * Huang et al. (2019) Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. 2019.  Clinicalbert: Modeling clinical notes and predicting hospital readmission.  _ArXiv_ , abs/1904.05342.
  * Ji et al. (2023) Shaoxiong Ji, Tianlin Zhang, Kailai Yang, Sophia Ananiadou, E. Cambria, and Jörg Tiedemann. 2023.  Domain-specific continued pretraining of language models for capturing long context in mental health.  _ArXiv_ , abs/2304.10447.
  * Jia et al. (2019) Robin Jia, Cliff Wong, and Hoifung Poon. 2019.  Document-level n-ary relation extraction with multiscale representation learning.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3693–3704, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Jiang et al. (2023) Lavender Yao Jiang, Xujin C. Liu, Nima Pour Nejatian, Mustafa Nasir-Moin, Duo Wang, Anas Abidin, Kevin Eaton, Howard A. Riina, Ilya Laufer, Paawan Punjabi, Madeline Miceli, Nora C. Kim, Cordelia Orillac, Zane Schnurman, Christopher Livia, Hannah Weiss, David Kurland, Sean N. Neifert, Yosef Dastagirzada, Douglas Kondziolka, Alexander T. M. Cheung, Grace Yang, Mingzi Cao, Mona G. Flores, Anthony B Costa, Yindalon Aphinyanaphongs, Kyunghyun Cho, and Eric Karl Oermann. 2023.  Health system-scale language models are all-purpose prediction engines.  _Nature_ , 619:357 – 362.
  * Johnson et al. (2016) Alistair E. W. Johnson, Tom J. Pollard, Lu Shen, Li wei H. Lehman, Mengling Feng, Mohammad Mahdi Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. 2016.  Mimic-iii, a freely accessible critical care database.  _Scientific Data_ , 3.
  * Kang et al. (2019) Tian Kang, Shirui Zou, and Chunhua Weng. 2019.  Pretraining to recognize pico elements from randomized controlled trial literature.  _Studies in health technology and informatics_ , 264:188 – 192.
  * Kweon et al. (2023) Sunjun Kweon, Junu Kim, Jiyoun Kim, Sujeong Im, Eunbyeol Cho, Seongsu Bae, Jungwoo Oh, Gyubok Lee, Jong Hak Moon, Seng Chan You, Seungjin Baek, Chang Hoon Han, Yoon Bin Jung, Yohan Jo, and E. Choi. 2023.  Publicly shareable clinical large language model built on synthetic clinical notes.  _ArXiv_ , abs/2309.00237.
  * Kwon et al. (2022) Sunjae Kwon, Zonghai Yao, Harmon Jordan, David Levy, Brian Corner, and Hong Yu. 2022.  MedJEx: A medical jargon extraction model with Wiki’s hyperlink span and contextualized masked language model score.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11733–11751, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Lehman et al. (2023) Eric P. Lehman, Evan Hernandez, Diwakar Mahajan, Jonas Wulff, Micah J. Smith, Zachary M. Ziegler, Daniel Nadler, Peter Szolovits, Alistair E. W. Johnson, and Emily Alsentzer. 2023.  Do we still need clinical language models? _ArXiv_ , abs/2302.08091.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Lewis et al. (2020) Patrick Lewis, Myle Ott, Jingfei Du, and Veselin Stoyanov. 2020.  Pretrained language models for biomedical and clinical tasks: Understanding and extending the state-of-the-art.  In _Proceedings of the 3rd Clinical Natural Language Processing Workshop_ , pages 146–157, Online. Association for Computational Linguistics.
  * Li et al. (2019) Fei Li, Yonghao Jin, Weisong Liu, Bhanu Pratap Singh Rawat, Pengshan Cai, and Hong Yu. 2019.  Fine-tuning bidirectional encoder representations from transformers (bert)–based models on large-scale electronic health record notes: An empirical study.  _JMIR Med Inform_ , 7(3):e14830.
  * Li and Yu (2020) Fei Li and Hong Yu. 2020.  Icd coding from clinical text using multi-filter residual convolutional neural network.  _Proceedings of the AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence_ , 34 5:8180–8187.
  * Li et al. (2022) Yikuan Li, Ramsey M. Wehbe, Faraz S. Ahmad, Hanyin Wang, and Yuan Luo. 2022.  Clinical-longformer and clinical-bigbird: Transformers for long clinical sequences.  _ArXiv_ , abs/2201.11838.
  * Lu et al. (2022) Qiuhao Lu, Dejing Dou, and Thien Nguyen. 2022.  ClinicalT5: A generative language model for clinical text.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 5436–5443, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Muennighoff et al. (2023) Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. 2023.  Scaling data-constrained language models.  _ArXiv_ , abs/2305.16264.
  * Mullenbach et al. (2018) James Mullenbach, Sarah Wiegreffe, Jon Duke, Jimeng Sun, and Jacob Eisenstein. 2018.  Explainable prediction of medical codes from clinical text.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1101–1111, New Orleans, Louisiana. Association for Computational Linguistics.
  * Parmar et al. (2023) Mihir Parmar, Aakanksha Naik, Himanshu Gupta, Disha Agrawal, and Chitta Baral. 2023.  Longbox: Evaluating transformers on long-sequence clinical tasks.  _ArXiv_ , abs/2311.09564.
  * Peng et al. (2023) C.A.I. Peng, Xi Yang, Aokun Chen, Kaleb E. Smith, Nima M. Pournejatian, Anthony B Costa, Cheryl Martin, Mona G. Flores, Ying Zhang, Tanja Magoc, Gloria P. Lipori, Duane A. Mitchell, Naykky Maruquel Singh Ospina, Mustafa Mamon Ahmed, William R. Hogan, Elizabeth A. Shenkman, Yi Guo, Jiang Bian, and Yonghui Wu. 2023.  A study of generative large language model for medical research and healthcare.  _NPJ Digital Medicine_ , 6.
  * Tay et al. (2020) Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2020.  Long range arena: A benchmark for efficient transformers.  _ArXiv_ , abs/2011.04006.
  * Taylor et al. (2023) Niall Taylor, Yi Zhang, Dan W. Joyce, Ziming Gao, Andrey Kormilitzin, and Alejo Nevado-Holgado. 2023.  Clinical prompt learning with frozen language models.  _IEEE Transactions on Neural Networks and Learning Systems_ , pages 1–11.
  * Wang et al. (2024) Jiaqi Wang, Junyu Luo, Muchao Ye, Xiaochen Wang, Yuan Zhong, Aofei Chang, Guanjie Huang, Ziyi Yin, Cao Xiao, Jimeng Sun, and Fenglong Ma. 2024.  Recent advances in predictive modeling with electronic health records.
  * Wang et al. (2023) Yuqing Wang, Yun Zhao, and Linda Petzold. 2023.  Are large language models ready for healthcare? a comparative study on clinical language understanding.
  * Wang and Sun (2022) Zifeng Wang and Jimeng Sun. 2022.  PromptEHR: Conditional electronic healthcare records generation with prompt learning.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 2873–2885, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Wiegreffe et al. (2019) Sarah Wiegreffe, Edward Choi, Sherry Yan, Jimeng Sun, and Jacob Eisenstein. 2019.  Clinical concept extraction for document-level coding.  In _Proceedings of the 18th BioNLP Workshop and Shared Task_ , pages 261–272, Florence, Italy. Association for Computational Linguistics.
  * Wornow et al. (2024) Michael Wornow, Alejandro Lozano, Dev Dash, Jenelle A. Jindal, Kenneth W. Mahaffey, and Nigam H. Shah. 2024.  Zero-shot clinical trial patient matching with llms.  _ArXiv_ , abs/2402.05125.
  * Wu et al. (2019) Stephen T Wu, Kirk Roberts, Surabhi Datta, Jingcheng Du, Zongcheng Ji, Yuqi Si, Sarvesh Soni, Qiong Wang, Qiang Wei, Yang Xiang, Bo Zhao, and Hua Xu. 2019.  Deep learning in clinical natural language processing: a methodical review.  _Journal of the American Medical Informatics Association : JAMIA_.
  * Yang et al. (2022a) Xi Yang, Aokun Chen, Nima M. Pournejatian, Hoo-Chang Shin, Kaleb E. Smith, Christopher Parisien, Colin B. Compas, Cheryl Martin, Anthony B Costa, Mona G. Flores, Ying Zhang, Tanja Magoc, Christopher A. Harle, Gloria P. Lipori, Duane A. Mitchell, William R. Hogan, Elizabeth A. Shenkman, Jiang Bian, and Yonghui Wu. 2022a.  A large language model for electronic health records.  _NPJ Digital Medicine_ , 5.
  * Yang et al. (2023) Zhichao Yang, Avijit Mitra, Weisong Liu, Dan Berlowitz, and Hong Yu. 2023.  Transformehr: transformer-based encoder-decoder generative model to enhance prediction of disease outcomes using electronic health records.  _Nature Communications_ , 14.
  * Yang et al. (2022b) Zhichao Yang, Shufan Wang, Bhanu Pratap Singh Rawat, Avijit Mitra, and Hong Yu. 2022b.  Knowledge injected prompt based fine-tuning for multi-label few-shot ICD coding.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 1767–1781, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Zhang and Jankowski (2022) Ning Zhang and Maciej Jankowski. 2022.  Hierarchical bert for medical document understanding.  _ArXiv_ , abs/2204.09600.

##  Appendix A Appendix

###  A.1 Text preprocessing

|  | Cohort selection | Code-rare | Code-common
---|---|---|---|---
shots | mean | 89 | 5 | 918
tokens | mean | 4924 | 4223 | 7062
median | 4632 | 3236 | 5177
99% | 10781 | 14345 | 13356
max | 13989 | 18480 | 14773
Table A.1: Number of instances per label (shots) and number of tokens per
input.

We followed Huang et al. (2019) to format notes during text preprocessing. But
We did not convert text to lowercase because Mamba tokenizer is able to
process both upper and lower cases. For notes on each patient’s hospital
visit, we sorted notes by their charted date and concatenated notes into one
string. We used string "- - {NoteType} note - -" to separate the notes. Table
A.2 shows comprehensive values of {NoteType}.

For pretraining data, we truncate notes with more than 16k tokens, however,
this is only less than 2%, a length distribution is provided in Figure A.1. We
exclude a small subset (6,049) of hospital visits due to the evaluation of
MIMIC ICD coding and MIMIC hospital readmission prediction, The visit ids
(hadm_id) are documented in the github.

Figure A.1: Long tail distribution of number of tokens per each visit. Y-axis is the density (sum to 1.0). Category | Count | % | Len
---|---|---|---
Nursing | 506,528 | 73 | 241
Radiology | 338,834 | 83.3 | 449
ECG | 123,042 | 61.3 | 43
Physician | 92,426 | 18.2 | 1369
Discharge summary | 47,572 | 96.7 | 2195
Echo | 34,064 | 45.8 | 464
Respiratory | 32,798 | 8.1 | 205
Nutrition | 7,971 | 6.4 | 602
General | 7,710 | 6.4 | 290
Rehab Services | 5,321 | 4.6 | 622
Social Work | 2,294 | 2.8 | 446
Case Management | 939 | 1.3 | 260
Pharmacy | 97 | 0.1 | 512
Consult | 78 | 0.1 | 1206
Table A.2: Statistic of note events documented in MIMIC-III dataset. Each
column represents a) the number of notes, b) proportion of visits, c) average
number of words for each note type.

###  A.2 Pretraining recipe

ClinicalMamba-2.8b is a selective state space model designed using replication
of the Mamba architecture (Gu and Dao, 2023). ClinicalMamba refers to the
class of models, while 2.8b represents the number of parameters of this
particular pretrained model. We also pretrained ClinicalMamba-130m using
pretraining data from the previous section. The specific values of
hyperparameters are shown in Table A.3. These models were trained for 763
million English tokens over 7000 steps (3 epochs) (Muennighoff et al., 2023).
It was trained as an autoregressive language model, using cross-entropy loss
(Brown et al., 2020). For learning rate scheduling, we followed Mamba and
chose linear learning rate warmup with cosine decay to $1e-5$. We found this
important setting to avoid loss overflow. It took under 60 hours to pretrain
ClinicalMamba-2.8b in on 4 Nvidia Tesla A100-80GB GPUs.

Hyperparameter | Value
---|---
num param | 130m/2.8b
num layer | 24/64
dim model | 768/2560
context len | 16k
num vocab | 50277
position emb | None
optimizer | Adam
beta1 | 0.9
beta2 | 0.95
epsilon | 1e-5
batch size | 32
weight decay | 0.1
gradient clipping | 1.0
peak learning rate | 1e-3/6e-4
Table A.3: Hyperparameters used to train ClinicalMamba.

###  A.3 Baselines

GPT-4 is a large language model designed to understand and generate human-like
text based on the input it receives. We applied zero-shot prompting to each
downstream task, using ACAN and original prompt introduced in Wornow et al.
(2024). GPT-4 (version 2023-12-01-preview) was accessed securely through the
Azure OpenAI API. We set the sampling temperature for decoding to 0.1.

Asclepius-R (Kweon et al., 2023) is a clinical generative language model
trained on MIMIC-III discharge summaries and corresponding instruction-answer
pairs. It has 7 billion parameters with a maximum input of 4096 tokens.

ClinicalLlama2 (CLlama2) is similar to Asclepius-R, but it was trained on all
types of MIMIC-III note with the same computation budget as ClinicalMamba-2.8b
(60 hours on 4 A100). It has 7 billion parameters with 4096 max context
length.

ClinicalLongformer (CLongformer) (Li et al., 2022) is a clinical knowledge
enriched version of Longformer that was further pretrained using MIMIC-III
clinical notes. It has 149 million parameters with a maximum input of 4096
tokens. We only used local attention and does not apply global attention for
computation efficiency.

Hierachical ClinicalRoberta (Hi-CRoberta) (Huang et al., 2022), utilizes
multiple embedding from clinical Roberta (Lewis et al., 2020). It first
segment clinical notes into chunks of 512 tokens to obtain their embeddings,
embeddings are then pooled by concatenation and finally a linear
classification head during downstream task. It has 110 million parameters with
a max of 16384 tokens.

MultiResCNN (Li and Yu, 2020) encode free text with Multi-Filter ResidualCNN,
and applied label code attention mechanism to enable each ICD code to attend
different parts of the document.

| Context Length | Perplexity | Throughput
---|---|---|---
Pythia-130m | 1k | 34.86 | 12042
| 4k | 69.79 | 14859
| 16k | 566.42 | 15713
Mamba-130m | 1k | 29.90 | 24695
| 4k | 26.45 | 34539
| 16k | 34.11 | 37009
ClinicalMamba-130m | 1k | 3.64 | 24709
| 4k | 3.29 | 33349
| 16k | 3.08 | 35486
ClinicalLlama2-7b | 1k | 3.28 | 1005
| 4k | 2.82 | 1013
| 16k | 94.02 | 951
Asclepius-R-7b | 1k | 4.01 | 1066
| 4k | 8.79 | 1064
| 16k | 97.37 | 995
ClinicalMamba-2.8b | 1k | 3.11 | 2932
| 4k | 2.84 | 3007
| 16k | 2.61 | 3027
Table A.4: Trade-off between language modeling abilities (perplexity) and
inference throughput (Tokens/s) for a number of models on MIMIC-III clinical
notes. Among 3 variations of 1k, 4k, and 16k context length, the variation
with best perplexity is bold for each model.