# Empowering Clinicians with Medical Decision Transformers: A Framework for
Sepsis Treatment

Aamer Abdul Rahman1,3, Pranav Agarwal3, Rita Noumeir3, Philippe Jouvet2,4,
Vincent Michalski ,1,2, Samira Ebrahimi Kahou∗,1,5,6
1Mila 2Université de Montréal 3École de Technologie Supérieure 4CHU Sainte-
Justine Hospital
5Canada CIFAR AI Chair 6 University of Calgary
<EMAIL_ADDRESS>Equal senior author contribution.

###### Abstract

Offline reinforcement learning has shown promise for solving tasks in safety-
critical settings, such as clinical decision support. Its application,
however, has been limited by the lack of interpretability and interactivity
for clinicians. To address these challenges, we propose the _medical decision
transformer (MeDT)_ , a novel and versatile framework based on the goal-
conditioned reinforcement learning paradigm for sepsis treatment
recommendation. _MeDT_ uses the decision transformer architecture to learn a
policy for drug dosage recommendation. During offline training, _MeDT_
utilizes collected treatment trajectories to predict administered treatments
for each time step, incorporating known treatment outcomes, target acuity
scores, past treatment decisions, and current and past medical states. This
analysis enables _MeDT_ to capture complex dependencies among a patient’s
medical history, treatment decisions, outcomes, and short-term effects on
stability. Our proposed conditioning uses acuity scores to address sparse
reward issues and to facilitate clinician-model interactions, enhancing
decision-making. Following training, _MeDT_ can generate tailored treatment
recommendations by conditioning on the desired positive outcome (survival) and
user-specified short-term stability improvements. We carry out rigorous
experiments on data from the MIMIC-III dataset and use off-policy evaluation
to demonstrate that _MeDT_ recommends interventions that outperform or are
competitive with existing offline reinforcement learning methods while
enabling a more interpretable, personalized and clinician-directed approach.

##  1 Introduction

Sepsis is a fatal medical condition caused by the body’s extreme response to
an infection. Due to the rapid progression of this disease, clinicians often
face challenges in choosing optimal medication dosages. Hence, there is
significant interest in developing clinical decision support systems that can
help healthcare professionals in making more informed decisions (Sutton et
al., 2020). In the medical field, many tasks involve sequential decision-
making, such as evaluating a patient’s evolving condition in the intensive
care unit (ICU) to make informed medical interventions. This is where
reinforcement learning (RL) comes in as a promising solution for developing
policies that recommend optimal treatment strategies for septic patients
(Raghu et al., 2017; Komorowski et al., 2018; Killian et al., 2020; Saria,
2018; Huang et al., 2022).

These tools are intended to bolster and assist healthcare workers rather than
replace them (Gottesman et al., 2018). Therefore, the reward function employed
by these RL algorithms ideally necessitates clinician input to ensure that the
policy generates decisions aligned with the domain expert’s intentions
(Gottesman et al., 2019). However, the majority of existing studies
predominantly depend on binary reward functions, signifying the patient’s
mortality (Komorowski et al., 2018; Killian et al., 2020; Tang et al., 2022).
In other words, the reward at each timestep in the patient’s history remains
zero until the final interval of the episode. This design leaves no room for
clinician input to modulate the policy toward the achievement of desirable
tasks, such as the stabilization of certain vital signs.

Existing works (Killian et al., 2020; Lu et al., 2020; Li et al., 2019) often
rely on modeling the patient’s medical history using recurrent neural networks
(RNNs). These networks struggle with complex and long medical records due to
vanishing or exploding gradients (Pascanu et al., 2013), leading to sub-
optimal RL policies (Parisotto et al., 2020). Sparse rewards also pose
challenges in the learning of optimal policies since it can be difficult to
identify a causal relationship between an action and a distant reward (Sutton
& Barto, 2018). The sequential design of RNNs aggravates this problem. The low
interpretability of model reasoning is another problem, given the high-stakes
nature of clinical decision making. It is essential to address these
challenges to create reliable decision support systems and improve clinical
uptake of machine learning solutions. Transformers (Vaswani et al., 2017) are
shown to effectively model long sequences, which enables learning of better
representations for treatment histories of patients, potentially yielding more
informed predictions.

In this paper, we propose the _medical decision transformer (MeDT)_ , an
offline RL framework where treatment dosage recommendation for sepsis is
framed as a sequence modeling problem. _MeDT_ , as shown in Figure 1, is based
on the decision transformer (DT) architecture (Chen et al., 2021). It
recommends optimal treatment dosages by autoregressively modeling a patient’s
state while conditioning on hindsight returns. To provide the policy with more
informative and goal-directed input, we also condition _MeDT_ on one-step
look-ahead patient acuity scores (Le Gall et al., 1993) at every time-step.
This enhances the potential for more granular conditioning while facilitating
the interaction of domain experts with the model.

Below we summarize the main contributions of this work:

  * •

We propose _MeDT_ , a transformer-based policy network that models the full
context of a patient’s clinical history and recommends medication dosages.

  * •

We develop a framework to enable clinicians to guide the generation of
treatment decisions by specifying short-term target improvements in patient
stability, which addresses the sparse reward issue.

  * •

We demonstrate that _MeDT_ outperforms or is competitive with popularly used
offline RL baselines over multiple methods of off-policy evaluation (OPE) such
as fitted Q-evaluation (FQE), weighted doubly robust (WDR) and weighted
importance sampling (WIS). Additionally, we leverage a transformer network,
the _state predictor_ , to serve as an approximate model to capture the
evolution of a patient’s clinical state in response to treatment. This model
enables autoregressive inference of _MeDT_ and also serves as an interpretable
evaluation framework of models used for clinical dosage recommendation.

##  2 Related Work

###  2.1 RL for Sepsis Treatment

The use of RL in sepsis treatment aims to deliver personalized, real-time
decision support. It involves modeling optimal strategies for the
administration of treatments, such as vasopressors (VPs) and intravenous
fluids (IVs), based on patient data and expert advice. This problem poses a
considerable challenge due to the potential for long-term effects associated
with these treatments, such as the accumulation of interstitial fluid and
subsequent organ dysfunction resulting from excessive fluid administration
(Gottesman et al., 2018).

To address this issue, Komorowski et al. (2018) propose a value-iteration
algorithm using discretized patient data from electronic health records (EHRs)
for treatment action selection. Subsequent work uses Q-learning with
continuous states and discrete actions and employs OPE to evaluate policies
(Raghu et al., 2017). Huang et al. (2022) uses deep deterministic policy
gradient (DDPG) with continuous states and actions to provide precise dosage
recommendations. Other works explore model-based RL (Peng et al., 2018) and
combined deep RL with kernel-based RL (Raghu et al., 2018) to further improve
treatment recommendations for septic patients. Yet, several significant issues
still need to be resolved, which currently impede the practical implementation
of RL for the treatment of sepsis. Most of these studies assume that agents
begin with a baseline reward of zero until the end of treatment. At the final
time-step in a patient’s history, a positive reward is given for survival and
a negative reward otherwise. Since the manifestation of treatment outcomes
(mortality) can occur with a delay of several days after decisions are made,
it is challenging to identify effective treatment strategies. Shorter-term
objectives, such as the stabilization of vital signs, are often overlooked.
Additionally, given the wealth of data being generated for each ICU patient,
identifying the most relevant aspects in the treatment history may not be
immediately apparent (Gottesman et al., 2018).

###  2.2 Transformer-based Policies

Another challenge in treatment modeling is introduced by the partial
observability of the patient’s state at each time-step. A single reading of
vital signs provides incomplete information on the patient’s well-being. RNNs
address this issue by sequentially processing multiple time-steps of data, but
face difficulties in capturing a patient’s complete state history due to
unstable gradients (Pascanu et al., 2013). This may result in incomplete
information and, consequently, inaccurate decision-making (Yu et al., 2021).
Recent research in RL (Parisotto et al., 2020; Parisotto & Salakhutdinov,
2021; Janner et al., 2021; Tao et al., 2022) is shifting towards attention-
based networks (Niu et al., 2021) like transformers (Vaswani et al., 2017; Lin
et al., 2022), which process information from past time-steps in parallel.

Transformers better capture long contexts and can be effectively trained in
parallel (Lin et al., 2022). This addresses challenges posed by the sequential
processing in RNNs (Wen et al., 2022). The self-attention mechanism in
transformers is particularly beneficial, addressing issues related to sparse
or distracting rewards. Self-attention, in short, first computes attention
weights for information in each time-step by matching their corresponding
_keys_ and _queries_ , which are learnable projections of input tokens.
Afterwards, these weights are used to compute weighted sums of _values_
corresponding to each time-step, potentially discovering dependencies between
distant time-steps. DT (Chen et al., 2021) leverages these advantages for
offline RL (Furuta et al., 2021; Xu et al., 2022; Meng et al., 2021), by
conditioning a policy on the full history of states, actions and an observed
or desired _reward-to-go_. Building on the DT architecture, we propose _MeDT_
, which integrates additional conditioning via short-term goals for
improvements in patient vital signs, yielding a framework for effective sepsis
treatment recommendation.

Figure 1:  _MeDT_ training: At each time-step $t$, the _MeDT_ policy attends
to the past treatment trajectory. This includes the desired treatment outcome
$r$ (at inference time fixed to $+1$ indicating survival), desired next-step
acuity scores $k_{1},\dots,k_{t}$ where
$k_{t}=(kc_{t},kr_{t},kn_{t},kl_{t},kh_{t},km_{t},ko_{t})$, patient states
$s_{1},\dots,s_{t}$, administered drug doses $a_{1},\dots,a_{t-1}$, and
outputs a dose prediction $\hat{a}_{t}$.  Figure 2: Autoregressive evaluation
pipeline: At each time-step $t$, the pre-trained state predictor attends to
past recommended doses $\hat{a}_{1},\dots,\hat{a}_{t}$, the initial patient
state $s_{1}$ and predicted patient states $\hat{s}_{2},\dots,\hat{s}_{t}$,
and outputs a prediction $\hat{s}_{t+1}$ of the patient state at time $t+1$.
Both dosage recommendations $\hat{a}_{t+1}$ and predicted states are fed back
to _MeDT_ to simulate treatment trajectories with multiple sequential
decisions.

###  2.3 Off-Policy Evaluation

OPE is a fundamental problem in RL concerned with estimating the expected
return of a given decision policy using historical data obtained by different
behavior policies (Uehara et al., 2022). Such an evaluation strategy is
particularly useful in situations where interacting with the environment is
costly, risky, or ethically challenging, like in healthcare (Sutton & Barto,
2018; Precup, 2000; Gottesman et al., 2020; Sheth et al., 2022). However, OPE
is inherently difficult because it necessitates counterfactual reasoning, i.e.
unraveling what would have occurred if the agent had acted differently based
on historical data. Nevertheless, while OPE may not necessarily help learn the
optimal policy, it can help identify policies with lower suboptimality (Tang &
Wiens, 2021).

A large subset of OPE methods are based on the concept of importance sampling
(IS). IS uses validation data to assess the evaluation policy’s value by
adjusting the weight of each episode based on its relative likelihood
(Păduraru et al., 2013; Voloshin et al., 2019). The WIS estimator is
considered more stable than IS (Păduraru et al., 2013; Voloshin et al., 2019).
On the other hand, we can directly estimate the evaluation policy’s
performance using the Q-function with FQE, rather than adjusting the weights
of observed experiences like WIS (Le et al., 2019). FQE predicts the expected
cumulative reward for taking a specific action in a given state. WDR is
another OPE technique that combines two approaches for estimating policy value
(Thomas & Brunskill, 2016; Jiang & Li, 2016). It leverages importance
sampling, which adjusts the weight of past experiences based on their
likelihood. Additionally, WDR incorporates value estimates at each step to
improve accuracy and reduce overall variation in the learning process.
Finally, approximate models (AMs) are another class of OPE that involves
directly modeling the dynamics of the environment (Jiang & Li, 2016; Voloshin
et al., 2019). This approximation, while not exact, may be sufficient to
evaluate the policy’s performance. Tang & Wiens (2021) empirically
demonstrated that FQE provided the most accurate and stable estimations over
varying data conditions. Given the difficult nature of evaluation in this
problem setting, we utilize each of the four mentioned OPE methods to infer
rigorous and robust policy evaluations.

###  2.4 Interpretability

The need for interpretability is more significant in safety-critical fields
such as healthcare (Amann et al., 2020). Despite extensive research, the
deployment of deep learning in healthcare has been met with resistance (Yin et
al., 2021). This is primarily due to the black-box nature of these networks,
resulting from their complexity and large number of parameters. Moreover,
attaining interpretability in RL has been a major challenge hindering its
deployment (Agarwal et al., 2023). Owing to their complexity, existing RL
algorithms fall short of being fully interpretable (Glanois et al., 2021).

One simple method of attempting to understand the inner workings of
transformers, is to visualize the computed attention weights. However, Serrano
& Smith (2019) show that attention weights only produce noisy predictions of
the relevance of each input token. Recent works delve into formulating methods
that more representatively capture the relevance of input tokens. Abnar &
Zuidema (2020) propose the rollout method, which considers paths over the
pairwise attention graph while assuming that attention is computed linearly.
However, this work is shown to assign importance to irrelevant tokens (Chefer
et al., 2021b). Chefer et al. (2021b) introduce a method based on layer-wise
relevance propagation, which is effective for encoder transformers. Chefer et
al. (2021a) present a generic interpretability method that is compatible with
every type of transformer architecture. The proposed method relies on the
concept of information flow. It involves monitoring the mixing and evolution
of attention to generate representative heatmaps, illustrating the importance
assigned to input tokens in the model’s decision. This method produces
interpretations that are similarly or more accurate than prior methods while
being simpler to implement. In this work, we utilize this method to generate
interpretations of _MeDT_ and visualize the relevance assigned in the input
space to aid clinicians in understanding the rationale behind the model’s
decision-making.

Figure 3: (a) Dosage recommended by _MeDT_ and clinician policy for different
SAPS2 scores. (b) Distribution of IV fluids and VPs given by the _MeDT_ and
clinician policies.

##  3 Medical Decision Transformer (MeDT)

We frame our problem as a Markov decision process (MDP), comprising a tuple
$\mathcal{(S,A,P,R,S^{\prime})}$, where $\mathcal{S}$ denotes the set of
possible patient states, $\mathcal{A}$ the set of possible dosage
recommendations, $\mathcal{P}$ the state transition function, $\mathcal{R}$
the reward function and $\mathcal{S^{\prime}}$ is the next patient state.
While this framework is well-suited for learning policies via trial-and-error
using RL methods, direct interaction with the environment can be risky in
safety-critical applications like clinical decision making. To mitigate this
risk, we use offline RL, a subcategory of RL that learns an optimal policy
using a fixed dataset of collected trajectories each containing the selected
actions, observed states and obtained rewards.

DT (Chen et al., 2021) uses transformers to model offline RL via an upside-
down RL approach, where a policy has to select actions, that are likely to
yield a specified future reward for a given past trajectory (Schmidhuber,
2019). Our proposed _MeDT_ architecture follows a similar approach for
learning policies.

The input tokens for the policy model encode past treatment decisions and
patient states, as well as the desired returns-to-go (RTG). The output at each
time-step is a distribution over possible actions. Specifically, we condition
the model using RTG $r_{t}=\sum_{t^{\prime}=t}^{T}R_{t^{\prime}}=R_{T}$, which
represents the singular positive or negative treatment outcome at the last
time-step, similar to DT. In addition, we propose to condition _MeDT_ on
short-term goals, such as future patient acuity scores, or acuity-to-go (ATG).
The acuity score provides an indication of the severity of the illness of the
patient in the ICU based on the status of the patient’s physiological systems
and can be inferred from vital signs of the corresponding time-step. Higher
acuity scores indicate a higher severity of illness. In this work, we opt to
use the SAPS2 (Le Gall et al., 1993) acuity score as opposed to popular scores
such as SOFA (Jones et al., 2009). This is because SAPS2 considers relatively
more physiological variables, which we believe will provide more informative
conditioning, allowing flexible user interactions (Morkar et al., 2022). This
formulation allows clinicians to input desired acuity scores for the next
state, providing additional context for treatment selection. This leads to
more information-dense conditioning, allowing clinicians to interact with the
model and guide the policy’s generation of treatment dosages.

To enable clinicians to provide more detailed inputs, we break down the SAPS2
score into constituent scores that correspond to specific organ systems (Le
Gall et al., 1993). Following the definitions provided by Schlapbach et al.
(2022), we define split scores $k$ = ($kc$, $kr$, $kn$, $kl$, $kh$, $km$,
$ko$) to represent the status of the cardiovascular, respiratory,
neurological, renal, hepatic, haematologic and other systems, respectively. A
more detailed breakdown of these scores can be found in the Appendix in Table
3.

In addition to the specification of the treatment outcome, our overall
framework empowers domain experts to establish and select a scheme for
interpretable short-term goal-conditioning, allowing clinicians to guide the
model using their knowledge of the relation between intermediate goals, such
as maintaining patients’ vital signs within a specified range, and favorable
long-term outcomes such as reduced mortality. This enhances the usability of
the model for clinicians, enabling efficient interaction with the model for
future dosage recommendations, considering the current state of the patient’s
condition. It is important to note that defining short-term goals presents a
challenge, given the ongoing complexity in determining ideal targets for
sepsis resuscitation (Simpson et al., 2017). Using these scores, the treatment
progress over $T$ time steps forms a trajectory

$\tau=\left((r_{1},k_{1},s_{1},a_{1}),(r_{2},k_{2},s_{2},a_{2}),\ldots,(r_{T},k_{T},s_{T},a_{T})\right),$
(1)

where, for each time-step $t$, $r_{t},k_{t},s_{t},a_{t}$ respectively
correspond to the RTG, the ATG, the patient state, and the treatment
decisions. We train our policy, a causal transformer network, to predict
ground truth dosages that were administered by a clinician at each time-step
given the treatment trajectory, ignoring future information and the prediction
target via masking (Figure 1). _MeDT_ aims to learn an optimal policy
distribution

$P_{\pi}(a_{t}|s_{\leq t},r_{\leq t},k_{\leq t},a_{<t}),$ (2)

inspired by the model architecture used in Chen et al. (2021). We use an
encoder with a linear layer and a normalization layer for each type of input
(i.e. RTG, ATG, state, action) to project raw inputs into token embeddings. To
capture the temporal dynamics of the patient’s trajectory, we use learned
position embeddings for each time-step, which are added to the token
embeddings. Finally, the resultant embeddings are fed into a causal
transformer, which autoregressively predicts the next action tokens.

###  3.1 Evaluation

In online RL, policies are typically assessed by having them interact with the
environment. However, healthcare involves patients, and employing this
evaluation method is unsafe. In this work, we evaluate the learned RL policy
in an observational setting, where the treatment strategy is assessed based on
historical data (Gottesman et al., 2018). Following the model-based OPE
approach, we introduce an additional predictor network based on the causal
transformer (Radford et al., 2018).

The predictor network, shown in Figure 2, acts as a stand-in for the simulator
during inference. It is trained to learn a state-prediction model defined by
the distribution

$P_{\theta}(s_{t}|a_{<t},s_{<t}),$ (3)

using a similar architecture as the policy network. This allows us to model
how a patient’s state changes in response to medical interventions. Rather
than introducing a termination model, we use a fixed rollout length of
$\mathcal{H}$. The estimated acuity scores can provide more clinically
relevant estimates because they indicate how the stability of the
physiological state of the patient may change given a treatment policy. While
not exact, this approximation can prove adequate for generating reasonable
estimates of a patient’s physiological dynamics. This enables inferring
estimates of patient acuity scores (SAPS2) from predicted states, which can
then be used for policy evaluation. Furthermore, during inference, this model
allows autoregressive generation of a sequence of actions by predicting how
the patient’s state evolves as a result of those actions. Figure 2 and
Algorithm 1 depict this rollout procedure.

Algorithm 1 Evaluation Loop

  1. 1.

Input: Initial patient state $s_{0}$

  2. 2.

Output: Acuity score $g_{1},\ldots,g_{T}$

  3. 3.

Set target return $r_{T}=1$

  4. 4.

Initialize state $s_{1}=s_{0}$, target return $r_{1:T}=r_{T}$ and action
sequence $a_{0}=\\{\\}$

  5. 5.

for $t=1,2,\ldots,T$

  6. 6.

Select desired Acuity To Go $k_{t}$

  7. 7.

Select action $a_{t}=\mathrm{MeDT}(r_{1:t},k_{1:t},s_{1:t},a_{0:t-1})$

  8. 8.

Append $a_{t}$ to the sequence of actions: $a_{1:t}=a_{0:t-1}+[a_{t}]$

  9. 9.

Estimate new state: $s_{t+1}=\mathrm{state\\_estimator}(s_{1:t},a_{1:t})$

  10. 10.

Evaluate acuity score $g_{t+1}$ for state $s_{t+1}$

  11. 11.

Append $s_{t}$ to the sequence of states: $s_{1:t}=s_{1:t-1}+[s_{t}]$

  12. 12.

end for

  13. 13.

return acuity score $g_{1},\ldots,g_{T}$

Additionally, we use WIS (Păduraru et al., 2013; Voloshin et al., 2019), WDR
(Jiang & Li, 2016; Voloshin et al., 2019) and FQE (Le et al., 2019) to
rigorously evaluate the performance of policies.

WIS uses a behavior policy $\pi_{b}$ to evaluate a policy $\pi$ by re-
weighting episodes according to their likelihood of occurrence (Păduraru et
al., 2013; Voloshin et al., 2019). With the per-step importance ratio
$\rho_{t}=\frac{\pi\left(a_{t}\mid s_{t}\right)}{\pi_{b}\left(a_{t}\mid
s_{t}\right)}$ and cumulative importance ratio
$\rho_{1:t}=\prod_{t^{\prime}=1}^{t}\rho_{t^{\prime}}$, WIS can be computed as

$\frac{1}{N}\sum_{n=1}^{N}\frac{\rho_{1:T^{(n)}}^{(n)}}{w_{T^{(n)}}}\left(\sum_{t=1}^{T^{(n)}}\gamma^{t-1}r_{t}^{(n)}\right),$
(4)

where $N$ is the total number of episodes, $T^{(n)}$ is the total number of
time-steps for episode $n$, $\gamma$ is the discount factor and the average
cumulative importance ratio $w_{t}=\frac{1}{N}\sum_{n=1}^{M}\rho_{1:t}^{(n)}$.

FQE is a value-based temporal difference algorithm that utilizes the Bellman
equation to compute bootstrapped target transitions from collected
trajectories and then uses function approximation to compute the $Q$ value of
policy $\pi$. This can be formalized as

$\frac{1}{N}\sum_{n=1}^{N}\sum_{a\in\mathcal{A}}\pi(a\mid
s_{1}^{(n)})\widehat{Q}_{\mathrm{FQE}}^{\pi}(s_{1}^{(n)},a),$ (5)

where $\widehat{Q}_{\mathrm{FQE}}$ is the estimated $Q$ function of $\pi$.

WDR utilizes both value estimators from FQE as well as importance sampling
from WIS in order to reduce the overall variance of estimations (Jiang & Li,
2016; Thomas & Brunskill, 2016). The WDR estimator is defined as follows:

$\displaystyle\frac{1}{N}\sum_{n=1}^{N}\sum_{t=1}^{T^{(n)}}\left[\frac{\rho_{1:t}^{(n)}}{w_{t}}\gamma^{t-1}r_{t}^{(n)}-\left(\frac{\rho_{1:t}^{(n)}}{w_{t}}\widehat{Q}^{\pi}\left(s_{t}^{(n)},a_{t}^{(n)}\right)-\frac{\rho_{1:t-1}^{(n)}}{w_{t-1}}\widehat{V}^{\pi}\left(s_{t}^{(n)}\right)\right)\right],$
(6)

where $\widehat{Q}_{\mathrm{FQE}}$ and $\widehat{V}_{\mathrm{FQE}}$ is the
estimated $Q$ and value function of policy $\pi$ respectively.

Table 1: Estimated final patient acuity scores (averaged over 2,898 patients) for BCQ: batch constrained Q-learning, NFQI: Neural Fitted Q-Learning, DDQN: Double Deep Q-Learning, CQL: Conservative Q-Learning, BC: behavior cloning, DT: decision transformer and _MeDT_ : medical decision transformer. Models | Overall $\downarrow$ | Low $\downarrow$ | Mid $\downarrow$ | High $\downarrow$
---|---|---|---|---
BCQ | 42.10$\pm$0.03 | 41.78$\pm$0.07 | 42.38$\pm$0.03 | 41.59$\pm$0.15
NFQI | 44.21$\pm$0.05 | 43.26$\pm$0.08 | 44.85$\pm$0.06 | 45.13$\pm$0.13
DDQN | 43.47$\pm$0.04 | 43.08$\pm$0.07 | 43.64$\pm$0.05 | 43.29$\pm$0.11
CQL | 40.42$\pm$0.03 | 40.18$\pm$0.07 | 40.59$\pm$0.04 | 41.03$\pm$0.11
BC | 40.50$\pm$0.03 | 40.33$\pm$0.07 | 40.56$\pm$0.03 | 40.29$\pm$0.12
DT | 40.38$\pm$0.03 | 40.16$\pm$0.06 | 40.49$\pm$0.03 | 40.06$\pm$0.12
MeDT | 40.31$\pm$0.03 | 40.05$\pm$0.06 | 40.40$\pm$0.03 | 40.35$\pm$0.14

###  3.2 Interpretability

We utilize the transformer interpretability method introduced by Chefer et al.
(2021a) which is based on the principle of information flow. We adapt this
algorithm for the decoder transformer architecture of _MeDT_ used in this
work. This subsection outlines the mechanisms underlying the computation of
relevance scores used to visualize interpretations.

Let $i$ refer to the input tokens of _MeDT_. $\mathbf{A}^{ii}$ represents the
self-attention interactions between these tokens. Based on these interactions,
we seek to compute the relevancy map $\mathbf{R}^{ii}$. Relevancy maps are
constructed with a forward pass through the self-attention layers, where these
layers attribute to aggregated relevance maps via the following propagation
rules.

Given that each token is self-contained prior to attention operations, self-
attention interactions are initialized with identity matrices. Thus, the
relevancy maps are also initialized as identity matrices:

$\mathbf{R}^{ii}=\mathbb{1}^{i\times i}.$ (7)

The attention matrix $\mathbf{A}$ from each layer is used to update the
relevance maps. The gradients $\nabla\mathbf{A}$ are used to average over the
heads $h$ dimension of the attention map, to account for the differing
importance assigned across the heads of the matrix (Voita et al., 2019).
$\nabla\mathbf{A}:=\frac{\partial y}{\partial\mathbf{A}}$, where $y$ refers to
the output for which we wish to visualize relevance. The aggregated attention
is then defined as:

$\bar{\mathbf{A}}=\mathbb{E}_{h}\left((\nabla\mathbf{A}\odot\mathbf{A})^{+}\right),$
(8)

where $\mathbb{E}_{h}$ is the mean over the $h$ dimension and $\odot$ is the
Hadamard product. + denotes that the negative values are replaced by zero
prior to computing the expectation.

At each attention layer, these aggregated attention scores are then used to
calculate the aggregated relevancy scores as follows:

$\mathbf{R}^{ss}=\mathbf{R}^{ss}+\bar{\mathbf{A}}\cdot\mathbf{R}^{ss}.$ (9)

These relevancy scores can then be used to visualize the importance assigned
across the input token space in the form of a heatmap. Since future tokens are
masked in transformer decoders, there is more attention toward initial tokens
in the input sequence. Hence, to apply these methods to _MeDT_ , we normalize
based on the receptive field of attention.

Figure 4: Box-plots of FQE, WIS and WDR off-policy evaluations for MeDT and
baselines.

##  4 Experiments

###  4.1 Experimental Settings

In this work, we train and evaluate the performance of _MeDT_ on a cohort of
septic patients. The cohort data is obtained from the medical information mart
for intensive care (MIMIC-III) dataset (Johnson et al., 2016), which includes
19,633 patients, with a mortality rate of 9%. These patients were selected on
fulfilling the sepsis-3 definition criteria (Singer et al., 2016). To pre-
process the data, we follow the pipeline defined by Killian et al. (2020). We
extract physiological measurements of patients recorded over 4-hour intervals
and impute missing values using the K-nearest neighbor algorithm. Multiple
observations within each 4-hour window are averaged.

The patient state consists of 5 demographic variables and 38 time-varying
continuous variables such as lab measurements and vital signs. This work
centers on the timing and optimal dosage of administering VP and IV fluids.
The administration of each drug for patients is sampled at 4-hour intervals.
We discretized the dosages for each drug into 5 bins, resulting in a
combinatorial action space of 25 possible treatment administrations. Limiting
our focus to IV fluids and vasopressors implies that these are the only
treatments within our control; other interventions like antibiotics that the
patient might receive are outside the scope of our consideration.

###  4.2 Baselines

We compare _MeDT_ to batch constrained Q-learning (BCQ), neural fitted
Q-learning (NFQI), double deep Q-learning (DDQN) and conservative Q-learning
(CQL) algorithms, which are commonly used baselines in recent works related to
offline reinforcement learning (Killian et al., 2020; Tang et al., 2022; Pace
et al., 2023). Additionally, we train and evaluate DT and a transformer-based
behaviour cloning (BC) algorithm. BC refers to a transformer that takes as
input past states and actions, guided by cross-entropy loss on predicted
actions, to directly imitate the behavior of the clinician’s policy. DT builds
on BC by conditioning on returns-to-go. The proposed _MeDT_ differs from DT in
that it also conditions on acuity-to-go at each time-step.

###  4.3 Training

The transformer policy is trained on mini-batches of fixed context length,
which are randomly sampled from a dataset of offline patient trajectories. In
our case, we chose a context length of 20, which is the longest patient
trajectory in the dataset following pre-processing. For trajectories shorter
than this length, we use zero padding to adjust them. During training, we use
teacher-forcing, where the ground-truth sequence is provided as input to the
model. At each time-step $t$, the ATG ($k_{t}$) is set to the actual acuity
scores of the state at time-step $t+1$ in the sequence. The prediction head of
the policy model, associated with the input token $s_{t}$, is trained to
predict the corresponding discrete treatment action $a_{t}$ using a cross-
entropy loss. The loss for a complete trajectory is averaged over time-steps.
Additionally, the state estimator is separately trained to predict the
patient’s state following the treatment actions. The prediction head of the
state predictor model, corresponding to the input token $a_{t}$, is trained to
estimate the continuous state $s_{t+1}$ using a mean square error loss. The
models are trained on NVIDIA V100 GPUs. We aggregate experimental results for
each model into mean and standard error over five random seeds. Additional
details on hyperparameter selection can be found in the Appendix in Section
A.3.

Figure 5: Visualization of 4 patient trajectories computed by the state
predictor following treatment recommendation from DT (red) and _MeDT_ (blue).

###  4.4 Results and Analysis

We evaluate our proposed _MeDT_ network in the autoregressive inference loop
with the state predictor (Table 1). As elaborated in the Appendix in Section
A.1, we use a naive heuristic to select ATG, and investigate whether the
network conditioned on these prompts results in more stable patient outcomes.
We compare our proposed approach to multiple baselines and run this loop over
only 10 time-steps to avoid the accumulation of state-prediction errors
resulting from the autoregressive nature of evaluation. We calculate the
average and standard error of the SAPS2 scores of the states estimated by the
predictor network for every patient in the test cohort. This cohort split
comprises 2,945 patients. We also evaluate all policies over additional
methods of OPE such as WIS, FQE and WDR.

####  4.4.1 Quantitative Analysis

From Table 1, we infer that the _MeDT_ policy, which is conditioned on both
positive RTGs and our chosen ATG heuristic, results in the most stable
estimated patient states. The DT framework conditioned only with positive RTGs
performs better than BC and other baselines. The learned policies are also
evaluated for patients with different severity of sepsis (denoted as low, mid
and high severity) based on the SAPS2 score of the initial state. Comparing
the models, we observe that the _MeDT_ policy results in more stable states
for low and mid-severity patients, while DT performs best for high-severity
patients. We hypothesize that, given there are far fewer data samples for
patients in the higher severity bracket, _MeDT_ was not able to learn an
accurate mapping of patient states to actions given the additional ATG
context. This suggests that _MeDT_ requires more samples relative to DT to
reach convergence.

We run an experiment to evaluate the sample efficiency of DT and _MeDT_ in
Figure 9 in the Appendix. We evaluate the performance of the policies when
trained on 50%, 75% and 100% of the data from the train split. DT performs
better when trained on the smallest 50% split, while _MeDT_ is performant on
the 75% and 100% splits. This supports the hypothesis that the additional
conditioning used in _MeDT_ has a negative impact on sample efficiency. It is
worth noting that given the small size of the sepsis cohort from the MIMIC-III
dataset, the 50%, 75% and 100% splits are all low training sample settings
relative to standard sizes of training data used in RL. Nevertheless, this is
an optimistic observation, given the potential for the exponential growth of
data available on large-scale EHRs.

Figure 4 depicts the results of the FQE, WIS and WDR evaluations. The _MeDT_
policy produces the highest estimated values on FQE and WDR while CQL performs
best on WIS. It is worth noting that the _MeDT_ and DT policies show
noticeably less variance than the baselines, suggesting they are more robust
models. These results indicate that the clinical dosage recommendations based
on our proposed conditioning method may have had the intended treatment
effects.

Figure 6: Relevancy maps depicting the importance assigned by the model to
input tokens upon predicting action $a_{5}$ for a sample patient trajectory.
Darker and lighter colors indicate lower and higher relevance scores,
respectively. In (a) the mean of the relevancy assigned to the ATG components
is taken to better visualize the relevance across time. Heatmap (b) depicts
relevance across each ATG component to visualize the importance assigned to
each conditioning.

####  4.4.2 Qualitative Analysis

We qualitatively evaluate the policy of _MeDT_ against the clinician’s policy.
To ensure accurate analysis, we use ground-truth trajectories as input
sequences instead of relying on autoregressive inference, which may lead to
compounding errors. In Figure 3a, we conduct a comparative analysis of the
mean dose of VPs and IVs recommended by the _MeDT_ policy and the clinician
policy, for patient states with varying SAPS2 scores. Figure 3b presents the
dosage distribution of IVs and VPs recommended by both the _MeDT_ and
clinician policies.

Our results show that the _MeDT_ policy generally aligns with the clinician’s
treatment strategy but recommends lower doses of IVs on average. Both policies
exhibit a similar trend of increasing medication doses with worsening patient
condition, for both VPs and IVs. Figure 3b reveals that the _MeDT_ policy uses
more zero dosage instances for both IVs and VPs, compared to the clinician
policy. We hypothesize the significant overlap between the _MeDT_ and
clinician policy is a byproduct of the imbalanced nature of the dataset, given
that over 91% of patient trajectories in the dataset resulted in positive
outcomes (survival). As a result, _MeDT_ decides to imitate the clinician
policy. Nevertheless, the alignment with the domain expert policy is ideal,
especially in this high-stakes task where the algorithm relies solely on pre-
existing static data for learning, as it is preferred to assess policies that
only recommend subtle changes and closely resemble those of physicians as a
precautionary measure (Gottesman et al., 2019).

Furthermore, previous studies have demonstrated a trend wherein lower dosages
are recommended for patients with higher acuity scores (Raghu et al., 2017).
This pattern can be linked to the common practice among clinicians of
administering elevated dosages to individuals with high acuity scores, often
associated with more severe medical conditions and, consequently, higher
mortality rates. The challenge arises when algorithms lack data samples
featuring high acuity scores coupled with minimal dosages. In such instances,
these algorithms default to advocating lower medication doses. Figure 3
demonstrates that the _MeDT_ policy diverges from prior research by refraining
from recommending minimal dosages for patients with elevated acuity scores.
This serves as an indicator of better generalization and sample efficient
properties from _MeDT_ given this negative behavior is not observed.

In Figure 5, we visualize the trajectories of multiple patients computed by
the state predictor, following treatment actions recommended by both the DT
and _MeDT_ policies. The impact of ATG conditioning on patient health is
evident, as _MeDT_ leads to more stable trajectories, demonstrating the
potential of our framework to generate targeted and improved treatment
recommendations by considering both the hindsight returns and ATG at each
time-step. In the Appendix in Figure 7, we provide visualizations of some
patient trajectories, where we observed that the _MeDT_ policy produced the
same or worse action policies relative to DT, with no discernible effect of
ATG conditioning. We hypothesize that this may be due to limitations of the
dataset, which may not sufficiently cover some regions of the joint space of
vital signs, treatment decisions and outcomes, causing the model to be unable
to discover some causal relations.

####  4.4.3 Interpretability

Currently, RL algorithms typically function as opaque systems (Gottesman et
al., 2019). They take in data and generate a policy as output, but these
policies are often challenging to interpret. This makes it difficult to
pinpoint the specific data features influencing a suggested action. The lack
of interpretability raises concerns, hindering experts from identifying errors
and potentially slowing down adoption. Thus, clinicians may be hesitant to
embrace recommendations that lack transparent clinical reasoning.

To improve the interpretability and reliability of our _MeDT_ model for users,
we illustrate the relevance assigned by the transformer to input tokens for an
example patient trajectory in Figure 6. The relevance across the ATG
components are averaged in Figure 6a to better depict the relevance assigned
over time, while Figure 6b visualizes the importance assigned to each ATG
component. We observe that _MeDT_ assigns relatively more importance to time-
steps 2 and 3 for this patient sample. Figure Figure 6b shows that the model
considers the conditioning for the Hepatic of higher relevance in its
prediction.

This allows clinicians to monitor the specific points in time when the model
assigns the highest importance, facilitating an assessment of its
reasonableness. If the model differs from ground-truth clinician actions, an
analysis may reveal which features carry the most weight in the decision-
making shift. Additionally, if the model relies on clinically irrelevant
features, it signals to clinicians that the recommendation may be unsound.
This not only enhances understanding of the model’s decision process but also
invites future research into the reliability of deep RL decision-making from a
clinical perspective.

##  5 Conclusion

In this work, we propose the _Medical Decision Transformer_ , a novel
reinforcement learning approach based on the transformer architecture. It
models the full context of a patient’s medical history to recommend effective
sepsis treatment decisions. During training, our framework conditions the
model not only on hindsight rewards but also on look-ahead patient acuity
scores at each time-step. This enables clinicians to later interact with the
model and guide its treatment recommendations by conditioning the model on
short-term goals for patient stability. For autoregressive evaluation of our
proposed approach, we present a separately trained state predictor that models
a patient’s clinical state evolution given a sequence of treatment decisions.
Our experimental results demonstrate the potential of _MeDT_ to bolster
clinical decision support systems by providing clinicians with an
interpretable and interactive intervention support system.

##  6 Acknowledgements

We are thankful to the Digital Research Alliance of Canada for computing
resources and CIFAR, Google, FRQ and NSERC for research funding.

## References

  * Abnar & Zuidema (2020) Samira Abnar and Willem Zuidema.  Quantifying attention flow in transformers.  In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 4190–4197, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.385.  URL https://aclanthology.org/2020.acl-main.385.
  * Agarwal et al. (2023) Pranav Agarwal, Aamer Abdul Rahman, Pierre-Luc St-Charles, Simon JD Prince, and Samira Ebrahimi Kahou.  Transformers in reinforcement learning: a survey.  _arXiv preprint arXiv:2307.05979_ , 2023.
  * Amann et al. (2020) Julia Amann, Alessandro Blasimme, Effy Vayena, Dietmar Frey, Vince I Madai, and Precise4Q Consortium.  Explainability for artificial intelligence in healthcare: a multidisciplinary perspective.  _BMC medical informatics and decision making_ , 20:1–9, 2020.
  * Chefer et al. (2021a) Hila Chefer, Shir Gur, and Lior Wolf.  Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 397–406, 2021a.
  * Chefer et al. (2021b) Hila Chefer, Shir Gur, and Lior Wolf.  Transformer interpretability beyond attention visualization.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 782–791, 2021b.
  * Chen et al. (2021) Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch.  Decision transformer: Reinforcement learning via sequence modeling.  _Advances in neural information processing systems_ , 34:15084–15097, 2021.
  * Furuta et al. (2021) Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu.  Generalized decision transformer for offline hindsight information matching.  _arXiv preprint arXiv:2111.10364_ , 2021.
  * Glanois et al. (2021) Claire Glanois, Paul Weng, Matthieu Zimmer, Dong Li, Tianpei Yang, Jianye Hao, and Wulong Liu.  A survey on interpretable reinforcement learning.  _arXiv preprint arXiv:2112.13112_ , 2021.
  * Gottesman et al. (2018) Omer Gottesman, Fredrik Johansson, Joshua Meier, Jack Dent, Donghun Lee, Srivatsan Srinivasan, Linying Zhang, Yi Ding, David Wihl, Xuefeng Peng, et al.  Evaluating reinforcement learning algorithms in observational health settings.  _arXiv preprint arXiv:1805.12298_ , 2018.
  * Gottesman et al. (2019) Omer Gottesman, Fredrik Johansson, Matthieu Komorowski, Aldo Faisal, David Sontag, Finale Doshi-Velez, and Leo Anthony Celi.  Guidelines for reinforcement learning in healthcare.  _Nature medicine_ , 25(1):16–18, 2019.
  * Gottesman et al. (2020) Omer Gottesman, Joseph Futoma, Yao Liu, Sonali Parbhoo, Leo Celi, Emma Brunskill, and Finale Doshi-Velez.  Interpretable off-policy evaluation in reinforcement learning by highlighting influential transitions.  In _International Conference on Machine Learning_ , pp. 3658–3667. PMLR, 2020.
  * Huang et al. (2022) Yong Huang, Rui Cao, and Amir Rahmani.  Reinforcement learning for sepsis treatment: A continuous action space solution.  In Zachary Lipton, Rajesh Ranganath, Mark Sendak, Michael Sjoding, and Serena Yeung (eds.), _Proceedings of the 7th Machine Learning for Healthcare Conference_ , volume 182 of _Proceedings of Machine Learning Research_ , pp. 631–647. PMLR, 05–06 Aug 2022.
  * Janner et al. (2021) Michael Janner, Qiyang Li, and Sergey Levine.  Offline reinforcement learning as one big sequence modeling problem.  _Advances in neural information processing systems_ , 34:1273–1286, 2021.
  * Jiang & Li (2016) Nan Jiang and Lihong Li.  Doubly robust off-policy value evaluation for reinforcement learning.  In _International Conference on Machine Learning_ , pp. 652–661. PMLR, 2016.
  * Johnson et al. (2016) Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark.  Mimic-iii, a freely accessible critical care database.  _Scientific data_ , 3(1):1–9, 2016.
  * Jones et al. (2009) Alan E Jones, Stephen Trzeciak, and Jeffrey A Kline.  The sequential organ failure assessment score for predicting outcome in patients with severe sepsis and evidence of hypoperfusion at the time of emergency department presentation.  _Critical care medicine_ , 37(5):1649–1654, 2009.
  * Killian et al. (2020) Taylor W. Killian, Haoran Zhang, Jayakumar Subramanian, Mehdi Fatemi, and Marzyeh Ghassemi.  An empirical study of representation learning for reinforcement learning in healthcare.  In _ML4H@NeurIPS_ , 2020.
  * Komorowski et al. (2018) Matthieu Komorowski, Leo Anthony Celi, Omar Badawi, Anthony C. Gordon, and Aldo A. Faisal.  The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care.  _Nature Medicine_ , 24:1716–1720, 2018.
  * Le et al. (2019) Hoang Le, Cameron Voloshin, and Yisong Yue.  Batch policy learning under constraints.  In _International Conference on Machine Learning_ , pp. 3703–3712. PMLR, 2019.
  * Le Gall et al. (1993) Jean-Roger Le Gall, Stanley Lemeshow, and Fabienne Saulnier.  A new simplified acute physiology score (saps ii) based on a european/north american multicenter study.  _Jama_ , 270(24):2957–2963, 1993.
  * Li et al. (2019) Luchen Li, Matthieu Komorowski, and Aldo A. Faisal.  Optimizing sequential medical treatments with auto-encoding heuristic search in pomdps.  _ArXiv_ , abs/1905.07465, 2019.
  * Lin et al. (2022) Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu.  A survey of transformers.  _AI Open_ , 2022.
  * Lu et al. (2020) Mingyu Lu, Zach Shahn, Daby M. Sow, Finale Doshi-Velez, and Li wei H. Lehman.  Is deep reinforcement learning ready for practical applications in healthcare? a sensitivity analysis of duel-ddqn for hemodynamic management in sepsis patients.  _AMIA … Annual Symposium proceedings. AMIA Symposium_ , 2020:773–782, 2020.
  * Meng et al. (2021) Linghui Meng, Muning Wen, Yaodong Yang, Chenyang Le, Xiyun Li, Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang, and Bo Xu.  Offline pre-trained multi-agent decision transformer: One big sequence model conquers all starcraftii tasks.  _arXiv preprint arXiv:2112.02845_ , 2021.
  * Morkar et al. (2022) Dnyanesh N Morkar, Manisha Dwivedi, and Priyanka Patil.  Comparative study of sofa, apache ii, saps ii, as a predictor of mortality in patients of sepsis admitted in medical icu.  _The Journal of the Association of Physicians of India_ , 70(4):11–12, 2022.
  * Niu et al. (2021) Zhaoyang Niu, Guoqiang Zhong, and Hui Yu.  A review on the attention mechanism of deep learning.  _Neurocomputing_ , 452:48–62, 2021.
  * Pace et al. (2023) Alizée Pace, Hugo Yèche, Bernhard Schölkopf, Gunnar Rätsch, and Guy Tennenholtz.  Delphic offline reinforcement learning under nonidentifiable hidden confounding.  _arXiv preprint arXiv:2306.01157_ , 2023.
  * Păduraru et al. (2013) Cosmin Păduraru, Doina Precup, Joelle Pineau, and Gheorghe Comănici.  An empirical analysis of off-policy learning in discrete mdps.  In _European Workshop on Reinforcement Learning_ , pp. 89–102. PMLR, 2013.
  * Parisotto & Salakhutdinov (2021) Emilio Parisotto and Ruslan Salakhutdinov.  Efficient transformers in reinforcement learning using actor-learner distillation.  _arXiv preprint arXiv:2104.01655_ , 2021.
  * Parisotto et al. (2020) Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al.  Stabilizing transformers for reinforcement learning.  In _International conference on machine learning_ , pp. 7487–7498. PMLR, 2020.
  * Pascanu et al. (2013) Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio.  On the difficulty of training recurrent neural networks.  In _International conference on machine learning_ , pp. 1310–1318. Pmlr, 2013.
  * Peng et al. (2018) Xuefeng Peng, Yi Ding, David Wihl, Omer Gottesman, Matthieu Komorowski, Li wei H. Lehman, Andrew Slavin Ross, A. Aldo Faisal, and Finale Doshi-Velez.  Improving sepsis treatment strategies by combining deep and kernel-based reinforcement learning.  _AMIA … Annual Symposium proceedings. AMIA Symposium_ , 2018:887–896, 2018.
  * Precup (2000) Doina Precup.  Eligibility traces for off-policy policy evaluation.  _Computer Science Department Faculty Publication Series_ , pp. 80, 2000.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.  2018\.
  * Raghu et al. (2017) Aniruddh Raghu, Matthieu Komorowski, Imran Ahmed, Leo Celi, Peter Szolovits, and Marzyeh Ghassemi.  Deep reinforcement learning for sepsis treatment.  _arXiv preprint arXiv:1711.09602_ , 2017.
  * Raghu et al. (2018) Aniruddh Raghu, Matthieu Komorowski, and Sumeetpal S. Singh.  Model-based reinforcement learning for sepsis treatment.  _ArXiv_ , abs/1811.09602, 2018.
  * Saria (2018) Suchi Saria.  Individualized sepsis treatment using reinforcement learning.  _Nature Medicine_ , 24:1641 – 1642, 2018.
  * Schlapbach et al. (2022) Luregn J. Schlapbach, Scott L. Weiss, Melania M. Bembea, Joseph A. Carcillo, Francis Leclerc, Stephane Leteurtre, Pierre Tissieres, James L. Wynn, Jerry Zimmerman, Jacques Lacroix, and Marie E Steiner.  Scoring systems for organ dysfunction and multiple organ dysfunction: The podium consensus conference.  January 2022.  doi: 10.1542/peds.2021-052888d.
  * Schmidhuber (2019) Juergen Schmidhuber.  Reinforcement learning upside down: Don’t predict rewards–just map them to actions.  _arXiv preprint arXiv:1912.02875_ , 2019.
  * Serrano & Smith (2019) Sofia Serrano and Noah A. Smith.  Is attention interpretable?  In Anna Korhonen, David Traum, and Lluís Màrquez (eds.), _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pp. 2931–2951, Florence, Italy, July 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1282.  URL https://aclanthology.org/P19-1282.
  * Sheth et al. (2022) Ivaxi Sheth, Aamer Abdul Rahman, Laya Rafiee Sevyeri, Mohammad Havaei, and Samira Ebrahimi Kahou.  Learning from uncertain concepts via test time interventions.  In _Workshop on Trustworthy and Socially Responsible Machine Learning, NeurIPS 2022_ , 2022.  URL https://openreview.net/forum?id=WVe3vok8Cc3.
  * Simpson et al. (2017) Nicholas Simpson, Francois Lamontagne, and Manu Shankar-Hari.  Septic shock resuscitation in the first hour.  _Current opinion in critical care_ , 23(6):561–566, 2017.
  * Singer et al. (2016) Mervyn Singer, Clifford S Deutschman, Christopher Warren Seymour, Manu Shankar-Hari, Djillali Annane, Michael Bauer, Rinaldo Bellomo, Gordon R Bernard, Jean-Daniel Chiche, Craig M Coopersmith, et al.  The third international consensus definitions for sepsis and septic shock (sepsis-3).  _Jama_ , 315(8):801–810, 2016.
  * Sutton et al. (2020) Reed T Sutton, David Pincock, Daniel C Baumgart, Daniel C Sadowski, Richard N Fedorak, and Karen I Kroeker.  An overview of clinical decision support systems: benefits, risks, and strategies for success.  _NPJ digital medicine_ , 3(1):17, 2020.
  * Sutton & Barto (2018) Richard S Sutton and Andrew G Barto.  _Reinforcement learning: An introduction_.  MIT press, 2018.
  * Tang & Wiens (2021) Shengpu Tang and Jenna Wiens.  Model selection for offline reinforcement learning: Practical considerations for healthcare settings.  In _Machine Learning for Healthcare Conference_ , pp. 2–35. PMLR, 2021.
  * Tang et al. (2022) Shengpu Tang, Maggie Makar, Michael Sjoding, Finale Doshi-Velez, and Jenna Wiens.  Leveraging factored action spaces for efficient offline reinforcement learning in healthcare.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=Jd70afzIvJ4.
  * Tao et al. (2022) Tianxin Tao, Daniele Reda, and Michiel van de Panne.  Evaluating vision transformer methods for deep reinforcement learning from pixels.  _arXiv preprint arXiv:2204.04905_ , 2022.
  * Thomas & Brunskill (2016) Philip Thomas and Emma Brunskill.  Data-efficient off-policy policy evaluation for reinforcement learning.  In _International Conference on Machine Learning_ , pp. 2139–2148. PMLR, 2016.
  * Uehara et al. (2022) Masatoshi Uehara, Chengchun Shi, and Nathan Kallus.  A review of off-policy evaluation in reinforcement learning.  _arXiv preprint arXiv:2212.06355_ , 2022.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Voita et al. (2019) Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov.  Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned.  _arXiv preprint arXiv:1905.09418_ , 2019.
  * Voloshin et al. (2019) Cameron Voloshin, Hoang M Le, Nan Jiang, and Yisong Yue.  Empirical study of off-policy policy evaluation for reinforcement learning.  _arXiv preprint arXiv:1911.06854_ , 2019.
  * Wen et al. (2022) Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun.  Transformers in time series: A survey.  _arXiv preprint arXiv:2202.07125_ , 2022.
  * Xu et al. (2022) Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan.  Prompting decision transformer for few-shot policy generalization.  In _International Conference on Machine Learning_ , pp. 24631–24645. PMLR, 2022.
  * Yin et al. (2021) Jiamin Yin, Kee Yuan Ngiam, and Hock Hai Teo.  Role of artificial intelligence applications in real-life clinical practice: systematic review.  _Journal of medical Internet research_ , 23(4):e25759, 2021.
  * Yu et al. (2021) Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin.  Reinforcement learning in healthcare: A survey.  _ACM Computing Surveys (CSUR)_ , 55(1):1–36, 2021.

##  Appendix A Appendix

###  A.1 Acuity Scores and Heuristic

While many works use SOFA scores for sepsis, recent work shows that both SOFA
and SAPS2 predict septic patient mortality effectively (Morkar et al., 2022).
We chose SAPS2 as it considers more physiological variables (12) compared to
SOFA (6), providing more informative conditioning and allowing flexible user
interactions.

Our approach to varying ATG for MeDT over the sequence involves a simple
heuristic: reducing the ATG linearly at each time-step in the sequence. By
varying the ATG score for individual systems, we found that the hepatic and
hematologic systems had the most positive effect on this cohort of patients.

###  A.2 Limitations and Future Work

The MIMIC-III dataset has some limitations, as it only represents a specific
geographic area, which could result in an over-representation of certain
patient populations and an under-representation of others. Consequently, using
the state predictor for evaluation may introduce biases inherent in the
dataset on which it was trained. To mitigate these potential biases, we will
investigate causal representation learning and pre-training techniques that
enhance model robustness. Despite these limitations, _MeDT_ provides a general
framework to harness the vast amount of data found in large-scale EHRs from
different modalities. Using the proposed framework, researchers can explore
the scalability of the transformer architecture to develop systems for
effective treatment recommendation for other medical conditions in the future.

###  A.3 Hyperparameters

The transformer architecture of MeDT consists of six layers. The models in
this work are trained with batch sizes of 64 and a learning rate of 0.0006.
The GELU activation function was used within the transformer architecture.

###  A.4 Ablation Study

From Table 4, we observe that DT makes a higher performance jump over behavior
cloning on patients with negative ground-truth trajectories. This can be
attributed to DT closely imitating BC for positive trajectories while being
able to adopt a more effective policy for negative trajectories due to RTG
conditioning. On the other hand, MeDT appears to make a higher leap on
positive ground-truth patients. This is likely due to the larger number of
positive samples available for MeDT to learn and condition using ATG.

###  A.5 Further Visualizations

Figure 7 shows the patient trajectories generated by the state predictor,
following treatment recommendations from DT (red) and MeDT (blue).
Specifically, we have visualized the trajectories where the MeDT model
produced the same or worse recommendation compared to the DT model. We
postulate that this outcome may be attributable to the limitation of the
dataset, which could have resulted in the MeDT model lacking adequate
information about the given ATG conditioning and state pair.

Figure 7: Visualization of six patient trajectories computed by the state predictor following treatment recommendation from DT (red) and _MeDT_ (blue).  Figure 8: Train-validation performance curve of state predictor averaged over 10 seeds.  Figure 9: Performance of _MeDT_ vs DT on scaling from training on 50% to a 100% of the train split averaged over 3 seeds. Table 2: List of demographic and observational patient variables in state space. | Observations |  | Demographics
---|---|---|---
Glasgow coma scale | Heart rate | Systolic BP | Re-admission
Mean BP | Diastolic BP | Respiratory rate | Gender
Body temp | FiO2 | Potassium | Weight
Sodium | Chloride | Glucose | Ventilation
Magnesium | Calcium | Hemoglobin | Age
WBC | Platelets count | Prothrombin time |
Partial thromboplastin time | Arterial pH | Magnesium |
PaO2 | PaCO2 | Arterial blood gas |
HCO3 | Arterial lactate | PaO2/FiO2 ratio |
SpO2 | SGOT | Creatinine |
Blood urea nitrogen | INR | Bilirubin |
SIRS | Shock index | Cumulative Fluid Balance |
Fluid input/4-hr | Fluid output $/4$-hr | Total fluid input |
Total fluid output |  |  |
Table 3: Observations considered for the split SAPS2 scores. Organ system | Variable
---|---
Cardiovascular $(kc)$ | Blood pressure; Heart rate; Fluid output
Respiratory $(kr)$ | Mechanical ventilation; PaO2 and FiO2
Neurological $(kn)$ | Glasgow coma scale
Renal $(kl)$ | Blood urea nitrogen
Hepatic $(kh)$ | Bilirubin
Haematologic $(km)$ | White blood cell count
Other $(k0)$ | Temperature; Potassium; Sodium
Table 4: Ablation study on patients with ground-truth positive and negative trajectories. Models | All | +1 | -1
---|---|---|---
BC | $40.50\pm 0.03$ | $40.36\pm 0.02$ | $41.91\pm 0.09$
DT | $40.38\pm 0.03$ | $40.27\pm 0.02$ | $41.73\pm 0.09$
MeDT | $\mathbf{40.31}\pm\mathbf{0.03}$ | $\mathbf{40.15}\pm\mathbf{0.02}$ | $\mathbf{41.72}\pm\mathbf{0.03}$


February 2024

The de Sitter swampland conjectures in the context of Chaplygin-inspired
inflation

Orfeu Bertolami1,2, Robertus Potting3,4 and Paulo M. Sá3,5

1 Departamento de Física e Astronomia, Faculdade de Ciências, Universidade do
Porto,

Rua do Campo Alegre s/n, 4169-007 Porto, Portugal

2 Centro de Física das Universidades do Minho e do Porto, Rua do Campo Alegre
s/n, 4169-007 Porto, Portugal

3 Departamento de Física, Faculdade de Ciências e Tecnologia, Universidade do
Algarve,

Campus de Gambelas, 8005-139 Faro, Portugal

4 CENTRA, Instituto Superior Técnico, Universidade de Lisboa, Avenida Rovisco
Pais, 1049-001 Lisboa, Portugal

5 Instituto de Astrofísica e Ciências do Espaço, Faculdade de Ciências,
Universidade de Lisboa, Campo Grande, 1749-016 Lisboa, Portugal

###### Abstract

In this work, we discuss the de Sitter swampland conjectures in the context of
the generalized Chaplygin-inspired inflationary model. We demonstrate that
these conjectures can be satisfied, but only in the region of the parameter
space far away from the General Relativity limit. The cosmic microwave
background data had already been found to restrict the allowed inflationary
potentials of this model. Our results impose a further limitation on the
possible potentials.

E-mail addresses<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
.

Swampland conjectures have been put forward in order to identify de Sitter
solutions that do not lie in the string theory landscape. Such apparently
consistent solutions do not admit a suitable ultraviolet completion (see Ref.
[1] for the original discussion and Ref. [2] for a review). These conjectures
are particularly relevant as they relate the intrinsic consistency of string
theory to conditions for obtaining our four-dimensional world, and also
address the notoriously difficult problem of obtaining inflation from the
fields that naturally arise in string theory.

In fact, quite involved scenarios are required in string theory in order to
get inflation (see, for instance, Ref. [3]), which is somewhat surprising as
in $N=1$ supergravity, presumably a low-energy limit of string theory,
inflationary solutions can be naturally implemented (see eg. Ref. [4]). From a
phenomenological point of view, viable string theory models, namely those that
have an intermediate-scale Grand Unified Theory, have been shown to require a
period of inflation for their implementation [5].

In broad terms, the swampland conjectures amount to a set of necessary
conditions that ensure general low-energy features, such as the presence of
local gauge symmetries, as well as the presence of at least one Planck-mass
particle in order to account for the weakness of gravity. They are also
required in order to assure that higher-derivative terms in the effective
action do not lead to superluminal propagation [6]. Not included among these
general requirements is the Strong Equivalence Principle, from which it
follows that gravity has to be described by General Relativity. Nevertheless,
in most of the applications of the swampland conjectures, this latter
assumption is tacitly made. Technically, this implies that when considering
the more general setting of alternative theories of gravity, they are
confronted with the swampland conjectures in the so-called Einstein frame.

Indeed, given that General Relativity has some limitations, it is natural to
consider alternative theories of gravity [7] and ask if the swampland
conjectures hold for inflationary models (either single-field or more
involved) arising from these theories. This issue has been recently analyzed
for theories of gravity with non-minimal coupling between curvature and matter
[8], and their inflationary solutions [9]. It was found that inflation in
these theories cannot be reconciled with the de Sitter swampland conjectures
[10].

In the current work, we examine another model in the context of the swampland
conjectures, namely the generalized Chaplygin-inspired inflationary model
[11]. This confrontation is particularly interesting, as it allows, through
inflation, to examine models that go beyond the Standard Model of particle
physics. An additional motivation to consider the generalized Chaplygin-
inspired inflationary model concerns the recent claim that a Chaplygin-like
equation of state can endow the vacuum with interesting features related to
entanglement entropy generation [12] and the suppression of the cosmological
constant due to inflation and the constraint set by the Bekenstein bound
between the entropy and the energy density [13].

The swampland conjectures can be expressed through constraints on scalar
fields in low-energy effective field theories, generically denoted by $\phi$
[14, 15], namely (in the Einstein frame),

$\displaystyle\dfrac{\Delta\phi}{M_{\rm P}}<c_{1},$ (1) $\displaystyle M_{\rm
P}\dfrac{|V^{\prime}|}{V}>c_{2},$ (2)

where $\Delta\phi$ denotes a scalar-field variation, $M_{\rm P}\equiv M_{\rm
Pl}/\sqrt{8\pi}$ is the reduced Planck’s mass, $V(\phi)$ is the scalar field
potential, $V^{\prime}\equiv\partial V/\partial\phi$, and $c_{1}$ and $c_{2}$
are $\mathcal{O}(1)$ constants. It has been argued [16, 17] that a more
refined condition should also be considered, namely the potential should
satisfy condition (2), or, alternatively,

$M_{\rm P}^{2}\dfrac{V^{\prime\prime}}{V}<-c_{3},$ (3)

where $V^{\prime\prime}\equiv\partial^{2}V/\partial\phi^{2}$ and the constant
$c_{3}$ is of order one.

The requirements expressed by Eqs. (2) and (3) are incompatible with the onset
conditions of single-field (cold) inflation, which require that the inflaton
satisfies the slow-roll conditions $\epsilon_{\phi}\ll 1$ and
$|\eta_{\phi}|\ll 1$ at the onset of inflation [18], where

$\epsilon_{\phi}=\dfrac{M_{\rm
P}^{2}}{2}\left(\dfrac{V^{\prime}}{V}\right)^{2}$ (4)

and

$\eta_{\phi}=M_{\rm P}^{2}\dfrac{V^{\prime\prime}}{V},$ (5)

so that at the end of inflation $\epsilon_{\phi}\sim|\eta_{\phi}|\sim 1$. It
is remarkable that the conditions for the onset of inflation match the
observational results [18]

$\epsilon_{\phi}<0.0044$ (6)

and

$\eta_{\phi}=-0.015\pm 0.006,$ (7)

which are incompatible with the requirements on $c_{2}$ and $c_{3}$.

However, this incompatibility can be alleviated or resolved if one considers
multi-field inflationary backgrounds which follow curved, non-geodesic
trajectories in field space [19] and also if one considers the presence of a
significant dissipation in the context of warm inflationary models [20] for
one [21, 22] or any number of scalar fields [23].

The generalized Chaplygin gas model has been originally proposed to unify dark
matter and dark energy by resorting to a fluid with an exotic equation of
state

$p=-\dfrac{A}{\rho^{\alpha}},$ (8)

where $p$ is the isotropic pressure, $\rho$ is the energy density, $A$ a
positive constant, and $0<\alpha\leq 1$. The original Chaplygin gas model
corresponds to $\alpha=1$. It is well known that this equation of state has
many interesting features [24, 25, 26]. The support of the Chaplygin equation
of state $(\alpha=1)$ is a Born-Infeld action that describes a brane [25]
which can be parametrised through the action of a complex [25, 26] or a real
scalar field [24, 27].

The evolution equation for the energy density,

$\dot{\rho}+3H(\rho+p)=0,$ (9)

where the dot denotes the derivative with respect to the cosmic time,
$H=\dot{a}/a$ is the expansion rate and $a(t)$ is the scale factor of the
Friedmann-Lemaître-Robertson-Walker metric, with $p$ given by Eq. (8), can be
easily integrated, yielding [24, 25, 26]

$\rho=\left[A+\dfrac{B}{a^{3(1+\alpha)}}\right]^{{\textstyle{{1}\over{1+\alpha}}}},$
(10)

where $B$ is an integration constant.

This type of behavior of the energy density $\rho$ can also arise from a
modification of gravity, particularly from a generalized Born-Infeld action
for a scalar field $\phi$ with energy density $\rho_{\phi}$, giving rise to a
modified Friedmann equation of the form [11]

$H^{2}=\dfrac{1}{3M_{\rm
P}^{2}}\left[A+\rho_{\phi}^{1+\alpha}\right]^{{\textstyle{{1}\over{1+\alpha}}}},$
(11)

where the scalar field satisfies the usual equation of motion

$\ddot{\phi}+3H\dot{\phi}+V^{\prime}=0,$ (12)

with $V$ a suitable inflationary potential.

Compatibility with the cosmic microwave background (CMB) data for monomial and
hilltop potentials has been examined for the generalized Chaplygin-inspired
inflationary model described by Eqs. (11) and (12) [28]. Interestingly, it was
shown that compatibility holds in the limit $A\gg V(\phi)^{1+\alpha}$ for
hilltop potentials, a situation where the inflationary features of the
generalized Chaplygin-inspired model can differ from the ones arising from the
General Relativity limit $A\ll V(\phi)^{1+\alpha}$. Monomial models are not
compatible with CMB data for $A\gg V(\phi)^{1+\alpha}$. We shall see below
that these results can be further restricted by the swampland conditions (2)
and (3).

Now let us assume that the potential satisfies condition (1) and investigate
if the generalized Chaplygin-inspired inflationary model described by Eqs.
(11) and (12) satisfies the de Sitter swampland conjectures (2) and (3) for a
generic form of the potential $V(\phi)$ and an arbitrary nonvanishing value of
the constant $A$.

From Eqs. (11) and (12) it is straightforward to obtain an expression for the
time derivative of the Hubble parameter, namely,

$\dot{H}=-\dfrac{1}{2M_{\rm
P}^{2}}\rho_{\phi}^{\alpha}(\rho_{\phi}+p_{\phi})\left(A+\rho_{\phi}^{1+\alpha}\right)^{-{\textstyle{{\alpha}\over{1+\alpha}}}},$
(13)

where $\rho_{\phi}=\dot{\phi}^{2}/2+V$ and $p_{\phi}=\dot{\phi}^{2}/2-V$ are
the energy density and pressure of the scalar field $\phi$, respectively.

In the slow-roll approximation, for which $\dot{\phi}^{2}/2\ll V$ and
$|\ddot{\phi}|\ll|3H\dot{\phi}|$, Eqs. (11), (12), and (13) can be written as

$\displaystyle H^{2}\simeq\dfrac{1}{3M_{\rm
P}^{2}}\left(A+V^{1+\alpha}\right)^{\textstyle{{1}\over{1+\alpha}}},$ (14)
$\displaystyle\dot{\phi}\simeq-\dfrac{M_{\rm
P}V^{\prime}}{\sqrt{3}}\left(A+V^{1+\alpha}\right)^{-{\textstyle{{1}\over{2(1+\alpha)}}}},$
(15)
$\displaystyle\dot{H}\simeq-\dfrac{1}{6}V^{\alpha}(V^{\prime})^{2}\left(A+V^{1+\alpha}\right)^{-1},$
(16)

where the symbol $\simeq$ means “equal within the slow-roll approximation”.

Using the above equations, the quantities $-\dot{H}/H^{2}$ and
$\ddot{\phi}/(H\dot{\phi})$ can be related to the slow-roll parameters
$\epsilon_{\phi}$ and $\eta_{\phi}$ defined in Eqs. (4) and (5), respectively.
We obtain

$\displaystyle-\dfrac{\dot{H}}{H^{2}}\simeq\epsilon_{\phi}\left(1+\dfrac{A}{V^{1+\alpha}}\right)^{-{\textstyle{{2+\alpha}\over{1+\alpha}}}},$
(17)
$\displaystyle\dfrac{\ddot{\phi}}{H\dot{\phi}}\simeq\epsilon_{\phi}\left(1+\dfrac{A}{V^{1+\alpha}}\right)^{-{\textstyle{{2+\alpha}\over{1+\alpha}}}}-\eta_{\phi}\left(1+\dfrac{A}{V^{1+\alpha}}\right)^{-{\textstyle{{1}\over{1+\alpha}}}}.$
(18)

Taking into account that in the slow-roll regime $|\dot{H}|/H^{2}\ll 1$ and
$|\ddot{\phi}/(H\dot{\phi})|\ll 1$, we conclude that

$\displaystyle\epsilon_{\phi}\ll\left(1+\dfrac{A}{V^{1+\alpha}}\right)^{{\textstyle{{2+\alpha}\over{1+\alpha}}}},$
(19)
$\displaystyle|\eta_{\phi}|\ll\left(1+\dfrac{A}{V^{1+\alpha}}\right)^{{\textstyle{{1}\over{1+\alpha}}}}.$
(20)

Note that the parameters $\epsilon_{\phi}$ and $\eta_{\phi}$ are related to
the constants $c_{2}$ and $c_{3}$ of the de Sitter swampland conjectures [see
Eqs. (2) and (3)] through the relations

$c_{2}^{2}<2\epsilon_{\phi}\quad\mbox{and}\quad c_{3}<|\eta_{\phi}|.$ (21)

In the regime $A\ll V^{1+\alpha}$, Eqs. (19) and (20) reduce to
$\epsilon_{\phi}\ll 1$ and $|\eta_{\phi}|\ll 1$, implying, in turn, that
$c_{2}\ll 1$ and $c_{3}\ll 1$, which goes against the de Sitter swampland
conjectures. This result coincides with that of General Relativity for the
single-field (cold) inflation case.

Much more interesting is the regime $A\gg V^{1+\alpha}$, for which Eqs. (19)
and (20) become

$\epsilon_{\phi}\ll\left(\dfrac{A}{V^{1+\alpha}}\right)^{{\textstyle{{2+\alpha}\over{1+\alpha}}}}\quad\mbox{and}\quad|\eta_{\phi}|\ll\left(\dfrac{A}{V^{1+\alpha}}\right)^{{\textstyle{{1}\over{1+\alpha}}}},$
(22)

implying that both $c_{2}$ and $c_{3}$ can be of order one during quasi-
exponential inflation, thus satisfying de Sitter swampland conjectures.

Compatibility of the generalized Chaplygin-inspired model with CMB data has
been investigated in Ref. [28]. It was found that for potentials of the form
$V(\phi)=V_{0}(\phi/M_{p})^{n}$ ($n=1,2$), compatibility with CMB data is not
possible for $x\equiv{\textstyle{{A}\over{V^{1+\alpha}}}}>1$. Hence, the
latter are incompatible with the swampland conjectures. On the other hand, for
quadratic and quartic hilltop potential models,
$V(\phi)=V_{0}(1-{\textstyle{{\gamma}\over{n}}}(\phi/M_{p})^{n})$ for
$0<\gamma<1$, $n=2,4$ and that satisfy swampland condition Eq. (1),
compatibility with CMB data was explicitly shown to be possible for
$\alpha=0.5$ or $1$ for a wide range (including large) values of $x$.
Therefore, these potentials can be made compatible with the de Sitter
swampland conditions.

In conclusion, in this work we have considered the generalized Chaplygin-
inspired inflationary model and shown that it is compatible with the swampland
conditions (2) and (3) provided the condition $A\gg V^{1+\alpha}$ is
satisfied. Interestingly, this limit corresponds to the situation where some
of the features of the model differ from the ones arising in the General
Relativity limit $A\ll V^{1+\alpha}$. As we have seen, compatibility of the
generalized Chaplygin-inspired inflationary model with the CMB data restricted
the allowed potentials. The de Sitter swampland conjectures impose a further
limitation on the possible potentials. In fact, these conjectures suggest that
one should stretch the model parameter $x$ to large values, far from the
General Relativity limit.

Acknowledgments

We gratefully acknowledge support from the Fundação para a Ciência e a
Tecnologia (Portugal) through the research grants doi.org/10.54499/CERN/FIS-
PAR/0027/2021 (OB), doi.org/10.54499/UIDB/00099/2020 (RP),
doi.org/10.54499/UIDB/04434/2020 and doi.org/ 10.54499/UIDP/04434/2020 (PMS).

## References

  * [1] C. Vafa, “The String Landscape and the Swampland”, hep-th/0509212.
  * [2] E. Palti, “The Swampland: Introduction and Review”, Fortschr. Phys. 67, 1900037 (2019).
  * [3] S. Kachru, R. Kallosh, A. Linde, and S. P. Trivedi, “de Sitter vacua in string theory”, Phys. Rev. D 68, 046005 (2003).
  * [4] J. A. Adams, G. G. Ross, and S. Sarkar, “Natural supergravity inflation”, Phys. Lett. B 391, 271(1997).
  * [5] O. Bertolami and G. G. Ross, “Inflation as a cure for the cosmological problems of superstring models with intermediate scale breaking”, Phys. Lett. B 183, 163(1987).
  * [6] N. Arkani-Hamed, L. Motl, A. Nicolis, and C. Vafa, “The String landscape, black holes and gravity as the weakest force,” J. High Energy Phys. 06, 060 (2007).
  * [7] T. Clifton, P. G. Ferreira, A. Padilla, and C. Skordis, “Modified Gravity and Cosmology,” Phys. Rept. 513, 1 (2012).
  * [8] O. Bertolami, C. G. Böhmer, T. Harko, and F. S. N. Lobo, “Extra force in $f(R)$ modified theories of gravity”, Phys. Rev. D 75, 104016 (2007).
  * [9] C. Gomes, J. G. Rosa, and O. Bertolami, “Inflation in non-minimal matter-curvature coupling theories”, J. Cosmol. Astropart. Phys. 06, 021 (2017).
  * [10] O. Bertolami, C. Gomes, and P. M. Sá, “Theories of gravity with nonminimal matter-curvature coupling and the de Sitter swampland conjectures”, Phys. Rev. D 107, 084009 (2023).
  * [11] O. Bertolami and V. Duvvuri, “Chaplygin inspired Inflation”, Phys. Lett. B 640, 121(2006).
  * [12] O. Bertolami, “Seeding the vacuum with entropy: the Chaplygin-like vacuum hypothesis”, Class. Quant. Grav. 40, 177002 (2023).
  * [13] O. Bertolami, “Inflation, phase transitions and the cosmological constant”, Gen. Relativ. Gravit. 53, 106 (2021).
  * [14] H. Ooguri and C. Vafa, “On the geometry of the string landscape and the swampland”, Nucl. Phys. B 766, 21(2007).
  * [15] G. Obied, H. Ooguri, L. Spodyneiko, and C. Vafa, “De Sitter Space and the Swampland”, arXiv:1806.08362 [hep-th] (2018).
  * [16] H. Ooguri, E. Palti, G. Shiu, and C. Vafa, “Distance and de Sitter conjectures on the Swampland”, Phys. Lett. B 788, 180(2019).
  * [17] S. K. Garg and C. Krishnan, “Bounds on slow roll and the de Sitter Swampland”, J. High Energy Phys. 11, 075 (2019).
  * [18] R. L. Workman et. al. [Particle Data Group], “Review of Particle Physics” Prog. Theor. Exp. Phys. 2022, 083C01 (2022).
  * [19] A. Achúcarro and G. A. Palma, “The string Swampland constraints require multi-field inflation”, J. Cosmol. Astropart. Phys. 02, 041 (2019).
  * [20] A. Berera, “Warm inflation”, Phys. Rev. Lett. 75, 3218(1995).
  * [21] M. Motaharfar, V. Kamali, and R. O. Ramos, “Warm inflation as a way out of the swampland”, Phys. Rev. D 99, 063513 (2019).
  * [22] R. Brandenberger, V. Kamali, and R. O. Ramos, “Strengthening the de Sitter Swampland conjecture in warm inflation”, J. High Energy Phys. 08, 127 (2020).
  * [23] O. Bertolami and P. M. Sá, “Multi-field cold and warm inflation and the de Sitter Swampland conjectures”, J. Cosmol. Astropart. Phys. 09, 001 (2022).
  * [24] A. Kamenshchik, U. Moschella, and V. Pasquier, “An alternative to quintessence”, Phys. Lett. B 511, 265 (2001).
  * [25] N. Bilić, G. B. Tupper, and R. D. Viollier, “Unification of dark matter and dark energy: the inhomogeneous Chaplygin gas”, Phys. Lett. B 535, 17 (2002).
  * [26] M. C. Bento, O. Bertolami, and A. A. Sen, “Generalized Chaplygin gas, accelerated expansion and dark energy-matter unification”, Phys. Rev. D 66, 043507 (2002).
  * [27] O. Bertolami, A. A. Sen, S. Sen, and P. T. Silva, “Latest supernova data in the framework of generalized Chaplygin gas model”, Mon. Not. Roy. Astron. Soc. 353, 329 (2004).
  * [28] C. Gomes, O. Bertolami, and J. G. Rosa, “Inflation with Planck data: A survey of some exotic inflationary models”, Phys. Rev. D 97, 104061 (2018).