11institutetext:  University of Bristol, Bristol BS8 1UB UK
11email<EMAIL_ADDRESS>22institutetext:
University Hospital Southampton, Southampton SO16 6YD UK
22email<EMAIL_ADDRESS>
# Offline Reinforcement Learning for Safer Blood Glucose Control in People
with Type 1 Diabetes

Harry Emerson 11 Matt Guy 22 Ryan McConville 11

###### Abstract

Hybrid closed loop systems represent the future of care for people with type 1
diabetes (T1D). These devices usually utilise simple control algorithms to
select the optimal insulin dose for maintaining blood glucose levels within a
healthy range. Online reinforcement learning (RL) has been utilised as a
method for further enhancing glucose control in these devices. Previous
approaches have been shown to reduce patient risk and improve time spent in
the target range when compared to classical control algorithms, but are prone
to instability in the learning process, often resulting in the selection of
unsafe actions. This work presents an evaluation of offline RL as a means for
developing clinically effective dosing policies without the need for patient
interaction. This paper examines the utility of BCQ, CQL and TD3-BC in
managing the blood glucose of nine virtual patients within the UVA/Padova
glucose dynamics simulator. When trained on less than a tenth of the data
required by online RL approaches, this work shows that offline RL can
significantly increase time in the healthy blood glucose range when compared
to the strongest state-of-art baseline. This is achieved without any
associated increase in low blood glucose events. Offline RL is also shown to
be able to correct for common and challenging scenarios such as incorrect
bolus dosing, irregular meal timings and sub-optimal training data. The code
for this work is available at: https://github.com/hemerson1/offline-glucose.

###### Keywords:

Type 1 diabetes Offline reinforcement learning Healthcare Reinforcement
learning

##  1 Introduction

Type 1 diabetes (T1D) is an autoimmune disease characterised by an
insufficiency of the hormone insulin, which is required for blood glucose
regulation. People with T1D must regularly monitor their blood glucose levels
and estimate the correct dosage of insulin to avoid dangerous instances of low
and high blood glucose. This includes taking bolus insulin to account for
ingested meal carbohydrates, in addition to adjusting basal insulin to account
for fluctuations between meals. Hybrid closed loop systems provide an
opportunity for people with T1D to automatically regulate their basal insulin
dosing [9, 1, 5]. These devices consist of an insulin pump connected to a
continuous glucose monitor (CGM) by a control algorithm. The CGM measures the
blood glucose level of the user and the control algorithm uses the CGM data to
instruct the insulin pump to deliver the required dosage. Trials of these
devices in patient populations have shown a significant association between
the use of hybrid closed loop systems and improvements to time spent in the
recommended blood glucose range [29, 17, 10].

The majority of commercially available hybrid closed loop systems utilise
predictive integral derivative (PID) controllers or model predictive
controllers (MPC) [24]. These algorithms are robust and easily interpretable,
but limit the utility of the devices. PID algorithms are prone to
overestimating insulin doses following meals and are unable to easily
incorporate additional factors which affect blood glucose, such as insulin
activity, time of day and exercise [28, 11]. In contrast, MPCs typically
utilise linear models of glucose dynamics, which are unable to capture the
full complexity of the task and struggle to predict future blood glucose over
the horizons required for effective control [18, 19]. Reinforcement learning
(RL) algorithms have been proposed as a means of addressing this problem;
using deep neural networks to construct sophisticated and personalised control
policies for individual patients. These algorithms have been demonstrated to
outperform their PID and MPC counterparts when trained and evaluated in
simulators of glucose dynamics but are impractical for clinical use in their
present state [33, 12, 13]. Current approaches predominantly utilise online RL
algorithms, which require interaction with a patient or simulator to develop
control policies. These agents typically start with a poor understanding of
their environment and are prone to learning instability, both of which can
contribute to the selection of dangerous insulin doses. This facet of online
RL limits its utility in real world hybrid closed loop systems; highlighting
the necessity for methods capable of learning accurate dosing policies without
the associated risk.

This work evaluates the use of offline RL as a method for developing safe and
high performing insulin dosing strategies without environmental interaction.
This entails a rigorous analysis of the offline RL algorithms: batch
constrained deep Q-learning (BCQ) [16], conservative Q-learning (CQL) [23] and
twin delayed deep deterministic policy gradient with behavioural cloning
(TD3-BC) [15] in their ability to improve blood glucose control in hybrid
closed loop systems. The presented algorithms are trained and tested on data
generated from a cohort of nine virtual patients and their performance and
sample-efficiency is scrutinised with respect to the current strongest online
RL and control baselines. Further examination is also performed to explore the
ability of offline RL for creating safer dosing policies in challenging
glucose control scenarios, such as accounting for an overdose in mealtime
insulin, responding to sporadic meal events and correcting for suboptimally
set PID parameters. This work shows that offline RL can yield safer and more
effective insulin dosing policies from significantly smaller samples of data
than required with the current standard of glucose control algorithms. This is
shown to be particularly significant when encountering potentially harmful
glucose control scenarios.

##  2 Related Work

Despite the prevalence of offline RL in other domains of medicine, its use in
glucose control has been limited [38]. Javad et al. used a q-learning
algorithm trained offline on samples of clinical data to select the optimal
daily basal dose for patients of a given demographic [20]. Shi et al.
presented an offline q-learning approach trained on the OhioT1DM dataset for
selecting discrete doses of basal insulin at hourly intervals [37]. Similarly,
Li et al. used a model-based RL algorithm to learn the blood glucose dynamics
of patients recovering from diabetic ketoacidosis [26]. This learned model was
then used to train an online q-learning algorithm to select the optimal basal
dose at three hour intervals over a 24 hour period. Although these approaches
presented methods for learning dosing policies offline, the timeframe over
which they act would be insufficient for managing the short-term blood glucose
fluctuations associated with meals and exercise. The most significant
application of offline RL for hybrid closed loop systems was presented in Fox,
in which the methods BCQ and bootstrapping error accumulation reduction (BEAR)
were compared to online RL approaches trained on an imperfect simulator of a
single adult patient [14]. Analysis showed that although offline RL was
capable of developing competent control policies, the achieved performance was
less than that obtained using an online approach trained on an imperfect
simulator.

Within online RL, several attempts have been made to address concerns around
safety and instability in glucose control. Fox et al. employed a transfer
learning approach to develop dosing policies from a general patient population
before fine-tuning them on target patients [13]. Lim et al. used a PID
controller to guide an online RL algorithm in the early stages of its
learning; progressively introducing a greater proportion of RL agent actions
[27]. Zhu et al. developed an online RL algorithm capable of integration with
a dual hormone pump, allowing control of both insulin and glucagon dosing and
hence for low blood glucose corrections to be made through glucagon infusion
[45]. These approaches all showed comparable or reduced time spent in the
potentially dangerous low blood glucose range when contrasted with MPC and PID
algorithms, but are limited by their reliance on glucose dynamics simulators.
These simulators represent a simplification of true blood glucose dynamics;
with almost all unable to incorporate common events such as exercise, stress
and illness. In moving towards use in hybrid closed loop systems, online RL
performance is likely to deteriorate as the complexity of the training
environment increases. In addition, glucose dynamics simulators allow for
unlimited training data generation. Of the approaches highlighted which
explicitly stated their sample size, the algorithms used 7,000 days ($>$19
years) [13] and 1,530 days ($>$4 years) [45] of simulated data to develop
personalised dosing policies. In an in vivo setting, allowing an RL algorithm
to control a patient’s blood glucose for several years without any associated
guarantee of safety would be unethical.

This work represents the first rigorous evaluation of offline RL for glucose
control; contrasting the performance of a diverse range of state-of-the-art
algorithms in a comprehensive cohort of virtual patients. Evaluation is
performed for a combined 12,150 virtual days (>33 years) and the presented
algorithms are evaluated inline with the current clinical guidance for
assessing patient glucose control. This analysis holds a particular focus on
exploring the practical and safety limitations of offline RL within hybrid
closed loop systems. In Silico evaluation of this nature is essential in
justifying trials of offline RL for glucose control in real patient
populations.

##  3 Methods

###  3.1 Problem Formulation

The task of glucose control in hybrid closed loop systems can be modelled as a
partially observable Markov decision process (POMDP) described by
($\mathcal{S}$, $\mathcal{A}$, $\Omega$, $\mathcal{T}$, $\mathcal{R}$,
$\mathcal{O}$). In each timestep $t$, the agent in a single state
$s\in\mathcal{S}$, performs an action $a\in\mathcal{A}$ and receives a reward
$r=\mathcal{R}(s)\in\mathbb{R}$ before transitioning into a new state
$s^{\prime}\in\mathcal{S}$ with probability
$P(s^{\prime}|s,a)=\mathcal{T}(s,a,s^{\prime})$. In a POMDP, the agent is
unable to directly view the next state $s^{\prime}\in\mathcal{S}$, and instead
receives an observation $o\in\Omega$ determined by probability
$P(o^{\prime}|s^{\prime},a)=\mathcal{O}(o^{\prime},s^{\prime},a)$ which
provides insight into the next state [35]. In the glucose control setting, the
state and action are defined by the patient blood glucose value $g_{t}$ and by
the basal dose $i_{t}$ in that timestep. The partial observability of the
state results from the inherent noise in the CGM devices used to make blood
glucose measurements and the dependency of blood glucose values on historical
data, such as ingested carbohydrates $c_{t}$, bolus doses $b_{t}$ and previous
blood glucose values [43]. Contextual information was incorporated in this
work by utilising the following state:

$s_{t}=[g_{t-1},g_{t-2},g_{t-3},...,g_{t-10},I_{t},M_{t}].$ (1)

This representation consists of the measured blood glucose values over the
previous four hour period at 30 minute intervals (each $g_{t}$), an estimation
of the combined insulin activity of basal and bolus insulin $I_{t}$ and an
estimation of carbohydrate activity $M_{t}$ given by [39]:

$I_{t}=\sum^{79}_{t^{\prime}=0}\left(1-\frac{t^{\prime}}{80}\right)\left[b_{t-t^{\prime}}+i_{t-t^{\prime}}\right],$
(2)
$M_{t}=\sum^{79}_{t^{\prime}=0}\left(1-\frac{t^{\prime}}{80}\right)\left[c_{t-t^{\prime}}\right].$
(3)

This state was selected in place of the full sequence of blood glucose,
carbohydrate and insulin data utilised in other approaches to avoid modifying
the offline RL methods to incorporate recurrency and to reduce state
dimensionality [13, 12, 44]. The reward for the agent was given by the
negative of the Magni risk function, which models the clinical risk for a
given blood glucose value [22]. An additional penalty of -1e5 was added for
blood glucose values beyond the realistically achievable range of 10 to 1,000
mg/dl. This modification was included as an incentive for the agent to not
purposefully terminate the environment and avoid future negative reward. The
parameters for the risk function are given as follows [13]:

$risk(b_{t})=10\cdot\left(3.5506\cdot\left(\log\left(b_{t}\right)^{0.8353}-3.7932\right)\right)^{2}.$
(4)

A reward function of this form ensures that low blood glucose events are
punished more severely than high blood glucose events; reflecting the greater
immediate risk low blood glucose events pose to patient health. The reward is
at a maximum when blood glucose is approximately in the centre of the target
range (70-180 mg/dl).

###  3.2 Offline Reinforcement Learning

In offline RL, the agent is incapable of environmental interaction and instead
must rely on samples generated by a demonstrator, such as a PID algorithm, to
build an understanding of the MDP [25]. These algorithms can be broadly
divided into model-free and model-based learning methods, which are
distinguished by the use of a dynamics model through which the transition
probability is approximated. This work applies the following model-free
approaches:

  * •

Batch Constrained Deep Q-learning (BCQ), which uses a variational auto-encoder
conditioned on the state to minimise the difference between selected actions
and those observed in the dataset [16].

  * •

Conservative Q-learning (CQL) prevents Q-function overestimation on out-of-
distribution state-action tuples by learning a conservative Q-function, which
acts as a lower bound on the true Q-value for a given policy [23]. Q-function
overestimation is of particular importance in safety-critical tasks such as
glucose control as it can result in the selection of potentially dangerous
out-of-distribution actions.

  * •

Twin Delayed DDPG with Behavioural Cloning (TD3-BC) modifies the established
off-policy online RL method TD3, to include a behavioural cloning term in its
loss; encouraging the policy to preferentially select actions observed in the
training distribution [15].

###  3.3 Baselines

The performance of the offline RL methods were compared to two baseline
algorithms: a tuned PID controller and the online RL method, recurrent soft
actor-critic (SAC-RNN). The PID algorithm is a robust control mechanism
capable of correcting for both short and long term deviations from a target
blood glucose value $g_{\text{target}}$ and operates according to [34]:

$i_{t}=k_{p}\cdot\left(g_{\text{target}}-g_{t}\right)+k_{i}\cdot\sum^{t}_{t^{\prime}=0}\left[g_{t^{\prime}}-g_{target}\right]+k_{d}\cdot\left(g_{t}-g_{t-1}\right),$
(5)

where $k_{p}$, $k_{i}$ and $k_{d}$ are parameters to be set. To ensure the
strongest comparison, the parameters were personalised to each patient and
were selected using a grid-search method to maximise the reward collected over
a 10-day test period. The SAC-RNN method represents one of the state-of-the-
art algorithms in glucose control and was recently presented in Fox et al.
[13]. The method used in this paper is a variation of this implementation,
using long-short-term memory (LSTM) layers in place of gated recurrent unit
(GRU) layers. This substitution was made as it was empirically found to
improve algorithmic performance and policy stability and thus provided a
stronger baseline.

###  3.4 Glucose Dynamics Simulation

The open source version of the UVA/Padova T1D glucose dynamics model was used
for training and evaluation in this work [42]. This software simulates the
human metabolic system using dietary models of glucose-insulin kinetics and is
designed as a substitute for pre-clinical trials in the development and
testing of T1D control algorithms [7]. The simulator can model a cohort of 30
virtual patients (10 children, 10 adolescents and 10 adults) and their
individualised responses to meal carbohydrates, basal/bolus insulin dosing and
interaction with CGM/pump devices. For the purpose of this study, all virtual
patients utilised a Dexcom CGM with a three-minute sampling rate joined to an
Insulet pump device. Basal doses were controlled by the RL/PID agent with
bolusing administered using the following controller [36]:

$I_{t}=\frac{c_{t}}{CR}+\left(\sum^{59}_{t^{\prime}=0}c_{t-t^{\prime}}=0\right)\cdot\left[\frac{g_{t}-g_{\text{target}}}{CF}\right],$
(6)

where $g_{\text{target}}=$ 144 mg/dl is the target blood glucose level and
corresponds to the greatest reward in the Magni risk function. In addition,
$CF$ and $CR$ are patient-specific parameters and were chosen from Fox et al.
[13]. Three meals and three snack events were included in the simulator with
the time and quantity of carbohydrate ingestion for each event being modelled
by a normal distribution.

###  3.5 Experimental Setup

Table 1: A description of the glycemic control metrics utilised for evaluation alongside their clinically recommended values. * For ages <25 this target decreases to >60%. Metric |  Description |  Target
---|---|---
Time-in-Range (TIR) |  The percentage time for which blood glucose measurements fall within the healthy glucose range (70-180 mg/dl). Increased TIR is strongly associated with a reduced risk of developing micro-vascular complications [3]. |  >70%* [2]
Time-Below-Range (TBR) |  The percentage time for which blood glucose measurements fall in the low blood glucose range (<70 mg/dl). Combined with TIR, this can additionally act as an indirect measure of time spent in the high blood glucose range (>180 mg/dl). |  <4% [2]
Coefficient of Variation (CV) |  The relative dispersion of blood glucose values around their mean. Increased CV is linked to an elevated risk of severe low blood glucose events (<54 mg/dl) and vascular tissue damage [6]. |  <36% [8]
Failure |  The percentage of test rollouts in which blood glucose levels reached values <10 mg/dl or >1000 mg/dl. For context, blood glucose <40 mg/dl is considered life-threatening and can result in major cardiovascular and cerebrovascular problems [21]. |  0%

####  3.5.1 Data Collection and Training.

Each algorithm was trained on $1\text{e}^{5}$ samples of data collected over
epochs of 10 days; the equivalent of 208 days of glucose data. Samples were
collected from nine simulated patients (three children, three adolescents and
three adults) and used to train each algorithm across three seeds. Training
and testing in adolescent and child populations is important for the adoption
of T1D technology as these groups are typically more susceptible to high blood
glucose events and increased glucose variability [31]. The training data was
collected using a PID algorithm tuned to achieve the maximum reward over a
10-day test period and noise was added to improve exploration within the
generated dataset. Basal noise was introduced by using an Ornstein-Uhlenbeck
process [4]. In addition, bolus noise was introduced by adding a 10%
estimation error to the simulated carbohydrate intake; this more closely
models the uncertainty observed in patient calculations of bolus doses. The
hyperparameters used for the offline RL algorithms were unchanged from their
original implementations.

####  3.5.2 Evaluation.

Performance was evaluated by monitoring blood glucose levels over a simulated
10-day test period and aggregating the results over three test seeds per
training seed. The metrics utilised for evaluation are given in Table 1 and
were used in addition to the sum of reward for assessing algorithmic
performance. Each measurement is accompanied by an error specifying the
standard error between test seeds. Further tests were employed to explore the
practical implications of using offline RL algorithms in hybrid closed loop
systems. This included evaluation on i) datasets containing demonstrations
from suboptimal PID demonstrators and ii) datasets of decreasing sample size.
These considerations are significant as patient insulin requirements are
continually changing and subsequently PID parameters are commonly suboptimal.
Similarly, patients are unlikely to adhere to lengthy periods of data
collection without gaining benefit within a reasonable timeframe, therefore
sample efficiency is of the utmost importance for implementation in hybrid
closed loop systems. Suboptimal PID parameters were selected as before but
instead of using the values yielding the highest reward, the parameters
producing the 10th and 20th greatest rewards were selected. To explore the
potential of offline RL for developing safer dosing policies, several common
scenarios were engineered to include features that are typically challenging
for users and control algorithms to account for. These included iii) a patient
consistently overestimating their bolus insulin for meals and iv) a patient
adhering to a more irregular meal schedule with greater uncertainty in meal
times.

##  4 Results

###  4.1 Offline Reinforcement Learning vs. Baseline Control Methods

Table 2: The mean performance of the offline RL algorithms: BCQ, CQL and TD3-BC against the online RL approach SAC-RNN and the control baseline PID. TD3-BC can be seen to significantly improve the proportion of patient time in the healthy glucose range (TIR) when compared to the PID and the SAC-RNN algorithms. This is done so without any associated increase in risk (Reward) or time in the low blood glucose range (TBR). $\dagger$, $\ddagger$ and $\mathsection$ indicate an offline RL, online RL and classical control algorithm respectively, with the best performing algorithm highlighted in bold. Algorithm | Reward | TIR (%) | TBR (%) | CV (%) | Failure (%)
---|---|---|---|---|---
BCQ$\dagger$ | -49,892 $\pm$ 907 | 65.8 $\pm$ 0.7 | 1.3 $\pm$ 0.4 | 40.5 $\pm$ 0.4 | 0.00
CQL$\dagger$ | -57,118 $\pm$ 2,153 | 55.6 $\pm$ 1.0 | 0.2 $\pm$ 0.1 | 35.2 $\pm$ 0.5 | 0.00
TD3-BC$\dagger$ | -48,205 $\pm$ 834 | 65.7 $\pm$ 0.7 | 0.2 $\pm$ 0.1 | 39.5 $\pm$ 0.4 | 0.00
SAC-RNN$\ddagger$ | -93,480 $\pm$ 11,255 | 29.5 $\pm$ 3.6 | 1.4 $\pm$ 0.4 | 31.5 $\pm$ 0.9 | 7.41
PID$\mathsection$ | -48,867 $\pm$ 843 | 63.5 $\pm$ 0.7 | 0.3 $\pm$ 0.1 | 37.0 $\pm$ 0.4 | 0.00

A comparison of the described offline RL methods with baseline approaches is
detailed in Table 2. Of the methods presented, the offline RL algorithm TD3-BC
achieved the best performance; obtaining the greatest reward and hence the
lowest Magni risk over the evaluation period. In addition, the algorithm
yields a $(2.2\pm 0.4)\%$ increase to TIR without any significant worsening in
TBR when compared to the PID algorithm. This equates to an additional 30
minutes per day in which patients would experience an improved quality of
life. BCQ also shows a similar level of improvement to TIR, however this is
coupled with a $(1.0\pm 0.1)\%$ increase to TBR. The use of BCQ and TD3-BC
also resulted in an increase to CV of $(3.6\pm 0.9)\%$ and $(2.5\pm 0.9)\%$
respectively. This constitutes a significant change as clinically this value
has been found to fall within the region of 31.0% to 42.3% for people with T1D
[32]. An elevated CV should indicate an increased risk of low blood glucose
events, however this is only observed in the BCQ approach. The origin of the
difference is most likely the use of the Magni risk function for defining
reward, as this value does not consider blood glucose variability within its
risk calculation.

The online RL algorithm SAC-RNN, performs comparably worse than the PID and
offline RL approaches in almost all metrics; terminating the environment and
thus harming the patient in 7.41% of the test rollouts. In this instance, CV
does fall within the recommended threshold of <36%, however this is likely a
consequence of patients having high blood glucose for $(69.2\pm 3.9)\%$ of the
evaluation period and therefore being closer to their blood glucose
equilibrium point [8]. The performance of SAC-RNN in this work significantly
differs from the results obtained previously under similar implementations
[13, 40]. This difference is most likely a result of differing evaluation
methods. In this work, SAC-RNN was trained for the full duration and evaluated
using the resulting weights, however in previous implementations performance
was measured on a validation environment after each episode and the highest
performing weights were selected for evaluation. One such online method
reported that using the final weights in place of the best performing weights
resulted in almost half of test rollouts ending in termination [13]. This work
elected to use the final weights for online evaluation, as it was concluded to
be more indicative of how the algorithm would be utilised in a patient
setting. Whereby, validating the performance of each online episode on a
patient would pose unnecessary risk.

###  4.2 Offline Reinforcement Learning Performance by Patient Age

Table 3: The mean glucose control performance by age group, as specified in
the UVA/Padova T1D Simulator. TD3-BC represents the only RL approach to
perform better than the PID across the three patient groups. The greatest
improvement is observed in adults where BCQ and TD3-BC yield a significant
increase to time in the target range, equivalent to an improvement of 85
mins/day and 65 mins/day. $\dagger$, $\ddagger$ and $\mathsection$ indicate an
offline RL, online RL and classical control algorithm respectively, with the
best performing algorithm highlighted in bold.

Algorithm | Reward | TIR (%) | TBR (%) | CV (%) | Failure (%)
---|---|---|---|---|---
BCQ$\dagger$ | -23,597 $\pm$ 419 | 73.6 $\pm$ 0.7 | 0.0 $\pm$ 0.0 | 31.9 $\pm$ 0.4 | 0.00
CQL$\dagger$ | -25,978 $\pm$ 375 | 66.7 $\pm$ 0.4 | 0.0 $\pm$ 0.0 | 29.5 $\pm$ 0.4 | 0.00
TD3-BC$\dagger$ | -25,532 $\pm$ 589 | 72.6 $\pm$ 0.6 | 0.2 $\pm$ 0.1 | 32.4 $\pm$ 0.4 | 0.00
SAC-RNN$\ddagger$ | -47,549 $\pm$ 3,828 | 43.8 $\pm$ 2.1 | 0.1 $\pm$ 0.0 | 25.7 $\pm$ 0.5 | 11.11
PID$\mathsection$ | -25,637 $\pm$ 410 | 68.0 $\pm$ 0.4 | 0.1 $\pm$ 0.0 | 27.4 $\pm$ 0.4 | 0.00
(a) Adults (aged 26 to 68 years).

Algorithm | Reward | TIR (%) | TBR (%) | CV (%) | Failure (%)
---|---|---|---|---|---
BCQ$\dagger$ | -46,939 $\pm$ 833 | 67.3 $\pm$ 0.7 | 1.7 $\pm$ 0.1 | 34.9 $\pm$ 0.4 | 0.00
CQL$\dagger$ | -46,002 $\pm$ 752 | 63.9 $\pm$ 1.7 | 0.1 $\pm$ 0.0 | 28.6 $\pm$ 0.6 | 0.00
TD3-BC$\dagger$ | -41,903 $\pm$ 774 | 69.0 $\pm$ 1.0 | 0.1 $\pm$ 0.1 | 34.6 $\pm$ 0.4 | 0.00
SAC-RNN$\ddagger$ | -90,230 $\pm$ 10,689 | 20.9 $\pm$ 5.6 | 0.0 $\pm$ 0.0 | 23.9 $\pm$ 0.7 | 11.11
PID$\mathsection$ | -42,001 $\pm$ 669 | 68.9 $\pm$ 0.4 | 0.1 $\pm$ 0.0 | 27.4 $\pm$ 0.4 | 0.00
(b) Adolescents (aged 14 to 19 years).

Algorithm | Reward | TIR (%) | TBR (%) | CV (%) | Failure (%)
---|---|---|---|---|---
BCQ$\dagger$ | -79,140 $\pm$ 1,470 | 56.6 $\pm$ 0.7 | 2.3 $\pm$ 0.3 | 54.7 $\pm$ 0.5 | 0.00
CQL$\dagger$ | -102,447 $\pm$ 3,255 | 36.2 $\pm$ 0.7 | 0.6 $\pm$ 0.2 | 47.5 $\pm$ 0.6 | 0.00
TD3-BC$\dagger$ | -77,180 $\pm$ 1,138 | 55.5 $\pm$ 0.5 | 0.3 $\pm$ 0.1 | 51.5 $\pm$ 0.5 | 0.00
SAC-RNN$\ddagger$ | -142,660 $\pm$ 19,249 | 23.7 $\pm$ 3.0 | 4.0 $\pm$ 1.1 | 44.9 $\pm$ 1.6 | 0.00
PID | -78,962 $\pm$ 1,449 | 53.6 $\pm$ 1.4 | 0.9 $\pm$ 0.2 | 52.4 $\pm$ 0.5 | 0.00
(c) Children (aged 7 to 12 years).

Table 3 presents a breakdown of glucose control performance by patient age. As
in Table 2, the offline RL algorithm TD3-BC performs the most consistently
across the patient groups, achieving a greater than or comparable reward to
the PID across all categories. The greatest improvements to TIR are observed
in adult patients, where BCQ and TD3-BC achieve an increase of $(5.6\pm
1.1)\%$ and $(4.6\pm 0.8)\%$ respectively. This observed difference is
significant enough to push TIR to within the recommended margin for that age
group, which if sustained would potentially lead to markedly better long-term
health outcomes for that population. It is also worth noting that in the adult
and adolescent group, TD3-BC and BCQ achieve values of CV within the
clinically recommended range, albeit elevated from those achieved under the
PID. The results in the child group are also particularly promising, whereby
the TD3-BC approach yields a $(1.9\pm 1.7)\%$ increase to TIR and a $(0.6\pm
0.3)\%$ reduction to TBR. Children represent one of the most challenging
control groups within the T1D simulator and in real life, as evidenced by the
significantly lower reward and greater glycemic variability obtained under the
PID in that group. Achieving an improvement of this magnitude is encouraging
for the transition of offline RL algorithms to real patient data in which
blood glucose relationships are likely to be more complex and depend on a
greater number of environmental factors. The TD3-BC algorithm was selected for
further evaluation due to the consistently high performance the approach
achieved across the nine virtual patients.

###  4.3 Addressing the Safety Concerns of Offline Reinforcement Learning in
Glucose Control

Figure 1: a) TD3-BC and PID performance for a varying number of training
samples. The offline RL approach TD3-BC can be seen to surpass PID performance
when trained on at least 5e4 samples of glucose data (100 days of glucose
data). With greater sample size, TD3-BC yields an increase to time in the
target glucose range (TIR) with an accompanying decrease to the proportion of
low blood glucose events (TBR). b) TD3-BC and PID performance for varying
levels of bolus overestimation. TD3-BC can be seen to avoid the pitfalls of
the PID algorithm; experiencing no significant increase to TBR for greater
overestimation.

PID was selected as the strongest safety benchmark as this algorithm has been
extensively evaluated in real patients and has been approved for clinical use
in hybrid closed loop systems across the world [41, 24].

####  4.3.1 Sample Size.

Figure 1 a) shows the effect of varying sample size on the performance of the
offline RL algorithm TD3-BC. TD3-BC achieves comparable or better TBR and
improvements to TIR for all sample sizes greater than or equal to 5e4
(approximately 100 days of glucose data). For the greatest number of samples
5e5 (> 1,000 days), TD3-BC achieves a $(2.2\pm 1.1)\%$ increase to TIR and a
$(0.2\pm 0.1)\%$ reduction to TBR. The presented findings are consistent with
the preliminary results in Fox, which show in a single adult patient that
capable glucose control policies can be developed from as little as two months
of data [14]. This result is significant for the application of offline RL to
future hybrid closed loop systems as 100 days represents a feasible timescale
for data collection in patient populations. This sample represents less than
one tenth of the data required for online RL approaches to surpass the PID
controller in glucose control [13, 45].

####  4.3.2 Meal Overestimation.

Figure 1 b) compares the ability of PID and TD3-BC in correcting for
overestimations in meal boluses. The TD3-BC approach can be seen to improve
glucose control in both TIR and TBR across all levels of overestimation. This
is particularly significant when considering a bolus overestimation of 40%,
which reduces TBR by $(0.6\pm 0.2)\%$. Mealtime miscalculation represents a
frequent problem for people with T1D. A study on the accuracy of bolus
calculations concluded that approximately 82% of participants overestimated
the carbohydrate content of common food choices, with the mean overestimation
amount being 40% [30]. In an in vivo setting, this would allow for the
correction of bias in bolus dosing without the inherent risk of using trial-
and-error to alter mealtime calculations or PID parameters manually.

Figure 2: a) TD3-BC and PID performance when trained on expert demonstrators
of different ability. TD3-BC can be seen to significantly improve TIR, while
achieving comparable or better TBR for all training policies. The reduction in
TBR is most significant when trained on the 20th best policy. b) TD3-BC and
PID performance in relation to greater uncertainty in meal times. TD3-BC
achieves significantly better performance when standard deviation is 60
minutes; significantly increasing TIR and reducing TBR over the evaluation
period.

####  4.3.3 Suboptimal Demonstrations.

Figure 2 a) shows the dependency of TD3-BC performance on the quality of the
training demonstrator. In all instances, TD3-BC can be seen to improve the
control of the demonstrator by a margin of at least $(1.3\pm 1.2)\%$ to TIR.
It also yielded a reduction of $(0.3\pm 0.2)\%$ to TBR when trained on the
10th ranked PID policy. The most significant difference is observed for the
20th ranked policy, whereby TD3-BC increases TIR by $(3.9\pm 1.4)$%, improves
CV by $(3.4\pm 1.1)\%$ and reduces TBR by $(4.7\pm 0.6)\%$. An improvement of
this magnitude could significantly improve the health outcomes of the user
without the need for manually altering PID parameters. The performance of the
TD3-BC approach does decline by a significant margin of $(13.2\pm 1.4)\%$ to
TIR and $(8.1\pm 0.7)\%$ for TBR between the 1st and 20th PID demonstrator.
Therefore, achieving glcyemic targets with offline RL in a hybrid closed loop
system would still be largely dependent on the performance of the demonstrator
in the training data.

####  4.3.4 Irregular Meal Schedules.

Figure 2 b) examines the ability of TD3-BC to exploit regular meal schedules
and adapt to greater uncertainty in meal timing. TD3-BC yields an improvement
in TIR of at least $(3.7\pm 1.2)\%$ regardless of meal time standard deviation
and without any significant worsening in TBR for non-zero meal deviation. The
performance of TD3-BC evidently improves with the removal of meal uncertainty
and snack events, as TIR is observed to increase by $(6.4\pm 1.1)\%$ (90
mins/day); the largest improvement across all the experiments presented in
section 4.3. This is also accompanied by a significant increase to TBR of
$(0.2\pm 0.2)\%$, which may explain the observed improvement. This
deterioration in policy could potentially be due to a lack of exploration in
the training samples, resulting from the high meal regularity. When using real
patient data, this flaw may become less apparent as this level of routine is
unlikely to be achievable in a realistic patient setting. Adapting dosing
policies to regular meal events may also transfer to other common routines in
daily life such as work schedules or exercise plans provided there is
sufficient contextual information in the state to intuit their occurrence.

##  5 Conclusion

This work examined the application of offline RL for safer basal insulin
dosing in hybrid closed loop systems. The experiments presented in this paper
demonstrated that the offline RL approaches BCQ, CQL and TD3-BC were capable
of learning effective control policies for adults, adolescents and children
simulated within the UVA/Padova T1D model. In particular, the TD3-BC approach
outperformed the widely-used and clinically validated PID algorithm across all
patient age groups with respect to TIR, TBR and glycemic risk. The improvement
was even more significant when TD3-BC was evaluated in potentially unsafe
glucose control scenarios. Further experiments on TD3-BC also highlighted the
ability of the approach to learn accurate and stable dosing policies from
significantly smaller samples of patient data than those utilised in current
online RL alternatives.

##  6 Future Work

This paper shows the potential of offline RL for creating safe and sample-
efficient glucose control policies in people with T1D. In moving towards an
algorithm capable of full implementation within hybrid closed loop systems
several avenues will first have to be explored. The most significant
limitation of the presented evaluation is the use of the T1D simulator. As
previously mentioned, these environments only capture a fraction of the
complexity involved in realistic blood glucose dynamics; neglecting events
such as stress, activity and illness. To confirm the scalability of offline RL
approaches to more complex environments, algorithms will have to be trained
and evaluated on real samples of patient data. This will require building on
the current state-of-art in the offline evaluation of RL algorithms. This
poses a particular challenge in glucose control compared to other fields of
medicine, as the nature of the task means the effect of actions may only
become apparent in some instances over several days, requiring blood glucose
to be modelled over long prediction horizons.

####  6.0.1 Acknowledgements

This work was supported by EPSRC Digital Health and Care Centre for Doctoral
Training (CDT) at the University of Bristol (UKRI grant no. EP/S023704/1).

## References

  * [1] Abraham, M.B., De Bock, M., Smith, G.J., et al.: Effect of a Hybrid Closed-Loop System on Glycemic and Psychosocial Outcomes in Children and Adolescents with Type 1 Diabetes: A Randomized Clinical Trial. JAMA Pediatrics (2021). https://doi.org/10.1001/jamapediatrics.2021.3965
  * [2] Battelino, T., Danne, T., Bergenstal, R.M., et al.: Clinical targets for continuous glucose monitoring data interpretation: Recommendations from the international consensus on time in range. Diabetes Care 42(8), 1593–1603 (8 2019). https://doi.org/10.2337/dci19-0028
  * [3] Beck, R.W., Bergenstal, R.M., Riddlesworth, T.D., et al.: Validation of time in range as an outcome measure for diabetes clinical trials. Diabetes Care 42(3), 400–405 (3 2019). https://doi.org/10.2337/dc18-1444
  * [4] Bibbona, E., Panfilo, G., Tavella, P.: The Ornstein-Uhlenbeck process as a model of a low pass filtered white noise. Metrologia 45(6) (12 2008). https://doi.org/10.1088/0026-1394/45/6/S17
  * [5] Breton, M.D., Kovatchev, B.P.: One Year Real-World Use of the Control-IQ Advanced Hybrid Closed-Loop Technology. Diabetes Technology and Therapeutics 23(9), 601–608 (9 2021). https://doi.org/10.1089/dia.2021.0097
  * [6] Ceriello, A., Monnier, L., Owens, D.: Glycaemic variability in diabetes: clinical and therapeutic implications (3 2019). https://doi.org/10.1016/S2213-8587(18)30136-0
  * [7] Dalla Man, C., Micheletto, F., Lv, D., et al.: The UVA/PADOVA type 1 diabetes simulator: New features. Journal of Diabetes Science and Technology 8(1), 26–34 (2014). https://doi.org/10.1177/1932296813514502
  * [8] Danne, T., Nimri, R., Battelino, T., et al.: International consensus on use of continuous glucose monitoring. Diabetes Care 40(12), 1631–1640 (12 2017). https://doi.org/10.2337/dc17-1600
  * [9] De Bock, M., McAuley, S.A., Abraham, M.B., et al.: Effect of 6 months hybrid closed-loop insulin delivery in young people with type 1 diabetes: A randomised controlled trial protocol. BMJ Open 8(8) (8 2018). https://doi.org/10.1136/bmjopen-2017-020275
  * [10] De Ridder, F., den Brinker, M., De Block, C.: The road from intermittently scanned continuous glucose monitoring to hybrid closed-loop systems. Part B: results from randomized controlled trials (2019). https://doi.org/10.1177/2042018819871903
  * [11] Forlenza, G.P.: Ongoing Debate about Models for Artificial Pancreas Systems and in Silico Studies (3 2018). https://doi.org/10.1089/dia.2018.0038
  * [12] Fox, I., Lee, J., Busui, R., Wiens, J.: Deep RL for Blood Glucose Control: Lessons, Challenges, and Opportunities (2019), https://openreview.net/forum?id=ryeN5aEYDH
  * [13] Fox, I., Lee, J., Pop-Busui, R., Wiens, J.: Deep Reinforcement Learning for Closed-Loop Blood Glucose Control (9 2020), http://arxiv.org/abs/2009.09051
  * [14] Fox, I.G.: Machine Learning for Physiological Time Series: Representing and Controlling Blood Glucose for Diabetes Management. Ph.D. thesis (2020)
  * [15] Fujimoto, S., Gu, S.S.: A Minimalist Approach to Offline Reinforcement Learning (6 2021), http://arxiv.org/abs/2106.06860
  * [16] Fujimoto, S., Meger, D., Precup, D.: Off-Policy Deep Reinforcement Learning without Exploration (12 2018), http://arxiv.org/abs/1812.02900
  * [17] Garg, S.K., Weinzimer, S.A., Tamborlane, W.V., et al.: Glucose Outcomes with the In-Home Use of a Hybrid Closed-Loop Insulin Delivery System in Adolescents and Adults with Type 1 Diabetes. Diabetes Technology and Therapeutics 19(3), 155–163 (3 2017). https://doi.org/10.1089/dia.2016.0421
  * [18] Hovorka, R., Allen, J.M., Elleri, D., et al.: Manual closed-loop insulin delivery in children and adolescents with type 1 diabetes: a phase 2 randomised crossover trial. The Lancet 375, 743–751 (2010). https://doi.org/10.1016/S0140, www.thelancet.com
  * [19] Incremona, G.P., Messori, M., Toffanin, C., et al.: Model predictive control with integral action for artificial pancreas. Control Engineering Practice 77, 86–94 (8 2018). https://doi.org/10.1016/j.conengprac.2018.05.006
  * [20] Javad, M.O.M., Agboola, S.O., Jethwani, K., Zeid, A., Kamarthi, S.: A reinforcement learning-based method for management of type 1 diabetes: Exploratory study (2019). https://doi.org/10.2196/12905
  * [21] Kalra, S., Mukherjee, J., Venkataraman, S., et al.: Hypoglycemia: The neglected complication. Indian Journal of Endocrinology and Metabolism 17(5), 819 (2013). https://doi.org/10.4103/2230-8210.117219
  * [22] Kovatchev, B.P., Cox, D.J., Gonder-Frederick, L.A., et al.: Symmetrization of the blood glucose measurement scale and its applications. Diabetes Care 20(11), 1655–1658 (1997). https://doi.org/10.2337/diacare.20.11.1655
  * [23] Kumar, A., Zhou, A., Tucker, G., et al.: Conservative Q-Learning for Offline Reinforcement Learning (6 2020), http://arxiv.org/abs/2006.04779
  * [24] Leelarathna, L., Choudhary, P., Wilmot, E.G., Lumb, A., Street, T., Kar, P., Ng, S.M.: Hybrid closed-loop therapy: Where are we in 2021? Diabetes, Obesity and Metabolism 23(3), 655–660 (3 2021). https://doi.org/10.1111/dom.14273
  * [25] Levine, S., Kumar, A., Tucker, G., et al.: Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (5 2020), http://arxiv.org/abs/2005.01643
  * [26] Li, T., Wang, Z., Lu, W., et al.: Electronic health records based reinforcement learning for treatment optimizing. Information Systems 104 (2 2022). https://doi.org/10.1016/j.is.2021.101878
  * [27] Lim, M.H., Lee, W.H., Jeon, B., et al.: A Blood Glucose Control Framework Based on Reinforcement Learning with Safety and Interpretability: In Silico Validation. IEEE Access 9, 105756–105775 (2021). https://doi.org/10.1109/ACCESS.2021.3100007
  * [28] Marchetti, G., Barolo, M., Jovanovic, L., et al.: An improved PID switching control strategy for type 1 diabetes. IEEE Transactions on Biomedical Engineering 55(3), 857–865 (3 2008). https://doi.org/10.1109/TBME.2008.915665
  * [29] McAuley, S.A., Lee, M.H., Paldus, B., et al.: Six months of hybrid closed-loop versus manual insulin delivery with fingerprick blood glucose monitoring in adults with type 1 diabetes: A randomizedcontrolled trial. Diabetes Care 43(12), 3024–3033 (12 2020). https://doi.org/10.2337/dc20-1447
  * [30] Meade, L.T., Rushton, W.E.: Accuracy of carbohydrate counting in adults. Clinical Diabetes 34(3), 142–147 (2016). https://doi.org/10.2337/diaclin.34.3.142
  * [31] Miller, K.M., Foster, N.C., Beck, R.W., et al.: Current state of type 1 diabetes treatment in the U.S.: Updated data from the t1d exchange clinic registry. Diabetes Care 38(6), 971–978 (6 2015). https://doi.org/10.2337/dc15-0078
  * [32] Monnier, L., Colette, C., Wojtusciszyn, A., et al.: Toward defining the threshold between low and high glucose variability in diabetes. Diabetes Care 40(7), 832–838 (7 2017). https://doi.org/10.2337/dc16-1769
  * [33] Myhre, J.N., Tejedor, M., Launonen, I.K., El Fathi, A., Godtliebsen, F.: In-silico evaluation of glucose regulation using policy gradient reinforcement learning for patients with type 1 diabetes mellitus. Applied Sciences (Switzerland) 10(18), 1–21 (2020). https://doi.org/10.3390/APP10186350
  * [34] Ngo, P.D., Wei, S., Holubová, A., Muzik, J., Godtliebsen, F.: Control of Blood Glucose for Type-1 Diabetes by Using Reinforcement Learning with Feedforward Algorithm (2018). https://doi.org/10.1155/2018/4091497
  * [35] Omidshafiei, S., Pazis, J., Amato, C., et al.: Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability. Tech. rep. (2017)
  * [36] Schmidt, S., Norgaard, K.: Bolus calculators (1 2014). https://doi.org/10.1177/1932296814532906
  * [37] Shi, C., Luo, S., Zhu, H., Song, R.: Statistically Efficient Advantage Learning for Offline Reinforcement Learning in Infinite Horizons (2 2022), http://arxiv.org/abs/2202.13163
  * [38] Tejedor Hernandez, M.A., Myhre, J.N.: Controlling Blood Glucose For Patients With Type 1 DiabetesUsing Deep Reinforcement Learning – The Influence OfChanging The Reward Function (2020). https://doi.org/10.7557/18.5166
  * [39] Toffanin, C., Zisser, H., Iii, F.J.D., Dassau, E.: Dynamic Insulin on Board: Incorporation of Circadian Insulin Sensitivity Variation. Tech. Rep. 4 (2013), www.journalofdst.org
  * [40] Viroonluecha, P., Egea-Lopez, E., Santa, J.: Evaluation of Blood Glucose Level Control in Type 1 Diabetic Patients Using Deep Reinforcement Learning (2021). https://doi.org/10.21203/rs.3.rs-1095721/v1, https://doi.org/10.21203/rs.3.rs-1095721/v1
  * [41] Weisman, A., Bai, J.W., Cardinez, M., et al.: Effect of artificial pancreas systems on glycaemic control in patients with type 1 diabetes: a systematic review and meta-analysis of outpatient randomised controlled trials. The Lancet Diabetes and Endocrinology 5(7), 501–512 (7 2017). https://doi.org/10.1016/S2213-8587(17)30167-5
  * [42] Xie, J.: Simglucose v0.2.1 (2018), https://github.com/jxx123/simglucose
  * [43] Xie, X., Doloff, J.C., Yesilyurt, V., et al.: Reduction of measurement noise in a continuous glucose monitor by coating the sensor with a zwitterionic polymer. Nature Biomedical Engineering 2(12), 894–906 (12 2018). https://doi.org/10.1038/s41551-018-0273-3
  * [44] Zhu, T., Li, K., Georgiou, P.: A Dual-Hormone Closed-Loop Delivery System for Type 1 Diabetes Using Deep Reinforcement Learning (10 2019), http://arxiv.org/abs/1910.04059
  * [45] Zhu, T., Li, K., Herrero, P., Georgiou, P.: Basal Glucose Control in Type 1 Diabetes Using Deep Reinforcement Learning: An in Silico Validation (2021). https://doi.org/10.1109/JBHI.2020.3014556