A Deployed Online Reinforcement Learning Algorithm In An Oral Health Clinical Trial
Anna L. Trella Kelly W. Zhang Hinal Jajal Inbal Nahum-Shani Vivek Shetty Finale Doshi-Velez Susan A. Murphy


Embedding theory contributions to average atom models for warm dense matter
    David A. Strubbe
    September 9, 2024
===========================================================================



§ ABSTRACT 
    Dental disease is a prevalent chronic condition associated with substantial financial burden, personal suffering, and increased risk of systemic diseases. Despite widespread recommendations for twice-daily tooth brushing, adherence to recommended oral self-care behaviors remains sub-optimal due to factors such as forgetfulness and disengagement. To address this, we developed Oralytics, a mHealth intervention system designed to complement clinician-delivered preventative care for marginalized individuals at risk for dental disease. Oralytics incorporates an online reinforcement learning algorithm to determine optimal times to deliver intervention prompts that encourage oral self-care behaviors. We have deployed Oralytics in a registered clinical trial. The  deployment required careful design to manage challenges specific to the clinical trials setting in the U.S. In this paper, we (1) highlight key design decisions of the RL algorithm that address these challenges and (2) conduct a re-sampling analysis to evaluate algorithm design decisions.
    
    A second phase (randomized control trial) of Oralytics is planned to start in spring 2025.






























§ INTRODUCTION


Dental disease is a prevalent chronic condition in the United States with significant preventable morbidity and economic impact <cit.>. Beyond its associated pain and substantial treatment costs, dental disease is linked to systemic health complications such as diabetes, cardiovascular disease, respiratory illness, stroke, and adverse birth outcomes. To prevent dental disease, the American Dental Association recommends systematic, twice-a-day tooth brushing for two minutes <cit.>. 
However, patient adherence to this simple regimen is often compromised by factors such as forgetfulness and lack of motivation <cit.>.



mHealth interventions and tools   can  be leveraged to prompt individuals to engage in high-quality oral self-care behaviors (OSCB) between clinic visits. This work focuses on Oralytics, a mHealth intervention designed to improve OSCB for individuals at risk for dental disease.

The intervention involves (i) a Bluetooth-enabled toothbrush to collect sensor data on an individual's brushing quality, and (ii) a smartphone application (app) to deliver treatments, one of which is engagement prompts to encourage individuals to remain engaged in improving their OSCB.
See Figure <ref> for screenshots from the  Oralytics app.
Oralytics includes multiple intervention components one of which is  an  online reinforcement learning (RL) algorithm  which is used to learn, online, a policy specifying when it is most useful to deliver engagement prompts. 
The algorithm should  avoid excessive burden and habituation by only  sending prompts at times they are  likely to be effective.
Before integrating a mHealth intervention

into broader healthcare programs, the  effectiveness of the intervention is deployed and tested in a clinical trial. However, the clinical trial setting introduces unique challenges for the design and deployment of online RL algorithms as part of the intervention.

 
might be best to delete this paragraph to prevent reader from going off-track. Including an online (as opposed to batch) reinforcement learning algorithm as part of the mHealth intervention is attractive when there are societal or other health trends that result in cohort effects  cite Rothman, K. J., Greenland, S., and Lash, T. L. (2008). Modern epidemiology (Third edition). Wolters Kluwer Health/Lippincott Williams and Wilkins or there is otherwise an absence of high quality batch data.  In these cases offline algorithms that learn from such batch data can result in poor performing policies when deployed.  







































































 §.§ Design & Deployment Challenges in Clinical Trials









First, clinical trials, conducted with US National Institutes of Health (NIH) funding, must adhere to the NIH policy on the dissemination of NIH-funded clinical trials <cit.>.
This policy requires pre-registration of the trial in order to enhance  transparency and replicability of trial results (Challenge 1).  The design of the health intervention, including any online algorithms that are components of the intervention, must be pre-registered. 


Indeed, changing any of the intervention components, including the online algorithm, during the conduct of the trial, makes it difficult for other scientists to know exactly what intervention was implemented and to replicate any results.   

Thus to enhance transparency and replicability, the online algorithm should be autonomous.  That is, the potential for major ad hoc changes that  alter the pre-registered protocol  should be minimized. 












    Second, the current setting involves  running an online algorithm which learns and updates the policy using incoming data throughout the trial. However,  by design, each individual only receives the mHealth intervention for a limited amount of time. 
    
    This proposes a challenge to the RL algorithm's ability to learn based on data from any single individual (Challenge 2).




Second, while the online algorithm learns and updates the policy using incoming data throughout the trial, the algorithm has, in total, a limited amount of data to learn from. 
By design, each individual only receives the mHealth intervention for a limited amount of time. 
Therefore, the RL algorithm only has data on a limited number of decision times for an individual.
This poses a challenge to the RL algorithm's ability to learn based on a small amount of data collected per individual (Challenge 2).

 
 
 













 §.§ Contributions

In this paper, we discuss how we addressed these deployment challenges in the design of an online RL algorithm – a generalization of a Thompson-sampling contextual bandit (Section <ref>) - as part of the Oralytics intervention to improve OSCB for individuals at risk for dental disease. The RL algorithm (1) learns online from incoming data and (2) makes decisions for individuals in real time as part of the intervention. Recently, the Oralytics intervention was deployed in a registered clinical trial <cit.>. Key contributions of our paper are:

    
  * We highlight key design decisions made for the Oralytics algorithm that deals with deploying an online RL algorithm as part of an intervention in a clinical trial (Section <ref>).
    
  * We conduct a re-sampling analysis[All code used in this paper can be found in GitHub: https://github.com/StatisticalReinforcementLearningLab/oralytics-post-deployment-analysis/tree/mainhere] using data collected during the trial to (1) re-evaluate design decisions made and (2) investigate algorithm behavior (Section <ref>). 
    



Further details about the clinical trial and algorithm design decisions can be found in <cit.>.


§ RELATED WORK






  
AI in Clinical Trials
A large body of work exists that incorporates AI algorithms to conduct clinical trials. AI can improve trial execution by automating cohort selection <cit.> and participant eligibility screening <cit.>. 

Prediction algorithms can be used to assist in maintaining retention by identifying participants who are at high risk of dropping out of the trial <cit.>. 
Recently, generative models have been considered to create digital twins <cit.> 
of participants to predict participant outcomes or simulate other behaviors. Online algorithms in adaptive trial design <cit.> can lead to more efficient trials (e.g., time and money saved, fewer participants required) by modifying the experiment design in real-time (e.g., abandoning treatments or redefining sample size). 
The above algorithms are part of the clinical trial design (experimental design)

while in our setting, the RL algorithm is a component of the  intervention.  






  
Online RL Algorithms in mHealth











Many online RL algorithms have been included in mHealth interventions deployed in a clinical trial. For example, online RL was used to optimize the delivery of prompts to encourage physical activity <cit.>, 


manage weight loss <cit.>, improve medical adherence <cit.>, 
assist with pain management <cit.>, reduce cannabis use amongst emerging adults <cit.>, and help people quit smoking <cit.>. 
There are also deployments of online RL in mHealth settings that are not formally registered clinical trials <cit.>. 


Many of these papers focus on algorithm design before deployment.  
Some authors <cit.>, compare outcomes between groups of individuals where each group is assigned a different algorithm or policy.   




Here we use a different analysis to inform further design decisions. Our analysis focuses on learning across time by a single online RL algorithm.  






























§ PRELIMINARIES






 §.§ Oralytics Clinical Trial

The Oralytics clinical trial (Table <ref>) enrolled participants recruited from UCLA dental clinics in Los Angeles[
The study protocol and consent procedures have been approved by the University of California, Los Angeles Institutional Review Board (IRB#21–001471) and the trial was registered on ClinicalTrials.gov (NCT05624489).]. Participants were recruited incrementally at about 5 participants every 2 weeks.
All participants received an electric toothbrush with WiFi and Bluetooth connectivity and integrated sensors. Additionally, they were instructed to download the Oralytics app on their smartphones. 



The RL algorithm dynamically decided whether to deliver an engagement  prompt for each participant twice daily, with delivery within an hour preceding self-reported morning and evening brushing times.



The clinical trial began in September 2023 and was completed in July 2024. A total of 79 participants were enrolled over approximately 20 weeks, with each participant contributing data for 70 days. 
However, due to an engineering issue, data for 7 out of the 79 participants was incorrectly saved and thus their data is unviable. Therefore, we restrict our analyses (in Section <ref>) to data from the 72 unaffected participants. For further details concerning the trial design, see <cit.> and <cit.>.



 §.§ Online Reinforcement Learning

Here we consider a setting involving sequential decision-making for N participants, each with T decision times.
Let subscript i ∈ [1:N] denote the participant and subscript t∈ [1:T] denote the decision time.  S_i, t denotes the current state of the participant.  At each decision time t, the algorithm selects action A_i, t after observing S_i, t, based on its policy π_θ(s) which is a function, parameterized by θ, that takes in input state s. After executing action A_i, t, the algorithm receives a reward R_i, t. In contrast to batch RL, where policy parameters are learned using previous batch data and fixed for all t ∈ [1: T], online RL learns the policy parameters with incoming data. At each update time τ, the algorithm updates parameters θ using the entire history of state, action, and reward tuples observed thus far ℋ_τ. The goal of the algorithm is to maximize the average reward across all participants and decision times, 𝔼[ 1/N· T∑_i = 1^N ∑_t = 1^T R_i, t].



 §.§ Oralytics RL Algorithm


The Oralytics RL algorithm is a generalization of a Thompson-Sampling contextual bandit algorithm <cit.>. The algorithm makes decisions at each of the T = 140 total decision times (2 every day over 70 days) on each participant.
The algorithm state (Table <ref>) includes current context information about the participant collected via the toothbrush and app (e.g., participant OSCB over the past week and prior day app engagement).
The RL algorithm makes decisions regarding whether or not to deliver an engagement prompt to  each participant twice daily, one hour before a participant's self-reported usual morning and evening brushing times. Thus the action space is binary, with A_i, t=1 denoting delivery of the prompt and A_i, t=0, otherwise. 

The reward, R_i, t, is constructed based on the proximal health outcome OSCB, Q_i, t, and a tuned approximation to the effects of actions on future states and rewards. This reward design allows a contextual bandit algorithm to approximate an RL algorithm that models the environment as a Markov decision process.
See <cit.> for more details on the reward designed for Oralytics. 


As part of the policy, contextual bandit algorithms use a model of the mean reward given state s and action a, parameterized by θ: r_θ(s, a). We refer to this as the reward model. While one could learn and use a reward model per participant i, in Oralytics, we ran a full-pooling algorithm (Section <ref>) that learns and uses a single reward model shared between all participants in the trial instead. In Oralytics, the reward model r_θ(s, a) is a linear regression model as in <cit.> (See Appendix <ref>). The Thompson-Sampling algorithm is Bayesian and thus the algorithm has a prior distribution θ∼𝒩(μ^prior, Σ^prior) assigned to parameter θ. See Appendix <ref> for the prior designed for Oralytics.


The RL algorithm updates the posterior distribution for parameter θ once a week on Sunday morning using all participants' data observed up to that time; denote these weekly update times by τ. 
Let n_τ be the number of participants that have started the trial before update time τ, and t(i, τ) be a function that takes in participant i and current update time τ and outputs the last decision time for that participant. Then to update posterior parameters μ^post_τ, Σ^post_τ, we use the history ℋ_τ := {(S_i, t', A_i, t', R_i, t')}_i = 1, t' = 1^n_τ, t(i, τ). 
Thus the RL algorithm is a full-pooling algorithm that pools observed data, ℋ_τ from all participants to update posterior parameters μ^post_τ, Σ^post_τ of θ.



Notice that due to incremental recruitment of trial participants, at a particular update time τ, not every participant will be on the same decision time index t and the history will not necessarily involve all N participants' data.







To select actions, the RL algorithm uses the latest reward model to model the advantage, or the difference in expected rewards, of action 1 over action 0 for a given state s. 
Since the reward model for Oralytics is linear, the model of the advantage is also linear:


    r_θ(s, a=1) - r_θ(s, a=0) = f(s)^⊤β


f(s) denotes the features used in the algorithm’s model for the advantage (See Table <ref>), and β is the subset of parameters of θ corresponding to the advantage. 

For convenience, let τ = τ(i, t) be the last update time corresponding to the current reward model used for participant i at decision time t. The RL algorithm micro-randomizes actions using ℙ(f(s)^⊤β > 0  | s = S_i, t, ℋ_τ) and therefore forms action-selection probability π_i, t:

    π_i,t := 𝔼_β∼𝒩(μ^β_τ, Σ^β_τ )[ρ(f(s)^⊤β) | s = S_i, t, ℋ_τ]

where μ^β_τ and Σ^β_τ are the sub-vector and sub-matrix of μ^post_τ and Σ^post_τ corresponding to advantage parameter β.

Notice that while classical posterior sampling uses an indicator function for ρ, the Oralytics RL algorithm instead uses a generalized logistic function for ρ to ensure that policies formed by the algorithm concentrate and enhance the replicability of the algorithm <cit.>. 


































Finally, the RL algorithm samples A_i, t from a Bernoulli distribution with success probability π_i, t:

    A_i, t|π_i, t∼Bern(π_i, t)












§ DEPLOYING ORALYTICS







 §.§ Oralytics Pipeline



  
Software Components

Multiple software components form the Oralytics software service. These components are (1) the main controller, (2) the Oralytics app, and (3) the RL service. The main controller is the central coordinator of the Oralytics software system that handles the logic for (a) enrolling participants, (b) pulling and formatting sensor data (i.e., brushing and app analytics data), and (c) communicating with the mobile app to schedule prompts for every participant. The Oralytics app is downloaded onto each participant's smartphone at the start of the trial. The app is responsible for (a) obtaining prompt schedules for the participant and scheduling them in the smartphone's internal notification system and (b) providing app analytics data to the main controller. The RL service is the software service supporting the RL algorithm to function properly and interact with the main controller. The RL service executes three main processes: (1) batch data update, (2) action selection, and (3) policy update. 

The main controller and RL service were deployed on infrastructure hosted on Amazon Web Services (AWS). Specifically, the RL service was wrapped as an application using Flask. A daily scheduler job first triggered the batch data update procedure and then the action-selection procedure and a weekly scheduler job triggered the policy update procedure. The Oralytics app was developed for both Android and iOS smartphones.



  
End-to-End Pipeline Description
We now describe interactions between clinical staff with components of the Oralytics software system  and between software components (See Figure <ref>). The Oralytics clinical trial staff recruits and registers participants (Step 1). The registration process consists of the participant downloading the Oralytics app and staff verifying that the participant had at least one successful brushing session from the toothbrush. Successfully registered participants are then entered into the participant enrollment database maintained by the main controller. The main controller maintains this database to track participants entering and completing the trial (i.e.,  at 70 days).

Every morning, a daily scheduler job first triggers the batch data update process and then the action-selection process (Step 2). The RL service begins by fetching the list of participants currently in the trial (Step 3) and the latest sensor data (i.e., brushing and app analytics data) for current participants (Step 4) from the main controller. 
Notice that this data contains rewards to be associated with previous decision times as well as current state information.
Rewards are matched with the correct state and action and these state, action, and reward tuples corresponding to previous decision times are added to the RL service's internal batch data table (Step 5).  During the action-selection process, the RL service first uses the latest sensor data to form states for all current participants (Step 6). Then, the RL service uses these states and the current policy to create a new schedule of actions for all current participants (Step 7). 
These states and actions are saved to the RL internal database to be added to the batch data table during Step 5, the next morning.
All new schedules of actions are pushed to the main controller and processed to be fetched (Step 8). When a participant opens their Oralytics app, the app fetches the new prompt schedule from the main controller and schedules prompts as notification messages in the smartphone's internal notification system (Step 9).

Every Sunday morning, a weekly scheduler job triggers the policy update process (Step 10). During this process, the RL system takes all data points (i.e., state, action, and reward tuples) in the batch data table and updates the policy (Step 11). Recall that the Oralytics RL algorithm is a Thompson sampling algorithm which means policy updates involve updating the posterior distribution of the reward model parameters (Section <ref>). The newly updated posterior distribution for the parameters is used to select treatments for all participants and all decision times for that week until the next update time.

Every morning, the Oralytics pipeline (Steps 6-8) produces a full 70-day schedule of treatment actions  for each participant starting at the current decision time (as opposed to a single action for the current decision time). The schedule of actions is a key design decision for the Oralytics system that enhances the transparency and replicability of the trial (Challenge 1). 
Specifically, this design decision mitigates networking or engineering issues if: (1) a new schedule of actions fails to be constructed or (2) a participant does not obtain the most recent schedule of actions. We further see the impact of this design decision during the trial in Section <ref>.



 §.§ Design Decisions To Enhance Autonomy and Thus Replicability


A primary challenge in our setting is the high standard for replicability and as a result the algorithm, and its components, should be autonomous (Challenge 1). However, unintended engineering or networking issues could arise during the trial. These issues could cause the intended RL system to function incorrectly compromising: (1) participant experience and (2) the quality of data for post-trial analyses. 


One way Oralytics dealt with this constraint is by implementing fallback methods. Fallback methods are pre-specified backup procedures, for action selection or updating, which are executed when an issue occurs. Fallback methods are part of a larger automated monitoring system <cit.> 

that detects and addresses issues impacting or caused by the RL algorithm in real-time. Oralytics employed the following fallback methods: 

    
  *  
    if any issues arose with a participant not obtaining the most recent schedule of actions, then the action for the current decision time will default to the action for that time from the last schedule pushed to the participant's app.
    
  *  if any issues arose with constructing the schedule of actions, then the RL service forms a schedule of actions where each action is selected with probability 0.5 (i.e., does not use the policy nor state to select action).
    
  *  for updating, if issues arise (e.g., data is malformed or unavailable), then the algorithm stores the data point, but does not add that data point to the batch data used to update parameters.






















 §.§ Design Decisions Dealing with Limited Decision Times Per Individual




Each participant is in the Oralytics trial for a total of 140 decision times, which results in a small amount of data collected per participant. Nonetheless, the RL algorithm needs to learn and select quality actions based on data from a limited number of decision times per participant (Challenge 2).






















A design decision to deal with limited data is full-pooling. Pooling refers to clustering participants and pooling all data within a cluster to update the cluster's shared policy parameters. Full pooling refers to pooling all N participants' data together to learn a single shared policy. 

Although participants are likely to be heterogeneous (reward functions are likely different), we chose a full-pooling algorithm like in <cit.> to trade off bias and variance in the high-noise environment of Oralytics. These pooling algorithms can reduce noise and speed up learning.    

We finalized the full-pooling decision after conducting experiments comparing no pooling (i.e., one policy per participant that only uses that participant's data to update) and full pooling. We expected the no-pooling algorithm to learn a more personalized policy for each participant later in the trial if there were enough decision times, but the algorithm is unlikely to perform well when there is little data for that participant. Full pooling may learn well for a participant's earlier decision times because it can take advantage of other participants' data, but may not personalize as well as a no-pooling algorithm for later decision times, especially if participants are heterogeneous. In extensive experiments, using simulation environments based on data from prior studies,  we found that full-pooling algorithms achieved higher average OSCB  than no-pooling algorithms across all variants of the simulation environment (See Table 5 in <cit.>).





§ APPLICATION PAYOFF


We conduct simulation and re-sampling analyses using data collected during the trial to evaluate design decisions made for our deployed algorithm. We focus on the following questions:


    
  * Was it worth it to invest in fallback methods? (Section <ref>)
    
  * Was it worth it to run a full-pooling algorithm? (Section <ref>) 
    
  * Despite all these challenges, did the algorithm learn? (Section <ref>) 




 §.§ Simulation Environment


One way to answer questions 2 and 3 is through a simulation environment built using data collected during the Oralytics trial. The purpose of the simulation environment is to re-simulate the trial by generating participant states and outcomes close to the distribution of the data observed in the real trial. This way, we can (1) consider counterfactual decisions (to answer Q2) and (2) have a mechanism for resampling to assess if evidence of learning by the RL algorithm is  due to random chance and thus spurious (to answer Q3). 

For each of the N=72 participants with viable data from the trial, we fit a model which is used to simulate OSCB outcomes.
Q_i, t given current state S_i, t and an action A_i, t. We also modeled participant app opening behavior and simulated participants starting the trial using the exact date the participant was recruited in the real trial. 
See Appendix <ref> 
for full details on the simulation environment.





 §.§ Was it worth it to invest in fallback methods?
 





During the Oralytics trial, various engineering or networking issues (Table <ref>) occurred that impacted the RL service's intended functionality. These issues were automatically caught and the pre-specified fallback method was executed. Figure <ref> shows that all 3 types of fallback methods were executed over the Oralytics trial.
Notice that fallback method (i), made possible by our design decision to produce a schedule of actions instead of just a single action, was executed 4 times during the trial and mitigated issues for more participants than any other method.
While defining and implementing fallback methods may take extra effort by the software engineering team, this is a worthwhile investment. Without fallback methods, the various issues that arose during the trial would have required ad hoc changes, to the RL algorithm reducing autonomy and thus replicability of the intervention. 



 §.§ Was it worth it to pool?





Due to the small number of decision points (T=140) per participant, the RL algorithm was a full-pooling algorithm (i.e., used a single reward model for all participants and updated using all participants' data). Even though before deployment we anticipated that trial participants would be heterogeneous (i.e., have different outcomes to the intervention), we still believed that full-pooling would learn better over a no-pooling or participant-specific algorithm. Here, we re-evaluate this decision.




  
Experiment Setup
Using the simulation environment (Section <ref>) we re-ran, with all other design decisions fixed as deployed in the Oralytics trial, an algorithm that performs full pooling with one that performs no pooling over 500 Monte Carlo repetitions. We evaluate algorithms based on:


    
  * average of participants' average (across time) OSCB: 
    1/N∑_i = 1^N 1/T∑_t = 1^T Q_i, t

    
  * first quartile (25th-percentile) of participants' average (across time) OSCB: 
    First Quartile({1/T∑_t = 1^T Q_i, t}_i = 1^N)





  
Results As seen in Table <ref>, the average and first quartile OSCB achieved by a full-pooling algorithm is slightly higher than the average OSCB achieved by a no-pooling algorithm. These results are congruent with the results for experiments conducted before deployment (Section <ref>). Despite the heterogeneity of trial participants, it was worth it to run a full-pooling algorithm instead of a no-pooling algorithm.



 §.§ Did We Learn?


























Lastly, we consider if the algorithm was able to learn despite the challenges of the clinical trial setting. We define learning as the RL algorithm successfully learning the advantage of action a = 1 over a = 0 (i.e., sending an engagement prompt over not sending one) in a particular state s. Recall that the Oralytics RL algorithm maintains a model of this advantage (Equation <ref>) to select actions via posterior sampling and updates the posterior distribution of the advantage model parameters throughout the trial.
One way to determine learning is to visualize the standardized predicted advantage in state s throughout the trial (i.e., using learned posterior parameters at different update times τ).

The standardized predicted advantage in state s using the policy updated at time τ is:

    predicted_adv(τ, s) := μ^β⊤_τ f(s)/√(f(s)^⊤Σ^β_τ f(s))

μ^β_τ and Σ^β_τ are 

the posterior parameters of advantage parameter β from Equation <ref>, and f(s) denotes the features used in the algorithm's model of the advantage (Table <ref>).
















For example, consider Figure <ref>. Using posterior parameters μ^β_τ, Σ^β_τ learned during the Oralytics trial, we plot the standardized predicted advantage over updates times τ in a state where it is
(1) morning, (2) the participant's exponential average OSCB in the past week is about 28 seconds (poor brushing), (3) the participant received prompts 45% of the times in the past week, and (4) the participant did not open the app the prior day.
Since this value is trending more positive, it appears that the algorithm learned that it is effective to send an engagement prompt for participants in this particular state. In the following section, we assess whether this  pattern is evidence that the RL algorithm learned or is purely accidental due to the stochasticity in action selection (i.e., posterior sampling).





  
Experiment Setup
We use the re-sampling-based parametric method developed in <cit.> to assess if the evidence of learning could have occurred by random chance. We use the simulation environment built using the Oralytics trial data (Section <ref>).
For each state of interest s, we run the following simulation.
(i) We rerun the RL algorithm in a variant of the simulation environment in which there is no advantage of action 1 over action 0 in state s (See Appendix <ref>) producing posterior means and variances, μ^β_τ and Σ^β_τ. 
Using μ^β_τ and Σ^β_τ, we calculate standardized predicted advantages for each update time τ. (ii) We compare the standardized predicted advantage (Equation <ref>) at each update time  from the real trial 
with the standardized predicted advantage from the simulated trials in (i).

We consider a total of 16 different states of interest. To create these 16 states, we consider different combinations of possible values for algorithm advantage features f(s) (Table <ref>). Features (1) and (4) are binary so we consider both values {0, 1} for each. Features (2) and (3) are real-valued between [-1, 1], so we consider the first and third quartiles calculated from the Oralytics trial data.[For feature (2), -0.7 corresponds to an exponential average OSCB in the past week of 28 seconds and 0.1 corresponds to 100 seconds; for feature (3), -0.6 corresponds to the participant receiving prompts 20% of the time in the past week and -0.1 corresponds to 45%.] 



  
Results
Key results are in Figure <ref> and additional plots are in Appendix <ref>.
Our results show that the Oralytics RL algorithm did indeed learn that sending a prompt is effective in some states and ineffective in others. This suggests that our state space design was a good choice because some state features helped the algorithm discern these states.


We highlight 3 interesting states in Figure <ref>: 

    
  * A state where the algorithm learned it is effective to send a prompt and the re-sampling indicates this evidence is real. The advantage features f(s) correspond to (1) evening, (2) the participant’s exponential average OSCB in the past week is about 28 seconds (poor brushing),
(3) the participant received prompts 20% of the time in the past week, and (4) the participant did not open the app the prior day.
    
  * A state where the algorithm learned it is ineffective to send a prompt and the re-sampling indicates this evidence is real. The advantage features f(s) correspond to (1) morning, (2) the participant’s exponential average OSCB in the past week is about 100 seconds (almost ideal brushing),
(3) the participant received prompts 45% of the time in the past week, and (4) the participant opened the app the prior day.
    
  * The state in Figure <ref> but the re-sampling method indicates the appearance of learning likely occurred by random chance. 


For (a) and (b) the re-sampling method suggests that evidence of learning is real because predicted advantages using posterior parameters updated during the actual trial are trending away from the simulated predictive advantages from re-sampled posterior parameters in an environment where there truly is no advantage in state s. For (c), however, the re-sampling method suggests that the appearance of learning likely occurred by random chance because predicted advantages using posterior parameters updated during the actual trial are extremely similar to those from re-sampled posterior parameters in an environment where there truly is no advantage in state s.




























§ DISCUSSION

We have deployed Oralytics, an online RL algorithm optimizing prompts to improve oral self-care behaviors.
As illustrated here, much is learned from the end-to-end development, deployment, and data analysis phases.
We share these insights by highlighting design decisions for the algorithm and software service and conducting a simulation and re-sampling analysis to re-evaluate these design decisions using data collected during the trial. Most interestingly, the re-sampling analysis provides evidence that the RL algorithm learned the advantage of one action over the other in certain states. 
We hope these key lessons can be shared with other research teams interested in real-world design and deployment of online RL algorithms.
From a health science perspective, pre-specified, primary analyses <cit.> will occur, which is out of scope for this paper. The re-sampling  analyses presented in this paper will inform design decisions for phase 2. The re-design of the RL algorithm for phase 2 of the Oralytics clinical trial  is currently under development and phase 2 is anticipated to start in spring 2025. 






§ ACKNOWLEDGMENTS

This research was funded by NIH grants IUG3DE028723, P50DA054039, P41EB028242, U01CA229437, UH3DE028723, and R01MH123804. SAM  holds concurrent appointments  at Harvard University and as an Amazon Scholar. This paper describes work performed at Harvard University and is not associated with Amazon.





























§ ADDITIONAL ORALYTICS RL ALGORITHM FACTS



 §.§ Algorithm State Space


S_i,t∈ℝ^d represents the ith participant's state at decision point t, where d is the number of variables describing the participant's state. 



  §.§.§ Baseline and Advantage State Features


Let f(S_i,t) ∈ℝ^5 denote the features used in the algorithm's model for both the baseline reward function and the advantage. 




  
 These features are:


    
  * Time of Day (Morning/Evening) ∈{0, 1}
    
  *  B: Exponential Average of OSCB Over Past 7 Days (Normalized) ∈ [-1, 1]
    
  *  A: Exponential Average of Engagement Prompts Sent Over Past 7 Days (Normalized)  ∈ [-1, 1]
    
  *  Prior Day App Engagement ∈{0, 1}
    
  * Intercept Term =1

Feature 1 is 0 for morning and 1 for evening. Features <ref> and <ref> are B̅_i,t = c_γ∑_j = 1^14γ^j-1 Q_i, t - j and A̅_i,t = c_γ∑_j = 1^14γ^j-1 A_i, t - j respectively, where γ=13/14 and c_γ = 1 - γ/1 - γ^14. Recall that Q_i, t is the proximal outcome of OSCB and A_i,t is the treatment indicator. Feature 4 is 1 if the participant has opened the app in focus (i.e., not in the background) the prior day and 0 otherwise. Feature 5 is always 1. For full details on the design of the state space, see Section 2.7 in <cit.>.



 §.§ Reward Model


The reward model (i.e., model of the mean reward given state s and action a) used in the Oralytics trial is a Bayesian linear regression model with action centering <cit.>:


    r_θ(s, a) = f(s)^T α_0 + π f(s)^T α_1 + (a - π) f(s)^T β + ϵ

where θ = [α_0, α_1, β] are model parameters, π is the probability that the RL algorithm selects action a = 1 in state s

and ϵ∼𝒩(0, σ^2). We call the term f(S_i, t)^T β the advantage (i.e., advantage of selecting action 1 over action 0) and f(S_i, t)^T α_0 + π_i,t f(S_i, t)^T α_1 the baseline.
The priors are α_0∼𝒩(μ_α_0, Σ_α_0), α_1∼𝒩(μ_β, Σ_β), β∼𝒩(μ_β, Σ_β). Prior values for μ_α_0, Σ_α_0, μ_β, Σ_β, σ^2 are specified in Section <ref>. For full details on the design of the reward model, see Section 2.6 in <cit.>.



 §.§ Prior



Table <ref> shows the prior distribution values used by the RL algorithm in the Oralytics trial. For full details on how the prior was constructed, see Section 2.8 in <cit.>.





§ SIMULATION ENVIRONMENT


We created a simulation environment using the Oralytics trial data in order to replicate the trial under different true environments. 
Although the trial ran with 79 participants, due to an engineering issue, data for 7 out of the 79
participants was incorrectly saved and thus their data is unviable. Therefore, the simulation environment is built off of data from the 72 unaffected participants.
Replications of the trial are useful to (1) re-evaluate design decisions that were made and (2) have a mechanism for resampling to assess if evidence of learning by the RL algorithm is due to random chance. 
For each of the 72 participants with viable data from the Oralytics clinical trial, we use that participant's data to create a participant-environment model. We then re-simulate the Oralytics trial by generating participant states, the RL algorithm selecting actions for these 72 participants given their states, the participant-environment model generating health outcomes / rewards in response, and the RL algorithm updating using state, action, and reward data generated during simulation. To make the environment more realistic, we also replicate each participant being recruited incrementally and entering the trial by their real start date in the Oralytics trial and simulate update times on the same dates as when the RL algorithm updated in the real trial (i.e., weekly on Sundays). 



 §.§ Participant-Environment Model

In this section, we describe how we constructed the participant-environment models for each of the N = 72 participants in the Oralytics trial using that participant's data. Each participant-environment model has the following components:

    
  * Outcome Generating Function (i.e., OSCB Q_i, t in seconds given state S_i, t and action A_i, t)
    
  * App Engagement Behavior (i.e., the probability of the participant opening their app on any given day)




  
Environment State Features The 
features used in the state space for each environment are 

a superset of the algorithm state features f(S_i, t) (Appendix <ref>). g(S_i,t) ∈ℝ^7 denotes the super-set of features used in the environment model. 

The features are:


    
  * Time of Day (Morning/Evening) ∈{0, 1}
    
  *  B: Exponential Average of OSCB Over Past 7 Days (Normalized) ∈ [-1, 1]
    
  *  A: Exponential Average of Prompts Sent Over Past 7 Days (Normalized)  ∈ [-1, 1]
    
  *  Prior Day App Engagement ∈{0, 1}
    
  * Day of Week (Weekend / Weekday) ∈{0, 1}
    
  * Days Since Participant Started the Trial (Normalized) ∈ [-1, 1]
    
  * Intercept Term =1


Feature 5 is 0 for weekdays and 1 for weekends. Feature 6 refers to how many days the participant has been in the Oralytics trial (i.e., between 1 and 70) normalized to be between -1 and 1.



  
Outcome Generating Function
The outcome generating function is a function that generates OSCB Q_i, t in seconds given current state S_i, t and action A_i, t. We use a zero-inflated Poisson to model each participant's outcome generating process because of the zero-inflated nature of OSCB found in previous data sets and data collected in the Oralytics trial. Each participant's outcome generating function is:


    Z ∼Bernoulli(1 - sigmoid( g(S_i, t)^⊤ w_i,b - A_i, t·max[ Δ_i,B^⊤ g(S_i, t), 0 ] ) )


    S ∼Poisson( exp( g(S_i, t)^⊤ w_i,p + A_i, t·max[ Δ_i,N^⊤ g(S_i, t), 0 ] ) )


    Q_i, t = ZS

where g(S_i, t)^⊤ w_i,b,g(S_i, t)^⊤ w_i,p are called baseline (aka when A_i,t=0) models with 
w_i,b, w_i,p as participant-specific baseline weight vectors, max[ Δ_i,B^⊤ g(S_i, t), 0 ], max[ Δ_i,N^⊤ g(S_i, t), 0 ] are called advantage models, with Δ_i,B, Δ_i,N as participant-specific advantage (or treatment effect) weight vectors. g(S_i, t) is  described in Appendix <ref>, and sigmoid(x) = 1/1 + e^-x.


The outcome generating function can be interpreted in two components: (1) the Bernoulli outcome Z models the participant's intent to brush given state S_i, t and action A_i, t and (2) the Poisson outcome S models the participant's OSCB value in seconds when they intend to brush, given state S_i, t and action A_i, t. Notice that the models for Z and S currently require the advantage/treatment effect of OSCB Q_i, t to be non-negative. Otherwise, sending an engagement prompt would yield a lower OSCB value (i.e., models participant brushing worse) than not sending one, which was deemed nonsensical in this mHealth setting.


Weights w_i,b, w_i,p, Δ_i,B, Δ_i,N for each participant's outcome generating function are fit that participant's state, action, and OSCB data from the Oralytics trial. We fit the function using MAP with priors w_i,b, w_i,p, Δ_i,B, Δ_i,N∼𝒩(0, I) as a form of regularization because we have sparse data for each participant. Finalized weight values were chosen by running random restarts and selecting the weights with the highest log posterior density. See Appendix <ref> for metrics calculated to verify the quality of each participant's outcome generating function.



  
App Engagement Behavior
We simulate participant app engagement behavior using that participant's app opening data from the Oralytics trial. Recall that app engagement behavior is used in the state for both the environment and the algorithm. More specifically, we define app engagement as the participant opening their app and the app is in focus and not in the background. Using this app opening data, we calculate p^app_i, the proportion of days that the participant opened the app during the Oralytics trial (i.e., number of days the participant opened the app in focus divided by 70, the total number of days a participant is in the trial for). During simulation, at the end of each day, we sample from a Bernoulli distribution with probability p^app_i for every participant i currently in the simulated trial.



 §.§ Assessing the Quality of the Outcome Generating Functions


Our goal is to have the simulation environment replicate outcomes (i.e., OSCB) as close to the real Oralytics trial data as possible. To verify this, we compute various metrics (defined in the following section) comparing how close the outcome data generated by the simulation environment is to the data observed in the real trial . Table <ref> shows this comparison on various outcome metrics.
Table <ref> shows various error values of simulated OSCB with OSCB observed in the trial.
For both tables, we report the average and standard errors of the metric across the 500 Monte Carlo simulations and compare with the value of the metric for the Oralytics trial data. Figure <ref> shows comparisons of outcome metrics across trial participants.



  §.§.§ Notation
 
𝕀{·} denotes the indicator function. Let Var({X_k}_k = 1^K) represent the empirical variance of X_1,...,X_K.



  §.§.§ Metric Definitions and Formulas

Recall that N=72 is the number of participants and T=140 is the total number of decision times that the participant produces data for in the trial. We consider the following metrics and compare the metric on the real data with data generated by the simulation environment.


    
  * Proportion of Decision Times with OCSB = 0:

    ∑_i=1^N∑_t=1^T𝕀{Q_i,t = 0}/N × T



  * Average of Average Non-zero Participant OSCB:


    1/N∑_i=1^NQ̅_i^non-zero


where 
    Q̅_i^non-zero = ∑_t=1^T Q_i,t·𝕀{Q_i,t > 0}/∑_t=1^T𝕀{Q_i,t > 0}



  * Average Non-zero OSCB in Trial:


    1/∑_i = 1^N ∑_t = 1^T 𝕀{Q_i, t > 0}∑_i = 1^N ∑_t = 1^T Q_i, t·𝕀{Q_i, t > 0}



  * Variance of Average Non-zero Participant OSCB:

    Var({Q̅_i^non-zero}_i = 1^N)


where 
    Q̅_i^non-zero = ∑_t=1^T Q_i,t·𝕀{Q_i,t > 0}/∑_t=1^T𝕀{Q_i,t > 0}








  * Variance of Non-zero OSCB in Trial: 


    Var({Q_i, t : Q_i, t > 0}_i = 1, t = 1^N, T)






  * Variance of Average Participant OCSB:


    Var({Q̅_i}_i = 1^N)


where Q̅_i = ∑_t = 1^T Q_i, t is the average OSCB for participant i




  * Average of Variances of Participant OSCB:


    1/N∑_i=1^NVar({Q_i, t}_t = 1^T)






We also compute the following error metrics. We use Q̂_i, t to denote the simulated OSCB and Q_i, t to denote the corresponding OSCB value from the Oralytics trial data. 



  * Mean Squared Error:

    1/N× T∑_i=1^N∑_t=1^T(Q̂_i,t - Q_i,t)^2


  * Root Mean Squared Error: 

    √(1/N× T∑_i=1^N∑_t=1^T(Q̂_i,t - Q_i,t)^2)


  * Mean Absolute Error: 

    1/N× T∑_i=1^N∑_t=1^T|Q̂_i,t - Q_i,t|




















 §.§ Environment Variants for Re-sampling Method


In this section, we discuss how we formed variants of the simulation environment used in the re-sampling method from Section <ref>. We create a variant for every state s of interest corresponding to algorithm advantage features f(s) and environment advantage features g(s). In each variant, outcomes (i.e., OSCB Q_i, t) and therefore rewards, are generated so that there is no advantage of action 1 over action 0 in the particular state s. 


To do this, recall that we fit an outcome generating function (Equation <ref>) for each of the N = 72 participants in the trial. Each participant i's outcome generating function has advantage weight vectors Δ_i,B, Δ_i,N that interact with the environment advantage state features g(s). Instead of using Δ_i,B, Δ_i,N fit using that participant's trial data, we instead use projections proj Δ_i,B, proj Δ_i,N of Δ_i,B, Δ_i,N that have two key properties:

    
  * for the current state of interest s, on average they generate treatment effect values that are 0 in state s with algorithm state features f(s) (on average across all feature values for features in g(s) that are not in f(s))
    
  * for other states s' ≠ s, they generate treatment effect values g(s')^⊤proj Δ_i,B, g(s')^⊤proj Δ_i,N close to the treatment effect values using the original advantage weight vectors g(s')^⊤Δ_i,B, g(s')^⊤Δ_i,N


To find proj Δ_i,B, proj Δ_i,N that achieve both properties, we use the SciPy optimize API[Documentation here: https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html] to minimize the following constrained optimization problem:


    min_proj Δ1/K∑_k = 1^K (g(s')_k^⊤proj Δ - g(s')_k^⊤Δ)^2


    subject to:  g(s)^⊤proj Δ = 0



{g(s')_k}_k = 1^K denotes a set of states we constructed that represents a grid of values that g(s') could take.
g(s) has the same state feature values as g(s) except the “Day of Week" and “Days Since Participant Started the Trial (Normalized)" features are replaced with fixed mean values 2/7 and 0. The objective function is to achieve property 2 and the constraint is to achieve property 1.

We ran the constrained optimization with Δ = Δ_i, B and Δ_i, N  to get proj Δ_i,B, proj Δ_i,N, for all participants i. All participants in this variant of the simulation environment produce OSCB Q_i, t given state S_i, t and A_i, t using Equation <ref> with Δ_i, B, Δ_i, N replaced by proj Δ_i,B, proj Δ_i,N.



§ ADDITIONAL DID WE LEARN? PLOTS




In Section <ref> we considered a total of 16 different states of interest. Results for all 16 states are in Figure <ref>. Recall each state is a unique combination of the following algorithm advantage feature values:

    
  * Time of Day: {0, 1} (Morning and Evening)
    
  * Exponential Average of OSCB Over Past Week (Normalized): {-0.7, 0.1} (first and third quartile in Oralytics trial data)
    
  * Exponential Average of Prompts Sent Over Past Week (Normalized): {-0.6, -0.1} (first and third quartile in Oralytics trial data)
    
  * Prior Day App Engagement: {0, 1} (Did Not Open App and Opened App)


Notice that since features (2) and (3) are normalized, for feature (2) the quartile value of -0.7 means the participant's exponential average OSCB in the past week is about 28 seconds and similarly 0.1 means its about 100 seconds. For feature (3), the quartile value of -0.6 means the participant received prompts 20% of the time in the past week and similarly -0.1 means it's 45% of the time.























§ DID WE LEARN?




    





























 §.§ Equations

Recall this is the prior on the advantage term that we deployed:

    μ_β = [0, 0, 0, 53, 0]^⊤


    Σ_β = diag(12^2, 33^2, 35^2, 56^2, 17^2)




  
What knowledge does this prior encode? 

    
  * For significant features as deemed by domain experts, we set the prior mean
to the empirical mean parameter value for that feature across the 9 participants in the Oralytics pilot study. Otherwise the prior mean is 0.
    
  * For significant features, we set the prior SD to
    the empirical SD for that feature across the 9 participants in the Oralytics pilot study. For non-significant parameters, we set the
    prior SD to the empirical SD divided by 2. We reduced the SD of the
    non-significant weights because we want to provide more shrinkage to the prior mean of 0. 
    
  * Prior day app engagement is the only feature domain experts deemed to be significant in the advantage










 §.§ Interesting Participant-Feature Graphs



  
Interesting Scores
We calculate two types of interesting scores depending on the type of value that each feature can take. In our setting, features are either binary {0, 1} or real-valued between [-1, 1] (See Appendix <ref>).

If feature z ∈{0, 1}, then the interesting score we compute for participant i is:

    score_int_z(i) := ∑_t = 1^T 𝕀[π̂_i, t(S_i, t(z = 1)) > π̂_i, t(S_i, t(z = 0))]/T


otherwise, if feature z ∈ [-1, 1], then the interesting score is:

    score_int_z(i) := ∑_t = 1^T 𝕀[π̂_i, t(S_i, t) > π̂_i, t(S_i, t(z = 0))]/T


π̂_i, t: s → [0.2, 0.8] is the estimated advantage function (i.e., action selection probability function) that produces an action-selection probability given state s. S_i, t(z = 1) denotes a state that takes the same value as S_i, t for all features except z, which is set to 1, and similarly for S_i, t(z = 0). Notice that for Equation <ref>, S_i, t is the actual observed state with no modifications, and we are contrasting with z = 0, which represents the average feature value.

We say that the trajectory for participant i is interesting for feature z if |score_int_z(i) - 0.5| ≥δ. The number of interesting participants for feature z is:


    #participant_int_z := ∑_i = 1^N 𝕀{|score_int_z(i) - 0.5| ≥δ}

where we set δ = 0.4 for our experiments.


















