# A guideline proposal for minimizing cybersickness in VR-based serious games
and applications
††thanks:

1st Thiago Porcino, 2nd Derek Reilly  Faculty of Computer Science
Dalhousie University
Halifax, Canada
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>3rd Esteban Clua, 4th Daniela Trevisan  Institute of
Computing
Universidade Federal Fluminense
Niteroi, Brazil
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Head-mounted displays (HMDs) are popular immersive tools in general, not
limited to entertainment but also for education, military, and serious games
for health. While these displays have strong popularity, they still have user
experience issues, triggering possible symptoms of discomfort to users. This
condition is known as cybersickness (CS) and is one of the most popular
research topics tied to virtual reality (VR) issues. We first present the main
strategies focused on minimizing cybersickness problems in virtual reality.
Following this, we propose a guideline framework based on CS causes such as
locomotion, acceleration, the field of view, depth of field, degree of
freedom, exposition use time, latency-lag, static rest frame, and camera
rotation. Additionally, serious games applications and broader categories of
games can also adopt it. Additionally, we categorized the imminent challenges
for CS minimization into four different items. Conclusively, this work
contributes as a consulting reference to enable VR developers and designers to
optimize their VR users’ experience and VR serious games.

###### Index Terms:

head-mounted displays, virtual reality, cybersickness, VR-based serious games,
challenges

##  I Introduction

Games and Virtual Reality may be efficient environments for teaching and
training in numerous fields, such as education, military, and health. In
particular, dedicated applications for health retain the purpose of training
health professionals and as tools to minimize health disorders. For example,
serious games can be developed for rehabilitation processes and therapies [12,
38]. Besides, with the immersive contribution of popular XR (extended reality)
supplies, such as head-mounted displays (HMDs) and the popularity of immersive
environments over the last years, serious games that can employ immersive
virtual conditions and gaming mechanics maintain a promising future, supported
by the market of vast hardware availability for immersive games [11].

HMDs devices are used for various purposes in the industry such as in games
that focus on health [10] and serious games [11]. However, VR consumers are
often complaining about frequent manifestations of discomfort in these devices
[25]. This situation is known as cybersickness (CS) and is a condition that
has been intrinsic in HMDs. CS can also trigger a scope of symptoms of
discomfort in users.

The discomfort effect can negatively impact the VR treatment and, in some
cases, even present dangerous situations for the current patient. For example,
while a VR-based game for upper limb rehabilitation [15] can be used as an
excellent therapy tool, it also can produce discomfort in patients immersed in
a virtual environment hearing HMDs. In addition, CS symptoms can vary across
individuals; some people are more susceptible than others. In summary, we
start from the principle that CS symptoms are a side effect of any immersive
application (including immersive serious games) that does not follow protocols
to minimize the CS.

In this work, we propose a guideline to improve the user experience and
minimize cybersickness symptoms. Additionally, we categorized the main
challenges in this context. To build this guideline, we focus on immersive
environments, including immersive serious games for health-based treatments,
VR entertainment games, and VR applications. Furthermore, we highlight some
recent immersive serious games that focus on rehabilitation but do not apply
any strategy to produce a better user experience related to minimizing
cybersickness in their application. We believe our guideline can improve
patients’ experience using immersive serious games and head-mounted displays.
More importantly, we look forward to answering the following research
questions:

  * •

Which are the main strategies reported in the literature to minimize CS in VR
serious games?

  * •

Which are the main challenges for mitigating CS in VR environments?

This paper is organized as follows: Section 2 clarifies the literature review
presenting the necessary background knowledge to understand the types of VR-
related illnesses. Section 3 explains the related works highlighting the
importance of using strategies to avoid CS in VR-based serious games for
rehabilitation. Section 4 describes our proposed guideline to mitigate
discomfort in VR-based environments and serious games, presenting the
strategies to overcome CS. Section 5 points out the main challenges to
minimizing CS in head-mounted displays. Section 6 presents a discussion of
this work’s findings. Finally, section 7 describes the conclusion and future
work.

##  II CS Background

This section presents a fundamental understanding of motion perception
concerning motion sickness’s (MS) primary distinctions and subcategories. MS
manifests itself because of the information divergence emitted by the human
sensory system. This occurs when conflicts between the sensory organs define
an individual’s orientation and spatial positioning. MS is defined as the
discomfort felt during a forced visual movement without equivalent body
movement, such as those present in the airplane, boats, or land vehicles
locomotions [21].

This discomfort also occurs in virtual environments and is called visually
induced motion sickness (VIMS). Merhi et al. [37] defined the event of VIMS
during experiments with video games as a game disease (gaming sickness).
Moreover, in VR, articles usually label VIMS that occurs in VR as
CyberSickness (CS) [35]. In contrast, VIMS that occurs during flight or drive
simulators is often called simulator sickness [5]. Overall, MS can be split
into two subcategories [21]: transportation sickness, which is tied to the
real world, and simulator sickness, which is associated with the virtual world
and includes cybersickness (CS).

Cybersickness symptoms, in turn, are comparable with MS symptoms occurring in
the real world, such as nausea, vertigo, dizziness, and upset stomach [19]. CS
Symptoms occur mainly with VR Head-Mounted Displays (HMDs) [55]. [25]
described more than 40 possible VIMS causes. These factors were grouped into
simulator, task, and individual factors. [56] expanded Kolasinski’s work by
tabling and dividing potential factors for CS manifestations. Rebenitsh [56]
stated that many factors and configurations related to discomfort are still
unknown.

##  III Related Works

CS attenuation for different users and games is not trivial. The first problem
is the lack of a unique variable for discomfort levels. VR users may
experience multiple symptoms. Besides, serious games focused on treatment use
VR environments and equipment (such as HMDs) to improve user experience and
patient engagement in therapy.

However, the lack of CS mitigation strategies in VR-based serious games may
impair therapy results, generating side effects on their patients. For
example, the CS symptoms generated in a VR-based game for therapy could be a
negative side effect in a VR game without CS minimization strategies. For this
reason, in this section, we separated related works considering works focused
on CS minimization and VR-based serious games for rehabilitation.

###  III-A CS minimization in VR

Several studies have been conducted using deep learning models, such as
convolutional neural networks (CNNs) and recurrent neural networks (RNNs).
[23], proposed a deep learning architecture to estimate the cognitive state
using brain signals and how they are related to CS levels. Their approach is
based on deep learning models, such as long short-term memory (LSTM), RNN, and
CNN [30, 18, 58]). The models learn the individual characteristics of the
participants that lead to the manifestation of CS symptoms when playing a VR
game.

In a previous work [49], we propose an approach to identifying causes of
cybersickness in different virtual reality games using head-mounted displays.
We used symbolic classifiers to analyze the causes of cybersickness during the
gameplay experience. Once the cause is identified, game designers can select
an adequate strategy to mitigate the impacts of CS.

Langbehn et al. [28], Farmani et al. [13], and Sarupuri et al.[61]
concentrates efforts on strategies to resolve CS tied to locomotion in virtual
reality environments. Besides, Berthoz et al., [2] Pavard et al., [43] and
Bouyer et al. [4] applied haptic feedback to cover this issue. Our proposed
guideline (section IV) detail these and other works considering their
strategies.

###  III-B VR-based serious games

In healthcare, serious games aimed at patient rehabilitation are top-rated for
many reasons [59]. In these rehabilitation activities, frequent patients give
up long-term treatments (such as physiotherapy) because they feel tedious
after many exercise sessions [39]. The serious games can motivate patients
during the rehabilitation tasks, mainly in long-term treatments where the
patient needs to train their motor abilities [7].

Some studies [15, 31] designed virtual reality-based serious games to enhance
the execution of these movements using virtual reality and HMDs. In Ferreira
et al. [15] work, they developed a serious VR-based game for upper limb
rehabilitation. In this case, the primary purpose of the developed games is to
keep patients engaged without developing any frustration or boredom during the
gameplay. They used an immersive Tetris-like game to execute movements that
strengthen patients’ muscles of the arm and shoulder, stimulate their
cognitive activity, and stimulate the patient’s sensory inputs. The 15
participants were exposed to a VR-based game for an average of 28 minutes and
46 seconds.

Liao et al. [31] also proposed a serious VR-based game for upper limb therapy.
The serious game developed is composed of three mini-games, where each one is
designed for a specific rehabilitation task related to daily life or a joint
event (i.e., fishing). They use a scoring system to reward patients after
completing tasks in games. The authors did not mention the number of
participants and the exposure time in their VR-based game.

Pereira et al. [45] developed the Stable-Hand VR, a serious game developed to
help patients in hand rehabilitation processes. This system uses an HMD to
render immersive content and a leap controller to track and manage the patient
hand’s movements. This serious VR-based game’s tasks are generally aimed at
pinching gestures. The experiment process in this work was made with 7
participants (in both works) in two 20 minutes sessions with one day break.
According Pereira et al. [45], 3 participants mentioned discomfort tied to HMD
use, specifically during the transition between scenes in the game menu.

Luo et al. [34] designed the VR driving serious game focused on rehabilitation
of patients with Hemiplegia, a condition caused by brain harm that results in
weakness, stiffness (spasticity), and lack of control in one side of the body.
They use a VR serious game to create rehabilitation sessions associated with
upper limp recovery treatment. In summary, the patient has the task of
controlling a virtual car to collect coins and avoid virtual obstacles. The
game has two different play modes, one with a fixed speed and the other with a
controllable speed (similar to conventional race games). They produced two
modes to adapt the application with different limb disorder levels. Also, the
application has an HMD and 3D annular projection screen modes. According to
the authors, compared to traditional rehabilitation activities, the VR game
increased the interest and confidence of patients during the rehabilitation
processes, but no CS mitigation technique was applied.

Although all the related works are focused on patient rehabilitation (and
related to health), only the work by Pereira et al. [45] mentioned the concern
with symptoms of the discomfort of their patients during the use of HMD
device. In addition, none of the studies mentioned applied one or more
strategies to avoid discomfort in their patients during HMDs use.
Additionally, a recent study related an option to use the same developed VR-
based game in a non-immersive way (with a standard screen) and the same
controllers (VR motion controllers) to avoid CS [29].

The studied serious games in this work do not endeavor to avoid CS using
strategies. On the other hand, commercial games essentially use CS
minimization techniques. We assume that VR-based serious games for
rehabilitation and traditional VR games have similar gameplay, design, and
development bases. For this reason, we propose a guideline framework with
strategies to combat CS in VR-based games (including those for
rehabilitation).

##  IV Proposed Guideline Framework

As previously mentioned, many related works have investigated the influence of
CS aspects in VR environments. However, in this study, we are interested in
investigating how such aspects of CS have impacted the user experience with
virtual reality serious games and how game designers could mitigate such
impacts.

Following a previous review of the literature [53, 52], we propose a guideline
strategy that may be used as a framework for avoiding CS in VR applications
and games. In this section, we present our proposed guideline framework based
on causes described in a previous work [53, 50]. Based on the referenced
serious games for health applications activities, we propose here that the
causes of CS may be influenced by the following variables: locomotion,
acceleration, the field of view, depth of field, degree of control, duration
use time, latency-lag (including rendering optimization), static rest frame,
and camera rotation. Our framework first suggests identifying when these
situations are present in the application, marking them as necessarily
desirable or optional, according to the serious game required activities. We
also suggest listing these situations in a priority list to give more
attention to those at the top of the list. Our framework suggests the designer
consider the following guidelines for each of these variables.

###  IV-A Locomotion

We start our framework with the consideration of the user movement. In this
sense, Teleportation strategies are one of the most effective strategies for
avoiding CS related to translation, and our solution claims that developers
should first consider if teleportation fits the application’s requirements
(Figure 1).

Figure 1: Example of teleportation strategy. (1) The user point to an specific
location in VR. (2) User are immediately transported to the aimed location.

In teleportation, users can travel great distances by specifying the trip’s
destination point with the help of a marker [28]. This strategy works as
follows: using a controller, the user points to the destination location and
squeezes a trigger button, which immediately transports the user to the new
location, also called ”pointing and teleport”. Another strategy called
”trigger walk” uses the concept of natural walking to reach a destination. In
this case, the user uses VR control triggers instead of legs to move around.
Each user’s hands handle each control in a relaxed and comfortable position
(with minimum energy consumption). The user moves closer to the direction
indicated at each pull of the trigger [61]. The VR-based serious games that
able players to virtually walk need to attempt locomotion techniques to avoid
CS.

###  IV-B Acceleration

Following locomotion, we suggest continuing the design process considering
acceleration issues. According to [2], it is possible to induce a sensation of
movement using a visual response (haptic feedback example in Figure 2) [57].

Figure 2: The virtual reality game (1) and a (2) HTC Vive controller enhanced
with haptic feedback actuators in Ryge et. al [57].

According to [43], the human visual system can adapt to illusive motion but
not acceleration. Various applications of VR (e.g., games and training
applications) require support for haptic perception. This is because haptic
perceptions can induce the sensation of acceleration in its users. When
correctly applied, artificial acceleration sensation can help avoid sensory
conflict. In some virtual environments (racing games), it is possible to
minimize CS problems using haptic responses. Haptics is a way of transmitting
physical sensations to the user, which are compatible with those captured by
the user’s visual system. [4] used haptic feedback outside a VR environment
while still managing to provide users with an enhanced sense of reality.
According to [46], measuring CS acceleration using electro-dermal activity
(EDA) is possible. Plouzeau et al. changed and adjusted the acceleration to
visualize EDA changes. When EDA values increase, the acceleration decreases
proportionately. According to research [64], the more predictable the camera
movement and acceleration, the lesser the CS effects will be. When
accelerations situations are necessary for the required simulation, our
framework suggests building or including proper interfaces, such as real
ergonomic bicycles, adapted wheelchairs, or adapted mats.

###  IV-C Field of view

Applying strategies that manipulate the field of view (FoV) in commercial
games is quite common [65]. Vignette is a strategy to shorten the FoV
gradually, thus reducing discomfort in VR environments [14]. This strategy is
a variation applied in [3], where the size of the vignette and dynamic FoV is
related to the camera acceleration values. Tunnel or Tunneling (illustrated in
Figure 3) is also used to solve locomotion problems.

Figure 3: (1) A VR race gameplay on a curved track with no appliance of
guidelines. (2) The race gameplay using rotational blurring and tunneling,
activated during the gameplay in [49].

Such a strategy reduces the size of the user’s FoV at the moment of the
locomotion, thereby minimizing sensory conflict problems. Similar to the
vignette, the tunnel significantly reduces the FoV. However, it is only
applied during locomotion. Design projects should select all the translation
stages and mark when such strategies could be used, considering the focus
elements required for that action. Developers should avoid presenting new
elements to the application during this moment. This strategy is required to
improve locomotion techniques if the VR-based therapy players have to walk
virtually. Additionally, VR-based therapies with camera movements can also
benefit from this strategy.

###  IV-D Depth of field

Some studies include a depth of field simulation (Figure 4) agent with image
blur to minimize the convergence and accommodation problems [9, 47].

Figure 4: (1) A VR application without depth of field effect. (2) The VR
application using depth of field focusing on the fountain [68].

The solution presented by [9] pointed to decreased discomfort in HMD
applications. Specifically, they suggested a GPU-based solution to simulate
depth of field in these applications. In our initial work, we developed a
focus and region of interest dynamics model for the visualization of objects
in VR [47]. Unlike Carnegie, we used the term ”dynamic” to suggest that the
model moves the region of interest in the 3D scene using the application. This
prototype simulates a visual focus self-extraction tool, limiting the region
of interest in the visual field. The model uses the region of interest to
determine the depth of field effects in real-time, minimizing discomfort when
using HMD. Additionally, we designed a methodological guide for CS
minimization on VR application [53]. On the other hand, field depth simulation
strategies can produce low frame rates, inducing high latency, thereby causing
CS. In the work of [26] and [42], an approach was adopted to apply a simulated
depth of field using an external interface, solving the problem of low frame
rate. However, such a strategy is contingent on the application of specific
hardware. A recent work reduced the spatial resolution in the HMD’s peripheral
regions (using foveated rendering) and applied the depth of field simulation
effect [20]. Most observed VR-based serious games for rehabilitation render
sharp images in HMDs, that trigger CS. In VR therapy, the lack of correct
simulation of DoF can negatively affect the VR therapy, causing discomfort in
the patient.

###  IV-E Duration use time

The study by [36] relates the exposure time with discomfort manifestation and
suggests short-term or interval virtual experiences. This principle suggests
that VR applications could avoid CS symptoms if paused periodically.
Consequently, the application should allow users to interrupt the experience
to take a rest and then be able to return to the exact point paused before.
According to [63], CS can be minimized by exposing non-experienced users
gradually to VR, which means a VR experienced user can have less
susceptibility to CS than non-experienced users. Our framework suggests
measuring the required time for the activities, and in cases when they take
more than 10 minutes [54], we suggest to divide activities into time slots
that may be interrupted without any immersive breakthrough.

###  IV-F Degree of freedom (control)

Degrees of freedom (3-Dof or 6-Dof) is an essential concept in VR that allows
natural human movement to be converted into simulated movement in the VR
application [41]. Low-cost HMDs (such as Google Cardboard, Oculus Go, and
Samsung Gear VR) has 3 degrees of freedom tracking (3Dof). In other words, it
is possible to look at 360 degrees, for example, looking left or right,
rotating their head up or down, and pivoting left or right. Consequently, the
user can feel disorientation by the limited movement. For 3-Dof HMDs,
anticipating and preparing the user’s visual motion experience can reduce the
lack of control by the user and, consequently, the discomfort. Having a
virtual guiding avatar, which prepares the user’s visual motion experience in
the virtual environment, is a result of VIMS reduction on Lin et al. [32]
(illustrated in Figure 5).

Figure 5: (1) Design of the Virtual guiding 3D avatar by [32]. (2) A view of
guiding avatar (3d arrows) use in first person perspective, and (3) a map of
virtual environment showing multiple guiding arrows around the VR environment
[8].

###  IV-G Latency-lag

The asynchronous time warp is a method for overcoming HMD’s latency [21] by
improving a rendered (warped) image based on the latest head-tracking data.
According to [66] , this method is based on augmented reality ”CamWarp” (that
is applied in see-through augmented reality devices) and also reduces
discomfort in VR environments. In other words, this strategy drives the user
to see their movement in VR without diverging between the physical movement
(made by the user’s body) and the virtual movement (visualized by the user’s
eyes on the HMD display).

Moreover, foveated rendering is a performance optimization based on the well-
known degradation of peripheral visual acuity (illustrated in Figure 6).

Figure 6: Albert et al. [1] produced a scene with different foveation
strategies. In (1) the original render without any effect, and in (2) they
exaggerated the magnitude of foveation to illustrate the effect. The center of
gaze (in yellow) is the region with high-resolution.

This strategy reduces the computational costs by rendering a high-resolution
image in the user’s center of attention (foveal region) and a low-resolution
image in the periphery [1]. In VR, foveated rendering is an excellent graphics
optimization that helps to reduce system latency and virtual reality
application performance. Similar to ”CamWarp” strategy, the ”Time-Warped
Foveated Rendering” [16] strategy produces a visually plausible result if
compared to regular rendering. In summary, Franke et al. [16] produced a
foveated rendered scene without being noticeable to the observer’s eye with a
speed-up of almost 1.6X than the original render. Consequently, both
strategies contribute to minimizing CS effects tied to latency. While VR-based
serious games for rehabilitation are simpler than commercial entertainment
games in the context of computational complexity (that affects the game
performance), in VR-based therapy, the game performance is also critical
because it directly affects the patient’s health in case of triggering CS
symptoms.

###  IV-H Static rest frame

According to studies [62, 24], people show longer tolerance to discomfort
during experiences based on VR projections (example: Cave Automatic Virtual
Environment [40]). One of the biggest differences between VR and projection-
based systems is rest frames. In projection-based systems, the screen edges
and visible real-world elements beyond the screens act as rest frames. This
raises the hypothesis that the simulation of rest frames in virtual
environments (Figure 7) can create comfortable experiences [32].

Figure 7: (1) A VR application without static rest frame. (2) The VR
application using static layer as rest frame [8].

However, adding elements to create a false rest frame that hides part of the
screen may not be a good strategy for all VR applications. It can work well
for racing games, where the player is naturally inserted into a car. However,
this approach may not work well for games with first-person cameras, as they
create strange circumstances for the player. Our framework suggests that the
designer consider when this can be possible or not during the experience.

###  IV-I Camera rotation

Several other works applied various strategies such as head movement
amplification, whereby individual movements are amplified in VR [27, 46].
Another example is the blurring rotation (Figure 8), a strategy implemented by
[6] that uniformly applies Gaussian blurs based on the magnitude of
acceleration and rotation values. There are also experiments deploying more
basic strategies that lock the users’ heads to avoid rotational movements.

According to [22], such a strategy reduces the CS manifested during rotation
by 30% compared with the use of controls to perform rotational movements.
Nevertheless, the authors concluded that participants found the strategy non-
intuitive because it reduced the sensation of presence in the virtual
environment. It is worth noting that both this strategy and rotation blurring
only apply to rotational movements. We suggest using the blurring effect in a
virtual reality experience during a rotation movement.

Figure 8: From (1) to (4), a blurring effect application during a rotation
movement in virtual reality environment [6].

##  V Challenges to Minimize CS

While many signs of progress related to immersion in VR had been achieved in
computer graphics visualization, audio, smart devices, video tracking, and
interfaces between many systems, there are still many remaining challenges to
minimize CS in HMDs, primarily related to the early identification of
situations that cause discomfort in the user. This section presents challenges
such as CS prediction, data labeling for machine learning studies, the lack of
underrepresented groups’ data analysis, and the variability of tasks in
virtual environments.

  1. 1.

Cybersickness prediction \- Although deep learning classifiers are the most
suitable approach for CS prediction, deep neural networks are black boxes that
are very difficult to grasp. On the other hand, symbolic machine learning
algorithms enable a straight understanding of decision paths [50]. The human-
readable characteristic of symbolic machine learning models such as decision
tree and random forest can help understand the discomfort manifestation
reasons [48]. Additionally, a recent approach applies strategies to make deep
learning models explainable [67], although the literature is still scarce on
the topic. Explainable AI can be an excellent way to better the two worlds:
the human understanding of symbolic machine learning with the accurate
prediction of deep neural networks.

  2. 2.

Data labeling \- Another critical problem in CS research is associated with
data labeling. Researchers generally collect verbal [50, 23], haptic, or brain
signal feedback to construct the ground truth of sickness. While verbal
feedback is highly subjective and different for each participant, collecting
haptic feedback when participants are in discomfort can often be corrupted by
the delay associated with participant feedback.

  3. 3.

Data from underrepresented groups \- Another not resolved challenge is
acquiring data considering underrepresented user-profiles [44] such as
specific impairments or health issues [33].

  4. 4.

Tasks in VR \- A recent work [17] pointed out that specific tasks can produce
different results of CS for different user profiles and groups. Additionally,
in a recent work [49] we suggested the importance of a better understanding of
the correlation between profile attributes (such as gender, age, XR
experience, and other individual characteristics) and gameplay elements, and
also how outcomes obtained from profile attributes can be used to tag XR
experiences according to distinct groups of users. In this context, studies
involving machine learning models combined with an evaluation of individual
tasks associated with CS causes in XR games may produce a more profound study
isolating any other XR possible influences on CS results.

##  VI Discussion

Although several strategies minimize cybersickness, it is still impossible to
eliminate all symptoms from HMD users. This problem occurs because each user
has a different level of susceptibility to discomfort, leading many
researchers to look for solutions involving machine learning (deep neural
networks), as seen in this work.

The immersive serious games described in the related works did not use any
strategy to minimize CS. This problem worsens when the serious game is
associated with some recovery therapy. While the treatment in a virtual
reality environment can be beneficial for the patient, it can generate
feelings of discomfort in users if CS issue is not considered, as reported by
Pereira et al. [45]. For this reason, immersive serious games (especially
those related to health) must follow guidelines to improve their users’
experience in VR environments.

Moreover, we did not find any standard methodology for CS data recording or
experiments conducted along with this study. Creating a standard methodology
to collect CS data can be challenging because the same sickness level can
differ among different participants. In other words, considering a scale from
0 to 3 (none, slight, moderate, and severe), one person can consider scale one
as slight, but another, more susceptible, will fall as three or severe.

Furthermore, collecting haptic feedback while participants are feeling
discomfort, this data can be corrupted by delayed (i.e., CS cases delays in
response time) or random responses. In other words, the moment the participant
feels the pain may not be the exact moment the CS cause was triggered in the
participant. Additionally, monitoring methodologies used to record the human
brain’s electrical activity (such as Electroencephalography) have more
accurate data, as they capture more precise data of the user’s physical and
mental state, and many diseases and brain problems are diagnosed through the
evaluation of such devices’ data [60]. However, these medical tools are
expensive and unavailable to the general public.

In summary, it is necessary to standardize the procedures, equipment, and
materials to collect CS data and the methods used to classify it.

##  VII Conclusion and Future Work

This work proposes an objective guideline process based on an extensive
literature review to avoid CS using software and hardware approaches. Besides,
CS can be triggered due to different factors [48]. Eventually, a single cause
might influence more in triggering CS than others. For this reason, different
combinations of strategies can be applied according to the game content or
user susceptibility to CS. In other words, different VR-based serious games
could lead to different strategy recommendations.

We covered the main strategies reported in the literature to minimize CS in VR
compatible with immersive serious games. Additionally, we categorize the main
challenges to minimizing CS into 4 different items, although others might be.
Specifically, this classification is not exhaustive, and many other aspects
could also be included in future works, such as human behaviors, hardware,
interface, and other related issues.

A straightforward way is to explore gender differences. A recent work [17]
pointed out that specific tasks can produce different results of discomfort
for different user profiles and user groups, regarding and not limited to:
gender, age, or health issues. Our symbolic machine learning approach can also
help to further analysis.

For this reason, the next step of this research is to propose a virtual
reality game that explores specific tasks individually tied to specific CS
causes and strategies with a long exposure. We intend to conduct an evaluation
study, considering individual tasks associated with CS causes and strategies.
We believe it is possible to produce a more profound study isolating any other
VR possible influences on CS results [51, 48].

At this moment, we believe this work can contribute to designers and
researchers to enable the right strategy to avoid CS in VR serious games and
other VR applications.

## References

  * [1] R. Albert, A. Patney, D. Luebke, and J. Kim.  Latency requirements for foveated rendering in virtual reality.  ACM Transactions on Applied Perception (TAP), 14(4):1–13, 2017\.
  * [2] A. Berthoz, B. Pavard, and L. Young.  Perception of linear horizontal self-motion induced by peripheral vision (linearvection) basic characteristics and visual-vestibular interactions.  Experimental brain research, 23(5):471–489, 1975.
  * [3] M. Bolas, J. A. Jones, I. McDowall, and E. Suma.  Dynamic field of view throttling as a means of improving user experience in head mounted virtual environments, May 9 2017.  US Patent 9,645,395.
  * [4] G. Bouyer, A. Chellali, and A. Lécuyer.  Inducing self-motion sensations in driving simulators using force-feedback and haptic motion.  In Virtual Reality (VR), 2017 IEEE, pages 84–90. IEEE, 2017.
  * [5] J. O. Brooks, R. R. Goodenough, M. C. Crisler, N. D. Klein, R. L. Alley, B. L. Koon, W. C. Logan Jr, J. H. Ogle, R. A. Tyrrell, and R. F. Wills.  Simulator sickness during driving simulation studies.  Accident Analysis & Prevention, 42(3):788–796, 2010.
  * [6] P. Budhiraja, M. R. Miller, A. K. Modi, and D. Forsyth.  Rotation blurring: Use of artificial blurring to reduce cybersickness in virtual reality first person shooters.  arXiv preprint arXiv:1710.02599, 2017.
  * [7] J. W. Burke, M. McNeill, D. K. Charles, P. J. Morrow, J. H. Crosbie, and S. M. McDonough.  Optimising engagement for stroke rehabilitation using serious games.  The Visual Computer, 25(12):1085–1099, 2009.
  * [8] Z. Cao, J. Jerald, and R. Kopper.  Visually-induced motion sickness reduction via static and dynamic rest frames.  In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pages 105–112. IEEE, 2018.
  * [9] K. Carnegie and T. Rhee.  Reducing visual discomfort with hmds using dynamic depth of field.  IEEE computer graphics and applications, 35(5):34–41, 2015.
  * [10] M. Carrión, M. Santorum, J. Benavides, J. Aguilar, and Y. Ortiz.  Developing a virtual reality serious game to recreational therapy using iplus methodology.  In 2019 International Conference on Virtual Reality and Visualization (ICVRV), pages 133–137. IEEE, 2019.
  * [11] D. Checa and A. Bustillo.  A review of immersive virtual reality serious games to enhance learning and training.  Multimedia Tools and Applications, 79(9):5501–5527, 2020.
  * [12] D. Deponti, D. Maggiorini, and C. E. Palazzi.  Droidglove: An android-based application for wrist rehabilitation.  In 2009 International Conference on Ultra Modern Telecommunications & Workshops, pages 1–7. IEEE, 2009.
  * [13] Y. Farmani.  Discrete Viewpoint Control to Reduce Cybersickness in Virtual Environments.  PhD thesis, Carleton University, 2018.
  * [14] A. S. Fernandes and S. K. Feiner.  Combating vr sickness through subtle dynamic field-of-view modification.  In 2016 IEEE Symposium on 3D User Interfaces (3DUI), pages 201–210. IEEE, 2016.
  * [15] B. Ferreira and P. Menezes.  An adaptive virtual reality-based serious game for therapeutic rehabilitation.  2020\.
  * [16] L. Franke, L. Fink, J. Martschinke, K. Selgrad, and M. Stamminger.  Time-warped foveated rendering for virtual reality headsets.  In Computer Graphics Forum, volume 40, pages 110–123. Wiley Online Library, 2021.
  * [17] S. Grassini and K. Laumann.  Are modern head-mounted displays sexist? a systematic review on gender differences in hmd-mediated virtual reality.  Frontiers in Psychology, 11, 2020.
  * [18] A. Graves, A.-r. Mohamed, and G. Hinton.  Speech recognition with deep recurrent neural networks.  In 2013 IEEE international conference on acoustics, speech and signal processing, pages 6645–6649. IEEE, 2013.
  * [19] P. Howarth and P. Costello.  The occurrence of virtual simulation sickness symptoms when an hmd was used as a personal viewing system.  Displays, 18(2):107–116, 1997.
  * [20] R. Hussain, M. Chessa, and F. Solari.  Mitigating cybersickness in virtual reality systems through foveated depth-of-field blur.  Sensors, 21(12):4006, 2021.
  * [21] A. Kemeny, J.-R. Chardonnet, and F. Colombet.  Getting Rid of Cybersickness: In Virtual Reality, Augmented Reality, and Simulators.  Springer Nature, 2020.
  * [22] A. Kemeny, P. George, F. Mérienne, and F. Colombet.  New vr navigation techniques to reduce cybersickness.  Electronic Imaging, 2017(3):48–53, 2017.
  * [23] J. Kim, W. Kim, H. Oh, S. Lee, and S. Lee.  A deep cybersickness predictor based on brain signal analysis for virtual reality contents.  In Proceedings of the IEEE International Conference on Computer Vision, pages 10580–10589, 2019.
  * [24] K. Kim, M. Z. Rosenthal, D. Zielinski, and R. Brady.  Comparison of desktop, head mounted display, and six wall fully immersive systems using a stressful task.  In 2012 IEEE Virtual Reality Workshops (VRW), pages 143–144. IEEE, 2012.
  * [25] E. M. Kolasinski.  Simulator sickness in virtual environments.  Technical report, DTIC Document, 1995.
  * [26] R. Konrad, N. Padmanaban, K. Molner, E. A. Cooper, and G. Wetzstein.  Accommodation-invariant computational near-eye displays.  ACM Transactions on Graphics (TOG), 36(4):88, 2017.
  * [27] R. Kopper, C. Stinson, and D. Bowman.  Towards an understanding of the effects of amplified head rotations.  In The 3rd IEEE VR Workshop on Perceptual Illusions in Virtual Environments, volume 2, 2011.
  * [28] E. Langbehn, P. Lubos, and F. Steinicke.  Evaluation of locomotion techniques for room-scale vr: Joystick, teleportation, and redirected walking.  In Proceedings of the Virtual Reality International Conference-Laval Virtual, page 4. ACM, 2018.
  * [29] D. Lanzoni, A. Vitali, D. Regazzoni, and C. Rizzi.  Design of customized virtual reality serious games for the cognitive rehabilitation of retrograde amnesia after brain stroke.  Journal of Computing and Information Science in Engineering, 22(3), 2022.
  * [30] S. Lawrence, C. L. Giles, A. C. Tsoi, and A. D. Back.  Face recognition: A convolutional neural-network approach.  IEEE transactions on neural networks, 8(1):98–113, 1997.
  * [31] K.-L. Liao, M. Huang, Y. Wang, Z. Wu, R. Yang, J. Zhang, and L. Wang.  A virtual reality serious game design for upper limb rehabilitation.  In 2021 IEEE 9th International Conference on Serious Games and Applications for Health (SeGAH), pages 1–5. IEEE, 2021.
  * [32] J. J. Lin, H. Abi-Rached, and M. Lahav.  Virtual guiding avatar: An effective procedure to reduce simulator sickness in virtual environments.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 719–726. ACM, 2004.
  * [33] M.-C. Lorenzini, A. M. Hämäläinen, and W. Wittich.  Factors related to the use of a head-mounted display for individuals with low vision.  Disability and Rehabilitation, 43(17):2472–2486, 2021.
  * [34] T. Luo, N. Cai, Z. Li, Z. Pan, and Q. Yuan.  Vr-dlr: a serious game of somatosensory driving applied to limb rehabilitation training.  In International Conference on Entertainment Computing, pages 51–64. Springer, 2020.
  * [35] M. E. McCauley and T. J. Sharkey.  Cybersickness: Perception of self-motion in virtual environments.  Presence: Teleoperators & Virtual Environments, 1(3):311–318, 1992\.
  * [36] M. Melo, J. Vasconcelos-Raposo, and M. Bessa.  Presence and cybersickness in immersive content: effects of content type, exposure time and gender.  Computers & Graphics, 71:159–165, 2018.
  * [37] O. Merhi, E. Faugloire, M. Flanagan, and T. A. Stoffregen.  Motion sickness, console video games, and head-mounted displays.  Human factors, 49(5):920–934, 2007.
  * [38] A. Min, H. Min, and S. Kim.  Effectiveness of serious games in nurse education: A systematic review.  Nurse education today, 108:105178, 2022.
  * [39] B. Moyà Alcover, A. Jaume-i Capó, J. Varona, P. Martinez-Bueso, and A. Mesejo Chiong.  Use of serious games for motivational balance rehabilitation of cerebral palsy patients.  In The proceedings of the 13th international ACM SIGACCESS conference on Computers and accessibility, pages 297–298, 2011.
  * [40] M. A. Muhanna.  Virtual reality and the cave: Taxonomy, interaction challenges and research directions.  Journal of King Saud University-Computer and Information Sciences, 27(3):344–361, 2015.
  * [41] Y. K. Nakka, R. C. Foust, E. S. Lupu, D. B. Elliott, I. S. Crowell, S.-J. Chung, and F. Y. Hadaegh.  A six degree-of-freedom spacecraft dynamics simulator for formation control research.  2018\.
  * [42] N. Padmanaban, R. Konrad, T. Stramer, E. A. Cooper, and G. Wetzstein.  Optimizing virtual reality for all users through gaze-contingent and adaptive focus displays.  Proceedings of the National Academy of Sciences, page 201617251, 2017.
  * [43] B. Pavard and A. Berthoz.  Linear acceleration modifies the perceived velocity of a moving visual scene.  Perception, 6(5):529–540, 1977.
  * [44] T. C. Peck, L. E. Sockol, and S. M. Hancock.  Mind the gap: The underrepresentation of female participants and authors in virtual reality research.  IEEE Transactions on Visualization and Computer Graphics, 26(5):1945–1954, 2020.
  * [45] M. F. Pereira, E. Oliveira, N. F. Rodrigues, M. Bressler, J. Kolbenschlag, and C. Prahm.  Hand rehabilitation with virtual reality: preliminary learning results.  In 2021 IEEE 9th International Conference on Serious Games and Applications for Health (SeGAH), pages 1–8. IEEE, 2021.
  * [46] J. Plouzeau, J.-R. Chardonnet, and F. Merienne.  Using cybersickness indicators to adapt navigation in virtual reality: A pre-study.  In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pages 661–662. IEEE, 2018.
  * [47] T. Porcino, E. Clua, C. Vasconcelos, and D. Trevisan.  Dynamic focus selection for first-person navigation with head mounted displays.  SBGames, 2016.
  * [48] T. Porcino, E. O. Rodrigues, F. Bernardini, D. Trevisan, and E. Clua.  A symbolic machine learning approach for cybersickness potential-cause estimation.  In International Conference on Entertainment Computing, pages 115–126. Springer, 2021.
  * [49] T. Porcino, E. O. Rodrigues, F. Bernardini, D. Trevisan, and E. Clua.  Identifying cybersickness causes in virtual reality games using symbolic machine learning algorithms.  Entertainment Computing, 41:100473, 2022.
  * [50] T. Porcino, E. O. Rodrigues, A. Silva, E. Clua, and D. Trevisan.  Using the gameplay and user data to predict and identify causes of cybersickness manifestation in virtual reality games.  In 2020 IEEE 8th International Conference on Serious Games and Applications for Health (SeGAH), pages 1–8. IEEE, 2020.
  * [51] T. Porcino, D. Trevisan, and E. Clua.  Minimizing cybersickness in head-mounted display systems: causes and strategies review.  In 2020 22nd Symposium on Virtual and Augmented Reality (SVR), pages 154–163. IEEE, 2020.
  * [52] T. Porcino, D. Trevisan, and E. Clua.  A cybersickness review: causes, strategies, and classification methods.  Journal on Interactive Systems, 12(1):269–282, 2021.
  * [53] T. M. Porcino, E. Clua, D. Trevisan, C. N. Vasconcelos, and L. Valente.  Minimizing cyber sickness in head mounted display systems: design guidelines and applications.  In Serious Games and Applications for Health (SeGAH), 2017 IEEE 5th International Conference on, pages 1–6. IEEE, 2017.
  * [54] A. Ramsey, S. Nichols, and S. Cobb.  Virtual reality induced symptoms and effects (vrise) in four different virtual reality display conditions.  In Proceedings of HCI International (the 8th International Conference on Human-Computer Interaction) on Human-Computer Interaction: Ergonomics and User Interfaces-Volume I-Volume I, pages 142–146. L. Erlbaum Associates Inc., 1999.
  * [55] L. R. Rebenitsch.  Cybersickness prioritization and modeling.  Michigan State University, 2015.
  * [56] H. Renkewitz and T. Alexander.  Perceptual issues of augmented and virtual environments.  Technical report, FGAN-FKIE WACHTBERG (GERMANY), 2007.
  * [57] A. Ryge, L. Thomsen, T. Berthelsen, J. S. Hvass, L. Koreska, C. Vollmers, N. C. Nilsson, R. Nordahl, and S. Serafin.  Effect on high versus low fidelity haptic feedback in a virtual reality baseball simulation.  In 2017 IEEE Virtual Reality (VR), pages 365–366. IEEE, 2017.
  * [58] H. Sak, A. W. Senior, and F. Beaufays.  Long short-term memory recurrent neural network architectures for large scale acoustic modeling.  2014\.
  * [59] M. Sandlund, S. McDonough, and C. Häger-Ross.  Interactive computer play in rehabilitation of children with sensorimotor disorders: a systematic review.  Developmental Medicine & Child Neurology, 51(3):173–179, 2009\.
  * [60] S. Sanei and J. A. Chambers.  Eeg signal processing.  2007\.
  * [61] B. Sarupuri, M. L. Chipana, and R. W. Lindeman.  Trigger walking: A low-fatigue travel technique for immersive virtual reality.  In 2017 IEEE Symposium on 3D User Interfaces (3DUI), pages 227–228. IEEE, 2017.
  * [62] S. Sharples, S. Cobb, A. Moody, and J. R. Wilson.  Virtual reality induced symptoms and effects (vrise): Comparison of head mounted display (hmd), desktop and projection display systems.  Displays, 29(2):58–69, 2008.
  * [63] K. M. Stanney and R. S. Kennedy.  The psychometrics of cybersickness.  Communications of the ACM, 40(8):66–68, 1997.
  * [64] H. T. Tran, N. P. Ngoc, C. T. Pham, Y. J. Jung, and T. C. Thang.  A subjective study on qoe of 360 video for vr communication.  In 2017 IEEE 19th International Workshop on Multimedia Signal Processing (MMSP), pages 1–6. IEEE, 2017.
  * [65] Ubisoft.  Eagle flight.  https://www.ubisoft.com/en-us/game/eagle-flight.  Accessed: 2022-03-14.
  * [66] J. Van Waveren.  The asynchronous time warp for virtual reality on consumer hardware.  In Proc. 22nd ACM Conference on Virtual Reality Software and Technology, pages 37–46, 2016.
  * [67] N. Xie, G. Ras, M. van Gerven, and D. Doran.  Explainable deep learning: A field guide for the uninitiated.  arXiv preprint arXiv:2004.14545, 2020.
  * [68] T. Zhou, J. X. Chen, and M. Pullen.  Accurate depth of field simulation in real time.  In Computer Graphics Forum, volume 26, pages 15–23. Wiley Online Library, 2007.


# Improving Text-based Early Prediction by Distillation from
Privileged Time-Series Text

Jinghui Liua,b Daniel Capurroa Anthony Nguyenb Karin Verspoorc,a
a School of Computing and Information Systems, The University of Melbourne
b Australian e-Health Research Centre, CSIRO
c School of Computing Technologies, RMIT
<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

Modeling text-based time-series to make prediction about a future event or
outcome is an important task with a wide range of applications. The standard
approach is to train and test the model using the same input window, but this
approach neglects the data collected in longer input windows between the
prediction time and the final outcome, which are often available during
training. In this study, we propose to treat this neglected text as privileged
information available during training to enhance early prediction modeling
through knowledge distillation, presented as Learning using Privileged tIme-
sEries Text (LuPIET). We evaluate the method on clinical and social media
text, with four clinical prediction tasks based on clinical notes and two
mental health prediction tasks based on social media posts. Our results show
LuPIET is effective in enhancing text-based early predictions, though one may
need to consider choosing the appropriate text representation and windows for
privileged text to achieve optimal performance. Compared to two other methods
using transfer learning and mixed training, LuPIET offers more stable
improvements over the baseline, standard training. As far as we are concerned,
this is the first study to examine learning using privileged information for
time-series in the NLP context.

##  1 Introduction

Time-series forecasting, or early prediction, is an important machine learning
task with a wide range of applications, such as weather prediction
(Krasnopolsky and Fox-Rabinovitz, 2006; Espeholt et al., 2022) and stock
forecasting (Xu and Cohen, 2018; Sharma et al., 2017). Predicting future
events or outcomes would enable timely responses that can bring significant
social and economic benefits. Meanwhile, most existing works on forecasting or
early prediction use structured measurements or features as input (Steyerberg,
2009), and studies to leverage unstructured text to explore temporal patterns
are still scarce (Assale et al., 2019). Moreover, user-generated, domain-
specific textual data, such as clinical and social media texts, can be noisy
and complex to model (Baldwin et al., 2013; Huang et al., 2020). This creates
challenges in utilizing text for early prediction.

(a) Standard training

(b) LuPIET

(c) Transfer learning

(d) Mixed training

Figure 1: Different methods to leverage later data from the time-series to
assist early prediction. LuPIET refers to learning using privileged time-
series text in training via knowledge distillation. Here 1-day is the baseline
prediction window at test time. Models may leverage data from the prolonged
training windows, e.g., 3-day, to enhance the performance for the shorter test
window.

The standard framework for early prediction trains and tests machine learning
models using the same input window, depicted in Figure 1(a). Though this is
the widely adopted approach, it discards data that are outside the prediction
window but are collected in practice as part of the training set. Ideally,
these data can be utilized to enhance early prediction, such as learning the
future trajectory of the time-series to assist modeling. Leveraging data
available at training time but not at test time – referred to as privileged
information – for training has been proposed as Learning using Privileged
Information (LuPI) (Vapnik and Vashist, 2009; Vapnik and Izmailov, 2015).
Recent studies have shown LuPI can be successfully applied to utilize time-
series privileged information for early prediction (Hayashi et al., 2019; K.A.
Karlsson et al., 2022). However, these experiments focus on structured
features from synthetic data or distributions under certain assumptions. It
remains unknown whether the approach applies to text-based early prediction
applications, where natural language presents distinct characteristics and
variation.

In this study, we adapt the time-series LuPI to textual data, presented as
Learning using Privileged tIme-sEries Text (LuPIET). We evaluate LuPIET on a
range of tasks to evaluate its efficacy. LuPIET trains a more performant model
using a longer predictive window that includes data created after the target
prediction point as a teacher model. This is applied to guide the training of
the early model through knowledge distillation. Figure 1(b) gives an example
where the prediction window is 1-day, and we aim to guide the training of the
early model with the teacher model trained from a 3-day window. To compare
with LuPIET, we also apply two other methods, common in other domains but not
well-examined for text-based time-series, to leverage data collected after the
prediction time but available for training. These are transfer learning and
mixed training, depicted in Figure 1(c) and Figure 1(d), respectively.

We examine LuPIET using two challenging and domain-specific datasets
containing clinical and social media text. Specifically, we explore four risk
and diagnosis prediction tasks with clinical text and two mental health status
prediction tasks with social media. The results show LuPIET can be an
effective and stable approach for improving early prediction based on textual
input.

In summary, this work examines the usefulness of privileged time-series
information in the NLP context to support early prediction. Our main
contributions include:

  1. 1.

Proposing LuPIET to improve time-series modeling for text-based early
prediction.

  2. 2.

Evaluating the performance of LuPIET on two domains using clinical and social
media corpora, presenting results on six prediction tasks. We show that when
the privileged text is appropriately chosen and represented, LuPIET can
improve over the baseline for early prediction by being more sample efficient.

  3. 3.

Benchmarking the performances of two other competitive methods to support
early prediction. We find although they can sometimes outperform LuPIET, they
are in certain cases detrimental to modeling. LuPIET offers more consistent
and stable improvements over the baseline.

##  2 Related Work

Early prediction with text Forecasting or early prediction has been widely
studied in various domains and applications. For example, Steyerberg (2009)
demonstrates the different facets and modeling strategies for clinical
prediction modeling. Most initial works in the field focus on structured
measurements as input features, with some attempts to extract and include
shallow textual features or topics (Suresh et al., 2017; Ghassemi et al.,
2015, 2014). More recent studies aim to put more stress on text by applying
more powerful models to handle the complexity of language (Matero and
Schwartz, 2020; Seinen et al., 2022). They have shown promise in modeling
various types of text to support the prediction of mental health issues
(Halder et al., 2017), stock market trends (Xu and Cohen, 2018), and clinical
outcomes (Hsu et al., 2020).

Learning using privileged information LuPI presents a framework to leverage
features only available at the train time but not at test time (Vapnik and
Vashist, 2009). It has shown improved results in a range of applications,
including recommendation (Xu et al., 2020) and image processing (Lee et al.,
2020). Recently, the approach has been applied to improve early prediction
using time-series data, which leverages data observed between the prediction
time and the future outcome as privileged information. Hayashi et al. (2019)
examines this approach on a synthetic dataset and a real-world dataset on air
conditions with eight variables. K.A. Karlsson et al. (2022) further
formalizes the framework for time-series as Learning using Privileged Time-
Series (LuPTS) and proves it is guaranteed to result in more efficient
learning when the time-series are drawn from a non-stationary Gaussian-linear
dynamic system. However, none of the prior works examines text as input.

Knowledge distillation In knowledge distillation (KD), a more performant
teacher model guides a smaller student model to achieve better results by
matching the distributions of their predictions or output logits (Hinton et
al., 2015). By training with the teacher output, the student model is provided
with soft targets that contain more nuanced information about the label
distribution compared to the true, hard labels. KD has been widely used for
model compression (Sanh et al., 2019; Tung and Mori, 2019; Jiao et al., 2020)
and other machine learning applications (Furlanello et al., 2018; Clark et
al., 2019) to transfer knowledge across models with different strengths,
sizes, or even architectures. In contrast, the classic transfer learning
focuses on a single model and transfers knowledge across datasets, often from
larger datasets to smaller ones (Devlin et al., 2019). Early works have shown
the connection between LuPI and KD, unified them under generalized
distillation (Lopez-Paz et al., 2016). Distillation has become a standard
implementating technique to leverage privileged information (Hayashi et al.,
2019; Xu et al., 2020).

##  3 Methods

Here we present the problem setting for early prediction and then describe how
learning using privileged time-series text (LuPIET) works.

###  3.1 Problem setting

The goal for early prediction is to learn a mapping function $f(\theta)$
between input $X_{t,n}\in\mathcal{R}^{n\times d}$ and future events or
outcomes $Y\in\mathcal{R}$, where $t=1\ldots T$ is a time point of the time-
series defined by the prediction window, and $n=1\ldots N$ is a textual note
or post available in the window $X_{t}$. Since $N$ can vary across prediction
windows, we neglect the notation of $n$ from now on for simplicity. Note that
$[X_{1},X_{2},...,X_{T}]$ share the same label $Y$ as they come from the same
sample, where $X_{t}$ is always a subset of $X_{t+1}$. We assume the baseline
prediction window by setting $t=1$, and the baseline model trained in Figure
1(a) is obtained as

$\theta_{base}=\mathop{\arg\min}\limits_{\theta\in\Theta}\mathcal{H}(f(X_{1}),Y)$
(1)

where $\mathcal{H}$ is the cross entropy loss. We then aim to improve
$\theta_{base}$ by leveraging texts created chronologically after $X_{1}$,
namely $[X_{2},X_{3},...X_{T}]$.

###  3.2 Learning with privileged time-series text

LuPIET optimizes a knowledge distillation loss that maps the predictions
between the baseline model and the new model trained with privileged text,
which can be viewed as a teacher model. We train this teacher model using
input from a prolonged prediction window compared to baseline, namely $X_{t}$
where $t\geq 2$, to obtain

$\theta_{t}=\mathop{\arg\min}\limits_{\theta\in\Theta}\mathcal{H}(f(X_{t}),Y)$
(2)

Then let $p_{base}(x)$ and $p_{t}(x)$ be the output logits from the base model
and teacher model, and we scale them with a temperature $\tau$ before taking
the softmax, as defined in the original setting of knowledge distillation
(Hinton et al., 2015):

$\displaystyle p_{base}^{\tau}(x_{i})$
$\displaystyle=\frac{e^{p_{base}(x_{i})/\tau}}{\sum_{j=1}^{K}e^{p_{base}(x_{j})/\tau}}$
(3) $\displaystyle p_{t}^{\tau}(x_{i})$
$\displaystyle=\frac{e^{p_{t}(x_{i})/\tau}}{\sum_{j=1}^{K}e^{p_{t}(x_{j})/\tau}}$
(4)

where $K$ is the number of labels. We calculate the distillation loss as the
KL-divergence between the two scaled logit distributions as

$\mathcal{L}_{KD}=\mathcal{D}_{KL}(p_{base}^{\tau}(x)||p_{t}^{\tau}(x))$ (5)

This distillation loss is then added to the cross entropy loss to train the
final model that consumes the baseline input $X_{1}$, which is obtained by
optimizing the following training objective:

$\mathcal{L}=(1-\alpha)\mathcal{H}(f(X_{1}),Y)+\alpha\mathcal{L}_{KD}$ (6)

where $\alpha$ is a hyperparameter and $0\leq\alpha\leq 1$.

###  3.3 Other options to improve early baseline

Besides LuPIET, we also examine two other simple methods to leverage
privileged text to assist early training — transfer learning and mixed
training. Transfer learning (Zhuang et al., 2021) refers to further fine-
tuning $\theta_{t}$ using $X_{1}$ by minimizing the standard cross entropy
loss. In other words, we initialize the training in Eq 1 using the model
parameters from Eq 2, as shown in Figure 1(c).

In the mixed training method, the approach is to mix $X_{t}$ with $X_{1}$ and
train the model from scratch. This can be considered as a data augmentation
approach (Wen et al., 2021) which enriches the training set and encourages the
model to learn from all variations of the same sample. This is depicted in
Figure 1(d).

##  4 Experiments

We examine LuPIET using datasets from two challenging textual domains:
clinical notes and social media posts. All datasets contain notes or posts
that are created chronologically, naturally forming the text-based time
series. Here we introduce them in more detail.

###  4.1 Clinical datasets and tasks

For clinical text, we use the MIMIC-III database (Johnson et al., 2016) to
construct the datasets to predict four clinical outcomes and targets, which
are in-hospital mortality, in-ICU mortality, length-of-stay (LOS) over 3 days,
and diagnostic related groups (DRG). These are popular tasks in the literature
for clinical early prediction and we follow previous works to define and
extract cohorts for them (Wang et al., 2020; Liu et al., 2021, 2022a). We note
that the early DRG prediction is different from the typical medical coding
performed post-discharge (Dong et al., 2022; Liu et al., 2022b) as it aims to
predict diagnosis and estimate care costs while patients are still in the
hospital (Gartner et al., 2015; Islam et al., 2021).

For each patient in the cohort, we extract all clinical notes during the
hospital course that are created before the prediction time, sort them
chronologically, and remove empty or duplicated notes. The prediction window
is defined by the number of days after the patient ICU admission. For example,
the 3-day window would include all clinical notes charted by the end of the
third day of ICU admission.

We define the baseline window for the clinical prediction tasks as 1-day,
which would allow timely interventions and resource arrangements to be made
for the hospitalized patients. This is also a common choice made in the
literature (Wang et al., 2020; Hsu et al., 2020). We then examine two extended
prediction windows to train LuPIET, which are 3-day and 7-day. Notice there
are cases whose time-series lengths are less than 3-day or 7-day. We did not
distinguish these cases from others in the data extraction process to ensure
we can directly compare with baseline results. For example, if a case has a
length of 2 days, then the input texts for this case under 3-day and 7-day are
the same. The numbers of train/validation/test cases are presented in Table 1.
Notice the two mortality predictions and LOS prediction share the same cohort.

###  4.2 Social media datasets and tasks

For social media texts, we focus on the eRisk 2018 datasets (Losada and
Crestani, 2016; Losada et al., 2018) to use Reddit posts to predict potential
mental health issues, which are depression and anorexia. Predicting mental
health status using social media data is an important yet challenging task
(Guntuku et al., 2017) that draws much attention in recent years from the NLP
community (Benton et al., 2017; Cohan et al., 2018). To parse the datasets, we
use both the title and the content as input text for the post. The datasets
present the posts in ten chunks, which are evenly split by time and sorted in
the chronological order. We follow this format to define the prediction window
by the number of chunks used as input, e.g., a 3-chunk window includes all
posts in the first three chunks.

Since social media text could present noisier temporal patterns, we examine
two baseline windows for the prediction tasks, which are 3-chunk and 7-chunk,
and only use the full 10-chunk as prolonged prediction window. We make this
choice also because the eRisk datasets are relatively small and the
performance can be unstable. We use all samples from the datasets for each
task and split them in a 0.8/0.1/0.1 ratio, and the numbers are again
presented in Table 1.

| # train | # validation | # test | # labels
---|---|---|---|---
Mortality & LOS | 26729 | 3407 | 3392 | 2
DRG | 16296 | 972 | 1866 | 570
eRisk Depression | 656 | 82 | 82 | 2
eRisk Anorexia | 376 | 47 | 48 | 2

Table 1: Number of cases for the train, validation, and test sets of the
examined prediction tasks.

###  4.3 Text representation and modeling

We examine two text representations for the text-based time-series modeling.
The first one is to model all input text as a single text string by
concatenating all notes or posts together. This allows us to model the input
as a sequence of words, which considers the word-level details. This is a
standard practice in NLP. The other representation is to encode all notes or
posts into document embeddings and model them at the document level. This
method, on the other hand, may lose details during the document encoding
process, but it helps to maintain the temporal patterns of the texts.

For word-level representation, we use domain-specific pretrained word
embeddings for the two datasets. Specifically, we use BioWordVec (Zhang et
al., 2019) for clinical text and GloVe-840B (Pennington et al., 2014) for
social media. Since the concatenated text string is rather long, we adapt
MultiResCNN (Li and Yu, 2020) for modeling, which is a CNN-based model that
has shown strong results in long-document classification. The model enhances
the vanilla text-CNN model by adding more filters with residual connections.
Given space restrictions here, we do not introduce the architecture in detail
and refer readers to the original paper for more information.

For document-level modeling, we first encode notes or posts using BERT (Devlin
et al., 2019) by extracting the [CLS] token as the final representation. We
again try to align BERT with the domains of the text by using ClinicalBERT
(Alsentzer et al., 2019) for clinical notes and RoBERTa (Liu et al., 2019) for
Reddit posts. We then apply LSTM (Hochreiter and Schmidhuber, 1997) to model
the sequence of notes and use the last hidden state for final prediction. We
use LSTM due to its power to extract and model temporal patterns.

###  4.4 Training and evaluation

We tune the hyperparameters for all examined methods based on the validation
set and then retrain the model with the best configuration with multiple
random seeds. Specifically, we tune the filter number and sizes for the CNN
model, hidden size for the LSTM model, and the number of layer, dropout,
learning rate, weight decay for both models. Adam (Kingma and Ba, 2014) is
used as the optimizer throughout the experiments. After obtaining the best
architectural configuration for the baseline model, we fix the setup and tune
$\tau$ and $\alpha$ for LuPIET. We adopt random search for the clinical
datasets and grid search for social media datasets given the former takes much
longer to run. We facilitate the hyperparameter searching with asynchronous
successive halving algorithm (Li et al., 2020), based on the implementation
from Ray Tune (Liaw et al., 2018). We perform all our experiments using
pytorch (Paszke et al., 2019) and pytorch-
lightning111https://www.pytorchlightning.ai/ on V100 GPUs.

For evaluation, we use area under the receiver operating characteristics curve
(AUROC) and precision-recall curve (AUPR) for the binary classification tasks,
and accuracy and macro F1 for multiclass classification. We run the final,
tuned model for each task with five random seeds for the clinical datasets and
average the results, and run with ten seeds for the social media datasets
given the small data size.

##  5 Results

| In-hospital Mortality | In-ICU Mortality | LOS$>$3days | DRG
---|---|---|---|---
AUROC | AUPR | AUROC | AUPR | AUROC | AUPR | Acc | MacroF1
Baseline |  |  |  |
1-day |  0.867 (0.0032) |  0.474 (0.0151) |  0.862 (0.0110) |  0.373 (0.0252) |  0.690 (0.0034) |  0.625 (0.0047) |  0.282 (0.0058) |  0.105 (0.0106)
LuPIET |  |  |  |
3-day $\rightarrow$ 1-day |  0.876 (0.0027) |  0.491 (0.0039) |  0.880 (0.0020) |  0.429 (0.0092) |  0.693 (0.0042) |  0.626 (0.0058) |  0.287 (0.0068) |  0.116 (0.0067)
7-day $\rightarrow$ 1-day |  0.879 (0.0029) |  0.501 (0.0093) |  0.880 (0.0024) |  0.413 (0.0076) |  0.692 (0.0035) |  0.627 (0.0041) |  0.290 (0.0097) |  0.115 (0.0130)
Transfer |  |  |  |
3-day $\rightarrow$ 1-day |  0.863 (0.0026) |  0.466 (0.0063) |  0.866 (0.0060) |  0.381 (0.0231) |  0.695 (0.0036) |  0.612 (0.0049) |  0.293 (0.0033) |  0.134 (0.0059)
7-day $\rightarrow$ 1-day |  0.865 (0.0040) |  0.482 (0.0073) |  0.860 (0.0051) |  0.379 (0.0089) |  0.691 (0.0046) |  0.616 (0.0068) |  0.284 (0.0027) |  0.113 (0.0072)
7-day $\rightarrow$ 3-day $\rightarrow$ 1-day |  0.864 (0.0040) |  0.482 (0.0110) |  0.855 (0.0073) |  0.374 (0.0098) |  0.683 (0.0031) |  0.606 (0.0054) |  0.285 (0.0093) |  0.112 (0.0081)
Mix-train |  |  |  |
1-day \+ 3-day \+ 7-day |  0.866 (0.0031) |  0.490 (0.0031) |  0.860 (0.0079) |  0.398 (0.0077) |  0.642 (0.0037) |  0.561 (0.0057) |  0.298 (0.0025) |  0.140 (0.0081)

Table 2: Results of word-level modeling for the four clinical prediction
tasks.

| Depression | Anorexia
---|---|---
AUROC | AUPR | AUROC | AUPR
Baseline - 3-chunk |  0.819 (0.0636) |  0.458 (0.1606) |  0.796 (0.0742) |  0.334 (0.1071)
\+ LuPIET |  0.868 (0.0203) |  0.431 (0.1150) |  0.830 (0.0742) |  0.339 (0.0963)
Baseline - 7-chunk |  0.836 (0.0303) |  0.470 (0.1374) |  0.798 (0.0401) |  0.429 (0.1217)
\+ LuPIET |  0.869 (0.0332) |  0.495 (0.0792) |  0.807 (0.0239) |  0.423 (0.0705)

Table 3: Results for the two mental health prediction tasks using social media
posts.

###  5.1 Comparing LuPIET with baseline

We present our main results with the word-level modeling for clinical
prediction tasks in Table 2. We observe LuPIET using the extended 3-day and
7-day windows improves over the standard baseline window on 1-day for in-
hospital mortality, in-ICU mortality, and DRG predictions. Meanwhile, LuPIET
does not provide much benefit for predicting LOS$>$3days. We believe this is
related to the nature of the task as the extended windows are already longer
than the target in consideration, so the teacher models can take advantage of
this shortcut to make accurate and confident predictions that are close to the
true labels and can no longer serve as soft targets (Hinton et al., 2015; Cho
and Hariharan, 2019).

The results for social media datasets are presented in Table 3. Here we
examine 3-chunk and 7-chunk as the baseline windows and apply LuPIET trained
using full length, i.e., 10-chunk. We again observe the benefit of LuPIET over
the baseline results, especially under AUROC. However, we note that due to the
much smaller dataset size, the variances in the results can be large. In a
couple of cases, LuPIET achieves slightly lower AUPR scores than the baseline,
but the results are still comparable given the variance.

###  5.2 Comparing LuPIET with other methods

We focus on the clinical datasets to compare LuPIET with transfer learning and
mixed training approaches given they have relatively sufficient data to
observe their behavior. In Table 2, we find LuPIET still performs strongly
when compared to these two approaches, but other methods may outperform
LuPIET, such as mixed training on DRG prediction. In this particular case, we
believe this is caused by the DRG dataset being a multiclassification task
with 570 labels, thus not having enough samples for training. By mixing
different windows of the same sample together serves as a data augmentation
strategy, which alleviates the low-resource situation for DRG to achieve
better results. Similarly, when modeling at the note level (Table 4), transfer
learning may be able to better transfer the temporal relations across windows
thus achieving slightly better results.

However, we observe that transfer learning and mixed training do not always
improve over the baseline, which is further shown when we examine the note-
level modeling results in Table 4. For example, though it could bring benefits
to tasks like DRG, mixed training is rather detrimental to the LOS$>$3days
prediction. On the other hand, LuPIET either improves over the baseline or at
least maintains the performance under both text representation and modeling
strategies.

We also see LOS$>$3days task does not benefit much from transfer learning and
mixed training either, similar to the results with LuPIET. This shows when
adopting these methods to improve the time-series modeling, it can be
important to consider the nature of the task and to choose proper windows
accordingly.

###  5.3 Comparing text representations

The results presented in Table 2 and Table 4 for word- and note-level modeling
are comparable for all the four tasks. Overall, we see modeling at word level
performs much better than at note level, demonstrating the need to attend to
the fine-grained textual details for these tasks. The LOS$>$3days is again an
exception where the two sets of results are similar, showing the task can be
inherently more difficult. Furthermore, modeling at word level tends to
benefit more from LuPIET. For example, we see much better mortality prediction
scores with LuPIET in Table 2. This indicates that the proper training of
LuPIET exploits the nuances in the data and it would benefit more when
modeling at a finer granularity.

We do not present the results with document embeddings for the social media
tasks as their AUROC results are sub-optimal and barely over 0.5. We suspect
this is because the Reddit posts are much noisier and the model needs to sift
away much unrelated information, so encoding all posts into embeddings is
unhelpful.

| In-hospital Mortality | In-ICU Mortality | LOS$>$3days | DRG
---|---|---|---|---
AUROC | AUPR | AUROC | AUPR | AUROC | AUPR | Acc | MacroF1
Baseline |  |  |  |
1-day |  0.844 (0.0058) |  0.417 (0.0122) |  0.860 (0.0049) |  0.369 (0.0172) |  0.695 (0.0059) |  0.629 (0.0061) |  0.228 (0.0057) |  0.056 (0.0043)
LuPIET |  |  |  |
3-day $\rightarrow$ 1-day |  0.850 (0.0040) |  0.428 (0.0108) |  0.861 (0.0060) |  0.362 (0.0178) |  0.696 (0.0034) |  0.631 (0.0054) |  0.238 (0.0063) |  0.056 (0.0055)
7-day $\rightarrow$ 1-day |  0.851 (0.0033) |  0.430 (0.0125) |  0.860 (0.0046) |  0.370 (0.0168) |  0.698 (0.0027) |  0.635 (0.0039) |  0.237 (0.0042) |  0.057 (0.0029)
Transfer |  |  |  |
3-day $\rightarrow$ 1-day |  0.853 (0.0038) |  0.439 (0.0080) |  0.862 (0.0069) |  0.384 (0.0093) |  0.688 (0.0022) |  0.615 (0.0044) |  0.232 (0.0067) |  0.059 (0.0043)
7-day $\rightarrow$ 1-day |  0.833 (0.0037) |  0.415 (0.0086) |  0.861 (0.0065) |  0.371 (0.0123) |  0.686 (0.0056) |  0.623 (0.0057) |  0.230 (0.0070) |  0.064 (0.0039)
7-day $\rightarrow$ 3-day $\rightarrow$ 1-day |  0.829 (0.0073) |  0.405 (0.0108) |  0.861 (0.0080) |  0.362 (0.0147) |  0.688 (0.0029) |  0.621 (0.0022) |  0.232 (0.0052) |  0.066 (0.0049)
Mix-train |  |  |  |
1-day \+ 3-day \+ 7-day |  0.838 (0.0019) |  0.416 (0.0113) |  0.866 (0.0055) |  0.382 (0.0125) |  0.671 (0.0035) |  0.605 (0.0066) |  0.236 (0.0073) |  0.061 (0.0043)

Table 4: Results of document-level modeling for the four clinical prediction
tasks.

##  6 Discussion

Figure 2: Learning curves for the four clinical prediction tasks under two
evaluation metrics. The blue curves depict the results of standard training
and the red curves depict those of LuPIET.

###  6.1 Sampling efficiency

K.A. Karlsson et al. (2022) shows under certain conditions, such as when the
time-series is from linear dynamical systems with Gaussian noise and is in
Markov structure, privileged information is guaranteed to improve the learning
efficiency of time-series models. Given these conditions do not necessarily
hold for the natural language that has distinctive data distribution, here we
empirically examine if privileged text can still benefit NLP modeling
efficiency without making other assumptions. Figure 2 shows the learning
curves for the four clinical prediction tasks, with x-axis is the ratio of the
full dataset used to train the model. We find LuPIET can lead to more sample
efficient learning in multiple tasks. For example, when training with only 10%
of the whole dataset, privileged learning achieves significant improvements
over the baseline on in-hospital mortality, LOS, and DRG. When the model is
fed with more samples, we see the difference between LuPIET and the baseline
gradually converge. This also happens to the much smaller social media
datasets, where we see the larger extent of improvements with LuPIET under
AUROC in Table 3. Furthermore, LuPIET could reduce the modeling variance
compared to baseline, consistent with findings on structured time-series (K.A.
Karlsson et al., 2022).

Meanwhile, in certain scenarios, such as for the in-ICU mortality prediction
and for the LOS prediction with sufficient data, we find the benefits of
LuPIET are much weaker. This may reflect our observations on LOS in Sec 5.1,
where the choice of input windows can be important. Extending input windows in
certain scenarios may not necessarily include more data. For instance, in the
ICU admissions stays are much shorter compared to hospital stays. This may
explain why little benefit is observed for in-ICU mortality prediction.

###  6.2 Limitations

There are a few limitations with our study. Firstly, we find it is important
to apply LuPIET to appropriate task settings but we did not formalize the
definition of appropriateness, though we offer some possible intuitions (e.g.,
on the case of LOS in Sec 5.1). Domain knowledge about the nature of the task
may be needed to realize optimal results with LuPIET, which could in some ways
correspond to the assumptions about data distribution made for successful LuPI
in previous works (Lopez-Paz et al., 2016; K.A. Karlsson et al., 2022). Future
works are needed to provide the theoretical explanation for the empirical
results and to guide more effective application of LuPIET.

Secondly, we find sometimes the teacher models do not benefit from extending
the input window and consuming more time-series data, for example, the shorter
3-chunk can outperform 7-chunk for anorexia prediction (Table 3). We did not
further investigate this phenomenon and leave it to future work to explore the
potential causes. We also focused on the specific time steps for baseline and
extended input windows and did not evaluate in more time steps, but in the
future we would like to consider various time steps in the time-series for
both training and evaluation. Similar evaluation setup has been examined in
prior work (Harutyunyan et al., 2019), such as framing patient deterioration
assessment as hourly patient mortality predictions. We regard this setup a
promising extension of our current experiments to examine LuPIET. Lastly, we
do not explore how factors in successful KD applications (Gou et al., 2021),
such as creating consistent teacher models (Beyer et al., 2022), affect
LuPIET, which we consider as a future direction to better utilize privileged
information and to enhance LuPI in general.

##  7 Conclusion

In this study, we present LuPIET, a framework to incorporate longer-range
time-series data available during training to improve text-based early
predictions. Though similar ideas have been examined recently for structured
time-series (Hayashi et al., 2019; K.A. Karlsson et al., 2022), we are not
aware of any previous studies on the use of this privileged information in the
context of text-based time-series. We find LuPIET is an effective strategy for
enhancing early prediction and for efficient time-series modeling when applied
to appropriate task settings.

LuPIET is implemented by simply optimizing a distillation loss. Therefore,
future works may extend LuPIET by training with more advanced distillation
techniques, e.g., matching hidden state instead of logits (Zhang et al.,
2018), or combining with other inputs, e.g., using multi-modal privileged
information.

## Acknowledgements

We would like to thank the anonymous reviewers for their helpful reviews and
comments. This research was undertaken using the LIEF HPC-GPGPU Facility
hosted at the University of Melbourne. This Facility was established with the
assistance of LIEF Grant LE170100200.

## References

  * Alsentzer et al. (2019) Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019.  Publicly available clinical BERT embeddings.  In _Proceedings of the 2nd Clinical Natural Language Processing Workshop_ , pages 72–78, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
  * Assale et al. (2019) Michela Assale, Linda Greta Dui, Andrea Cina, Andrea Seveso, and Federico Cabitza. 2019.  The revival of the notes field: Leveraging the unstructured content in electronic health records.  _Frontiers of medicine_ , 6:66.
  * Baldwin et al. (2013) Timothy Baldwin, Paul Cook, Marco Lui, Andrew MacKinlay, and Li Wang. 2013.  How noisy social media text, how diffrnt social media sources? In _Proceedings of the Sixth International Joint Conference on Natural Language Processing_ , pages 356–364, Nagoya, Japan. Asian Federation of Natural Language Processing.
  * Benton et al. (2017) Adrian Benton, Margaret Mitchell, and Dirk Hovy. 2017.  Multitask learning for mental health conditions with limited social media data.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_ , pages 152–162, Valencia, Spain. Association for Computational Linguistics.
  * Beyer et al. (2022) Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. 2022.  Knowledge distillation: A good teacher is patient and consistent.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10915–10924.
  * Cho and Hariharan (2019) Jang Hyun Cho and Bharath Hariharan. 2019.  On the efficacy of knowledge distillation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 4794–4802.
  * Clark et al. (2019) Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D Manning, and Quoc V Le. 2019.  BAM! Born-Again Multi-Task networks for natural language understanding.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5931–5937, Florence, Italy. Association for Computational Linguistics.
  * Cohan et al. (2018) Arman Cohan, Bart Desmet, Andrew Yates, Luca Soldaini, Sean MacAvaney, and Nazli Goharian. 2018.  SMHD: a Large-Scale resource for exploring online language usage for multiple mental health conditions.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 1485–1497, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dong et al. (2022) Hang Dong, Matúš Falis, William Whiteley, Beatrice Alex, Joshua Matterson, Shaoxiong Ji, Jiaoyan Chen, and Honghan Wu. 2022.  Automated clinical coding: what, why, and where we are? _NPJ digital medicine_ , 5(1):159.
  * Espeholt et al. (2022) Lasse Espeholt, Shreya Agrawal, Casper Sønderby, Manoj Kumar, Jonathan Heek, Carla Bromberg, Cenk Gazen, Rob Carver, Marcin Andrychowicz, Jason Hickey, Aaron Bell, and Nal Kalchbrenner. 2022.  Deep learning for twelve hour precipitation forecasts.  _Nature communications_ , 13(1):5145.
  * Furlanello et al. (2018) Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. 2018.  Born again neural networks.  In _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pages 1607–1616. PMLR.
  * Gartner et al. (2015) Daniel Gartner, Rainer Kolisch, Daniel B Neill, and Rema Padman. 2015.  Machine learning approaches for early DRG classification and resource allocation.  _INFORMS journal on computing_ , 27(4):718–734.
  * Ghassemi et al. (2014) Marzyeh Ghassemi, Tristan Naumann, Finale Doshi-Velez, Nicole Brimmer, Rohit Joshi, Anna Rumshisky, and Peter Szolovits. 2014.  Unfolding physiological state: mortality modelling in intensive care units.  In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_ , KDD ’14, pages 75–84, New York, NY, USA. Association for Computing Machinery.
  * Ghassemi et al. (2015) Marzyeh Ghassemi, Marco A F Pimentel, Tristan Naumann, Thomas Brennan, David A Clifton, Peter Szolovits, and Mengling Feng. 2015.  A multivariate timeseries modeling approach to severity of illness assessment and forecasting in ICU with sparse, heterogeneous clinical data.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2015:446–453.
  * Gou et al. (2021) Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. 2021.  Knowledge distillation: A survey.  _International journal of computer vision_ , 129(6):1789–1819.
  * Guntuku et al. (2017) Sharath Chandra Guntuku, David B Yaden, Margaret L Kern, Lyle H Ungar, and Johannes C Eichstaedt. 2017.  Detecting depression and mental illness on social media: an integrative review.  _Current Opinion in Behavioral Sciences_ , 18:43–49.
  * Halder et al. (2017) Kishaloy Halder, Lahari Poddar, and Min-Yen Kan. 2017.  Modeling temporal progression of emotional status in mental health forum: A recurrent neural net approach.  In _Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis_ , pages 127–135, Copenhagen, Denmark. Association for Computational Linguistics.
  * Harutyunyan et al. (2019) Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. 2019.  Multitask learning and benchmarking with clinical time series data.  _Scientific data_ , 6(1):96.
  * Hayashi et al. (2019) Shogo Hayashi, Akira Tanimoto, and Hisashi Kashima. 2019.  Long-Term prediction of small Time-Series data using generalized distillation.  In _2019 International Joint Conference on Neural Networks (IJCNN)_ , pages 1–8.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015.  Distilling the knowledge in a neural network.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780.
  * Hsu et al. (2020) Chao-Chun Hsu, Shantanu Karnwal, Sendhil Mullainathan, Ziad Obermeyer, and Chenhao Tan. 2020.  Characterizing the value of information in medical notes.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 2062–2072, Online. Association for Computational Linguistics.
  * Huang et al. (2020) Rongtao Huang, Bowei Zou, Yu Hong, Wei Zhang, Aiti Aw, and Guodong Zhou. 2020.  NUT-RC: Noisy user-generated text-oriented reading comprehension.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 2687–2698, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Islam et al. (2021) Md Mohaimenul Islam, Guo-Hung Li, Tahmina Nasrin Poly, and Yu-Chuan Jack Li. 2021\.  DeepDRG: Performance of artificial intelligence model for Real-Time prediction of Diagnosis-Related groups.  _Healthcare (Basel, Switzerland)_ , 9(12).
  * Jiao et al. (2020) Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020.  TinyBERT: Distilling BERT for natural language understanding.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4163–4174, Online. Association for Computational Linguistics.
  * Johnson et al. (2016) Alistair E W Johnson, Tom J Pollard, Lu Shen, Li-Wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016.  MIMIC-III, a freely accessible critical care database.  _Scientific data_ , 3:160035.
  * K.A. Karlsson et al. (2022) Rickard K.A. Karlsson, Martin Willbo, Zeshan M Hussain, Rahul G Krishnan, David Sontag, and Fredrik Johansson. 2022.  Using time-series privileged information for provably efficient learning of prediction models.  In _Proceedings of The 25th International Conference on Artificial Intelligence and Statistics_ , volume 151 of _Proceedings of Machine Learning Research_ , pages 5459–5484. PMLR.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.
  * Krasnopolsky and Fox-Rabinovitz (2006) Vladimir M Krasnopolsky and Michael S Fox-Rabinovitz. 2006.  Complex hybrid models combining deterministic and machine learning components for numerical climate modeling and weather prediction.  _Neural networks: the official journal of the International Neural Network Society_ , 19(2):122–134.
  * Lee et al. (2020) Wonkyung Lee, Junghyup Lee, Dohyung Kim, and Bumsub Ham. 2020.  Learning with privileged information for efficient image Super-Resolution.  In _Computer Vision – ECCV 2020_ , pages 465–482. Springer International Publishing.
  * Li and Yu (2020) Fei Li and Hong Yu. 2020.  ICD coding from clinical text using Multi-Filter residual convolutional neural network.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(5):8180–8187.
  * Li et al. (2020) Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Jonathan Ben-tzur, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. 2020.  A system for massively parallel hyperparameter tuning.  In _Proceedings of Machine Learning and Systems_ , volume 2, pages 230–246.
  * Liaw et al. (2018) Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Stoica. 2018.  Tune: A research platform for distributed model selection and training.
  * Liu et al. (2021) Jinghui Liu, Daniel Capurro, Anthony Nguyen, and Karin Verspoor. 2021.  Early prediction of diagnostic-related groups and estimation of hospital cost by processing clinical notes.  _NPJ digital medicine_ , 4(1):103.
  * Liu et al. (2022a) Jinghui Liu, Daniel Capurro, Anthony Nguyen, and Karin Verspoor. 2022a.  “Note Bloat” impacts deep learning-based NLP models for clinical prediction tasks.  _Journal of biomedical informatics_ , 133:104149.
  * Liu et al. (2022b) Leibo Liu, Oscar Perez-Concha, Anthony Nguyen, Vicki Bennett, and Louisa Jorm. 2022b.  Hierarchical label-wise attention transformer model for explainable ICD coding.  _Journal of biomedical informatics_ , 133:104161.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized BERT pretraining approach.
  * Lopez-Paz et al. (2016) David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, and Vladimir Vapnik. 2016\.  Unifying distillation and privileged information.  In _International Conference on Learning Representations_.
  * Losada and Crestani (2016) David E Losada and Fabio Crestani. 2016.  A test collection for research on depression and language use.  In _Proc. of Experimental IR Meets Multilinguality, Multimodality, and Interaction, 7th International Conference of the CLEF Association, CLEF 2016_ , pages 28–39. Springer International Publishing.
  * Losada et al. (2018) David E Losada, Fabio Crestani, and Javier Parapar. 2018.  Overview of erisk: Early risk prediction on the internet.  In _Proc. of Experimental IR Meets Multilinguality, Multimodality, and Interaction, International Conference of the CLEF Association, CLEF 2018_ , pages 343–361. Springer International Publishing.
  * Matero and Schwartz (2020) Matthew Matero and H Andrew Schwartz. 2020.  Autoregressive affective language forecasting: A Self-Supervised task.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 2913–2923, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  PyTorch: An imperative style, High-Performance deep learning library.  In _Advances in Neural Information Processing Systems 32_ , pages 8024–8035. Curran Associates, Inc.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  GloVe: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.
  * Seinen et al. (2022) Tom M Seinen, Egill A Fridgeirsson, Solomon Ioannou, Daniel Jeannetot, Luis H John, Jan A Kors, Aniek F Markus, Victor Pera, Alexandros Rekkas, Ross D Williams, Cynthia Yang, Erik M van Mulligen, and Peter R Rijnbeek. 2022.  Use of unstructured text in prognostic clinical prediction models: a systematic review.  _Journal of the American Medical Informatics Association: JAMIA_ , 29(7):1292–1302.
  * Sharma et al. (2017) Ashish Sharma, Dinesh Bhuriya, and Upendra Singh. 2017.  Survey of stock market prediction using machine learning approach.  In _2017 International conference of Electronics, Communication and Aerospace Technology (ICECA)_ , volume 2, pages 506–509. ieeexplore.ieee.org.
  * Steyerberg (2009) E W Steyerberg. 2009.  Applications of prediction models.  In Ewout W Steyerberg, editor, _Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating_ , pages 11–31. Springer New York, New York, NY.
  * Suresh et al. (2017) Harini Suresh, Nathan Hunt, Alistair Johnson, Leo Anthony Celi, Peter Szolovits, and Marzyeh Ghassemi. 2017.  Clinical intervention prediction and understanding with deep neural networks.  In _Proceedings of the 2nd Machine Learning for Healthcare Conference_ , volume 68 of _Proceedings of Machine Learning Research_ , pages 322–337. PMLR.
  * Tung and Mori (2019) Frederick Tung and Greg Mori. 2019.  Similarity-preserving knowledge distillation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1365–1374.
  * Vapnik and Izmailov (2015) Vladimir Vapnik and Rauf Izmailov. 2015.  Learning using privileged information: Similarity control and knowledge transfer.  _Journal of machine learning research: JMLR_ , 16(61):2023–2049.
  * Vapnik and Vashist (2009) Vladimir Vapnik and Akshay Vashist. 2009.  A new learning paradigm: learning using privileged information.  _Neural networks: the official journal of the International Neural Network Society_ , 22(5-6):544–557.
  * Wang et al. (2020) Shirly Wang, Matthew B A McDermott, Geeticka Chauhan, Marzyeh Ghassemi, Michael C Hughes, and Tristan Naumann. 2020.  MIMIC-Extract: a data extraction, preprocessing, and representation pipeline for MIMIC-III.  In _Proceedings of the ACM Conference on Health, Inference, and Learning_ , CHIL ’20, pages 222–235, New York, NY, USA. Association for Computing Machinery.
  * Wen et al. (2021) Qingsong Wen, Liang Sun, Fan Yang, Xiaomin Song, Jingkun Gao, Xue Wang, and Huan Xu. 2021.  Time series data augmentation for deep learning: A survey.  In _Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21_ , pages 4653–4660. International Joint Conferences on Artificial Intelligence Organization.
  * Xu et al. (2020) Chen Xu, Quan Li, Junfeng Ge, Jinyang Gao, Xiaoyong Yang, Changhua Pei, Fei Sun, Jian Wu, Hanxiao Sun, and Wenwu Ou. 2020.  Privileged features distillation at taobao recommendations.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, KDD ’20, pages 2590–2598, New York, NY, USA. Association for Computing Machinery.
  * Xu and Cohen (2018) Yumo Xu and Shay B Cohen. 2018.  Stock movement prediction from tweets and historical prices.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1970–1979, Melbourne, Australia. Association for Computational Linguistics.
  * Zhang et al. (2019) Yijia Zhang, Qingyu Chen, Zhihao Yang, Hongfei Lin, and Zhiyong Lu. 2019.  BioWordVec, improving biomedical word embeddings with subword information and MeSH.  _Scientific data_ , 6(1):52.
  * Zhang et al. (2018) Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. 2018.  Deep mutual learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 4320–4328.
  * Zhuang et al. (2021) Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. 2021.  A comprehensive survey on transfer learning.  _Proceedings of the IEEE_ , 109(1):43–76.