# Hierarchical Reinforcement Learning for Modeling User Novelty-Seeking Intent
in Recommender Systems

Pan Li  New York University44 West 4th StreetNew YorkUSA<EMAIL_ADDRESS>,
Yuyan Wang  Google ResearchMountain ViewCaliforniaUSA<EMAIL_ADDRESS>,  Ed
H. Chi  Google ResearchMountain ViewCaliforniaUSA<EMAIL_ADDRESS>and  Minmin
Chen  Google ResearchMountain ViewCaliforniaUSA<EMAIL_ADDRESS>
###### Abstract.

Recommending novel content, which expands user horizons by introducing them to
new interests, has been shown to improve users’ long-term experience on
recommendation platforms (Chen et al., 2021). Users however are not constantly
looking to explore novel content. It is therefore crucial to understand their
novelty-seeking intent and adjust the recommendation policy accordingly. Most
existing literature models a user’s propensity to choose novel content or to
prefer a more diverse set of recommendations at individual interactions.
Hierarchical structure, on the other hand, exists in a user’s novelty-seeking
intent, which is manifested as a static and intrinsic user preference for
seeking novelty along with a dynamic session-based propensity. To this end, we
propose a novel hierarchical reinforcement learning-based method to model the
hierarchical user novelty-seeking intent, and to adapt the recommendation
policy accordingly based on the extracted user novelty-seeking propensity. We
further incorporate diversity and novelty-related measurement in the reward
function of the hierarchical RL (HRL) agent to encourage user exploration
(Chen et al., 2021). We demonstrate the benefits of explicitly modeling
hierarchical user novelty-seeking intent in recommendations through extensive
experiments on simulated and real-world datasets. In particular, we
demonstrate that the effectiveness of our proposed hierarchical RL-based
method lies in its ability to capture such hierarchically-structured intent.
As a result, the proposed HRL model achieves superior performance on several
public datasets, compared with state-of-art baselines.

User Novelty-Seeking Intent, Hierarchical Reinforcement Learning, Recommender
System, User Modeling

##  1\. Introduction

Recommender system constitutes one of the most important information filtering
systems that provide users with the most relevant content. Classic
recommendation models focus primarily on matching users with the most relevant
items based on their historical activities (Adomavicius and Tuzhilin, 2005).
Recent literature (Chen et al., 2022, 2021; Wang et al., 2022) however pointed
out the need for user exploration, when designing recommender systems. In
particular, users might get bored with repeated types of item recommendations
and would therefore prefer to seek novel content. By producing diversified and
unexpected recommendations for the targeted users, we could expand their
horizons, address their novelty-seeking desire and improve their online
experience as a result. To this end, multiple user exploration-based
recommendation models have been proposed, where researchers utilize bandit-
based methods (Li et al., 2010) and reinforcement learning-based methods
(Afsar et al., 2021) to exploit existing user interests while simultaneously
exploring new user interests. These models have achieved significant
recommendation performance improvements, leading to industrial adoption by a
number of major recommendation platforms (Chen et al., 2021).

Despite the great success achieved by these user exploration models,
understanding the user novelty-seeking intent and adapting recommendation
policies accordingly still remains challenging and under-explored.
Specifically, existing methods focus on identifying suitable exploratory
content for the user, without explicitly modeling the propensity of each user
to select novel and diverse content. There is a missed opportunity here as
user novelty-seeking intent can be affected by both the static, intrinsic user
preference and a dynamic session-based factor. As a result, the user novelty-
seeking intent can vary significantly across users, and for the same user, it
can fluctuate across different recommendation sessions or even within the same
recommendation session. For example, a power user with broad interests could
be content with consuming only content from their known interests, while a new
user could be drawn to explore and discover new interests. Meanwhile, the same
user could enjoy browsing curiosity-inducing content during the morning rush
hour on public transportation to kill time, while on a relaxing Saturday
night, he or she might prefer to pick up the TV series where they left off
last week. In another example, the user might want to switch to other genres,
after binge-watching the same TV series. The complex and hierarchical
structure of user novelty-seeking intent has not been explicitly and
systematically captured in the existing methods.

To this end, we propose a novel hierarchical reinforcement learning (HRL)
method to model the user novelty-seeking intent in recommendations, and to
update the recommendation policy accordingly. Specifically, motivated by the
Deep Reinforcement Learning technique (Mnih et al., 2015), we formulate two
modules in our proposed recommendation model: (1) the Session-Level DDPG,
which captures high-level, abstract, and session-based user novelty-seeking
intent. It stays static throughout the whole session, and gets updated when
the user enters a new recommendation session. (2) the Interaction-Level DQN,
which captures the dynamic and personalized user novelty-seeking intent
towards each item. It is updated upon every new interaction between a user and
an item. By taking into account both the session-level and interaction-level
novelty-seeking intent when modeling a user’s decision-making process, our
proposed method is able to produce more effective user exploration strategies
and superior recommendation performance.

We in addition study the design of reward functions in our proposed method. We
found it beneficial to explicitly incorporate novelty and diversity-based
metrics in the reward functions for optimizing the Session-Level DDPG and the
Interaction-Level DQN respectively, in order to further encourage user
interest exploration. We validate the effectiveness of our proposed method on
a simulation dataset and three real-world industrial datasets, where it
achieves significant recommendation performance improvement over selected
state-of-the-art recommendation baselines. We also conducted an extensive set
of ablation studies to understand the importance of each component in our
proposed method.

In summary, we make the following research contributions in this paper:

  * •

We provide empirical evidence to demonstrate the importance of modeling
hierarchical user novelty-seeking intent in the design of recommender systems.

  * •

We propose a novel hierarchical reinforcement learning-based recommendation
model that consists of a Session-level DDPG and an Interaction-level DQN to
capture the hierarchical user novelty-seeking intent and adapt the
recommendation policy accordingly.

  * •

We test our proposed method through extensive simulation and offline
experiments, showcasing that our model can effectively capture hierarchical
user novelty-seeking intent in recommendations, and achieves significant
performance improvement over state-of-the-art baselines.

##  2\. Related Work

###  2.1. User Novelty-Seeking Intent in Recommendations

Classic recommendation models focus primarily on matching contents similar to
known user interests (Adomavicius and Tuzhilin, 2005; Xiao and Benbasat,
2007). They often overlook the dispersion of the user’s recommendations, as
well as users’ desire to seek novel recommendations (Givon, 1984; Adamopoulos
and Tuzhilin, 2014). As a matter of fact, users can be interested in the
unshown items on the platform. Optimizing purely the relevance metric can lead
to feedback-loop biases (Sun et al., 2019) and the filter bubble phenomenon
(Pariser, 2011), reducing user satisfaction in the long run.

The ”exploration-exploitation” trade-off (Balabanović, 1998) has been
extensively studied in the context of recommender systems. The recommendation
agent faces the dilemma of determining whether to exploit the known user
interests by recommending items similar to their historical consumption, or to
explore new user interests by recommending novel items. Multi-arm bandits (Li
et al., 2010; Auer et al., 2002; Chapelle and Li, 2011) have long been used to
make the trade-off, in which one allocates a certain proportion of online
traffic for exploring user interests while exploiting the known interests in
the rest of the traffic (Agrawal and Goyal, 2012). Meanwhile, reinforcement
learning techniques (Mnih et al., 2015) formulate the Markov Decision Process
(MDP) (Shani et al., 2005) for decision making. These RL-based recommendation
models, which aim at identifying optimal recommendation actions to maximize
the long-term objectives, such as user retention rates and churn rates (Zhao
et al., 2019), make the trade-off implicitly as well. Some representative deep
reinforcement learning-based recommendation models include (Zheng et al.,
2018; Zou et al., 2019; Xin et al., 2020; Chen et al., 2021).

While these user exploration models have achieved great success in various
recommendation applications, they only focus on identifying suitable
exploratory content for the user, without properly modeling the propensity of
each user to select novel and diverse content. In particular, they do not
explicitly capture the hierarchical structure of user novelty-seeking intent.
As a result, the long horizon span of multi-session online user interactions
can easily render these methods sub-optimal. In this paper, we propose a novel
hierarchical reinforcement learning-based model to capture the hierarchical
user novelty-seeking intent and achieve significantly better performance as a
result.

###  2.2. Reinforcement Learning for Recommendation

As discussed in (Mnih et al., 2015; Sutton and Barto, 2018), reinforcement
learning provides a mathematical framework to capture dynamic user preferences
and learn recommendation policy to optimize long-term business objectives
(Zheng et al., 2018; Zou et al., 2019), and many reinforcement learning-based
models have been proposed for recommendation purposes (Xin et al., 2020; Zhang
et al., 2022; Antaris and Rafailidis, 2021; Xie et al., 2021; Xiao and Wang,
2021). For example, Liebman et al. (2019) proposed a reinforcement learning
framework to generate playlists according to the current context by adapting
to a listener’s sequential preferences within a listening session. Our work is
significantly different from all existing work in that, we explicitly model
hierarchical user novelty-seeking intent through a reinforcement learning-
based method to produce more effective exploration strategies in
recommendations. Deep reinforcement learning combines the modeling capacity of
deep neural networks and the MDP formulation of classic reinforcement learning
have achieved great success in many domains (Chen et al., 2019b; Afsar et al.,
2021).

###  2.3. Hierarchical Reinforcement Learning

Hierarchical Reinforcement Learning intends to address the sample inefficiency
of RL, especially in long-horizon problems, through temporal abstraction. One
class of HRL methods integrates hierarchical action-value functions that
operate at different temporal scales (Kulkarni et al., 2016). A top-level
q-value function learns a policy over intrinsic goals, while a lower-level
function learns a policy over atomic actions to satisfy the given goals. It
allows for flexible goal specifications, such as functions over entities and
relations, and provides an efficient space for exploration in complicated
environments. It has achieved great success across different applications,
such as robotics (Kulkarni et al., 2016), arcade learning (Bacon et al.,
2017), self-driving (Codevilla et al., 2018), and data-efficient learning
(Nachum et al., 2018). Motivated by the effectiveness of hierarchical
reinforcement learning methods, we propose a novel recommendation model to
capture the hierarchical user novelty-seeking intent by estimating the Q-value
in each recommendation session.

##  3\. Method

In this section, we introduce the hierarchical reinforcement learning-based
model to capture the hierarchical user novelty-seeking intent, and to improve
recommendation quality. We first formulate the problem under the Q-Learning
framework, and present the Hierarchical Q-learning solution. We then describe
the learning process of the Session-DDPG and Interaction-DQN respectively.
Finally, we discuss the design of reward functions and summarize our proposed
model. An overview of the model is shown in Figure 1.

Figure 1. Overview of the proposed Hierarchical Reinforcement Learning-Based
Model.

###  3.1. The Hierarchical Reinforcement Learning Framework

To start with, we formulate the task of determining the optimal recommendation
policy as a Markov Decision Process (MDP), and use Q-Learning (Watkins and
Dayan, 1992) to learn the policy. The Q-value of taking action $a$ in state
$s$ under a recommendation policy $\pi$ equals to

(1) $Q^{\pi}(s_{t};a_{t})=E_{\tau\sim
p_{\pi}(\tau|s_{t},a_{t})}[\sum_{t^{\prime}=t}\gamma^{t^{\prime}-t}r(s_{t^{\prime}},a_{t^{\prime}})]$

where $r(s_{t},a_{t})$ represents the immediate reward for taking action
$a_{t}$ under state $s_{t}$, and the discount factor $\gamma$ controls the
relative importance of the immediate reward and the future reward. At each
step $t$, the agent chooses the action that maximizes the Q-value, i.e.,
$a^{\ast}_{t}=\arg\max_{a}{Q^{\pi}(s_{t};a_{t})}$.

To estimate the Q-value in each recommendation session, and to capture the
hierarchical user novelty-seeking intent in recommendations, we adopt the Deep
Hierarchical Reinforcement Learning method from the robotics literature
(Kulkarni et al., 2016), and apply it to the recommendation tasks in a novel
manner. The tabular setting in classic reinforcement learning cannot handle
the enormous state and action spaces in recommendation settings. Furthermore,
the highly complex and nonlinear nature of the values of user-item
interactions requires high-capacity models. The deep neural network, on the
other hand, is capable of learning personalized and dynamic user-item
relationships in recommendations. In this work, we follow the Deep Q-Network
(DQN) technique (Mnih et al., 2015) and the Deep Deterministic Policy Gradient
(DDPG) technique (Lillicrap et al., 2015) that both utilize deep neural
networks to estimate the Q-function.

Specifically, our model consists of the following two modules: (1) the
Session-Level DDPG, which captures the high-level, abstract, and personalized
user novelty-seeking intent in each session; and (2) the Interaction-Level
DQN, which captures the dynamic and personalized user novelty-seeking intent
at each item interaction. The Session-Level DDPG takes the initial user state
at the beginning of each session as the input, and produces the session policy
$g_{t}$, which is a latent vector that controls the overall recommendation
policy within the current recommendation session. The Session-Level DDPG and
the latent policy vector $g_{t}$ will stay static through the same session,
and only get updated when the user enters a new recommendation session. We
explain the motivation of the design choice in Section 3.2 below. The
Interaction-Level DQN takes the dynamic user state as the input, and produces
the optimal action $a_{t}$ as the recommended product that will be provided to
the consumer. As the result, the Interaction-Level DQN and the produced action
$a_{t}$ will be updated upon every user-item interaction. By considering both
the session-level and interaction-level user novelty-seeking intent when
modeling users’ decision-making process, our algorithm is able to produce more
effective user exploration strategies and better recommendations. Our proposed
deep hierarchical reinforcement learning-based model is visualized in Figure
1.

We use a Dueling Double Deep Q-Network, which combines two variants of the
DQN, namely Double DQN (Van Hasselt et al., 2016) and Dueling DQN (Wang et
al., 2016) to construct both the Session-Level DDPG and the Interaction-Level
DQNs. The Dueling DQN decomposes the Q-value estimation into two separate
components: $V(s)$, the value of being in state $s$; and $A(s;a)$, the
advantage of taking action $a$ in state $s$. This decomposition leads to
better learning efficiency, more accurate Q estimation, and better policy than
the traditional DQN, as discussed in (Wang et al., 2016). The Double DQN
introduces a separate target network $Q^{\prime}$ for target Q-value
generation apart from the $Q$ network used for action selection. The $Q$
network is the primary network that is updated at every learning step, while
the target network $Q^{\prime}$ adopts periodical syncing to the $Q$ network.
Decoupling the two and allowing the slow update to the target network reduces
the overestimation of Q-values and stabilizes learning.

###  3.2. The Learning Process

As shown in Figure 1, the Session-level DDPG learns a session policy or goal
$g_{t}$ that supervises the interaction-level policy within this session. Note
that $g_{t}$ is a continuous, latent vector, and we cannot traverse all
possible options of $g_{t}$ to select the optimal values, as was done in the
classical DQN settings. Therefore, we follow the learning paradigm in DDPG
(Lillicrap et al., 2015) and produce the optimal session policy $g_{t}$
through a separate action network $\mu_{\phi}$. The action network consists of
multiple fully-connected hidden layers, which take the user state as the input
and output a latent vector $g_{t}$ as the goal. The parameters of the action
network will be updated through back-propagation when we minimize the Temporal
Difference (TD) loss described below.

When a user arrived at the beginning of a recommendation session $t$ with
state $s_{t}$, We learn the Session-level DDPG to maximize the expected
session return, which can be expressed by the following Bellman equation:

(2) $\begin{split}Q_{\theta_{ses}}(s_{t},g_{t})=r_{ses}(s_{t},g_{t})+\gamma
Q^{\prime}_{\theta^{\prime}_{ses}}(s_{t+N},g_{t+N})\\\ \mbox{ where
}g_{t}=\mu_{\phi_{ses}}(s_{t}),\
g_{t+N}=\mu_{\phi^{\prime}_{ses}}(s_{t+N})\end{split}$

where $g_{t}\in\mathcal{R}^{d}$ is a latent vector representing session-level
policy and will be learned through back-propagation from the action network
$\mu_{\phi_{ses}}$; $g_{t+N}$ is the session policy vector for the next
recommendation session, which is learned from the target action network
$\mu_{\phi^{\prime}_{ses}}$ by polyak averaging the action network parameters
over the course of training; $r_{ses}(s_{t},g_{t})$ represents the session-
level immediate reward for taking session goal $g_{t}$; $s_{t+N}$ is the user
state representation at the beginning of next recommendation session, when
acting according to $g_{t}$ in the current session; $Q_{\theta_{ses}}$ and
$Q^{\prime}_{\theta^{\prime}_{ses}}$ are the primary and target networks in
the Session-Level DDQG model that we have previously described.

Following the Double Dueling DQN design, we further decompose
$Q_{\theta_{ses}}(s_{t},a_{t})$ as

(3)
$\begin{split}Q_{\theta_{ses}}(s_{t},g_{t})=V_{\theta_{ses}}(s_{t})+A_{\theta_{ses}}(s_{t},g_{t})\end{split}$

Temporal Different (TD) learning is employed to learn the parameters of the Q
networks and action networks.

(4)
$\begin{split}&\ell_{ses}(\theta_{ses},\phi_{ses},\theta^{\prime}_{ses},\phi^{\prime}_{ses})=\\\
&\left[Q_{\theta_{ses}}(s_{t},g_{t})-\left(r_{ses}(s_{t},g_{t})+\gamma
Q^{\prime}_{\theta^{\prime}_{ses}}(s_{t+N},g_{t+N})\right)\right]^{2}\end{split}$

We further plug-in equation (3) to replace the Q networks with the value
networks (V networks) and advantage networks (A networks), and optimize these
two networks accordingly to produce the Q-value estimation for each action
$g_{t}$ under user state $s_{t}$.

Denote $\mathcal{A}$ as the discrete action space and $a\in\mathcal{A}$ as the
recommended item. We pick $a_{t}$ according to the interaction-level DQN,
conditioning on the Session-level policy $g_{t}$. By conditioning, we
concatenate the latent vector $g_{t}$ to the state vector $s_{t}$. The
interaction-level Q value, similarly, follows the Bellman equation as follows:

(5)
$\begin{split}Q_{\theta_{int}}(s_{t},a_{t};g_{t})=r_{int}(s_{t},a_{t})+\gamma
Q^{\prime}_{\theta^{\prime}_{int}}(s_{t+1},a_{t+1}^{\ast};g_{t})\\\ \mbox{
where
}a_{t+1}^{\ast}=\arg\max_{a^{\prime}}Q^{\prime}_{\theta^{\prime}_{int}}(s_{t+1},a^{\prime};g_{t})\end{split}$

Here $r_{int}(s_{t},a_{t})$ represents the interaction-level immediate reward.
$s_{t+1}$ is the user state transitioned from $s_{t}$ when taking action
$a_{t}$. We again use dueling to decompose the
$Q_{\theta_{int}}(s_{t},a_{t};g_{t})$, and learn the interaction-level value
network $V_{\theta_{int}}(s_{t})$ and advantage network
$A_{\theta_{int}}(s_{t},a_{t})$ using TD learning.

(6) $\begin{split}&\ell_{int}(\theta_{int},\theta^{\prime}_{int})=\\\
&\left[Q_{\theta_{int}}(s_{t},a_{t};g_{t})-\left(r_{int}(s_{t},a_{t})+\gamma
Q^{\prime}_{\theta^{\prime}_{int}}(s_{t+1},a_{t+1}^{\ast};g_{t})\right)\right]^{2}\end{split}$

We learn the Session-level DDPG and the Interaction-level DQN by jointly
minimizing the combined TD losses.

(7)
$\begin{split}\ell=\ell_{ses}(\theta_{ses},\phi_{ses},\theta^{\prime}_{ses},\phi^{\prime}_{ses})+\ell_{int}(\theta_{int},\theta^{\prime}_{int})\end{split}$

The user states are encoded through the GRU network (Chung et al., 2014),
where we feed the explicit user features and user-item interaction history as
inputs, and produce the dynamic user state representations from the sequential
neural network accordingly.

###  3.3. The Reward Function

To enable learning of our proposed model, we need to formulate the session-
level reward $r_{ses}(s_{t},g_{t})$ and the interaction-level reward
$r_{int}(s_{t},a_{t})$ based on the user-item interaction records. As user
reward cannot be explicitly observed in the offline experiment settings,
existing literature (Afsar et al., 2021) typically use the average item
ratings in a recommendation session as the proxy of the session-level reward
$r_{ses}(s_{t},g_{t})$, and the item rating as the proxy of the interaction-
level reward $r_{int}(s_{t},a_{t})$. However, these reward designs do not
capture the exploration-related objectives of the user. To further encourage
user exploration in recommendations, we propose to incorporate novelty-based
and diversity-based metrics in the reward functions of our proposed deep
hierarchical reinforcement learning-based model. In particular, the session-
level reward $r_{ses}(s_{t},g_{t})$ is formulated as

(8) $r_{ses}(s_{t},g_{t})=R_{ses}(s_{t},g_{t})+D_{ses}(s_{t},g_{t}),$

and the interaction-level reward $r_{int}(s_{t},a_{t})$ is formulated as

(9) $r_{int}(s_{t},a_{t})=R_{int}(s_{t},a_{t})+N_{int}(s_{t},a_{t}),$

where $R_{ses}$ is the average rating of all the interactions within a
session, and $R_{int}$ is the rating of the item in the interaction. $D_{ses}$
computes the average pairwise dissimilarity111See Section 4.1 below for the
dissimilarity measures. within the list of items in the current recommendation
session, and $N_{int}$ represents the deviation of the current recommended
item from the last item consumed by the user222See Section 4.1 below for the
novelty measures.. We empirically validated these designs through extensive
experiments on the simulation datasets and three real-world datasets. In the
experiments, we will show that explicitly incorporating novelty-based and
diversity-based metrics in the reward functions significantly improves user
exploration.

##  4\. Experiment on Simulation Datasets

To demonstrate the effectiveness of our proposed model, we conduct a series of
experiments on the simulation dataset, and compare its performance with
selected state-of-the-art recommendation baselines. We also conducted
extensive ablation experiments to shed light on the importance of different
components. We will now start by introducing the setup for our simulation
experiment.

###  4.1. Simulation Experiment Settings

In our simulation experiment, the user states will be dynamically updated as
the user interacts with the recommendation agent. Specifically, at _each_
timestamp $t$, we generate the interaction-level reward from the selected item
$i$ for user $u$ using the discrete choice model in the item response theory
(Embretson and Reise, 2013) as:

(10) $r_{ui,t}=A_{ui}+E_{u,t}*N_{ui}+r_{u}+r_{i}+e_{ui,t}$

where $r_{ui,t}$ represents the simulated reward of recommending item $i$ to
user $u$ at time $t$, $A_{ui}$ stands for the relevance (affinity) score
between user $u$ and item $i$, $E_{u,t}$ is a continuous value capturing the
novelty-seeking intent of the user in the current recommendation session, of
which the generation process is explained below. $N_{ui}$ is the novelty score
of item $i$ with regard to the user $u$, $r_{u}$ and $r_{i}$ represent the
intrinsic rating levels (fixed effects) for user $u$ and item $i$,
respectively. $e_{ui,t}$ is the random bias term that is added to the model to
simulate the fluctuation of product utility values for each consumer in
practice. In our simulation experiment, these variables are all sampled from
normal distributions with predetermined mean and variance values.
Specifically, we draw $r_{u}\backsim N(0.5,1)$, $r_{i}\backsim N(0.5,1)$ and
$e_{uit}\backsim N(0,0.1)$.

Meanwhile, the relevance objective in our simulation experiment is modeled as

(11) $A_{ui}=e_{u}^{T}e_{i},$

where $e_{u}$ and $e_{i}$ constitute the latent representations of explicit
user features and item features which are normalized to the unit sphere
(i.e.$||e_{u}||_{2}=1$, $||e_{i}||_{2}=1$). Each dimension of $e_{u}$ and
$e_{i}$ is sampled from the uniform distribution $U[0,1]$, and the similarity
function is defined as the inner product between the user representation and
item representation. We also model the user novelty-seeking intent as a
combination of user-based intrinsic intent and session-based intent, i.e.,

(12) $E_{ut}=E_{u}^{0}+E_{t}^{s(t)},$

where $E_{u}^{0}$ represents the intrinsic user propensity to explore novel
items, while $E_{t}^{s(t)}$ represents the session-level user novelty-seeking
intent for the current session $s(t)$. $E_{u}^{0}$ is sampled from $N(0,1)$,
and $E_{t}^{s(t)}$ for each session $s(t)$ is also sampled from $N(0,1)$,
where the session length is fixed at 5 in our experiments 333This can be
easily generalized to variable session length, or other definitions of
session. We have also conducted additional simulation experiments using
session lengths of 10 and 20, and observed similar levels of performance
improvements that we illustrate in this paper. Finally, we model the novelty
objective as

(13) $N_{ui}=||e_{i}-e_{i_{t-1}}||_{2},$

where $e_{i_{t-1}}$ is the embedding of the last consumed item, and
$||\cdot||_{2}$ is the Euclidean distance (item embeddings are normalized to
the unit sphere). Intuitively, $N_{ui}$ captures the dissimilarity between the
current item and the last consumed item by the user.

In our simulation experiment, we first generate the rating matrix $R_{u,i}$
with 10,000 consumers and 10,000 products, thus having 100,000,000 ratings in
total. At step 1, we recommend the top-1 product with the highest reward for
each user to be used as the interaction history; and then from step 2 to 40,
we construct the training set by recommending one product to the user each
time based on the policy from each individual agent; we then simulate the
reward as defined in equation (10) assuming the user would interact the
recommendation. We summarize the details for generating the trajectories for
each user in Algorithm 1, which is an online learning procedure when the agent
is learned through our proposed method. Algorithm 2 describes step 41-50,
where we roll out the learned policy from Algorithm 1 for another 10 steps,
and compute evaluation metrics as introduced in Section 4.2 below. The
evaluation results are reported as the average of 10 independent runs.

Input: $e_{u}$, $e_{i}$ for all users $u=1,...,U$ and all items $i=1,...,I$;
Model parameters $\theta_{ses}$, $\theta^{\prime}_{ses}$, $\phi_{ses}$,
$\phi^{\prime}_{ses}$ for the session-level DDPG network, and $\theta_{int}$,
$\theta^{\prime}_{int}$ for the interaction-level DQN; Reward discounting
factor $\gamma$; Learning rate $\alpha$; Session length $L=5$; Training steps
$n_{train}$ = 40.

1 Initiate the networks $Q_{\theta_{ses}}$ and $Q^{\prime}_{\theta_{ses}}$ for
session-level DDPG, $Q_{\theta_{int}}$ and $Q^{\prime}_{\theta_{int}}$ for
interaction-level DQN, and initial state $s_{u1}$, $g_{1}$.

2 for  _$u=1,...,U$_ do

3 for  _$t=1,...,n_{train}$_ do

4 Obtain $g_{t}$ from the action network $\mu_{\phi}(s_{ut})$ for
$Q_{\theta_{ses}}$, where $s_{ut}$ is the user state representation obtained
through the GRU network

5 Pick action
$a_{t}=\text{argmax}_{a^{\prime}}Q_{\theta_{int}}(s_{ut},a^{\prime};g_{t})$.

6 Observe $r^{int}_{ui,t}$ according to Eq.(9).

7 Update the Interaction-level DQN parameters $\theta_{int}$ through TD
learning using the newly observed sample
$(s_{u,t-1},a_{t-1},r^{int}_{ui,t},a_{t})$, and the target network parameters
$\theta^{\prime}_{int}$ using polyak averaging.

8 if  _$t\%L=0$_ then

9 Compute session-level reward $r^{ses}_{uit}$ as in Eq.(8).

10 Update the session-level DDPG parameters $\theta_{ses},\phi_{ses}$ through
TD learning using the newly observed sample
$(s_{u,t-L},g_{t-L},r^{ses}_{ui,t},g_{t})$, and the target network parameters
$\theta^{\prime}_{ses},\phi^{\prime}_{ses}$ using polyak averaging.

11

12 end if

13

14 end for

15

16 end for

Output: Updated model parameters $\theta_{ses}$, $\theta^{\prime}_{ses}$,
$\phi_{ses}$, $\phi^{\prime}_{ses}$, $\theta_{int}$ and
$\theta^{\prime}_{int}$.

Algorithm 1 Simulating the Training Data

1

Input: Learned parameters $\theta_{ses}$, $\theta^{\prime}_{ses}$,
$\phi_{ses}$, $\phi^{\prime}_{ses}$, $\theta_{int}$ and
$\theta^{\prime}_{int}$; $e_{u}$, $e_{i}$ for all users $u=1,...,U$ and all
items $i=1,...,I$; K=10 as number of recommendations per user; Number of test
steps $n_{test}=10$.

2 for  _$t=n_{train}+1,...,n_{train}+n_{test}$_ do

3 Get $s_{t}$ from the GRU component and Obtain $g_{t}$ from session-level
action network $\mu_{\phi_{ses}}(s_{t})$, and therein.

4 Generate the predicted Q-values for every action $a_{i}$, i.e.
$Q_{\theta_{int}}(s_{t},a_{t};g_{t})$.

5 Select the top K actions $a^{*}_{1},...,a^{*}_{K}$ based on the Q-values.

6 Sample $r_{u}$ $\backsim N(0.5,1)$

7 for  _$i\in\\{a^{*}_{1},...,a^{*}_{K}$}_ do

8 Sample $r_{i}\backsim N(0.5,1)$; Compute $A_{ui}$, $E_{u,t}$ and $N_{ui}$
according to Eq (11), Eq (12) and Eq (13); Sample $e_{ui,t}\backsim N(0,0.1)$.

9

10 end for

11 Obtain ground-truth rewards for the top K actions based on Eq.(10)

12 Compute Average Reward, Hit Rate@10, Diversity, and Novelty metrics defined
in Section 4.2.

13 Add the top-1 item into the history of the user, and update the user state
representation $s_{t}$ through the GRU network accordingly.

14 end for

15

Output: Average Reward, Hit Rate@10, Diversity and Novelty metrics.

Algorithm 2 Simulating the Test Data (for a single user).

###  4.2. Simulation Experiment Baselines and Metrics

To demonstrate the effectiveness of our proposed model, we compare its
performance with selected state-of-the-art recommendation baselines, ranging
from relevance-oriented baseline methods of DIN, DeepFM, Wide & Deep, and PNN,
to state-of-the-art reinforcement learning-based baseline methods of HRL-Rec,
REINFORCE and DRN. We summarize these baseline models below:

  * •

HRL-Rec (Xie et al., 2021) The Hierarchical reinforcement learning framework
for integrated recommendation (HRL-Rec) model produces the integrated
recommendation into two agents: the low-level agent is a channel selector,
which generates a personalized channel list; while the high-level agent is an
item recommender, which recommends specific items from heterogeneous channels
under the channel constraints.

  * •

REINFORCE (Sutton and Barto, 2018) The classic REINFORCE algorithm, which has
been successfully applied in a large-scale commercial recommender system (Chen
et al., 2019a) with the additional off-policy correction to address data
biases in learning from logged feedback.

  * •

DRN (Zheng et al., 2018) The DRN model is constructed based on Deep
Q-Learning, which explicitly models future rewards. The DRN model also
considers the user return pattern as a supplement to clicking labels to
capture more user feedback information and to establish an effective
exploration strategy.

  * •

DIN (Zhou et al., 2018) Deep Interest Network designs a local activation unit
to adaptively learn the representation of user interests from historical
behaviors with respect to a certain item.

  * •

DeepFM (Guo et al., 2017) DeepFM combines the power of factorization machines
for recommendation and deep learning for feature learning in a new neural
network architecture.

  * •

Wide & Deep (Cheng et al., 2016) Wide & Deep utilizes the wide model to handle
the manually designed cross-product features, and the deep model to extract
nonlinear relations among features.

  * •

PNN (Qu et al., 2016) Product-based Neural Network model introduces an
additional product layer to serve as the feature extractor.

In addition, we compare our method with several bandit-based recommendation
models that could balance between exploration and exploitation strategies in
recommendations, which include the following:

  * •

LinUCB (Li et al., 2010) LinUCB models the personalized recommendation task as
a contextual bandit problem, where the learning algorithm sequentially selects
items to serve users based on contextual information, while simultaneously
adapting its strategy based on user feedback to maximize total rewards.

  * •

TS (Chapelle and Li, 2011) Thompson Sampling (TS) is a method for choosing
actions to address exploration-exploitation in the multi-armed bandit problem
by choosing the action that maximizes the expected reward with respect to a
randomly drawn belief.

  * •

COFIBA (Li et al., 2016) The Collaborative Filtering Bandit method takes into
account the collaborative effects that arise due to the interaction of the
users with the items, and also takes advantage of preference patterns in the
data in the bandit-learning process.

Finally, we also construct multiple variants of our proposed model to shed
light on the importance of each component in our model design:

  * •

Ablation 1 (No Session Intent) In this ablation model, the user novelty-
seeking intent is solely determined by the user’s intrinsic novelty-seeking
level, without taking into account the session-based novelty-seeking intent.

  * •

Ablation 2 (No Hierarchical RL) In this ablation model, the recommendations
are provided following the classical reinforcement learning method, instead of
the hierarchical reinforcement learning method. That is to say, we remove the
Session-DDPG from our proposed model.

  * •

Ablation 3 (No Hierarchical RL+No Session Intent) In this ablation model, we
remove the Session-DDPG from our proposed model and formulate the user
novelty-seeking intent only based on the user’s intrinsic exploration level.

  * •

Ablation 4 (Vanilla DQN) In this ablation model, we formulate the networks of
both Session-DDPG and Interaction-DQN through the vanilla DQN method, instead
of the Double Dueling DQN method in our proposed model.

  * •

Ablation 5 (No Novelty) In this ablation model, we remove the novelty metric
from the interaction-level reward, i.e.
$r_{int}(s_{t},a_{t})=R_{int}(s_{t},a_{t})$ in Eq.(9).

  * •

Ablation 6 (No Diversity) In this ablation model, we remove the diversity
metric from the session-level reward, i.e.
$r_{ses}(s_{t},g_{t})=R_{ses}(s_{t},g_{t})$ in Eq.(8).

To evaluate the recommendation performance in the simulation experiment, we
consider the following four metrics:

  * •

Average Reward, which measures the average reward that the user could get from
the top-K (K=10) item recommendations generated by the recommender system. The
reward values are simulated following Eq.(10).

  * •

Hit Rate@K, which measures the percentage of ”positive” items (reward greater
than 0.5) in the top-K item recommendations generated by the recommender
system.

  * •

Diversity, which measures the pairwise dissimilarity in the top-K item
recommendations, calculated as the Euclidean distance between their latent
embeddings.

  * •

Novelty, which measures the deviation of the current item recommendation to
the last purchased item of the user, calculated as 1 minus the Euclidean
distance between their latent embeddings.

###  4.3. Simulation Experiment Results

We present the results of the simulation experiment in Table 1. We see that
our proposed model has achieved significantly better performance over all
selected state-of-the-art recommendation baselines in terms of all four
evaluation metrics (Average Reward, Hit Rate@10, Diversity, Novelty). These
superior recommendation performance results indicate that it is beneficial to
explicitly model the hierarchical user novelty-seeking intent when producing
recommendations, and that our proposed model could effectively capture such
novelty-seeking intent to provide significantly better recommendation
performance. We also verify that there is no significant difference in terms
of computational resources between our proposed model and the baseline
methods.

In addition, our proposed model also achieves significant and consistent
performance improvements over the multiple ablation variants. By comparing the
recommendation performance of our model with Ablation 1, we verify that the
session-level novelty-seeking intent is an important component that needs to
be properly modeled in the recommendation process. Ablation studies 2-3
further confirm that hierarchical reinforcement learning is an essential
component in our proposed model, removing which resulted in significantly
worsened recommendation performance. Compared with Ablation 4, we can see
introducing Double Dueling DQNs to reduce over-estimation and improve
stability in the classic DQN leads to better recommendation policy. Finally,
ablation settings 5 and 6 verify the hypothesis in the paper that it is
important to explicitly incorporate the novelty-based metric and diversity-
based metric in reward functions.

Algorithm | Avg-Reward | HR@10 | Diversity | Novelty
---|---|---|---|---
Our Model | 0.5080* | 0.5270* | 0.2567* | 0.2600*
| (0.0007) | (0.0010) | (0.0005) | (0.0005)
(Improvement over the best below) | +0.36% | +1.25% | +2.56% | +2.85%
HRL-Rec | 0.4916 | 0.4930 | 0.2496 | 0.2510
REINFORCE | 0.4924 | 0.4948 | 0.2501 | 0.2526
DRN | 0.4877 | 0.4774 | 0.2488 | 0.2496
DIN | 0.5012 | 0.5132 | 0.1734 | 0.1895
DeepFM | 0.5003 | 0.5118 | 0.1730 | 0.1906
Wide & Deep | 0.4916 | 0.5096 | 0.1728 | 0.1925
PNN | 0.4905 | 0.5077 | 0.1736 | 0.1925
LinUCB | 0.4773 | 0.4882 | 0.2488 | 0.2489
TS | 0.4816 | 0.4916 | 0.2462 | 0.2501
COFIBA | 0.4879 | 0.4997 | 0.2506 | 0.2473
Ablation 1 | 0.5044 | 0.5169 | 0.2503 | 0.2528
Ablation 2 | 0.5062 | 0.5205 | 0.2460 | 0.2509
Ablation 3 | 0.5034 | 0.5170 | 0.2472 | 0.2506
Ablation 4 | 0.4978 | 0.4147 | 0.2496 | 0.2496
Ablation 5 | 0.4998 | 0.5098 | 0.2275 | 0.2428
Ablation 6 | 0.4972 | 0.5066 | 0.2314 | 0.2444

Table 1. Comparison of recommendation performance in the simulation dataset.
‘*’ represents statistical significance at the 0.95 level. Improvement
percentages are computed over the best baseline model (including the ablation
studies) for each metric.

###  4.4. Visualization

To further demonstrate that our proposed hierarchical reinforcement learning-
based method indeed captures the intended construct of hierarchical user
novelty-seeking intent, we visualize the session policy $g_{t}$ of 100
randomly selected users from the simulation experiment (Figure 2). The
visualization is achieved using the TSNE technique (Van der Maaten and Hinton,
2008), which maps the high-dimensional latent session policy vector $g_{t}$
into the 2-dimensional latent space, in such a way that similar objects are
modeled by nearby points and dissimilar objects are modeled by distant points
with high probability. We then compare the session policy $g_{t}$ with the
ground truth user novelty-seeking intent in each recommendation session in our
simulation dataset. For better visualization purposes, we classify the ground
truth intent into three categories based on the intent values: ”novelty-
seeking intent ¿ 0.5”, ”novelty-seeking intent ¡ -0.5”, and ”-0.5 ¡ novelty-
seeking intent ¡ 0.5”. As we show in Figure 2, the TSNE visualization of the
session policy matches well with the hierarchical user novelty-seeking intent.
In particular, the sessions with high, medium, and low novelty-seeking intent
are clearly clustered into three disjoint groups. This means that the learned
session-level policy $g_{t}$ is indeed capturing the session-level novelty-
seeking intent. This validates the effectiveness of our proposed hierarchical
reinforcement learning model in capturing the hierarchical structures of user
novelty-seeking intent.

Figure 2. TSNE Visualization of the Session Policy

##  5\. Experiment on Real-World Datasets

###  5.1. Data and Experiment Settings

We further test our model on three real-world datasets: the Yelp Challenge
Dataset 444https://www.yelp.com/dataset/challenge, which is the Round 8
restaurant review dataset, which contains check-in information of users and
restaurants, and the user rating information; the MovieLens Dataset
555https://grouplens.org/datasets/movielens/, which contains information of
the users, movies, and ratings; and the Youku dataset collected from the major
online video platform Youku, which contains rich information of users, videos,
clicks, and their corresponding features. We list the descriptive statistics
of these datasets in Table 2. We normalize the ratings in Yelp and MovieLens
datasets into the scale of between 0 and 1 to construct the reward of
recommending each item to the user.

Dataset | Yelp | MovieLens | Youku
---|---|---|---
# of Ratings | 2,254,589 | 19,961,113 | 1,806,157
# of Users | 76,564 | 138,493 | 46,143
# of Items | 75,231 | 15,079 | 53,657
Sparsity | 0.039% | 0.956% | 0.073%

Table 2. Descriptive Statistics of Three Datasets

Algorithm | Alibaba-Youku | Yelp | MovieLens
---|---|---|---
Avg-Reward | HR@10 | Diversity | Novelty | Avg-Reward | HR@10 | Diversity | Novelty | Avg-Reward | HR@10 | Diversity | Novelty
Our Model | 0.4796* | 0.4383* | 0.2571* | 0.2488* | 0.4642* | 0.3854* | 0.2568* | 0.2486* | 0.4700* | 0.4030* | 0.2562* | 0.2487*
| (0.0009) | (0.0010) | (0.0005) | (0.0005) | (0.0008) | (0.0009) | (0.0005) | (0.0005) | (0.0008) | (0.0009) | (0.0005) | (0.0005)
(Improve %) | +1.89% | +1.55% | +5.02% | +5.42% | +2.16% | +2.47% | +4.86% | +4.50% | +2.55% | +3.71% | +9.02% | +2.09%
HRL-Rec | 0.4451 | 0.4181 | 0.2443 | 0.2349 | 0.4418 | 0.3639 | 0.2303 | 0.2343 | 0.4428 | 0.3811 | 0.2279 | 0.2306
REINFORCE | 0.4476 | 0.4189 | 0.2448 | 0.2360 | 0.4429 | 0.3651 | 0.2315 | 0.2364 | 0.4446 | 0.3832 | 0.2305 | 0.2333
DRN | 0.4438 | 0.4166 | 0.1774 | 0.1752 | 0.4401 | 0.3628 | 0.2306 | 0.2325 | 0.4423 | 0.3810 | 0.2298 | 0.2317
DIN | 0.4656 | 0.4284 | 0.1776 | 0.1758 | 0.4489 | 0.3677 | 0.1814 | 0.1816 | 0.4521 | 0.3864 | 0.1782 | 0.1754
DeepFM | 0.4648 | 0.4280 | 0.1793 | 0.1744 | 0.4486 | 0.3676 | 0.1830 | 0.1831 | 0.4526 | 0.3857 | 0.1782 | 0.1756
Wide & Deep | 0.4642 | 0.4271 | 0.1788 | 0.1766 | 0.4473 | 0.3668 | 0.1832 | 0.1830 | 0.4517 | 0.3857 | 0.1784 | 0.1770
PNN | 0.4610 | 0.4255 | 0.1806 | 0.1795 | 0.4451 | 0.3651 | 0.1855 | 0.1859 | 0.4508 | 0.3854 | 0.1790 | 0.1774
LinUCB | 0.4374 | 0.4166 | 0.2440 | 0.2338 | 0.4387 | 0.3619 | 0.2301 | 0.2330 | 0.4428 | 0.3832 | 0.2310 | 0.2321
TS | 0.4396 | 0.4169 | 0.2440 | 0.2316 | 0.4395 | 0.3628 | 0.2297 | 0.2341 | 0.4436 | 0.3830 | 0.2316 | 0.2327
COFIBA | 0.4428 | 0.4177 | 0.2412 | 0.2338 | 0.4417 | 0.3628 | 0.2301 | 0.2347 | 0.4436 | 0.3841 | 0.2316 | 0.2321
Ablation 1 | 0.4688 | 0.4287 | 0.2430 | 0.2336 | 0.4521 | 0.3742 | 0.2428 | 0.2371 | 0.4561 | 0.3816 | 0.2338 | 0.2430
Ablation 2 | 0.4707 | 0.4316 | 0.2445 | 0.2358 | 0.4544 | 0.3761 | 0.2449 | 0.2379 | 0.4583 | 0.3886 | 0.2349 | 0.2435
Ablation 3 | 0.4670 | 0.4270 | 0.2426 | 0.2341 | 0.4508 | 0.3740 | 0.2420 | 0.2376 | 0.4540 | 0.3878 | 0.2350 | 0.2436
Ablation 4 | 0.4643 | 0.4301 | 0.2306 | 0.2289 | 0.4516 | 0.3738 | 0.2299 | 0.2287 | 0.4544 | 0.3833 | 0.2315 | 0.2418
Ablation 5 | 0.4626 | 0.4289 | 0.1976 | 0.1968 | 0.4484 | 0.3710 | 0.2018 | 0.2016 | 0.4510 | 0.3810 | 0.2074 | 0.2088
Ablation 6 | 0.4608 | 0.4286 | 0.2058 | 0.2012 | 0.4450 | 0.3701 | 0.2034 | 0.2041 | 0.4499 | 0.3796 | 0.2063 | 0.2071

Table 3. Comparison of recommendation performance in three real-world
datasets. ‘*’ represents statistical significance at the 0.95 level.
Improvement percentages are computed over the best baseline model (including
the ablation studies) for each metric.

Note that, similarly to all other archival datasets in the field of
recommender systems, we can only observe a small portion of the ratings or
reward signal between all available user-item pairs. This is problematic for
unbiased offline evaluations of recommendation models, especially
reinforcement learning-based ones, which require the ground truth reward for
all possible user-item pairs. Therefore, in our real-world experiment, we
first impute the missing reward values by fitting our reward function as shown
in the simulation experiment (i.e., Eq.(10)) on the three real-world datasets.
Different from the simulation experiment where we draw the values of $r_{u}$,
$r_{i}$, and $e_{ij}$ from the predetermined normal distribution, these values
are determined by fitting the reward function on the observed reward in these
datasets. In addition, the user and item embeddings are generated from the
offline datasets using the Neural Collaborative Filtering (NCF) algorithm (He
et al., 2017), rather than generated randomly as in the simulation experiment,
due to the powerfulness and popularity of NCF-based methods shown in the
literature. We then report the experimental results as the average of 10
independent runs. The entire model is trained in Python using Tensorflow as
the backend on an MX450 GPU.

###  5.2. Experiment Results

Table 3 shows the performance of our proposed model as well as the baseline
models on the three real-world datasets. Similar to our findings in the
simulation experiment, our proposed hierarchical reinforcement learning-based
model achieves significant recommendation performance improvements over all
selected state-of-the-art baselines, and in terms of all four evaluation
metrics (Average Reward, Hit Rate@10, Diversity, Novelty). In addition,
similar to the results in Section 4.3, our model also significantly
outperforms the ablated alternatives, further indicating the importance of
each component in our model design.

##  6\. Conclusions

Balancing between exploiting users’ known interests, and exploring to help
users discover new interests, is critical to the design of modern industrial
recommender systems. While extensive works have been proposed to address the
challenge through classic bandits or reinforcement learning-based approaches,
they do not explicitly capture user novelty-seeking intent, nor adapt the
recommendation policy accordingly. We argue that hierarchical structure exists
in users’ novelty-seeking intent, and proposed a novel hierarchical
reinforcement learning-based model to capture such user novelty-seeking
intent. Our hierarchical agent includes a Session-DDPG to model user session-
level novelty-seeking intent and produce session policy/goal to guide an
interaction-DQN agent to make individual recommendations.

We conducted extensive simulation studies and experiments on three industrial
datasets, where we observed significant recommendation performance
improvements over selected state-of-the-art recommendation models. The
simulation study also validates that our hierarchical RL-based model can
indeed recover the hierarchical user novelty-seeking intent as constructed. We
also performed extensive ablation studies to shed light on the importance of
different components used in our model. We find that adopting deep
hierarchical reinforcement learning methods significantly would increase the
performance of recommender systems, and that it is beneficial to explicitly
model the novelty-based and diversity-based metrics in the design of the
reward functions used for learning. We hope the work can inspire future
research on discovering even better user exploration techniques.

For future work, we plan to strengthen our results further by studying the
impact of our proposed model in different recommendation setups where the
structure of user novelty-seeking intent differs, including e-commerce
platforms with monetary transactions vs organic content platforms. We also
plan to systematically study the long-term impact of our method.

## References

  * (1)
  * Adamopoulos and Tuzhilin (2014) Panagiotis Adamopoulos and Alexander Tuzhilin. 2014.  On unexpectedness in recommender systems: Or how to better expect the unexpected.  _ACM Transactions on Intelligent Systems and Technology (TIST)_ 5, 4 (2014), 1–32.
  * Adomavicius and Tuzhilin (2005) Gediminas Adomavicius and Alexander Tuzhilin. 2005.  Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions.  _IEEE transactions on knowledge and data engineering_ 17, 6 (2005), 734–749.
  * Afsar et al. (2021) M Mehdi Afsar, Trafford Crump, and Behrouz Far. 2021\.  Reinforcement learning based recommender systems: A survey.  _ACM Computing Surveys (CSUR)_ (2021).
  * Agrawal and Goyal (2012) Shipra Agrawal and Navin Goyal. 2012.  Analysis of thompson sampling for the multi-armed bandit problem. In _Conference on learning theory_. JMLR Workshop and Conference Proceedings, 39–1.
  * Antaris and Rafailidis (2021) Stefanos Antaris and Dimitrios Rafailidis. 2021.  Sequence adaptation via reinforcement learning in recommender systems. In _Proceedings of the 15th ACM Conference on Recommender Systems_. 714–718.
  * Auer et al. (2002) Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. 2002\.  Finite-time analysis of the multiarmed bandit problem.  _Machine learning_ 47, 2 (2002), 235–256.
  * Bacon et al. (2017) Pierre-Luc Bacon, Jean Harb, and Doina Precup. 2017\.  The option-critic architecture. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 31.
  * Balabanović (1998) Marko Balabanović. 1998\.  Exploring versus exploiting when learning user models for text recommendation.  _User Modeling and User-Adapted Interaction_ 8, 1 (1998), 71–102.
  * Chapelle and Li (2011) Olivier Chapelle and Lihong Li. 2011.  An empirical evaluation of thompson sampling.  _Advances in neural information processing systems_ 24 (2011).
  * Chen et al. (2019a) Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed H Chi. 2019a.  Top-k off-policy correction for a REINFORCE recommender system. In _Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining_. 456–464.
  * Chen et al. (2021) Minmin Chen, Yuyan Wang, Can Xu, Ya Le, Mohit Sharma, Lee Richardson, Su-Lin Wu, and Ed Chi. 2021.  Values of User Exploration in Recommender Systems. In _Fifteenth ACM Conference on Recommender Systems_. 85–95.
  * Chen et al. (2022) Minmin Chen, Can Xu, Vince Gatto, Devanshu Jain, Aviral Kumar, and Ed Chi. 2022\.  Off-Policy Actor-critic for Recommender Systems. In _Proceedings of the 16th ACM Conference on Recommender Systems_. 338–349.
  * Chen et al. (2019b) Xinshi Chen, Shuang Li, Hui Li, Shaohua Jiang, Yuan Qi, and Le Song. 2019b.  Generative adversarial user model for reinforcement learning based recommendation system. In _International Conference on Machine Learning_. PMLR, 1052–1061.
  * Cheng et al. (2016) Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. 2016\.  Wide & deep learning for recommender systems. In _Proceedings of the 1st workshop on deep learning for recommender systems_. 7–10.
  * Chung et al. (2014) Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  _arXiv preprint arXiv:1412.3555_ (2014).
  * Codevilla et al. (2018) Felipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy. 2018\.  End-to-end driving via conditional imitation learning. In _2018 IEEE international conference on robotics and automation (ICRA)_. IEEE, 4693–4700.
  * Embretson and Reise (2013) Susan E Embretson and Steven P Reise. 2013.  _Item response theory_.  Psychology Press.
  * Givon (1984) Moshe Givon. 1984\.  Variety seeking through brand switching.  _Marketing Science_ 3, 1 (1984), 1–22.
  * Guo et al. (2017) Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017.  DeepFM: a factorization-machine based neural network for CTR prediction.  _arXiv preprint arXiv:1703.04247_ (2017).
  * He et al. (2017) Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017\.  Neural collaborative filtering. In _Proceedings of the 26th international conference on world wide web_. 173–182.
  * Kulkarni et al. (2016) Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. 2016.  Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation.  _Advances in neural information processing systems_ 29 (2016).
  * Li et al. (2010) Lihong Li, Wei Chu, John Langford, and Robert E Schapire. 2010\.  A contextual-bandit approach to personalized news article recommendation. In _Proceedings of the 19th international conference on World wide web_. 661–670.
  * Li et al. (2016) Shuai Li, Alexandros Karatzoglou, and Claudio Gentile. 2016\.  Collaborative filtering bandits. In _Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval_. 539–548.
  * Liebman et al. (2019) Elad Liebman, Maytal Saar-Tsechansky, and Peter Stone. 2019\.  The right music at the right time: Adaptive personalized playlists based on sequence modeling.  _MIS Quarterly_ 43, 3 (2019).
  * Lillicrap et al. (2015) Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015.  Continuous control with deep reinforcement learning.  _arXiv preprint arXiv:1509.02971_ (2015).
  * Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015\.  Human-level control through deep reinforcement learning.  _nature_ 518, 7540 (2015), 529–533.
  * Nachum et al. (2018) Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. 2018.  Data-efficient hierarchical reinforcement learning.  _Advances in neural information processing systems_ 31 (2018).
  * Pariser (2011) Eli Pariser. 2011\.  _The filter bubble: How the new personalized web is changing what we read and how we think_.  Penguin.
  * Qu et al. (2016) Yanru Qu, Han Cai, Kan Ren, Weinan Zhang, Yong Yu, Ying Wen, and Jun Wang. 2016.  Product-based neural networks for user response prediction. In _2016 IEEE 16th International Conference on Data Mining (ICDM)_. IEEE, 1149–1154.
  * Shani et al. (2005) Guy Shani, David Heckerman, Ronen I Brafman, and Craig Boutilier. 2005.  An MDP-based recommender system.  _Journal of Machine Learning Research_ 6, 9 (2005).
  * Sun et al. (2019) Wenlong Sun, Sami Khenissi, Olfa Nasraoui, and Patrick Shafto. 2019.  Debiasing the human-recommender system feedback loop in collaborative filtering. In _Companion Proceedings of The 2019 World Wide Web Conference_. 645–651.
  * Sutton and Barto (2018) Richard S Sutton and Andrew G Barto. 2018.  _Reinforcement learning: An introduction_.  MIT press.
  * Van der Maaten and Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-SNE.  _Journal of machine learning research_ 9, 11 (2008).
  * Van Hasselt et al. (2016) Hado Van Hasselt, Arthur Guez, and David Silver. 2016\.  Deep reinforcement learning with double q-learning. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 30.
  * Wang et al. (2022) Yuyan Wang, Mohit Sharma, Can Xu, Sriraj Badam, Qian Sun, Lee Richardson, Lisa Chung, Ed H Chi, and Minmin Chen. 2022.  Surrogate for Long-Term User Experience in Recommender Systems. In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_. 4100–4109.
  * Wang et al. (2016) Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. 2016\.  Dueling network architectures for deep reinforcement learning. In _International conference on machine learning_. PMLR, 1995–2003.
  * Watkins and Dayan (1992) Christopher JCH Watkins and Peter Dayan. 1992.  Q-learning.  _Machine learning_ 8, 3 (1992), 279–292.
  * Xiao and Benbasat (2007) Bo Xiao and Izak Benbasat. 2007.  E-commerce product recommendation agents: Use, characteristics, and impact.  _MIS quarterly_ (2007), 137–209.
  * Xiao and Wang (2021) Teng Xiao and Donglin Wang. 2021.  A general offline reinforcement learning framework for interactive recommendation. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 35. 4512–4520.
  * Xie et al. (2021) Ruobing Xie, Shaoliang Zhang, Rui Wang, Feng Xia, and Leyu Lin. 2021.  Hierarchical reinforcement learning for integrated recommendation. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 35. 4521–4528.
  * Xin et al. (2020) Xin Xin, Alexandros Karatzoglou, Ioannis Arapakis, and Joemon M Jose. 2020.  Self-supervised reinforcement learning for recommender systems. In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval_. 931–940.
  * Zhang et al. (2022) Qihua Zhang, Junning Liu, Yuzhuo Dai, Yiyan Qi, Yifan Yuan, Kunlun Zheng, Fan Huang, and Xianfeng Tan. 2022\.  Multi-Task Fusion via Reinforcement Learning for Long-Term User Satisfaction in Recommender Systems. In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_. 4510–4520.
  * Zhao et al. (2019) Xiangyu Zhao, Long Xia, Jiliang Tang, and Dawei Yin. 2019\.  Deep reinforcement learning for search, recommendation, and online advertising: a survey.  _ACM sigweb newsletter_ Spring (2019), 1–15.
  * Zheng et al. (2018) Guanjie Zheng, Fuzheng Zhang, Zihan Zheng, Yang Xiang, Nicholas Jing Yuan, Xing Xie, and Zhenhui Li. 2018.  DRN: A deep reinforcement learning framework for news recommendation. In _Proceedings of the 2018 world wide web conference_. 167–176.
  * Zhou et al. (2018) Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018\.  Deep interest network for click-through rate prediction. In _Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining_. 1059–1068.
  * Zou et al. (2019) Lixin Zou, Long Xia, Zhuoye Ding, Jiaxing Song, Weidong Liu, and Dawei Yin. 2019\.  Reinforcement learning to optimize long-term user engagement in recommender systems. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 2810–2818.


11institutetext: Research Institute for Signals, Systems and Computational
Intelligence, sinc(i), CONICET, FICH-UNL (Argentina) 22institutetext: Division
of Anaesthesia, Department of Medicine, University of Cambridge (UK)
33institutetext: BioMedIA, Imperial College London (UK)

# Cranial Implant Design via Virtual Craniectomy with Shape Priors

Franco Matzkin 11 Virginia Newcombe 22 Ben Glocker 33 Enzo Ferrante 11

###### Abstract

Cranial implant design is a challenging task, whose accuracy is crucial in the
context of cranioplasty procedures. This task is usually performed manually by
experts using computer-assisted design software. In this work, we propose and
evaluate alternative automatic deep learning models for cranial implant
reconstruction from CT images. The models are trained and evaluated using the
database released by the AutoImplant challenge, and compared to a baseline
implemented by the organizers. We employ a simulated virtual craniectomy to
train our models using complete skulls, and compare two different approaches
trained with this procedure. The first one is a direct estimation method based
on the UNet architecture. The second method incorporates shape priors to
increase the robustness when dealing with out-of-distribution implant shapes.
Our direct estimation method outperforms the baselines provided by the
organizers, while the model with shape priors shows superior performance when
dealing with out-of-distribution cases. Overall, our methods show promising
results in the difficult task of cranial implant design.

###### Keywords:

Skull reconstruction self-supervised learning decompressive craniectomy shape
priors

##  1 Introduction

Crainoplasty is a surgical procedure aimed at repairing a skull vault defect
by insertion of a bone or nonbiological implant (e.g. metal or plastic) [1].
Such skull defect may exist due to different reasons, like a brain tumor
removal procedure or a decompressive craniectomy surgery following a traumatic
brain injury [12]. Cranial implant design is usually performed by experts
using computer-aided design software specifically tailored for this task [2].
The AutoImplant challenge, organized for the first time at MICCAI 2020, aims
at bench-marking the latest developments in computational methods for cranial
implant reconstruction. In this work, we propose and evaluate two approaches
to solve this task using deep learning models.

Previous works on skull and cranial implant reconstruction suggest that deep
learning models are good candidates to solve this task. In [13] a denoising
autoencoder was used to perform skull reconstruction, following an approach
similar to the recently proposed Post-DAE method [7, 6]. In this case, a
denoising autoencoder is trained to reconstruct full skulls from corrupted
versions. However, the model proposed in [13] works with skulls extracted from
magnetic resonance images, can only handle low resolution images and was
evaluated on the full-skull reconstruction task. Here we focus on
reconstructing the flap only, on skulls extracted from high resolution and
anisotropic computed tomography (CT) images. Other approaches rely on a head
symmetry assumption and propose to take advantage of it to reconstruct the
missing parts by mirroring the complete side of the skull [4]. However, this
is not a realistic assumption since missing flaps may occur in both sides
simultaneously. Another alternative could be the subtraction of the aligned
pre- and post-operative CT scans. Unfortunately, this requires to have access
to the pre-operative image, which may not be the case in real clinical
scenarios.

Recently, we have proposed [11] a simple virtual craniectomy procedure which
enables training different deep learning models in a self-supervised way,
given a dataset composed of full skulls. In this work, we compared two
different approaches: direct estimation of the implant, or reconstruct-and-
subtract (RS) strategies where the full skull is first reconstructed, and then
the original image is subtracted from it to generate a difference map. We
evaluated different architectures and concluded that direct estimation
produces more accurate estimates than RS strategies, since the latter one
tends to generate noise in areas far from the flap. A different approach has
been introduced by the AutoImplant challenge organizers [9] which also employs
deep learning models, but it works in two steps. First, a low resolution
version of the image is reconstructed to localize the area where the defected
region is located. Then, they extract a 3D patch from the high resolution
image and process it using a second neural network trained for fine implant
prediction.

In this work, based on the conclusions from [11], we employ a direct
estimation method that operates on full skulls which are rigidly registered to
an atlas and resampled to an intermediate resolution. Aligning the images
allow us to work in a common space which simplifies the reconstruction task.
We adapt the virtual craniectomy procedure to account for more realistic flap
shapes, similar to the ones introduced in the AutoImplant challenge. Moreover,
we propose to incorporate anatomical priors into the standard direct
estimation model introduced in [11] by feeding the registered skull atlas as
an extra image channel. Previous works [8] have shown that incorporating
approximate shape priors as additional image channels is a simple yet
effective way to increase the anatomical plausibility of the segmentations,
since it provides supplementary context information to the network. We compare
the results of our two methods with those obtained by the baseline benchmark
model introduced in [9], showing the superiority of our approach.

Figure 1: Examples of images from the $\mathcal{D}_{\mathrm{test}}$ set and
$\mathcal{D}_{\mathrm{test-extra}}$ (out-of-distribution cases). As it can be
observed, images from $\mathcal{D}_{\mathrm{test}}$ follow a common pattern,
while those in $\mathcal{D}_{\mathrm{test-extra}}$ present different defects
with various shapes.

##  2 Challenge description and database

The AutoImplant challenge organizers provided 100 images for training
($\mathcal{D}_{\mathrm{train}}$) and 110 images for testing. From the 110 test
images, 100 of them (denoted here as $\mathcal{D}_{\mathrm{test}}$) have
simulated surgical defects which follow the same distribution as the ones on
the training images, while the remaining 10 (denoted as
$\mathcal{D}_{\mathrm{test-extra}}$) have defects which do not follow the same
distribution (see Figure 1). The images were selected from the CQ500 public
database111The database can be accessed at: http://headctstudy.qure.ai/dataset
[3]. They have fixed image dimension in the axial plane (512 x 512) and a
variable number of axial slices Z.

The training dataset ($\mathcal{D}_{\mathrm{train}}$) is composed of triplets
($\mathcal{X}^{\mathrm{full}},\mathcal{X}^{\mathrm{defected}},\mathcal{Y}$),
where $\mathcal{X}^{\mathrm{full}}$ is the full skull,
$\mathcal{X}^{\mathrm{defected}}$ corresponds to the defected skull and
$\mathcal{Y}$ to the removed defect that we aim at reconstructing. For the
test images, only the $\mathcal{X}^{\mathrm{defected}}$ images were released.
We evaluated the proposed methods in the test images and submitted the results
to the organizers, who computed the metrics reported in this paper. It is
important to note that, in order to avoid overfitting to the test data, we
could submit our results a maximum of 5 times.

##  3 Methods

Figure 2: Modified virtual craniectomy procedure. We incorporated new template
shapes for the virtual craniectomy to account for the pattern found in the
AutoImplant challenge dataset.

The proposed cranial implant reconstruction methods operate on the space of
binary volumetric masks. Such binary skull can be obtained by simply
thresholding a brain CT image according to the Hounsfield scale, or applying
more sophisticated methods. In the AutoImplant challenge, the skulls were
already provided as binary volumes, extracted from the CT images using
thresholding and additional post-processing steps (for further details we
refer to [9]). Since the training data includes the full skulls, we leveraged
the virtual craniectomy procedure proposed in [11] to train our models.

###  3.1 Virtual Craniectomy and Data Augmentation

Given a full skull, we designed a virtual craniectomy procedure which consists
in removing a bone flap using a template located in a random position along
its upper part. In [11], spherical template shapes were used. By visual
inspection of the AutoImplant training data, we observed that defects tend to
follow a pattern given by the intersection of the skull with a cube with two
cylinders over the edges perpendicular to the axial planes. So, we designed a
variable-size template shape which produces similar defects, as shown in
Figure 2. To increase the diversity of our training procedure, we also
included spherical and cubic templates of random sizes (all of the three
shapes were selected with equal probability).

The virtual craniectomy was used as a data augmentation mechanism to generate
a variety of training samples from a limited amount of full skulls, resulting
in a self-supervised learning approach, where no annotated skull defects are
required for training. We also included salt and pepper noise in the input
images with probability 0.01. Moreover, we also considered the defective
skulls provided by the organizers as part of our datasets (in these cases,
virtual craniectomy was not performed). During training, we sampled images
coming from both sources: simulated virtual craniectomies and defective skulls
provided by the organizers.

###  3.2 Common space alignment

Before training, all the images were rigidly registered to a common space
determined by a full skull atlas. It consists in a thresholded version of a
full-skull head CT atlas constructed by averaging several healthy head CT
images. Such atlas allowed us to normalize the images by resampling them to an
intermediate resolution. We chose this resolution to be 0.695 x 0.695 x 0.715
mm (resulting in a volume of 304 x 304 x 224 voxels) because it was the
maximum size we managed to fit in GPU memory. Moreover, aligning the images in
a common space simplifies the reconstruction task for the neural network,
since it can focus on shape variations which are more relevant to the
reconstruction task than translations and rotations. We used the FLIRT
software package [5] for rigid registration. At test time, given a test
defective image $\mathcal{X}^{\mathrm{defected}}_{i}$, we apply the same
registration procedure which returns a transformation $\mathcal{T}$ and its
inverse $\mathcal{T}^{-1}$. The transformation is applied to the original
image $\mathcal{T}\circ\mathcal{X}^{\mathrm{defected}}_{i}$. The estimated
skull defect $\hat{\mathcal{Y}}_{i}$ is reconstructed in the common space, and
the final estimate in the original space is recovered by applying the inverse
transformation $\mathcal{T}^{-1}\circ\hat{\mathcal{Y}}_{i}$.

Figure 3: (a) The images are first registered to an atlas space, and resampled
to a common resolution. We store the resulting transform $\mathcal{T}$ and its
inverse $\mathcal{T}^{-1}$. (b) We compare two different approaches for the
implant reconstruction task. The first one is a standard DE-UNet model. The
second one incorporates a shape prior by considering the atlas as an extra
input channel to the network. After prediction, the segmentation mask is
mapped-back to the original image space using the inverse transform
$\mathcal{T}^{-1}$.

###  3.3 Direct estimation

Our first method is a direct estimation model which follows the same
architecture as the DE-UNet used in [11]. It is a standard 3D UNet encoder-
decoder architecture with skip connections, trained using a compound loss
which combines Dice and cross-entropy terms [14] (for more details, we refer
to our work in [11]). After reconstruction, the segmentation is re-mapped to
the original resolution using the inverse transform $\mathcal{T}^{-1}$ as
previously discussed.

The model is trained using batches with full volume images, pre-aligned in the
common space and resampled to an intermediate resolution as previously
discussed.

###  3.4 Direct estimation with shape priors

Since the DE-UNet model is a fully convolutional architecture, the receptive
field of the model is mainly determined by the amount of layers and parameters
of the pooling and convolution operations. In other words, the local support
of the output predictions is restricted to a certain area in the input image.
When we have to reconstruct big or out-of-distribution skull defects, it may
happen that most of the image support for certain parts of it are background,
so the network may have no context to infer the implant shape. To overcome
this limitation and make our model robust, we propose to incorporate context
via shape priors given as an extra channel to the segmentation network.
Previous works [8] have shown that this simple extension can boost the
robustness of existing state-of-the-art pixel-wise approaches in medical image
segmentation tasks.

We take advantage of the fact that images are co-registered to a common space,
and use the same skull atlas as shape prior. After registration, we
concatenate the resampled image with the atlas as a extra input channel, and
train the network following the same strategy discussed before. In this case,
the shape prior acts as a kind of initialization for the network’s output,
providing additional context that will be useful specially to reconstruct out-
of-distribution defects. We refer to this model as DE-Shape-UNet.

Figure 4: Comparison of the results for the proposed methods in terms of Dice
and Hausdorff Distance (HD). HD is shown in log scale for better
visualization.

###  3.5 Implementation details

The models were implemented in Pyhton, using the PyTorch 1.4 library. We
trained and evaluated the CNNs using an NVIDIA TITAN Xp GPU with 12GB of RAM.
The same virtual craniectomy and data augmentation procedure was used to train
both models. In both cases we used a compound loss function which combines
Dice loss and Binary Cross Entropy (BCE) as $L=L_{Dice}+\lambda L_{BCE}$
(parameter $\lambda$ was set to $\lambda=1$ by grid search). Both models
followed the DE-UNet architecture described in [11]; the only difference
between them was that we concatenated the atlas as an extra input channel in
the DE-Shape-UNet model. For optimization, we used Adam with initial learning
rate of 1e-4. The batch-size was set to 1 for memory restrictions. The models
were trained for 50 epochs. The 100 training images were split in 95 images
for training and 5 for validation. After 50 epochs, we kept the model that
achieved best accuracy in the validation fold.

##  4 Results

Figure 4 and Table 1 include a quantitative comparison of the results. We
report Dice coefficient and Hausdorff distance measured in the
$\mathcal{D}_{\mathrm{test}}$ (100 images), $\mathcal{D}_{\mathrm{test-
extra}}$ (10 images) and the whole test dataset. We observe that DE-Shape-UNet
presents better performance for out-of-distribution cases
($\mathcal{D}_{\mathrm{test-extra}}$), while DE-UNet outperforms the other
model in the $\mathcal{D}_{\mathrm{test}}$ set. Since the whole test dataset
is composed of 100 images from $\mathcal{D}_{\mathrm{test}}$ and only 10
images from $\mathcal{D}_{\mathrm{test-extra}}$, the DE-UNet model shows
better performance in the overall comparison. Moreover, DE-UNet model
outperforms the two baseline models (N1 and N2) reported by the organizers in
[9]. Figure 5 provides some visual examples for reconstructions obtained with
both methods in samples from $\mathcal{D}_{\mathrm{test}}$ and
$\mathcal{D}_{\mathrm{test-extra}}$.

Table 1: Quantitative results obtained for the two proposed methods (DE-UNet
and DE-Shape-UNet) compared with the two baselines reported by the challenge
organizers in [9]. We report the mean Dice and HD values, and the standard
deviation in parentheses.

Method |  $\mathcal{D}_{\mathrm{test}}$ (100) |  $\mathcal{D}_{\mathrm{test-extra}}$ (10) | Overall
---|---|---|---
Dice | HD (mm) | Dice | HD (mm) | Dice | HD (mm)
Baseline N1 [9] | 0.809 | 5.440 | - | - | - | -
Baseline N2 [9] | 0.855 | 5.182 | - | - | - | -
DE-UNet | 0.913 (0.038) | 4.067 (1.762) | 0.769 (0.126) | 8.585 (5.128) | 0.900 (0.067) | 4.477 (2.626)
DE-Shape-UNet | 0.845 (0.107) | 6.414 (9.060) | 0.816 (0.078) | 5.952 (1.258) | 0.842 (0.105) | 6.372 (8.648)

Figure 5: Examples of different reconstructions from
$\mathcal{D}_{\mathrm{test}}$ (cases which follow the same pattern than the
training dataset, shown in rows 1 and 2) and $\mathcal{D}_{\mathrm{test-
extra}}$ (out-of-distribution case, shown in row 3). As we can observe, both
methods performed well in the image depicted in row 1. For the case in the 2nd
row, even if the DE-Shape-UNet model managed to reconstruct the implant, the
quality of the reconstruction is lower than that of the DE-UNet. The opposite
happened with the image in row 3 (an out-of-distribution case from
$\mathcal{D}_{\mathrm{test-extra}}$) where the model which incorporated shape
priors managed to reconstruct the implant, while the DE-UNet failed in this
task.

##  5 Conclusions

In this work, we evaluated two different approaches for cranial implant
reconstruction based on deep learning: a direct estimation method and an
alternative strategy which incorporates shape priors. We adapted the virtual
craniectomy procedure proposed in [11] to the defect distribution of the
AutoImplant challenge. We found that the simple DE-UNet method produces more
accurate results for the skull defects which follow the same distribution as
those in the training dataset. However, for out-of-distribution cases where
the DE-UNet model tends to fail, the use of shape priors increases the
robustness of the model, providing additional context to the network. In our
implementation, this gain in robustness for out-of-distribution cases was
achieved to the detriment of the overall accuracy. In future work, we plan to
study alternative ways to introduce shape priors, e.g. considering deformable
registration with anatomical constraints [10] to the atlas space instead of
rigid transformations, or incorporating shape priors in a co-registration and
segmentation process [15].

##  6 Acknowledgments

The authors gratefully acknowledge NVIDIA Corporation with the donation of the
Titan Xp GPU used for this research, and the support of UNL (CAID-
PIC-50220140100084LI) and ANPCyT (PICT 2018-03907).

## References

  * [1] Andrabi, S.M., Sarmast, A.H., Kirmani, A.R., Bhat, A.R.: Cranioplasty: Indications, procedures, and outcome–an institutional experience. Surgical neurology international 8 (2017)
  * [2] Chen, X., Xu, L., Li, X., Egger, J.: Computer-aided implant design for the restoration of cranial defects. Scientific reports 7(1), 1–10 (2017)
  * [3] Chilamkurthy, S., Ghosh, R., Tanamala, S., Biviji, M., Campeau, N.G., Venugopal, V.K., Mahajan, V., Rao, P., Warier, P.: Development and validation of deep learning algorithms for detection of critical findings in head ct scans. arXiv preprint arXiv:1803.05854 (2018)
  * [4] Hieu, L., Bohez, E., Sloten, J.V., Phien, H., Vatcharaporn, E., Binh, P., An, P., Oris, P.: Design for medical rapid prototyping of cranioplasty implants. Rapid Prototyping Journal 9(3), 175–186 (Aug 2003). https://doi.org/10.1108/13552540310477481, https://doi.org/10.1108/13552540310477481
  * [5] Jenkinson, M., Bannister, P., Brady, M., Smith, S.: Improved optimization for the robust and accurate linear registration and motion correction of brain images. Neuroimage 17(2), 825–841 (2002)
  * [6] Larrazabal, A.J., Martínez, C., Glocker, B., Ferrante, E.: Post-dae: Anatomically plausible segmentation via post-processing with denoising autoencoders. IEEE Transactions on Medical Imaging (2020). https://doi.org/10.1109/TMI.2020.3005297
  * [7] Larrazabal, A.J., Martinez, C., Ferrante, E.: Anatomical priors for image segmentation via post-processing with denoising autoencoders. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 585–593. Springer (2019)
  * [8] Lee, M.C.H., Petersen, K., Pawlowski, N., Glocker, B., Schaap, M.: Tetris: template transformer networks for image segmentation with shape priors. IEEE transactions on medical imaging 38(11), 2596–2606 (2019)
  * [9] Li, J., Pepe, A., Gsaxner, C., von Campe, G., Egger, J.: A baseline approach for autoimplant: the miccai 2020 cranial implant design challenge. arXiv preprint arXiv:2006.12449 (2020)
  * [10] Mansilla, L., Milone, D.H., Ferrante, E.: Learning deformable registration of medical images with anatomical constraints. Neural Networks 124, 269–279 (2020)
  * [11] Matzkin, F., Newcombe, V., Stevenson, S., Khetani, A., Newman, T., Digby, R., Stevens, A., Glocker, B., Ferrante, E.: Self-supervised skull reconstruction in brain ct images with decompressive craniectomy. MICCAI 2020 (2020)
  * [12] Monteiro, M., Newcombe, V.F., Mathieu, F., Adatia, K., Kamnitsas, K., Ferrante, E., Das, T., Whitehouse, D., Rueckert, D., Menon, D.K., et al.: Multiclass semantic segmentation and quantification of traumatic brain injury lesions on head ct using deep learning: an algorithm development and multicentre validation study. The Lancet Digital Health (2020)
  * [13] Morais, A., Egger, J., Alves, V.: Automated computer-aided design of cranial implants using a deep volumetric convolutional denoising autoencoder. In: World Conference on Information Systems and Technologies. pp. 151–160. Springer (2019)
  * [14] Patravali, J., Jain, S., Chilamkurthy, S.: 2d-3d fully convolutional neural networks for cardiac mr segmentation. In: Pop, M., Sermesant, M., Jodoin, P.M., Lalande, A., Zhuang, X., Yang, G., Young, A., Bernard, O. (eds.) Statistical Atlases and Computational Models of the Heart. ACDC and MMWHS Challenges. pp. 130–139. Springer International Publishing, Cham (2018)
  * [15] Shakeri, M., Ferrante, E., Tsogkas, S., Lippe, S., Kadoury, S., Kokkinos, I., Paragios, N.: Prior-based coregistration and cosegmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 529–537. Springer (2016)