# Reinforcement Learning based Negotiation-aware Motion Planning of Autonomous
Vehicles

Zhitao Wang*, Yuzheng Zhuang*, Qiang Gu, Dong Chen, Hongbo Zhang, Wulong Liu
* denotes for equally contribution to this paper. Z. Wang, Y. Zhuang, Q. Gu,
D. Chen, H, Zhang and W. Liu are all with Noah’s Ark Lab, Huawei Technologies,
Beijing, China. ({wangzhitao5; zhuangyuzheng; qiang.gu; chendong48;
zhanghongbo888; liuwulong}@huawei.com).

###### Abstract

For autonomous vehicles integrating onto roadways with human traffic
participants, it requires understanding and adapting to the participants’
intention and driving styles by responding in predictable ways without
explicit communication. This paper proposes a reinforcement learning based
negotiation-aware motion planning framework, which adopts RL to adjust the
driving style of the planner by dynamically modifying the prediction horizon
length of the motion planner in real time adaptively w.r.t the event of a
change in environment, typically triggered by traffic participants’ switch of
intents with different driving styles. The framework models the interaction
between the autonomous vehicle and other traffic participants as a Markov
Decision Process. A temporal sequence of occupancy grid maps are taken as
inputs for RL module to embed an implicit intention reasoning. Curriculum
learning is employed to enhance the training efficiency and the robustness of
the algorithm. We applied our method to narrow lane navigation in both
simulation and real world to demonstrate that the proposed method outperforms
the common alternative due to its advantage in alleviating the social dilemma
problem with proper negotiation skills.

##  I Introduction

With promising potential to improve traffic efficiency and safety, the history
of Autonomous Vehicles (AVs) could be traced back to the first half of the
twentieth century [1]. Motion planning is one of the critical capabilities for
AVs [2]. In order to take its responsibility to find a dynamically feasible,
collision free, and comfortable resulting trajectory, motion planning module
need to identify uncertainties in dynamic environments [3]. General
formulation is built as optimization problems, which fulfill the requirements
of safety and comfortability by maximizing a complex reward function. However,
AVs are exposed to complex interactions among traffic participants in real
world driving where prediction and re-planning are required to generate safe
trajectories [4]. It requires the module capable of understanding and adapting
to the participants’ intents and driving styles by responding in predictable
ways without explicit communication.

In order to overcome the challenge, in this paper we propose a negotiation-
aware planning framework that integrates reinforcement learning (RL) as a
negotiation skill selector into motion planning. We model the interaction
between the AV and the other traffic participants as a Markov Decision Process
(MDP) under such framework. The key idea is adopting RL to adjust planning
modality by dynamically modifying hyper-parameters of the motion planner in
real time according to the event of a change in environment, typically
triggered by traffic participants’ switch of intents with different driving
styles. By taking advantages of RL, which tends to have a promising
performance in dealing with dynamic environments, the approach embeds implicit
prediction and reasoning for the intentions of other vehicles with different
driving styles in our framework with a temporal sequence of occupancy grid
maps (OGMs) as inputs [5].

Narrow lane navigation is one of the typical real world scenarios for
autonomous driving, which could be formulated as a sidewalk problem. The
sidewalk problem is a benchmark problem for cooperative stochastic games and
interactive decision making, in which the agents have to deviate from the
optimal path considering their original intersect guidance trajectories [6].
It is usually hard to reason intentions of other agents let alone interacting
to drive efficiently. For the reason of biased estimation, interactive
vehicles may perform aggressively and reach to a bottleneck situation or both
vehicles yield to others and driving tentatively, which lead to the social
dilemma problem [7]. We applied our method to narrow lane navigation to deal
with the sidewalk problem in both simulation and real-world driving to
demonstrate that the proposed method outperforms the common alternative due to
its advantage in alleviating the social dilemma problem with proper
negotiation skills.

Thus the main contributions of this paper are as follows: (1) We introduce an
interactive motion planning approach and establish a mapping from the length
of planner’s prediction horizon to the negotiation skills. (2) Based on the
mapping relations, we propose a negotiation-aware planning framework by
regulating the prediction horizon via RL to adjust the planning modality
adaptively in real time. (3) The framework is implemented in narrow lane
navigation and demonstrated the capability for dealing with the sidewalk
problem. The rest of this paper is organized as follows. Section II reviews
related literatures. Section III proposes our problem statement and
negotiation-aware planning framework. Then we look into simulation and
experiment results in section IV and conclude this work in section V.

##  II Related Work

Many works have investigated the methods for motion planning based on graph
search, sampling and optimization. A*, RRT, and their variations are classical
approaches applied in autonomous driving [8, 9]. Considering the structured
constraints of road lane, decomposing lateral and longitudinal movement
planning in Frenet-frame is a promising method to simplify the optimization
based planning in practice [10]. To model the inherent unpredictability of
other traffic and the resulting uncertainty, a semi-reactive trajectory
generation method based on polynomial approximation is proposed, which can be
tightly integrated into the behavioral layer [11].

Most of the planning schemes are integrated with prediction of other vehicles’
future trajectories to avoid collisions [12]. However, the interaction
scenarios in real driving is too complex to be handled perfectly by the
prediction module [13]. Besides deducing the intentions of other human traffic
participants, driving behavior of the AV also need to be inferred by other
vehicles [4]. Modeling interactions as multi-agent game [14] or cooperative
optimization [15, 16], which will generate behavior more interactive rather
than reactive. However, both modeling the game and dealing with the complexity
of solution are all intriguing problems. Partially observable Markov decision
process (POMDP) is also a normal way of executing the interaction problem,
which assuming that the intentions and re-planning procedures of the other
agents are not directly observable and are encoded in hidden variables [17,
18]. In such situation, finding a suitable symbolic representation for POMDP
is challenging [19].

Learning-based approach is a promising way for autonomous driving [20, 21].
Reinforcement learning (RL) has shown its benefits in many fields, especially
in dynamic environment with complex interactions [22, 23]. By fitting a policy
for sequential decision problems, RL has been applied to self-driving [24].
The framework integrating RL and planning could combine the properties of both
algorithms, in which planning is responsible for safety by restricting
collision actions and RL is used to reason the intentions of other vehicles
and handle the interaction uncertainty. Kesleman _et al._ used RL to train a
heuristic represented by a deep neural networks and combine it with A* [25].
Scholz _et al._ proposed a method in which the RL aims to learn the
optimization criteria for model-based planning [26]. For tactical decision
making, Hoel _et al._ introduced a general framework, which combines planning
and learning in the form of Monte Carlo tree search and RL [27].

In this work, we propose a framework integrating RL and planning which takes a
sequential of OGM as inputs of the learning algorithm. Compared with other
combined schemes which learn cost map, heuristic search space or subgoals [21,
25, 26, 27], our approach provides a way to adaptively adjust planning
modality referring to negotiation skill selection w.r.t the event of a change
in the environment. We utilize the benefits of RL to implicitly reason the
intentions of other traffic participants under different driving styles and
select negotiation skill respectively while taking advantage of the planner’s
property to keep safe from collisions in the dynamic environment. As a result,
our framework has the advantage of reasoning behaviors of other traffic
participants and adopting an appropriate planning skill to avoid social
dilemmas compared to traditional planners which are usually predefined with
one fixed driving fashion limited to prior knowledges of algorithm designers.

##  III Negotiation-aware Adaptive Planning

The negotiation-aware planning framework integrates planning and RL to
alleviate driving social dilemmas caused by interactive driving scenarios. In
this section, we formulate the narrow lane navigation problem as an MDP under
our proposed framework and introduce details of the framework.

Figure 1: The narrow lane navigation scenario. Both of the vehicles wish to
pass the narrow corridor as soon as possible and keep safe at the same time.

###  III-A Narrow lane navigation

We focus on investigating the narrow lane navigation, a typical real world
scenario that could be formulated as a benchmark sidewalk problem. As shown in
Fig 1, there exists a narrow corridor due to the on street parking, which is
common in residential communities. The goal for both the AV and the social
vehicle is to pass through the corridor and reach their targets on the other
side considering both safety and efficiency.

###  III-B Negotiation-aware Planning Framework

Figure 2: An illustration of the relationship between the planning behavior
and the prediction horizon. When considering shorter future trajectories of
social vehicles, the ego vehicle tends to be more aggressive and vice versa.

Our proposed framework consists of two parts including a RL module that
outputs planner’s prediction horizon and an interactive motion planning module
responsible for generating feasible trajectories. As shown in Fig 2, at every
time step, a temporal sequential OGM generated by a perception system that
represents the current state of the environment is given to the network, then
taking into the selection of prediction horizon from RL module as well as
guidance route and objects information from perception system, the planner
will give a future trajectory.

The RL module is composed of a CNN encoder, a value network and a policy
network. The CNN encoder is used to distill the information of history
environment dynamics from the sequential OGM images. Both of the value and
policy networks are fully connected networks trained to obtain the optimal
policy for regulating planner’s prediction horizon by achieving the desired
goal. We applied ACKTR algorithm, a policy gradient algorithm to optimize our
training procedure [28]. As for the various of interactive situations, we
leveraged curriculum learning to handle it [29].

###  III-C Interactive Motion Planning

Motion planning could be formulated as an optimization problem as follows:

$\displaystyle\underset{\bm{x}}{\text{argmin}}$
$\displaystyle\sum_{t\in[0,H_{\rm plan}]}c(x_{t})$ (1) subject to
$\displaystyle f_{t}(x_{t},\dot{x}_{t},...)\leq 0\quad t\in[0,H_{\rm plan}],$

where $t$ is time step, $H_{\rm plan}$ is the planning horizon,
$\bm{x}=\\{x_{1},x_{2},...,x_{H_{\rm plan}}\\}$ represents the planning
trajectory, $c$ and $f_{t}$ are cost function and constraints considering
safety, vehicle dynamics and comfort respectively. Since finding an optimal
solution of the problem (2) is PSPACE-hard with the nonholonomic constraints,
we approximate the solution by sampling a set of physically valid trajectories
and pick the one with minimum cost.

We adopt a semi-reactive motion planner by means of optimal-control strategies
[11]. The planner takes inputs include HD map and LiDAR sweeps for navigation
guidance reference and objects array collision checking. The trajectory set is
generated by polynomial approximation that quintic polynomials are the jerk-
optimal connection between two secondary order states
$[x_{0},\dot{x}_{0},\ddot{x}_{0}]$ and $[x_{H_{\rm plan}},\dot{x}_{H_{\rm
plan}},\ddot{x}_{H_{\rm plan}}]$. Lateral and longitudinal movements are
generated by sampling along Frenet-Frame independently and checked against
acceleration values of dynamic limitation before combination. The trajectory
with the lowest conjoint cost function and free of collision is picked out for
the downstream controller to track.

In order to consider the interaction with other social vehicles, a prediction
module is embedded into our motion planner. We use
$\bm{\hat{x}}=\\{\hat{x}_{1},\hat{x}_{2},...,\hat{x}_{H_{\rm pred}}\\}$ to
denote the $H_{\rm pred}$ steps rollout for a prediction trajectory of each
vehicle. The prediction trajectory is projected onto the SLT frame for the
collision checking. As default, the length of prediction horizon $H_{\rm
pred}$ should be the same as planning horizon $H_{\rm plan}$. The prediction
trajectories of other vehicles are hardly to be precise, longer prediction
horizon normally lead to a relatively conservative choice of plans and shorter
ones could otherwise lead to a more aggressive driving behavior as shown in
Fig 3.

AVs requires conservative behaviors for safe operations because of lacking
capability of precise predictions for dynamics. However, social vehicles with
various driving styles could react to our AV differently, thus being
conservative constantly could have a passive influence on driving efficiency
or even lead to traffic bottlenecks, especially when the other vehicles also
drive in a modest fashion. On the other hand, being blindly aggressive without
proper negotiations could still cause traffic dilemmas.

###  III-D RL formulation

#### • Markov Decision Process

MDP is an mathematical framework for modeling decision making process for an
agent to achieve a goal by learning from a continual interaction with the
environment. The agent selects actions and the environment responds to these
actions with rewards and presents new situations to the agent. An MDP is a
tuple $\langle S,A,P,R,\gamma\rangle$, where $S$ is a finite set of states
represent the environment’s situations, $A$ is a finite set of possible
actions for the agent, $P$ denotes the state transition probability function
$P\\!=\\!P(S_{t+1}\\!=\\!s^{\prime}|S_{t}\\!=\\!s,A_{t}\\!=\\!a)$, $R$ denotes
for the reward function $R\\!=\\!E[R_{t+1}|S_{t}\\!=\\!s,A_{t}\\!=\\!a]$, and
$\gamma$ is a discount factor that $\gamma\in[0,1]$. Policy $\pi$ describes
the agent behavior that is a distribution over actions given states. The goal
is to find the optimal policy $\pi^{*}$ in an MDP by maximizing the cumulative
discounted rewards with finite horizon $H$:

$\displaystyle\bm{\pi^{*}}=\underset{\bm{\pi}}{\text{argmax}}$
$\displaystyle\,E[\sum_{t=0}^{H}\gamma^{t}R(s_{t},\pi(s_{t}))]$ (2)

where the reward function $R$ receives the current state $s_{t}$ and the
action determined by current policy $\pi$ given $s_{t}$.

#### • State and action representation

An OGM is a grid map of size $L_{\rm W}\times L_{\rm H}$ pixels rendered into
the top-down coordinate system, which represent the occupancy state of regions
around the ego-vehicle. It is easy to generate in the simulation environment
or could be constructed by the meta data form LiDAR in real world driving.
Compared with the raw sensor data representation, OGM shows its advantages in
flexibility and generalization[30].

As the state representation of RL, a sequence of temporal OGMs are sampled
according to a fixed time step $\Delta t$ from past $T_{\rm ogm}$ seconds,
thus the inputs dimension is $[T_{\rm ogm}\times L_{\rm W}\times L_{\rm H}]$.
Coordination transformation based on current vehicle-link frame is implemented
in order to make the temporal sequential representation could indicate the
potential dynamics from current view.

According to the mapping relations from planner’s prediction horizon to
planning modality, we discretized the prediction horizon into a $H_{\rm pred}$
dimension vector which refers to different driving styles from aggressive to
conservative.

Figure 3: An illustration of the relationship between planning behavior and
prediction horizon. When considering shorter future trajectories of
interactive vehicles, the ego vehicle tends to be more aggressive and vice
versa.

#### • Reward setting

The task is designed to terminate and reset either all vehicles reach their
target positions or collision. Since the step by step interaction process
between the AV and the social vehicle is hard to be intuitively described, we
consider the safety, efficiency and task completion for reward function design
in a sparse manner. A negative reward is given to penalize collision and a
positive reward for the vehicles both finish their driving tasks. Also, a
negative per-step reward is accumulatively given with fixed episodic horizon
$H$ to encourage efficiency. A detailed reward setting is shown in (3).

$R=\begin{cases}-0.25*t,&\rm\textit{per-step penalty,}\\\
-5,&\rm\textit{collision penalty,}\\\ 10,&\rm\textit{reached
targets}\end{cases}$ (3)

#### • Algorithm

The RL algorithm we adopt in this work named actor critic using Kronecker-
factored trust region (ACKTR) [28]. ACKTR is a policy gradient method with the
trust region optimization method use second order optimizer. By applying the
Kronecker-factored approximation to optimize both actor and critic, ACKTR will
speed up the optimization and stabilize the training procedure.

With an actor critic architecture, we need to optimize a stochastic policy
$\pi_{\theta}:S\times A\rightarrow\mathbb{R}_{+}$ that parameterized by
$\theta$. Given the state $s_{t}$ from $S$, which are assumed to be the OGM
based observations of the environment, with actions $a_{t}$ from the action
space $A$ chosen by the policy distribution $\pi_{\theta}(\cdot|s_{t})$. The
AV interact with the environment with prediction horizon assigned motion
planner and perceived next state $s_{t+1}\sim P(s_{t+1}|s_{t},a_{t})$ and the
immediate reward $R(s_{t},a_{t})\in\mathbb{R}$ according to the reward
function defined in (3). Altogether, it can be formalized as an
$\gamma$-discounted MDP with the expected discounted cumulative return is:

$\displaystyle
J(\Theta)=E_{\pi}[\sum_{t=0}^{H}\gamma^{t}R(s_{t},\pi(s_{t}))].$ (4)

Update the $\theta$ w.r.t the policy gradient defined as:

$\displaystyle\nabla_{\Theta}J(\Theta)=E_{\pi}[\sum_{t=0}^{H}\Psi^{t}\nabla_{\Theta}log\pi_{\theta}(a_{t}|s_{t})],$
(5)

where $\Psi^{t}$ denotes for the advantage function $A^{\pi}(s_{t},a_{t})$.
The advantage is calculated as follows:

$\displaystyle
A^{\pi}(s_{t},a_{t})=\sum_{i=0}^{k-1}(\gamma^{i}R(s_{t+i},a_{t+i})+\gamma^{k}V^{\pi}_{\phi}(s_{t+k}))-V^{\pi}_{\phi}(s_{t}),$
(6)

where $V^{\pi}_{\phi}(s_{t})$ denotes for the value network that estimate the
expected reward with policy $\pi$. We train the network by performing temporal
difference updates to minimize the squared difference between the $k$-step
bootstrapped returns $\hat{R}_{t}$ and $V^{\pi}_{\phi}(s_{t})$.

#### • Curriculum Learning

In order to enhance the training efficiency and robustness of our algorithm,
we employ curriculum learning that could help intelligent agents to learn
better with organized examples which gradually illustrate more and more
complex concepts.

In our training procedure, the initial position of the social vehicle is
selected from a certain range with a driving style assigned. We deploy
interactive motion planning method to control social vehicles and define the
driving styles from aggressive to conservative w.r.t the mapping from
prediction horizons to planning behaviors. Different distributions of both the
initial position and the driving style significantly affect the difficulty of
our task and the curriculum is designed by decomposing the learning procedure
into multiple phases accordingly.

For instance, in first phase we set initial positions far away from the narrow
corridor for conservative social vehicles and nearby positions for aggressive
ones, than in order to increase the difficulty, the distributions of initial
positions will be set closer to each other. Finally, our AV learns to
reasoning about the social vehicle’s intent according to its driving style and
road geometry, and thus react with proper choice of skill w.r.t task situation
and goal. The corresponding training procedure is shown in Algorithm 1.

Algorithm 1 Curriculum learning procedure.

1: distribution $D_{0}$,…, $D_{s}$, iteration numbers $K_{0}$,…, $K_{s}$, task
batch size $M$, number of trajectories $N$

2:policy and value networks $\pi_{\theta}$, $V_{\phi}$

3:Randomly initialize $\theta$, $\phi$

4:while  _not done_ do

5: for $(K,D)\in\\{(K_{0},D_{0}),...,(K_{s},D_{s})\\}$ do

6: for $k\in\\{(1,...,K)\\}$ do

7: Sample M number of tasks $T\in D$

8: for all $T$ do

9: Sample $N$ trajectories $\tau^{1:N}$ with $\pi_{\theta}$

10: end for

11: update network $\pi_{\theta}$ and $V_{\phi}$ using ACKTR

12: end for

13: end for

14:end while

##  IV Experiment

(a) driving time

(b) suspend number

(c) reward

Figure 4: Performance comparisons of our framework in push experiment setting.
As 4(a) 4(b) show, all the algorithm could complete driving task in limit
time, while the adaptive behavior planner has better driving efficiency and
take less suspend time; Thus the proposed framework tends to have a higher
reward in 4(c).

(a) driving time

(b) suspend number

(c) reward

Figure 5: Performance comparisons of our framework in yield experiment
setting. Because of a closer social vehicle distance and the social vehicle’s
intent of pushing, ego vehicle will yield to keep safe. 5(a) shows that except
the proposed algorithm, other schemes could not reach the goal position in
limit time. To yield to social vehicles, the suspend time tends to be more
than that in push experiment setting as shown in 5(b).

###  IV-A Experimental settings

The proposed algorithm was trained and tested in the narrow lane navigation
scenario. Relative distances from the narrow corridor and driving styles of
the two vehicles are the key features to impact the driving behavior. The
initial position of the social vehicle is randomly sampled in a range. As for
the training and testing in the simulator, a polynomial planner introduced in
interactive motion planning section is arranged for the social vehicle which
is identical with the one used in our framework. We also implemented and
evaluated our work on a real autonomous vehicle in a campus scenario.

###  IV-B Implementation details

The initial positions of the social vehicle are sampled from $11.99$m to
$14.02$m for conservative behavior and $16.33$m to $18.80$m for aggressive
behavior. As for the ego autonomous vehicle, the position is fixed with
$13.35$m. The frequency of the implementation is 5Hz and we sampled sequential
OGM of $84\times 84$ size every second in the last 4 seconds, which means that
4 channels of OGM are given to our CNN encoder at a time. The CNN encoder
which extracts features from OGM sequence has 3 convolution layers with the
filter size of $32\times 8\times 8$, $64\times 4\times 4$ and $64\times
3\times 3$, followed with 1 fully connected layer with 512 output neurons.
Both value and policy networks are fully connected networks with 2 hidden
layers with 128 units. The Policy network has 4 discrete outputs units to
represent time horizon $\\{0,2,4,6\\}$ while the value network has only one
output. We use Adam as the optimizer and a learning rate of $10^{-4}$ and
$10^{-3}$ for policy network and value networks respectively. For the setting
of RL, we use mini-batch size of $m=512$, trajectory length of $H=128$, number
of trajectories to collect for each iteration of $N=200$, and $\gamma=0.99$ as
the discount factor.

###  IV-C Results and discussion

We tested the negotiation-aware framework in both push and yield experiment
settings in narrow lane navigation and demonstrate the effectiveness of our
proposed framework compared to the polynomial planner with different fixed
prediction horizons in this section.

Fig 4 and Fig 5 illustrate the performance comparisons in the two experiment
settings respectively. In the push setting where the social vehicle is further
from narrow corridor than ego vehicle, most of the algorithms could dominate
autonomous vehicle reach its target within time limit while our proposed
algorithm takes less time and suspend times to reach the target. In the yield
setting scenario, the vehicles with polynomial planner tends to get into a
social dilemma with the almost identical relative distances. On the contrary,
our algorithm could decide to yield in advance before stuck in a dilemma by
reasoning the other vehicle’s behavior.

Generally, compared with the polynomial planner with fixed prediction horizon,
our adaptive framework could complete the driving task in shorter time. One
reason could explain the outstanding performance in driving efficiency is the
less hesitate in driving interaction as shown in Fig 4(b) and Fig 5(b).
Another reason is that our framework learns the approximating optimum yield
position from the sequential OGM images which describes the environment
dynamics. This makes the autonomous vehicle has less chance to get into
dilemma situations with social vehicles. As a result, our framework has a
higher reward than the polynomial planner with fixed prediction horizon. The
statistical results also show that the performance of our proposed framework
has better stability because of the robustness w.r.t to environment dynamics
consists of social participants and road geometry.

(a) Behavior index in the push experiment setting.

(b) Behavior index in the yield experiment setting

Figure 6: The change of driving behavior index in narrow lane navigation
scenario. In the push experiment setting, the index value is always negative.
However in the yield experiment setting, the index value shows that the
autonomous vehicle will yield to the social vehicle at first.

The behavior of prosocial or egoistic could be measured by Social Value
Orientation (SVO), which is an index represents the weight assignment for the
interests of ego vehicle and others in the formation of angles [7]. Inspired
by the work in [7], we also use an angle like index as the measurement for the
social behavior in sidewalk scenario,

$\displaystyle\psi=\text{atan}{\frac{s_{\rm soc}-s_{\rm ego}}{d_{\rm
ini,ego}/d_{\rm ini,soc}}},$ (7)

where $d$ is the initial distance from the corridor, $s$ is the travel
distance of ego and social vehicles respectively. Both distances are counted
at the time step since the social vehicle enter the ego vehicle’s perception.
Such index works the same as SVO, that the behavior tends to be prosocial for
positive values and competitive for a negative ones. The numerator of (7)
describes the fact that which vehicle dominants the driving interaction and
the denominator considers the impact of the initial position.

Fig 6 shows the change of driving behavior index $\phi$ in push and yield
experiment setting respectively. The behavior of autonomous vehicle in the
interactive driving and the driving behavior index could verify each other. In
the push scenario the adaptive framework will regulate the prediction horizon
to performance aggressively to pass the narrow corridor as soon as possible
when the social vehicle is further from the corridor. Fig 6(a) shows negative
index values during driving which reflects the aggressive driving behavior.
And in the yield experiment setting. To keep safe and retain driving
efficiency at the same time, ego vehicle will pause at a reasonable position
to yield to the social vehicle. After the interactive vehicle passes the
corridor, our ego vehicle will perform aggressively to reach to the goal. Fig
6(b) illustrates the driving behavior tunning procedure in this scenario, that
the index value is positive and then negative.

We implemented the framework onto a real autonomous vehicle and tested in a
real world narrow lane navigation scenario as shown in Fig 7. Our proposed
approach demonstrates a good generalization in real world driving. A
supplementary video is provided to show the performance of our work111See
videos at https://youtu.be/j9FK_RNbbI0.

Figure 7: Testing of negotiation-aware framework in real-world driving.

##  V Conclusions

In this paper, we proposed a RL-based negotiation-aware adaptive motion
planning framework to deal with the sidewalk problem by alleviating social
dilemmas that exist in autonomous driving. We employ ACKTR and curriculum
learning to train the policy network with sequential OGM images as inputs.
According to the prediction horizon policy generated by the policy network, we
use polynomial planner with prediction module to generating the optimal
trajectories. The effectiveness of our framework is evaluated both in
simulation and real-driving scenarios. We find that our framework could
regulating driving social behavior and outperform the common alternatives in
driving efficiency with safety properly considered.

## References

  * [1] F. Kröger, “Automated driving in its social, historical and cultural contexts,” in Autonomous Driving, pp. 41–68, Springer, 2016.
  * [2] C. Katrakazas, M. Quddus, W.-H. Chen, and L. Deka, “Real-time motion planning methods for autonomous on-road driving: State-of-the-art and future research directions,” Transportation Research Part C: Emerging Technologies, vol. 60, pp. 416–442, 2015.
  * [3] J.-C. Latombe, Robot motion planning, vol. 124.  Springer Science & Business Media, 2012.
  * [4] W. Schwarting, J. Alonso-Mora, and D. Rus, “Planning and decision-making for autonomous vehicles,” Annual Review of Control, Robotics, and Autonomous Systems, 2018.
  * [5] N. Mohajerin and M. Rohani, “Multi-step prediction of occupancy grid maps with recurrent neural networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10600–10608, 2019.
  * [6] D. Isele and K. Fujimura, “The sidewalk problem,”
  * [7] W. Schwarting, A. Pierson, J. Alonso-Mora, S. Karaman, and D. Rus, “Social behavior for autonomous vehicles,” Proceedings of the National Academy of Sciences, vol. 116, no. 50, pp. 24972–24978, 2019.
  * [8] A. Bacha, C. Bauman, R. Faruque, M. Fleming, C. Terwelp, C. Reinholtz, D. Hong, A. Wicks, T. Alberi, D. Anderson, et al., “Odin: Team victortango’s entry in the darpa urban challenge,” Journal of field Robotics, vol. 25, no. 8, pp. 467–492, 2008.
  * [9] J. Leonard, J. How, S. Teller, M. Berger, S. Campbell, G. Fiore, L. Fletcher, E. Frazzoli, A. Huang, S. Karaman, et al., “A perception-driven autonomous urban vehicle,” Journal of Field Robotics, vol. 25, no. 10, pp. 727–774, 2008.
  * [10] H. Fan, F. Zhu, C. Liu, L. Zhang, L. Zhuang, D. Li, W. Zhu, J. Hu, H. Li, and Q. Kong, “Baidu apollo em motion planner,” arXiv preprint arXiv:1807.08048, 2018.
  * [11] M. Werling, J. Ziegler, S. Kammel, and S. Thrun, “Optimal trajectory generation for dynamic street scenarios in a frenet frame,” in Robotics and Automation (ICRA), 2010 IEEE International Conference on, 2010.
  * [12] Q. Tran and J. Firl, “Modelling of traffic situations at urban intersections with probabilistic non-parametric regression,” in 2013 IEEE Intelligent Vehicles Symposium (IV), pp. 334–339, IEEE, 2013.
  * [13] C. Dong, J. M. Dolan, and B. Litkouhi, “Intention estimation for ramp merging control in autonomous driving,” in 2017 IEEE Intelligent Vehicles Symposium (IV), pp. 1584–1589, IEEE, 2017.
  * [14] M. Bahram, A. Lawitzky, J. Friedrichs, M. Aeberhard, and D. Wollherr, “A game-theoretic approach to replanning-aware interactive scene prediction and planning,” IEEE Transactions on Vehicular Technology, vol. 65, no. 6, pp. 3981–3992, 2015.
  * [15] Z. Wang, Y. Zheng, S. E. Li, K. You, and K. Li, “Parallel optimal control for cooperative automation of large-scale connected vehicles via admm,” in 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pp. 1633–1639, IEEE, 2018.
  * [16] D. Lenz, T. Kessler, and A. Knoll, “Tactical cooperative planning for autonomous highway driving using monte-carlo tree search,” in 2016 IEEE Intelligent Vehicles Symposium (IV), pp. 447–453, IEEE, 2016.
  * [17] Y. Luo, P. Cai, A. Bera, D. Hsu, W. S. Lee, and D. Manocha, “Porca: Modeling and planning for autonomous driving among many pedestrians,” IEEE Robotics and Automation Letters, vol. 3, no. 4, pp. 3418–3425, 2018.
  * [18] C. Hubmann, M. Becker, D. Althoff, D. Lenz, and C. Stiller, “Decision making for autonomous driving considering interaction and uncertain prediction of surrounding vehicles,” in 2017 IEEE Intelligent Vehicles Symposium (IV), pp. 1671–1678, IEEE, 2017.
  * [19] A. Somani, N. Ye, D. Hsu, and W. S. Lee, “Despot: Online pomdp planning with regularization,” in Advances in neural information processing systems, pp. 1772–1780, 2013.
  * [20] W. Zeng, W. Luo, S. Suo, A. Sadat, B. Yang, S. Casas, and R. Urtasun, “End-to-end interpretable neural motion planner,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8660–8669, 2019.
  * [21] S. Rosbach, V. James, S. Großjohann, S. Homoceanu, and S. Roth, “Driving with style: Inverse reinforcement learning in general-purpose planning for automated driving,” arXiv preprint arXiv:1905.00229, 2019.
  * [22] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al., “Mastering the game of go with deep neural networks and tree search,” nature, vol. 529, no. 7587, p. 484, 2016.
  * [23] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015.
  * [24] A. E. Sallab, M. Abdou, E. Perot, and S. Yogamani, “Deep reinforcement learning framework for autonomous driving,” Electronic Imaging, vol. 2017, no. 19, pp. 70–76, 2017.
  * [25] A. Keselman, S. Ten, A. Ghazali, and M. Jubeh, “Reinforcement learning with a* and a deep heuristic,” arXiv preprint arXiv:1811.07745, 2018.
  * [26] J. Scholz and M. Stilman, “Combining motion planning and optimization for flexible robot manipulation,” in 2010 10th IEEE-RAS International Conference on Humanoid Robots, pp. 80–85, IEEE, 2010.
  * [27] C.-J. Hoel, K. Driggs-Campbell, K. Wolff, L. Laine, and M. Kochenderfer, “Combining planning and deep reinforcement learning in tactical decision making for autonomous driving,” IEEE Transactions on Intelligent Vehicles, 2019.
  * [28] Y. Wu, E. Mansimov, R. B. Grosse, S. Liao, and J. Ba, “Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation,” in Advances in neural information processing systems, pp. 5279–5288, 2017.
  * [29] Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” in Proceedings of the 26th annual international conference on machine learning, pp. 41–48, 2009.
  * [30] M. Bansal, A. Krizhevsky, and A. Ogale, “Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst,” arXiv preprint arXiv:1812.03079, 2018.


2022

[1]M. Jorge Cardoso Equal contribution Equal contribution Equal contribution
Equal contribution

1]School of Biomedical Engineering & Imaging Sciences, King’s College London,
London, U.K 2]NVIDIA Corporation, Santa Clara and Bethesda, USA 3]School of
Electrical and Computer Engineering, Rice University, Houston, USA
4]Department of Engineering Science, University of Oxford, Oxford, UK
5]Department of Medical Physics and Biomedical Engineering, University College
London, London, UK 6]Department of BioAI, InstaDeep Ltd, London, UK
7]Department of Electrical & Computer Engineering, University of Iowa, Iowa
City, USA 8]Mayo Clinic, Jacksonville, USA 9]Mars Incorporated, , USA 10]Div.
Intelligent Medical Systems, German Cancer Research Center, Heidelberg,
Germany 11]Interactive Machine Learning Group, German Cancer Research Center,
Heidelberg, Germany 12]Division of Medical Image Computing, German Cancer
Research Center, Heidelberg, Germany 13]Pattern Analysis and Learning Group,
Department of Radiation Oncology, Heidelberg University Hospital, Heidelberg
14]School of Biomedical Engineering & Suzhou Institute for Advanced Research,
University of Science and Technology of China, Suzhou, China 15]Department of
Medical Physics, Guy’s & St Thomas’ NHS Foundation Trust, London, UK
16]National Cancer Institute, Bethesda, USA 17]Kitware, Inc., Clifton Park,
USA 18]Department of Ophthalmology, University of Colorado, Aurora, USA
19]Frederick National Laboratory for Cancer Research, Frederick, USA
20]Department of Pathology, Northwestern University, Chicago, USA

# MONAI: An open-source framework for deep learning in healthcare

Wenqi Li  Richard Brown  Nic Ma  Eric Kerfoot  Yiheng Wang  Benjamin Murrey
Andriy Myronenko  Can Zhao  Dong Yang  Vishwesh Nath  Yufan He  Ziyue Xu  Ali
Hatamizadeh  Andriy Myronenko  Wentao Zhu  Yun Liu  Mingxin Zheng  Yucheng
Tang  Isaac Yang  Michael Zephyr  Behrooz Hashemian  Sachidanand Alle
Mohammad Zalbagi Darestani  Charlie Budd  Marc Modat  Tom Vercauteren  Guotai
Wang  Yiwen Li  Yipeng Hu  Yunguan Fu  Benjamin Gorman  Hans Johnson  Brad
Genereaux  Barbaros S. Erdal  Vikash Gupta  Andres Diaz-Pinto  Andre Dourson
Lena Maier-Hein  Paul F. Jaeger  Michael Baumgartner  Jayashree Kalpathy-
Cramer  Mona Flores  Justin Kirby  Lee A.D. Cooper  Holger R. Roth  Daguang Xu
David Bericat  Ralf Floca  S. Kevin Zhou  Haris Shuaib  Keyvan Farahani  Klaus
H. Maier-Hein  Stephen Aylward  Prerna Dogra  Sebastien Ourselin  Andrew Feng
[  [  [  [  [  [  [  [  [  [  [  [  [  [  [  [  [  [  [  [

###### Abstract

Artificial Intelligence (AI) is having a tremendous impact across most areas
of science. Applications of AI in healthcare have the potential to improve our
ability to detect, diagnose, prognose, and intervene on human disease. For AI
models to be used clinically, they need to be made safe, reproducible and
robust, and the underlying software framework must be aware of the
particularities (e.g. geometry, physiology, physics) of medical data being
processed. This work introduces MONAI, a freely available, community-
supported, and consortium-led PyTorch-based framework for deep learning in
healthcare. MONAI extends PyTorch to support medical data, with a particular
focus on imaging, and provide purpose-specific AI model architectures,
transformations and utilities that streamline the development and deployment
of medical AI models. MONAI follows best practices for software-development,
providing an easy-to-use, robust, well-documented, and well-tested software
framework. MONAI preserves the simple, additive, and compositional approach of
its underlying PyTorch libraries. MONAI is being used by and receiving
contributions from research, clinical and industrial teams from around the
world, who are pursuing applications spanning nearly every aspect of
healthcare.

###### keywords:

AI, Healthcare, Open-source

##  1 Introduction

Evidence-based medicine is the de facto standard approach guiding clinical
management and decision-making, drawing recommendations directly from robust
and bias-controlled evidence, and utilizing statistical measures to quantify
the clinical benefit of these recommendations. As patterns become more complex
and data more numerous, advanced data analytics and pattern recognition
methods are needed; artificial intelligence, and more specifically, machine
learning, delivers on this need.

Artificial intelligence (AI), when applied to healthcare data, carries the
promise of improving and automating the detection and diagnosis of diseases,
enabling the prognosis of disease outcomes, and supporting the delivery of
clinical interventions. AI can also be used to speed image reconstruction, to
automate clinical data curation and preparation Kraljevic:2021aa , to improve
auditing and clinical safety Brzezicki:2020aa , and to optimise patient
scheduling and other hospital operations Nelson:2019aa . In order for this
vision to be achieved, several problems need to be mitigated Panch:2019aa ;
among them, the evidence supporting the safety and efficacy of these AI
systems needs to outweigh the potential risks while representing a good value
proposition. While several AI risks are often problem-specific and require
substantial evidence to validate their accuracy, the safety and quality of the
underlying software frameworks and tools used to develop these AI models are
equally important and pervasive.

There are a plethora of competing high-quality, general-purpose machine
learning libraries and software frameworks that are used by researchers and
developers to create, train, and deploy AI models. Examples of high-quality
frameworks for AI development include actively supported frameworks such as
Tensorflow tensorflowpaper , Keras, PyTorch pytorchpaper , JAX and Apache
MXNet mxnetpaper , as well as many deprecated frameworks such as Theano,
Torch, Caffe, and CNTK. The two most popular frameworks (Tensorflow and
PyTorch) have significant feature parity, demonstrating the maturity of the
field. However, as general-purpose frameworks, they often do not support
field-specific or data-specific functionality. Healthcare data, such as
medical images and digital pathology reports, have a physiological basis that
must be understood and maintained as the data is processed, e.g., tabular
blood data transformations must be aware of the expected numerical ranges, or
clinical x-ray computed tomography images are recorded in DICOM object files
with rich metadata descriptions of the acquisition protocol and with voxel
values recorded in Hounsfield units at specific within-slice and inter-slice
distances. Correspondingly, the associated AI models for processing such data
(e.g. 3D multimodal medical imaging data) require purpose-specific
architectures, augmentation, and training mechanisms. If healthcare AI models
are to be built using a general-purpose framework, significant functionality
would need to be developed and tested, thus increasing the length of and the
risks associated with the full R&D life-cycle.

With the increasing popularity of AI in healthcare, several open-source
healthcare-specific frameworks have also been developed by academic groups
(NiftyNet Gibson:2018aa , DLTK DLTKpaper , DeepNeuro deepneuropaper ) and
industry teams (NVIDIA Clara, Microsoft Project InnerEye). However, this
disjointed development of several platforms resulted in a fragmented software
field, diluting development efforts, reducing code quality, and slowing the
pace of research.

Project MONAI (https://monai.io) is a joint initiative aiming at defining,
standardizing, developing, and exchanging best practices for AI in healthcare;
it is unifying the fragmented healthcare AI software field. Initially led by
NVIDIA and King’s College London, Project MONAI has since evolved into a
growing consortium currently with 16 different universities and industrial
partners. Project MONAI consists of several components, including MONAI Label
for AI-assisted image annotation, MONAI Deploy for integrating AI models into
clinical workflows, and MONAI Core for deep learning model research and
development. This paper is focused on MONAI Core, but much of the design and
community-oriented philosophies of MONAI Core are shared with the other
Project MONAI components.

MONAI Core is a freely available, community-supported, consortium-led,
component of Project MONAI for deep learning in healthcare. It provides
domain-optimized foundational capabilities for developing healthcare AI model
training workflows, with a focus on imaging, video, and other forms of
structured data (e.g. tabular data, HL7 FIHR, EEG signals). MONAI Core has
strong community backing, with several thousand active users and more than 120
contributors. It was created with several principles in mind: open-source
design, standardized, user-friendly, reproducible, high quality, and easy to
integrate. This paper describes the MONAI Core software framework and its
open-source strategy, design philosophy, implementation choices, abstractions,
and capabilities. This paper also provides examples of how MONAI Core, and
Project MONAI in general, is being applied to solve a variety of healthcare
challenges.

###  1.1 Design considerations for deep learning in healthcare

Compared with generic deep learning software packages, healthcare has its own
characteristics and technical challenges. These aspects are carefully
considered throughout the design of the project. First, the data are often in
the form of high-dimensional arrays, with the processing of the data often
being coupled with the underlying physical interpretations of the data
acquisition processes and the anatomical analysis. Thus, the design should
consider the accompanying metadata and the relevant annotations. Second, due
to data variability and highly flexible requirements, it is necessary to build
simple and robust low-level data processing components, with each component
focusing on a key problem implemented to be efficient and tested to be robust;
such an approach allows end-users to build complex and flexible compositional
pipelines using these components. Lastly, while exposing the low-level
component APIs, the design decisions should also be aware of the widely-used,
high-level workflows. The introduction of standardized pipeline elements
reduces the effort needed to reproduce algorithmic research baselines,
potentially accelerating research and development by allowing users to focus
on their innovative contributions while reusing most of the existing
pipelines.

##  2 Methods

###  2.1 Design choices and structure of the framework

As alluded to in the previous section, MONAI Core (also referred to simply as
MONAI in the remainder of this paper) has been through a very deliberate
design process with a clear design philosophy as its foundation. Project MONAI
has a set of principles to which its design adheres and which guide the
addition of functionality across the core team and the broader base of
contributors.

The three key design principles are: it looks and feels like PyTorch; it is
opt-in and incremental over PyTorch, and it fully integrates with the PyTorch
ecosystem. These principles are detailed in the following sections, with
concrete examples throughout subsequent sections.

####  2.1.1 MONAI looks and feels like PyTorch

PyTorch is a highly-successful deep learning framework, particularly amongst
the research community he2019mlframeworks . This research focus was a key
factor in why PyTorch was chosen as the MONAI underlying framework, with ease
of use when building novel networks and a rich ecosystem built up around the
core PyTorch functionality, further motivating this decision.

MONAI has adopted the principle that features should be designed and
implemented following the core guidelines and principles of PyTorch
(docs.monai.io/en/stable/contrib.html), or in the case of the quality-of-life
frameworks, such as Ignite (pytorch.org/ignite), as if it was done by the team
developing that library. This approach was chosen to ensure developers using
MONAI have a minimal learning curve and cognitive burden beyond understanding
PyTorch.

Exceptions to this rule are additive in nature. As an example of such an
exception, a number of MONAI transforms which rely on a random number
generation have a behaviour typical of transforms provided by libraries such
as Torchvision (pytorch.org/vision), the de-facto imaging preprocessing
library for PyTorch, but expose extra options that can provide additional
functionality to enhance determinism.

####  2.1.2 MONAI is opt-in and incremental over PyTorch

A core tenet of MONAI is to be opt-in and incremental, and emphases low
coupling between components. Rather than rebuilding a network architecture or
model in MONAI, individual components can be selected to enhance vanilla
PyTorch, Ignite-based projects, or other deep learning pipelines. MONAI has a
smooth barrier-to-entry, allowing the user to gradually and seamlessly adopt
individual transforms, layers, loss functions, and other elements of a deep
learning model into a vanilla PyTorch model.

In particular, MONAI does not enforce overly stringent design philosophies or
offer cumbersome APIs, nor does its implementation use architectural features
that are unfamiliar to or introduce unexpected constraints on Pytorch users.
MONAI definitions are almost entirely standard Python, inheriting from PyTorch
or Ignite types and exhibiting the same loose coupling expected of these
libraries. Only PyTorch, and by extension Numpy, are mandatory dependencies of
MONAI; if libraries, such as Ignite, are absent from the environment, then
dependent definitions will use alternative or dummy implementations with
appropriate error management.

####  2.1.3 MONAI collaborates with the PyTorch ecosystem

There is a wealth of medically-focused libraries built around PyTorch, which
are seldom compatible with one another. For example, while there is a rich
ecosystem of medical transform and augmentation libraries, combining these
libraries (e.g., through PyTorch Compose) is not directly supported. In
collaboration with the developers of these libraries, MONAI now allows methods
from those libraries to be mixed within a single compose command, promoting
their usage and integrating their benefits and user communities.

MONAI also integrates and seamlessly extends several core PyTorch ecosystem
tools. As an example, several PyTorch-based quality-of-life frameworks, such
as Ignite, exist to provide higher-order engines and abstractions for the
training and validation of neural networks. MONAI has adopted Ignite and
extended it following an Ignite-native approach, all while ensuring two-way
compatibility.

###  2.2 Open-Source Strategy

MONAI is a software framework for both research and commercial product
development, with the ultimate objective of accelerating, simplifying, and
reducing risks associated with AI model development and associated downstream
clinical deployment. With this in mind, MONAI is licensed under the Apache-2.0
License, a permissive, altruistic, free-software license that allows the
software to be used for any purpose, to be distributed and modified, without
Project MONAI charging royalties or licensing fees. Copyleft licenses were
explicitly avoided as the redistribution requirement would result in reduced
commercial adoption and contributions, which have significantly benefited the
quality, capabilities, and impact of MONAI.

While MONAI is being developed with minimal required dependencies, namely
PyTorch and NumPy, it provides wrappers and adaptors that allow popular
healthcare AI tools to be used from within MONAI. Such an approach improves
access to these tools and brings the community together in a manner that
preserves software quality and integration.

Several other software initiatives are being developed under the Project MONAI
consortium, building on the foundation provided by MONAI Core: MONAI Label, an
AI-assisted data labeling tool MONAILabel ; MONAI Deploy, a framework and
software development toolkit allowing AI models to be deployed and integrated
into a clinical environment; MONAI FL, providing standards for Federated
Learning; MONAI Education, an educational package that can be used or adapted
to teach AI in healthcare; with others on the way. The Project MONAI
consortium is also working with key societies in the field, such as MICCAI and
MIDL, with the aim of supporting educational initiatives and hackathons and
bringing the research community together.

###  2.3 System Overview and components

Figure 1: Pictorial representation of the MONAI Core modules (top) and MONAI
workflows components (bottom).

The MONAI core modules include the following (as illustrated in Fig 1-Top):
monai.data, targeting the datasets, readers/writers, and synthetic data;
monai.losses, targeting classes defining loss functions and following the
pattern of torch.nn.modules.loss; monai.networks, containing network
definitions, component definitions, and PyTorch-specific utilities;
monai.transforms, defining data transforms for preprocessing and
postprocessing; monai.csrc and monai._extensions, providing C++/CUDA
extensions for MONAI core Python APIs; monai.visualize, containing utilities
for data visualization; monai.metrics, defining metric tracking and analysis
tools; and monai.optimizers, containing classes defining optimizers and
following the pattern of torch.optim. To assemble and integrate different
components, as depicted in Fig 1 (bottom), core modules also include
monai.engines and monai.handlers, defining workflow engines for training and
evaluation, and event handlers for implementing functionality at various
stages in the training and inferencing processes.

###  2.4 Transforms

Data transformation and augmentation is of paramount importance in the domain
of deep learning, primarily when applied to imaging data. They can help with
data processing, from simple use cases such as loading an image from a file or
normalizing the intensity of an image to a given range, to complex geometrical
data transformations taking the image geometry into account. Equally,
transforms can be used in the context of data augmentation to minimize model
overfitting, for example, by randomly rotating or flipping images.

MONAI aims to support users by providing both domain-specific transforms (see
next sections), as well as those commonly used in volumetric medical-imaging
deep learning pipelines. Medical images require highly specialized methods for
I/O, pre-processing, and augmentation. Images are often stored in complex
formats with rich meta-information, with the data volumes being high-
dimensional and requiring carefully designed manipulation procedures.

MONAI provides comprehensive medical image-specific transformations for IO,
spatial, intensity, crop/pad, etc. They can be easily composed into optimized
medical data processing pipelines emphasizing reproducible results. A non-
exhaustive list of examples is provided as follows: LoadImage, which loads
medical images with an automatically-selected package (ITK itk2014 , Nibabel
nibabel , PIL PIL , etc.); Spacing, which resamples input images into a
specified voxel spacing; Orientation, which transforms images to a specified
orientation axis (e.g., RAS); RandGaussianNoise, which perturbs image
intensities by adding statistical noise; NormalizeIntensity, which normalizes
data intensity based on mean and standard deviation; Affine, which transforms
images based on affine parameters; and Rand3DElastic, which performs both a
random elastic and affine deformations in 3D. Some of these examples are also
depicted in Figure 2.

(a) Orientation (RAS)

(b) RandGaussianNoise

(c) RandRotate

(d) RandSpatialCrop

Figure 2: Each block (a-d) represents some data before (top) and after
(bottom) a transformation.

####  2.4.1 Physics-specific transforms

In a medical imaging setting, users often require transforms that have a
grounding in the underlying physics of the acquisition. For example, data
acquired from magnetic resonance (MR) imaging is recorded in $k$-space, and
therefore it is desirable to be able to augment MR images in the $k$-space
domain. The RandKSpaceSpikeNoise transform, which randomly applies localized
radio-frequency spikes in $k$-space, is just one example of many transforms in
MONAI that are inspired by the physics of how medical images are acquired.

####  2.4.2 Invertible transforms

Within deep learning workflows, it is often desirable to invert or revert
previously-applied spatial transforms (resize, flip, rotate, zoom, crop, pad,
etc.) in order to return transformed data to a previous geometrical state.
This is useful if, for example, in test-time augmentation (TTA)
testtimeaugmentation2019 , augmentation-consistency based domain adaptation,
or when the user wishes to save an inferred segmented image in a way that
preserves the original geometry and promotes transformation equivariance.
Doing so requires a full understanding of the tensor geometry and attached
rasterization grid as to ensure the geometry is preserved after inversion.

####  2.4.3 Array and dictionary transforms

The design of MONAI transforms emphasizes code readability and usability.
MONAI transforms can either be applied to a tensor or array of data (e.g., a
single image), or to a data dictionary. The former focuses on simplicity,
whilst the latter allows for more complex pipelines, such as applying the same
random instantiation of a transform to sets of paired data (i.e. input image
and associated ground truth segmentation). Storing data in a dictionary also
allows for meta-information to be used, updated and tracked, for example, to
ensure that the physical attributes of an image are preserved after a resize
transform is applied.

####  2.4.4 Utility transforms

The ability to compose and chain transforms is naturally required when
applying a series of data processing steps; this is achieved in MONAI using
the Compose class. The provided OneOf transform allows the user to build on
Compose by executing one of multiple transforms with some user-defined
probability.

To convert the data shapes or types, utility transforms such as ToTensor,
ToNumpy, SqueezeDim are also provided. These are not only convenient for users
but also enable compatibility with other packages, as described in the
following sections.

####  2.4.5 Simplifying CPU, GPU, PyTorch & NumPy interactions

The majority of transforms in MONAI are compatible with both PyTorch and NumPy
input, do not require explicit conversion between the two formats, and support
both CPU and GPU inputs. Users can also arbitrarily switch between CPU and GPU
devices during their chain of transforms with the ToDevice transform, allowing
for computation-heavy transforms (e.g. those requiring image resampling) to be
performed on the GPU, with other transforms performed on the CPU to save GPU
memory for model training.

####  2.4.6 Compatibility with community-led libraries

MONAI provides adapter tools to accommodate the use of third-party transforms
from packages such as ITK itk2014 , torchIO perez-garcia_torchio_2021 , Kornia
eriba2019kornia , BatchGenerator isensee_fabian_2020_3632567 , Rising rising ,
and cuCIM cuCIM .

For example, cuCIM has implementations of optimized versions of several common
transforms that are often used in digital pathology pipelines. These
transforms are executed on the GPU and act on CuPy arrays, with the CuCIM
class provided to adapt cuCIM operations to MONAI transform types.

###  2.5 Engines, Loss Functions, and Metrics

MONAI focuses on PyTorch-like implementations, and thus, introducing additive
concepts on top of its architecture allows its components to be used in
typical existing workflows. The common pattern for a PyTorch training loop,
consisting of an explicit iteration over input data which is passed to the
model and loss calculation before the optimizer is manually stepped, can
incorporate MONAI networks, loss function, metrics, and other features.
Following our design principles, MONAI training and inference workflows extend
the Engine class of Ignite. These types encapsulate the training loop while
supporting online metric calculation, visualization, and saving network state.
Functionality is added on top of that provided by Ignite, such as the support
for specialized transform handling, provide default training functions, and
provide extra event handling functionality.

Loss functions in MONAI adhere to PyTorch API conventions and can be used in
any existing PyTorch workflow. The added functions introduce specialized
losses not present in the base library, such as Dice and its variants
(generalized, masked, focal, etc.) Sudre_2017 , focal loss, Tversky loss,
contrastive loss, those used for image registration (bending energy, multi-
scale loss, etc.), and several others. Where possible, inheritance is used to
limit code duplication and ease the extension of the provided classes, and
many aspects of the loss calculations, such as choice of reduction, are
controlled via constructor arguments. In addition to PyTorch compatibility,
the design of the loss functions emphasizes flexibility of use and ease of
extension.

Lastly, metrics are used to calculate values for assessing the performance of
an AI model during training or validation. These are typically non-
differentiable, unlike loss functions, allowing them to calculate measures of
performance that would be unsuitable as losses. MONAI has been among the lead
initiatives of the Metrics Reloaded consortium, comprising academic and
industrial image analysis thought leaders from all over the world
MaierHein2022 . The mission of Metrics Reloaded is to foster reliable
algorithm validation through problem-aware choice of metrics with the long-
term goal of (1) enabling the reliable tracking of scientific progress and (2)
bridging the current chasm between AI research and translation into biomedical
imaging practice. Contributions include a comprehensive collection of metric-
related pitfalls Reinke2021  and a corresponding paper on recommendations
MaierHein2022 . Reference implementations for all recommended metrics are
currently being integrated into MONAI. Following the metric definitions
provided by Ignite, MONAI metric classes can be used with an evaluator engine
class to assess the network at key points in training while supporting
distributed data-parallel operations.

###  2.6 Network Architectures

Reference | General Purpose
---|---
AHNet | SegResNet | AutoEncoder
BasicUNet | SegResNetVAE | Regressor
DenseNet | SENet | Classifier
DiNTS | Transchex | Discriminator
DynUNet | UNETR | Critic
EfficientNet | ViT | FullyConnectedNet
HighResNet | ViTAutoEnc | VarFullyConnectedNet
RegUNet | VNet | Generator
ResNet | SwinUNETR | UNet
|  | VarAutoEncoder
Table 1: MONAI Network Definitions

Table 1 lists a subset of the network definitions provided by MONAI and
divided into two primary categories: those defining a reference implementation
of a published architecture, and those defined for general use following what
is considered best design and practice. Reference implementations include
networks such as ResNet, BasicUNet, EfficientNet, and several transformer-
based networks, with implementations explicitly adhering to their published
definitions and settings, rather than providing an abstract architecture.
Where possible, these are also defined with some configurability in place,
such as allowing 1/2/3D inputs and outputs based on constructor parameters,
and a varying number of input and output channel counts.

General purpose networks, conversely, primarily adhere to an architectural
structure emphasizing inheritance for reuse, configurability of number of
spatial dimensions for input and output data, of internal structure (e.g., the
number of levels or layers), and of specific network layers and blocks. The
intent of this class of networks is to define sensible defaults, inheriting
structure or components from other similar networks, while permitting
customization and reuse in other applications without requiring major
rewriting or adaptation.

####  2.6.1 Working network example and extensions

As an example, the UNet class can be instantiated without modification to
operate on one, two, or three-dimensional data as input and output. The number
of levels the network has can also be set to values other than that found in
the reference implementation Ronneberger2015 , subject to the constraints of
downsampling (as discussed later). Each level is by default defined using
blocks of convolution-normalization-dropout-activation layers, but these can
be replaced with residual units or sequences of such layers. Constructor
arguments also set the type of activation or normalization used throughout the
network. As an explicit example, a network segmenting 3D volumetric image data
into four labels can be created by setting the constructor parameters for
dimensions to 3, in_channels to 1 for single-channel inputs, and out_channels
to 4. The number of layers is set by the channels parameter whose values
define the number of output channels for the layers of the encoding side of
the network, e.g., five layers, comprising four encoding layers and a final
bottleneck layer, would be given as (4, 8, 16, 32, 64).

By default, downsampling is performed in the encoding part of the network
using strided convolutions, as defined by the strides parameter for every
layer save the bottleneck. A strides value of (2, 2, 2, 2) implies the network
will downsample the activation map by half at each of the first four layers.
Note that an input whose spatial dimensions are multiples of $2^{4}$ is
necessary for the decoding path to concatenate skip connections and get an
output with the same dimensions as the input.

A much simpler network producing a single channel 2D output from the same sort
of input can be created by changing the dimensions argument to 2 and
out_channels to 1. Using fewer layers can be accomplished with fewer values in
channels and strides. Activation layers are uniformly the same type throughout
the network, by default PReLU https://doi.org/10.48550/arxiv.1502.01852 , but
can be changed by setting the act argument to a name or a type for a different
layer. MONAI provides factory objects to query types from a central location
by names, thus setting act to the string “GELU” gelu  will cause the network
to be constructed with that activation in place of the default.

All general-purpose networks are compatible with Torchscript as scripted
objects when using either the default or a subset of the layer parameters,
maximizing portability and usability of pre-trained networks. This allows
trained networks to be loaded in a process where there is no Python
dependency, thus facilitating downstream deployment and integration. A
scripted network saved to a Torchscript archive can be instantiated and used
for training or inference without MONAI being installed in the host
environment, thus decoupling the library and its version from the stored
network so that requirements and incompatibilities with the host environment
can be minimized.

These network factories are used to easily refer to built-in PyTorch or custom
types, as well as choose the appropriate type for the desired spatial
dimensionality. This latter property is useful for PyTorch layers, which
define separate classes for each dimension as the factory can be queried by
name and other arguments such as dimension. Without this feature, definitions
parameterized on number of dimensions would have to include code to choose
which layer to instantiate based on that value.

Typically MONAI definitions using these factories permit constructor argument
values to be passed along with the name so that the layer can be constructed
in a configurable way. For example, the act argument for UNet can be given
(Act.SELU, "inplace": True) to use torch.nn.SELU as an in-place operation
https://doi.org/10.48550/arxiv.1706.02515 .

###  2.7 Datasets and IO

####  2.7.1 Data representations for imaging

Medical imaging data requires metadata to preserve geometric and other data-
specific, information. As such, MONAI provides first-class support for imaging
data and related networks and transforms. A typical approach to implementing
such domain-specific functionality is to provide concrete types for entities
such as medical images that are then hardwired through the framework. This,
however, conflicts the MONAI design goal that it should be interoperable with
PyTorch; it also limits the reusability of components in cases where the
assumptions made about such specialized types do not hold.

To this end, MONAI has a class named MetaTensor. It inherits from the PyTorch
torch.Tensor class, meaning it can be used wherever tensors are used in
PyTorch. It further extends upon PyTorch functionality by also storing image
metadata. An example of this is the orientation information stored in DICOM or
NIfTI images, which when loaded with MONAI can then be stored alongside the
images, and the MONAI transforms can use and update this meta data as
necessary.

The MetaTensor also stores information, such as transformations that have
already been applied to an image. This is particularly useful when data
augmentations are only applied some of the time, parameterized by user-defined
probabilities (e.g., randomly flipping an image 50% of the time), or where the
user would like to inspect the parameters of a randomly applied transform
(e.g., knowing the rotation matrix that was applied during RandRotate). It
also enables MONAI to perform actions such as inverse transforms (as mentioned
in 2.4.2) all whilst resting in the confines of PyTorch interoperability.

####  2.7.2 Extended dataset functionality

Medical datasets, particularly those for various imaging modalities, are
amongst the most memory-intensive per-sample datasets in deep learning.
Conversely, the relative lack of high-quality labeling means that, at least
for supervised deep learning tasks, training must often be carried out on
relatively small number of samples and data augmentation techniques are very
important for effective learning. As a result, pre-processing of data can be a
significant overhead unless careful attention is paid to the data input and
pre-processing aspects of model training.

MONAI provides extensions to the standard PyTorch Dataset class which help
solve the above problems. In particular, MONAI provides integrated caching and
persistence solutions that can reduce the computational expense of data pre-
processing without adding complexity to the user experience.

CacheDataset provides the ability to cache deterministic operations in a pre-
processing pipeline that can be costly with certain large types of medical
data. Operations such as deterministic pixel/voxel resampling and rescaling
are carried out the first time when a particular sample is loaded, and the
results of those operations are then cached _in memory_ for future training
steps. Note that only preprocessing steps occurring before stochastic steps
can be cached in this way. CacheDataset provides effective performance gains
as long as the cached data does not exceed RAM, and the user can tune the
proportion of the dataset that is cached in this manner.

PersistentDataset provides analogous functionality to CacheDataset, but with
the output of the deterministic stages stored in an intermediary file system
representation rather than to memory. The use of PersistentDataset is
recommended for 3D datasets or where the overall dataset size is much larger
than RAM.

####  2.7.3 Wrappers for reference datasets

As with machine learning in other domains, reference datasets exist for
medical deep learning applications. For example, MedNIST 111Made available by
Dr. Bradley J. Erickson M.D., PhD (Department of Radiology, Mayo Clinic) under
the Creative Commons CC BY-SA 4.0 license. serves as an MNIST handwritten
digit dataset analogue lecun2010mnist , providing a standard dataset that is
suitable for initial training and benchmarking of various medically-focused
deep learning applications. Another example, the Medical Segmentation
Decathlon decathlon , represents a set of 10 challenge datasets, widely used
to demonstrate network performance across a range of segmentation tasks. MONAI
provides Dataset extension classes that simplify the downloading, storage and
partitioning of the MedNIST and Medical Segmentation Decathlon datasets, and
demonstrate a best practice for wrapping other such datasets. These datasets
are extensions of MONAI CacheDataset, facilitating rapid training. In addition
to providing an extended dataset class for the Medical Segmentation Decathlon,
the MONAI Consortium also manages a msd-for-monai S3 bucket in US and EU
regions, via AWS Open Data, with the aim of facilitating cloud-based model
training and data distribution. A plan is currently in place to further extend
beyond the current dataset pool and provide a common platform for large
research data sharing and ingestion for the community. Recently MONAI took a
first step towards this by introducing TciaDataset to automatically download
and extract publicly-accessible datasets from The Cancer Imaging Archive with
accompanying DICOM segmentations, and act as PyTorch datasets to generate
training, validation, and test data.

###  2.8 Training, Inference Engines and Event Handlers

To quickly set up training and evaluation experiments, MONAI provides a set of
workflows to significantly simplify the setup of a MONAI script and allow for
fast prototyping.

These features decouple the domain-specific components and the generic
machine-learning processes. They also provide a set of unified APIs for
higher-level applications (such as AutoML, and Federated Learning). The
trainers and evaluators of the workflows are compatible with the Ignite engine
and event-handler mechanism. There are rich event handlers in MONAI to
independently attach to the trainer or evaluator, and users can register
additional custom events to workflows.

Distributed data-parallel is an important feature of PyTorch to connect
multiple GPU devices, on single or multiple nodes, to train or evaluate
models. The distributed data-parallel APIs of MONAI are compatible with the
native PyTorch distributed module, Ignite distributed module, Horovod, XLA,
and SLURM.

For model inferences on large volumes, the sliding window approach is a
popular choice to achieve high performance while having flexible memory
requirements. A typical process comprises the selection of continuous windows
on the original image, followed by iteratively running batched window
inferences until all windows are analyzed. The inference outputs are then
aggregated into a single segmentation map, and the results are saved to file
or compute some evaluation metrics. This approach also supports ‘overlap‘ and
‘blending_mode‘ configurations to better handle window overlap, resulting in
improved performance.

The Ignite library provides event handlers to decouple training/inference
pipelines from additional logic which gets triggered at set times or events
during execution. Users can extend handlers for specific requirements, use
pre-existing handlers for logging, metric calculation, or other tasks, or
attach their own defined handler routines. These will be triggered by specific
events during execution such as at the completion of a batch or an epoch, or
during data loading, allowing data to be analyzed, model parameters adjusted,
or other operations at these key moments.

###  2.9 Visualisations and utilities

####  2.9.1 Tensorboard

When developing large workflows, it is crucial that users are able to
visualize, track and understand processes and progress. To this end, MONAI
provides many helper functions to the popular package, Tensorboard
abadi2016tensorflow . Images in medical imaging are often 3D, so MONAI wrapper
functions enable the presentation of these images in the form of GIFs, a set
of 2D slices, or as interactive 3D renderings major2022tensorboard3d . These
can be used in conjunction with the engines mentioned in the previous section.

####  2.9.2 Traceable transformations

As mentioned in Section 2.4.3, MONAI transforms can be applied to both arrays
and dictionaries of data. When dictionaries are used, as well as storing the
input metadata corresponding to each image, a stack of transforms that have
been applied to that image can also be stored. This is particularly useful
when data augmentations are only applied some of the time, parameterized by
user-defined probability (e.g., randomly flipping an image 50% of the time),
or where the user would like to inspect the parameters of a randomly applied
transform (e.g., knowing the rotation matrix that was applied during
RandRotate). It also provides a mechanism for augmentation auditing and
traceability, a subject of increasing importance when developing AI models for
clinical use. Internally, MONAI also uses this functionality when performing
inverse transformations, when it is appropriate to invert with a last in,
first out strategy.

####  2.9.3 Interpretability

Figure 3: Examples of interpretability with MONAI. The MedNIST dataset was
used here, and the network correctly classified each image, which from left to
right were head CT, chest x-ray and hand x-ray. From top to bottom, the
columns are the input image and then the outputs from occlusion sensitivity
and GradCAM++ (a GradCAM variant). Blue indicates areas of the image that the
network relied on more heavily in the decision-making process.

MONAI supports popular variants of network interpretability that are often
employed in classification tasks. These include occlusion sensitivity OccSens
, GradCAM GradCAM  and Smoothgrad SmoothGrad . These methods help the
researchers address the question: “which parts of the image were important in
arriving to a given decision of a classification?”. This allows users to
better visualize the inner workings of their models. Examples of occlusion
sensitivity and GradCAM++ (a GradCAM variant) can be seen in Figure 3.

Another form of interpretability is test-time augmentation (TTA), which is
also available in MONAI. TTA works by repeatedly applying data augmentations
and averaging the inferred outputs. If spatial transformations have been
applied, the inverse transformation is required — as previously mentioned,
this functionality is also provided by MONAI.

####  2.9.4 Utility functions

Lastly, a wealth of other functions exist in MONAI for helping users visualize
their results. An example of this is the blend_image function, which allows
users to easily create RGB images from the superposition of images and labels
(ground truth or inferred), examples of which can be seen in Figure 2. Another
example of utility functions used for visualizations is the matshow3d
function, which creates a 3D volume figure as a grid of images.

##  3 Experiments and Applications

This section introduces four applications with the aim of demonstrating the
breadth and benefits of MONAI.

###  3.1 Segmentation

The typical PyTorch training workflow consists of a training loop which feeds
data into the network and optimizes its parameters, followed by an evaluation
loop where validation data is used in conjunction with a metric to assess
training progress. PyTorch provides the relatively low-level components for
this workflow but relies on libraries such as Ignite for higher-level
definitions. Types which encapsulate the training process offer ease of use at
the expense of some flexibility but represent a significant acceleration of
development for most users.

Training a segmentation network with ground truth annotated data is one
instance of this common supervised process. The network is tasked with
predicting the segmentation image from the input image, and this is compared
using a loss such as Dice against the known actual segmentation. A supervised
segmentation training workflow in PyTorch can be sketched as the following:

PyTorch Training Loop ⬇ for epoch in range(max_epochs):  network.train()  for
inputs, labels in train_loader:  optimizer.zero_grad()  outputs =
network(inputs)  loss = loss_function(outputs, labels)  loss.backward()
optimizer.step()  \parnetwork.eval()  with torch.no_grad():  for val_inputs,
val_labels in val_loader:  val_outputs = network(val_inputs)
metric(y_pred=val_outputs, y=val_labels)  \parmetric =
metric.aggregate().item()  print(”Validation result:”, metric)

MONAI provides types inheriting from Ignite definitions to encapsulate this
process. The purpose is to simplify and regularize how workflows are created
to make the process quicker, easier, and more reproducible. For a supervised
training workflow, the SupervisedTrainer and SupervisedEvaluator types define
the training and validation components discussed previously. The previous
training script is thus transformed into an instantiation of these types
followed by a call to run to hand control of the process to the internal
training mechanism:

MONAI Training Loop ⬇ evaluator = SupervisedEvaluator(
val_data_loader=val_loader,  network=network,  key_val_metric={ ”metric”:
metric },  …  )  \partrainer = SupervisedTrainer(  max_epochs=num_epochs,
train_data_loader=train_loader,  network=network,  optimizer=optimizer,
loss_function=loss_function,  train_handlers=[ValidationHandler(1,evaluator)],
…  )  \partrainer.run() # do the training run for 10 epochs

In addition to simplifying the code down to a few lines of object
instantiations, this form of workflow aids in reproducibility and correctness
by limiting the number of details and definitions the scientist needs to
implement for training. With more moving parts and structural definitions a
user must implement, even simple things like loops and method calls, the more
opportunity there is for variation or error to be introduced into the code.
Reproduction for software experiments requires rerunning or repurposing pre-
existing code for new situations, and so introduced variations between code
bases that are expected to be equivalent complicate the process of deriving
and verifying scientific results.

###  3.2 Classification

For many training and evaluation tasks, deterministic behavior is important
for consistent and reproducible results. MONAI provides routines for managing
determinism in workflows by setting random seed values for PyTorch, Numpy, and
the native Python random library, as well as setting backend flags, to
appropriate values. This ensures that many stochastic processes, including the
evaluation of data transforms and network processes like dropout, can be
expected to produce identical results between runs.

For example, a fixed seed can be set for all involved libraries as well as
setting flags for deterministic behavior, followed by computation, and then
returning the state of the running program to non-determinism:

MONAI Training Loop ⬇ random_seed = 42  monai.utils.set_determinism(
seed=random_seed,  use_deterministic_algorithms=True )  \par# perform
computation \parmonai.utils.set_determinism(  seed=None,
use_deterministic_algorithms=False )

Classification problems benefit from this determinism when interpreting
network behavior and results. MONAI provides networks, losses, and metrics for
classification problems, and visualization routines for displaying occlusion
sensitivity and the GradCAM method. Making sense of the interpretations these
methods provide requires experimentation and minor changes to network
parameters to observe results. Being able to fix a deterministic training
process allows the user to minimize the variability that would be introduced
with randomized transforms or internal PyTorch/Numpy processes in their code,
and allows consistent results to be generated and visualized.

###  3.3 Registration

DeepReg Fu2020  is an open-source toolkit of networks and algorithms for image
registration. A number of these components have been ported into MONAI and are
provided as standard loss functions and network definitions.

MONAI provides a semi-supervised example of registering lung CT images at
inspiration to expiration. During training, a deformation field is estimated
by the network to deform the moving image and its segmentation to the fixed
image and segmentation. A combined loss scores the results based on the
similarity between the deformed and fixed images, the Dice score of the
deformed and fixed segmentation calculated at various scales, and an energy-
based regularisation term.

The image loss is the standard PyTorch MSE function, and the Dice loss is that
provided by MONAI and used in other segmentation applications. The
regularisation function is a bending loss term based on second-order
derivatives and is a MONAI definition.

Deformation is performed using a MONAI class which can use resampling routines
included in PyTorch, or MONAI’s compiled code for doing the same. Note that
these routines implement interpolation models (e.g. tricubic), which are not
present in PyTorch, in addition to having speed advantages. MONAI thus
provides a number of important tools for training deep learning registration
algorithms not present in the base PyTorch library.

###  3.4 Benchmarks

Typically, model training is a time-consuming step during deep learning
development, especially in medical imaging applications. Volumetric medical
images are usually large (as multi-dimensional arrays), and the model training
process can be complex. Even with powerful hardware platforms (e.g., CPU/GPU
with large RAM), it is not easy to fully leverage them to achieve high
performance. MONAI provides not only optimized tools and code but also a
detailed guide based on best practices to achieve the best performance.

NVIDIA GPUs have been widely applied in many areas of deep learning training
and evaluation, and the CUDA parallel computation library shows obvious
acceleration when compared to traditional computation methods. To fully
leverage GPU features, many popular mechanisms are provided in PyTorch, like
automatic mixed precision (AMP), distributed data-parallel, etc. MONAI can
support these features and provides rich examples. Even with CacheDataset, one
still needs to copy the same data to GPU memory for random transforms or
network computation for every epoch. To address this issue, an efficient
approach is to cache the data to GPU memory directly, allowing for more
efficient and immediate GPU computation.

As an example, one first converts to PyTorch Tensor with the EnsureTyped
transform and moves data to GPU with the ToDeviced transform. CacheDataset
then caches the transform results until ToDeviced, so caches are stored in GPU
memory. Then in every epoch, the program fetches cached data from GPU memory
and only executes the following random transforms (like
RandCropByPosNegLabeld) on GPU directly.

A fast model training guide is available as a MONAI tutorial, providing an
overview of the fast training techniques in practice, covering aspects such as
profiling the pipelines, optimizing data loading function, algorithmic
improvement, optimizing GPU utilization, and leveraging multi-GPU and multi-
node distributed training.

##  4 Conclusions

We present MONAI, an open-source and community-supported, consortium-led
PyTorch-based framework for deep learning in healthcare, with a particular
focusing providing a standardized and flexible platform for developing
biomedical and clinical applications. This paper introduces key design
considerations and implementation choices that have driven the development of
MONAI, presents the rationale behind the MONAI open-source strategy and
license, and introduces MONAI’s major system components. Finally, example use
cases of MONAI are provided, highlighting the benefits of the platform.
Overall, we believe MONAI can help accelerate and simplify the development of
AI models and contribute towards more impactful research, development, and
clinical deployment.

##  5 Acknowledgments

JC, EK, TV, and SO acknowledge funding from the Wellcome/EPSRC Centre for
Medical Engineering (WT203148/Z/16/Z), Wellcome Flagship Programme
(WT213038/Z/18/Z). JC, EK, SO and HS also acknowledge funding from the London
Medical Imaging and AI Centre for Value-based Healthcare. SA was funded, in
part, from the National Institutes of Health via NIBIB and NIGMS R01EB021396,
NIBIB R01EB014955, NCI R01CA220681, and NINDS R42NS086295. This project has
been funded in whole or in part with federal funds from the National Cancer
Institute, National Institutes of Health, under Contract No.
HHSN261200800001E. The content of this publication does not necessarily
reflect the views or policies of the Department of Health and Human Services,
nor does mention of trade names, commercial products, or organizations imply
endorsement by the U.S. Government.

## References

  *   * (1) Kraljevic, Z. _et al._ Multi-domain clinical natural language processing with medcat: The medical concept annotation toolkit.  _Artificial Intelligence in Medicine_ 117, 102083 (2021).  URL https://www.sciencedirect.com/science/article/pii/S0933365721000762.  https://doi.org/10.1016/j.artmed.2021.102083 .
  * (2) Brzezicki, M. A. _et al._ Artificial intelligence outperforms human students in conducting neurosurgical audits.  _Clinical Neurology and Neurosurgery_ 192, 105732 (2020).  URL https://www.sciencedirect.com/science/article/pii/S0303846720300755.  https://doi.org/10.1016/j.clineuro.2020.105732 .
  * (3) Nelson, A., Herron, D., Rees, G. & Nachev, P.  Predicting scheduled hospital attendance with artificial intelligence.  _npj Digital Medicine_ 2 (1), 26 (2019).  URL https://doi.org/10.1038/s41746-019-0103-3.  10.1038/s41746-019-0103-3 .
  * (4) Panch, T., Mattie, H. & Celi, L. A.  The “inconvenient truth”about ai in healthcare.  _npj Digital Medicine_ 2 (1), 77 (2019).  URL https://doi.org/10.1038/s41746-019-0155-4.  10.1038/s41746-019-0155-4 .
  * (5) Abadi, M. _et al._ Tensorflow: A system for large-scale machine learning (2016).  URL https://www.usenix.org/system/files/conference/osdi16/osdi16-abadi.pdf.
  * (6) Paszke, A. _et al._ Pytorch: An imperative style, high-performance deep learning library (2019).  URL https://arxiv.org/abs/1912.01703.
  * (7) Chen, T. _et al._ Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems (2015).  URL https://arxiv.org/abs/1512.01274.
  * (8) Gibson, E. _et al._ Niftynet: a deep-learning platform for medical imaging.  _Computer Methods and Programs in Biomedicine_ 158, 113–122 (2018).  URL https://www.sciencedirect.com/science/article/pii/S0169260717311823.  https://doi.org/10.1016/j.cmpb.2018.01.025 .
  * (9) Pawlowski, N. _et al._ Dltk: State of the art reference implementations for deep learning on medical images (2017).  URL https://arxiv.org/abs/1711.06853.
  * (10) Beers, A. _et al._ Deepneuro: an open-source deep learning toolbox for neuroimaging.  _Neuroinformatics_ 19 (1), 127–140 (2021).  10.1007/s12021-020-09477-5 .
  * (11) He, H.  The state of machine learning frameworks in 2019.  _The Gradient_ (2019) .
  * (12) Diaz-Pinto, A. _et al._ Monai label: A framework for ai-assisted interactive labeling of 3d medical images (2022).  URL https://arxiv.org/abs/2203.12362.
  * (13) McCormick, M., Liu, X., Ibanez, L., Jomier, J. & Marion, C.  ITK: enabling reproducible research and open science.  _Frontiers in Neuroinformatics_ 8, 13 (2014).  URL https://www.frontiersin.org/article/10.3389/fninf.2014.00013.  10.3389/fninf.2014.00013 .
  * (14) Brett, M. _et al._ nipy/nibabel: 3.2.1 (2020).  URL https://doi.org/10.5281/zenodo.4295521.
  * (15) van Kemenade, H. _et al._ python-pillow/pillow: 8.4.0 (2021).  URL https://doi.org/10.5281/zenodo.5571504.
  * (16) Wang, G. _et al._ Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks.  _Neurocomputing_ 338, 34–45 (2019).  URL https://www.sciencedirect.com/science/article/pii/S0925231219301961.  https://doi.org/10.1016/j.neucom.2019.01.103 .
  * (17) Pérez-García, F., Sparks, R. & Ourselin, S.  Torchio: a python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning.  _Computer Methods and Programs in Biomedicine_ 106236 (2021).  URL https://www.sciencedirect.com/science/article/pii/S0169260721003102.  https://doi.org/10.1016/j.cmpb.2021.106236 .
  * (18) Riba, E., Mishkin, D., Ponsa, D., Rublee, E. & Bradski, G.  Kornia: an open source differentiable computer vision library for pytorch (2020).  URL https://arxiv.org/pdf/1910.02190.pdf.
  * (19) Isensee, F. _et al._ batchgenerators - a python framework for data augmentation (2020).  URL https://doi.org/10.5281/zenodo.3632567.
  * (20) Schock, J., Baumgartner, M. & Weninger, L.  Phoenixdl/rising: High-performance differentiable medical data augmentation.  https://github.com/PhoenixDL/rising.  Accessed: 2021-12-20.
  * (21) Lee, G., Bae, G., Zaitlen, B., Kirkham, J. & Choudhury, R.  cucim - a gpu image i/o and processing library (2021).  URL https://doi.org/10.25080/majora-1b6fd038-022.
  * (22) Sudre, C. H., Li, W., Vercauteren, T., Ourselin, S. & Jorge Cardoso, M.  Cardoso, M. J. _et al._ (eds) _Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations_.  (eds Cardoso, M. J. _et al._) _Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support_ , 240–248 (Springer International Publishing, Cham, 2017).
  * (23) Maier-Hein, L. _et al._ Metrics reloaded: Pitfalls and recommendations for image analysis validation (2022).  URL https://arxiv.org/abs/2206.01653.
  * (24) Reinke, A. _et al._ Common limitations of image processing metrics: A picture story (2021).  URL https://arxiv.org/abs/2104.05642.
  * (25) Ronneberger, O., Fischer, P. & Brox, T.  in U-Net: Convolutional Networks for Biomedical Image Segmentation (eds Navab, N., Hornegger, J., Wells III, W. M. & Frangi, A. F.) _MICCAI 2015_ , Vol. 9351 of _LNCS_ 234–241 (Springer, 2015).
  * (26) He, K., Zhang, X., Ren, S. & Sun, J.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification (2015).  URL https://arxiv.org/abs/1502.01852.
  * (27) Hendrycks, D. & Gimpel, K.  Gaussian error linear units (gelus) (2016).  URL https://arxiv.org/abs/1606.08415.
  * (28) Klambauer, G., Unterthiner, T., Mayr, A. & Hochreiter, S.  Self-normalizing neural networks (2017).  URL https://arxiv.org/abs/1706.02515.
  * (29) LeCun, Y., Cortes, C. & Burges, C.  Mnist handwritten digit database.  _ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist_ 2 (2010) .
  * (30) Antonelli, M. _et al._ The medical segmentation decathlon (2021).  URL https://arxiv.org/abs/2106.05735.
  * (31) Abadi, M. _et al._ Tensorflow: Large-scale machine learning on heterogeneous distributed systems (2016).  1603.04467.
  * (32) Major, B., McCormick, M. & Aylward, S.  Tensorboardplugin3d: 3d tensor visualization.  https://github.com/KitwareMedical/tensorboard-plugin-3d (2022).
  * (33) Zeiler, M. D. & Fergus, R.  Fleet, D., Pajdla, T., Schiele, B. & Tuytelaars, T. (eds) _Visualizing and Understanding Convolutional Networks_.  (eds Fleet, D., Pajdla, T., Schiele, B. & Tuytelaars, T.) _Computer Vision – ECCV 2014_ , 818–833 (Springer International Publishing, Cham, 2014).
  * (34) Selvaraju, R. R. _et al._ Grad-cam: Visual explanations from deep networks via gradient-based localization (2017).
  * (35) Smilkov, D., Thorat, N., Kim, B., Viégas, F. & Wattenberg, M.  Smoothgrad: removing noise by adding noise (2017).  URL https://arxiv.org/abs/1706.03825.
  * (36) Fu, Y. _et al._ DeepReg: a deep learning toolkit for medical image registration.  _Journal of Open Source Software_ 5 (55), 2705 (2020).  URL https://doi.org/10.21105/joss.02705.  10.21105/joss.02705 .