11institutetext: Helsinki Institute for Information Technology HIIT,
Department of Computer Science, Aalto University 22institutetext: Department
of Computer Science, University of Manchester 33institutetext: Bitville Oy,
Espoo, Finland

# Learning to Assist Agents by Observing Them

Antti Keurulainen 1133 Isak Westerlund 33 Samuel Kaski 1122 Alexander Ilin 11

###### Abstract

The ability of an AI agent to assist other agents, such as humans, is an
important and challenging goal, which requires the assisting agent to reason
about the behavior and infer the goals of the assisted agent. Training such an
ability by using reinforcement learning usually requires large amounts of
online training, which is difficult and costly. On the other hand, offline
data about the behavior of the assisted agent might be available, but is non-
trivial to take advantage of by methods such as offline reinforcement
learning. We introduce methods where the capability to create a representation
of the behavior is first pre-trained with offline data, after which only a
small amount of interaction data is needed to learn an assisting policy. We
test the setting in a gridworld where the helper agent has the capability to
manipulate the environment of the assisted artificial agents, and introduce
three different scenarios where the assistance considerably improves the
performance of the assisted agents.

###### Keywords:

Deep reinforcement learning, cooperative AI, helper agent, meta-learning,
modelling other agents.

##  1 Introduction

The ability to build an AI system that can assist other AI agents and humans
to reach their goals is an obviously important and ambitious goal. There are
numerous possible applications that would benefit directly of such a
capability. For example, helper agents could be harnessed to guide self-
driving cars to allow smoother traffic flow. An example of cooperation with
humans could be a helper agent, which is able to infer the goals of a human
navigating a web site, and manipulate the website structure dynamically to
help the human users reach their goals faster.

In this paper, we consider building such helper agents. We construct settings,
where an artificial agent (a helper agent) observes the behavior of an
assisted agent (goal-driven agent), and learns to perform useful assisting
actions by manipulating the environment dynamics.

We formulate the challenge as a meta-reinforcement learning problem. We first
train a small population of goal-driven agents to reach their goals as fast as
possible. The helper agent observes the goal-driven agents in many instances
of the environments, and is trained by meta-reinforcement learning to perform
useful assisting actions by receiving a positive reward when the goal-driven
agents succeed in their tasks.

As a contribution of our work, we show that similar capabilities to inverse
reinforcement learning arise spontaneously as a result of meta-learning. We
demonstrate this effect and its usefulness for producing helping actions in
three different scenarios:

  1. 1.

By learning representations of the behavior of the goal-driven agent from the
past partial trajectories in an unsupervised way.

  2. 2.

By learning representations of the behavior of the goal-driven agent from the
past partial trajectories in a self-supervised way, by predicting the actions
of the goal-driven agent.

  3. 3.

By learning representations of the behavior of the goal-driven agent from the
past partial trajectories in a supervised way, by predicting the goal of the
goal-driven agent.

##  2 Related work

There is a rich literature on reasoning about the behavior and inferring the
goals of other agents [1], which is closely related to our work. Unlike most
of the work related to modelling other agents, our specific goal was to build
agents which can do assisting actions, based on the inferred representations
of the behavior of the other agent.

Similar settings have been studied in multi-agent scenarios. As an example, in
[5] a two-agent helper-AI scenario is presented, where both agents have the
same reward function, but disagree on the transition dynamics of the Markov
decision process (MDP). In our setting, in contrast, the action spaces of the
agents are different, and the assisted agent is unaware of the assisting
agent. The authors in [8] consider learning representations of agent behaviors
in multi-agent scenarios. Our approach is similar in that we also learn to
represent agents’ behaviors, but rather than communicating cooperatively, the
helper agent attempts to change the environment in a beneficial manner.

An individual is considered to have a Theory of Mind (ToM), if they impute a
mental state to themselves or others [12, 19]. An inferred mental state can be
used to infer goals and make predictions of behavior of other agents. Our work
is inspired by Theory of Mind, as we are training behavioral embeddings of the
goal-driven agents by only observing them. Examples of previous work
implementing Theory of Mind in a machine learning context include [3], a
Bayesian Theory of Mind model, and [20], which is based on deep learning. In
our approach, a separate helper agent infers the goal-driven agent behavior
and produces behavior embeddings as in [20], which are used for manipulating
the environment in a beneficial way.

The problem of learning fast from limited amounts of data can be tackled with
meta-learning methods, where the goal is to train a system in such a way that
it is capable of performing fast adaptation to new situations [11, 22]. We
train the helper agent to be able to do few-shot learning of a new goal-driven
agent in a new sampled environment by creating a behavior embedding of the
agent based on small amounts of observations. Commonly used meta-learning
methods include gradient based methods [7] and memory-based methods [6, 21,
24]. Our meta-learning solution resembles mostly attention-based meta-learning
[14], since in our case the behavior embeddings are generated by averaging the
state representations over the time steps without recurrent connections.

Our third scenario is partly inspired by the centralized training and
decentralized execution (CTDE) concept [10, 16], where additional information
can be used during the training but not during the execution. Instead of using
private information of the goal-driven agent, we use information of the goal-
driven agent’s goal for pre-training the helper agent policy.

Similar challenges could be solved by transfer learning or by using auxiliary
tasks. In transfer learning, the goal is to transfer what has been learned in
one task to some other final task, possibly by partially sharing parameters
between the models [18, 25]. Numerous success cases have demonstrated benefits
of transfer learning in the areas of computer vision [17] and natural language
processing [2, 4]. Our solution is one type of transfer learning, where the
goal is specifically to pre-train one specific part of the policy function to
allow faster learning for producing assisting actions. When using auxiliary
tasks, the common representation among the original task and auxiliary tasks
can be improved by adding extra auxiliary heads to the representation [9, 13,
26]. In our solution, instead of adding an extra auxiliary head to the common
representation, the helper agent is pre-trained on the task of making
predictions of the goal-driven agents’ actions in self-supervised manner or
goals in a supervised manner. After completing the pre-training task, the
final task training benefits from this ability.

Inverse Reinforcement Learning (IRL) [15] has similar goals, but does so by
recovering the unknown reward function of the expert user or any agent it
tries to model. The difference in our method is that we do not attempt to
recover the reward function but rather model the agent behavior directly by
observing the agent actions, and then build the helper policy, which uses the
behavior embeddings. This approach is useful when moving to behavior where
reward functions are complex or intractable, as often is the case when
attempting to model human-like complex behavior.

##  3 Setting

We use a meta-reinforcement learning formulation for training both the goal-
driven agents and the helper agent. The MDP is defined by a tuple
$M=\langle\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma,\rho_{0},H\rangle$,
where $\mathcal{S}$ is the state space and $\mathcal{A}$ is the action space,
$\gamma$ is the discount factor, $\rho_{0}:\mathcal{S}\rightarrow[0,1]$ is the
initial state distribution,
$\mathcal{P}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,1]$ is
the transition function,
$\mathcal{R}:\mathcal{S}\times\mathcal{A}\rightarrow{\rm I\\!R}$ is the reward
function and $H$ is the horizon.

The agents maximize the expected sum of discounted rewards over the episodes
$J(\pi_{\theta})=E_{\tau\sim
p_{\pi}(\tau)}[\sum_{t=0}^{H}\gamma^{t}\mathcal{R}(s_{t},a_{t})]$, where
$\tau=(s_{0},a_{0},...)$ is the trajectory of states $s_{t}$ and actions
$a_{t}$ of the agent at time step $t$ for an episode of length $H$.

The initial state is sampled from the distribution $\rho_{0}$ and the agent
$j$ samples the action $a_{t}$ from its policy function
$\pi_{j}(a_{t}|s_{t})$. The next state is sampled from the transition dynamics
function $s_{t+1}\sim\mathcal{P}(s_{t+1}|s_{t},a_{t})$. The agents learn
through many episodes. For each episode, a new MDP $M_{i}$ is sampled from a
family $\mathcal{M}$ of MDPs. The MDPs for the helper agent and the goal-
driven agents share the same state space $\mathcal{S}$ but have different
actions spaces and reward functions denoted as $\mathcal{A}^{helper}$ and
$\mathcal{A}^{goal-driven}$, and $\mathcal{R}^{helper}$ and
$\mathcal{R}_{j}^{goal-driven}$. From the helper agent point of view, the
behavior of the goal-driven agent is embedded in the transition dynamics of
the sampled MDP, and each sampled MDP refers to different task in the meta-
reinforcement learning framework. The meta-learning goal is to maximize the
expected returns over the task distribution.

##  4 Methods

###  4.1 The gridworld

We test our architectures in various gridworld scenarios. Every new sampled
MDP has a 7 x 7 gridworld environment, which includes four possible goals in
random locations, an L-shaped wall in a random location and a goal-driven
agent starting in a random position. Each goal-driven agent is trained to
navigate to one of the goals with the shortest route. The wall has always the
same basic shape, and its location is sampled but restricted so that the long
edge of the wall needs to be at least two steps from the border of the
gridworld. The left side of Figure 1 illustrates a sampled gridworld
environment.

The environment delivers a set of tensors as a state representation, one
tensor per one time step. The state representation is a collection of 7x7x6
tensors, where the different channels in one tensor represent goal-driven
agent location (1 channel), wall elements (1 channel) and goals (4 channels).

The episodes are divided in two distinct stages. In the first stage, the
helper agent observes several partial trajectories of the unassisted goal-
driven agent navigating to its goal in different sampled environments. In the
second stage, a new environment is sampled and the same goal-driven agent is
set in a random location. In the beginning of the second stage, the helper
agent observes the state (a single tensor) and will decide on an action based
on this observation. The right side of Figure 1 illustrates a situation where
the helper agent has acted and opened a shortcut for the goal-driven agent.

Figure 1: Illustration of the 7x7 gridworld. An agent trajectory is
illustrated with a dotted line, when the goal-driven agent is walking towards
the red goal. Left: the helper agent has not opened a shortcut. Right: The
helper agent has opened a shortcut

###  4.2 The goal-driven agents

The goal-driven agents, which are part of the environment of the helper agent,
are trained with the Proximal Policy Optimization (PPO) algorithm [23]. We
train four different agent types which have different target goals. The action
space $\mathcal{A}^{goal-driven}$ consists of five possible actions (move up,
down, left, right and stay). The episode ends with a positive reward when the
agent reaches the correct goal from four possible goals. Each step has a small
negative reward. During the training of the goal-driven agent, there is a
shortcut in the wall with the probability 0.5.

When the goal-driven agent $j$ follows its policy, it will produce a
trajectory consisting of a list of state-action pairs

$\tau_{j}=\Big{\\{}\big{(}s_{t},a_{t}\big{)}\Big{\\}}_{t=0}^{H}$

where $H$ is the length of the episode.

###  4.3 The helper agent

Once a population of goal-driven agents exists, the helper agent is trained
with the PPO algorithm to infer the agent types by observing them, and to
conduct an assisting action by manipulating the transition dynamics (creating
a shortcut in a specified location). The action space $\mathcal{A}^{helper}$
of the helper agent consists of two actions, either to act or not to act, and
the action is performed once at a specified time point depending on the
scenario in question, thus the horizon $H$ for the helper agent is a single
time step. The reward function $\mathcal{R}^{helper}$, is the same as with the
goal-driven agent but augmented with a negative reward for making an action to
assist the goal-driven agent. In other words, the helper agent gets a reward
when the goal-driven agent succeeds in its task. The helper agent has access
only to the partial trajectories of $\tau_{j}$ of the goal-driven agents and
thus not to their reward function $R_{j}^{goal-driven}$ or policy
$\pi_{j}^{goal-driven}$.

##  5 Experiments and results

###  5.1 End-to-end training without agent-specific inference.

We first evaluate the performance of the helper agent in a simplified
situation where it does not have access to the past observations of unassisted
behavior of the goal-driven agents. The results of this experiment serve as a
lower bound for the helper agent performance. The helper agent does not
observe the first stage of each episode and needs to decide on its actions
only based on the initial state of the second stage. Thus, in this case the
helper agent does not have the capability of inferring the behavior of the
goal-driven agents.

In this scenario, the helper agent is a neural network with two convolutional
layers and one fully-connected layer (see Figure 3, left). More details of the
neural network implementation is available in the appendix.

The green line in Figure 2 illustrates the training result in this setting.
The helper agent learns to improve its policy slightly despite not being able
to infer the agent behavior. In this case, the helper agent tries to find the
optimal performance by observing the general behavior of all agent types and
by finding a more general policy that does not depend on a single agent’s
goal. As an example, if all goals are on the other side of the wall, it would
be beneficial to open the wall. Likewise, if all goals are on the same side of
the wall as the goal-driven agent, it is not useful to open the wall and take
the penalty. These two scenarios are illustrated in Figure 3.

Figure 2: The average learning curves of simulations with 8 random seeds with
various helper agent architectures. The green line represents the learning
curve without behavior embeddings and serves as a lower bound for the helper
agent performance. The orange line shows the performance when the policy
network uses ground truth information about the agent behavior, and serves as
the upper bound for the helper agent performance. The blue line represents
end-to-end training with embeddings (scenario 1), the purple line with action
prediction pre-training (scenario 2) and the red line with goal prediction
pre-training (scenario 3). The shaded region illustrates the standard
deviation over 8 seeds. Figure 3: Left: A simplified version of the helper
agent that is not capable of inferring the goal-driven agent behavior. Middle:
The helper agent has not opened the wall because all the goals are on the same
side as the agent. Right: The helper has opened the wall because all the goals
are on the other side. The colored squares indicate the goals, black lines
represent the wall and the black circle is the location of the goal-driven
agent.

###  5.2 Scenario 1: End-to-end training with agent-specific inference.

In the first scenario, the helper agent is exposed to the past observations of
unassisted behavior of the goal-driven agents. In the first stage, it now
observes four first steps from four different episodes of the same agent but
with different sampled layout of the gridworld. By observing these partial
trajectories before conducting the assisting action, the helper agent has a
chance to learn about the goal-driven agents’ goals based on the agents’
unassisted behavior. The helper agent architecture is illustrated in Figure 5.
During the first stage, the helper agent uses the behavior inference part of
the policy. The embeddings of partial episodes are formed by using a
convolutional network, followed by mean pooling and a fully connected layer.
The behavior embeddings are formed in a similar manner from the episode
embeddings but without convolutional layers. During the training, the behavior
inference part learns to construct 2-dimensional behavior embeddings by
clustering the agents based on their goals, where each cluster contains agents
with the same goal (Figure 4).

Figure 4: Visualization of the 2-dimensional behavior embedding in scenario 1.
The behavior inference part of the policy has spontaneously clustered the
goal-driven agents based on their goals. Figure 5: Scenario 1. The helper
agent architecture. The behavior inference part infers the goal driven agent
behavior by observing partial unassisted behavior. The policy head produces
the action distribution from a query state and the behavior embedding.

The policy head part of the helper agent has the same structure as the simpler
case shown in the left side of Figure 3. The task of the policy head is to
produce the action distribution but now it additionally receives the behavior
embedding from the behavior inference part. The training of this scenario is
conducted end-to-end.

The result of this experiment is illustrated by the blue line in Figure 2. The
helper agent is capable to use the information produced by the behavior
inference part and as a result the performance is improved considerably. Since
there is significant variance present in simulations with different seeds, we
can assume that the end-to-end training is more sensitive to the
initialization than the other scenarios.

###  5.3 Scenario 2: Self-supervised pre-training with agent-specific
inference.

In this scenario the helper agent has an initial pre-training phase where it
observes multiple partial episodes where the goal-driven agents behave without
assistance, simulating a situation where previously collected offline data of
the assisted agent behavior is available. The pre-training is conducted in a
self-supervised fashion by predicting the action distributions of the goal-
driven agents. After the partial rollouts of episodes the distribution over
the next action is predicted by using a linear classification layer on top of
the 2-dimensional behavior embedding. The cross-entropy loss is calculated
between the observed one-hot action and the predicted action distribution.
Once the pre-training is complete, the linear classification layer is
discarded and the setting is converted to a similar one as in scenario 1.

Pre-training of the behavior inference part with action predictions clearly
helps the policy function to learn assisting actions as the learning is faster
and there is less variation across different seeds, as is visible by the
purple line in Figure 2.

###  5.4 Scenario 3: Supervised pre-training with agent-specific inference.

In our final scenario the helper agent has again an initial pre-training phase
where it observes multiple partial episodes where goal-driven agents behave
without assistance. Inspired by the centralized training decentralized
execution (CTDE) method [16, 10], the helper agent is now provided with the
information about the agent’s goal in each episode during the pre-training
phase, but not during the execution after the pre-training. The pre-training
is conducted in a similar manner as in scenario 2 by using the linear
classifier on top of the 2-dimensional behavior embedding, and by calculating
the cross-entropy loss between the one-hot goal label and the predicted goal
distribution. In this case, the pre-training has strong supervision, and the
helper agent is able to cluster the goal-driven agents based on their goals
already during the pre-training phase. This allows faster learning as is
visible with the red line in Figure 2.

###  5.5 Summary of results

In order to evaluate the performance of the helper agent in various scenarios,
we made additional measurements that are useful for gaining insight of the
helper agent performance. We measured the mean return over episodes when the
wall was permanently closed, when it was permanently open and the optimal
policy by making rollouts from the query state. For each query state, we made
a rollout both with wall open and wall closed scenarios. By selecting higher
return from these rollouts, we measured the optimal policy. All key results
are summarized in Table 1. The results show, that even when the helper agent
does not have the possibility to observe the behavior of the goal-driven
agent, the performance is improved. The performance is further increased when
the helper agent can observe partial episodes of the goal driven agent
(scenario 1). The self-supervised pre-training (scenario 2) and supervised
pre-training (scenario 3) achieve highest overall performance as they are
closest to the optimal performance. Interestingly, the self-supervised
scenario reach a similar performance as the supervised pre-training, which
suggests that self-supervised method is a potential alternative for assessing
behavior when labels are not available for the pre-training.

Table 1: Helper agents with progressively more capabilities are able to help the goal-driven agents more. Model | Mean return
---|---
Baseline: Wall always open (with penalty for opening the wall) | -5.19
Baseline: Wall always closed | -5.85
Baseline: Helper agent with ground truth goal | -4.48 $\pm$0.03
(oracle) optimal policy | -4.47
Baseline: End-to-end without agent specific inference | -4.82 $\pm$0.03
Scenario 1: End-to-end with agent-specific inference | -4.66 $\pm$0.08
Scenario 2: Self-supervised pre-training with action predictions | -4.59 $\pm$0.04
Scenario 3: Supervised pre-training with goal predictions | -4.59 $\pm$0.03

##  6 Discussion

In this paper, we introduced several scenarios, where a helper agent can
assist goal-driven agents by manipulating the transition dynamics of the
environment. We showed that by using meta-learning, similar capabilities to
inverse reinforcement learning arise spontaneously. Our method is also an
example how previously collected data can be used to enhance the performance
of a helper agent when offline learning methods are not possible. This kind of
situation arises if the previously collected data is from the assisted agent
behavior, not from a helper agent behavior.

The results indicated that by learning to infer the agent’s behavior, the
helper agent’s policy can be significantly improved. Furthermore, the training
can be made faster and more robust by a separate pre-training phase. The pre-
training can be done either in self-supervised manner by predicting the
actions of the assisted agent, or by using additional information about the
agents’ goals that is available during the training but not during the
execution. This kind of scenario is valid when it is of interest to predict a
known feature that explains the behavior, such as the goal, skill level, level
of stochasticity, level of visibility, or level of memory. An interesting
continuation to this work could be extending to modeling more complex human-
like behaviors in scenarios that are closer to real-world human-computer
interactions (for example, web-site navigation).

##  7 Acknowledgements

This work was supported by the Academy of Finland (Flagship programme: Finnish
Center for Artificial Intelligence FCAI, and grants 319264, 292334).

## References

  * [1] Stefano V Albrecht and Peter Stone.  Autonomous agents modelling other agents: A comprehensive survey and open problems.  Artificial Intelligence, 258:66–95, 2018.
  * [2] Radford Alec, Narasimhan Karthik, Salimans Tim, Ilya Sutskever Openai, Andrew M. Dai, Quoc V. Le, Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, Anonymous Emnlp, Alex Graves, Matthew E. Peters, Waleed Ammar, Chandra Bhagavatula, Russell Power, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer.  Improving Language Understanding by Generative Pre-Training.  OpenAI, 2018.
  * [3] Chris L. Baker, Julian Jara-Ettinger, Rebecca Saxe, and Joshua B. Tenenbaum.  Rational quantitative attribution of beliefs, desires and percepts in human mentalizing.  Nature Human Behaviour, 2017.
  * [4] Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, 2019.
  * [5] Christos Dimitrakakis, David C. Parkes, Goran Radanovic, and Paul Tylkin.  Multi-view decision processes: The helper-AI problem.  In Advances in Neural Information Processing Systems, 2017.
  * [6] Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel.  ${RL}^{2}$: Fast reinforcement learning via slow reinforcement learning.  arXiv preprint arXiv:1611.02779, 2016.
  * [7] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In 34th International Conference on Machine Learning, 2017.
  * [8] Aditya Grover, Maruan Ai-Shedivat, Jayesh K. Gupta, Burda Yura, and Harrison Edwards.  Learning policy representations in multiagent systems.  In 35th International Conference on Machine Learning, 2018.
  * [9] Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu.  Reinforcement learning with unsupervised auxiliary tasks.  In 5th International Conference on Learning Representations, 2017\.
  * [10] Landon Kraemer and Bikramjit Banerjee.  Multi-agent reinforcement learning as a rehearsal for decentralized planning.  Neurocomputing, 2016.
  * [11] Christiane Lemke, Marcin Budka, and Bogdan Gabrys.  Metalearning: a survey of trends and technologies.  Artificial Intelligence Review, 2015.
  * [12] Alan M. Leslie, Ori Friedman, and Tim P. German.  Core mechanisms in ’theory of mind’.  Trends in Cognitive Sciences, 2004.
  * [13] Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, et al.  Learning to navigate in complex environments.  arXiv preprint arXiv:1611.03673, 2016.
  * [14] Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel.  A simple neural attentive meta-learner.  arXiv preprint arXiv:1707.03141, 2017.
  * [15] Andrew Ng and Stuart Russell.  Algorithms for inverse reinforcement learning.  Proceedings of the Seventeenth International Conference on Machine Learning, 2000.
  * [16] Frans A. Oliehoek.  Decentralized POMDPs.  In Adaptation, Learning, and Optimization. 2012.
  * [17] Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic.  Learning and transferring mid-level image representations using convolutional neural networks.  In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2014.
  * [18] Sinno Jialin Pan and Qiang Yang.  A survey on transfer learning.  In IEEE Transactions on Knowledge and Data Engineering, 2010.
  * [19] David Premack and Guy Woodruff.  Does the chimpanzee have a theory of mind?  Behavioral and Brain Sciences, 1978.
  * [20] Neil C. Rabinowitz, Frank Perbet, H. Francis Song, Chiyuan Zhang, and Matthew Botvinick.  Machine Theory of mind.  In 35th International Conference on Machine Learning, 2018.
  * [21] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap.  Meta-Learning with Memory-Augmented Neural Networks.  In 33rd International Conference on Machine Learning, 2016.
  * [22] Jurgen Schmidhuber.  Evolutionary Principles in Self-Referential Learning. On Learning now to Learn: The Meta-Meta-Meta…-Hook.  PhD thesis, 1987.
  * [23] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347, 2017.
  * [24] Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick.  Learning to reinforcement learn.  arXiv preprint arXiv:1611.05763, 2016.
  * [25] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson.  How transferable are features in deep neural networks?  In Advances in Neural Information Processing Systems, 2014.
  * [26] Yuting Zhang, Kibok Lee, and Honglak Lee.  Augmenting supervised neural networks with unsupervised objectives for large-scale image classification.  In International conference on machine learning. ICML, 2016.

##  Appendix 0.A Neural Network details

###  0.A.1 helper agent

The helper agent behavior inference part has two convolutional layers, batch
normalization layers, and two fully connected layers. Convolutional layers 1
and 2: 8 filters, 3x3 kernels, stride 1, zero padding to maintain the
dimensions. Followed by ReLU activations and batch normalization. The output
of the convolutional layers of each episode is averaged across the time
dimension to produce the representations of the partial episodes. The
representations are fed through a linear layer, averaged across the episodes
of the agent and fed through another fully connected layer to produce the
2-dimensional behavior embeddings.

The policy head part of the helper agent has a shared policy and value head
backbone with 2 convolutional layers with 16 filters, 3x3 kernels, stride 1
and zero padding. The output of the convolution is followed by a policy head
with Softmax to produce action probabilities and a separate value head to
produce value estimates.

The helper agent is trained using the Adam optimizer with a learning rate of
1e-4. Reward for opening the wall: -1. Reward of every step: -1. Reward of
reaching the goal +1.

Using a batch size of 256 simulations were run for 60 000 PPO training steps
and pre-training was done using 10 000 classification steps. The baseline
results where the wall is always closed, the wall is always open and the
optimal policy were calculated as the mean return over 512 000 rollouts. The
helper agent results were calculated as the mean of the last 51 200 rollouts
during training.

###  0.A.2 Goal-driven agents

The goal-driven agent policy network has two convolutional layers and a fully
connected layer.

Convolutional layer 1: 16 filters, 3x3 kernels, stride 1, ReLU activations,
zero padding to maintain the dimensions. Convolutional layer 2: 32 filters,
3x3 kernels, stride 1, ReLU activations, zero padding. Fully connected layer
with 32 hidden units: ReLU activation followed by a policy head with Softmax
to produce action probabilities and a value head to produce value estimates.

Reward of every step: -1. Reward of reaching the goal +1.


11institutetext:  King’s College London, School of Biomedical Engineering &
Imaging Sciences, London, SE1 7EU, UK 11email<EMAIL_ADDRESS>
22institutetext: Ultromics Ltd., Oxford, OX4 2SU, UK 22email:
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
# Echo from noise: synthetic ultrasound image generation using diffusion
models for real image segmentation

David Stojanovski  This work was supported by the Wellcome/EPSRC Centre for
Medical Engineering [WT203148/Z/16/Z] and by the National Institute for Health
Research (NIHR) Biomedical Research Centre at Guy’s and St Thomas’ NHS
Foundation Trust and King’s College London. The views expressed are those of
the author(s) and not necessarily those of the NHS, the NIHR or the Department
of Health. 11 0000-0002-7912-5249 Uxio Hermida 11 0000-0003-1671-3489 Pablo
Lamata 11 0000-0002-3097-4928 Arian Beqiri 1122 0000-0002-9169-1788 Alberto
Gomez 1122 0000-0002-7897-7589

###### Abstract

We propose a novel pipeline for the generation of synthetic ultrasound images
via Denoising Diffusion Probabilistic Models (DDPMs) guided by cardiac
semantic label maps. We show that these synthetic images can serve as a viable
substitute for real data in the training of deep-learning models for
ultrasound image analysis tasks such as cardiac segmentation. To demonstrate
the effectiveness of this approach, we generated synthetic 2D echocardiograms
and trained a neural network for segmenting the left ventricle and left
atrium. The performance of the network trained on exclusively synthetic images
was evaluated on an unseen dataset of real images and yielded mean Dice scores
of 88.6 $\pm 4.91$ , 91.9 $\pm 4.22$, 85.2 $\pm 4.83$ % for left ventricular
endocardium, epicardium and left atrial segmentation respectively. This
represents a relative increase of $9.2$, $3.3$ and $13.9$ % in Dice scores
compared to the previous state-of-the-art. The proposed pipeline has potential
for application to a wide range of other tasks across various medical imaging
modalities.

###### Keywords:

Diffusion Models Image synthesis Ultrasound

##  1 Introduction

###  1.1 Background and motivation

Echocardiography (echo) is the most widely used method for evaluating the
heart, because it is more cost-effective and safer than other imaging
modalities, while providing high resolution images in real-time. However, a
major drawback of echo is its heavy dependence on operators’ expertise to
obtain high-quality images and associated anatomical and functional
measurements.

Convolutional neural networks (CNNs) have shown great potential for automating
medical image analysis tasks and are capable of accurately learning complex
relevant features from large sets of data. A major challenge in the use of
CNNs for medical imaging tasks is the need for labelling such large sets of
data for model training. Further, CNN accuracy can be limited by the quality
of the labels used during training, particularly in the presence of noise or
other artefacts that can lead to inter-observer errors. Experienced
cardiologists have been shown to have inter-observer errors up to 22% when
labelling common measurements in echo [2].

Previous research has shown the feasibility of generating realistic natural
and medical images [21, 5, 1]. Until recently, the state-of-the-art (SOTA)
results were achieved with Generative Adversarial Networks (GANs) and
CycleGANs. GANs are known to be notoriously difficult to train, due to
training instability as the generator and discriminator are trained
simultaneously, and the loss function can be highly non-convex, making it
challenging to find the global minimum. Additionally, GANs can suffer from
vanishing gradients, which can lead to slow or non-existent convergence. This
often leads to failed training runs and great difficulty in reproducing
results. GANs are also prone to mode collapse, where the generator learns to
produce a limited set of outputs, which can be repeated instead of generating
diverse samples [4]. This can happen when the discriminator is too strong in
comparison to the generator, and rejects any samples that do not match the
training data, forcing the generator to produce only a limited set of outputs.
When using a guide image for synthetic image synthesis (e.g., to generate a
synthetic medical image guided by an anatomical representation), CycleGANs
have been the go-to technique of choice, but typically fail to reproduce the
anatomy under large transformations of the guide images and collapse to
anatomy seen in the training set, as illustrated later in this paper.

Denoising Diffusion Probabilistic models (DDPMs) are more recent generative
models that are far less susceptible to the pitfalls of GAN based methods
[10]. However, limited research has been performed on medical images, with a
few notable examples [14, 9]. No prior research studies have explored their
use in semantically guided image synthesis, or to ultrasound imaging at all.
In this paper, we use DDPMs to generate synthetic echo images and train a
segmentation model.

###  1.2 Related works

The two primary Deep Learning (DL) techniques for generating synthetic images
are GANs and DDPMs.

####  1.2.1 Generative Adversarial Networks:

GANs and the CycleGAN subclass have shown success in generating synthetic
medical images. Examples include Cycle-MedGAN [1] that achieved $\>0.91$
Structural Similarity Index Measure (SSIM) in a Positron Emission Tomography
to Computed Tomography (CT) unsupervised translation task. The SOTA in
segmentation is presented by Gilbert et al. who used a CycleGAN architecture
to generate images for training a network to segment an unseen real cardiac
ultrasound test set. They achieved 79.4, 88.6 and 71.3 % mean Dice scores on
the left ventricle endocardium, left ventricle epicardium and left atrium
respectively [6].

####  1.2.2 Diffusion Models:

Sohl-Dickstein et al. initially proposed DDPMs [18], which have been used
successfully for unsupervised image-to-image translation in both the natural
and medical image domains. Pinaya et al. achieved a Fréchet inception distance
(FID) of $7.8\times 10^{-3}$ for brain Magnetic Resonance Imaging (MRI)
generation versus an FID of $5\times 10^{-4}$ for real brain MRI [14]. Lyu et
al. achieved $>0.85$ SSIM score for their conversion of CT to MRI images [9].

These works relate to the problem of image-to-image domain adaption. Instead,
we use diffusion models to address the issue of limited labelled training data
in echo image segmentation, by using a semantically guided network that
receives an anatomical semantic label map to generate conditional synthetic
echo images which adhere to the anatomy. Our work aims to combine the benefits
of DDPMs with semantically guided medical imaging to synthetically increase
dataset size and enable the creation of out-of-distribution images.

###  1.3 Contributions of this study

This is the first work to utilize DDPMs for generating medical images using
semantic label maps as a source image for conditioning the generated image. To
summarize our contributions, we: 1) Demonstrate that a semantic label map
guided diffusion model can be trained to synthesize cardiac ultrasound images
and matching semantic labels that can be used to train a segmentation model
that then performs to high accuracy on real echo data, and 2) Release the
generated datasets, as well as the code, for public usage.

The code is available at https://github.com/david-stojanovski/echo_from_noise.
The generated diffusion model dataset, as described in Section 2.1 is
available at https://zenodo.org/record/7921055#.ZGYS_9LMLmE.

##  2 Methods

As an overview, our image synthesis pipeline implements the Semantic Diffusion
Model (SDM) proposed by Wang et al. [20], the details of which are described
in Section 2.1. Subsequently, these generated images are used to train and
validate a segmentation model. This model is then tested on an unseen dataset
of real echocardiographic images, as described in Section 2.2.2. An overview
of the pipeline is given in Fig. 1.

Figure 1: Pipeline for generating synthetic ultrasound images from a semantic
diffusion model (SDM), to be used in segmentation training and testing on real
data. SDM Training: The SDM is trained to transform the noise to the realistic
image through an iterative denoising process. SDM inference: The trained SDM
is inferenced on augmented labels to generate corresponding realistic
synthetic ultrasound images. Segmentation: The set of generated synthetic
ultrasound images are used to train a segmentation network. The segmentation
performance is tested on real data. x denotes the semantic label map; $y_{t}$
denotes the noisy image at each time step $t$.

###  2.1 Synthetic ultrasound generation: Semantic Diffusion Model

####  2.1.1 Data:

We used the CAMUS echocardiography dataset [8] which contains 500 patients in
total with semantically segmented left ventricle myocardium, endocardium, and
left atrial surface for 4 chamber and 2 chamber images at both end-diastole
(ED) and end-systole (ES) frames. The official test subset, of 50 patients,
was reserved for testing segmentation networks. We used the remaining 450
patients for training and validating the generative models, by splitting them
into 400 training and 50 validation. Note that only the ED frames were used
for training and validating the two diffusion models (2C and 4C), totaling 400
training + 50 validation frames used to train each model.

An additional label was added to the label maps to describe the ultrasound
sector (see Fig. 1), by applying a simple thresholding to the ultrasound
images.

####  2.1.2 Proposed model:

We made a few notable modifications to the Semantic Diffusion Model (SDM),
including some best practices proposed by Nichol et al. [11].

Firstly we used a cosine noise schedule, instead of a linear schedule, to
reduce the rate at which noise is added, thus increasing the contribution of
the noise in later steps of the forward process. Secondly, the objective
function being optimized is the summation of the predicted noise given an
input label map at each time point and the Kullback–Leibler (KL) divergence
between the estimated distribution and diffusion process posterior. We
modified the weight of the KL divergence to $0.001$ to stabilize the
optimization.

During inference, we removed the classifier-free guidance sampling for
disentanglement, as it was found to add minimal perceptible difference to
generated images. This allowed us to approximately halve the inference time.
Briefly, the predicted noise of the model from a semantic label map is
represented by:

$\hat{\epsilon}_{\theta}(y_{t}|x)=\epsilon_{\theta}(y_{t}|x)+s\cdot(\epsilon_{\theta}(y_{t}|x)-\epsilon_{\theta}(y_{t}|\emptyset))$
(1)

where $\hat{\epsilon}_{\theta}(y_{t}|x)$ is the total-estimated noise in a
ground truth image $y$ at time step $t$, given an input semantic label map
$x$, $s$ is a user-defined guidance scale and
$\epsilon_{\theta}(y_{t}|\emptyset)$ is estimated noise in a ground truth
image given a null input, $\emptyset$.

Two separate models were trained, a model on 4 chamber (4C) end diastolic
frames and a model on 2 chamber (2C) end diastolic frames. Training and
inference was performed using Pytorch 1.13 [13] on 8 $\times$ Nvidia A100
graphics processing units for 50,000 steps using an annealing learning rate
and a batch size of 12.

###  2.2 Segmentation

####  2.2.1 Data:

From the 400+50 CAMUS patient images used to build the diffusion models, we
took the semantic maps (4 per patient: 2C and 4C both ED and ES) to produce
four synthetic datasets: 2CED, 2CES, 4CED and 4CES. For each dataset, we took
the 400+50 semantic maps and applied five random transformations (each being a
combination of random affine and elastic deformation), to produce a total of
2000 (training) + 250 (validation) semantic maps per dataset. Affine
transformation ranges for rotation degrees, translate, scale and shear were:
$(-5,5)^{\circ}$, $(0,0.05)$, $(0.8,1.05)$ and $5^{\circ}$ respectively. The
number of control points and max displacement were $(10,10,4)$ and
$(0,30,30)$. The corresponding echo images were generated by using the
previously trained SDMs (the same SDM was used for ED and ES frames from a
given view). A fifth dataset is built by aggregating the other four.

The exact generated datasets constituency is shown in Fig. 2. The results
given for the segmentation tasks are the values from testing on the official
test split of the CAMUS dataset.

Figure 2: Flow chart of semantic diffusion model dataset design. ED: end
diastolic frames, ES: end systolic frames, 2-ch: echo 2 chamber images, 4-ch:
echo 4 chamber images.

####  2.2.2 Model:

We implemented an 8 layer U-Net model [16], identical to the one in [5] for
fair comparison. Five instances of this model were trained with the 5
aforementioned datasets. Segmentation accuracy was assessed using the 2D Dice
score [17]. The segmentation model was an in-house implementation of the
standard U-Net using MONAI [3], adapted for multiple labels and image
resolution of (256, 256) as input. The network contains 8 layers (L) with
$2^{L}$ channels in each layer and was trained for 300 epochs with the Adam
optimizer and a learning rate of $1\times 10^{-3}$, $\beta_{1}$ of 0.9 and
$\beta_{2}$ of 0.999. These hyperparameters were chosen based on best
practices proposed in literature [7].

###  2.3 Baseline CycleGAN model and data

The CycleGAN network from [5] was used as a benchmark for the SDM ultrasound
image generation. The CycleGAN was trained with 2C and 4C slices from a public
dataset of 1000 synthetic cardiac meshes [15]. Data was divided into a
70/15/15% train/validation/test split and trained for 200 epochs. The training
procedure was implemented as described in [19]. The Dice score comparisons
were made with the originally published results by Gilbert et al. [6].

##  3 Experiments and Results

The final Dice scores on the unseen CAMUS test set of 50 patients are shown in
Table 1. Example images generated with the SDMs in all frames for a single
patient are shown in Fig. 3. We also present extreme augmentations in Fig. 4
to illustrate the robustness of the SDM model to inputs that would be out of
the distribution of the training set. Fig. 5 shows a comparison of our SDM
model against a CycleGAN model for inferencing across a range of
augmentations.

Table 1: Dice scores for the CAMUS test set. $LV_{endo}$, $LV_{epi}$ and $LA$ denote left ventricular endocardium, epicardium and atrium respectively. Rows stating all frames show mean and standard deviation for each label from a network trained on all views. All frames refers to 2CH and 4CH images at both systole and diastole (4 images total). Train/Validation Data | Dice Score (%)
---|---
$LV_{endo}$ | $LV_{epi}$ | LA | Mean
SDM 4 Chamber ED | 91.3$\pm 4.4$ | 94.0$\pm 3.2$ | 82.6$\pm 16.9$ | 89.3$\pm 4.9$
SDM 4 Chamber ES | 87.8$\pm 5.5$ | 92.7$\pm 2.7$ | 88.0$\pm 5.3$ | 89.5$\pm 2.3$
SDM 2 Chamber ED | 88.0$\pm 4.9$ | 89.3$\pm 5.5$ | 77.5$\pm 16.8$ | 85.0$\pm 5.3$
SDM 2 Chamber ES | 87.0$\pm 5.7$ | 92.9$\pm 2.4$ | 87.9$\pm 7.3$ | 89.3$\pm 2.6$
All SDM frames | 88.6$\pm 5.8$ | 91.9$\pm 4.2$ | 85.2$\pm 13.2$ | 88.6$\pm 2.7$
CycleGAN [6] all frames | $79.4\pm 9.9$ | $88.6\pm 5.4$ | $71.3\pm 23.7$ | 79.7$\pm 7.1$
All real CAMUS frames | $87.8\pm 6.8$ | $90.9\pm 5.4$ | $82.6\pm 14.4$ | 87.1$\pm 3.4$

Using the CAMUS pretrained All SDM frames model and performing testing on the
EchoNet-Dynamic Atrial 4 Chamber at End Diastole dataset [12], the model
achieved $87.83\pm 7.21\%$ vs $85.6\pm 7.0\%$ obtained by Gilbert et al. when
training a dataset-specific CycleGAN model.

Qualitatively, our results suggest that our SDMs are able to generate
ultrasound images with superior overall image realism and propensity for
adhering to anatomical input constraints, as well as the ability to generate
images from extreme out-of-distribution semantic label maps. Representative
examples of generated images are shown in Fig. 3, 4 and 5.

Figure 3: Example images generated from our trained SDM networks. A) SDM
synthetic images; B) SDM synthetic images overlaid with input semantic label
map; C) Synthetic image with contour of input semantic label map. ED: End
diastole; ES: End systole.

Figure 4: Example images generated from extreme semantic map distortions. Top
row: Synthetic images; Bottom row: Synthetic images with semantic label map
contours. Anatomical defects: a hand-crafted septal defect (left), and missing
left atrium (right).

An example of our model vs CycleGAN (previous SOTA) is shown in Fig. 5.
Columns 2 and 3 represent examples of CycleGAN’s limited ability to generate
anatomically realistic images, even with a realistic anatomical guiding image.
CycleGAN outputs show atrial walls merging, hallucinated septal wall
thickening, or complete collapse of the ventricle.

Figure 5: Comparison of generated images from the SDM and our CycleGAN
network. Top row: Generated synthetic images, bottom row: Overlay of the
corresponding label map. Red dashed lines and arrows point to anatomical
features missed or wrongly predicted by the CycleGAN model. Green dashed lines
and arrows point to anatomical features correctly predicted by the SDM model.

##  4 Discussion and Conclusions

Our anatomy-guided diffusion models generate realistic echo images that,
compared to SOTA methods, adhere better to the anatomical constraints of a
given label map, even when the prescribed anatomy is very different from the
training set. Indeed, While CycleGAN can generate realistic ultrasound images,
we observed that their ability to reproduce the anatomy under large
deformation of the guide images is limited. Further, the anatomically accurate
synthetic data generated with our model significantly improves the performance
of the segmentation model on real images, showing the potential to address
challenges involving rare medical conditions, data privacy, or limited data
availability.

A segmentation network trained on the SDM-generated synthetic data
significantly outperformed SOTA methods, and even training on the original
real data, in segmentation of real 2 and 4 chamber ultrasound images. The LV
endocardium, epicardium, and atrium were segmented with high accuracy
(88.6$\pm 5.8$, 91.9$\pm 4.2$, 85.2$\pm 13.2$ % Dice score respectively).
These results represent an 11.6, 3.7 and 19.5% improvement in relation to
previous SOTA results. Moreover, our results showed reduced standard deviation
for all labels, suggesting our model yields realistic images more
consistently, leading to less variation in segmentation performance. These
results also highlight the adaptability of SDMs for generating new data.

Future work will include addressing variation across devices and clinical
centers and investigating temporal aspects of synthetic data generation. The
results presented in this paper show promise for synthetic data generation
that can be used to train deep neural networks to high performance, addressing
a crucial problem in medical imaging such as the availability of expert-
labeled data.

## References

  * [1] Armanious, K., Jiang, C., Abdulatif, S., Küstner, T., Gatidis, S., Yang, B.: Unsupervised medical image translation using cycle-medgan (3 2019), http://arxiv.org/abs/1903.03374http://dx.doi.org/10.23919/EUSIPCO.2019.8902799
  * [2] Armstrong, A.C., Ricketts, E.P., Cox, C., Adler, P., Arynchyn, A., Liu, K., Stengel, E., Sidney, S., Lewis, C.E., Schreiner, P.J., Shikany, J.M., Keck, K., Merlo, J., Gidding, S.S., Lima, J.A.: Quality control and reproducibility in m-mode, two-dimensional, and speckle tracking echocardiography acquisition and analysis: The cardia study, year 25 examination experience. Echocardiography 32, 1233–1240 (8 2015). https://doi.org/10.1111/echo.12832
  * [3] Consortium, T.M.: Project monai (Dec 2020), https://doi.org/10.5281/zenodo.4323059
  * [4] Feng, R., Lin, Z., Zhu, J., Zhao, D., Zhou, J., Zha, Z.J.: Uncertainty principles of encoding gans. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 139, pp. 3240–3251. PMLR (18–24 Jul 2021), https://proceedings.mlr.press/v139/feng21c.html
  * [5] Gilbert, A., Marciniak, M., Rodero, C., Lamata, P., Samset, E., McLeod, K.: Generating synthetic labeled data from existing anatomical models: An example with echocardiography segmentation. IEEE Transactions on Medical Imaging 40, 2783–2794 (10 2021). https://doi.org/10.1109/TMI.2021.3051806
  * [6] Gilbert, A., Marciniak, M., Rodero, C., Lamata, P., Samset, E., McLeod, K.: Supplementary materials for generating synthetic labeled data from existing anatomical models: An example with echocardiography segmentation, https://github.com/adgilbert/data-generation/blob/main/SupplementaryMaterial.pdf
  * [7] Kingma, D.P., Ba, J.L.: Adam: A method for stochastic optimization. 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings pp. 1–15 (2015)
  * [8] Leclerc, S., Smistad, E., Pedrosa, J., Østvik, A., Cervenansky, F., Espinosa, F., Espeland, T., Berg, E.A.R., Jodoin, P.M., Grenier, T., Lartizien, C., D’hooge, J., Lovstakken, L., Bernard, O.: Deep learning for segmentation using an open large-scale dataset in 2d echocardiography. IEEE Transactions on Medical Imaging 38(9), 2198–2210 (2019)
  * [9] Lyu, Q., Wang, G.: Conversion between ct and mri images using diffusion and score-matching models (9 2022), http://arxiv.org/abs/2209.12104
  * [10] Müller-Franzes, G., Niehues, J.M., Khader, F., Arasteh, S.T., Haarburger, C., Kuhl, C., Wang, T., Han, T., Nebelung, S., Kather, J.N., Truhn, D.: Diffusion probabilistic models beat gans on medical images (2022), https://arxiv.org/abs/2212.07501
  * [11] Nichol, A., Dhariwal, P.: Improved denoising diffusion probabilistic models (2 2021), http://arxiv.org/abs/2102.09672
  * [12] Ouyang, D., He, B., Ghorbani, A., Yuan, N., Ebinger, J., Langlotz, C.P., Heidenreich, P.A., Harrington, R.A., Liang, D.H., Ashley, E.A., et al.: Video-based ai for beat-to-beat assessment of cardiac function. Nature 580(7802), 252–256 (2020). https://doi.org/10.1038/s41586-020-2145-8
  * [13] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems 32(NeurIPS) (2019)
  * [14] Pinaya, W.H.L., Tudosiu, P.D., Dafflon, J., da Costa, P.F., Fernandez, V., Nachev, P., Ourselin, S., Cardoso, M.J.: Brain imaging generation with latent diffusion models (9 2022), http://arxiv.org/abs/2209.07162
  * [15] Rodero, C., Strocchi, M., Marciniak, M., Longobardi, S., Whitaker, J., O’Neill, M.D., Gillette, K., Augustin, C., Plank, G., Vigmond, E.J., Lamata, P., Niederer, S.A.: Linking statistical shape models and simulated function in the healthy adult human heart. PLoS Computational Biology 17(4), 1–28 (2021), http://dx.doi.org/10.1371/journal.pcbi.1008851
  * [16] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation (5 2015), http://arxiv.org/abs/1505.04597
  * [17] Skrifter, B., ind, B.V., Thorvald, B.S., København, R.: Det kongelige danske videnskabernes selskab a method of e st a b l ish in g groups of equal amplitude in plant sociology b a se d on sim ilarity of s p e c ie s content and its application to analyses of the vegetation on danish commons
  * [18] Sohl-Dickstein, J., Weiss, E.A., Maheswaranathan, N., Ganguli, S., Edu, S.: Deep unsupervised learning using nonequilibrium thermodynamics
  * [19] Stojanovski, D., Hermida, U., Muffoletto, M., Lamata, P., Beqiri, A., Gomez, A.: Efficient pix2vox++ for 3d cardiac reconstruction from 2d echo views. In: Aylward, S., Noble, J.A., Hu, Y., Lee, S.L., Baum, Z., Min, Z. (eds.) Simplifying Medical Ultrasound. pp. 86–95. Springer International Publishing, Cham (2022)
  * [20] Wang, W., Bao, J., Zhou, W., Chen, D., Chen, D., Yuan, L., Li, H.: Semantic image synthesis via diffusion models (6 2022), http://arxiv.org/abs/2207.00050
  * [21] Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks (3 2017), http://arxiv.org/abs/1703.10593