# GOMA: Proactive Embodied Cooperative Communication via Goal-Oriented Mental
Alignment

Lance Ying1,2, Kunal Jha3, Shivam Aarya4, Joshua B. Tenenbaum2, Antonio
Torralba2, Tianmin Shu4 1Harvard University, Cambridge, MA 02138, USA
<EMAIL_ADDRESS>Institute of Technology, Cambridge,
MA 01239, USA<EMAIL_ADDRESS><EMAIL_ADDRESS>College, Hanover,
NH 03755, USA<EMAIL_ADDRESS>Hopkins University,
Baltimore, MD 21218, USA {saarya1<EMAIL_ADDRESS>
###### Abstract

Verbal communication plays a crucial role in human cooperation, particularly
when the partners only have incomplete information about the task,
environment, and each other’s mental state. In this paper, we propose a novel
cooperative communication framework, Goal-Oriented Mental Alignment (GOMA).
GOMA formulates verbal communication as a planning problem that minimizes the
misalignment between the parts of agents’ mental states that are relevant to
the goals. This approach enables an embodied assistant to reason about when
and how to proactively initialize communication with humans verbally using
natural language to help achieve better cooperation. We evaluate our approach
against strong baselines in two challenging environments, Overcooked (a
multiplayer game) and VirtualHome (a household simulator). Our experimental
results demonstrate that large language models struggle with generating
meaningful communication that is grounded in the social and physical context.
In contrast, our approach can successfully generate concise verbal
communication for the embodied assistant to effectively boost the performance
of the cooperation as well as human users’ perception of the assistant.

##  I Introduction

Rich verbal communication naturally emerges from human cooperation when people
only have partial information about the environments and/or about each other’s
mental states [1]. It serves as a complementary source of information, in
addition to the visual inputs, to help achieve better cooperation by aligning
each other’s mental states (including goals, beliefs, and eventually plans [2,
3, 4]). Recent advances in large language models (LLM) and machine Theory of
Mind (ToM) have sparked interest in building cooperative robots that can not
only physically cooperate with humans but also verbally communicate with
humans using natural language [5, 6]. However, it remains challenging to
enable robots to actively initiate verbal communication that is both concise
(only communicate when necessary) and consistent with the physical environment
and the social context (e.g., what humans want to do, believe, know, and need
to know).

Figure 1: Illustration of cooperation with a shared mind or misaligned minds
and communication optimized via goal-oriented mental alignment. (a) When human
and robot minds are perfectly signed (i.e., a shared mind), they share the
same belief of the physical state and the same goal, which leads to the same
joint plan shared by both agents. This is the ideal condition for reaching
optimal cooperation. (b However, in real-world cooperation, human and robot
minds are typically unaligned, leading to two different (and often
conflicting) joint plans in their minds. (c) To achieve a shared joint plan
that optimizes cooperation, we optimize verbal communication initiated by the
robot to actively align the joint plans in both agents’ minds.

A long history of research in psychology has shown that proactive verbal
communication serves to align the mental states of agents [7]. Imagine you are
going to get some groceries for your mom. As you put on your shoes and walk
towards the door, your mom gets out of the kitchen and says “It’s going to
rain, get your umbrella, and don’t forget about the avocados.” In this
scenario, your mom decides to communicate with you because she is uncertain
whether you have the same beliefs regarding the weather forecast and the
required grocery items as you walk out the door.

When cooperating with one another, each agent not only needs to plan for
itself but also has to imagine the plans of its partners. Such planning
process is termed as joint planning [8, 9]. To achieve joint planning, prior
works typically assumed that both agents have full observability and complete
knowledge about the task. In other words, they have a shared mind, based on
which they can derive the same joint plan (Fig. 1(a)). However, in real-world
embodied cooperation, robot assistants only have partial observations and
often do not know the true human goals (Fig. 1(b)).

The goal of cooperative communication is then to reach a shared mind (two
agents’ are perfectly aligned) so that the resulting joint plans in both
agents’ minds are the same (Fig. 1(c)). Once we reach such mental alignment
condition, both agents know exactly what each other plans to do, and therefore
achieve optimal cooperation. However, an agent belief can be about any part of
a state. If the state is high dimensional (such as the state in a real-world
home), it is extremely difficult to make sure two beliefs are the same. Our
key insight is that we only need to align the part of the belief that is
relevant to reaching the goal.

Following this insight, we propose a novel cooperative communication
framework, Goal-Oriented Mental Alignment (GOMA). In this framework, we aim to
generate optimal communication in the belief space. That is, verbal
communication, by exchanging information, can help reshape agents’ beliefs. In
particular, GOMA first seeks to detect misalignment in agents’ goal-relevant
beliefs via divergence between the joint plans based on an agent’s own belief
and a simulated hypothetical shared mind after acquiring additional knowledge
from another agent via hypothetical communication. We then optimize the
communication using a proxy reward derived from the divergence between the
plans. The resulting communication can then help us minimize the difference
between the joint plan in each agent’s mind and the true joint plan given a
true shared mind. By doing so, we can optimize the cooperation.

We evaluate GOMA in two popular human-AI cooperation domains, Overcooked and
VirtualHome. Our experimental results with a simulated human agent and real
human participants show that our GOMA outperforms strong baselines (including
a recent LLM-based baseline). The GOMA-enabled assistant also receives higher
subjective ratings from human participants.

In sum, our contributions include (1) a novel embodied cooperative
communication framework – GOMA, (2) extensive evaluation of strong baselines
and GOMA in two challenging domains, and (3) a human user study that evaluates
the task performance of AI assistants and humans’ perception of them.

##  II Related Work

###  II-A Communication in Collaboration

Human communication is grounded in cooperative intentions. [7] argues that
language communication is a joint activity that attempts to achieve mutual
understanding. [1] proposes three communicative motives: requesting help or
information, informing the other agents, and sharing feelings or attitudes.
These communicative motives help to align the mental states of the agents.
Through verbal communication, agents can assess others’ goals, knowledge,
emotions, and beliefs, which they can then use to plan for the next actions.

However, verbal communication can also be costly, as it demands cognitive
resources and distracts agents when performing actions [10]. Prior works on
multi-agent teaming have formalized communication costs in collaborative
settings [10, 11, 12], showing that excessive communication can degrade the
performance of the team. Therefore, when designing communication policies, the
AI assistant needs to communicate useful, concise, and relevant information
yet not too frequently.

###  II-B Collaborative and Communicative AI Agent

Communication between humans and robots has also been extensively studied.
Most existing literature has focused on one-directional communication where
the human instructs the robot [13, 14]. Some recent studies have proposed bi-
directional communication. For example, [15] proposes a bi-directional human-
robot collaborative communication framework that allows the robot to
communicate decisions with explanations from human feedback. [12] introduced
CommPlann, a bi-directional communication framework that allows the robot to
ask for human’s intent, share the robot’s intent, and give commands to humans.
There have been recent works that use LLMs as a communication module in bi-
directional human-robot communication, (e.g., [5, 16, 17, 2, 18]). While these
recent LLM-based agents can achieve certain success, the communication
generated by LLMs is often redundant and/or not grounded in agents’ mental
states, actions, and plans.

In addition, most human-robot communication frameworks, such as [12, 19, 20],
assume full agent observability. The resulting communication is thus only
restricted to informing and inquiring about goals and plans. Our work attempts
to extend to scenarios where both agents have partial observability of the
environment and allow the robot to communicate to resolve partial knowledge
and false beliefs about the environment state. This requires agents to model
and reason about each other’s mental states recursively (e.g., the robot
thinks the human thinks the glass is in the fridge, but it knows that the
glass is actually in the cabinet), which remains a challenge for LLMs today
[21]. As a result, such cooperative communication capacity remains an open
research question in embodied cooperation.

###  II-C Theory of Mind for Cooperative Robot Planning

There have been many studies on inferring an agent’s goals and beliefs (e.g.,
[22, 23, 24, 25, 26, 2, 27]), commonly referred to as the Theory of Mind
reasoning, to better coordinate with humans in collaborative tasks. Previous
studies have leveraged explicit mental reasoning to improve cooperative robot
planning. This includes generating more expressive or explainable plans to
improve humans’ understanding of robots’ plans [28, 29, 30, 31, 4, 32] or
better understanding of humans’ cooperative actions [3], all via reasoning
about humans’ mental models of the robot. There have also been works on
developing a shared joint planner in two agents’ minds to reach optimal
coordination by reasoning about one agent’s own plan and the other agent’s
plan jointly. However, existing works do not allow verbal communication
between humans and robots in addition to action planning. Our work aims to
fill this gap by jointly planning for actions that change the physical state
but verbal communication that changes the mental states of humans and robots.

##  III Problem Formulation

In this work, we consider two agents, a human user and a robot assistant. To
successfully communicate and cooperate, the two agents must infer each other’s
mind. We adopt the Interactive Partially Observable Markov Decision Process
(I-POMDP) [33, 34] to formulate the mental reasoning between the human and the
robot.

###  III-A Background: I-POMDP

I-POMDP is a framework that enables an agent to recursively model other
agents, which captures complex social interactions between agents. Here, we
consider the interactions between two agents, $i$ and $j$, in which agent $i$
infers agent $j$’s mental state recursively. In an I-POMDP, there are states
$s^{t}$; agents’ observations, $o_{i}^{t}$ and $o_{j}^{t}$, sampled from their
conditional observation probabilities, $O_{i}(o^{t}_{i}|s^{t})$ and
$O_{j}(o^{t}_{j}|s^{t})$; and agents’ actions $a_{i}^{t}$ and $a_{j}^{t}$.
Agents have their beliefs, $b_{i}^{t}$ and $b_{j}^{t}$, and goals,
$\theta_{i}$ and $\theta_{j}$. To model the recursive mental reasoning, we
define interactive states for the agents, i.e., $is_{i,\ell}$ and
$is_{j,\ell}$, at level-$\ell$. From agent $i$’s perspective, we define its
interactive state at each level as

  * •

Level $0$: $is_{i,0}=s$

  * •

Level $1$: $is_{i,1}=(s,b_{j,0},\theta_{j})$

  * •

$\cdots$

  * •

Level $\ell$: $is_{i,\ell}=(s,b_{j,\ell-1},\theta_{j})$

The level-$\ell$ inference for agent $i$ is to infer the belief
$b_{i,\ell}^{t}=p(is_{i,\ell}^{t}\lvert o_{i}^{1:t},a_{i}^{1:t-1})$. Since the
level-$\ell$ agent $i$’s interactive state,
$is_{i,\ell}^{t}=(s^{t},b_{j,\ell-1}^{t},\theta_{j})$, includes $j$’s belief
at level $\ell-1$ ($b_{j,\ell-1}^{t}$), the inference at level $\ell$ depends
on inference at level $\ell-1$ which depends on inference at level $\ell-2$,
and so on. This recursive inference terminates at level $0$. That is, the
belief at level-0 is only about the physical state, $b_{i,0}=p(s^{t}\lvert
o_{i}^{1:t},a_{i}^{1:t-1})$. This becomes a standard POMDP [35] which does not
model other agents.

###  III-B Two-level Reasoning for Embodied Cooperative Cooperation

Theoretically, the level of agents’ reasoning about other agents’ minds can go
to infinity (e.g. robot thinks human thinks robot thinks…) yet we cap the
depth at two in our model, which is in line with most empirical evidence
suggesting that humans rarely engage in greater than 2 levels of recursive
Theory of Mind reasoning [36]. Therefore, we adopt a two-level I-POMDP for
modeling the mental reasoning between a human user and a robot assistant in
embodied cooperation. In particular, we define the mind of each agent as the
belief of the level-1 interactive state of the agent. For the human user’s
mind, we have $m_{H}=b(is_{H,1})=\\{b_{H,0},b(b_{R,0}),b(g_{R})\\}$, where
$b_{R,0}$ is the robot’s interactive state at level 0, i.e., its belief about
the physical state; and $g_{R}$ is the robot’s goal. Similarly, for the robot
assistant, we define its mind as
$m_{R}=b(is_{R,1})=\\{b_{R,0},b(b_{H,0}),b(g_{H})\\}$, where $b_{H,0}$ is the
human’s belief about the physical state, and $g_{H}$ is the robot’s goal.
Intuitively, each mind models the agent’s belief about (1) the physical state,
(2) another agent’s belief about the physical state, and (3) the goal of
another agent. Due to the cooperative nature of our problem setting, we
further constrain the goal inference to be either one of the following two
conditions:

  * •

Condition 1: Both agents share a known common goal;

  * •

Condition 2: The robot’s goal is the human goal inferred by the robot, and the
human user knows that the robot is trying to help with the inferred human
goal.

Condition 1 models human-robot teaming, in which the human and robot agents
are teammates who work on the same task assigned to them a priori. Condition 2
models robot assistance, in which the human’s true goal is unknown to the
robot a priori, thus the robot must infer the human’s goal and provide
assistance. In both cases, agents only have partial observability of the
physical state, and thus they have to infer both the physical state and each
other’s belief about the physical state. It is worth noting that our
formulation departs from most previous assistance-game setups, which either
assume that the agents have full observability or that they share a known
goal. As in collaborative tasks, agents often do not have perfect knowledge of
the environment and thus need to represent other agents’ beliefs differently
from theirs and communicate and coordinate their actions, our formulation is
more aligned with real-world embodied cooperation.

##  IV Goal-Oriented Mental Alignment

As Fig. 1 illustrates, when there is a shared mind, two agents will share the
same joint plan. In our Goal-oriented Mental Alignment (GOMA) framework, we
formulate communication optimization as the convergence of the current joint
plan and the joint plan given a shared mind achieved by exchanging information
through verbal communication. In particular, we consider two types of
communication – sharing information and requesting information. These are two
dominant types of verbal communication in human cooperation [1]. We
hypothesize that these are also two types of communication that a robot
assistant can proactively initiate to achieve joint plan alignment. To reason
whether to communicate and what to communicate, we define a proxy reward for
minimizing the divergence between plans before and after one type of
communication. We summarize GOMA in Algorithm 1, which works with any off-the-
shelf action planner. We introduce key components of the algorithm in the rest
of the section.

Algorithm 1 GOMA

1: Input: Planner(), $T_{\text{max}}$

2: Initialization: $b(g_{H})$, $b_{R,0}(s^{0})$, particles of sampled human
beliefs: $\\{b_{H,0}^{(l)}(s^{0})\\}_{l=1}^{L}$

3: $t\leftarrow 1$, $u_{R}^{0}=\text{None}$

4: repeat

5: Observe $o_{R}^{t}$ and receive human message $u_{H}^{t-1}$

6: Update level-0 belief: $b_{R,0}(s^{t})$ based on both $o_{R}^{t}$ and
$u_{H}^{t-1}$

7: Robot knowledge: $K_{R}^{t}=K_{R}(b_{R,0}(s^{t}))$

8: Update human goal inference:

9: $b(g_{H})\propto P(a_{H}^{t-1}|g_{H})P(u_{H}^{t-1}|g_{H})b(g_{H}),\forall
g_{H}\in\mathcal{G}$

10: for all $l=1,\cdots,L$ do

11: Sample a human goal based on the goal inference: $\hat{g}^{(l)}_{H}\sim
b(g_{H})$

12: Set the robot goal as the inferred human goal:
$g_{R}^{(l)}\leftarrow\hat{g}^{(l)}_{H}$

13: Sample an environment state $s^{t}\sim b_{R,0}(s^{t})$

14: Sample inferred human observations $\hat{o}_{H}^{t}\sim
O_{H}(\hat{o}_{H}^{t}|s^{t})$

15: Update $b_{H,0}^{(l)}(s^{t})$ based on both $\hat{o}_{H}^{t}$ and
$u_{R}^{t-1}$

16: Human plan given the inferred human belief:

17:
$\pi_{H}(a_{H}^{t}|b_{H,0},\hat{g}^{(l)}_{H})\leftarrow\textbf{Planner}(b_{H,0},\hat{g}^{(l)}_{H})$

18: Human plans given the shared minds augmented by different sub-states in
robot knowledge:

19:
$\\{\pi_{H}(a_{H}^{t}|b^{+s_{n}}_{H,0},\hat{g}^{(l)}_{H})\leftarrow\textbf{Planner}(b^{+s_{n}}_{H,0},\hat{g}^{(l)}_{H})$;
$\forall s_{n}^{t}\in K_{R}^{t}\\}$

20: Robot plan given the robot belief:

21:
$\pi_{R}(a_{R}^{t}|b_{R,0},g_{R}^{(l)})\leftarrow\textbf{Planner}(b_{R,0},g_{R}^{(l)})$

22: Robot plans given the shared minds augmented by different sub-states in
human knowledge:

23:
$\\{\pi_{R}(a_{R}^{t}|b^{+s_{n}}_{R,0},g_{R}^{(l)})\leftarrow\textbf{Planner}(b^{+s_{n}}_{R,0},g_{R}^{(l)})$;
$\forall s_{n}^{t}\in K(b_{H,0}^{(l)}(s^{t}))\\}$

24: end for

25:
$m_{R}\leftarrow(b_{R,0}(s^{t}),\\{b_{H,0}^{(l)}(s^{t})\\}_{l=1}^{L},b(g_{H}))$

26: All possible human knowledge:
$\hat{K}_{H}^{t}=\cup_{l=1}^{L}K(b_{H,0}^{(l)}(s^{t}))$

27: Construct the utterance space $U$ based on the robot knowledge $K_{R}^{t}$
and all possible kuman knowledge $\hat{K}_{H}^{t}$

28: Compute $R(u,M_{R})$, $\forall u\in U$ using the plans generated above
based on Eq. (IV-D-4)

29: Select robot utterance based on the proxy reward:

30: $u_{R}^{t}=\operatorname*{arg\,max}_{u\in U}R(u,m_{R})$

31: Select robot action based on the average plan:

32:
$a_{R}^{t}=\operatorname*{arg\,max}_{a_{R}\in\mathcal{A_{R}}}\sum_{l=1}^{L}\pi_{R}(a_{R}|b_{R,0},g^{(l)}_{R})/L$

33: Execute the robot action $a_{R}^{t}$ and send the robot utterance
$u_{R}^{t}$

34: $t\leftarrow t+1$

35: until $t=T_{\text{max}}$ or the true goal has not been reached

###  IV-A Gaol Inference and Joint Planning for the Robot

Unless the human goal is given to the robot a priori (i.e., condition 1
defined in Section III-B), the robot must infer the human goal. We adopt the
approach introduced by [2], which leverages an LLM to conduct goal inference
based on the observed human actions and messages (Line 8-9 in Algorithm 1). We
then sample the possible goals of humans

The joint plan for the robot includes two components. First, the robot’s
policy given its goal and its belief, i.e., $\pi_{R}(a_{R}|b_{R,0},g_{R})$.
Second, the expected human’s policy inferred by the robot, i.e.,
$\mathrm{E}_{b(b_{H,0}),b(g_{H})}[\pi_{H}(a_{H}|b_{H,0},g_{H})]$. In practice,
we can estimate this expectation via sampling particles of possible human
beliefs (i.e., $\\{b_{H,0}^{(l)}(s^{t})\\}_{l=1}^{L}$ in Algorithm 1) and
possible goals (Line 11 in Algorithm 1).

###  IV-B Agent Knowledge From Level-0 Belief

Recall that the level-0 belief of an agent $b_{i,0}$ represents the agent’s
belief of the physical state $s$. If we partition the state $s$ into multiple
sub-states such as states of all objects in the environment, then we can
evaluate the uncertainty in the belief of each sub-states. We define the sub-
states that have certain belief distributions as knowledge of an agent.
Formally, let us denote a state partition as $s=\\{s_{n}\\}_{n=1}^{N}$ with N
sub-states and $b_{i,0}(s_{n})$ as the level-0 belief of the sub-state
$s_{i}n$. For instance, if $s_{n}$ is object $n$’s state, then
$b_{i,0}(s_{n})$ is the belief of the object $i$’ state. Consequently, we
define the knowledge of agent $i$ as

$\displaystyle K_{i}$ $\displaystyle=K(b_{i,0})$
$\displaystyle=\\{b_{i,0}(s_{n}):\mathcal{H}(b_{i,0}(s_{n}))<\mathcal{H}_{\text{max}},n=1,\cdots,N\\},$
(1)

where $\mathcal{H}$ is the entropy of a belief distribution and
$\mathcal{H}_{\text{max}}$ is maximum entropy that is considered to be
certain. In the example of object states as sub-states, knowledge consists of
objects over which the agent has beliefs with high certainty.

###  IV-C Shared Mind Augmented by An Agent’s Knowledge

An agent $i$ can imagine a shared mind after acquiring knowledge about a sub-
state, $b_{j,0}(s_{n})\in K_{i}$, from another agent $j$ via verbal
communication, as both agents would share this knowledge after the
communication. We define this as the belief merge operation
$b^{+s_{n}}_{i,0}=\text{Merge}(b_{i,0},b_{j,0}(s_{n}))$. Specifically, this
merge operation will set the belief of sub-state $n$ of agent $i$ to that of
agent $j$, i.e., $b^{+s_{n}}_{i,0}(s_{n})=b_{j,0}(s_{n})$.

###  IV-D Divergence Between Plans as Proxy Reward

It is hard to directly estimate the effect of an utterance on the overall task
performance. To directly reason what knowledge is critical for aligning the
joint plans between agents, we define a proxy reward for communicating about
the knowledge of an agent’s knowledge. Since the goal of this work is to
generate proactive communication initiated by the robot, we model the proxy
reward from the perspective of the robot.

We first define the reward of sharing the robot’s knowledge of sub-state
$s_{n}$ with the human user as follows:

$\displaystyle R(\text{share }s_{n},M_{R})=$
$\displaystyle\text{KL}\left(\mathrm{E}[\pi_{H}(a_{H}|b^{+s_{n}}_{H,0},g_{H})]||\mathrm{E}[\pi_{H}(a_{H}|b_{H,0},g_{H})]\right)-C,$
(2)

where $b^{+s_{n}}_{H,0}=\text{Merge}(b_{H,0},b_{R,0}(s_{n}))$ and $C$ is the
cost for communication at a time step.

We then define the reward of requesting possible human knowledge of sub-state
$s_{n}$ to inform the robot’s plan:

$\displaystyle R(\text{request }s_{n},M_{R})=$
$\displaystyle\text{KL}\left(\pi_{R}(a_{R}|b^{+s_{n}}_{R,0},g_{R})||\pi_{R}(a_{R}|b_{R,0},g_{R})\right)-C,$
(3)

where $b^{+s_{n}}_{R,0}=\text{Merge}(b_{R,0},b_{H,0}(s_{n}))$.

The plans used to compute the KL-divergence for the proxy rewards can be
generated by running an off-the-shelf planner given the corresponding beliefs
and goals (Line 16-23 in Algorithm 1).

We also define the reward for not communicating at a step as follows:

$R(\text{None},M_{R})=0.$ (4)

###  IV-E Communication Optimization

Given the proxy rewards defined above, we can then choose whether and what to
communicate based on the robot’s mind at each step (Line 27-30 in Algorithm
1). In particular, the utterance space is
$U=\\{\text{None}\\}\cup\\{\text{share }s_{n};s_{n}\in
K_{R}\\}\cup\\{\text{request }s_{n};s\in\hat{K}_{H}\\}$, where $\hat{K}_{H}$
is the inferred human knowledge estimated from the human belief particles:
$\hat{K}_{H}=\cup_{l=1}^{L}K(b_{H,0}^{(l)})$. We select the best robot
utterance at step $t$ as follows:

$u_{R}^{t}=\operatorname*{arg\,max}_{u\in U}R(u,M_{R}).$ (5)

We can further generate a natural language message based on the utterance
$u_{R}^{t}$ to enable communication with real humans. This can be achieved by
using GPT-4 [37] to translate $u_{R}^{t}$ to natural language through few-show
prompting.

###  IV-F Multimodal Mental Update

At each step, the robot will update its mind based on both its observation
$o^{t}_{R}$ and the messages it sends and receives. In particular, we extract
human knowledge $b_{H,0}(s_{n})$ from the human message $u_{h}^{t}$ via GPT-4
and use it to update the robot’s level-0 belief $b_{R,0}$ jointly with
$o^{t}_{R}$ (Line 6 in Algorithm 1) operation. For instance, if the human
informs the robot of the location of an object, we can update the robot’s
level-0 belief with the knowledge of the object’s location. Additionally, if
the robot shares knowledge $b_{R,0}(s_{n})$ in its utterance, then the robot
can assume that the human’s level-0 belief will also be updated accordingly.
Thus, in robot mind $M_{R}$, we can update $b(b_{H,0})$ using both the shared
robot knowledge and the human observation (Line 15 in Algorithm 1). Note that
we can sample possible human observations based on the state inferred by the
robot’s level-0 belief (Line 13-14 in Algorithm 1). All beliefs are
initialized with a uniform distribution (Line 2 in Algorithm 1).

##  V Experiments

We evaluate our model in two human-AI domains Overcooked and VirtualHome.
These two domains cover two distinct alignment objectives. In both domains,
there are two agents – a human user and an embodied AI assistant. In
Overcooked, the agent’s goal is to align their plans temporally so that
certain joint actions can be performed at similar time steps, whereas in
VirtualHome, the agents align their beliefs about the location of the objects
they try to collect. We describe each in detail below.

Recipe Name |  Ingredient List
---|---
Burger |  Cooked(Patty), Cooked(Potato), Chopped(Lettuce), Chopped(Tomato)
Pasta |  Cooked(Spaghetti), Cooked(Mushroom), Cooked(Cream), Chopped(Basil)
Ramen |  Cooked(Noodle), Cooked(Mushroom), Cooked(Egg), Chopped(Scallion)
Steak & Fries |  Cooked(Beef), Cooked(Potato), Chopped(Parsley)
TABLE I: Overcooked recipe specifications. Figure 2: Example Overcooked
environment. In each environment, there are two rooms. The two agents are
always in different rooms. An agent cannot observe the other room. Thus it has
to rely on verbal communication to infer the states of the objects in the
other room.

###  V-A Overcooked

Overcooked is a popular multiagent game where agents need to collaborate to
prepare and cook ingredients, which is also widely used for evaluating human-
AI cooperation (e.g., [38, 9]). In the original game, agents have full
observability. In this study, we extended the Overcooked simulator from [9] by
assuming partial observability where each agent cannot observe the other room
as shown in Fig. 2. At each step, the AI assistant may share its progress on
the task or ask about the human’s progress.

The goal of the collaborating agents is to complete the dishes in the shortest
amount of time. To simulate more realistic cooking scenarios, we augment the
existing simulator with dynamics that cooked ingredients will gradually cool
down. If cooked ingredients are not at the ideal temperature when the dish is
served, the team will receive a penalty. This requires both agents to
coordinate better to avoid misalignment in their plans for cooking the
ingredients. For instance, one agent cannot finish making the burger too early
if the other agent has not started cooking the French fries. Therefore, the
agents need to coordinate and align their plans such that they finish cooking
at the same time. The agents can align their plan by choosing to wait for the
other agents (e.g. I will start cooking A as soon as the other agent finishes
B). There are four recipes in our experiment (Table I): Burger, Spaghetti,
Ramen, and Steak, each in a unique room layout. We simulate a human agent
using the planner in [9], which does not proactively communicate with the AI
Assistant. Each recipe is run 10 times with different seeds and we report the
aggregate results.

Baselines. We evaluate three baselines: Single-agent, No-Communication (No-
Comm), and Heuristic-based Communication (Heur-Comm). In Single-Agent, the
human completes all the tasks alone. In the No-Comm baseline, no messages are
exchanged. In the Heur-Comm baseline, the AI Assistant follows a simple
heuristic that shares updates every time a sub-goal has been completed and
periodically asks for the human’s progress. The action planner in all methods
including GOMA is the same as the planner in [9].

Metrics. We use two performance metrics: speedup and total plan costs. Speedup
is calculated by comparing the plan length in each team condition, where the
human is working with one of the four collaborative AI models, to the single
agent baseline, i.e. $Speedup=L_{single}/L_{team}-1$.

Total plan cost is the sum of all action and communication costs with
penalties applied for sub-optimal dish states due to time lapse between the
completion of a hot sub-task (e.g. cooked noodle) and the end of the trial,
i.e. $TotalCost=L+U+\sum_{i\in hot\\_items}{\Delta(L_{i},L)}$ where $U$ is the
total number of utterances in a trial, $L$ is the plan length and $L_{i}$
refers to the time step where item i is completed.

Goals |  Goal Specification
---|---
Set up table |  Put [N forks, N plates, N waterglasses or wineglasses] on [kitchentable, coffeetable]
Put groceries |  Put [N apple, N salmon, N pudding, N cupcakes] inside [cabinet, fridge]
Prepare food |  Put [N apple, N salmon, N pudding, N cupcakes] on [kitchentable, coffeetable]
Load dishwasher |  Put [N forks, N plates, N waterglasses or wineglasses] inside [dishwasher]
TABLE II: VirtualHome goal specifications.

(a) Overcooked Simulation

(b) Virtual-Home Simulation

(c) Virtual-Home Human Experiment

(d) Virtual-Home Human Rating

Figure 3: Experimental results in Overcooked and VirtualHome. The quantitative
results from experiments (a, b, c) demonstrate that GOMA led to the greatest
speedup (left) and least plan cost (right) compared to other baselines. In
human subjective ratings (d), participants find GOMA to be more helpful and
communicate more useful information than other models. Figure 4: Example of
typical communication enabled by GOMA in VirtualHome. (a) Once the human (in
the blue shirt) gives a command to the AI Assistant (in the orange shirt), it
infers the human goal and reasons that the human needs 2 plates and 2 forks.
(b) As the AI watches the human agent opening the fridge, GOMA informs the
human that the plates are on the coffee table. Consequently, the human goes to
the coffee table to pick up a plate. Figure 5: Agents’ trajectories with No-
Comm (left) and with GOMA (right) in a VirtualHome environment. In this
example, the AI Assistant needs to find a plate and a fork while the human is
looking for a water glass. Both agents have knowledge about the items that the
other agent is looking for but not their own goal objects. In the No-Comm
setting, the agents cannot share knowledge and have to open many containers to
search for goal items. By inferring the other agent’s goal and communicating
goal-relevant knowledge, GOMA drastically reduces the total number of steps
taken to complete the task.

###  V-B VirtualHome

VirtualHome [39] is a multiagent household simulator. In VirtualHome, agents
collaborate to complete daily household tasks. In our experiments, we include
four common types of household tasks: Set Table, Load Dishwasher, Get Snacks,
Stock Fridge. The goal for each task is defined as a set of goal predicates
and their counts as defined in Table II. In VirtulHome, each object is
associated with a unique object ID, which we use in agents’ communication to
distinguish the referent from others (e.g. cabinet.145).

Simulation Experiment. We simulate 25 collaborative scenarios in VirtualHome
across 4 goal types and 5 simulated apartments. Each episode is run 3 times
and we report the averaged results. We simulate the human agent using the MCTS
planner from [39]. The simulated human agent requests help by sampling a
subset of the goal predicates and replies to the AI assistant’s questions. We
compare our proposed method against four baselines: Single-agent, No-
Communication (No-Comm), Goal-Agnostic (Goal-Ag), and LLM agent. The first two
are identical to the ones in Overcooked. The Goal-Ag baseline does not infer
the joint goals and plan and instead randomly shares information about any
objects that the human doesn’t know. For the LLM agent, we use COELA [5],
which achieved state-of-the-art performance on human-AI cooperation in
VirtualHome.

Human Experiment. We developed an online human interface to conduct a human
experiment. The interface follows the same task setup as the simulation study
with 5 conditions: Single-Agent, No-Comm, Goal-Ag, CoELA, and GOMA (Ours). The
participants controlled the human user agent to either perform the task alone
(Single-Agent) or to work with an AI assistant driven by one of the methods.
In all collaborative conditions, the interface includes a chatbox that allows
the participant and the AI agent to send messages to each other. We recruited
10 participants who had no prior experience with the simulator. They completed
60 trials over 20 tasks. After completing a trial with an AI assistant, the
participants were asked to rate the AI assistant based on four criteria: 1)
the assistant is helpful; 2) the assistant understands your goal; 3) the
assistant’s communication is useful; and 4) the assistant communicates more
than necessary. Each criterion is rated on a 7-point Likert scale (1 =
Strongly Disagree, 7 = Strongly Agree).

Metrics. In line with previous studies on VirtualHome [39], we evaluate the
models’ performance by computing 1) speedups: counting the number of steps
taken to complete the task, and 2) total costs: an overall cost metric that
sums up the action and communication cost over the episode.

##  VI Results

###  VI-A Simulation Experiment

The simulation results are shown in Fig. 3ab. The advantage of collaboration
is evident as the Single-agent baseline performed significantly worse than all
other collaborative models. Overall, we find that across both Overcooked and
Virtual-Home experiments, our model outperformed other baselines in all
metrics. The differences between GOMA and other baselines are all
statistically significant with $p<0.01$ across two performance metrics.

In Overcooked, GOMA took on average 46.76 steps to complete the task,
achieving a 44.61% speedup. Our model completed the tasks with the lowest
costs (M = 58.06) compared to the Heuristic-based model (M = 72.0) and No-Comm
baseline (M = 65.05). Additionally, GOMA delivered the dishes in the best
condition among all tested models, as signaled by the lowest coldness penalty
(7.85).

In VirtualHome, GOMA took on average 20.08 steps to complete the task with a
55.8% speedup. Despite having observed objects relevant to the human agent’s
goal, CoELA made few utterances (Mean = 3.03) and focused exclusively on
communicating observations of its own goal. For example, when given a command
”Please help me find a fork.”, CoELA would respond later ”I found fork 323 in
cabinet 132.” and did not share any knowledge that may be useful for the
human’s subgoal and plan. The Goal-Agnostic model makes frequent (Mean = 5.41)
but mostly irrelevant utterances about possible goal objects. However, it did
perform slightly better than the No-Comm baseline because, with enough
utterances, it occasionally mentions useful information to the human agent.

Unlike baselines, GOMA can communicate and inquire about useful goal-relevant
information with the human, leading to improved team performance. We include
two qualitative examples of GOMA in VirtualHome simulations in Fig. 4 and 5.
In these examples, we show that due to partial observability, the AI Assistant
and the human have exclusive knowledge about certain objects relevant to other
agent’s subgoals. GOMA allows the AI Assistant to inquire and inform another
agent about this goal-relevant information. As a result, the agents can find
the goal objects quickly without exhaustively opening and checking all
containers.

###  VI-B Human Experiment

The human experiment results are shown in Fig. 3cd. Similar to the simulation
results, our proposed method had the greatest speedup over a single agent and
outperformed all baselines in terms of plan costs. In contrast to the
simulation study, the Goal-agnostic model here performed no better than No-
Comm and CoELA as participants stopped paying attention to the assistant after
it made too many statements irrelevant to the goal. This is shown in the
participants’ subjective ratings where participants reported that the Goal-
Agnostic baseline communicated more than necessary.

The participants gave a higher subjective rating to our model than other
baselines on all 4 items. Interestingly, even though CoELA and GOMA performed
goal inference with the same method, the participants thought that only GOMA
understood the human’s goal. This is because by communicating goal-relevant
information, GOMA implicitly expressed its understanding of the user’s goal,
whereas CoELA only communicated the progress of its own subgoal.

##  VII Conclusion

In this paper, we introduce GOMA, which enables an embodied AI assistant to
efficiently and effectively communicate with a human user to achieve optimal
cooperation. GOMA achieves this by reasoning about the other agent’s mental
state, assessing the misalignment between mental states, and then proactively
initiating necessary communication to exchange goal-relevant information. Our
experiments in Overcooked and VirtualHome demonstrate that embodied AI
assistants built with GOMA can not only help achieve the human goal faster
with lower total plan cost but also receive higher subjective ratings from
human participants.

Our study is not without limitations. We have not evaluated GOMA on real-world
robot assistants, which we intend to study in the future. We also plan to
enhance the flexibility of the communication generation, so that it can
communicate about any information relevant to the task in an open-ended
manner. Finally, we also aim to investigate more general belief
representations that go beyond object states.

## References

  * [1] M. Tomasello, _Origins of human communication_. MIT press, 2010.
  * [2] L. Ying, T. Zhi-Xuan, V. Mansinghka, and J. B. Tenenbaum, “Inferring the goals of communicating agents from actions and instructions,” in _Proceedings of the AAAI Symposium Series_ , vol. 2, no. 1, 2023, pp. 26–33.
  * [3] D. Hadfield-Menell, S. J. Russell, P. Abbeel, and A. Dragan, “Cooperative inverse reinforcement learning,” in _Advances in Neural Information Processing Systems_ , 2016, pp. 3909–3917.
  * [4] X. Gao, R. Gong, Y. Zhao, S. Wang, T. Shu, and S.-C. Zhu, “Joint mind modeling for explanation generation in complex human-robot collaborative tasks,” in _2020 29th IEEE international conference on robot and human interactive communication (RO-MAN)_. IEEE, 2020, pp. 1119–1126.
  * [5] H. Zhang, W. Du, J. Shan, Q. Zhou, Y. Du, J. B. Tenenbaum, T. Shu, and C. Gan, “Building cooperative embodied agents modularly with large language models,” _arXiv preprint arXiv:2307.02485_ , 2023.
  * [6] A. Hong, N. Lunscher, T. Hu, Y. Tsuboi, X. Zhang, S. F. dos Reis Alves, G. Nejat, and B. Benhabib, “A multimodal emotional human–robot interaction architecture for social robots engaged in bidirectional communication,” _IEEE transactions on cybernetics_ , vol. 51, no. 12, pp. 5954–5968, 2020.
  * [7] H. H. Clark, _Using language_. Cambridge university press, 1996.
  * [8] M. Kleiman-Weiner, M. K. Ho, J. L. Austerweil, M. L. Littman, and J. B. Tenenbaum, “Coordinate to cooperate or compete: abstract goals and joint intentions in social interaction,” in _COGSCI_ , 2016.
  * [9] S. A. Wu, R. E. Wang, J. A. Evans, J. B. Tenenbaum, D. C. Parkes, and M. Kleiman-Weiner, “Too many cooks: Bayesian inference for coordinating multi-agent collaboration,” _Topics in Cognitive Science_ , vol. 13, no. 2, pp. 414–432, 2021.
  * [10] J. MacMillan, E. E. Entin, and D. Serfaty, “Communication overhead: The hidden cost of team cognition.” 2004.
  * [11] E. Horvitz and J. Apacible, “Learning and reasoning about interruption,” in _Proceedings of the 5th international conference on Multimodal interfaces_ , 2003, pp. 20–27.
  * [12] V. V. Unhelkar, S. Li, and J. A. Shah, “Decision-making for bidirectional communication in sequential human-robot collaborative tasks,” in _Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction_ , 2020, pp. 329–341.
  * [13] E. C. Williams, N. Gopalan, M. Rhee, and S. Tellex, “Learning to parse natural language to grounded reward functions with weak supervision,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 4430–4436.
  * [14] T. Zhi-Xuan, L. Ying, V. Mansinghka, and J. B. Tenenbaum, “Pragmatic instruction following and goal assistance via cooperative language-guided inverse planning,” _arXiv preprint arXiv:2402.17930_ , 2024.
  * [15] L. Yuan, X. Gao, Z. Zheng, M. Edmonds, Y. N. Wu, F. Rossano, H. Lu, Y. Zhu, and S.-C. Zhu, “In situ bidirectional human-robot value alignment,” _Science robotics_ , vol. 7, no. 68, p. eabm4183, 2022.
  * [16] C. Zhang, J. Chen, J. Li, Y. Peng, and Z. Mao, “Large language models for human-robot interaction: A review,” _Biomimetic Intelligence and Robotics_ , p. 100131, 2023.
  * [17] B. Ichter, A. Brohan, Y. Chebotar, C. Finn, K. Hausman, . …, and C. Kelly, “Do as i can, not as i say: Grounding language in robotic affordances,” in _Proceedings of The 6th Conference on Robot Learning_ , ser. Proceedings of Machine Learning Research, K. Liu, D. Kulic, and J. Ichnowski, Eds., vol. 205. PMLR, 14–18 Dec 2023, pp. 287–318.
  * [18] Z. Mandi, S. Jain, and S. Song, “Roco: Dialectic multi-robot collaboration with large language models,” _arXiv preprint arXiv:2307.04738_ , 2023.
  * [19] S. Devin and R. Alami, “An implemented theory of mind to improve human-robot shared plans execution,” in _2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_. IEEE, 2016, pp. 319–326.
  * [20] K. E. Schaefer, E. R. Straub, J. Y. Chen, J. Putney, and A. W. Evans III, “Communicating intent to develop shared situation awareness and engender trust in human-agent teams,” _Cognitive Systems Research_ , vol. 46, pp. 26–39, 2017.
  * [21] T. Ullman, “Large language models fail on trivial alterations to theory-of-mind tasks,” _arXiv preprint arXiv:2302.08399_ , 2023.
  * [22] C. L. Baker, J. Jara-Ettinger, R. Saxe, and J. B. Tenenbaum, “Rational quantitative attribution of beliefs, desires and percepts in human mentalizing,” _Nature Human Behaviour_ , vol. 1, no. 4, pp. 1–10, 2017.
  * [23] T. Zhi-Xuan, J. Mann, T. Silver, J. Tenenbaum, and V. Mansinghka, “Online bayesian goal inference for boundedly rational planning agents,” _Advances in Neural Information Processing Systems_ , vol. 33, 2020.
  * [24] T. Shu, A. Bhandwaldar, C. Gan, K. Smith, S. Liu, D. Gutfreund, E. Spelke, J. Tenenbaum, and T. Ullman, “Agent: A benchmark for core psychological reasoning,” in _International conference on machine learning_. PMLR, 2021, pp. 9614–9625.
  * [25] C. Jin, Y. Wu, J. Cao, J. Xiang, Y.-L. Kuo, Z. Hu, T. Ullman, A. Torralba, J. B. Tenenbaum, and T. Shu, “Mmtom-qa: Multimodal theory of mind question answering,” _arXiv preprint arXiv:2401.08743_ , 2024.
  * [26] L. Ying, K. M. Collins, M. Wei, C. E. Zhang, T. Zhi-Xuan, A. Weller, J. B. Tenenbaum, and L. Wong, “The neuro-symbolic inverse planning engine (nipe): Modeling probabilistic social inferences from linguistic inputs,” _arXiv preprint arXiv:2306.14325_ , 2023.
  * [27] L. Ying, T. Zhi-Xuan, L. Wong, V. Mansinghka, and J. Tenenbaum, “Grounding language about belief in a bayesian theory-of-mind,” _arXiv preprint arXiv:2402.10416_ , 2024.
  * [28] A. Dragan and S. Srinivasa, “Generating legible motion,” 2013.
  * [29] F. Stulp, J. Grizou, B. Busch, and M. Lopes, “Facilitating intention prediction for humans by optimizing robot motions,” in _2015 IEEE/RSJ international conference on intelligent robots and systems (IROS)_. IEEE, 2015, pp. 1249–1255.
  * [30] M. Kwon, S. H. Huang, and A. D. Dragan, “Expressing robot incapability,” in _Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction_ , 2018, pp. 87–95.
  * [31] Y. Zhang, S. Sreedharan, A. Kulkarni, T. Chakraborti, H. H. Zhuo, and S. Kambhampati, “Plan explicability and predictability for robot task planning,” in _2017 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2017, pp. 1313–1320.
  * [32] X. Gao, L. Yuan, T. Shu, H. Lu, and S.-C. Zhu, “Show me what you can do: Capability calibration on reachable workspace for human-robot collaboration,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 2644–2651, 2022.
  * [33] P. J. Gmytrasiewicz and P. Doshi, “A framework for sequential planning in multi-agent settings,” _Journal of Artificial Intelligence Research_ , vol. 24, pp. 49–79, 2005.
  * [34] P. Doshi and P. J. Gmytrasiewicz, “Monte Carlo sampling methods for approximating interactive POMDPs,” _Journal of Artificial Intelligence Research_ , vol. 34, pp. 297–337, 2009.
  * [35] L. P. Kaelbling, M. L. Littman, and A. R. Cassandra, “Planning and acting in partially observable stochastic domains,” _Artificial intelligence_ , vol. 101, no. 1-2, pp. 99–134, 1998.
  * [36] A. Bosch-Domenech, J. G. Montalvo, R. Nagel, and A. Satorra, “One, two,(three), infinity,…: Newspaper and lab beauty-contest experiments,” _American Economic Review_ , vol. 92, no. 5, pp. 1687–1701, 2002.
  * [37] OpenAI, “Gpt-4 technical report,” 2023.
  * [38] M. Carroll, R. Shah, M. K. Ho, T. Griffiths, S. Seshia, P. Abbeel, and A. Dragan, “On the utility of learning about humans for human-ai coordination,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [39] X. Puig, T. Shu, S. Li, Z. Wang, Y.-H. Liao, J. B. Tenenbaum, S. Fidler, and A. Torralba, “Watch-and-help: A challenge for social perception and human-ai collaboration,” _arXiv preprint arXiv:2010.09890_ , 2020.


CharacterGAN: Few-Shot Keypoint Character Animation and Reposing

Tobias Hinz^1,2

Matthew Fisher^2

^1University of Hamburg

Oliver Wang^2

Eli Shechtman^2

^2Adobe Research

Stefan Wermter^1

Input                                         Input

figureWe train a generative model in a low-data setting (8 to 15 training samples) to repose and animate characters based on keypoint positions. The first row shows video results (indicated throughout the paper by ), of our method driven by linearly interpolating input keypoints. Please view with Adobe Acrobat to see animations. The second and third rows show interpolated frames generated by our method between a single start and end frame (left and right columns).

We introduce CharacterGAN, a generative model that can be trained on only a few samples (8 – 15) of a given character.
Our model generates novel poses based on keypoint locations, which can be modified in real time while providing interactive feedback, allowing for intuitive reposing and animation.
Since we only have very limited training samples, one of the key challenges lies in how to address (dis)occlusions, e.g. when a hand moves behind or in front of a body.
To address this, we introduce a novel layering approach which explicitly splits the input keypoints into different layers which are processed independently.
These layers represent different parts of the character and provide a strong implicit bias that helps to obtain realistic results even with strong (dis)occlusions.
To combine the features of individual layers we use an adaptive scaling approach conditioned on all keypoints.
Finally, we introduce a mask connectivity constraint to reduce distortion artifacts that occur with extreme out-of-distribution poses at test time.
We show that our approach outperforms recent baselines and creates realistic animations for diverse characters.
We also show that our model can handle discrete state changes, for example a profile facing left or right, that the different layers do indeed learn features specific for the respective keypoints in those layers, and that our model scales to larger datasets when more data is available.
Code is available at <https://github.com/tohinz/CharacterGAN>.

§ INTRODUCTION

Modifying and animating 2D artistic characters is a task that requires experts to manually create many instances of the same character in different poses, which is a time-consuming and expensive process.
As a result, often only very few examples of a specific character are available, as the cost of acquisition of additional examples is high.
Using video frame interpolation methods to create new poses can reduce the overhead, but these methods do not leverage character-specific priors and so can only be used for small motions.
Similarly, warping input frames directly with 2D handles cannot account for disocclusions or appearance changes between poses.
We believe that control over 2D animated characters is a challenging open problem, due to the sparse example set and challenging topology changes.
In this work we have two main goals: (1) to generate high quality frames of an animated character based on a small number of examples, and (2) to generate these images based on a sparse set of keypoints that can be easily modified in real time.

To address both issues, we propose to train a conditional Generative Adversarial Network [10] (GAN) that allows us to create new images of a character based on a set of given keypoint locations.
We show that, with the right form of implicit biases, such a model can be trained in a few-shot setting (i.e., 10s of training images).
In character animation, each training image has to be created manually, making it expensive to acquire the number of images required for training most conditional GAN models [14], and unlike recent work on single image generative models [33], we desire precise control over the generated image.
Our approach makes it possible for the broad community to animate and repose any 2D character without the need for specialized skills.

Our method consists of a GAN that is trained on 8 – 15 images of a given character and its associated keypoints.
One of the key challenges is that it is difficult for the model to learn which parts of a character should be occluded by other parts.
E.g. when the hand of a humanoid character moves in front of the torso, either the hand should be visible at all times (if it is in the foreground) or only the torso should be visible if the hand moves “behind” the torso.
To learn this ordering in a purely data-driven approach we need many images which we do not have in our setting.
Instead, we propose to use user-specified layers for our keypoints, i.e. each keypoint lies in a given layer and we can introduce an ordering over those layers.
For example, a humanoid character could be described by three layers, consisting of 1) the arm and leg in the “back” (i.e. occluded by the torso and other arm and leg), 2) the torso and head which are occluded by one arm and one leg and occlude one arm and one leg, and 3) the other arm and leg which occlude every other part of the character.
Our model processes each of these layers independently, i.e. generates features for each layer without knowing the location of the keypoints in the other layers.
We then use an adaptive scaling approach, conditioned on all keypoints, to spatially scale the features of each layer, before concatenating and using them to generate the final image.

Additionally, we can train our generator to predict the mask for a generated character which we found to be a robust way to identify and automatically fix unrealistic keypoint layouts at test time.
Our model naturally learns to associate the keypoints with roughly semantically meaningful body parts for each layer, and can handle discrete state points that arise as a function of keypoint locations (e.g., switching between a profile facing left or right).
We show via a number of qualitative and quantitative experiments that our resulting model allows for real-time reposing and animation of diverse characters and that our layering approach outperforms the more traditional conditioning approach of using all keypoints at once.
Since we assume no prior knowledge about the modeled character our model can be applied to any shape and does not require additional data such as a 3D model or a character mesh.
This allows our model to be applied in domains for which only limited data is available (e.g. artistic drawings or sprite sheets) without the need for additional manual input besides the keypoint labels.
In summary, we introduce the following main contributions:

* We show that it is possible to train a GAN on only few images (8 – 15) of a given character to allow for few-shot character reposing and animation. By only conditioning the training on keypoints (instead of e.g. semantic maps) the trained model allows for character reposing in real-time without expert knowledge.

* By using a layered approach that explicitly encodes the ordering of different keypoints our model is able to model occlusions with only very limited training data.

* We introduce a mask connectivity constraint, where a jointly predicted mask can be used at test time to automatically fix keypoint layouts for which the model produces unrealistic outputs.

§ RELATED WORK

Conditional GANs make it possible to control the output of the generator to varying degrees and have also been shown to help with the training process.
The conditioning input can come in several forms, such as class conditioning [25], semantic maps [14], keypoints [31], or bounding boxes [12, 13].
However, most conditional GANs are trained with large datasets and are applied to broader domains, whereas we are interested in animating a specific character.
In the following, we first focus on approaches in how to leverage small amounts of data to train GANs and conclude with a section about how the chosen conditioning method directly affects how the model can be used at test time.

Our model processes keypoints which are split into individual layers. The resulting features for each keypoint layer are then scaled, concatenated, and used to generate the final image.

Few-Shot Learning with GANs
One promising approach to few-shot learning with GANs is to fine-tune GANs that are pre-trained on large datasets.
This can be achieved by fine-tuning a given model [42], by only training parts of the pre-trained model [27, 32, 26, 19], or transfering knowledge between different - but related - domains [41, 45].
However, these methods still rely on a model that is pre-trained on a large dataset in a very similar domain.
In contrast to this, we do not fine-tune a pre-trained model but instead train our model from scratch on limited available data.

Recent approaches show that applying data augmentation techniques during training of GANs directly is useful, especially when the dataset is small [37, 17, 46, 47].
One of the main insights of these approaches is that it is not sufficient to only augment real images since this leads the discriminator to learn that these augmented images are part of the real data.
We also make heavy use of data augmentation in our training process but use much less data (8 – 15 images) than the approaches mentioned here (usually 100+ images).

It is also possible to learn useful features from a single image [51, 2] and tasks such as texture synthesis [15, 18, 5, 49], image retargeting [34], inpainting and segmentation [38, 9], and unconditional image synthesis [33, 11] are possible with only a single image.
Other approaches train GANs for image-to-image translation with only one pair of matching images [20, 4, 28, 39].
However, a model trained on just a single image can only generate limited variations of the training input.
Therefore, we propose to increase the available information by slightly increasing the training set's size, which allows the model to generate more variations of a learned object.

An important characteristic of models for character reposing and animation is how the character is controlled.
Existing approaches use driving videos [35], learn a distribution over poses for inbetweening [30], or map puppets to skeletons [7].
However, it is difficult to achieve a desired pose with these approaches [35, 30] or requires expert knowledge to obtain the training data [7].
Another approach is to condition the model on a semantic map [6, 39] which can be changed at test time.
However, modifying semantic maps or edge maps is not something that can be done on-the-fly, but instead takes time and skill.
Finally, one could start with the character in a given pose and warp or stretch it until it reaches the desired pose [21] which might lead to unrealistic results if the end pose is too different from the starting pose.
Given the previous limitations we only condition our model on high-level keypoints which are provided for a given character, making it easy and fast to generate new poses at test time by moving keypoints.

§ METHODOLOGY

We focus on character animation and reposing where the images show the same character in different poses from the same viewpoint.
Since we work in low data scenarios, we focus on cases with a small amount of images of the given character.
Furthermore, the individual images can depict discrete appearance variations, e.g. different facial expressions.
We do not make any other prior assumptions about the structure of the character.
Our method requires an input image set, keypoints per image, keypoint connectivity information (essentially a skeleton), and a layer ordering for the keypoints.
This information can be obtained easily as we only need labels for very few images.
While the keypoints have to be labeled manually, all other information is image independent and, thus, only needs to be defined once.

We experimented with various ways to input keypoints to the network, such as as individual channels, but found that representing keypoints as RGB Gaussian blobs performed the best.
This also means that the input condition always has the same input dimensionality, independent of the number of keypoints, which is helpful as our model uses the same parameters to process these individual layers.
Each keypoint is defined by a position $\{x, y\}$, three randomly chosen color values for the three RGB channels, and the values $\sigma$ and $r$, where $\sigma$ represents the falloff of the Gaussian distribution we use for blurring the keypoint, and $r$ represent the radius of the final keypoint, which we define later.
Discrete keypoints are modelled with individual RGB values, and in cases where different keypoints overlap we sum the respective colors.

§.§ Model

Our model consists of a generator and a discriminator (see <ref>).
The generator generates the image based on the keypoint locations while the discriminator is trained to distinguish between real and fake image-keypoint pairs.
To address the challenge of having only a limited number of training images we incorporate several problem-specific implicit biases into our generator design and combine this with data augmentation techniques.

Generator and Discriminator
We base our architecture on pix2pixHD [40].
However, while the discriminator is similar to the original pix2pixHD discriminator, we add several implicit biases to the generator reflecting our prior knowledge about the problem.
Concretely, we know that the modeled characters are inherently three-dimensional, i.e. if some body parts are occluded by others they still exist even though they may not be visible.
To address this, we split our characters into different layers, e.g. representing the “left” side of the character (e.g. left arm and leg), the “middle” part of the character (e.g. head and torso) and the “right” side of the character (e.g. right arm and leg).
These layers can be modeled individually and are then be composed to form a final image.
Our generator processes each keypoint layer individually and, thus, learns layer-specific representations.

Intuitively, we could set some features to zero if they are occluded by other features.
For example, if the left hand is “behind” the torso for a given image, zeroing out the features for the left hand might make it easier for the generator to generate a realistic image.
Conversely, if the right hand is “in front of” the torso, zeroing out the torso features at the location of the right hand might improve the performance.
To address this, we incorporate an adaptive scaling technique [29] in which we scale the features of each layer before concatenating them.
For this, we first learn an embedding of the keypoints and their layers (“Global” in Adaptive Scaling of <ref>).
Based on this embedding we then learn scaling parameters for each keypoint layer and use them to scale the features of each each layer.
These scaled layer features are then concatenated and used to generate the final image.

Our discriminator takes the keypoint conditioning $k$ concatenated with an RGB image as input and classifies it as either real or fake.
We use two patch-discriminators [14], one of which operates on the full resolution image, while the second operates on the image down-scaled by a factor of two.
We use a feature matching and an adversarial loss during training as defined by the pix2pixHD model [40].
The adversarial loss is the standard GAN loss $L_{\text{adv}}$:

\begin{equation}
\begin{aligned}
\underset{G}{min}\ \underset{D}{max}\  \mathcal{L}_{\text{adv}} = \mathbb{E}_{(k,x)}[\text{log}D(k,x)] + \\
\mathbb{E}_{(k)}[\text{log}(1-D(k, G(k))]
\end{aligned}
\end{equation}

where $k$ is the keypoint condition and $x$ is the corresponding real image.
The feature matching loss $L_{\text{fm}}$ stabilizes training by forcing the generator to produced realistic features at multiple scales and is defined as:

\begin{equation}
\underset{G}{min}\ \ \mathcal{L}_{\text{fm}} = \mathbb{E}_{(k,x)}\sum_{i}^{T}\frac{1}{N_i}[\vert\vert D(k,x) - D(k, G(k))\vert\vert_1 ]
\end{equation}

where $T$ is the number of layers in the discriminator and $N_i$ is the number of elements in each layer.
We add a perceptual loss [16, 44] to further improve the image quality.
We use a VGG net to extract features from real and generated images and compute the perceptual loss $L_{\text{perc}}$ as defined by [16].
Our final loss is the combination of these losses:

\begin{equation}
\underset{G}{min}\ \underset{D}{max}\  \mathcal{L}_{\text{adv}} + \mathcal{L}_{\text{fm}} + \mathcal{L}_{\text{perc}}.
\end{equation}

Dataset     2cSinGAN [33]     2cConSinGAN [11]     2cDeepSIM [39]     2cCharacterGAN

PSNR $\uparrow$     LPIPS $\downarrow$     PSNR $\uparrow$     LPIPS $\downarrow$     PSNR $\uparrow$     LPIPS $\downarrow$     PSNR $\uparrow$     LPIPS $\downarrow$

Watercolor Man     18.03$\pm$0.09     0.149$\pm$0.0032     21.65$\pm$0.32     0.102$\pm$0.0151     21.92$\pm$0.03     0.066$\pm$0.0002     24.33$\pm$0.05     0.042$\pm$0.0002

Ape     13.94$\pm$0.02     0.234$\pm$0.0098     18.09$\pm$0.08     0.132$\pm$0.0058     17.01$\pm$0.01     0.132$\pm$0.0009     18.94$\pm$0.04     0.089$\pm$0.0004

Sprite Man     17.14$\pm$0.08     0.156$\pm$0.0063     23.19$\pm$0.17     0.087$\pm$0.0045     22.08$\pm$0.07     0.071$\pm$0.0001     25.23$\pm$0.02     0.038$\pm$0.0006

Dog     15.01$\pm$0.28     0.195$\pm$0.0078     19.24$\pm$0.27     0.125$\pm$0.0089     20.08$\pm$0.07     0.087$\pm$0.0010     22.22$\pm$0.04     0.062$\pm$0.0006

Ostrich     16.54$\pm$0.03     0.200$\pm$0.0023     23.30$\pm$0.18     0.103$\pm$0.0020     21.54$\pm$0.13     0.079$\pm$0.0006     23.80$\pm$0.09     0.063$\pm$0.0005

Cow     13.58$\pm$0.03     0.220$\pm$0.0016     18.71$\pm$0.12     0.133$\pm$0.0037     17.65$\pm$0.03     0.115$\pm$0.0013     19.59$\pm$0.01     0.085$\pm$0.0005

Results of cross validation for the different models.

2cCharacterGAN     2cCharacterGAN

Dataset     2cNo Layer, No Scaling     2cLayer, No Scaling

PSNR $\uparrow$     LPIPS $\downarrow$     PSNR $\uparrow$     LPIPS $\downarrow$

Watercolor Man     23.81 $\pm$ 0.15     0.049 $\pm$ 0.0050     24.29 $\pm$ 0.02     0.044 $\pm$ 0.0004

Ape     18.11 $\pm$ 0.06     0.109 $\pm$ 0.0009     18.92 $\pm$ 0.03     0.092 $\pm$ 0.0003

Sprite Man     24.63 $\pm$ 0.12     0.041 $\pm$ 0.0020     25.12 $\pm$ 0.01     0.039 $\pm$ 0.0004

Dog     21.71 $\pm$ 0.10     0.068 $\pm$ 0.0007     22.14 $\pm$ 0.05     0.064 $\pm$ 0.0005

Ostrich     22.95 $\pm$ 0.04     0.068 $\pm$ 0.0007     22.97 $\pm$ 0.04     0.067 $\pm$ 0.0004

Cow     18.95 $\pm$ 0.08     0.094 $\pm$ 0.0021     19.52 $\pm$ 0.01     0.086 $\pm$ 0.0009

Watercolor Man     Ape     Ostrich

Sprite Man     Dog     Cow

Ablation study: results of cross validation for different parts of our model.

Data Augmentation

We employ both affine transformations and thin-plate-spline augmentation [39].
We use a mixture of horizontal and vertical translations and horizontal flipping and randomly sample a subset from these augmentation approaches at each training iteration.
Thin-plate-spline (TPS) augmentation was introduced by Vinker et al. [39].
For this approach, the image is modeled as a grid and each grid point is then shifted by a random distance sampled from a uniform distribution.
After this, a TPS is used to smooth the transformed grid into a more realistic warp.
Using TPS augmentation results in warped images where parts of the image are stretched and elongated, adding further variation to the training data.
All augmentations are applied to the given image and the associated keypoints and skeleton.

§.§ Editability

After our model is trained it offers a straightforward way to modify the pose of the character.
Given an image of the character the user can drag keypoints to novel positions and the generator will generate the character in the new pose.
We can also easily switch between different discrete states, and we provide two ways to do this.
First, it can be handled completely automatic where the discrete state is determined solely by keypoint positions (e.g., for facing left vs right).
Second, we provide the ability to have specific keypoints for individual states, such as smile vs frown, so a user can choose the desired expression at inference time.
We also allow the user to optionally enable a mask-based connectivity correction.
In this case, if the user positions keypoints too far from their input distribution, such that they could lead to unrealistic or undesired results, we can automatically modify the keypoint locations to achieve more realistic results.

Ensuring Mask Connectivity
If the image background is of uniform color, or we have a segmentation network, we can automatically extract a foreground-background mask.
We can use this mask as additional conditioning information during training, i.e. in this case the generator does not only generate the RGB image but also the mask, while the discriminator gets as input an image and its associated keypoints and mask.
At test time, if a keypoint is moved in a way that results in a layout that is too different from the ones seen at train time it can happen that the generator generates either “disconnected” body parts (e.g. the hand is not connected to the body) or introduces unwanted artifacts.
Since the generator predicts the mask for the generated character we can use connected component analysis [8, 43] to check whether the generated mask is connected.
We found two cases that often lead to a disconnected mask; if the disconnected part contains a keypoint the resulting character will be ripped apart or, alternatively, if the disconnected part does not contain a keypoint the generated image will often contain unwanted artifacts.
In either case, we fix the result by automatically moving nearby keypoints in an iterative procedure until the predicted mask is fully connected again.

Given the last moved keypoint that caused the a disconnected region to appear, we identify the closest keypoint and move this keypoint in the same direction as the keypoint moved by the user by a fraction $\delta$ of the absolute distance.
If $\delta = 1$ the keypoint is moved exactly in the same direction and by the same amount as the original keypoint and if $\delta = 0$ no keypoints are moved automatically at all.
We found that a default of $\delta = 0.5$ achieved good results in a single iteration in most cases, however, this process can be repeated iteratively until the mask is connected (<ref>).

Original     Generated        Original     Generated        Original     Generated        Original     Generated

Examples from our model which was trained on only 8 – 12 images for each of the characters. Odd columns show the original image and our intended modifications, even columns show the output of our model.

Original     Generated        Original     Generated        Original     Generated        Original     Generated

Examples from our model which was trained on only 8 – 12 images for each of the characters. Even columns show the original image and our intended modifications, odd columns show the output of our model.

SinGAN     ConSinGAN     DeepSIM      2cCharacterGAN     Ground

w/o layer     w/ layer     Truth

Qualitative examples of reconstructing held-out test images based on their keypoint locations.

§ EXPERIMENTS

Since this specific problem has not been addressed in the literature there are no standard baselines available to compare against and, therefore, we chose to use three single-image models as our baselines as they are closest to our settings regarding needed data.
SinGAN [33] and ConSinGAN [11] are trained on a single image for tasks such as image generation and harmonization.
Both models are trained in a multi-stage manner where the image resolution increases with each stage.
We adapt both models by additionally conditioning the training at each stage on the image keypoints.
DeepSIM [39] trains a model specifically for image manipulation, where the conditioning input consists either of edge-maps, semantic labels, or a combination of both and the training set consists of a single image and its condition.
We adapt the DeepSIM model to our setting by replacing the edge-map conditioning with keypoints.

DAIN     DeepSIM     Ours     Input        DAIN     DeepSIM     Ours     Input

Comparison of our approach to frame interpolation [3] and DeepSIM [39] using the displayed input frames ().

We perform experiments on several different characters, including humans and animals.
Our data comes from different sources such as drawings by artists [7] and characters from sprite sheets.
For our characters we have 8 – 15 images which we manually label with keypoint locations.
All manual labeling can be done without any specific skills and our approach requires only RGB images as input.
In contrast, traditional animation pipelines would likely need additional expert drawings, as well as a more structured character representation (e.g. a mesh).
As a result, our method is less labor-intensive, and is more amenable to unskilled users compared to traditional animation approaches.

We use the same layer definition and number of layers (three) for all our examples, roughly corresponding to “front”, “middle”, and “back”.
Based on our experiments, the keypoint labeling and model training for a new character can be done in well under one hour.
Manually labeling new keypoints takes roughly 30-60 seconds per image (depending on the number of keypoints and used interface) while training the model takes roughly 30-40 minutes on an NVIDIA RTX 2080Ti.
Our model itself can generate images in real-time and takes about 0.01 seconds for one image on an NVIDIA RTX 2080Ti while one iteration of evaluating the mask connectivity takes about 0.00002 seconds on CPU.

§.§ Quantitative Evaluation

To the best of our knowledge there are no established quantitative evaluation metrics for few-shot character animation.
[33] introduced the Single-Image FID (SIFID) score to evaluate single-image generative models.
However, [11] report large variance in the SIFID scores and [32] report that models in the few-shot setting overfit to the FID metric.

We use the Peak Signal-to-Noise Ratio (PSNR) and LPIPS [44] as metrics to evaluate our model.

To evaluate our model we design an $N$-fold cross-validation for a given character with $N$ images.
Given a character we train our model $N$ times on $N-1$ images were each image in the dataset is left out of training exactly once.
At test time we generate the left-out image based on its keypoint layout and calculate the PSNR and LPIPS between the generated and ground-truth image.
For each character we run the full $N$-fold cross-validation three times and report the average and standard deviation across the three runs.
Some examples are shown in <ref>.

No Layer     Layer        No Layer     Layer

Comparison of layered vs non-layered at occlusions/overlaps.

<ref> shows the results of our model compared to the baselines.
Our model achieves the best LPIPS and PSNR for all characters.
We observe that the PSNR is not always predictive of the (perceptive) quality of the generated image.
In particular, SinGAN and ConSinGAN often generate images where the character exhibits disconnected body parts (e.g. the feet are not connected to the main body), but this is not represented in the PSNR, as feet and legs cover a relatively small area of the image.

<ref> shows ablation studies with our model, where we train the same model without any layering, and with layering but no adaptive scaling.
We can see that the layering approach improves the performance in all cases.
Finally, adding the adaptive scaling further improves the performance, albeit not as much as the keypoint layering.
While the difference in the quantitative evaluation may not seem large between our layered and baseline model, this is partly due to the quantitative metrics which evaluate the global image, while we show that the improvements are mainly in areas of overlapping keypoints, leading to clear qualitative improvements (e.g. see <ref>).

§.§ Qualitative Evaluation

<ref> shows visualizations of our model's capabilities on several different characters.
All examples in this figure are trained on sprite sheets which contain 8 – 12 examples of the given character.
Our model learns to generate realistic samples of four-legged animals, two-legged animals, and humanoid shapes.
Furthermore, we see that our model can handle the movement of keypoints that relate to relatively small character parts (e.g. individual feet) as well as keypoints that represent large body parts (e.g. head and torso).
Even when we move multiple keypoints, the resulting image is realistic, adheres to the novel keypoint layout, and leaves areas of the character that were not modified unchanged.

Visualization of what different layers learn ().

<ref> and <ref> show how our model can be used for character animation.
We use the poses from the training set as starting point and linearly interpolate the keypoints to generate the intermediate frames.
Our model produces smooth and realistic results.
We note that these intermediate keypoint locations could also be derived from other data e.g. by extracting keypoints from driving videos [1].
For comparison, we also show the results of a recent general purpose frame-interpolation model DAIN [3].

Mask-based Keypoint Refinement
When moving certain keypoints to a new location we sometimes observe that this leads to “rips” in the generated character since the keypoint is too far away from the main body and such a configuration does not occur in training data.
This can usually be fixed by moving the connected body part in the same direction as the original keypoint.
<ref> shows examples of rips in the generated character (first two rows) and introduced artifacts (third row).
The first column shows the original image and user specified modifications.
The second and third columns show the predicted mask and generated image.
The last column shows our model's final output after the respective connected keypoint was moved automatically.
We can see that by enforcing the mask connectivity constraint we can get more realistic results in all cases.

Layered vs Non-layered Architecture
<ref> shows qualitative comparisons between images generated by our architecture while using either our layered approach or a non-layered approach.
As discussed previously, our layered approach is beneficial when several keypoints overlap in the 2D space, e.g. if a hand passes in front of a body or if two body parts are very close to each other.
<ref> visualizes the features that our model learns for each keypoint layer.
As we can see the model learns to only model the relevant keypoints and their associated body parts for each layer.

Human Reposing
<ref> shows how our model, trained on 10 images of the given character, compares to human reposing models which are pretrained on large datasets.
To make the comparison fairer we chose a relatively “simple” human character, i.e., with a standard body shape and tight clothing.
The first two models (PATN [50] and XinGAN [36]) are pretrained on the Market-1501 dataset [48] and are not capable of modeling this human who is slightly outside of the original training distribution.
We also evaluated these models pretrained on the DeepFashion dataset [24] but the performance was worse.
The LWG [23], pretrained on the iPER dataset [22], performs a fine-tuning step on the same training images as we use before running the animation and is much better at modeling this person.
These results indicate that current human reposing models are not (yet) capable of generalizing to humans outside of their training distribution unless a fine-tuning step on the individual human is performed, hinting at the possibility of using our approach for modeling specific humans or use-cases.
We assume that future human reposing models will be able to model examples similar to the one chosen here and, therefore, suggest that our approach might be better suited to model highly individual persons (e.g., wearing unconventional clothes) or a person performing a very specific movement.

PATN        XinGAN         LWG        Character        Original

[50]        [36]         [23]        GAN

Animation of a real person ().

Discrete appearance changes ().

Original     Predicted Mask     Before Fix     Fixed

Enforcing mask connectivity at test time results in more realistic images.

Automatic Appearance Switching
Our model does not only learn to associate discrete keypoints with given features, but also learns to associate different features with a keypoint based on its location relative to other keypoints.
<ref> shows several examples in which we can see that individual parts get  “flipped” as a function of the location of that keypoint with respect to the others.
As in our previous examples, the features of unrelated keypoints are unaffected by this.

Appearance-specific Keypoints
<ref> shows how our model is able to switch between discrete appearance states for given keypoints.
Each of the characters shows different visual features during training which we encode as different keypoint conditions.
At test time we can switch between these different states to combine novel poses with any of the discrete visual expressions.
Note that, again, our model learns to associate good features with the given keypoints, allowing us to model the poses independently of the discrete keypoint states.

Original     Generated        Original     Generated

Discrete appearance change based on keypoint selection. In this example, the user not only moves keypoints to generate a new pose, but switches the IDs of keypoints to change expression, such as the color and rotation of the characters' faces.

7cNumber of Training Images

5     15     89       5     15     85

Here we show how performance improves with more training images ().

Scaling to Larger Datasets
While we show that our model performs well with only 8 – 15 training images, we also evaluate our model on characters for which we have more training images ($\ge 50$).
<ref> shows how our model scales with larger datasets.
We see that more data is especially helpful when there are overlaps and occlusions.
While the models trained on only 5 or 15 images have difficulty modeling this (e.g. when the hand moves in front of the body), the models which are trained on more images perform much better.

Limitations and Future Work
Our model addresses the main challenge of correctly modeling (dis)occlusions based on limited training data.
However, our model still has no explicit understanding about any underlying 3D representation of the character and <ref> shows how modeling occlusions gets worse with fewer training images.
Extending the model to handle 3D characters is an interesting avenue for future work.
We hypothesize that this may benefit from additional information available in 3D models (such as camera pose).
Additionally, the concept of layering would have to be made more adaptive since different viewpoints would necessitate different layers of keypoints.

§ CONCLUSION

In this work we show how to train GANs on few examples (8 – 15 images) of a given character for few-shot character reposing and animation.
The model is easy to use, requires no expert knowledge, and our layering approach produces realistic results for novel poses and occlusions.
We can perform character reposing in real-time through moving around keypoints and can animate character by interpolating keypoints.
In the future, this can be used with other approaches, e.g. by extracting keypoints from driving videos for character animation.
Through the use of a predicted foreground mask we can also automatically fix keypoint layouts that lead to unrealistic character poses.
Finally, we show that our model learns discrete state changes based on keypoint locations, associates keypoints and their layers with semantic body parts, and scales to larger datasets.

§ ACKNOWLEDGEMENTS

The authors gratefully acknowledge partial support from the German Research Foundation DFG under project CML (TRR 169). We also thank Zuzana Studená who produced some of the artwork used in this work.

[1]
Kfir Aberman, Jing Liao, Mingyi Shi, Dani Lischinski, Baoquan Chen, and Daniel
Neural best-buddies: Sparse cross-domain correspondence.
ACM Transactions on Graphics (TOG), 37(4):69, 2018.

[2]
Yuki M Asano, Christian Rupprecht, and Andrea Vedaldi.
A critical analysis of self-supervision, or what we can learn from a
single image.
In International Conference on Learning Representations, 2020.

[3]
Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan
Depth-aware video frame interpolation.
In Proceedings of the IEEE Computer Vision and Pattern
Recognition, pages 3703–3712, 2019.

[4]
Sagie Benaim, Ron Mokady, Amit Bermano, Daniel Cohen-Or, and Lior Wolf.
Structural-analogy from a single image pair.
arXiv preprint arXiv:2004.02222, 2020.

[5]
Urs Bergmann, Nikolay Jetchev, and Roland Vollgraf.
Learning texture manifolds with the periodic spatial gan.
In International Conference on Machine Learning, pages
469–477, 2017.

[6]
Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros.
Everybody dance now.
In Proceedings of the IEEE International Conference on Computer
Vision, pages 5933–5942, 2019.

[7]
Marek Dvorožňák, Wilmot Li, Vladimir G. Kim, and Daniel Sýkora.
ToonSynth: Example-based synthesis of hand-colored cartoon
ACM Transactions on Graphics, 37(4), 2018.

[8]
Christophe Fiorio and Jens Gustedt.
Two linear time union-find strategies for image processing.
Theoretical Computer Science, 154(2):165–181, 1996.

[9]
Yossi Gandelsman, Assaf Shocher, and Michal Irani.
"double-dip": Unsupervised image decomposition via coupled
In Proceedings of the IEEE Computer Vision and Pattern
Recognition, 2019.

[10]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,
Sherjil Ozair, Aaron Courville, and Yoshua Bengio.
Generative adversarial nets.
In Advances in Neural Information Processing Systems, pages
2672–2680, 2014.

[11]
Tobias Hinz, Matthew Fisher, Oliver Wang, and Stefan Wermter.
Improved techniques for training single-image gans.
In IEEE Winter Conference on Applications of Computer Vision,
pages 1300–1309, 2021.

[12]
Tobias Hinz, Stefan Heinrich, and Stefan Wermter.
Generating multiple objects at spatially distinct locations.
International Conference on Learning Representations, 2019.

[13]
Tobias Hinz, Stefan Heinrich, and Stefan Wermter.
Semantic object accuracy for generative text-to-image synthesis.
IEEE Transactions on Pattern Analysis and Machine Intelligence,

[14]
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.
Image-to-image translation with conditional adversarial networks.
In Proceedings of the IEEE Computer Vision and Pattern
Recognition, pages 1125–1134, 2017.

[15]
Nikolay Jetchev, Urs Bergmann, and Roland Vollgraf.
Texture synthesis with spatial generative adversarial networks.
In Advances in Neural Information Processing Systems Workshop,

[16]
Justin Johnson, Alexandre Alahi, and Li Fei-Fei.
Perceptual losses for real-time style transfer and super-resolution.
In European Conference on Computer Vision, pages 694–711.
Springer, 2016.

[17]
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and
Timo Aila.
Training generative adversarial networks with limited data.
In Advances in Neural Information Processing Systems, 2020.

[18]
Chuan Li and Michael Wand.
Precomputed real-time texture synthesis with markovian generative
adversarial networks.
In European Conference on Computer Vision, pages 702–716,

[19]
Yijun Li, Richard Zhang, Jingwan Cynthia Lu, and Eli Shechtman.
Few-shot image generation with elastic weight consolidation.
Advances in Neural Information Processing Systems, 33, 2020.

[20]
Jianxin Lin, Yingxue Pang, Yingce Xia, Zhibo Chen, and Jiebo Luo.
Tuigan: Learning versatile image-to-image translation with two
unpaired images.
In European Conference on Computer Vision, 2020.

[21]
Songrun Liu, Alec Jacobson, and Yotam Gingold.
Skinning cubic bézier splines and catmull-clark subdivision
ACM Transactions on Graphics (TOG), 33(6):1–9, 2014.

[22]
Wen Liu, Zhixin Piao, Jie Min, Wenhan Luo, Lin Ma, and Shenghua Gao.
Liquid warping gan: A unified framework for human motion imitation,
appearance transfer and novel view synthesis.
In Proceedings of the IEEE International Conference on Computer
Vision, 2019.

[23]
Wen Liu, Zhixin Piao, Zhi Tu, Wenhan Luo, Lin Ma, and Shenghua Gao.
Liquid warping gan with attention: A unified framework for human
image synthesis.
arXiv preprint arXiv:2011.09055, 2020.

[24]
Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang.
Deepfashion: Powering robust clothes recognition and retrieval with
rich annotations.
In Proceedings of the IEEE Computer Vision and Pattern
Recognition, 2016.

[25]
Mehdi Mirza and Simon Osindero.
Conditional generative adversarial nets.
arXiv preprint arXiv:1411.1784, 2014.

[26]
Sangwoo Mo, Minsu Cho, and Jinwoo Shin.
Freeze discriminator: A simple baseline for fine-tuning gans.
arXiv preprint arXiv:2002.10964, 2020.

[27]
Atsuhiro Noguchi and Tatsuya Harada.
Image generation from small datasets via batch statistics adaptation.
In Proceedings of the IEEE International Conference on Computer
Vision, pages 2750–2758, 2019.

[28]
Taesung Park, Alexei A Efros, Richard Zhang, and Jun-Yan Zhu.
Contrastive learning for unpaired image-to-image translation.
In European Conference on Computer Vision, 2020.

[29]
Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu.
Semantic image synthesis with spatially-adaptive normalization.
In Proceedings of the IEEE Computer Vision and Pattern
Recognition, pages 2337–2346, 2019.

[30]
Omid Poursaeed, Vladimir Kim, Eli Shechtman, Jun Saito, and Serge Belongie.
Neural puppet: Generative layered cartoon characters.
In IEEE Winter Conference on Applications of Computer Vision,
pages 3346–3356, 2020.

[31]
Scott E Reed, Zeynep Akata, Santosh Mohan, Samuel Tenka, Bernt Schiele, and
Honglak Lee.
Learning what and where to draw.
In Advances in Neural Information Processing Systems, pages
217–225, 2016.

[32]
Esther Robb, Wen-Sheng Chu, Abhishek Kumar, and Jia-Bin Huang.
Few-shot adaptation of generative adversarial networks.
arXiv preprint arXiv:2010.11943, 2020.

[33]
Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli.
Singan: Learning a generative model from a single natural image.
In Proceedings of the IEEE International Conference on Computer
Vision, pages 4570–4580, 2019.

[34]
Assaf Shocher, Shai Bagon, Phillip Isola, and Michal Irani.
Ingan: Capturing and retargeting the dna of a natural image.
In Proceedings of the IEEE International Conference on Computer
Vision, pages 4492–4501, 2019.

[35]
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and
Nicu Sebe.
First order motion model for image animation.
In Advances in Neural Information Processing Systems, 2019.

[36]
Hao Tang, Song Bai, Li Zhang, Philip HS Torr, and Nicu Sebe.
Xinggan for person image generation.
In European Conference on Computer Vision, 2020.

[37]
Ngoc-Trung Tran, Viet-Hung Tran, Ngoc-Bao Nguyen, Trung-Kien Nguyen, and
Ngai-Man Cheung.
Towards good practices for data augmentation in gan training.
arXiv preprint arXiv:2006.05338, 2020.

[38]
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky.
Deep image prior.
In Proceedings of the IEEE Computer Vision and Pattern
Recognition, pages 9446–9454, 2018.

[39]
Yael Vinker, Eliahu Horwitz, Nir Zabari, and Yedid Hoshen.
Deep single image manipulation.
arXiv preprint arXiv:2007.01289, 2020.

[40]
Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan
High-resolution image synthesis and semantic manipulation with
conditional gans.
In Proceedings of the IEEE Computer Vision and Pattern
Recognition, pages 8798–8807, 2018.

[41]
Yaxing Wang, Abel Gonzalez-Garcia, David Berga, Luis Herranz, Fahad Shahbaz
Khan, and Joost van de Weijer.
Minegan: effective knowledge transfer from gans to target domains
with few images.
In Proceedings of the IEEE Computer Vision and Pattern
Recognition, pages 9332–9341, 2020.

[42]
Yaxing Wang, Chenshen Wu, Luis Herranz, Joost van de Weijer, Abel
Gonzalez-Garcia, and Bogdan Raducanu.
Transferring gans: generating images from limited data.
In European Conference on Computer Vision, pages 218–234,

[43]
Kesheng Wu, Ekow Otoo, and Arie Shoshani.
Optimizing connected component labeling algorithms.
In Medical Imaging 2005: Image Processing, volume 5747, pages
1965–1976, 2005.

[44]
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.
The unreasonable effectiveness of deep features as a perceptual
In Proceedings of the IEEE Computer Vision and Pattern
Recognition, pages 586–595, 2018.

[45]
Miaoyun Zhao, Yulai Cong, and Lawrence Carin.
On leveraging pretrained gans for generation with limited data.
In International Conference on Machine Learning, 2020.

[46]
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han.
Differentiable augmentation for data-efficient gan training.
In Advances in Neural Information Processing Systems, 2020.

[47]
Zhengli Zhao, Zizhao Zhang, Ting Chen, Sameer Singh, and Han Zhang.
Image augmentations for gan training.
arXiv preprint arXiv:2006.02595, 2020.

[48]
Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian.
Scalable person re-identification: A benchmark.
In Proceedings of the IEEE International Conference on Computer
Vision, pages 1116–1124, 2015.

[49]
Yang Zhou, Zhen Zhu, Xiang Bai, Dani Lischinski, Daniel Cohen-Or, and Hui
Non-stationary texture synthesis by adversarial expansion.
ACM Transactions on Graphics (TOG), 37(4):1–13, 2018.

[50]
Zhen Zhu, Tengteng Huang, Baoguang Shi, Miao Yu, Bofei Wang, and Xiang Bai.
Progressive pose attention transfer for person image generation.
In Proceedings of the IEEE Computer Vision and Pattern
Recognition, pages 2347–2356, 2019.

[51]
Maria Zontak and Michal Irani.
Internal statistics of a single natural image.
In Proceedings of the IEEE Computer Vision and Pattern
Recognition, pages 977–984, 2011.