# Learning Cooperative Multi-Agent Policies with Partial Reward Decoupling

Benjamin Freed∗1, Aditya Kapoor∗1, Ian Abraham2, Jeff Schneider1, and Howie
Choset1 Manuscript received: September 9, 2021; Revised: November 25, 2021;
Accepted: November 29, 2021.This paper was recommended for publication by
Editor Jens Kober upon evaluation of the Associate Editor and Reviewers’
comments.1 These authors are with the Robotics Institute at Carnegie Mellon
University, Pittsburgh, PA 15213, USA.
<EMAIL_ADDRESS>Ian Abraham is with the
Mechanical Engineering Department at Yale University<EMAIL_ADDRESS>
###### Abstract

One of the preeminent obstacles to scaling multi-agent reinforcement learning
to large numbers of agents is assigning credit to individual agents’ actions.
In this paper, we address this credit assignment problem with an approach that
we call partial reward decoupling (PRD), which attempts to decompose large
cooperative multi-agent RL problems into decoupled subproblems involving
subsets of agents, thereby simplifying credit assignment. We empirically
demonstrate that decomposing the RL problem using PRD in an actor-critic
algorithm results in lower variance policy gradient estimates, which improves
data efficiency, learning stability, and asymptotic performance across a wide
array of multi-agent RL tasks, compared to various other actor-critic
approaches. Additionally, we relate our approach to counterfactual multi-agent
policy gradient (COMA), a state-of-the-art MARL algorithm, and empirically
show that our approach outperforms COMA by making better use of information in
agents’ reward streams, and by enabling recent advances in advantage
estimation to be used.

###### Index Terms:

Reinforcement Learning, Multi-Robot Systems, Cooperating Robots

††publicationid: pubid: 0000–0000/00$00.00 © 2021 IEEE

##  I Introduction

Many difficult problems faced in robotics and AI involve large multi-agent
systems, such as networks of autonomous vehicles, drone swarms, factory robot
teams, or network routing. Recently, multi-agent reinforcement learning (MARL)
has shown promise on many of these problems, by enabling highly complex and
cooperative strategies to be automatically learned, as opposed to manually
programmed. For example, MARL algorithms have achieved super-human performance
on complex multi-agent strategy games such as DOTA 2 [1], Starcraft II [2],
and capture the flag [3].

While MARL approaches have achieved impressive performance on these tasks,
they require immense amounts of data and computing resources to train
effectively, often requiring on the order of millions or billions of
environmental interactions, necessitating costly GPU clusters. The
computational difficulty of large multi-agent RL (MARL) problems stems largely
from the credit assignment problem: identifying which agents’ actions had a
positive or negative impact on total group reward becomes increasingly
difficult as the number of agents increases [4, 5].

Figure 1: Examples of problems in which PRD is applicable. Driving Scenario:
depicted in the left side, cars receive a negative reward for colliding. The
two orange and blue cars are on a collision course, meaning that each one’s
actions at the current timestep impact the future reward of the other, and
they therefore cannot be decoupled using PRD. Note that neither orange car is
at risk of colliding with either blue car, and therefore the orange and blue
cars can be decoupled from one another using PRD. Factory Scenario: depicted
in the right side, robot 1 is rewarded for placing wheels on the conveyor
belt, robot 2 is rewarded for placing car bodies on the belt, and robot 3 is
rewarded for successfully assembling the components. Here, robots 1 and 2 may
be decoupled using PRD as each one does not impact each other’s rewards.
However, robot 1 and robot 2 cannot be decoupled from robot 3, because robot
3’s ability to assemble cars and achieve reward is dependent on the actions of
robots 1 and 2.

In this paper, we address the difficulties associated with learning in large
team sizes by streamlining credit assignment. In model-free policy gradient-
style algorithms (such as Actor Critic [6], Proximal Policy Optimization [7],
Trust Region Policy Optimization [8], and Soft Actor-Critic [9]), we argue
that the credit assignment problem manifests itself as a decrease in the
signal-to-noise ratio in the learning update as the team size increases. This
viewpoint unifies two concurrent lines of work in model-free RL: credit
assignment, and variance reduction.

One way to combat the increased difficulty of credit assignment employs reward
shaping [10], wherein the reward function is meticulously hand-engineered by
the algorithm designer to help guide the learning process. For example, if it
is known that an action taken by a particular agent has a deleterious impact
on team performance, one may assign an explicit negative reward to that agent.
Many prominent MARL approaches incorporate some form of reward shaping.
However, reward shaping hardly offers a general solution to the credit
assignment problem; for one, it may require a high degree of domain-specific
knowledge, or tedious hyper-parameter tuning. Additionally, a poorly designed
reward function may limit the asymptotic performance of the team of agents,
for example by making agents act selfishly or competitively when full
cooperation is ideal.

To address the credit assignment issue, we introduce a technique that we term
partial reward decoupling (PRD), which dynamically decomposes large groups of
agents solving a cooperative task into smaller subgroups. This decomposition
is performed in such a way that if each agent learns how to optimally
cooperate with agents within its subgroup, then the agents will also achieve
optimal group-level cooperation. Decomposing large multi-agent problems into
smaller subproblems involving fewer agents reduces the number of agents among
which credit must be distributed, thereby simplifying credit assignment.
Practically, within a policy gradient-style algorithm, we find that this
streamlining of credit assignment results in lower variance policy-gradient
estimates, therefore improving the data efficiency and stability of learning.

PRD relies on the fact that, for many problems, it is the case that not every
agent’s actions are relevant to all other agents’ rewards. More precisely, if
a particular agent’s action only impacts the future rewards of a subset of
other agents’ individual rewards, then that agent need only explicitly attempt
to cooperate with that subset of agents. We show how this property can be used
to improve the efficiency of the actor-critic algorithms [6], by filtering out
the contributions of non-essential agents from policy gradient estimates. We
use a value function with an attention mechanism to efficiently estimate which
agents can be decoupled from one another. We empirically show that our
modified actor-critic algorithm (which we term partial reward-decoupled actor-
critic, PRD-AC) demonstrates more efficient and stable learning across a wide
array of benchmarks, compared to conventional actor-critic, and a naive
“greedy” version of actor-critic, wherein each agent attempts to maximize its
own individual reward. The code used to perform all experiments is available
https://github.com/biorobotics/PRD.

##  II Background

###  II-A Multi-Agent Markov Decision Process

We consider multi-agent sequential decision-making problems that can be
modeled as multi-agent Markov decision processes (MMDPs) containing $M$ agents
[11]. An MMDP is defined by
$(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\rho_{0},\gamma)$, where
$\mathcal{S}$ is the state space, $\mathcal{A}$ is the joint action space,
consisting of every possible combination of individual agents’ actions,
$\mathcal{P}(s_{t+1}|s_{t},a_{t})$ specifies the state transition probability
distribution, $\mathcal{R}(r_{t}|s_{t},a_{t})$ specifies the reward
distribution, $\rho_{0}(s_{0})$ denotes the initial state distribution, and
$\gamma\in(0,1]$ denotes a discount factor. At each timestep, each agent
$i\in\\{1,...,M\\}$ selects an action independently according to its state-
conditioned policy $\pi_{i}(a_{t}^{(i)}|s^{(i)}_{t};\theta_{i})$, where
$s^{(i)}_{t}$ denotes the state information available to agent $i$ (which may
be the full state, as assumed in Sec. V), and $\theta_{i}$ denotes the
parameters for agent $i$. Each agent subsequently earns an individual reward
$r^{(i)}_{t}$ sampled from $\mathcal{R}$ (that is,
$r^{(1)}_{t},...,r^{(M)}_{t}\sim\mathcal{R}(\cdot|s_{t},a_{t})$), and the
environment undergoes a state transition,
$s_{t+1}\sim\mathcal{P}(\cdot|s_{t},a_{t})$.

We focus on efficiently solving fully cooperative tasks, wherein each agent
attempts to maximize the sum of all agents’ individual rewards. More
precisely, we wish to find the optimal agent policy parameters
$\theta^{*}=\\{\theta_{1}^{*},...,\theta_{M}^{*}\\}$ that solve
$\theta^{*}=\underset{\theta}{\mathrm{argmax}}\ J(\theta)$, where

$J(\theta)=\mathbb{E}\Big{[}\sum_{t=0}^{T}\sum_{j=1}^{M}\gamma^{t}r_{t}^{(j)}\Big{]},$
(1)

with $T$ denoting the length of the episode. By the policy gradient theorem
[12], the gradient of $J$ with respect to the policy parameters $\theta_{i}$
of a particular agent $i$ is given by

$\nabla_{\theta_{i}}J(\theta)=\mathbb{E}\Big{[}\nabla_{\theta_{i}}\log\pi_{i}(a_{t}^{(i)}|s_{t})\\\
\sum_{j=1}^{M}\left(Q_{j}^{\pi}(s_{t},a_{t})-b_{ij}(s_{t},a_{t}^{-i})\right)\Big{]},$
(2)

where $Q_{j}^{\pi}(s_{t},a_{t})$ denotes the expected future reward for a
particular agent $j$, and $b_{ij}(s_{t},a_{t}^{-i})$ is commonly known as a
baseline function, which can be any any function that depends on the state,
and the actions of all agents except $i$ at time $t$ [13]. $b_{ij}$ cannot
depend on the action of agent $i$ because this would result in biased policy-
gradient estimates. Often a learned value function is used for the baseline.

In the MMDP formulation we consider, agents earn individual rewards; that is,
each agent has its own unique stream of rewards. Several past approaches to
cooperative MARL have considered settings with global rewards, in which the
entire team earns one scalar reward value at each timestep. We focus on the
individual reward setting for two main reasons: 1) individual rewards are more
general than global rewards, and 2) individual rewards are relatively common
in real-world settings.

Individual rewards are more general than global rewards because any global
reward function can be expressed as an individual reward function in which
individual rewards across all agents are the same (i.e.,
$r^{(i)}_{t}=r^{(j)}_{t}=r^{global}_{t},\quad\forall i,j\in\\{1,...,M\\}$).
However, not all individual reward functions can be expressed as a global
reward function, particularly in competitive or mixed cooperative-competitive
problems such as those considered in [14].

We also argue that individual rewards are relatively common in real-world
settings. For example, in vehicle routing, a natural reward function for a
given agent is the rate at which it delivers goods to a customer. Similarly,
in a soccer game, individual rewards may be assigned to an agent that scores a
goal. Critically, we do not assume that a satisfactory cooperative solution
may be achieved if each agent selfishly maximizes its own reward, which is why
this paper proposes algorithms for maximizing total group reward. Note that
credit assignment is still necessary even if individual rewards are assumed,
because in general it remains unclear what the influence of a particular agent
was on each agents’ individual rewards.

##  III Partial Reward Decoupling

Many popular RL algorithms, such as SAC [9], PPO [7], TRPO [8], D4PG [15],
MADDPG [14], and COMA [16] work by estimating the gradient specified in eq.
(2). However, in a MARL context, these gradient estimates are often very noisy
due to the fact that they are essentially a sum over contributions from
(potentially many) agents, each of which contributes noise to the policy
gradient estimate. Luckily, it is often the case that large-scale cooperative
multi-agent problems can be thought of as a collection of decoupled
subproblems, involving only a fraction of the entire group. For example, in
the robot assembly problem depicted in the right of Figure 1, imagine that
robots 1 and 2 are both rewarded for placing items on the belt, and do not
influence the quality with which the other can perform its job, when
functioning properly. Therefore, robots 1 and 2 may be decoupled (that is,
treated as belonging to separate subteams), and do not need to explicitly
cooperate. However, both robots 1 and 2 influence the quality with which robot
3 completes a downstream process, and therefore neither robots 1 or 2 can be
decoupled from robot 3. If robot 1 were to malfunction and interfere with
robot 2’s ability to perform its function by blocking it, this would also
introduce coupling between robots 1 and 2.

While in the robot assembly example the possible decouplings are fairly
obvious, for many problems the decoupling is non-obvious, and may vary with
time. For example, in the driving scenario depicted on the left of the figure,
for a portion of the time it may be possible to decouple an agent from all
other agents. For example, if we are confident that two vehicles do not
influence each other at a particular time, or any time in the future, the
vehicles may be decoupled and treated as parts of two separate RL problems.
However, when a vehicle enters a busy intersection, or is at risk of colliding
with another vehicle, it should be considered coupled to other vehicles, at
least momentarily.

Our proposed approach, PRD, decomposes multi-agent problems based on which
agents’ actions impact the future reward of other agents. If a particular
agent $i$ does not impact the expected future reward of another agent $j$,
then agent $i$ can be decoupled from agent $j$. In the context of a policy
gradient-style algorithm, this decoupling means that the contribution of agent
$j$’s rewards can be removed from the gradient estimate of agent $i$, without
introducing bias. We claim PRD addresses the credit assignment problem,
because credit for agent i’s rewards is assigned only to those agents that had
an influence on agent i’s rewards. PRD can also be seen as a variance
reduction technique, because in a policy gradient-style algorithm, we expect
this decoupling to result in lower variance policy gradient estimates, as each
policy gradient estimate would consider only the contributions of a smaller
subset of agents, thus reducing the amount of noise that is injected.

######  Definition III.1.

The subset of agents that are impacted by the action of agent $i$ at time $t$
is defined as the relevant set of agent $i$, $R^{\pi}_{i}(s_{t},a_{t})$, which
is defined implicity such that

$\mathbb{E}\big{[}\sum_{\tau=t}^{T}r_{\tau}^{(j)}|s_{t},a_{t}\big{]}=\\\
\mathbb{E}\big{[}\sum_{\tau=t}^{T}r_{\tau}^{(j)}|s_{t},\\{a^{(i)}_{t}|i\in\\{1,...,M\\},j\in
R_{i}^{\pi}(s_{t},a_{t})\\}\big{]}.$ (3)

That is, the expected future reward of a particular agent $j$, given the
current state of the environment, only depends on the actions of agents that
include $j$ in their relevant set. Note that $R_{i}^{\pi}$ changes with time.

We include the joint policy $\pi$ in the superscript because the relevant set
depends on the joint policy of all the agents.

######  Theorem III.1.

Using the definition of the relevant set presented in (3), the gradient update
of agent $i$ can be simplified by removing the contributions of agents not in
the relevant set of agent $i$. That is, the policy gradient of agent $i$ can
be simplified to (omitting the arguments of the Q functions, baselines, and
the relevant set for brevity)

$\nabla_{\theta_{i}}J(\theta)=\mathbb{E}\Big{[}\nabla_{\theta_{i}}\log\pi_{i}(a_{t}^{(i)}|s_{t})\sum_{j\in
R^{\pi}_{i}}\big{(}Q_{j}^{\pi}-b_{ij}\big{)}\Big{]}.$ (4)

###### Proof.

If agent $j$ is outside of the relevant set of agent $i$, then $Q^{\pi}_{j}$
does not depend on $a_{t}^{(i)}$, and can therefore be subtracted out as an
action-independent baseline without biasing the policy gradient. That is,
$b_{ij}(s_{t},a_{t}^{-i})=Q^{\pi}_{j}$, given that $j\notin
R^{\pi}_{i}(s_{t},a_{t})$. Therefore, terms corresponding to $j\notin
R^{\pi}_{i}(s_{t},a_{t})$ do not contribute to summation in (2), and may
therefore be removed. ∎

Omitting the contributions of irrelevant agents from the learning update of
agent $i$ can be seen as decomposing a (potentially very large) cooperative
multi-agent problem involving $M$ agents into a set of smaller ones, involving
only $|R_{i}(s_{t},a_{t})|$ agents, at each timestep $t$. However, unlike a
static decomposition that may attempt to assign agents to static subgroups,
partial reward decoupling assigns agents to subgroups that are overlapping
(i.e., an agent may belong to multiple subgroups simultaneously) and dynamic
(i.e., relevant set assignments change from timestep to timestep). In the
following section we explain how we use learned value functions with attention
mechanisms to efficiently infer the relevant set for each agent.

###  III-A Estimating the Relevant Set

In practice, because the dynamics of the environment and reward function are
unknown to agents, the relevant set must be estimated empirically through
interactions with the environment. To estimate the relevant set of each agent,
we utilize the common technique of value function estimation [12, 14, 6, 17].
During training, we maintain a learned value function $V^{\pi}_{ij}$ that
attempts to estimate the expected individual future reward for each agent $j$,
given the current state and the actions of all agents except $i$, which we
explain further in Sec. IV-A. We acknowledge that this definition of a value
function differs from past works where value was conditioned only on the
state, we find this extension useful because it enables us to use
$V^{\pi}_{ij}$ both as a baseline $b_{ij}$, and for relevant set estimation.
While it may appear that estimating $V^{\pi}_{ij}\quad\forall i,j$ requires
$M^{2}$ different learned value functions (one for every combination of $i$
and $j$), in practice we find that we can efficiently estimate $V^{\pi}_{ij}$
for any $(i,j)$ with one set of value function parameters, using the
architecture described in Sec. IV-A and by sharing weights across agents.

Recall that, if the expected future reward of agent $j$ does not depend on the
action of agent $i$ at a particular time $t$, then $j$ is not in agent $i$’s
relevant set at time $t$. This property justifies the use of the value
function to estimate the relevant set; if our value estimate for agent $j$
does not depend on $a_{t}^{(i)}$, we infer $j\notin R^{\pi}_{i}(s_{t},a_{t})$.
We use a value function with an attention mechanism over agent actions, which
allows dependence on particular agents’ actions to be “shut off” as a function
of the current state of the environment, by setting the attention weight for
the corresponding action to 0.

In the following section, we use this observation to modify the popular actor-
critic algorithm [6] to help solve the credit assignment problem, by
selectively removing the contributions of agents not within the relevant set
from estimated policy gradients.

##  IV Partial Reward Decoupled Actor-Critic

To create an efficient MARL algorithm using PRD, we combine PRD with the
actor-critic algorithm [6], which we refer to as PRD Actor-Critic (PRD-AC). In
PRD-AC, value function estimation is used to determine which advantage terms
may be eliminated from the learning updates. Here, PRD can be seen as a
variance reduction technique, because it removes unnecessary advantage terms
that inject noise into the gradients.

###  IV-A Actor and Critic Architectures

Each agent maintains identical copies of an actor network and a critic
network. The actor network is a multi-layer perceptron (MLP), which takes the
local observation of the current agent, concatenated with all other agents’
local observations. To support decentralized execution, local observations
along with messages from other agents may be provided as input to the actor
network instead [18, 19]. The critic network is a graph neural network (GNN)
with a graph attention mechanism [20] that outputs $V^{\pi}_{ij}$. We do not
condition $V_{ij}^{\pi}$ on the action of agent $i$, so that in addition to
estimating the relevant set, $V_{ij}^{\pi}$ can be used as a baseline
($b_{ij}$ form eq. (2)) [6, 13]. Note that, in general, $V_{ij}^{\pi}\neq
V_{kj}^{\pi}$ for some $k\neq i$, because $V_{ij}^{\pi}$ and $V_{kj}^{\pi}$
condition on the actions of different sets of agents.

The graph attention mechanism used in the critic network is based on scaled
dot product attention [21]. For each agent (corresponding to a node in the
GNN), at each timestep, state components pertaining to individual agents
(i.e., position, velocity, and goal position) are pre-processed using a MLP,
and then used to compute keys and queries. Attention values for each agent $k$
are computed using the pre-processed state and action of agent $k$, with the
exception that $a_{t}^{(i)}$ is replaced with $\pi^{(i)}(s_{t})$, to avoid
conditioning agent $i$’s baseline $V_{ij}^{\pi}$ on its own action. Keys,
queries, and values are subsequently passed through a single-head self-
attention (SSA) mechanism, which combines attention values from each agent
according to a weighted sum [21]. The output of the SSA is passed through a
linear layer to compute $V^{\pi}$, an $M$-by-$M$ matrix of value estimates
$V_{ij}^{\pi}$ for each $i,j\in\\{1,...,M\\}$. The SSA also outputs the weight
matrix $W$, where $W[k,j]$ (which we refer to as $w_{kj}(s_{t})$) indicates
how much agent $j$ attends to the observation-action of agent $k$ while
computing $V_{ij}^{\pi}$, for each possible value of $i$ (that is,
$V_{ij}^{\pi}$ places the same attention weight on the observation-action for
agent $k$ when estimating the expected future rewards for agent $j$,
regardless of $i$. While theorem III.1 motivates using attention weights over
actions only, we found that attending over observations and actions stabilized
training.

###  IV-B Actor Update

The policy update for agent $i$ in PRD-AC is similar to that of a standard
actor-critic algorithm, with the primary exception being that advantage terms
outside of agent $i$’s relevant set are removed. Note that, because $w_{ij}$
varies with time, so does the estimated relevant set, enabling decoupling to
be dynamic with time. After each episode, we compute one gradient-based policy
update. The gradient estimate for the policy parameters of agent $i$ is given
by

$g_{i}=\sum_{t=0}^{T}\nabla_{\theta_{i}}\log\pi_{i}(a^{(i)}_{t}|s_{t})\sum_{j:w_{ij}(s_{t})>\epsilon}\hat{A}_{ij}(s_{t},a_{t}),$
(5)

where $\hat{A}_{ij}(s_{t},a_{t})$ is the advantage estimate computed using the
actual rewards of agent $j$, and the critic-estimated expected rewards of
agent $j$, not conditioned on the actions of agent $i$. There are many ways
this advantage may be estimated, but we choose to use generalized advantage
estimation (GAE) [22], with $\lambda=0.98$. Here, $\epsilon$ is the threshold
above which we consider $w_{ij}$ to be relevant. In practice, we initialize
$\epsilon$ to 0 (making the algorithm initially equivalent to standard actor-
critic), and linearly increase it over the first 15K episodes to a maximum
value of 0.01, which we found empirically to stabilizes training, by allowing
the critic time to learn how to correctly assign attention weights.

###  IV-C Critic Update

The output of the critic network is regressed against the TD($\lambda$) target
value estimate, with $\lambda=0.8$ [5]. After each episode of environmental
interaction, we apply one gradient update to the parameters of the critic
network. The gradients used to update the critic loss are obtained by
differentiating the Huber loss [23] between the critic network outputs and
TD($\lambda$) value estimates [5] for the last episode, which is used to
compute the critic learning update. Huber loss is similar to the more
commonly-used MSE, but is more robust to outliers, which we found stabilized
training.

###  IV-D Relation to Counterfactual Multi-Agent Policy Gradient

Counterfactual multi-agent policy gradient (COMA) is another actor-critic
algorithm that addresses the credit assignment problem, and to the best of our
knowledge is the existing algorithm most closely related to PRD-AC. PRD-AC
retains the benefits of COMA, and extends them in two key ways: 1) when
individual rewards are available, PRD-AC can take advantage of the additional
information contained in the individual reward streams to further streamline
the gradient updates, and 2) PRD enables a wide range of advantage estimation
techniques to be used (for example, GAE, which we use in our experiments). In
practice, we find that both of these advantages yield substantial improvements
in learning speed and asymptotic performance (see sec. V-C).

Adapting notation to fit the rest of the paper, the COMA gradient estimator
can be expressed as

$g_{i}^{\text{COMA}}=\nabla_{\theta_{i}}\log\pi_{i}(a_{t}^{(i)}|s_{t}^{(i)})A^{(\text{COMA},i)}(s_{t},a_{t}),$
(6)

where the COMA advantage estimate is given by

$A^{(\text{COMA},i)}(s_{t},a_{t})=Q^{\pi}(s_{t},a_{t})-\mathbb{E}_{a_{t}^{(i)}}[Q^{\pi}(s_{t},a_{t})].$
(7)

COMA has a similar benefit to PRD-AC in that, if the action of agent $i$ does
not impact the expected future group reward, its influence from the gradient
estimate is eliminated. This is captured implicitly in COMA by virtue of the
fact that the COMA advantage estimate $A^{(COMA,i)}$ is $0$ when agent $i$’s
action does not impact expected future group reward (i.e., when
$Q^{\pi}(s_{t},a_{t})=\mathbb{E}_{a_{t}^{(i)}}[Q^{\pi}(s_{t},a_{t})$. This
effect is captured in PRD-AC because if agent $i$’s action does not impact
group rewards (that is, if
$Q_{j}^{\pi}(s_{t},a_{t})=\mathbb{E}_{a_{t}^{(i)}}[Q_{j}^{\pi}(s_{t},a_{t})],\quad\forall
j$, then agent $i$’s relevant set is empty, and agent $i$’s advantage terms
are omitted from the gradient estimate. However, PRD-AC extends COMA by making
use of additional information in individual agents’ reward streams, if they
are available. While COMA can only eliminate advantage terms when the
corresponding agents’ actions have no impact on total expected group reward,
PRD-AC can eliminate advantage terms on an individual-agent basis; for
example, if agent $i$’s action does not impact the expected future reward of
agent $j$, then $\hat{A}_{ij}$ may be eliminated. The ability to remove
advantages on an individual-agent basis is beneficial because often it may be
the case that a particular agent’s action impacts only a subset of other
agents’ rewards, allowing us to streamline that agent’s learning update,
despite the fact that agent $i$’s action may still impacts future group
reward, for instance by impacting other agents besides $j$.

The second way PRD-AC extends COMA is by enabling a wider range of advantage
estimation techniques to be used. While COMA assumes advantage estimates of
the form shown in eq. (7), PRD-AC places no restriction on the advantage
estimation technique used, and we find that other advantage estimation
techniques (in particular, GAE) offer improved performance.

##  V Experimental Results

We experimentally evaluate the performance of PRD-AC on a range of benchmarks
designed to require various levels and types of cooperation (e.g., cooperation
based on teams, or based on physical proximity). We compare average episode
reward vs. elapsed training episodes PRD-AC to COMA, as well as two other
baseline actor-critic algorithms, one that encourages full cooperation
(Shared-AC), and one that encourages agents to act selfishly (Greedy-AC). To
validate our claim that PRD-AC reduces policy gradient estimator variance, we
compare the variance of gradient estimates of PRD-AC to an actor-critic
algorithm in which no decoupling occurs (Shared-AC). Finally, to better
understand the behavior of PRD-AC, we perform a series of ablation experiments
that interpolate between PRD-AC and COMA.

###  V-A Comparison to Greedy and Shared-Reward Actor-Critic

To evaluate the effect of PRD on learning efficiency and asymptotic
performance, we compare our PRD-AC algorithm (see Section IV) to the following
baseline approaches on a set of multi-agent RL tasks:

  * •

Greedy Actor-Critic (Greedy-AC): each agent attempts to maximize its own
individual reward.

  * •

Shared-Reward Actor-Critic (Shared-AC): each agent attempts to maximize the
sum of group rewards.

  * •

COMA.

We considered greedy-AC as a baseline because it represents a naive form of
reward shaping, that may work well for many problems in which little to no
cooperation is required. Greedy-AC simplifies the learning problem, because
each agent need not consider its impact on the rewards of other agents.
However, it does not encourage cooperation among agents. We additionally
considered shared-AC as a baseline because a shared reward function in which
every agent optimizes its policy for total group reward should discover fully
cooperative policies that are optimal on a group level. The primary drawback
to shared-reward AC is that it has a higher variance learning signal, because
it must consider the rewards from all agents. We hypothesize that PRD-AC will
combine the benefits of both greedy and shared-reward actor-critic by allowing
each agent to selectively consider only the rewards of other agents that are
impacted by its behavior in its learning updates. Intuitively, for each agent
$i$, PRD-AC attempts to find the smallest set of other agents that agent $i$
must cooperate with (i.e., its relevant set) so as to to yield group-level
solutions; in essence, PRD-AC acts as greedily as possible while still being
fully cooperative.

We evaluate PRD-AC, greedy-AC, and shared-AC in three distinct task settings,
which we refer to as paired-agent goal finding, collision avoidance, and
social dilemma. Each task requires agents to navigate from randomly-selected
start locations to randomly-selected goal locations. In each task, agents
modeled as point masses in a 2-dimensional planar environment. The chosen
evaluation tasks require varying degrees of cooperation; additionally, each
task involves unique strategies for ideal relevant set selection. For each
experiment, we run 5 distinct trials of each algorithm (unless otherwise
indicated) with distinct random seeds. We plot the mean episode reward across
all trials, as well as a shaded region indicating 1 standard deviation above
and below the mean.

Paired-Agent Goal-Finding: agents are randomly paired with exactly 1 other
partner agent. Agents must navigate from their initial location to their goal
location, but are not rewarded themselves for reaching their goal. Instead,
each agent is rewarded based on how much closer their partner gets to its
goal. This task is intended to be a somewhat contrived toy problem, designed
to present a situation in which a greedy approach fails catastrophically
because each agent’s individual reward is independent of its actions.
Therefore, the task cannot be effectively solved by having every agent attempt
to maximize its own individual reward.

We compare PRD-AC, greedy-AC, and shared-AC in a task with 30 agents. As
predicted, greedy-AC fails to make significant progress towards solving the
task (Fig. 2). PRD-AC outperforms the shared reward strategy, achieving a
higher reward more rapidly. COMA learned more rapidly than PRD-AC, but
suffered from instability.

Figure 2: Mean episode reward vs. episodes for paired agent, collision
avoidance, and social dilemma tasks. PRD-AC reliably solved all of the tasks,
and tended to outperform the other algorithms we tested. COMA tended to suffer
from instability despite extensive hyperparameter tuning, and underperformed
PRD-AC in all but the simplest (paired agent) environment. Neither Shared-AC
nor Greedy-AC reached the same level of asymptotic performance as PRD-AC on
most tasks, with Greedy-AC failing to make progress on tasks that required
high degrees of cooperation, such as paired agent and social dilemma.

Collision Avoidance: Agents are spawned on the perimeter of a square planar
environment, and must navigate to a goal location diametrically opposite to
them, while avoiding collisions with other agents. Agents’ rewards are
composed of a component that is positive when agents move closer to their
goals, and a negative collision penalty. We consider three distinct methods
for assigning collision penalties:

  1. 1.

Collision penalties on colliding agents (v1): any two agents that collide are
both penalized (M=8).

  2. 2.

Collision penalties on non-colliding agents (v2): all agents except the
colliding agents are penalized (M=8).

  3. 3.

Collision penalties on non-colliding team-members (v3): Agents are divided
into 3 teams of 8 agents. When two agents of the same team collide, all other
non-colliding agents of the same team are penalized.

Assigning collision penalties to colliding agents can be seen as a form of
reward shaping, wherein the offending agent is directly penalized for impeding
the progress of other agents. We expect this task to favor greedy-AC, since
individual agents’ rewards already encourage some amount of cooperation. On
the other hand, assigning collision penalties to non-colliding agents (as is
done in v2 and v3) simulates externalities, a concept from economics wherein
an agent’s actions have a negative impact on other agents without negatively
affecting the offending agent. In this case, we expect greedy-AC to fail,
because agents’ individual rewards do not encourage them to avoid collisions,
which we hypothesize will result in selfish behavior. We expect shared-AC
outperform greedy-AC, since agents attempt to maximize group reward rather
than individual reward. However, we expect PRD-AC to outperform both greedy-AC
and shared-AC because it should learn cooperative policies while benefiting
from lower-variance gradient updates compared to shared-reward actor-critic.

In v1, while greedy initially learns most rapidly out of the four approaches,
and shared reward appears to match the performance of PRD, both greedy and
shared suffer from learning instability and experience a reward crash just
prior to convergence. In the case of shared-AC, we hypothesize that this crash
is due to high-variance gradient estimates. COMA learns more slowly than the
other 3 approaches. In v2, we find that greedy-AC learns a suboptimal policy
as predicted, and that PRD-AC slightly outperforms shared-AC. COMA suffers
from instability and fails. In v3, PRD-AC outperforms shared-AC and greedy-AC
by a much larger margin than in v2; we hypothesize that this is due to the
fact that in v3, each agent’s actions impact a smaller number of other agents,
allowing decoupling to be more effective. COMA converges to a sub-optimal
solution in this experiment.

Social Dilemma: 2 teams of agents each are spawned in random locations on one
side of a square and 2 goal regions (one for each team) are placed on the
opposite side of the square. Agents receive a positive reward for moving
closer to either of the goals, and an additional positive reward for being
within a goal region. Agents are incentivized to move as quickly as possible
to a goal location, because rewards earned further in the future are
discounted more heavily. However, if an agent enters a goal region that
belongs to the other team, every agent belonging to the other team receives a
negative penalty. The strategy which maximizes total group reward is for each
agent to travel to the goal associated with its team; however, the strategy
that maximizes individual reward is for each agent to go to the goal to which
it is closest initially, which will be the opposite team’s goal roughly 50% of
the time, thereby imposing a penalty on the other team and lowering group
reward. We therefore expect greedy-AC to learn a suboptimal strategy.

As predicted, greedy learns a severely suboptimal strategy. PRD achieves a
higher maximal reward than shared. COMA converges to a suboptimal strategy.

Figure 3: Results for COMA/PRD comparison. PRD-AC outperforms all other
algorithms. COMA is outperformed by global-GAE, indicating that COMA’s
advantage estimation is suboptimal.

###  V-B Policy Gradient Estimator Variance

To better understand the reasons for the performance improvement of PRD-AC
over shared-AC, we plot the variance of the policy gradient estimates for both
PRD-AC and shared-AC at various points in the training process for the paired
agent task with M=30. Because PRD-AC is able to eliminate the stochastic
advantage estimates of agents outside of the relevant set from its policy
gradient estimate, and these advantage estimates contribute noise to the
gradient estimates, we hypothesize that the gradients estimated by PRD-AC will
have a significantly lower variance than those estimated by shared-AC.

For maximal comparability between the two different gradient estimation
techniques, we use the identical actor and critic networks (i.e., with the
same parameters) to compute the gradient variance estimates for both PRD-AC
and shared-AC. That way, any differences in variance can only be explained by
differences in gradient estimation technique, rather than differences in actor
or critic network parameters, differences in policy stochasticity, or any
other confounding variables that may influence the variance of gradient
estimates. When computing the gradient variance for both PRD-AC and shared-AC,
we use actor and critic network parameters that were saved every 1000 episodes
during one run of shared-AC. We estimate variances empirically by taking the
variance across 100 independent gradient estimates, each computed using data
from an independent episode.

We observe that our hypothesis that PRD-AC provides lower variance gradient
estimates is correct (Fig. 4), especially in later episodes, where the
magnitude of the variance differs by over an order of magnitude. These
observations give a strong indication that the reduction in variance enabled
by PRD is responsible for the improvements in the learning efficiency and
stability of PRD-AC over shared-AC.

Figure 4: Comparison of gradient estimator variance between PRD-AC and shared-
AC. PRD-AC has significantly lower gradient estimator variance than shared-AC.

###  V-C Comparison to COMA

To better understand the advantages of PRD-AC over COMA, we run both
algorithms on the same task (paired agent task with M=8), and additionally
perform a series of ablation experiments that combine different aspects of the
two algorithms. The experiments are labeled as follows:

  * •

Shared-AC-GAE: ”default” implementation of shared-AC used to generate the
results in sec. V-A. Shared-AC-GAE is equivalent to PRD-AC, except all
advantage terms are retained in the gradient update.

  * •

Shared-AC-MC: Equivalent to shared-AC-GAE, except Monte-Carlo (MC) advantage
estimates are used; that is, advantages are estimated by subtracting a learned
value-function baseline (identical to the one used in PRD-AC and Shared-AC-
GAE) from a Monte-Carlo estimate of expected future returns (that is, simply
the sum of future returns for one episode).

  * •

Global-GAE: Equivalent to COMA (uses global rewards) but estimates advantages
using GAE instead of COMA’s advantage estimation technique. Note that we do
not consider this algorithm to be a variant of COMA, because COMA is
essentially defined by a particular method of advantage estimation. An
additional difference between Global-GAE and COMA is that COMA fits a
Q-function while Global-GAE fits a value function.

  * •

COMA-indiv: Equivalent to COMA, except Q functions are decomposed on an
individual-agent basis. COMA-indiv is also equivalent to shared-AC-GAE and
shared-AC-MC, except for the fact that COMA-style advantage estimates are
used, as opposed to GAE or MC advantage estimates. This experiment represents
an adaptation of COMA to a setting with individual rewards.

  * •

COMA [16].

We find that PRD-AC significantly outperforms COMA, with PRD-AC achieving
higher reward than COMA during the entirety of training (Fig. 3). We
additionally find that shared-AC-GAE, shared-AC-MC, and global-GAE all
outperform COMA, despite not utilizing any sophisticated credit-assignment
schemes, suggesting that the advantage estimation scheme used by COMA could be
improved. Interestingly, global-GAE outperforms COMA, because global-GAE is
equivalent to COMA other than the fact that it uses GAE. We hypothesize that
COMA’s heavy reliance on learned value functions for advantage estimation
contributes to its performance, due to the gradient bias introduced as a
result of approximation errors in the Q function [22]. Finally, we found that
COMA-indiv, which is essentially a naive implementation of COMA to an
individual reward scenario, was highly unstable and failed to make progress
towards solving the task, despite extensive hyper-parameter search. We
hypothesize that this issue is again due to biases introduced during advantage
estimation, which compounded due to having one Q function estimate for every
individual agent, instead of just one for the entire group.

These results, and in particular the results of COMA and COMA-indiv, highlight
two important advantages that PRD-AC offers over COMA. The first is
flexibility in advantage estimation strategy; whereas COMA is tethered to a
particular (biased) advantage estimation strategy (which may fail in certain
situations, as indicated by COMA-indiv), PRD is agnostic to advantage
estimation strategy. PRD can therefore take advantage of recent advances in
advantage function estimation, such as GAE [22] (which effectively balances
bias and variance) and action-conditioned baselines [13] (which offers further
variance reduction while avoiding bias). The second advantage of PRD is that
it is able to make better use of individual rewards, when they are available,
as indicated by the fact that COMA fails when we attempt to incorporate
individual rewards. PRD, on the other hand, is able to use this additional
information to significantly improve learning efficiency.

##  VI Conclusions

In this paper, we address the credit assignment problem in MARL by studying
how large cooperative MARL problems can be decomposed into smaller decoupled
subproblems, which may be solved independently to yield a fully cooperative
group-level solution. Additionally, we demonstrated how this decomposition,
which we refer to as partial reward decoupling (PRD), can be incorporated into
an actor-critic algorithm to yield an efficient model-free MARL algorithm that
we term PRD actor-critic (PRD-AC). We show that PRD-AC consistently
outperforms actor-critic algorithms that naively attempt to maximize either
individual reward (greedy-AC) or total group reward (shared-AC), on a diverse
set of tasks that require varying degrees of cooperation. Finally, we relate
PRD to an existing state-of-the-art multi-agent actor-critic algorithm (COMA),
and show that for problems in which individual rewards are available, PRD
outperforms COMA by making better use of this additional information, and by
allowing more advanced advantage estimation strategies.

## References

  * [1] C. Berner, G. Brockman, B. Chan, V. Cheung, P. Debiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse _et al._ , “Dota 2 with large scale deep reinforcement learning,” _arXiv preprint arXiv:1912.06680_ , 2019.
  * [2] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev _et al._ , “Grandmaster level in starcraft ii using multi-agent reinforcement learning,” _Nature_ , vol. 575, no. 7782, pp. 350–354, 2019.
  * [3] M. Jaderberg, W. M. Czarnecki, I. Dunning, L. Marris, G. Lever, A. G. Castaneda, C. Beattie, N. C. Rabinowitz, A. S. Morcos, A. Ruderman _et al._ , “Human-level performance in 3d multiplayer games with population-based reinforcement learning,” _Science_ , vol. 364, no. 6443, pp. 859–865, 2019.
  * [4] M. Minsky, “Steps toward artificial intelligence,” _Proceedings of the IRE_ , vol. 49, no. 1, pp. 8–30, 1961.
  * [5] R. S. Sutton, A. G. Barto _et al._ , _Introduction to reinforcement learning_. MIT press Cambridge, 1998, vol. 135.
  * [6] V. R. Konda and J. N. Tsitsiklis, “Actor-critic algorithms,” in _Advances in neural information processing systems_ , 2000, pp. 1008–1014.
  * [7] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _arXiv preprint arXiv:1707.06347_ , 2017\.
  * [8] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy optimization,” in _International conference on machine learning_. PMLR, 2015, pp. 1889–1897.
  * [9] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in _International conference on machine learning_. PMLR, 2018, pp. 1861–1870.
  * [10] A. Y. Ng, D. Harada, and S. Russell, “Policy invariance under reward transformations: Theory and application to reward shaping,” in _Icml_ , vol. 99, 1999, pp. 278–287.
  * [11] C. Boutilier, “Planning, learning and coordination in multiagent decision processes,” in _TARK_ , vol. 96. Citeseer, 1996, pp. 195–210.
  * [12] R. S. Sutton, D. A. McAllester, S. P. Singh, and Y. Mansour, “Policy gradient methods for reinforcement learning with function approximation,” in _Advances in neural information processing systems_ , 2000, pp. 1057–1063.
  * [13] C. Wu, A. Rajeswaran, Y. Duan, V. Kumar, A. M. Bayen, S. Kakade, I. Mordatch, and P. Abbeel, “Variance reduction for policy gradient with action-dependent factorized baselines,” in _International Conference on Learning Representations_ , 2018.
  * [14] R. Lowe, Y. WU, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitive environments,” _Advances in Neural Information Processing Systems_ , vol. 30, pp. 6379–6390, 2017.
  * [15] G. Barth-Maron, M. W. Hoffman, D. Budden, W. Dabney, D. Horgan, T. Dhruva, A. Muldal, N. Heess, and T. Lillicrap, “Distributed distributional deterministic policy gradients,” in _International Conference on Learning Representations_ , 2018.
  * [16] J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson, “Counterfactual multi-agent policy gradients,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 32, no. 1, 2018.
  * [17] S. Iqbal and F. Sha, “Actor-attention-critic for multi-agent reinforcement learning,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 2961–2970.
  * [18] B. Freed, R. James, G. Sartoretti, and H. Choset, “Sparse discrete communication learning for multi-agent cooperation through backpropagation,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 7993–7998.
  * [19] Q. Li, W. Lin, Z. Liu, and A. Prorok, “Message-aware graph attention networks for large-scale multi-robot path planning,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 3, pp. 5533–5540, 2021.
  * [20] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph attention networks,” in _International Conference on Learning Representations_ , 2018.
  * [21] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in neural information processing systems_ , 2017, pp. 5998–6008.
  * [22] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel, “High-dimensional continuous control using generalized advantage estimation,” _arXiv preprint arXiv:1506.02438_ , 2015.
  * [23] P. J. Huber, “Robust estimation of a location parameter,” in _Breakthroughs in statistics_. Springer, 1992, pp. 492–518.


Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.
10.1109/ACCESS.2017.DOI

# Offset-Guided Attention Network for Room-Level Aware Floor Plan Segmentation

ZHANGYU WANG1 and NINGYUAN SUN2  Avon Old Farms School, CT, USA (e-mail:
<EMAIL_ADDRESS> University of Wisconsin-Madison, WI, USA, (e-mail:
<EMAIL_ADDRESS>
###### Abstract

Recognition of floor plans has been a challenging and popular task. Despite
that many recent approaches have been proposed for this task, they typically
fail to make the room-level unified prediction. Specifically, multiple
semantic categories can be assigned in a single room, which seriously limits
their visual quality and applicability. In this paper, we propose a novel
approach to recognize the floor plan layouts with a newly proposed Offset-
Guided Attention mechanism to improve the semantic consistency within a room.
In addition, we present a Feature Fusion Attention module that leverages the
channel-wise attention to encourage the consistency of the room, wall, and
door predictions, further enhancing the room-level semantic consistency.
Experimental results manifest our approach is able to improve the room-level
semantic consistency and outperforms the existing works both qualitatively and
quantitatively.

###### Index Terms:

floor plan recognition, attention, offset-guided, convolutional neural network

=-15pt

##  I Introduction

The recognition of floor plan elements has been a useful task that provides
benefits to various applications such as indoor robotics navigation[45],
computer-aided home design[1, 3], and VR/AR[46, 47]. However, it is a
challenging task beyond the general semantic segmentation problem. First,
different from the common objects in natural images, rooms in the floor plan
images typically share the similar rectangular shape as shown in Fig 1, so it
is challenging to identify their semantic categories simply based on their
appearances. Second, the relative spatial locations of rooms are crucial to
floor plan image understanding. However, traditional CNNs for semantic
segmentation typically possess the translational invariance property,
incapable of effectively capturing the rooms’ relative spatial information in
floor plan images. Third, the floor plan images typically lack textures, which
plays a key role in CNNs’ recognition as suggested by existing works like
[49]. Therefore, directly adopting the semantic segmentation networks [20, 21]
designed for natural images to this task cannot produce satisfactory results
due to the above issues, and quantitative proofs are provided in our
experiments.

Recently, many approaches[1, 2, 14] have been proposed to address the floor
plan recognition task through convolutional neural networks and provided
promising results. However, their models typically lack the capability of
room-level understanding. To be more specific, they predict the semantic label
of each pixel individually and may assign multiple semantic categories in one
single room as shown in Fig 1 (b, c), which severely limits their
applicability in practical applications. The above issue is potentially
induced by the following causes. First, the rooms vary from each other
significantly in terms of size; yet, their approaches cannot effectively
capture both the global and local information to handle the large and tiny
sized rooms respectively. Second, they typically predict the room and boundary
categories (wall, door) parallelly, thus the predictions on these categories
may be inconsistent with each other. A simple approach to address this issue
is to leverage flood fill, i.e., assign all the pixels in a room to be the
same category according to majority voting. However, this approach still
cannot work well due to the reasons that will be discussed in Section II.

(a) Ground Truth

(b) DFP

(c) DeepLabV3+

(d) Ours

Figure 1: Existing works including Deep Floor Plan(DFP) and DeepLabV3+ [1, 20]
have inconsistent room-level prediction (b,c). Our approach addresses this
issue and can produce plausible results (d).

In this paper, we propose a novel approach to address the above issues in
existing approaches to improve the prediction consistency within a single room
and the overall floor plan segmentation performance. First, we design an
Offset-Guided Attention network to encourage consistency within each room. To
do this, we utilize a simple prior that “all the pixels of the same room share
the same semantic category”, which always holds true in all the existing
datasets as far as we know, but is ignored by the current state of the arts.
Our network explicitly learns the correlation of pixels within each room with
our newly proposed offset prediction module, and the learned correlation is
further utilized to refine the output and improve the consistency of
predictions within each room. In addition, our Offset-Guided Attention module
helps the network to learn long-range context information and increase the
effective receptive field of the network; hence, our approach can handle large
and long rooms and have consistent semantic predictions in them (see Fig 1
(d)).

Besides, we propose a Feature Fusion Attention module to further improve the
consistency between room and boundary predictions. Existing works [1] propose
to predict the room and boundary categories in individual branches to improve
the results, which has been proven to be effective. However, this strategy has
a side-effect that the prediction of the two branches may be inconsistent with
each other. As shown in Fig 1 (b), the blue room is dilated into the orange
room while the wall prediction does not recognize any boundaries between blue
and orange, which shows that the prediction of the boundary categories is not
aligned with the room. Similarly, there are also cases where a boundary is
successfully recognized by the boundary branch, but not recognized by the room
branch. This is another cause of the semantic inconsistency within a single
room. To address this issue, we present a Feature Fusion Attention module to
combine the prediction of two branches and yield consistent predictions (Fig 1
(d)).

Quantitative experiments manifest that our approach outperforms the existing
works on two commonly used floor plan recognition datasets[4, 13]
consistently. In addition, qualitative comparisons show that our Offset-Guided
Attention and Feature Fusion Attention effectively improve the room-level
consistency. Code and models will be released upon publication.

To summarize, this paper has the following contributions to floor plan
recognition :

  1. 1.

We propose an effective and interpretable framework for floor plan
segmentation beyond the existing works;

  2. 2.

We propose an Offset-Guided Attention module and a channel-wise Feature Fusion
Attention module to improve the room-level semantic prediction;

  3. 3.

We achieve state-of-the-art performance on the two common floor plan
recognition datasets R2V [4] and R3D [13].

[t!][scale = 0.75]wang5.pdfOverview of our framework. We extract features
using a CNN backbone(1); then use the feature to generate an offset
prediction(2), which will later be used to determine the affinity in the
Offset-Guided room attention module(3); besides, an additional feature
extraction branch is used to enrich the room and boundary features by fusing
them with the Feature Fusion Attention module(FFA)(4), and then produce the
final boundary and room prediction(5). Note that the English labels on the
“input image” are added for reader’s reference and do not exist in the
original image.

##  II Related Work

###  II-A Semantic Segmentation

Semantic segmentation aims to recognize the pixel-wise semantic categories in
the image. After Long et al.[17] proposes to utilize the Fully Convolutional
Networks to address this issue, remarkable signs of progress are made in this
field [18, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 41, 42, 43, 44]. As
discussed in Section I, floor plan segmentation is drastically different from
the general semantic segmentation. In this work, we propose some particularly
designed novel modules for floor plan segmentation and manifest their
effectiveness in this challenging task.

###  II-B Traditional Approaches on Floor Plan Recognition

Floor plan recognition aims to derive the pixel-level prediction of a floor
plan image. However, it is notably different from semantic segmentation, as
discussed in Section I. This task attracts a lot of attention from both
academia and industry in the past few years [6, 7, 5, 8, 2, 1, 14, 34, 35, 36,
37]. Traditional methods [6, 7] use low-level information like graphical
shapes to predict rooms and boundaries. Ahamed et al. [5] propose to decouple
the boundary recognition and text recognition from the room recognition, and
they introduce to extract line segments of different thicknesses – the thicker
ones are considered to be the walls and the thinner ones are the texts. In
addition, another approach [8] utilizes a heuristics method for this task.
However, the performance of this heuristics-based approach is far from
satisfactory as the assumptions of this approach cannot cover all the
circumstances. Besides, models using this heuristics-based approach need to be
fine-tuned manually in a very frequent manner and many parameters need to be
tuned for specific datasets.

###  II-C Room-level consistency in Floor Plan Recognition

Recently, a variety of approaches[1, 2, 14] are proposed to address the floor
plan recognition task using convolutional neural networks, which yielded
promising results. However, as discussed in Section I, they typically predict
inconsistent semantic categories within one single room (see Fig 1 (a,c,d)),
which implies that their models lack of the room-level understanding.

An intuitive idea to address this issue is to leverage a post-processing
method flood fill, which simply assigns all pixels within a certain boundary
to the same semantic label according to the majority voting. An intuitive idea
to address this issue is to leverage a post-processing method, flood fill,
which assigns all pixels within a specific boundary to the same semantic label
according to the majority voting. However, it cannot achieve satisfactory
performance. That is because the boundary (door and wall categories)
segmentation that flood fill is based upon a challenging task. Specifically,
in many cases, the boundaries are predicted in an inaccurate or even wrong
location. In addition, flood fill requires the boundary to be continuous and
form a closed space to conduct the majority voting – any slight failure in
boundary prediction could cause a huge room-level prediction error, as shown
in Fig 1 (d).

Some existing works try to enhance the accuracy of boundary prediction.
However, flood fill still cannot work well based on their approaches due to
their unsatisfactory boundary prediction performance. However, due to
unsatisfactory boundary prediction performance, flood fill still cannot work
well on the top of their approaches. Liu et al.[4] design a network that
recognizes the junction point of the floor plan and determines the position of
boundaries based on such prediction. However, the method can only recognize
horizontal and vertical boundaries and cannot work well for others, which
significantly limits its applicability. Besides, Zeng et al.[1] design a
network with boundary-aware kernels and successfully bring up the accuracy of
boundary predictions. Nevertheless, their boundary accuracy is still not high
enough for flood fill, and the network has an inferior accuracy in certain
types of rooms, where in many cases the majority of pixels predicted in a room
are of the wrong category. In a word, even though existing works [4, 1]
address on the boundary accuracy, the simple approach like flood filling still
fails to improve the room prediction consistency and the overall accuracy.

###  II-D Attention Mechanism in Floor Plan Recognition

Early works like [11] propose using self-attention mechanism to increase
awareness of the correlation between different words in a sentence. Recently,
some existing works adopt the attention mechanism in floor plan recognition.
Zeng et al.[1] is the first to use attention in this task. They design a
direction-aware kernel supported by room-boundary guided attention to
encourage recognition of the spatial relationship between boundaries and
rooms. With such a method, the accuracy of wall prediction is largely
increased, but room prediction remains to be an issue. Recently, Lv et al.[3]
further enhance the accuracy of the room and boundary prediction with the
architecture to jointly detect and learn text, symbol, structural element, and
scale information. However, since their approach relies on the text and symbol
prediction, [3] only works on their own dataset but cannot be applied to the
general floor plan datasets like R3D and R2V, since the symbols in these
datasets have different styles and the texts are even in different languages.
Besides, text and symbol recognition usually cause significant increase in the
network complexity.

[t!][scale = 0.6]wang6.pdfArchitecture of the Offset-Guided Room Attention
module. The (1) and (2) in the image refers to Equation 1 and Equation 2
respectively. We use the offset prediction (a) to generate a position
correlation map among pixels (b), which will then be used to derive the
affinity of pixels in the attention map (c) and fused with room feature map
using batch multiplication at (d). The module outputs the enhanced room
feature map (e).

[t!][scale = 0.6]wang7.pdfIllustration of the offset prediction module in the
Offset-Guided Attention. This module predicts the offset from each pixel to
the center of room it belongs to. The correlation between the two pixels is
measured by the distance of the them after moving along according to the
offset towards the room center.

##  III Method

###  III-A Overview

Fig 2 shows the overall architecture of our network. We adopt the ResNet-101
[19] backbone to extract feature maps from the input image. After the feature
extraction, the prediction head of the network contains four branches:
boundary prediction, room prediction, Offset-Guided Attention, and Feature-
Fusion Attention. Boundary prediction (Fig 1 (4)) is to predict of boundary
categories, i.e., walls and doors. Room prediction (Fig 1 (5)) is to predict
the semantic category of the pixels within rooms. Offset-Guided Attention (Fig
II-D) refers to the prediction of the vertical and horizontal offset between
the specific pixel $p$ and the geometrical center $c$ of the room $R$ it
belongs to. Feature-Fusion Attention (Fig III-B) aims to improve the
consistency between the room and boundary predictions. In the following, we
will detail the key modules including the boundary and room prediction in
Section III-B, Offset-Guided Attention in Section III-C, and Feature-Fusion
Attention in Section III-D.

###  III-B Boundary and Room Prediction

We adopt ResNet-101 with 8$\times$ down-sampling rate as the backbone network
for feature extraction. Inspired by [1], we use two individual branches on the
top of the backbone for room and boundary prediction respectively, as shown in
Fig 1(4, 5) to improve their performances.

[t!][scale=0.8]wang8.pdfIllustration of the Feature Fusion Attention module,
where Ⓐ denotes average pooling layer, $\bigotimes$ denotes multiplication,
and $\bigoplus$ denotes concatenation.

###  III-C Offset-Guided Attention Module

In the floor plan image, the semantic correlation between two pixels cannot be
directly represented by their distance on the floor plan. As shown in Fig
II-D, pixels that are close to each other (p2, p3) can be hardly related in
terms of semantic category since they belong to different rooms, whereas those
that are far away (p1, p2) are semantically correlated with each other since
they belong to the same room. Inspired by the above observation, we propose an
Offset-Guided Attention module to better measure the pixel correlation.

Fig II-D shows the architecture of Offset-Guided Room Attention Module. First,
as shown in figure II-D (a), the offset prediction module predicts the the
offset $o_{p}$ from each pixel $p=(x,y)$ to the center of the room $c_{p}$ it
belongs to, i.e., $o_{p}=c_{p}-p$. With equation

$p^{\prime}=p+o_{p}$ (1)

denoting the offset-guided shifted position of $p$, we can measure the
correlation $A_{i,j}$ of two pixels $p_{i},p_{j}$ according to the Equation 2.

$A_{i,j}=||p^{\prime}_{i}-p^{\prime}_{j}||_{2}^{2}$ (2)

If the pixels $p_{i},p_{j}$ are close to each other after the offset-guided
shifting, then they are strongly correlated with each other, and hence have
better chance to be in the same room and also share the same semantic
category. Then, $A$ further serves as the attention map of our Offset-Guided
Attention module. In Fig II-D (c), we illustrate the mutual attention map of
the pixels in one row $I_{i}$ of the image $I$. This sub attention map has the
shape $w\times w$ where $w$ indicates the width of the image. It is the
symmetric matrix where the diagonal values indicate the correlation to the
pixel itself. In addition, the attention of nearby pixels (in the same room)
are significantly larger than others that are located in different room, and
the sharp change of attention values in Fig II-D (c) manifest the
effectiveness of our attention map $A$ to distinguish the pixels within the
same room or are in different rooms.

Then, the attention map $A$ is normalized by softmax to make sure that
$\sum_{j}A_{i,j}$, i.e., the sum of attention values of all other pixels $j$
to the target pixel $i$, to be 1. Then the features from the room prediction
branch $f_{j}$ are weighted summed according to $A$, and produce the enhanced
feature $\hat{f_{i}}$ for the target pixel $i$, as shown in Equation 3.

$\hat{f}_{i}=\sum_{j}A_{i,j}f_{j}$ (3)

At last, the enhanced feature $\hat{f}_{i}$ is concatenated with $f_{i}$ for
the final semantic prediction. Note that here we take the pixel $i$ as an
example, and in our approach, all the pixels are operated in the same way.

To improve the memory and calculation efficiency, we adopt crisscross
attention technique[12], where only the correlation between the target pixel
$i$ and other pixels $j$ on its horizontal and vertical axis contribute to the
attention map. Following [12], this module is applied for two times
successively in order to simulate the full attention. Please refer to [12] for
more details.

###  III-D Feature Fusion Attention Module

As discussed in Section I, the predictions of room and boundary might be
inconsistent with each other, since the two branches introduced in Section
III-B predict these two kinds of categories individually, as shown in Fig 1.
To address this issue, we propose a Feature Fusion Attention module to fuse
the features of the two branches with an additional shared branch, and enhance
the prediction consistency of room and boundary branches. Fig III-B shows the
architecture of Feature Fusion Attention module. First, the feature map from
the two branches $f_{1},f_{2}\in R^{d\times h\times w}$ are concatenated in
the channel dimension as $f=f_{1}\oplus f_{2}\in R^{2d\times h\times w}$, and
a global average pooling layer reduces the size of $f$ to be $\in R^{2d\times
1\times 1}$. Then one fully connected layer learns a channel-wise attention
map $A_{c}$ of size $2d\times 2d$, indicating how each channel correlates to
others, and outputs a weight feature $w\in R^{2d\times 1\times 1}$, one
element for each channel. At last, the enhanced feature map is derived with
Equation 4.

$\hat{f}=f\otimes w$ (4)

where the $\otimes$ denotes the element-wise multiplication.

To summarize, the Feature Fusion Attention module leverage the shared features
of the boundary and room prediction branches and feed the mutual information
between them to improve their prediction consistency.

###  III-E Loss function

For both the boundary and room prediction, we use weighted softmax cross
entropy loss with specific weight for different categories, as shown in
Equation 5.

$\mathcal{L}_{s}=\sum_{i=1}^{C}-w_{i}\mathbbm{1}(y_{i})\log(p_{i})$ (5)

where $C$ is the number of categories, $w_{i}$ is the weight applied to the
specific category, $\mathbbm{1}(y_{i})$ is an indicator function and equals to
1 when the ground truth category of $y$ is the category $i$. $p_{i}$ is the
predicted possibility for the category normalized by the Softmax function.
$w_{i}$ is derived by calculating the percent of pixels of the specific
category in the training set:

$w_{i}=\frac{V_{i}}{V_{total}}$ (6)

where $V_{i}$ is number of pixels of the specific category, and $V_{total}$ is
the total number of pixels.

For the offset prediction, we use $L_{1}$ loss to measure the distance of two
pixels after offset-guided shifting, as shown in Equation 7.

$\mathcal{L}_{o}=|g_{x}-o_{x}|+|g_{y}-o_{y}|$ (7)

where $g_{x}$ and $g_{y}$ are the offset ground truth in x and y axis, and
$o_{x}$ and $o_{y}$ are the predicted offset in x and y axis.

The overall loss to train the network is $\mathcal{L}_{s}+\mathcal{L}_{o}$.

(a) Input

(b) Ground Truth

(c) Ours

(d) DeepFloorPlan

(e) R2V

(f) DeepLabV3+

(g) PSPNet

Figure 2: Visual comparison of our result(c) with existing works(d-g),
including Deep Floor Plan[1], R2V(Raster-to-Vector)[4], DeepLabV3+[20], and
PSPNet[21], on the R2V dataset. Note that the English labels in (a) do not
exist in the original image, and are added for reader’s reference.

##  IV Experiments

###  IV-A Datasets

We evaluate our Offset-Guided Attention Networks on two datasets that were
commonly used for floor plan recognition, the R3D and R2V dataset [13, 4]. The
R3D dataset consists of 214 images from [13], and the R2V dataset consists of
815 images from [4]. To have a fair comparison with existing works, we follow
the same train-test split as [1].

###  IV-B Implementation Details

Our network is trained on a single NVIDIA Tesla P100-PCIE GPU and is trained
for 7k iterations in total. We adopted a fixed initial learning rate of $0.01$
and used a weight decay of rate $0.0002$. Besides, we use SGD with Nesterov
Momentum and update parameters. The training is conducted in two steps. First,
we initialize our network using the ResNet model pretrained on ImageNet and
optimize the offset prediction module with other weights frozen. Then, we
optimize other modules in our network with offset prediction branch frozen. In
inference, we derive the final result by a single forward pass, without any
post processing like existing works [1, 3].

###  IV-C Metrics

Following the existing works [1, 4], we adopt $overallaccuracy$ and
$classaccuracy$ as the metrics to evaluate the performance quantitatively as
shown in Equation 8 and 9.

$overall\\_acc=\sum^{i}_{j=0}\frac{C(j)}{N(j)},$ (8)
$class\\_acc(j)=\frac{C(j)}{N(j)}$ (9)

where $overall\\_acc$ denotes overall accuracy, $class\\_acc$ denotes class
accuracy of a specific class $j$, $i$ is the number of categories, $C$ is the
correctly predicted number of pixels of specific category $j$, and $N$ is the
total number of pixels of specific category $j$.

In addition, we also adopt $m\\_IoU$(mean Intersection over Union) as an
additional metric, defined as the following equation:

$m\\_IoU=\frac{\sum^{n}_{j=0}\frac{C_{j}}{P_{j}+G_{j}-C_{j}}}{n},$ (10)

where $n$ denotes total number of categories, $C_{j}$ is the correctly
predicted number of pixels in certain category, $P_{j}$ is the total number of
pixels predicted in certain category, and $G_{j}$ is total number of pixels in
the ground truth of certain category.

###  IV-D Comparison with the Existing Works

We compare our network with existing floor plan recognition approaches Raster-
to-Vector [4] and Deep Floor Plan [1], as well as general semantic
segmentation approaches PSPNet [21] and DeepLabV3+ [20].

Fig 2 (c, e) show visual comparisons of our result and Raster-to-Vector.
Comparing with the ground truth (b), Raster-to-Vector (e) fails to determine
the room type of a major room in the top row. In addition, their model fails
to recognize some room regions and leaves it empty. In contrast, our model
shows perfect room type prediction and accurate recognition of boundary
categories. As shown in Table I “Raster-to-Vector”, our result has superior
performances in all categories and has significantly higher overall accuracy.

|  | R2V | R3D
---|---|---|---
Ours | Raster-to-Vector | DeepLabV3+ | PSPNet | DFP | Ours | DeepLabV3+ | PSPNet | DFP
| _overall accuracy_ | 0.93 | 0.84 | 0.88 | 0.88 | 0.89 | 0.91 | 0.85 | 0.84 | 0.89
| _mean IoU_ | 0.77 | N/A | 0.69 | 0.70 | 0.74 | 0.74 | 0.50 | 0.50 | 0.63
_class accuracy_ | wall | 0.90 | 0.53 | 0.80 | 0.84 | 0.89 | 0.95 | 0.93 | 0.91 | 0.98
door & window | 0.89 | 0.58 | 0.72 | 0.76 | 0.89 | 0.83 | 0.60 | 0.54 | 0.83
closet | 0.88 | 0.78 | 0.78 | 0.80 | 0.81 | 0.71 | 0.24 | 0.45 | 0.61
bathroom | 0.92 | 0.83 | 0.90 | 0.90 | 0.87 | 0.90 | 0.76 | 0.70 | 0.81
living room | 0.94 | 0.72 | 0.85 | 0.83 | 0.88 | 0.93 | 0.76 | 0.76 | 0.87
bedroom | 0.96 | 0.89 | 0.82 | 0.86 | 0.83 | 0.89 | 0.56 | 0.55 | 0.75
hall | 0.84 | 0.64 | 0.55 | 0.78 | 0.68 | 0.87 | 0.72 | 0.61 | 0.59
balcony | 0.91 | 0.71 | 0.87 | 0.87 | 0.90 | 0.83 | 0.08 | 0.41 | 0.44
TABLE I: Quantitative comparison with Raster-to-Vector[4], DeepLabv3+[20],
PSPNet[21], and Deep Floor Plan(DFP)[1] on both R2V and R3D. Our method
produces superior results in both datasets and has higher accuracy in almost
all sub-classes. We did not report the m_IoU of Raster-to-Vector model since
m_IoU metric is not conducted in their original paper[4] and the model’s
relatively weak performance under accuracy metric implies that is has
relatively low m_IoU.

Next, we compare our result with general segmentation networks including
DeepLabV3+[20] and PSPNet[21]. We utilize the author-released code to produce
their results. First, as shown in Fig 2 (c, f, g), in the top and bottom case,
both DeepLabV3+ and PSPNet are not able to predict one or more rooms to be the
unified category. Besides, there are inconsistencies between their boundary
and room predictions, such as some nearby room categories are dilated into
each other. In contrast, our network produces the uniform and consistent
results within rooms. Also, Fig 3 shows our network’s ability to produce
superior results in the R3D dataset over the existing works. Table I
“DeepLabV3+” and “PSPNet” show the quantitative comparison between our results
and results of the two existing works on the R2V and R3D datasets. As shown in
the table, our network produces significantly superior results in overall
accuracy and all sub-classes in comparison with the other two networks in both
R2V and R3D datasets. This result explicitly shows the superiority of our
network over the general semantic segmentation networks, and also demonstrates
the significant difference between the floor plan recognition task and the
general semantic segmentation task.

(a) Input

(b) Ground Truth

(c) Ours

(d) DeepFloorPlan

(e) DeepLabV3+

(f) PSPNet

Figure 3: Visual comparison of our result(c) with existing works(d-g),
including Deep Floor Plan[1], R2V(Raster-to-Vector)[4], DeepLabV3+[20], and
PSPNet[21], on the R3D dataset

(a) Input

(b) Ours

(c) Ground Truth

Figure 4: More results produced by our network on the R3D and R2V dataset

We further compare our work with a recent floor plan recognition approach Deep
Floor Plan [1]. Still, we utilize the author-released model and adopt the same
experimental setting as in their paper. As shown in Fig 2 (d), in the middle
case, there is a blue part predicted inside the orange room, indicating that
their approach is incapable to have the consistent room-type prediction, and
can make the wrong prediction when the room has the irregular shape. In
addition, some predictions are dilated to the nearby rooms as shown in the
bottom row, leading to inconsistent predictions with a single room. On the
contrary, as shown in Fig 2 (c), our model is able to generate consistent and
correct predictions with our Offset-Guided Attention Mechanism combined with
Feature Fusion Attention module, no matter what the shape or size of the room
is. In R3D dataset shown as Fig 3 (c, d), our results achieve the superior
room uniformity and consistency over Deep Floor Plan [1]. Quantitatively,
according to Table I, our model has higher accuracy for almost all sub-classes
for both R2V and R3D datasets.

###  IV-E More results

We further proceed to analyse our network’s ability in various specific cases.
We divide R3D floor plans into two types, irregular (first and third case) and
rectangular (second and last case), and selected two cases of each type. As
shown in Fig 4 (a, b), our model is able to produce highly accurate and
unified room predictions in both situations, manifesting the ability of our
network in handling rooms with different shapes. Besides, our model is able to
recognize walls of different thicknesses accurately. As for the R2V dataset,
it is mostly composed of rectangular floor plans, so we select the challenging
cases that have complicated layouts and contain multiple rooms that look
similar. Fig 4 (c, d) shows that our model can consistently generate precise
and concrete predictions on all of them.

##  V Ablation study

###  V-A Offset-Guided Attention Module

| Ours | Without OGA | Without FFA
---|---|---|---
overall acc | 0.93 | 0.91(-0.02) | 0.92(-0.01)
wall | 0.90 | 0.93(+0.03) | 0.90
door & window | 0.89 | 0.86(-0.03) | 0.84(-0.05)
closet | 0.88 | 0.87(-0.01) | 0.87(-0.01)
bathroom | 0.92 | 0.91(-0.01) | 0.91(-0.01)
living room | 0.94 | 0.90(-0.04) | 0.90(-0.04)
bedroom | 0.96 | 0.95(-0.01) | 0.96
hall | 0.84 | 0.82(-0.02) | 0.87(+0.03)
balcony | 0.91 | 0.86(-0.05) | 0.91
TABLE II: Comparison of results produced by our network with both modules,
without Offset-Guided Attention(OGA) module, and without Feature Fusion
Attention(FFA) module on the R2V dataset.

First, we evaluate the effectiveness of Offset-Guided Attention module on
floor plan recognition task. As shown in Table II and Fig 5, all of the room
categories attain inferior accuracy when the Offset-Guided Attention module is
disabled, whereas the results of the boundary categories are not largely
impacted. Notably, the performances of balcony, living room, and hall
prediction decrease more significantly than the other categories since they
are typically in either irregular shapes or have large areas. Our Offset-
Guided Attention module helps to handle the irregular-shaped rooms by fusing
the multiple predictions of the same room, and improve the large-area room
performance by increasing the network’s receptive field.

(a) Input image

(b) Ground truth

(c) Ours without FFA

(d) Ours

Figure 5: Results produced by our model with and without Feature Fusion
Attention(FFA) module. The red boxes denote the disagreement of the two
branches without the Feature Fusion Attention module.

###  V-B Analysis of Feature Fusion Attention module

Then, we evaluate the effectiveness of our Feature Fusion Attention module.
Similarly, we train and test our network without Feature Fusion Attention
module on the R2V dataset and analyze the results. As shown in Fig 5 (c),
without Feature Fusion Attention module, some orange regions dilate into the
green room. In addition, the room on the top right corner is correctly
predicted by the boundary prediction but not recognized by room prediction
branch. Such kind of disagreement between the two branches harms the overall
accuracy and also the visual quality. The above issues are addressed by our
Fusion Attention module as shown in Fig 5 (d). Next, we study the quantitative
results produced by the network with and without Feature Fusion Attention
Table II shows quantitative comparisons. From the table, the Feature Fusion
Attention module improves most boundary and door sub-class accuracy and
improved overall accuracy by 1%, manifesting the effectiveness of this module.

###  V-C More ablation studies

At last, we conduct additional evaluations on some designs of our proposed
modules. To begin with, in the Offset-Guided Attention module, we study the
effectiveness of the softmax normalization and the room-aware attention
mechanism. Without the softmax normalization, the sum of attention values of
all other pixels in the image to the target pixel is not necessarily to be 1,
making the feature aggregation unstable. As shown in Table III, the overall
accuracy drops by 5% when softmax normalization is absent. To validate the
effectiveness of the room-aware attention mechanism, we use an alternative
attention mechanism based on the absolute distance between two pixels. As
shown in Table III “Absolute Distance”, the overall accuracy has a 2% decrease
when Offset-Guided Attention is replaced by absolute distance based attention,
illustrating the superiority of our Offset-Guided room attention module.

Then, we evaluate several key components in our Feature Fusion Attention
module, including the effectiveness of the Feature Fusion Attention module in
each branch and average pooling. Table IV shows results when a specific part
is absent or altered. Notably, the absence of the Feature Fusion Attention
module in either room or boundary branch results in the general performance
drop in multiple sub-classes, implying that the FFA module only expresses its
functionality when applied on both branches, since this module is designed to
enhance the prediction consistency of the two branches. Max pooling layer
yields comparable results as our network with average pooling, since both of
the pooling strategies are able to extract a semantic-meaningful global
feature from the feature map for the subsequent attention operations.

##  VI Conclusion

In this work, we present a novel method for floor plan recognition task that
has the following contributions: first, we design an Offset-Guided Attention
module to encourage the consistency of predictions within a single room;
second, we design a Feature Fusion Attention module that further enhances the
performance by enhancing the prediction consistency between room and boundary
branches. Further, we extensively evaluate our approach on two commonly used
datasets R2V and R3D. Qualitative and quantitative evaluations show the
superiority of our network over the existing works on floor plan recognition.
In the future, we will extend the work to more tasks and practical
applications such as floor plan instance segmentation and 3D reconstruction,
et.al.

| Ours | Without softmax | Absolute Distance
---|---|---|---
overall acc | 0.93 | 0.88(-0.05) | 0.91(-0.02)
wall | 0.90 | 0.81(-0.09) | 0.93(+0.03)
door & window | 0.89 | 0.81(-0.08) | 0.90(+0.01)
closet | 0.88 | 0.73(-0.15) | 0.83(-0.05)
bathroom | 0.92 | 0.84(-0.08) | 0.87(-0.05)
living room | 0.94 | 0.95(+0.01) | 0.91(-0.03)
bedroom | 0.96 | 0.92(-0.04) | 0.95(-0.01)
hall | 0.84 | 0.47(-0.37) | 0.75(-0.09)
balcony | 0.91 | 0.86(-0.05) | 0.89(-0.02)
TABLE III: Additional ablation studies on Offset-Guided Attention module. “Absolute Distance” means applying a absolute-distance-based attention instead of Offset-Guided Attention in the module. | Ours | No R-FFA | No B-FFA | Max pooling
---|---|---|---|---
overall acc | 0.93 | 0.92(-0.01) | 0.92(-0.01) | 0.93
wall | 0.90 | 0.90 | 0.90 | 0.90
door & window | 0.89 | 0.86(-0.03) | 0.87(-0.02) | 0.86(-0.03)
closet | 0.88 | 0.88 | 0.87(-0.01) | 0.88
bathroom | 0.92 | 0.91(-0.01) | 0.90(-0.02) | 0.93(+0.01)
living room | 0.94 | 0.93(-0.01) | 0.92(-0.02) | 0.92(-0.02)
bedroom | 0.96 | 0.96 | 0.95(-0.01) | 0.96
hall | 0.84 | 0.84 | 0.83(-0.01) | 0.85(+0.01)
balcony | 0.91 | 0.89(-0.02) | 0.91 | 0.91
TABLE IV: Additional ablation studies on Feature Fusion Attention module.
R-FFA represents the Feature Fusion Attention module applied to the room
branch, and B-FFA represents the Feature Fusion Attention module applied to
the boundary branch. Max pooling means replacing our average pooling layer
with a max pooling layer.

## References

  * [1] Zhiliang Zeng, Xianzhi Li, Ying Kin Yu, and Chi-Wing Fu. 2019. Deep floor plan recognition using a multi-task network with room-boundary-guided Attention. In _International Conference on Computer Vision_. 2019.
  * [2] Sheraz Ahmed, Marcus Liwicki, Markus Weber, and Andreas Dengel. Improved automatic analysis of architectural floor plans. In _International Conference on Document Analysis and Recognition (ICDAR)_. IEEE, 2011.
  * [3] X. Lv, S. Zhao, X. Yu and B. Zhao, ”Residential floor plan recognition and reconstruction,” _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (CVPR), 2021, pp. 16712-16721, doi: 10.1109/CVPR46437.2021.01644.
  * [4] Chen Liu, Jiajun Wu, Pushmeet Kohli, and Yasutaka Furukawa. Raster-to-Vector: Revisiting floorplan transformation. In _IEEE International Conference on Computer Vision_ (ICCV), 2017.
  * [5] Sheraz Ahmed, Marcus Liwicki, Markus Weber, and Andreas Dengel. Improved automatic analysis of architectural floor plans. In 2011 International Conference on Document Analysis and Recognition, pages 864–869. IEEE, 2011.
  * [6] Sheraz Ahmed, Marcus Liwicki, Markus Weber, and Andreas Dengel. Automatic room detection and room labeling from architectural floor plans. In _2012 10th IAPR International Workshop on Document Analysis Systems_ , pages 339– 343. IEEE, 2012.
  * [7] Christian Ah-Soon and Karl Tombre. Variations on the analysis of architectural drawings. In _International Conference on Document Analysis and Recognition_ (ICDAR).IEEE, 1997.
  * [8] Lucile Gimenez, Sylvain Robert, Fred´ eric Suard, and Khal- ´ doun Zreik. Automatic reconstruction of 3D building models from scanned 2D floor plans. _Automation in Construction_ , 63:48–56, 2016.
  * [9] Xinyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He, and Jiajun Liang. East: an efficient and accurate scene text detector. In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , pages 5551–5560, 2017.
  * [10] Toshihiko Yamasaki, Jin Zhang, and Yuki Takada. Apartment structure estimation using fully convolutional networks and graph model. In _Proceedings of the 2018 ACM Workshop on Multimedia for Real Estate Tech_ , 2018.
  * [11] Bahdanau, Dzmitry and Cho, Kyunghyun and Bengio, Yoshua. Neural Machine Translation by Jointly Learning to Align and Translate. In _International Conference on Learning Representations_ , 2015.
  * [12] Zilong Huang, Xinggang Wang, Yunchao Wei, Lichao Huang, Humphrey Shi, Wenyu Liu, Thomas S. Huang. CCNet: Criss-Cross Attention for Semantic Segmentation. In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * [13] Chenxi Liu, Alex Schwing, Kaustav Kundu, Raquel Urtasun, and Sanja Fidler. Rent3D: Floor-plan priors for monocular layout estimation. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015.
  * [14] Sebastien Mac ´ e, Herv ´ e Locteau, Ernest Valveny, and Salva- ´tore Tabbone. A system to detect rooms in architectural floor plan images. In _Proceedings of the 9th IAPR International Workshop on Document Analysis Systems_ , 2010.
  * [15] Fix, E., Hodges, J.L. Discriminatory analysis, nonparametric discrimination: Consistency properties. Technical Report 4, USAF School of Aviation Medicine, Randolph Field, Texas, 1951. _(Unpublished)_
  * [16] X. Wang, S. Wang, Y. Zhu and X. Meng, ”Image segmentation based on Support Vector Machine,” In _Proceedings of 2012 2nd International Conference on Computer Science and Network Technology_ , 2012, pp. 202-206, doi: 10.1109/ICCSNT.2012.6525921.
  * [17] J. Long, E. Shelhamer and T. Darrell, ”Fully convolutional networks for semantic segmentation”, In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 3431-3440, 2015.
  * [18] Ronneberger, Olaf, Philipp Fischer and Thomas Brox. “U-Net: Convolutional Networks for Biomedical Image Segmentation.” In _MICCAI_ , 2015.
  * [19] He, Kaiming, X. Zhang, Shaoqing Ren and Jian Sun. “Deep Residual Learning for Image Recognition.” In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ (2016): 770-778.
  * [20] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In _European Conference on Computer Vision (ECCV)_ , 2018.
  * [21] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * [22] Wang Jingdong, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu and Bin Xiao. “Deep High-Resolution Representation Learning for Visual Recognition.” In _IEEE Transactions on Pattern Analysis and Machine Intelligence 43 (2021)_ : 3349-3364.
  * [23] Chen Liang-Chieh, George Papandreou, Iasonas Kokkinos, Kevin P. Murphy and Alan Loddon Yuille. “DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs.” In _IEEE Transactions on Pattern Analysis and Machine Intelligence 40 (2018)_ : 834-848.
  * [24] Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. In _International Conference on Learning Representations (ICLR)_ , 2016
  * [25] Adam Paszke, Abhishek Chaurasia, Sangpil Kim, and Eugenio Culurciello. ENet: A deep neural network architecture for real-time semantic segmentation. arXiv preprint arXiv:1606.02147, 2016.
  * [26] Xiaojuan Qi, Zhengzhe Liu, Jianping Shi, Hengshuang Zhao, and Jiaya Jia. Augmented feedback in semantic segmentation under image level supervision. In _European Conference on Computer Vision (ECCV)_ , pages 90–105. Springer, 2016.
  * [27] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. SegNet: A deep convolutional encoder-decoder architecture for image segmentation. In _IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI)_ , 39(12):2481–2495, 2017\.
  * [28] Hengshuang Zhao, Xiaojuan Qi, Xiaoyong Shen, Jianping Shi, and Jiaya Jia. ICNet for real-time semantic segmentation on high-resolution images. In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 405–420, 2018\.
  * [29] Hengshuang Zhao, Yi Zhang, Shu Liu, Jianping Shi, Chen Change Loy, Dahua Lin, and Jiaya Jia. PSANet: Point-wise spatial attention network for scene parsing. In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 267–283, 2018.
  * [30] Zhengzhe Liu, Xiaojuan Qi, and Chi-Wing Fu. 3D-to-2D distillation for indoor scene parsing. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * [31] Yuan Yuhui, Chen Xilin, and Wang Jingdong. Objectcontextual representations for semantic segmentation. arXiv preprint arXiv:1909.11065, 2019.
  * [32] Yuan Yuhui and Wang Jingdong. OCNet: Object context network for scene parsing. arXiv preprint arXiv:1809.00916, 2018\.
  * [33] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 7794–7803, 2018.
  * [34] Chuhang Zou, Alex Colburn, Qi Shan, and Derek Hoiem. LayoutNet: Reconstructing the 3D room layout from a single RGB image. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * [35] Yinda Zhang, Shuran Song, Ping Tan, and Jianxiong Xiao. PanoContext: A whole-room 3D context model for panoramic scene understanding. In _European Conference on Computer Vision (ECCV)_ , 2014.
  * [36] Shang-Ta Yang, Fu-En Wang, Chi-Han Peng, Peter Wonka, Min Sun, and Hung-Kuo Chu. DuLa-Net: A dual-projection network for estimating room layouts from a single RGB panorama. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [37] Toshihiko Yamasaki, Jin Zhang, and Yuki Takada. Apartment structure estimation using fully convolutional networks and graph model. In _Proceedings of the 2018 ACM Workshop on Multimedia for Real Estate Tech_ , pages 1–6. ACM, 2018
  * [38] Cheng Sun, Chi-Wei Hsiao, Min Sun, and Hwann-Tzong Chen. HorizonNet: Learning room layout with 1D representation and pano stretch data augmentation. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [39] Kathy Ryall, Stuart Shieber, Joe Marks, and Murray Mazer. Semi-automatic delineation of regions in floor plans. In _International Conference on Document Analysis and Recognition (ICDAR)_. IEEE, 1995
  * [40] Siu-Hang Or, Kin-Hong Wong, Ying-Kin Yu, and Michael Ming-Yuan Chang. Highly automatic approach to architectural floorplan image understanding and model generation. In _Proc. of Vision, Modeling, and Visualization 2005 (VMV2005)_ , pages 25–32, 2005.
  * [41] Xiaojuan Qi, Zhengzhe Liu, Jianping Shi, Jiaya Jia. Augmented Feedback in Semantic Segmentation under Image Level Supervision In _European Conference on Computer Vision (ECCV), 2016._ , pages 90-105.
  * [42] Liu, Zhengzhe, Xiaojuan Qi, and Chi-Wing Fu. ”One thing one click: A self-training approach for weakly supervised 3d semantic segmentation.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1726-1736.
  * [43] Xie, Enze, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. ”SegFormer: Simple and efficient design for semantic segmentation with transformers.” Advances in Neural Information Processing Systems 34 (2021), pages 12077-12090.
  * [44] Cheng, Bowen, Alex Schwing, and Alexander Kirillov. ”Per-pixel classification is not all you need for semantic segmentation.” Advances in Neural Information Processing Systems 34 (2021): 17864-17875.
  * [45] Federico Boniardi, Abhinav Valada, Rohit Mohan, Tim Caselitz, and Wolfram Burgard, “Robot localization in floor plans using a room layout edge extraction network,” in _2019 IEEE/RSJ International Conference on Intelligent Robots andSystems (IROS)_. IEEE, 2019, pp. 5291–5297.
  * [46] Georg Gerstweiler, Lukas Furlan, Mikhail Timofeev and Hannes Kaufmann. ”Extraction of Structural and Semantic Data from 2D Floor Plans for Interactive and Immersive VR Real Estate Exploration,” in _Technologies, Volume 6 (2018)_ pp.101.
  * [47] So Clifford Kwok-Fung, George Baciu and Hanqiu Sun. “Reconstruction of 3D virtual buildings from 2D architectural floor plans.” In _VRST ’98 (1998)_.
  * [48] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015
  * [49] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In _International Conference on Learning Representations (ICLR)_ , 2019.

|  Zhangyu Wang is currently a rising Senior in Avon Old Farms School, CT,
USA. His research interests mainly focus on computer vision and artificial
intelligence.
---|---
|  Ningyuan Sun is currently a student in University of Wisconsin-Madison, WI,
USA, pursuing the bachelor degree in Computer Science. His research interest
mainly focuses on computer vision and semantic segmentation.
---|---