# The Faber-Krahn inequality for the Hermite operator with Robin boundary
condition

Francesco Chiacchio  Francesco Chiacchio
Università degli Studi di Napoli Federico II
Dipartimento di Matematica e Applicazioni “R. Caccioppoli”
Complesso di Monte Sant’Angelo, Via Cintia, 80126 Napoli, Italia.
<EMAIL_ADDRESS>and  Nunzia Gavitone  Nunzia Gavitone
Università degli Studi di Napoli Federico II
Dipartimento di Matematica e Applicazioni “R. Caccioppoli”
Complesso di Monte Sant’Angelo, Via Cintia, 80126 Napoli, Italia.
<EMAIL_ADDRESS>
(Date: August 27, 2024)

###### Abstract.

In this paper we prove a Faber-Krahn type inequality for the first eigenvalue
of the Hermite operator with Robin boundary condition. We prove that the
optimal set is an half-space and we also address the equality case in such
inequality.

MSC 2020:35P15, 35J25.
Keywords: Faber-Krahn inequality, Hermite operator, Robin boundary condition.

##  1\. Introduction

This paper deals with the eigenvalue problem for the Hermite operator with
Robin boundary condition. Let us denote by

$d\gamma_{N}=\phi_{N}(x)\,dx$

the normalized Gaussian measure in $\mathbb{R}^{N}$, where its density,
$\phi_{N}(x)$, is given by

$\phi_{N}(x)=\frac{1}{\left(2\pi\right)^{\frac{N}{2}}}\exp\left(-\frac{|x|^{2}}{2}\right).$

Let $\Omega$ be a sufficiently smooth domain of $\mathbb{R}^{N}$ and let $\nu$
be the unit outer normal to $\partial\Omega$. We consider the following
eigenvalue problem

(1.1) $\left\\{\begin{array}[]{ll}-\text{div}\left(\phi_{N}(x)\nabla
u(x))\right)=\lambda(\Omega)\phi_{N}(x)u(x)&\text{in }\Omega\\\\[5.69046pt]
&\\\ \dfrac{\partial u}{\partial\nu}+\beta u=0&\text{on
}\partial\Omega,\end{array}\right.$

where $\beta>0$. Our aim is to prove a Faber-Krahn inequality for the first
eigenvalue, $\lambda_{1}(\Omega)$, of problem (1.1). More precisely, we are
interested in finding, in the class of the sets having prescribed Gaussian
measure, the one which minimizes $\lambda_{1}(\Omega)$.

The Faber-Krahn inequality for the first Robin eigenvalue, for the classical
Laplace operator, has been established in the planar case in [9].
Subsequently, such a result has been generalized in any dimension in [20] and
for the $p$-Laplace operator in [12]. In all these cases the ball turns out to
be the optimal set, when the Robin parameter $\beta$ is positive.

For $\beta<0$, the problem appears more delicate. Indeed many phenomena can
occur, depending also on the dimension and the topological properties of the
domain (see, e.g., [24], [23], [4] and the references therein).

The difficulties of the problem when $\beta>0$ are twofold. From on hand the
level sets of the first eigenfunction are not closed, on the other, there is
no monotonicity of the first eigenvalue with respect to the inclusion of sets.
These facts prevent one from adapting the classical symmetrization methods,
which work for the Dirichlet boundary condition. In order to overcome these
difficulties, in the papers quoted above, the authors used a different
approach. More precisely, in [9], Bossel introduced a sort of
“desymmetrization” technique together with a representation formula for the
first Robin eigenvalue. Concerning problem (1.1), some further issues arise
since, in general, the domain $\Omega$ is unbounded and the first
eigenfunction is not in $L^{\infty}(\Omega).$ These circumstances do not allow
to apply, in a straightforward way, the Bossel’s arguments. Our analysis
requires a detailed study of problem (2.5) in one dimension (see Section 3)
and a suitable modification of the proof of the representation formula (see
Section 4). One crucial point, in the study of the one-dimensional problem, is
the asymptotic behavior of the first eigenfunction, since it allows to prove,
among other things, its $\log$-concavity which, in turn, enables to prove the
monotonicity property, with respect the inclusion of half lines, of the first
eigenvalue.

Let us spend a few words about some motivations of the present note. First of
all, the Hermite operator, as well known, enter in the description of the
harmonic oscillator in quantum mechanics (see, e.g., [13] and the references
therein). Moreover the interest in the Hermite operator comes from the fact
that Gaussian measure in $\mathbb{R}^{N}$ can be obtained as a limit, as $k$
goes to infinity, of the normalized surface measures on
$\mathbb{S}_{\sqrt{k}}^{k+N+1}$, the sphere in $\mathbb{R}^{k+N+2}$ of radius
$\sqrt{k}$ (a process known in literature as “Poincarè limit”).

Note also that the Robin problem is often regarded as interpolating between
the Dirichlet ($\beta=0$) and Neumann ($\beta=+\infty$) cases.

As far as the Hermite operator is considered and $\Omega$ varies in the class
of domains with fixed Gaussian measure, it is known (see [21] and [5]) that
the set which minimizes the first Dirichlet eigenvalue is given by an half-
space. Hence it coincides with the isoperimetric set in the Gaussian
Isoperimetric inequality (see, e.g., [30], [8], [14], [17] and the references
therein). On the other hand, when Neumann boundary condition are imposed, the
situation is quite different. As well known, the problem of minimizing the
first non-trivial Neumann eigenvalue, $\mu_{1}(\Omega)$, is meaningless and,
hence, one tries to maximize it, instead. In [15] it has been proved that
among all smooth domains with prescribed Gaussian measure, symmetric with
respect to the origin the ball centered at the origin maximizes
$\mu_{1}(\Omega)$. Furthermore in [15] it is shown that, even removing such a
topological assumption, the half spaces do not maximize it.

This phenomenon is quite surprising since in the Euclidean case the ball
minimizes the first Dirichlet eigenvalue (the classical Faber-Krahn
inequality) and, at same time, it maximizes the first nontrivial Neumann
eigenvalue (the classical Szegö-Weiberger inequality).

So it is natural to investigate which is the optimal set in the “Gaussian-
Faber-Krahn inequality” when Robin boundary condition are imposed.

We finally point out that in [19] the authors prove an isoperimetric
inequality for the Robin torsional rigidity related to the Hermite operator.

Our main result, Theorem 1.1 below, requires the validity of some functional
embedding Theorems. In Section 2 (see definition 2.1) we introduce and
describe the family $\mathcal{G}$ of those domains of $\mathbb{R}^{N}$ for
which such results hold true.

######  Theorem 1.1.

Let $\Omega\in\mathcal{G}$, with $\mathcal{G}$ as in Definition 2.1. Then

(1.2) $\lambda_{1}(\Omega^{{}^{\\#}})\leq\lambda_{1}(\Omega),$

where $\Omega^{{}^{\\#}}=S_{\sigma^{{}^{\\#}}}$ is any half-space such that
$\gamma_{N}(\Omega^{{}^{\\#}})=\gamma_{N}(\Omega)$. Moreover, equality holds

in (1.2), if and only if, $\Omega$ is an half-space, modulo a rotation about
the origin.

The paper is organized as follows. Section 2 contains the isoperimetric
inequality with respect to the Gaussian measure, along with some preliminary
results on the eigenvalue problem (1.1). In the third Section we analyze the
problem in one dimension. As said before, among other things, we prove the
$\log$-concavity of any positive first eigenfunction. Note that in
$\mathbb{R}^{N}$ such a property does not hold true, in general, even for the
classical Laplace operator (see [3]). Section 4 is devoted to the
representation formula for the first eigenvalue of the problem (1.1). Finally,
in the last Section, we prove our main result, Theorem 1.1.

##  2\. Preliminary results

###  2.1. The Gaussian isoperimetric inequality

Let $\phi_{N}(x)$ denotes the density of the normalized Gaussian measure in
$\mathbb{R}^{N}\,$ i.e.

$\phi_{N}(x)=\frac{1}{\left(2\pi\right)^{\frac{N}{2}}}\exp\left(-\frac{|x|^{2}}{2}\right).$

One defines the Gaussian perimeter of any Lebesgue measurable set $A$ of
$\mathbb{R}^{N}$ as follows

$P_{\phi_{N}}(A)=\left\\{\begin{array}[]{cc}\displaystyle\int_{\partial
A}\phi_{N}(x)\text{ }d\mathcal{H}^{N-1}(x)&\text{ if }\partial A\text{ is
}(N-1)-\text{rectifiable}\\\ &\\\ +\infty&\text{otherwise,}\end{array}\right.$

where $d\mathcal{H}^{N-1}$ denotes the $(N-1)-$dimensional Hausdorff measure
in $\mathbb{R}^{N}$.

While the Gaussian measure of $A$ is given by

(2.1) $\gamma_{N}(A)=\int_{A}\phi_{N}(x)\,dx\in\left[0,1\right].$

The celebrated Gaussian isoperimetric inequality (see [30], [8] and [21])
states that among all Lebesgue measurable sets in $\mathbb{R}^{N},$ with
prescribed Gaussian measure, the half-spaces minimize the Gaussian perimeter.
Furthermore the isoperimetric set is unique, clearly, up to a rotation with
respect to the origin (see [14] and [17]).

Let $a\in\mathbb{R}$, in the sequel we will use the following notation

(2.2) $S_{a}:=\\{x=(x_{1},x_{2},\cdots,x_{n})\in\mathbb{R}^{N}\text{
}\colon\text{ }x_{1}<a\\}.$

If $\Omega\subset\mathbb{R}^{N}$ is a Lebesgue measurable set, its Gaussian
symmetrized, $\Omega^{{}^{\\#}},$ is

(2.3) $\Omega^{{}^{\\#}}=S_{\sigma^{{}^{\\#}}}$

where $\sigma^{{}^{\\#}}$ is such that

(2.4)
$\gamma_{N}(\Omega)=\gamma_{N}(\Omega^{{}^{\\#}})=\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\sigma^{{}^{\\#}}}\,\text{exp}\left(-\dfrac{t^{2}}{2}\right)dt=\frac{1}{2}+\frac{1}{2}\operatorname{erf}\left(\frac{\sigma^{{}^{\\#}}}{\sqrt{2}}\right),$

where $\operatorname{erf}$ stands for the standard error function.

The isoperimetric function in Gauss space, $g(s),$ is given by

$g:s\in\left[0,1\right]\rightarrow
g(s)=\frac{1}{\sqrt{2\pi}}\exp\left[-\frac{\left(\operatorname{erf}^{-1}(2s-1)\right)^{2}}{2}\right].$

Note, indeed, that the Gaussian perimeter of any half-space of Gaussian
measure $s\in\left[0,1\right]$ is equal to $g(s).$ The isoperimetric property
of the half-spaces can finally be stated also in the following analytic form.

######  Theorem 2.1.

If $\Omega\subset\mathbb{R}^{N}$ is any Lebesgue measurable set it holds that

$P_{\phi_{N}}(\Omega)\geq
P_{\phi_{N}}(\Omega^{{}^{\\#}})=g\left(\gamma_{N}(\Omega)\right),$

where equality holds, if and only if, $\Omega$ is equivalent to an half-space.

###  2.2. The Sobolev space $H^{1}(\Omega,\phi_{N})$

Let $\Omega\subset\mathbb{R}^{n}$ be an open connected set. We will denote by
$L^{2}(\Omega,\phi_{N})$ the set of all real measurable functions defined in
$\Omega$, such that

$\left\|u\right\|_{L^{2}(\Omega,\phi_{N})}^{2}:=\int_{\Omega}u^{2}(x)\phi_{N}(x)dx<+\infty.$

For our future purposes we need also to introduce the following weighted
Sobolev space

$H^{1}(\Omega,\phi_{N}):=\left\\{u\in
W_{\text{loc}}^{1,1}(\Omega):(u,\left|\nabla u\right|)\in
L^{2}(\Omega,\phi_{N})\times L^{2}(\Omega,\phi_{N})\right\\},$

endowed with the norm

$\left\|u\right\|_{H^{1}(\Omega,\phi_{N})}=\left\|u\right\|_{L^{2}(\Omega,\phi_{N})}+\left\|\nabla
u\right\|_{L^{2}(\Omega,\phi_{N})}.$

In the sequel of the paper, we need to introduce the following family of
sets..

######  Definition 2.1.

A Lipschitz domain $\Omega$ of $\mathbb{R}^{N}$ is in $\mathcal{G}$ if
$\gamma_{N}(\Omega)\in(0,1)$ and the following conditions are fulfilled:

  * (i)

$H^{1}(\Omega,\phi_{N})$ is compactly embedded in $L^{2}(\Omega,\phi_{N})$.

  * (ii)

The trace operator $T$

$T:u\in H^{1}(\Omega,\phi_{N})\rightarrow\left.u\right|_{\partial\Omega}\in
L^{2}(\partial\Omega,\phi_{N}),$

is well defined;

  * (iii)

The trace operator defined in the previous point is compact from
$H^{1}(\Omega,\phi_{N})$ onto $L^{2}(\partial\Omega,\phi_{N})$.

In (ii) and (iii) the functional space $L^{2}(\partial\Omega,\phi_{N})$ is
endowed with the norm

$\left\|u\right\|_{L^{2}(\partial\Omega,\phi_{N})}^{2}=\int_{\partial\Omega}u^{2}(x)\phi_{N}(x)d\mathcal{H}^{N-1}(x).$

######  Remark 2.1.

Observe that $\mathcal{G}$ is non empty. Indeed it contains, at least, the
following families of sets

  * (j)

All the bounded and Lipschitz domains of $\mathbb{R}^{N}$.

  * (jj)

All the convex domains of $\mathbb{R}^{N}$, not necessarily bounded.

  * (jjj)

All the Lipschitz domains of $\mathbb{R}^{N}$, not necessarily bounded, for
which an extension Theorem holds true (see, e.g., [22], [27], [11] and [28]).

The study of functional inequalities related to the Gaussian measure, also
because they are often generalizable to infinite-dimensional spaces, has given
rise to a rich line of research, starting from the seminal paper by Gross (see
[26]). The related bibliography is very wide, we remind the interested reader
to [1], [2], [6], [7], [10], [16], [18], [29] and the references therein.

###  2.3. The eigenvalue problem

Let $\Omega\in\mathcal{G}$, with $\mathcal{G}$ as in Definition 2.1. We
finally consider the eigenvalue Robin boundary value problem

(2.5) $\left\\{\begin{array}[]{ll}-\operatorname{div}\left(\phi_{N}(x)\nabla
u(x))\right)=\lambda(\Omega)\phi_{N}(x)u(x)&\text{in }\Omega,\\\\[5.69046pt]
\displaystyle\frac{\partial u}{\partial\nu}+\beta u=0&\text{on
}\partial\Omega.\end{array}\right.$

where $\nu$ is the unit outer normal to $\partial\Omega$ and $\beta>0$. A real
number $\lambda(\Omega)$ and a function $u\in H^{1}(\Omega,\phi_{N})$ are
respectively called eigenvalue and corresponding eigenfunction of the problem
(2.5) if the following equality holds

(2.6) $\int_{\Omega}\langle\nabla
u,\nabla\psi\rangle\,\phi_{N}\,dx+\beta\int_{\partial\Omega}u\psi\phi_{N}\,\,d\mathcal{H}^{N-1}(x)=\lambda(\Omega)\int_{\Omega}u\,\psi\,\phi_{N}\,dx,\quad\psi\in
H^{1}(\Omega,\phi_{N}).$

By the definition of $\mathcal{G}$, one can apply the standard theory on
compact, self-adjoint operator to problem (2.5). Hence we can arrange the
eigenvalues of (2.5) in a non-decreasing sequence
$\\{\lambda_{n}(\Omega)\\}_{n\in\mathbb{N}}$, such that
$\lim_{n\to\infty}\lambda_{n}(\Omega)=+\infty$. Moreover, the smallest
eigenvalue of (2.5), $\lambda_{1}(\Omega)$, has the following variational
characterization

(2.7) $\lambda_{1}(\Omega)=\inf_{v\in
H^{1}(\Omega,\phi_{N})\setminus\\{0\\}}J[\beta,v],$

where

(2.8) $J[\beta,v]=\frac{\displaystyle\int_{\Omega}|\nabla
v|^{2}\phi_{N}\,dx+\beta\displaystyle\int_{\partial\Omega}v^{2}\phi_{N}\,d\mathcal{H}^{N-1}(x)}{\displaystyle\int_{\Omega}v^{2}\phi_{N}\,dx}.$

Conversely, any function on which the functional $J[\beta,v]$ achieves its
minimum is an eigenfunction of (2.5). Furthermore the following result holds.

######  Theorem 2.2.

Let $\Omega\in\mathcal{G}$, with $\mathcal{G}$ as in Definition 2.1 and let
$\lambda_{1}(\Omega)$ be the first eigenvalue of (2.5). Then the corresponding
eigenfunctions are smooth, have constant sign in $\Omega$ and
$\lambda_{1}(\Omega)$ is simple, that is the first eigenfunction is unique up
to a multiplicative constant.

###### Proof.

By classical regularity results the first eigenfunctions are smooth in
$\Omega$. Let $u$ be an eigenfunction corresponding to $\lambda_{1}(\Omega)$,
then both $u$ and $|u|$ minimize (2.7), since $J(\beta,u)=J(\beta,|u|)$.
Therefore $|u|$ is a first eigenfunction too. By Harnack’s inequality we
deduce that $|u|$ can not vanishes inside $\Omega$ and therefore $u$ has one
sign in $\Omega$. The simplicity of $\lambda_{1}(\Omega)$ follows repeating
the standard arguments for the classical Laplace operator. ∎

######  Remark 2.2.

Since eigenfunctions corresponding to different eigenvalues are mutually
orthogonal in $L^{2}(\Omega,\phi_{N})$, the previous result implies that any
positive function $v\in H^{1}(\Omega,\phi_{N})$, which solves problem (2.5)
for some $\lambda\in\mathbb{R}$, is a first eigenfunction, that is
$\lambda=\lambda_{1}(\Omega)$.

##  3\. The eigenvalue problem in one dimension

In this Section we consider problem (2.5) in one dimension, that is

(3.1)
$\left\\{\begin{array}[]{ll}-w^{\prime\prime}+tw^{\prime}=\lambda(\sigma)w&\,\,t\in
I_{\sigma}\\\\[5.69046pt] w^{\prime}(\sigma)+\beta
w(\sigma)=0,\end{array}\right.$

where $I_{\sigma}=(-\infty,\sigma)$.

By Theorem 2.2 the first eigenvalue $\lambda_{1}(\sigma)$ is simple and the
corresponding eigenfunctions $w(t)$ are smooth and have one sign in
$I_{\sigma}$. In this case, the variational characterization reads as

(3.2) $\lambda_{1}(\sigma)=\min_{\begin{subarray}{c}v\in
H^{1}\left(I_{\sigma},\phi_{1}\right)\setminus\\{0\\}\end{subarray}}\displaystyle\frac{\displaystyle\int_{-\infty}^{\sigma}(v^{\prime}(t))^{2}e^{-\frac{t^{2}}{2}}\,dt+\beta(v(\sigma))^{2}e^{-\frac{\sigma^{2}}{2}}}{\displaystyle\int_{-\infty}^{\sigma}(v^{\prime}(t))^{2}e^{-\frac{t^{2}}{2}}\,dt},$

By standard theory on hypergeometric functions (see for example [21],[31]),
the minimizers $w$ of $\lambda_{1}(\sigma)$ have the following Taylor
expansion

(3.3)
$w(t)=C\sum_{m=0}^{\infty}\binom{\lambda_{1}(\sigma)}{m}\displaystyle\frac{\left(t/\sqrt{2}\right)^{m}}{\Gamma\left(\frac{1-\lambda_{1}(\sigma)+m}{2}\right)},$

where $C\in\mathbb{R}$ and $\Gamma$ is the usual Gamma function. In particular
when $\lambda_{1}(\sigma)\in\mathbb{N}$, the right hand-side in (3.3) reduces
to the following Hermite polynomial of degree $\lambda_{1}(\sigma)$

$w(t)=C\text{exp}\left(\frac{t^{2}}{2}\right)\displaystyle\frac{d^{\lambda_{1}(\sigma)}}{dt^{\lambda_{1}(\sigma)}}\text{exp}\left(-\frac{t^{2}}{2}\right).$

Moreover it is possible to show (see, for instance, [31], pp. 34-35 and pp.
72-76) that $w$ has the following asymptotic behavior

(3.4) $w\propto t^{\lambda_{1}(\sigma)}\left(1+O(t^{-2})\right)\quad\text{for
}t\to-\infty.$

Problem (3.1) is strictly related to problem (2.5) when $\Omega$ is an half-
space. More precisely let $\sigma\in\mathbb{R}$ and $S_{\sigma}$ as in (2.2),
and let us consider the following problem

(3.5) $\left\\{\begin{array}[]{ll}-\operatorname{div}\left(\phi_{N}(x)\nabla
u)\right)=\lambda_{1}(S_{\sigma})\phi_{N}(x)u(x)&\text{in
}S_{\sigma},\\\\[5.69046pt] \displaystyle\frac{\partial u}{\partial
x_{1}}+\beta u=0&\text{on }\\{x_{1}=\sigma\\},\end{array}\right.$

where $\lambda_{1}(S_{\sigma})$ is the first eigenvalue given by

(3.6) $\lambda_{1}(S_{\sigma})=\min_{v\in
H^{1}(S_{\sigma},\phi_{N})\setminus\\{0\\}}\displaystyle\frac{\operatorname{\displaystyle\int}_{S_{\sigma}}|\nabla
v|^{2}\phi_{N}\,dx+\beta\operatorname{\displaystyle\int}_{\begin{subarray}{c}\\{x_{1}=\sigma\\}\end{subarray}}v^{2}\phi_{N}\,d\mathcal{H}^{N-1}(x)}{\operatorname{\displaystyle\int}_{S_{\sigma}}v^{2}\phi_{N}\,dx}$

Theorem 2.2 ensures that $\lambda_{1}(S_{\sigma})$ is simple and it admits a
positive corresponding eigenfunction $u(x)$. In what follows we observe that
the eigenfunctions $u$ are determined by the ones of problem (3.1).

######  Theorem 3.1.

Let $u$ be a positive eigenfunction corresponding to $\lambda_{1}(\sigma)$.
Then there exists a function $w(t)\colon I_{\sigma}\to(0,+\infty)$ such that
$u(x)=w(x_{1})$. Moreover $w$ is strictly monotone decreasing in $I_{\sigma}$
and it solves (3.1), with $\lambda_{1}(\sigma)=\lambda_{1}(S_{\sigma})$.

###### Proof.

Let $w(t)$ be a positive eigenfunction of problem (3.1) corresponding to
$\lambda_{1}(\sigma)$. Then it solves

(3.7)
$\left\\{\begin{array}[]{ll}-\left(w^{\prime}\phi_{1}(t)\right)^{\prime}=\lambda_{1}(\sigma)\phi_{1}(t)w&\,\,t\in
I_{\sigma},\\\\[8.5359pt] w^{\prime}(\sigma)+\beta
w(\sigma)=0.\end{array}\right.$

Since $w>0$ in $I_{\sigma}$, we clearly have that

(3.8) $\left(w^{\prime}\phi_{1}(t)\right)^{\prime}<0,\,\,\,\,t\in I_{\sigma}.$

We claim that

(3.9) $\lim_{t\to-\infty}w^{\prime}(t)e^{-\frac{t^{2}}{2}}=0.$

Note that, by (3.8), such a limit exists. Now assume, by absurd, that

$\lim_{t\to-\infty}w^{\prime}(t)e^{-\frac{t^{2}}{2}}=L\in(-\infty,+\infty]\setminus\\{0\\}.$

This would imply that

$\int_{-\infty}^{\sigma}(w^{\prime}(t))^{2}e^{-\frac{t^{2}}{2}}\,dt=+\infty,$

an absurd, since $w\in H^{1}(I_{\sigma},\phi_{1})$. The claim (3.9) is
therefore proved.

From (3.9) and (3.8) it follows that $w^{\prime}(t)<0$ in $I_{\sigma}$.
Defining $u(x)=w(x_{1})$, $x\in S_{\sigma}$, by Theorem 2.2 and the simplicity
of the first eigenvalue, we get that
$\lambda_{1}(\sigma)=\lambda_{1}(S_{\sigma})$.

This concludes the proof of the Theorem. ∎

Let $w(t)$ as in Theorem 3.1 and let us define the following function

(3.10) $\beta(t)=\displaystyle-\frac{w^{\prime}(t)}{w(t)},\,\,\,\,t\in
I_{\sigma},$

then it holds

######  Proposition 3.1.

The function $\beta(t)$ defined in (3.10) is positive, strictly increasing in
$I_{\sigma}$ and $\beta(\sigma)=\beta$.

###### Proof.

Being w a solution to (3.1), Theorem 3.1 implies that $\beta>0.$ Moreover,
since $w^{\prime}$ verifies in $I_{\sigma}$ the following equation

(3.11)
$-(w^{\prime})^{\prime\prime}+t(w^{\prime})^{\prime}=\left(\lambda(\sigma)-1\right)w^{\prime},$

taking into account the asymptotic behavior given in (3.4), we have that

(3.12) $\lim_{t\to-\infty}\beta(t)=0.$

Moreover we have that

$\beta^{\prime}(t)=\displaystyle\frac{-w^{\prime\prime}w+(w^{\prime})^{2}}{w^{2}}\qquad
t\in I_{\sigma}.$

Since $w^{\prime\prime}$ is an eigenfunction of the one-dimensional Hermite
problem corresponding to the eigenvalue $\lambda(\sigma)-2$, again (3.12)
yields

(3.13) $\lim_{t\to-\infty}\beta^{\prime}(t)=0.$

Moreover by (3.1) it holds

$\beta^{\prime}(t)=\displaystyle\frac{-tww^{\prime}+(w^{\prime})^{2}+\lambda(\sigma)w^{2}}{w^{2}}=t\beta(t)+\lambda(\sigma)+\beta^{2}(t),\qquad
t\in I_{\sigma}.$

Hence denoted by $z=\beta^{\prime}$ we have

$z^{\prime}=tz+\beta(t)+2\beta(t)z>z(2\beta(t)+t),$

with $\lim_{t\to-\infty}z=0$. Then $z>0$ that is, $\beta(t)$ is strictly
increasing and this completes the proof. ∎

######  Remark 3.1.

We observe that Theorem 3.1 implies that any positive eigenfunction of problem
(3.1) is log-concave and the same clearly, also true for problem (3.5). Indeed
if we consider

$f(t)=\log(w(t)),\quad t\in I_{\sigma}$

then

$f^{\prime}(t)=-\beta(t)\text{ and }f^{\prime\prime}(t)=-\beta^{\prime}(t)<0.$

Let $\lambda_{1}(\sigma)$ be as in (3.2). Proposition 3.1 implies the
following monotonicity result for the first eigenvalue $\lambda_{1}(\sigma)$
of the half-spaces.

######  Proposition 3.2.

Let $r,\,\sigma\in\mathbb{R}$ such that $r\leq\sigma$. Then
$\lambda_{1}(r)\geq\lambda_{1}(\sigma)$.

###### Proof.

Let $\beta(r)$ be the function defined in (3.10) and $\beta$ the parameter
which appears in the Robin boundary condition in (3.1), we have

(3.14)
$\displaystyle\begin{split}\lambda_{1}(\sigma)&=\displaystyle\frac{\operatorname{\displaystyle\int}_{-\infty}^{r}(w^{\prime}(t))^{2}e^{-\frac{t^{2}}{2}}\,dt+\beta(r)(w(r))^{2}e^{-\frac{r^{2}}{2}}}{\operatorname{\displaystyle\int}_{-\infty}^{r}(w^{\prime}(t))^{2}e^{-\frac{t^{2}}{2}}\,dt}\\\
&=\min_{v\in
H^{1}\left(I_{\sigma},\phi_{1}(t)\right)\setminus\\{0\\}}\displaystyle\frac{\operatorname{\displaystyle\int}_{-\infty}^{r}(v^{\prime}(t))^{2}e^{-\frac{t^{2}}{2}}\,dt+\beta(r)(v(r))^{2}e^{-\frac{r^{2}}{2}}}{\operatorname{\displaystyle\int}_{-\infty}^{r}(v^{\prime}(t))^{2}e^{-\frac{t^{2}}{2}}\,dt}\\\
&\leq\min_{v\in
H^{1}\left(I_{\sigma},\phi_{1}(t)\right)\setminus\\{0\\}}\displaystyle\frac{\operatorname{\displaystyle\int}_{-\infty}^{r}(v^{\prime}(t))^{2}e^{-\frac{t^{2}}{2}}\,dt+\beta\,\,(v(r))^{2}e^{-\frac{r^{2}}{2}}}{\operatorname{\displaystyle\int}_{-\infty}^{r}(v^{\prime}(t))^{2}e^{-\frac{t^{2}}{2}}\,dt}\\\
&=\lambda_{1}(r),\end{split}$

where the last inequality follows by Proposition 3.1. ∎

##  4\. A representation formula for $\lambda_{1}(\Omega)$

Let $\Omega\in\mathcal{G}$, with $\mathcal{G}$ as in Definition 2.1 and let
$u$ be the first positive eigenfunction of (2.5) such that
$\|u\|_{L^{2}(\Omega,\phi_{N})}=1$. From now on, for every $t\in[0,1]$, we
will use the following notation

$\begin{array}[]{l}U_{t}=\\{x\in\Omega\colon u>t\\},\\\\[2.84544pt] \partial
U^{\text{int}}_{t}=\\{x\in\Omega\colon u=t\\},\\\\[2.84544pt] \partial
U^{\text{ext}}_{t}=\\{x\in\partial\Omega\colon u>t\\}.\end{array}$

Let $\psi\in L^{2}(\Omega,\phi_{N})$ be a non-negative function and let us
consider the following functional

(4.1)
$\mathcal{F}_{\Omega}(U_{t},\psi)=\frac{1}{\gamma_{N}(U_{t})}\left(-\int_{U_{t}}\psi^{2}\phi_{N}\,dx+\int_{\partial
U^{\text{int}}_{t}}\psi\phi_{N}\,d\mathcal{H}^{N-1}(x)+\beta\int_{\partial
U^{\text{ext}}_{t}}\phi_{N}\,d\mathcal{H}^{N-1}(x)\right).$

The following level set representation formula for $\lambda_{1}(\Omega)$
holds.

######  Theorem 4.1.

Let $\Omega\in\mathcal{G}$, with $\mathcal{G}$ as in Definition 2.1 and let
$u$ be the positive minimizer of (2.5) such that
$\|u\|_{L^{2}(\Omega,\phi_{N})}=1$. Then, for a.e. $t>0$, it holds true

(4.2) $\lambda_{1}(\Omega)=\mathcal{F}_{\Omega}\left(U_{t},\bar{\psi}\right),$

where $\bar{\psi}=\displaystyle\frac{|\nabla u|}{u}$ and
$\mathcal{F}_{\Omega}$ is defined in (4.1).

###### Proof.

Being $u$ smooth and positive in $\Omega$, we can divide both terms in the
equation in (2.5) by $u$, and integrate on $U_{t}$. Then, by the boundary
condition we get

(4.3)
$\begin{split}\lambda(\Omega)\gamma_{N}(U_{t})&=\displaystyle\int_{U_{t}}\displaystyle\frac{-\operatorname{div}\left(\phi_{N}(x)\nabla
u)\right)}{u}\,dx=-\displaystyle\int_{\partial U_{t}}\frac{\partial
u}{\partial\nu}\frac{1}{u}\phi_{N}(x)\,d\mathcal{H}^{N-1}(x)-\int_{U_{t}}\displaystyle\frac{|\nabla
u|^{2}}{u^{2}}\phi_{N}(x)\,dx\\\ &=\displaystyle\int_{\partial
U^{\text{int}}_{t}}\frac{|\nabla
u|}{u}\phi_{N}(x)\,d\mathcal{H}^{N-1}(x)+\beta\displaystyle\int_{\partial
U^{\text{ext}}_{t}}\phi_{N}(x)\,d\mathcal{H}^{N-1}(x)-\int_{U_{t}}\displaystyle\frac{|\nabla
u|^{2}}{u^{2}}\phi_{N}(x)\,dx\\\
&=\displaystyle\mathcal{F}_{\Omega}\left(U_{t},\bar{\psi}\right),\end{split}$

and this concludes the proof. ∎

######  Theorem 4.2.

Let $\Omega\in\mathcal{G}$, with $\mathcal{G}$ as in Definition 2.1 and let
$\bar{\psi}$ as in Theorem 4.1. Let $\psi\in L^{2}(\Omega,\phi_{N})$ be a
nonnegative function such that $\psi\not\equiv\bar{\psi}$ and let
$\mathcal{F}_{\Omega}$ be as in (4.1). Set

$w(x):=\psi-\bar{\psi},\qquad I(t):=\int_{U_{t}}w\bar{\psi}\phi_{N}\,dx,$

then $I\colon]0,+\infty[\rightarrow\mathbb{R}$ is locally absolutely
continuous and

(4.4)
$\mathcal{F}_{\Omega}(U_{t},\psi)\leq\lambda_{1}(\Omega)-\frac{1}{\gamma_{N}(U_{t})}\frac{1}{t}\Big{(}\frac{d}{dt}t^{2}I(t)\Big{)},$

for almost every $t>0$.

###### Proof.

In order to prove (4.4), writing the representation formula (4.2) in terms of
$w$, it follows that, for a.e. $t>0$,

(4.5)
$\begin{split}\mathcal{F}_{\Omega}(U_{t},\psi)&=\lambda_{1}(\Omega)+\frac{1}{\gamma_{N}(U_{t})}\left(\int_{\partial
U_{t}^{\text{int}}}w\phi_{N}\,d\mathcal{H}^{N-1}(x)-\int_{U_{t}}\Big{(}\psi^{2}-\bar{\psi}^{2}\Big{)}\phi_{N}\,dx\right)\\\
&\leq\lambda_{1}(\Omega)+\frac{1}{\gamma_{N}(U_{t})}\left(\int_{\partial
U_{t}^{\text{int}}}w\phi_{N}\,d\mathcal{H}^{N-1}(x)-2\int_{U_{t}}w\bar{\psi}\phi_{N}\,dx\right)\\\
&=\lambda_{1}(\Omega)+\frac{1}{\gamma_{N}(U_{t})}\left(\int_{\partial
U_{t}^{\text{int}}}w\phi_{N}\,d\mathcal{H}^{N-1}(x)-2\,I(t)\right)\end{split}$

where the inequality in (4.5) follows since $\psi,\bar{\psi}\geq 0$. Applying
the coarea formula, it is possible to rewrite $I(t)$ as

$I(t)=\int_{U_{t}}w\bar{\psi}\phi_{N}\,dx=\int_{t}^{+\infty}\frac{1}{\tau}d\tau\int_{\partial
U_{\tau}^{\text{int}}}w\,\phi_{N}\,d\mathcal{H}^{N-1}(x).$

This assures that $I(t)$ is locally absolutely continuous in $]0,+\infty[$
and, for almost every $t>0$ we have

$-\frac{d}{dt}\big{(}t^{2}I(t)\big{)}=t\left(\int_{\partial
U_{t}^{\text{int}}}w\,\phi_{N}\,d\mathcal{H}^{N-1}(x)-2I(t)\right).$

Substituting in (4.5), the inequality (4.4) follows. ∎

######  Theorem 4.3.

Let $\Omega\in\mathcal{G}$, with $\mathcal{G}$ as in Definition 2.1, and let
$\bar{\psi}$ be as in Theorem 4.1. Let $\psi\in L^{2}(\Omega,\phi_{N})$ be a
nonnegative function such that $\psi\not\equiv\bar{\psi}$ and let
$\mathcal{F}_{\Omega}$ be as in (4.1). Then there exists a set
$T\subset]0,+\infty[$ with positive Lebesgue measure such that for every $t\in
T$ it holds that

(4.6) $\lambda_{1}(\Omega)\geq\mathcal{F}_{\Omega}(U_{t},\psi).$

###### Proof.

Let $u$ be the first positive eigenfunction of (2.5) such that
$\|u\|_{L^{2}(\Omega,\phi_{N})}=1$. We have to discuss two cases. If $u\in
L^{\infty}(\Omega)$ then the claim follows by repeating line by line the
arguments in [20] and [12]. Hence, in the proof below, we will assume that $u$
is not bounded. Note that, by the asymptotic behaviour given in (3.4), this
case occurs surely when $\Omega$ is any half-space. In order to prove (4.6),
let us proceed by contradiction. Then we assume that there exists a
nonnegative function $\psi\in L^{2}(\Omega,\phi_{N})$ such that the reverse
inequality holds, that is

(4.7) $\lambda_{1}(\Omega)<\mathcal{F}_{\Omega}(U_{t},\psi),$

for almost all $t>0$. Let $\psi_{n}\in C^{\infty}_{c}(\Omega)$ be a sequence
of nonnegative functions such that $\psi_{n}\to\psi$ in
$L^{2}(\Omega,\phi_{N})$. Then by Fatou’s lemma we have

$\lambda_{1}(\Omega)<\mathcal{F}_{\Omega}(U_{t},\psi)\leq\liminf_{n\to\infty}\mathcal{F}_{\Omega}(U_{t},\psi_{n}).$

Let $\psi_{n_{k}}$ be a subsequence such that

$\lim_{k\to\infty}F_{\Omega}(U_{t},\psi_{n_{k}})=\liminf_{n\to\infty}\mathcal{F}_{\Omega}(U_{t},\psi_{n}).$

In order to simplify the notation, we will still denote by $\psi_{n}$ such a
subsequence the subsequence. Then by (4.7) and Theorem 4.3 we have

$\lambda_{1}(\Omega)<\lim_{n\to\infty}\mathcal{F}_{\Omega}(U_{t},\psi_{n})\leq\lambda_{1}(\Omega)-\lim_{n\to\infty}\frac{1}{\gamma_{N}(U_{t})}\frac{1}{t}\Big{(}\frac{d}{dt}t^{2}I_{n}(t)\Big{)},$

where

$I_{n}(t)=\int_{U_{t}}w_{n}\bar{\psi}\phi_{N}\,dx$

with $w_{n}=\psi_{n}-\bar{\psi}$. Then for $n$ large enough it has to hold

(4.8) $\frac{d}{dt}t^{2}I_{n}(t)<0,$

almost everywhere in $]0,+\infty[$. Since

(4.9)
$\begin{split}\lim_{t\to+\infty}t^{2}|I_{n}(t)|&\leq\lim_{t\to+\infty}t^{2}\int_{U_{t}}|\psi_{n}-\bar{\psi}|\bar{\psi}\phi_{N}\,dx\leq\lim_{t\to+\infty}\int_{U_{t}}u^{2}|\psi_{n}-\bar{\psi}|\bar{\psi}\phi_{N}\,dx\\\
&\leq\lim_{t\to+\infty}\int_{U_{t}\cap\text{supp}\psi_{n}}\psi_{n}u|\nabla
u|\phi_{N}\,dx+\int_{U_{t}}|\nabla u|^{2}\phi_{N}\,dx=0,\end{split}$

and then for sufficiently large $n$ we have

(4.10) $\lim_{t\to+\infty}t^{2}I_{n}(t)=0.$

On the other hand denoted by $m=\min_{\bar{\Omega}}u>0$, it holds

$|I_{n}(t)|\leq\int_{\Omega}|\psi_{n}-\bar{\psi}|\bar{\psi}\phi_{N}\,dx\leq\frac{\gamma_{N}(\Omega)^{1/2}}{m}\|\psi_{n}\|_{L^{\infty}(\Omega\cap\text{supp}(\psi_{n}))}\|\nabla
u\|_{L^{2}(\Omega,\phi_{N})}+\frac{1}{m^{2}}\|\nabla
u\|^{2}_{L^{2}(\Omega,\phi_{N})}.$

Then for sufficiently large $n$ we have

(4.11) $\lim_{t\to 0^{+}}t^{2}I_{n}(t)=0$

Since, by (4.8), the function $t^{2}I_{n}(t)$ is strictly decreasing,
equations (4.10) and (4.11) give an absurd. This concludes the proof. ∎

##  5\. Proof of the main result

In this Section we prove the Faber-Krahn inequality stated in Theorem 1.1.

###### Proof of Theorem 1.1.

We first construct a suitable test function defined in $\Omega$ for (4.6). Let
$v$ be a positive eigenfunction to the problem (3.5) in $\Omega^{{}^{\\#}}$
and $w$ the solution to (3.1) such that $v(x)=w(x_{1})$. Then if we consider
the function

$\varphi^{*}(r)=-\displaystyle\frac{w^{\prime}(r)}{w(r)}\quad\text{with }r\in
I_{\sigma^{{}^{\\#}}}.$

By Theorem 4.1, we have

$\lambda_{1}(\Omega^{{}^{\\#}})=\mathcal{F}_{\Omega^{{}^{\\#}}}(S_{r},\varphi^{*})$

As before, let $u$ be the first positive eigenfunction of (2.5) in $\Omega$
such that $\|u\|_{L^{2}(\Omega,\phi_{N})}=1$. Using the same notation of the
previous Section, for any $t>0$ we consider $\mathcal{S}_{r(t)}$, the half-
space such that $\gamma_{N}(U_{t})=\gamma_{N}(S_{r(t)})$. Then, if
$x\in\Omega$ and $u(x)=t$, we define the following test function

$\psi(x):=\beta(r(t))=\psi^{*}(r(t)),$

where $\beta(r(t))$ is defined in (3.10).

We claim that the following inequality holds

(5.1)
$\mathcal{F}_{\Omega^{{}^{\\#}}}(S_{r(t)},\psi^{*})\leq\mathcal{F}_{\Omega}(U_{t},\psi)$

for every $t>0$. Being

$\mathcal{F}_{\Omega^{{}^{\\#}}}(S_{r(t)},\psi^{*})=\frac{1}{\gamma_{N}(S_{r(t)})}\left(-\int_{S_{r(t)}}\psi^{2}\phi_{N}dx+\int_{\\{x_{1}=\sigma^{{}^{\\#}}\\}}\psi\phi_{N}\,d\mathcal{H}^{N-1}(x)\right),$

in order to prove inequality (5.1), being
$\gamma_{N}(U_{t})=\gamma_{N}(S_{r(t)})$ we have to show that

(5.2)
$\displaystyle-\int_{S_{r(t)}}\psi^{2}\phi_{N}dx+\int_{\\{x_{1}=\sigma^{{}^{\\#}}\\}}\psi\phi_{N}\,d\mathcal{H}^{N-1}(x)\leq$
$\displaystyle-\int_{U_{t}}\psi^{2}\phi_{N}\,dx+\int_{\partial
U_{t}^{\text{int}}}\psi\phi_{N}\,d\mathcal{H}^{N-1}(x)+\beta\int_{\partial
U_{t}^{\text{ext}}}\phi_{N}\,d\mathcal{H}^{N-1}(x).$

Regarding the first integral in the left and right hand-sides in (5.2), by
construction we observe that the functions $\psi$ and $\psi^{*}$ are
equimeasurable, with respect to the Gaussian measure, and therefore

(5.3)
$\int_{U_{t}}\psi^{2}\phi_{N}dx=\int_{S_{r(t)}}(\psi^{*})^{2}\phi_{N}dx.$

Moreover, by the isoperimetric inequality (2.1), and being, by Lemma 3.1,
$\beta_{r(t)}\leq\beta$ for all $t>0$, we have that

(5.4)
$\int_{\\{x_{1}=r(t)\\}}\psi^{*}\phi_{N}\,d\mathcal{H}^{N-1}(x)=\beta_{r(t)}P_{\phi_{N}}(S_{r(t)})\leq\\\
\leq\beta_{r(t)}P_{\phi_{N}}(U_{t})\leq\int_{\partial
U_{t}^{\text{int}}}\psi\phi_{N}\,d\mathcal{H}^{N-1}(x)+\beta\int_{\partial
U_{t}^{\text{ext}}}\phi_{N}\,d\mathcal{H}^{N-1}(x).$

Hence, combining (5.3) and (5.4) we get (5.2) and then (5.1). Then, by
Theorems 4.1, 4.3 and equation (5.1) we get, for $t\in T\in]0,+\infty[$
defined in Theorem 4.3, it happens that

$\lambda_{1}(\Omega^{{}^{\\#}})=\mathcal{F}_{\Omega^{{}^{\\#}}}(S_{r(t)},\psi^{*})\
\leq\mathcal{F}_{\Omega}(U_{t},\psi)\leq\lambda_{1}(\Omega)$

which gives the claim.

In order to conclude the proof, we have to consider the equality case. Let us
suppose that $\lambda_{1}(\Omega)=\lambda_{1}(\Omega^{{}^{\\#}})$. Then all
the inequalities appearing in the first part of the proof become equalities.
In particular $P_{\phi_{N}}(S_{r(t)})=P_{\phi_{N}}(U_{t})$. By Theorem 2.1 we
have that $U_{t}$ are half-spaces for each $t$. Since the level sets of any
function are always nested, we have that $u$ depends on $x_{1}$ only and it is
a decreasing function. Therefore $\Omega=\Omega^{{}^{\\#}}$ and $u(x_{1})$
coincides with $w$ modulo a constant. ∎

## Acknowledgements

This work has been partially supported by the PRIN project 2017JPCAPN (Italy)
grant: “Qualitative and quantitative aspects of nonlinear PDEs”and by GNAMPA
of INdAM.

## References

  * [1] R. A. Adams.  General Logarithmic Inequalities and Orlicz imbeddings.  Journal of Functional Analysis, 34 (1979), 292–303.
  * [2] L. Ambrosio, G. Da Prato and D. Pallara.  BV functions in a Hilbert space with respect to a Gaussian measure.  Atti Accad. Naz. Lincei Rend. Lincei Mat. Appl., 21, no. 4, 405–414, 2010.
  * [3] B. Andrews, J. Clutterbuck and D. Hauer.  Non-concavity of the Robin ground state.  Camb. J. Math., 8, no. 2, 243–310, 2020.
  * [4] P. R. S. Antunes, P. Freitas and D. Krejčiřík.  Bounds and extremal domains for Robin eigenvalues with negative boundary parameter. Adv. Calc. Var., 10 (2017), 357–380.
  * [5] M. F. Betta, F. Chiacchio and A. Ferone.  Isoperimetric estimates for the first eigenfunction of a class of linear elliptic problems  Z. Angew. Math. Phys., 58, no. 1, 37–52, 2007.
  * [6] S. G. Bobkov, C. Houdrè.  Some connections between isoperimetric and Sobolev-type inequalities.  Mem. Am. Math. Soc., 129(616), 1997.
  * [7] V. I. Bogachev.  Gaussian measures.  Mathematical Surveys and Monographs, 62\. American Mathematical Society, Providence, RI, 1998.
  * [8] C. Borell.  The Brunn-Minkowski inequality in Gauss space.  Invent. Math., 30 , 207–216, 1975.
  * [9] M. H. Bossel.  Membranes ĺastiquement liés: extension du thórème de Rayleigh-Faber-Krahn et de l’inǵalité de Cheeger.  C. R. Acad. Sci. Paris Sér. I Math., 302(1):47–50, 1986.
  * [10] B. Brandolini, F. Chiacchio and C. Trombetti.  Hardy type inequalities and Gaussian measure.  Commun. Pure Appl. Anal. , 6(2), 411–428, 2007.
  * [11] B. Brandolini, F. Chiacchio, D. Krejčiřík and C. Trombetti.  The equality case in a Poincaré-Wirtinger type inequality.  Atti Accad. Naz. Lincei Rend. Lincei Mat. Appl. , 27, no. 4, 443–464, 2016.
  * [12] D. Bucur and D. Daners.  An alternative approach to the Faber-Krahn inequality for Robin problems.  Calc. Var., 37:75–86, 2010.
  * [13] F. W. Byron and R. W. Fuller.  Mathematics of classical and quantum physics. Dover Publications, Inc., New York, 1992.
  * [14] E. A. Carlen and C. Kerce.  On the cases of equality in Bobkov’s inequality and Gaussian rearrangement.  Calc. Var. Partial Differential, Equations 13, 1–18, 2001.
  * [15] F. Chiacchio and G. di Blasio.  Isoperimetric inequalities for the first Neumann eigenvalue in Gauss space.  Ann. Inst. H. Poincaré Anal. Non Linéaire , 29, no. 2, 199–216, 2012.
  * [16] A. Cianchi.  Moser-Trudinger trace inequalities.  Adv. Math., 217(5), 2005–2044, 2008.
  * [17] A. Cianchi, N. Fusco, F. Maggi, A. Pratelli.  On the isoperimetric deficit in Gauss space.  Amer. J. Math., 133, no. 1, 131–186, 2011.
  * [18] A. Cianchi, V. Musil, L. Pick.  Moser inequalities in Gauss space.  Math. Ann., 377, no. 3-4, 1265–1312, 2020.
  * [19] F. Chiacchio, N. Gavitone, C. Nitsch, C. Trombetti.  in preparation.
  * [20] D. Daners.  A Faber-Krahn inequality for Robin problems in any space dimension.  Mathematische Annalen, 335:767–785, June 2006.
  * [21] A. Ehrhard.  Inègalitès isopèrimètriques et intègrales de Dirichlet gaussiennes.  Ann. Sci. Ècole Norm. Sup., 17 no. 2, 317–332, 1984.
  * [22] F. Feo and M. R. Posteraro.  Logarithmic Sobolev trace inequalities.  Asian J. Math., Vol. 17 (2013), No. 3, pp. 569–582.
  * [23] V. Ferone, C. Nitsch and C. Trombetti.  On a conjectured reverse Faber-Krahn inequality for a Steklov-type Laplacian eigenvalue  Commun. Pure Appl. Anal., 14, 63–81, 2015.
  * [24] P. Freitas and D. Krejčiřík.  The first Robin eigenvalue with negative boundary parameter.  Adv. Math., 280, 322–339, 2015.
  * [25] E. Gagliardo.  Caratterizzazioni delle tracce sulla frontiera relative ad alcune classi di funzioni in n variabili.  Rendiconti del Seminario Matematico della Universita di Padova, 27, 284–305, 1957.
  * [26] L. Gross.  Logarithmic Sobolev inequalities.  Amer. J. Math., 97, 1061–1083, 1975.
  * [27] P. Harrington and A. Raich.  Sobolev spaces and elliptic theory on unbounded domains in $\mathbb{R}^{n}$.  Adv. Differential Equations, 19, no. 7-8, 635–692, (2014).
  * [28] G. V. Livshyts.  On a conjectural symmetric version of Ehrhard’s inequality. arXiv:2103.11433.
  * [29] E. Pelliccia, G. Talenti.  A proof of a logarithmic Sobolev inequality.  Calc. Var. Partial Differ. Equ., 1(3), 237–242, 1993.
  * [30] V. N. Sudakov and B.S. Cirel’son.  Extremal properties of half-spaces for spherically invariant measures (Russian). Problems in the Theory of Probability Distributions, II, Zap. Naučn. Sem. Leningrad. Otdel. Mat. Inst. Steklov. (LOMI) 41, 14–24, 165, 1974
  * [31] F. G. Tricomi.  Fonctions Hypergèomètriques Confluentes. Gauthier Villars, Paris 1960.


# iShape: A First Step Towards Irregular Shape Instance Segmentation

Lei Yang
<EMAIL_ADDRESS>
&Ziwei Yan
<EMAIL_ADDRESS>
&Yisheng He
<EMAIL_ADDRESS>&Wei Sun
<EMAIL_ADDRESS>
&Zhenhang Huang
<EMAIL_ADDRESS>&Haibin Huang
<EMAIL_ADDRESS>&Haoqiang Fan
<EMAIL_ADDRESS>
###### Abstract

In this paper, we introduce a brand new dataset to promote the study of
instance segmentation for objects with irregular shapes. Our key observation
is that though irregularly shaped objects widely exist in daily life and
industrial scenarios, they received little attention in the instance
segmentation field due to the lack of corresponding datasets. To fill this
gap, we propose iShape, an irregular shape dataset for instance segmentation.
iShape contains six sub-datasets with one real and five synthetics, each
represents a scene of a typical irregular shape. Unlike most existing instance
segmentation datasets of regular objects, iShape has many characteristics that
challenge existing instance segmentation algorithms, such as large overlaps
between bounding boxes of instances, extreme aspect ratios, and large numbers
of connected components per instance. We benchmark popular instance
segmentation methods on iShape and find their performance drop dramatically.
Hence, we propose an affinity-based instance segmentation algorithm, called
ASIS, as a stronger baseline. ASIS explicitly combines perception and
reasoning to solve Arbitrary Shape Instance Segmentation including irregular
objects. Experimental results show that ASIS outperforms the state-of-the-art
on iShape. Dataset and code are available at http://ishape.github.io

##  1 Introduction

(a) iShape-Wire

(b) Ground Truth

Figure 1: A typical scene of objects with irregular shape and similar
appearance. It has many characteristics that challenge instance segmentation
algorithms, including the large overlaps between bounding boxes of objects,
extreme aspect ratios (bounding box of the grey mask), and large numbers of
connected components in one instance (green and blue masks).

Instance segmentation aims to predict the semantic and instance labels of each
image pixel. Compared to object detection [1, 2, 3, 4, 5, 6, 7, 8] and
semantic segmentation [9, 10, 11], instance segmentation provides more fine-
grained information but is more challenging and attracts more and more
research interests of the community. Many methods [12, 13, 14, 15] and
datasets [16, 17, 18, 19] continue to emerge in this field. However, most of
them focus on regularly shaped objects and only a few [20, 18] study irregular
ones, which are thin, curved, or having complex boundary and can not be well-
represented by regularly rectangular boxes. A more clear definition of
irregular shape is “the area of the bounding box is much larger than the area
of instance mask or the aspect ratio of the bounding box is large”. We think
the insufficient exploration of this direction is caused by the lack of
corresponding datasets.

In this work, we present iShape, a new dataset designed for irregular Shape
instance segmentation. Our dataset consists of six sub-datasets, namely
iShape-Antenna, iShape-Branch, iShape-Fence, iShape-Log, iShape-Hanger, and
iShape-Wire. As shown in Figure 2, each sub-dataset represents scenes of a
typical irregular shape, for example, strip shape, hollow shape, and mesh
shape. iShape has many characteristics that reflect the difficulty of instance
segmentation for irregularly shaped objects. The most prominent one is the
large overlaps between bounding boxes of objects, which is hard for proposal-
based methods[12, 14] due to feature ambiguity and non-maximum suppression
(NMS [21]). Meanwhile, overlapped objects that share the same center point
challenge center-based methods[22, 23, 24]. Another characteristic of iShape
is a large number of objects with similar appearances, which makes embedding-
based methods[25, 26] hard to learn discriminative embedding. Besides, each
sub-dataset has some unique characteristics. For example, iShape-Fence has
about 53 connected components per instance, and iShape-Log has a large object
scale variation due to various camera locations and perspective
transformations. We hope that iShape can serve as a complement of existing
datasets to promote the study of instance segmentation for irregular shape as
well as arbitrary shape objects.

We also benchmark popular instance segmentation methods on iShape and find
their performance degrades significantly. To this end, we introduce a stronger
baseline considering irregular shape in this paper, which explicitly combines
perception and reasoning. Our key insight is to simulate how a person
identifies an irregular object. Taking the wire shown in Figure 1 for example,
one natural way is to start from a local point and gradually expand by
following the wire contour and figure out the entire object. The behavior of
such “following the contour” procedure is a process of continuous iterative
reasoning based on local clues, which is similar to the recent affinity-based
approaches [27, 28]. Under such observation, we propose a novel affinity-based
instance segmentation baseline, called ASIS, which includes principles of
generating effective and efficient affinity kernel based on dataset property
to solve Arbitrary Shape Instance Segmentation. Experimental results show that
the proposed baseline outperforms popular methods by a large margin on iShape.

Our contribution is summarized as follows:

  * •

We propose a brand new dataset, named iShape, which focuses on irregular shape
instance segmentation and has many characteristics that challenge existing
methods. In particular, we analyzed the advantages of iShape over other
instance segmentation datasets.

  * •

We benchmark popular instance segmentation methods on iShape to reveal the
drawbacks of popular methods on irregularly shaped objects.

  * •

Inspired by human’s behavior on instance segmentation, we propose ASIS as a
stronger baseline on iShape, which explicitly combines perception and
reasoning to solve Arbitrary Shape Instance Segmentation.

(a) Antenna

(b) Branch

(c) Fence

(d) Hanger

(e) Log

(f) Wire

Figure 2: The six sub-datasets in iShape.

##  2 Related Work

###  2.1 Existing Datasets

There are several benchmark datasets collected to promote the exploration of
instance segmentation. The COCO [16] and the Cityscapes [17] are the most
popular ones among them. However, the shapes of target objects in these
datasets are too regular. The connected components per instance (CCPI) and
average MaxIoU are low in the datasets and state-of-the-art algorithms
selected from them can not generalize to more challenging scenarios. Instead,
in the scenario of human detection and segmentation, the OC human [20] and the
Crowd Human [29] introduce datasets with larger MaxIoU. Nevertheless, the OC
human dataset only provides a small number of images for testing, and the
number of instances per image is too small to challenge instance segmentation
algorithms. While the crowd human dataset only provides annotations of object
bounding boxes, limiting their application to the instance segmentation field.
In the area of photogrammetry, the iSAID [18] dataset is proposed to lead
algorithms to tackle objects with multi scales. However, only a few instances
in this dataset are of irregular shape, most of which are rectangular, and the
lack of instance overlapping reduces its challenge to instance segmentation
algorithms as well. Under the observation that these existing regular datasets
are not enough to challenge algorithms for more general scenarios, we propose
iShape, which contains irregularly shaped objects with large overlaps between
bounding boxes of objects, extreme aspect ratios, and large numbers of CCPI to
promote the capabilities of instance segmentation algorithms.

###  2.2 Instance Segmentation Algorithms

Existing instance segmentation algorithms can be divided into two classes,
proposal-based and proposal-free.

Proposal-based approaches One line of these approaches [12, 14, 30] solve
instance segmentation within a two-stage manner, by first propose regions of
interests (RoIs) and then regress the semantic labels of pixels within them.
The drawback of these approaches comes from the loss of objects by NMS due to
large IoU. Instead, works like [15] tackle the problem within a single-stage
manner. For example, PolarMask [15] models the contours based on the polar
coordinate system and then obtain instance segmentation by center
classification and dense distance regression. But the convex hull setting
limits its accuracy.

Proposal-free approaches To shake off the rely on proposals and avoid the
drawback caused by them, many bottom-up approaches like [23, 24, 25, 26] are
introduced. These works are in various frameworks. The recent affinity-based
methods obtain instance segmentation via affinity derivation [27] and graph
partition[31]. This formulation is more similar to the perception and
reasoning procedure of human beings and can handle more challenging scenarios.
GMIS [27] utilizes both region proposals and pixel affinities to segment
images and SSAP [28] outputs the affinity pyramid and then performs cascaded
graph partition. However, the affinity kernels of GMIS and SSAP are sparse in
angle and distance, leading to missing components of some instances due to
loss of affinity connection. To this end, we propose ASIS which includes
principles of generating effective and efficient affinity kernel based on
dataset property to solve Arbitrary Shape Instance Segmentation and achieve
great improvement on iShape.

##  3 iShape Dataset

iShape consists of six sub-datasets. One of them, iShape-Antenna, is collected
from real scenes, which are used for antenna counting and grasping in
automatic production lines. The other five sub-datasets are synthetic datasets
that try to simulate five typical irregular shape instance segmentation
scenes.

###  3.1 Dataset Creation

iShape-Antenna Creation. For the creation of iShape-Antenna, we first prepare
a carton with a white cushion at the bottom, then randomly and elaborately
place antennas in it to generate various scenes. Above the box, there is a
camera with a light that points to the inside of the box to capture the scene
images. We collect 370 pictures and annotate 3,036 instance masks then split
them equally for training and testing. The labeling is done by our supplier.
We have checked all the annotations ourselves, and corrected the wrong
annotations. Although iShape-Antenna only contains 370 images, the number of
instances reaches 3,036 which is more than most categories in Cityscapes [17]
and PASCAL VOC [32].

Synthetic Sub-datasets Creation There are lots of typical irregular shape
instance segmentation scenes. Consequently, it is impractical to collect a
natural dataset for each typical scene. Since it is traditional to study
computer vision problems using synthetic data [33, 34, 35], we synthesize five
sub-datasets of iShape which include iShape-Branch, iShape-Fence, iShape-Log,
iShape-Hanger, and iShape-Wire, by using CG software Blender. Besides, our
dataset focuses on the evaluation of instance segmentation methods on
irregularly shaped objects, for which the synthetic images can serve well. In
particular, we build corresponding 3D models and place them appropriately in
Blender with optional random background and lighting environment, optional
physic engine, and random camera position. The creation configs of synthesis
sub-datasets are listed below. After setting up the scene, we use a ray
tracing render engine to render the RGB image. Besides, We build and open
source a blender module, bpycv [36], to generate instance annotation. We
generate 2500 images for each sub-dataset, 2000 for training, 500 for testing.

Table 1: Comparison of statistics with different datasets. Dataset | Images | Instances | Instances | OoS | Average | Aspect | CCPI
---|---|---|---|---|---|---|---
/image | bbox | convex | MaxIoU | ratio
Cityscapes | 2,975 | 52,139 | 17.52 | 0.14 | 0.07 | 0.394 | 2.29 | 1.34
COCO | 123,287 | 895,795 | 7.26 | 0.15 | 0.09 | 0.210 | 2.59 | 1.41
CrowdHuman | 15,000 | 339,565 | 22.64 | - | - | - | - | -
OC Human | 4,731 | 8,110 | 1.71 | 0.25 | 0.20 | 0.424 | 2.28 | 3.11
iSAID | 2,806 | 655,451 | 233.58 | - | - | - | 2.40 | -
Antenna | 370 | 3,036 | 8.20 | 0.62 | 0.23 | 0.655 | 9.86 | 2.45
Branch | 2,500 | 26,046 | 10.14 | 0.62 | 0.52 | 0.750 | 2.47 | 10.88
Fence | 2,500 | 7,870 | 3.15 | 0.65 | 0.63 | 0.983 | 1.05 | 53.65
Hanger | 2,500 | 49,275 | 19,71 | 0.53 | 0.34 | 0.685 | 3.28 | 4.94
Log | 2,500 | 72,144 | 28.86 | 0.73 | 0.06 | 0.843 | 34.14 | 2.64
Wire | 2,500 | 17,469 | 6.99 | 0.74 | 0.60 | 0.795 | 3.32 | 4.76
iShape | 12,870 | 175,840 | 13.66 | 0.65 | 0.42 | 0.806 | 15.84 | 6.99

###  3.2 Dataset Characteristics

In this sub-section, we analyze the characteristics of iShape and compare it
with other instance segmentation datasets. Since each sub-dataset represents
irregularly shaped objects in different scenes, we present the statistical
results of each sub-dataset separately.

Dataset basic information. As summarized in Table 1, iShape contains 12,870
images with 175,840 instances. All images are 1024$\times$1024 pixels and
annotated with pixel-level ground truth instance masks. Since iShape focus on
evaluating the performance of algorithms on the irregular shape, each scene
consists of multiple instances of one class, which is also common cases in
industrial scenarios.

Instance count per image. A larger instance count is more challenging. Despite
iSAID getting the highest instance count per image, it is unfair for extremely
high-resolution images and normal-resolution images to be compared on the
indicator. Among iShape, the instance count per image of iShape-Log reaches
28.86 that significantly higher than other normal-resolution datasets.

The large overlap between objects. We introduce a new indicator, Overlap of
Sum (OoS), which aims to measure the degree of occlusion and crowding in a
scene, defined as follows:

$Overlap\;of\;Sum=\left\\{\begin{array}[]{cc}1-\frac{|\bigcup_{i=1}^{n}C_{i}|}{\sum_{i=1}^{n}|C_{i}|},&n>0\\\
0,&n=0\end{array}\right.$ (1)

where $C$ means bounding boxes(bbox) or convex hulls(convex) of all instances
in the image, $n$ means number of instances, $\bigcup$ means union operation,
and $|C_{i}|$ means to get the area of $C_{i}$. The statistics of average OoS
for bounding box and convex hull are presented in Table. 1. For bounding box
OoS, All iShape sub-datasets are higher than other datasets, which reflects
the large overlap characteristic of iShape. Thanks to the large-area hollow
structure, iShape-Fence gets the highest average convex hull OoS 0.63.
Moreover, The Average MaxIoU [20] of all images also reflects the large
overlap characteristic of iShape.

The similar appearance between object instances. Instances from the same
object class in iShape share similar appearance, which is challenging to
embedding-based algorithms. In particular, any two object instance in iShape-
Antenna, iShape-Fence and iShape-Hanger are indistinguishable according to
their appearance. They are generated from either industrial standard antennas
or copies of the same mesh models. Meanwhile, the appearance of objects in
iShape-Branch, iShape-Log, and iShape-Wire are slightly changeable to add some
variances, but appearances of different instances are still much more similar
than those from other existing datasets in Table 1.

Aspect ratio. Table 1 presents statistics on the average aspect ratio of the
object’s minimum bounding rectangle for each dataset. Among them, iShape-Log’s
aspect ratio reaches 34.14, which is more than 10 times of other regularly
shaped datasets. Such a gap is caused by two following reasons: Firstly, the
shape of logs has a large aspect ratio. Secondly, partially occluding logs
leads to a higher aspect ratio. iShape-Antenna also has a high aspect ratio,
9.86, which exceeds other regularly shaped datasets.

Connected Components Per Instance (CCPI). Larger CCPI poses a larger challenge
to instance segmentation algorithms. Due to the characteristics of irregular
shaped objects and the occlusion of scenes, the instance appearance under the
mesh shape tends to be divided into many pieces, leading to large CCPI of
iShape-Fence. As is shown in Table 1, the result on CCPI of iShape-Fence is
53.65, about 5 times higher than the second place. iShape-Branch, iShape-
Hanger, and iShape-Wire also have a large CCPI that exceeds other regularly
shaped datasets.

##  4 Baseline Approach

Figure 3: Overview of ASIS. In the training stage, the network learns to
predict the semantic segmentation as well as the affinity map. In the
inference stage, first, build graph operation transforms the predicted
affinity map into a sparse undirected graph by setting pixels as nodes and the
affinity between pixels as edges. Then the graph merge algorithm is applied to
the graph. The algorithm will cluster the pixels to yield class-agnostic
instance segmentation. Finally, the class assign module will add a category
with confidence to each instance using the result of semantic segmentation.

Inspired by how a person identifies a wire shown in Figure 1, We propose an
affinity-based instance segmentation baseline, called ASIS, to solve Arbitrary
Shape Instance Segmentation by explicitly combining perception and reasoning.
Besides, ASIS includes principles of generating effective and efficient
affinity kernel based on dataset property. In this section, an overview of the
pipeline is firstly described in Subsection 4.1, then design principles of the
ASIS affinity kernel are explained in Subsection 4.2.

###  4.1 Overview of ASIS

As shown in Figure 3, we firstly employ the PSPNet [11] as the backbone and
remove its last softmax activation function to extract features. The semantic
head, which combines a single convolution layer and a softmax activation
function, will input those features and output a $C\times H\times W$ semantic
segmentation probability map where $C$ means the total categories number. The
affinity head that consists of a single convolution layer and a sigmoid
activation function will output a $N\times H\times W$ affinity map, where $N$
is the neighbor number of affinity kernel. Affinity kernel [27] defines a set
of neighboring pixels that needs to generate affinity information. Examples of
affinity kernels can be found in Figure 4. Each channel of the affinity map
represents a probability of whether the neighbor pixel and the current one
belong to the same instance.

During the training stage, we apply the affinity kernel on the instance
segmentation ground truth to generate the affinity ground truth. Since
affinity ground truth is imbalanced, an OHEM [37] loss is calculated between
the predicted affinity map and the affinity ground truth to effectively
alleviate the problem. Affinities that connect segments of fragmented
instances are important but hard to learn. Thanks to the difficulty of
learning these affinities, the OHEM loss pays more attention to these
important affinities.

For the inference stage, we firstly take pixels as nodes and affinity map as
edges to build an undirected sparse graph. The undirected sparse graph in
Figure 3 shows an example of how a pixel node on the spoon should connect the
other pixel nodes. Then, we apply the graph merge algorithm [27] on the
undirected sparse graph. The algorithm will merge nodes that have a positive
affinity to each other into one supernode, by contrast, keep nodes independent
if their affinity is negative. Pixels that merged to the same supernodes are
regarded as belonging to the same instance. In this way, we obtain a class-
agnostic instance map. A class assign module [27] will take the class-agnostic
instance map and the semantic segmentation result as input, then assign a
class label with a confidence value to each instance.

(a) ASIS

(b) GMIS (c) SSAP

Figure 4: Illustration for affinity kernels. (a) ASIS affinity kernel could
connect these two segments with two neighbors (blue points). (b) GMIS affinity
kernel cannot reach the right segment. (c) Examples of failure case for SSAP
affinity kernel. For higher resolutions (top), $5\times 5$ affinity window
cannot reach the segment on the right. For lower resolutions (bottom), the
view of thin antennas are lost in the resized feature maps.

###  4.2 ASIS Affinity Kernel

Since instances could be divided into many segments, it is important to design
an appropriate affinity kernel to connect those segments that belong to the
same instance. As shown in Figure 4(b) and Figure 4(c), The yellow point is
the current pixel. Red points belong to different instances and blue points
belong to the same instance of the current pixel. The antenna that the current
pixel (yellow point) belongs to has two segments that need to be connected by
affinity neighbor. The previous affinity-based approaches [27, 28] don’t take
into account such problems and cause some failures. Hence, we propose
principles of generating effective and efficient affinity kernel based on
dataset property to solve Arbitrary Shape Instance Segmentation. Our affinity
kernel is shown in 4(a).

Affinity kernels of GMIS and SSAP are centered symmetric, unfortunately, that
will cause redundant outputs. For example, the affinity of pixel $(1,1)$ with
its right side pixel and the affinity of pixel $(1,2)$ with its left side
pixel both mean the probability of these two pixels belonging to one instance.
A detailed description of redundant affinity can be found in the appendix. To
reduce the network’s outputs, redundant affinity neighbors are discarded in
the ASIS affinity kernel. As shown in 4(a), affinity neighbors of ASIS are
distributed in an asymmetric semicircle structure. Besides, the area covered
by asymmetric semicircle affinity kernel is reduced by half, in other words,
the demand for receptive fields is reduced, which further reduces the
difficulty of CNN learning affinity.

Two main parameters determine the shape of the ASIS affinity kernel. Kernel
radius $r_{k}$ controls the radius of the kernel and determines how far the
farthest of two segments can be reached. Affinity neighbor gap $g$ represents
the distance between any two nearly affinity neighbors, thus, $g$ controls the
sparseness of the affinity neighbor. Since each dataset has its optimal
affinity kernel, we propose another algorithm that could adaptively generate
appropriate $r_{k}$ and $g$ based on the dataset property. Detailed
descriptions of these two algorithms can be found in the appendix.

##  5 Experiments

We choose representative instance segmentation methods in various paradigms
and benchmark them on iShape to reveal the drawbacks of those methods on
irregularly shaped objects. All the existing methods are trained and tested on
six iShape sub-datasets with their recommended setting. And we further study
the effect of our baseline method, ASIS.

Implementation of GMIS*. Since there is no open-source implementation of GMIS
[27] available, we reproduced GMIS as GMIS*. There are only two differences
between ASIS and GMIS*. One is the different affinity kernel, and another is
GMIS* does not apply OHEM loss.

Implementation Details. The input image resolution of our framework is 512
$\times$ 512\. The image data augmentation is flipped horizontally or
vertically with a probability of 0.5. We use the ResNet-50 [38] as our
backbone network and the weight is initialized with ImageNet [39] pretrained
model. All experiments are trained in 4 2080Ti GPUs and batch size is set to
8. The stochastic gradient descent (SGD) solver is adopted in 50K iterations.
The momentum is set to 0.9 and weight decay is set to 0.0005. The learning
rate is initially set to 0.01 and decreases linearly.

Table 2: Qualitative results on iShape. We report the mask mmAP of six sub-datasets and the average of mmAP. All methods use ResNet-50 as the backbone. “w/o” denotes “without”. For “62.93/0.05”, the small number on the right is the standard deviation of four experiments. Method | Antenna | Branch | Fence | Hanger | Log | Wire | Avg
---|---|---|---|---|---|---|---
SOLOv2 [22] | 6.6 | 27.5 | 0.0 | 28.8 | 22.2 | 0.0 | 14.07
PolarMask [15] | 0.0 | 0.0 | 0.0 | 0.0 | 18.6 | 0.0 | 3.10
SE [23] | 38.3 | 0.0 | 0.0 | 49.8 | 20.9 | 0.0 | 18.17
Mask RCNN [12] | 16.9 | 4.2 | 0.0 | 22.1 | 32.6 | 0.0 | 12.63
DETR [40] | 2.1 | 2.6 | 0.0 | 32.2 | 46.2 | 0.0 | 13.85
GMIS* [27] | 67.6 /0.4 | 14.9 /0.2 | 30.6 /0.1 | 24.8 /0.3 | 63.2 /0.9 | 46.1 /1.1 | 41.21 /0.23
ASIS w/o OHEM | 82.1 /0.9 | 17.6 /0.4 | 48.0 /0.3 | 40.5 /0.4 | 66.4 /0.4 | 66.5 /0.5 | 53.51 /0.12
ASIS(ours) |  88.5 /0.3 | 24.6 /0.4 |  60.4 /0.4 |  57.4 /0.2 |  69.4 /0.2 |  77.3 /0.4 |  62.93 /0.05

###  5.1 Experiment Results

We evaluate the proposed ASIS and other popular approaches on iShape. The
quantitative results are shown in Table 2 and some qualitative results are
reported in Figure 7.

As is shown in Table 2, the performance of Mask R-CNN [12] is far from
satisfactory on iShape. We think the drop in performance mainly comes from
three drawbacks of the design. Firstly, the feature maps suffer from ambiguity
when the IoU is large, which is a common characteristic of crowded scenes of
irregular shape objects. Also, Mask R-CNN depends on the proposals of RoI,
which may be abandoned by the NMS algorithm due to large IoU and lead to
missing of some target objects. Moreover, many thin objects can not be
segmented by Mask R-CNN because of its RoI pooling, which resizes the feature
maps and lost the view of thin objects. The recent proposed end-to-end object
detection approach, DETR [40], shake of the reliance of NMS and can better
deal with objects with large IoU and achieve better performance, as shown in
the table. However, DETR still suffers from the RoI pooling problems and
performs badly on thin objects, as shown in Figure 7.

We also report some qualitative results of SE [23] in Figure 7. As is shown in
the figure, one common failure case of SE is that when the length of irregular
objects is longer than a threshold, the object will be split into multi
instances, for example, the wire in Figure 7. We think that’s because SE will
regress a circle of the target instance and then calculate its IoU with the
mask for supervision. However, for long and thin irregular objects, the radius
of the center circle can not reach the length of the target object, leading to
a multi-split of a long instance. Also, instances that share the same center
may cause ambiguity to SE, such as hanger and fence in Figure 7. Moreover,
many centers of irregular objects lie outside the mask, making it hard to
match them to the objects themselves.

We evaluate SOLO v2 [22] on the proposed iShape and find that it failed to
segment instances that share the same center, for example, fences in Figure 7.
Also, since SOLO V2 depends on the center point as SE, it also suffers from
performance drop caused by object centers that lie outside the mask.

Figure 5: Two example false cases of ASIS on iShape-Antenna. (a) Two antennas
merged into one (blue and orange). (b) ASIS fails to connect the right parts
of an object (red and sky blue).

(a) ASIS

(b) GMIS

Figure 6: Activation maps of different affinity kernels. We calculate the loss
for affinities that connects to the yellow point and perform backpropagate,
then visualize the gradient on the input image to reveal the activated maps.

In Table 2, we report the performance of PolarMask [15] on our dataset. As is
shown in the table, PolarMask can not solve the instance segmentation of
irregular objects. That is because PolarMask can only represent a thirty-six-
side mask due to its limited number of rays. Hence, it can not handle objects
with hollow, for example, the fences. Also, they distinguish different
instances according to center regression, which, however, can not handle
instances that share the same center. We also find that PolarMask can only
tackle some cases of logs in iShape, which looks like circles on the side and
fit its convex hull mask setting.

Thanks to the perception and reasoning mechanism as well as the well-designed
affinity kernels of our ASIS, it obtained the best performance on iShape. In
Table 2, ASIS advances GMIS* [27] by 21.7%, advances other popular approaches
by 44.2%. However, there are still some drawbacks to the design of ASIS and
some failure cases caused by them. For example in Figure 6, two instances are
merged into one. We think that’s because the graph merge algorithm is a kind
of greedy algorithm, while the greedy algorithm makes optimal decisions
locally instead of looking for a global optimum. Hence, ASIS is not robust to
false-positive (FP) with high confidence. Also, ASIS fails to connect the two
parts of an object if they are far away from each other, for example, the
antenna on Figure 6. We think that’s because CNN is not good at learning long-
range affinity. Another failure case is the branch in Figure 7, ASIS results
on iShape-Branch include many parts that have not been merged. Those parts
lead to a large number of small FPs, which causes a serious performance drop.

Figure 7: Qualitative results of different instance segmentation approaches on
iShape. More results are shown in https://ishape.github.io or the appendix.

###  5.2 Ablation Study

Table 3: Ablation Study of GMIS and ASIS on iShape-Antenna. “SY” and “ASY” indicate a centrosymmetric or asymmetric affinity kernel respectively. $\surd$ denotes equipped with and $\circ$ not. Affinity Kernel | Neighbors | Affinity GT | OHEM | mAP
---|---|---|---|---
GMIS [27] | 56 (SY) | $\circ$ | $\circ$ | 67.6
$\circ$ | $\surd$ | 81.5
$\surd$ | - | 90.2
28 (ASY) | $\circ$ | $\circ$ | 77.7
ASIS(ours) | 53 (ASY) | $\circ$ | $\circ$ | 82.1
$\circ$ | $\surd$ | 88.5
$\surd$ | - | 98.5

Effect of ASIS kernel. In Table 2, ASIS kernel advances GMIS kernel by 12.3%
in iShape. We think that is because our well-designed affinity kernels based
on dataset property can better discover the connectivity of different parts of
an object. While the GMIS kernel suffers from its sparsity in distance and
angle, examples are shown in Figure 8(b) and Figure 9 shows the reason for its
failure through the affinity visualization. In Table 3, we also use ground
truth affinity map to explore the upper bound of ASIS, where a 98.5% mAP is
achieved, showing its great potential. Moreover, we find our non-
centrosymmetric design of affinity kernel outperform centrosymmetric one by
10% in the table. We think such a design cut off the output and calculation
redundancy and reduce the requirement of large receptive field from CNN,
simplifying representation learning. Inspired by Grad-CAM [41], we use the
gradient of the input image to visualize the model’s activation map. As shown
in Figure 6, ASIS pays more attention to the discriminative information such
as the instance outline, while GMIS focuses more on the area around the center
point.

Effect of OHEM. Table 2 and Table 3 show that OHEM boosts the performance of
ASIS and GMIS by a large margin. We think firstly that is because OHEM can
ease problems caused by the imbalance distribution of positive and negative
affinity. Besides, affinities that connect segments of fragmented instances
are important but hard to learn, which means the OHEM loss pays more attention
to these important affinities.

(a) ASIS

(b) GMIS

Figure 8: Results compared with GMIS kernel. As shown in (b), GMIS kernel fail
to connect segments that belong to one instance.

(a) ASIS

(b) GMIS

Figure 9: Affinity visualization. The yellow point is the kernel center. The
ASIS kernel has a more reasonable affinity distribution and can connect the
two separated parts, while GMIS fails to reach the right segment.

##  6 Conclusion

In this work, we introduce a new irregular shape instance segmentation dataset
(iShape). iShape has many characteristics that challenge existing instance
segmentation methods, such as large overlaps, extreme aspect ratios, and
similar appearance between objects. We evaluate popular algorithms on iShape
to establish the benchmark and analyze their drawbacks to reveal possible
improving directions. Meanwhile, we propose a stronger baseline, ASIS, to
better solve Arbitrary Shape Instance Segmentation. Thanks to the combination
of perception and reasoning as well as the well-designed affinity kernels,
ASIS outperforms the state-of-the-art methods on iShape. We believe that
iShape and ASIS can serve as a complement to existing datasets and methods to
promote the study of instance segmentation for irregular shape as well as
arbitrary shape objects.

## References

  * [1] Joseph Redmon and Ali Farhadi.  Yolo9000: Better, faster, stronger, 2016.
  * [2] Ross Girshick.  Fast r-cnn, 2015.
  * [3] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation, 2014.
  * [4] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks, 2016.
  * [5] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C. Berg.  Ssd: Single shot multibox detector.  Lecture Notes in Computer Science, page 21–37, 2016.
  * [6] Jifeng Dai, Yi Li, Kaiming He, and Jian Sun.  R-fcn: Object detection via region-based fully convolutional networks, 2016.
  * [7] Zhaowei Cai and Nuno Vasconcelos.  Cascade r-cnn: Delving into high quality object detection, 2017.
  * [8] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection, 2016.
  * [9] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation, 2015.
  * [10] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs, 2017.
  * [11] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network, 2017.
  * [12] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [13] Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee.  Yolact: Real-time instance segmentation, 2019.
  * [14] Zhaowei Cai and Nuno Vasconcelos.  Cascade r-cnn: High quality object detection and instance segmentation, 2019.
  * [15] Enze Xie, Peize Sun, Xiaoge Song, Wenhai Wang, Ding Liang, Chunhua Shen, and Ping Luo.  Polarmask: Single shot instance segmentation with polar representation, 2020.
  * [16] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [17] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding, 2016.
  * [18] Syed Waqas Zamir, Aditya Arora, Akshita Gupta, Salman Khan, Guolei Sun, Fahad Shahbaz Khan, Fan Zhu, Ling Shao, Gui-Song Xia, and Xiang Bai.  isaid: A large-scale dataset for instance segmentation in aerial images, 2019.
  * [19] Jiyang Qi, Yan Gao, Yao Hu, Xinggang Wang, Xiaoyu Liu, Xiang Bai, Serge Belongie, Alan Yuille, Philip Torr, and Song Bai.  Occluded video instance segmentation: Dataset and challenge.  2021\.
  * [20] Song-Hai Zhang, Ruilong Li, Xin Dong, Paul L. Rosin, Zixi Cai, Han Xi, Dingcheng Yang, Hao-Zhi Huang, and Shi-Min Hu.  Pose2seg: Detection free human instance segmentation, 2019.
  * [21] A. Neubeck and L. Van Gool.  Efficient non-maximum suppression.  In 18th International Conference on Pattern Recognition (ICPR’06), volume 3, pages 850–855, 2006.
  * [22] Xinlong Wang, Rufeng Zhang, Tao Kong, Lei Li, and Chunhua Shen.  Solov2: Dynamic and fast instance segmentation, 2020.
  * [23] Davy Neven, Bert De Brabandere, Marc Proesmans, and Luc Van Gool.  Instance segmentation by jointly optimizing spatial embeddings and clustering bandwidth.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8837–8845, 2019.
  * [24] Bowen Cheng, Maxwell D Collins, Yukun Zhu, Ting Liu, Thomas S Huang, Hartwig Adam, and Liang-Chieh Chen.  Panoptic-deeplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation.  In CVPR, 2020.
  * [25] Bert De Brabandere, Davy Neven, and Luc Van Gool.  Semantic instance segmentation with a discriminative loss function.  arXiv preprint arXiv:1708.02551, 2017.
  * [26] Shu Kong and Charless C Fowlkes.  Recurrent pixel embedding for instance grouping.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9018–9028, 2018.
  * [27] Yiding Liu, Siyu Yang, Bin Li, Wengang Zhou, Jizheng Xu, Houqiang Li, and Yan Lu.  Affinity derivation and graph merge for instance segmentation, 2018.
  * [28] Naiyu Gao, Yanhu Shan, Yupei Wang, Xin Zhao, Yinan Yu, Ming Yang, and Kaiqi Huang.  Ssap: Single-shot instance segmentation with affinity pyramid.  In Proceedings of the IEEE International Conference on Computer Vision, pages 642–651, 2019.
  * [29] Shuai Shao, Zijian Zhao, Boxun Li, Tete Xiao, Gang Yu, Xiangyu Zhang, and Jian Sun.  Crowdhuman: A benchmark for detecting human in a crowd, 2018.
  * [30] Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia.  Path aggregation network for instance segmentation, 2018.
  * [31] Jianbo Shi and Jitendra Malik.  Normalized cuts and image segmentation.  IEEE Transactions on pattern analysis and machine intelligence, 22(8):888–905, 2000.
  * [32] M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman.  The pascal visual object classes challenge: A retrospective.  International Journal of Computer Vision, 111(1):98–136, January 2015.
  * [33] Sergey I Nikolenko.  Synthetic data for deep learning.  arXiv preprint arXiv:1909.11512, 2019.
  * [34] Stephan R. Richter, Zeeshan Hayder, and Vladlen Koltun.  Playing for benchmarks.  In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 2232–2241, 2017.
  * [35] Yuan-Ting Hu, Hong-Shuo Chen, Kexin Hui, Jia-Bin Huang, and A. Schwing.  Sail-vos: Semantic amodal instance level video object segmentation – a synthetic dataset and baselines.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3100–3110, 2019.
  * [36] Lei Yang.  bpycv: Computer vision utils for blender.  https://github.com/DIYer22/bpycv.
  * [37] Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick.  Training region-based object detectors with online hard example mining, 2016.
  * [38] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition, 2015.
  * [39] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  Imagenet large scale visual recognition challenge, 2015.
  * [40] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers, 2020.
  * [41] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017.