# Graph Mover’s Distance: An Efficiently Computable Distance Measure for
Geometric Graphs

Sushovan Majhi School of Information, University of California, Berkeley, USA,
<EMAIL_ADDRESS>
###### Abstract

Many applications in pattern recognition represent patterns as a geometric
graph. The geometric graph distance (GGD) has recently been studied in [13] as
a meaningful measure of similarity between two geometric graphs. Since
computing the GGD is known to be $\mathcal{NP}$–hard, the distance measure
proves an impractical choice for applications. As a computationally tractable
alternative, we propose in this paper the Graph Mover’s Distance (GMD), which
has been formulated as an instance of the earth mover’s distance. The
computation of the GMD between two geometric graphs with at most $n$ vertices
takes only $O(n^{3})$-time. Alongside studying the metric properties of the
GMD, we investigate the stability of the GGD and GMD. The GMD also
demonstrates extremely promising empirical evidence at recognizing letter
drawings from the LETTER dataset [18].

##  1 Introduction

Graphs have been a widely accepted object for providing structural
representation of patterns involving relational properties. While hierarchical
patterns are commonly reduced to a string [7] or a tree representation [6],
non-hierarchical patterns generally require a graph representation. The
problem of pattern recognition in such a representation then requires
quantifying (dis-)similarity between a query graph and a model or prototype
graph. Defining a relevant distance measure for a class of graphs has been
studied for almost five decades now and has a myriad of applications including
chemical structure matching [21], fingerprint matching [16], face
identification [11], and symbol recognition [12].

Depending on the class of graphs of interest and the area of application,
several methods have been proposed. Graph isomorphisms [5] or subgraph
isomorphisms can be considered. These, however, cannot cope with (sometimes
minor) local and structural deformations of the two graphs. To address this
issue, several alternative distance measures have been studied. We
particularly mention _edit distance_ [20, 9] and _inexact matching distance_
[3]. Although these distance measures have been battle-proven for attributed
graphs (i.e., combinatorial graphs with finite label sets), the formulations
seem inadequate in providing meaningful similarity measures for geometric
graphs.

A geometric graph belongs to a special class of attributed graphs having an
embedding into a Euclidean space $\mathbb{R}^{d}$, where the vertex labels are
inferred from the Euclidean locations of the vertices and the edge labels are
the Euclidean lengths of the edges.

In the last decade, there has been a gain in practical applications involving
comparison of geometric graphs, such as road-network or map comparison [1],
detection of chemical structures using their spatial bonding geometry, etc. In
addition, large datasets like [18] are being curated by pattern recognition
and machine learning communities.

###  1.1 Related Work and Our Contribution

We are inspired by the recently developed geometric graph distance (GGD) in
[4, 13]. Although the GGD succeeds to be a relevant distance measure for
geometric graphs, its computation, unfortunately, is known to be
$\mathcal{NP}$-hard. Our motivation stems from applications that demand an
efficiently computable measure of similarity for geometric graphs. The
formulation of our graph mover’s distance is based on the theoretical
underpinning of the GGD. The GMD provides a meaningful yet computationally
efficient similarity measure between two geometric graphs.

In Section 2, we revisit the definition of the (GGD) to investigate its
stability under Hausdorff perturbation. Section 3 is devoted to the study of
the GMD. The GMD has been shown to render a _pseudo_ -metric on the class of
(ordered) geometric graphs. Finally, we apply the GMD to classify letter
drawings in Section 4. Our experiment involves matching each of $2250$ test
drawings, modeled as geometric graphs, to $15$ prototype letters from the
English alphabet. For the drawings with LOW distortion, the correct letter has
been found among the top $3$ matches at a rate of $98.93\%$, where the
benchmark accuracy is $99.6\%$ obtained using a $k$-nearest neighbor
classifier ($k$-NN) with the graph edit distance [3].

##  2 Geometric Graph Distance (GGD)

We first formally define a geometric graph. Throughout the paper, the
dimension of the ambient Euclidean space is denoted by $d\geq 1$. We also
assume that the cost coefficients $C_{V}$ and $C_{E}$ are positive constants.

######  Definition 2.1 (Geometric Graph).

A _geometric graph_ of $\mathbb{R}^{d}$ is a (finite) combinatorial graph
$G=(V^{G},E^{G})$ with vertex set $V^{G}\subset\mathbb{R}^{d}$, and the
Euclidean straight-line segments $\\{\overline{ab}\mid(a,b)\in E^{G}\\}$
intersect (possibly) at their endpoints.

We denote the set of all geometric graphs of $\mathbb{R}^{d}$ by
$\mathcal{G}(\mathbb{R}^{d})$. Two geometric graphs $G=(V^{G},E^{G})$ and
$H=(V^{H},E^{H})$ are said to be _equal_ , written $G=H$, if and only if
$V^{G}=V^{H}$ and $E^{G}=E^{H}$. We make no distinction between a geometric
graph $G=(V^{G},E^{G})$ and its _geometric realization_ as a subset of
$\mathbb{R}^{d}$; an edge $(u,v)\in E^{G}$ can be identified as the line-
segment $\overline{uv}$ in $\mathbb{R}^{d}$, and its length by the Euclidean
length ${\lvert\overline{uv}\rvert}$.

Following the style of [13], we first revisit the definition of GGD. The
definition uses the notion of an inexact matching. In order to denote a
deleted vertex and a deleted edge, we introduce the _dummy vertex_
$\epsilon_{V}$ and the _dummy edge_ $\epsilon_{E}$, respectively.

######  Definition 2.2 (Inexact Matching).

Let $G,H\in\mathcal{G}(\mathbb{R}^{d})$ be two geometric graphs. A relation
$\pi\subseteq(V^{G}\cup\\{\epsilon_{V}\\})\times(V^{H}\cup\\{\epsilon_{V}\\})$
is called an (inexact) matching if for any $u\in V^{G}$ (resp. $v\in V^{H}$)
there is exactly one $v\in V^{H}\cup\\{\epsilon_{V}\\}$ (resp. $u\in
V^{G}\cup\\{\epsilon_{V}\\}$) such that $(u,v)\in\pi$.

The set of all matchings between graphs $G,H$ is denoted by $\Pi(G,H)$.
Intuitively, a matching $\pi$ is a relation that covers the vertex sets
$V^{G},V^{H}$ exactly once. As a result, when restricted to $V^{G}$ (resp.
$V^{H}$), a matching $\pi$ can be expressed as a map $\pi:V^{G}\to
V^{H}\cup\\{\epsilon_{V}\\}$ (resp. $\pi^{-1}:V^{H}\to
V^{G}\cup\\{\epsilon_{V}\\}$). In other words, when $(u,v)\in\pi$ and
$u\neq\epsilon_{V}$ (resp. $v\neq\epsilon_{V}$), it is justified to write
$\pi(u)=v$ (resp. $\pi^{-1}(v)=u$). It is evident from the definition that the
induced map

$\pi:\\{u\in V^{G}\mid\pi(u)\neq\epsilon_{V}\\}\to\\{v\in
V^{H}\mid\pi^{-1}(v)\neq\epsilon_{V}\\}$

is a bijection. For edges $e=(u_{1},u_{2})\in E^{G}$ and $f=(v_{1},v_{2})\in
E^{H}$, we introduce the short-hand $\pi(e):=(\pi(u_{1}),\pi(u_{2}))$ and
$\pi^{-1}(f):=(\pi^{-1}(v_{1}),\pi^{-1}(v_{2}))$.

Another perspective of $\pi$ is to view it as a matching between portions of
$G$ and $H$, (possibly) after applying some edits on the two graphs. For
example, $\pi(u)=\epsilon_{V}$ (resp. $\pi^{-1}(v)=\epsilon_{V}$) encodes
deletion of the vertex $u$ from $G$ (resp. $v$ from $H$), whereas
$\pi(e)=\epsilon_{E}$ (resp. $\pi^{-1}(f)=\epsilon_{E}$) encodes deletion of
the edge $e$ from $G$ (resp. $f$ from $H$). Once the above deletion operations
have been performed on the graphs, the resulting subgraphs of $G$ and $H$
become isomorphic, which are finally matched by translating the remaining
vertices $u$ to $\pi(u)$. Now, the cost of the matching $\pi$ is defined as
the total cost for all of these operations:

######  Definition 2.3 (Cost of a Matching).

Let $G,H\in\mathcal{G}(\mathbb{R}^{d})$ be geometric graphs and
$\pi\in\Pi(G,H)$ an inexact matching. The cost of $\pi$, is
$\mathrm{Cost}(\pi)=$

$\begin{split}&\underbrace{\sum_{\begin{subarray}{c}u\in V^{G}\\\
\pi(u)\neq\epsilon_{V}\end{subarray}}C_{V}|u-\pi(u)|}_{\text{vertex
translations}}+\underbrace{\sum_{\begin{subarray}{c}e\in E^{G}\\\
\pi(e)\neq\epsilon_{E}\end{subarray}}C_{E}\big{|}|e|-|\pi(e)|\big{|}}_{\text{edge
translations}}+\quad\quad\underbrace{\sum_{\begin{subarray}{c}e\in E^{G}\\\
\pi(e)=\epsilon_{E}\end{subarray}}C_{E}|e|}_{\text{edge
deletions}}+\underbrace{\sum_{\begin{subarray}{c}f\in E^{H}\\\
\pi^{-1}(f)=\epsilon_{E}\end{subarray}}C_{E}|f|}_{\text{edge
deletions}}.\end{split}$ (1)

######  Definition 2.4 ($\mathrm{GGD}$).

For geometric graphs $G,H\in\mathcal{G}(\mathbb{R}^{d})$, their geometric
graph distance, $\mathrm{GGD}(G,H)$, is

$\mathrm{GGD}(G,H)\stackrel{{\scriptstyle\text{def}}}{{=}}\min_{\pi\in\Pi(G,H)}\mathrm{Cost}(\pi)\;.$

###  2.1 Stability of GGD

A distance measure is said to be _stable_ if it does not change much if the
inputs are _perturbed_ only slightly. Usually, the change is expected to be
bounded above by the amount of perturbation inflicted on the inputs. The
perturbation is measured under a suitable choice of metric. In the context of
geometric graphs, it is natural to wonder if the GGD is stable under the
Hausdorff distance between two graphs. To our disappointment, we can easily
see for the graphs shown in Fig. 1 that the GGD is positive, whereas the
Hausdorff distance between their realizations is zero. So, the Hausdorff
distance between the graphs can not bound their GGD from above.

$v_{1}$$v_{2}$$u_{1}$$u_{2}$$u_{3}$$H$$G$ Figure 1: The graphs $G$ (top) and
$H$ (bottom) are embedded in the real line; the distance between consecutive
ticks is $1$ unit. The Hausdorff distance between $G$ and $H$ is zero, however
$\mathrm{GGD}(G,H)=C_{V}+C_{E}$ is non-zero. The optimal matching is given by
$\pi(u_{1})=v_{1}$, $\pi(u_{2})=v_{2}$, and $\pi(u_{3})=\epsilon_{V}$.

One might think that the $\mathrm{GGD}$ is stable when the Hausdorff distance
only between the vertices is considered. However, the graphs in Fig. 2
indicate otherwise.

$0$$1$$2$$3$$0$$1$$2$$3$$u_{1}$$u_{3}$$u_{2}$$0$$1$$2$$3$$0$$1$$2$$3$$v_{3}$$v_{1}$$v_{2}$
Figure 2: For the graphs $G,H\in\mathcal{G}(\mathbb{R}^{2})$, the Hausdorff
distance between the vertex sets is zero, however $\mathrm{GGD}(G,H)=4C_{E}$
is non-zero. The optimal matching is given by $\pi(u_{1})=v_{1}$,
$\pi(u_{3})=v_{3}$, $\pi(u_{2})=\epsilon_{V}$, and
$\pi^{-1}(v_{2})=\epsilon_{V}$.

Under strong requirements, however, it is not difficult to prove the following
result on the stability of $\mathrm{GGD}$ under the Hausdorff distance.

######  Theorem 1 (Hausdorff Stability of GGD).

Let $G,H\in\mathcal{G}(\mathbb{R}^{d})$ be geometric graphs with a graph
isomorphism $\pi:V^{G}\to V^{H}$. If $\delta>0$ is such that
$|u-\pi(u)|\leq\delta$ for all $u\in V^{G}$, then

$\mathrm{GGD}(G,H)\leq C_{V}|V^{G}|\delta.$

###### Proof.

The given graph isomorphism $\pi$ is a bijective mapping between the vertices
of $G$ and $H$. So, $\pi\in\Pi(G,H)$, i.e., it defines an inexact matching.
Since $\pi$ is a graph isomorphism, it does not delete any vertex or edge.
More formally, for all $u\in V^{G}$ and $v\in V^{H}$, we have
$\pi(u)\neq\epsilon_{V}$ and $\pi^{-1}(v)\neq\epsilon_{V}$, respectively.
Also, for all $e\in E^{G}$ and $f\in E^{H}$, we have $\pi(e)\neq\epsilon_{E}$
and $\pi^{-1}(f)\neq\epsilon_{E}$, respectively. From (1), the cost

$\mathrm{Cost}(\pi)=\sum_{u\in V^{G}}C_{V}|u-\pi(u)|\leq C_{V}|V^{G}|\delta.$

So, $\mathrm{GGD}(G,H)\leq\mathrm{Cost}(\pi)\leq C_{V}|V^{G}|\delta$. ∎

##  3 Graph Mover’s Distance (GMD)

We define the _Graph Mover’s Distance_ for two ordered geometric graphs. A
geometric graph is called _ordered_ if its vertices are ordered or indexed. In
that case, we denote the vertex set as a (finite) sequence
$V^{G}=\\{u_{i}\\}_{i=1}^{m}$. Let us denote by
$\mathcal{G}^{O}(\mathbb{R}^{d})$ the set of all ordered geometric graphs of
$\mathbb{R}^{d}$. The formulation of the GMD uses the framework known as the
earth mover’s distance (EMD).

###  3.1 Earth Mover’s Distance (EMD)

The EMD is a well-studied distance measure between weighted point sets, with
many successful applications in a variety of domains; for example, see [8, 10,
17, 19]. The idea of the EMD was first conceived by Monge [14] in 1781, in the
context of transportation theory. The name “earth mover’s distance” was coined
only recently, and is well-justified due to the following analogy. The first
weighted point set can be thought of as piles of earth (dirt) lying on the
point sites, with the weight of a site indicating the amount of earth;
whereas, the other point set as pits of volumes given by the corresponding
weights. Given that the total amount of earth in the piles equals the total
volume of the pits, the EMD computes the least (cumulative) cost needed to
fill all the pits with earth. Here, a unit of cost corresponds to moving a
unit of earth by a unit of “ground distance” between the pile and the pit.

The EMD can be cast as a transportation problem on a bipartite graph, which
has several efficient implementations, e.g., the network simplex algorithm [2,
15]. Let the weighted point sets $P=\\{(p_{i},w_{p_{i}})\\}_{i=1}^{m}$ and
$Q=\\{(q_{j},w_{q_{j}})\\}_{j=1}^{n}$ be a set of suppliers and a set of
consumers, respectively. The weight $w_{p_{i}}$ denotes the total supply of
the supplier $p_{i}$, and $w_{q_{j}}$ the total demand of the consumer
$q_{j}$. The matrix $[d_{i,j}]$ is the matrix of ground distances, where
$d_{i,j}$ denotes the cost of transporting a unit of supply from $p_{i}$ to
$q_{j}$. We also assume the _feasibility condition_ that the total supply
equals the total demand:

$\sum_{i=1}^{m}w_{p_{i}}=\sum_{j=1}^{n}w_{q_{j}}\;.$ (2)

A _flow_ of supply is given by a matrix $[f_{i,j}]$ with $f_{i,j}$ denoting
the units of supply transported from $p_{i}$ to $q_{j}$. We want to find a
flow that minimizes the overall cost

$\sum_{i=1}^{m}\sum_{j=1}^{n}f_{i,j}d_{i,j}$

subject to:

$\displaystyle f_{i,j}\geq 0\text{ for any }i=1,\ldots,m\text{ and
}j=1,\dots,n$ (3) $\displaystyle\sum_{j=1}^{n}f_{i,j}=w_{i}\text{ for any
}i=1,\ldots,m$ (4) $\displaystyle\sum_{i=1}^{m}f_{i,j}=w_{j}\text{ for any
}j=1,\ldots,n,$ (5)

Constraint (3) ensures a flow of units from $P$ to $Q$, and not vice versa;
constraint (4) dictates that a supplier must send all its supply—not more or
less; constraint (5) guarantees that the demand of every consumer is exactly
fulfilled.

The _earth mover’s distance_ (EMD) is then defined by the cost of the optimal
flow. A solution always exists, provided condition (2) is satisfied. The
weights and the ground distances can be chosen to be any non-negative numbers.
However, we choose them appropriately in order to solve our graph matching
problem.

$1$$1$$1$$2$$u_{1}$$1$$u_{2}$$1$$u_{3}$$1$$u_{4}$$2$$v_{1}$$1$$v_{2}$$1$$v_{3}$$3$
Figure 3: The bipartite network used by the $\mathrm{GMD}$ is shown for two
ordered graphs $G,H$ with vertex sets $V^{G}=\\{u_{1},u_{2},u_{3}\\}$ and
$V^{H}=\\{v_{1},v_{2}\\}$, respectively. The dummy nodes $u_{4}$ for $G$ and
$v_{3}$ for $H$, respectively, have been shown in gray. Below each node, the
corresponding weights are shown. A particular flow has been depicted here. The
gray edges do not transport anything. A red edge has a non-zero flow with the
transported units shown on them.

###  3.2 Defining the GMD

Let $G,H\in\mathcal{G}^{O}(\mathbb{R}^{d})$ be two ordered geometric graphs of
$\mathbb{R}^{d}$ with $V^{G}=\\{u_{i}\\}_{i=1}^{m}$ and
$V^{H}=\\{v_{j}\\}_{j=1}^{n}$. For each $i=1,\ldots,m$, let $E^{G}_{i}$ denote
the (row) $m$–vector containing the lengths of (ordered) edges incident to the
vertex $u_{i}$ of $G$. More precisely, the

$k\text{th element of }E^{G}_{i}=\begin{cases}|e^{G}_{i,k}|,\text{ if
}e^{G}_{i,k}:=(u_{i},u_{k})\in E^{G}\\\ 0,\text{ otherwise.}\end{cases}$

Similarly, for each $j=1,\ldots,n$, we define $E^{H}_{j}$ to be the (row)
$n$–vector with the

$k\text{th element of }E^{H}_{j}=\begin{cases}|e^{H}_{j,k}|,\text{ if
}e^{H}_{j,k}:=(v_{j},v_{k})\in E^{H}\\\ 0,\text{ otherwise.}\end{cases}$

In order to formulate the desired instance of the EMD, we take the point sets
to be $P=\\{u_{i}\\}_{i=1}^{m+1}$ and $Q=\\{v_{j}\\}_{j=1}^{n+1}$. Here,
$u_{m+1}$ and $v_{n+1}$ have been taken to be a dummy supplier and dummy
consumer, respectively, to incorporate vertex deletion into our GMD framework.
The weights on the sites are defined as follows:

$w_{u_{i}}=1\text{ for }i=1\ldots,m\text{ and }w_{u_{m+1}}=n\;.$

And,

$w_{v_{j}}=1\text{ for }j=1\ldots,n\text{ and }w_{v_{m+1}}=m\;.$

We note that the feasibility condition (2) is satisfied: $m+n$ is the total
weight for both $P$ and $Q$. An instance of the transportation problem is
depicted in Fig. 3.

Finally, the ground distance from $u_{i}$ to $v_{j}$ is defined by:

$d_{i,j}=\begin{cases}C_{V}|u_{i}-v_{j}|+&C_{E}\|E^{G}_{i}D_{m\times
p}-E^{H}_{j}D_{n\times p}\|_{1},\\\ &\text{ if }1\leq i\leq m,1\leq j\leq n\\\
C_{E}\|E^{H}_{j}\|_{1},&\text{ if }i=m+1\text{ and }1\leq j\leq n\\\
C_{E}\|E^{G}_{i}\|_{1},&\text{ if }1\leq i\leq m\text{ and }j=n+1\\\ 0,&\text{
otherwise}.\end{cases}$

Here, $p=\min\\{m,n\\}$, the $1$–norm of a row vector is denoted by
$\|\cdot\|_{1}$, and $D$ denotes a diagonal matrix with the all diagonal
entries being $1$.

$0$$1$$2$$0$$1$$2$$u_{4}$$u_{1}$$u_{5}$$u_{2}$$u_{3}$$0$$1$$2$$0$$1$$2$$v_{3}$$v_{1}$$v_{5}$$v_{2}$$v_{4}$$G$$H$
Figure 4: For the geometric graph $G,H\in\mathcal{G}^{O}(\mathbb{R}^{2})$, the
GMD is zero. The optimal flow is given by the matching $\pi(u_{1})=v_{2}$,
$\pi(u_{2})=v_{1}$, $\pi(u_{3})=v_{4}$, $\pi(u_{4})=v_{3}$, and
$\pi(u_{5})=v_{5}$.

###  3.3 Metric Properties

We can see that the GMD induces a pseudo-metric on the space of ordered
geometric graphs $\mathcal{G}^{O}(\mathbb{R}^{d})$. Non-negativity, symmetry,
and triangle inequality follow from those of the cost matrix $[d_{i,j}]$
defined in the GMD.

In addition, we note that $G=H$ (as ordered graphs) implies that $d_{i,j}=0$
whenever $i=j$. The trivial flow, where each $u_{i}$ sends its full supply to
$v_{i}$, has a zero cost. So, $\mathrm{GMD}(G,H)=0$. The GMD does not,
however, satisfy the separability condition on
$\mathcal{G}^{O}(\mathbb{R}^{d})$.

For the graphs $G,H$ shown in Fig. 4, we have $\mathrm{GMD}(G,H)=0$. We note
that $G,H$ have the following adjacency length matrices $[E^{G}_{i}]_{i}$ and
$[E^{H}_{j}]_{j}$, respectively:

$\begin{bmatrix}0&0&0&2&\sqrt{2}\\\ 0&0&2&0&\sqrt{2}\\\ 0&2&0&0&0\\\
2&0&0&0&0\\\ \sqrt{2}&\sqrt{2}&0&0&0\end{bmatrix}\text{ and
}\begin{bmatrix}0&0&2&0&\sqrt{2}\\\ 0&0&0&2&\sqrt{2}\\\ 2&0&0&0&0\\\
0&2&0&0&0\\\ \sqrt{2}&\sqrt{2}&0&0&0\end{bmatrix}\;.$

It can be easily checked that the flow that transports a unit of supply from
$u_{1}\mapsto v_{2}$, $u_{2}\mapsto v_{1}$, $u_{3}\mapsto v_{4}$,
$u_{4}\mapsto v_{3}$, $u_{5}\mapsto v_{5}$, and five units from $u_{6}\mapsto
v_{6}$ has total cost zero. So, $\mathrm{GMD}(G,H)=0$. However, the graphs $G$
and $H$ are not the same geometric graph. The fact that $\mathrm{GGD}(G,H)\neq
0$ implies the GGD is not stable under the GMD.

One can easily find even simpler configurations for two distinct geometric
graphs with a zero GMD—if the graphs are allowed to have multiple connected
components.

We conclude this section by stating a stability result for the GMD under the
Hausdorff distance. We omit the proof, since it uses a similar argument
presented in Theorem 1.

######  Theorem 2 (Hausdorff Stability of GMD).

Let $G,H\in\mathcal{G}^{O}(\mathbb{R}^{d})$ be ordered geometric graphs with a
bijection $\pi:V^{G}\to V^{H}$ such that $e^{G}_{i,j}=e^{H}_{\pi(i),\pi(j)}$
for all $i,j$. If $\delta>0$ is such that $|u_{i}-\pi(u_{i})|\leq\delta$ for
all $u_{i}\in V^{G}$, then

$\mathrm{GMD}(G,H)\leq C_{V}|V^{G}|\delta.$

###  3.4 Computing the GMD

As pointed out earlier, the GMD can be computed as an instance of
transportation problem—using, for example, the network simplex algorithm. If
the graphs have at most $n$ vertices, computing the ground cost matrix
$[d_{i,j}]$ takes $O(n^{3})$-time. Since the bipartite network has $O(n)$
vertices and $O(n^{2})$ edges, the simplex algorithm runs with a time
complexity of $O(n^{3})$, with a pretty good constant. Overall, the time
complexity of the GMD is $O(n^{3})$.

##  4 Experimental Results

We have implemented the GMD in Python, using network simplex algorithm from
the networkx package. We ran a pattern retrieval experiment on letter drawings
from the IAM Graph Database [18]. The repository provides an extensive
collection of graphs, both geometric and labeled.

In particular, we performed our experiment on the LETTER database from the
repository. The graphs in the database represent distorted letter drawings.
The database considers only $15$ uppercase letters from the English alphabet:
A, E, F, H, I, K, L, M, N, T, V, W, X, Y, and Z. For each letter, a prototype
line drawing has been manually constructed. On the prototypes, distortions are
applied with three different level of strengths: LOW, MED, and HIGH, in order
to produce $2250$ letter graphs for each level. Each test letter drawing is a
graph with straight-line edges; each node is labeled with its two-dimensional
coordinates. Since some of the graphs in the dataset were not embedded, we had
to compute the intersections of the intersecting edges and label them as
nodes. The preprocessing guaranteed that all the considered graphs were
geometric; a prototype and a distorted graph are shown in Fig. 5.

$0$$1$$2$$3$$0$$1$$2$$3$$u_{3}$$u_{4}$$u_{1}$$u_{2}$$u_{5}$$0$$1$$2$$3$$0$$1$$2$$3$$v_{1}$$v_{2}$$v_{3}$$v_{4}$$v_{5}$$v_{6}$$v_{7}$
Figure 5: The prototype geometric graph of the letter A is shown on the left.
On the right, a (MED) distorted letter A is shown.

We devised a classifier for these letter drawings using the GMD. For this
application, we chose $C_{V}=4.5$ and $C_{E}=1$. For a test letter, we
computed its GMD from the $15$ prototypes, then sorted the prototypes in an
increasing order of their distance to the test graph. We then check if the
letter generating the test graph is among the first $k$ prototypes. For each
level of distortion and various values of $k$, we present the rate at which
the correct letter has been found in the first $k$ models. The summary of the
empirical results have been shown in Table 1. Although the graph edit distance
based $k$-NN classifier still outperforms the GMD by a very small margin, our
results has been extremely satisfactory.

| correct letter in first $k$ models ($\%$)
---|---
Distortion |  $k=1$ |  $k=3$ |  $k=5$
LOW |  $96.66\%$ |  $98.93\%$ |  $99.37\%$
MED |  $66.66\%$ |  $85.37\%$ |  $91.15\%$
HIGH |  $73.73\%$ |  $90.48\%$ |  $95.51\%$
Table 1: Empirical result on the LETTER dataset

One possible reason why the GMD might fail to correctly classify some of the
graphs is that lacks the separability property as a metric.

##  5 Discussions

We have successfully introduced an efficiently computable and meaningful
similarity measure for geometric graphs. However, the GMD lacks some of the
desirable properties, like separability and stability. The currently presented
stability results for the GGD and GMD have a factor that depends on the size
of the input graphs. The question remains if the distance measures are in fact
stable under much weaker conditions, possibly with constant factors on the
right side. It will also be interesting to study the exact class of geometric
graphs for which the GMD is, in fact, a metric.

## References

  * [1] M. Ahmed, S. Karagiorgou, D. Pfoser, and C. Wenk.  Map Construction Algorithms.  Springer International Publishing, first edition, 2015.
  * [2] R. Ahuja, T. Magnanti, and J. Orlin.  Network Flows: Theory, Algorithms, and Applications.  Always learning. Pearson, 2013.
  * [3] H. Bunke and G. Allermann.  Inexact graph matching for structural pattern recognition.  Pattern Recognition Letters, 1(4):245–253, May 1983.
  * [4] O. Cheong, J. Gudmundsson, H.-S. Kim, D. Schymura, and F. Stehn.  Measuring the Similarity of Geometric Graphs.  In J. Vahrenhold, editor, Experimental Algorithms, volume 5526, pages 101–112. Springer, 2009.
  * [5] D. G. Corneil and C. C. Gotlieb.  An efficient algorithm for graph isomorphism.  J. ACM, 17(1):51–64, 1970.
  * [6] K.-S. Fu and B. Bhargava.  Tree systems for syntactic pattern recognition.  IEEE Transactions on Computers, C-22(12):1087–1099, 1973.
  * [7] K.-S. Fu and P. Swain.  On syntactic pattern recognition.  In J. T. Tou, editor, Computer and Information Sciences – 1969, volume 2 of SEN Report Series Software Engineering, pages 155–182. Elsevier, 1971.
  * [8] C. J. Hargreaves, M. S. Dyer, M. W. Gaultois, V. A. Kurlin, and M. J. Rosseinsky.  The Earth Mover’s Distance as a Metric for the Space of Inorganic Compositions.  Chemistry of Materials, 32(24):10610–10620, Dec. 2020.
  * [9] D. Justice and A. Hero.  A binary linear programming formulation of the graph edit distance.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(8):1200–1214, Aug. 2006.
  * [10] M. Kusner, Y. Sun, N. Kolkin, and K. Weinberger.  From Word Embeddings To Document Distances.  In Proceedings of the 32nd International Conference on Machine Learning, pages 957–966. PMLR, June 2015.  ISSN: 1938-7228.
  * [11] J. Liu and Y. T. Lee.  Graph-based method for face identification from a single 2d line drawing.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(10):1106–1119, 2001.
  * [12] J. Llados, E. Marti, and J. Villanueva.  Symbol recognition by error-tolerant subgraph matching between region adjacency graphs.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(10):1137–1143, 2001.
  * [13] S. Majhi and C. Wenk.  Distance measures for geometric graphs.  arXiv preprint arXiv:2209.12869, 2022.
  * [14] G. Monge.  Mémoire sur la théorie des déblais et des remblais.  Imprimerie royale, 1781.
  * [15] O. Pele and M. Werman.  A Linear Time Histogram Metric for Improved SIFT Matching.  In D. Forsyth, P. Torr, and A. Zisserman, editors, Computer Vision – ECCV 2008, Lecture Notes in Computer Science, pages 495–508, Berlin, Heidelberg, 2008. Springer.
  * [16] J. W. Raymond and P. Willett.  Effectiveness of graph-based and fingerprint-based similarity measures for virtual screening of 2D chemical structure databases.  Journal of Computer-Aided Molecular Design, 16(1):59–71, 2002.
  * [17] Z. Ren, J. Yuan, and Z. Zhang.  Robust hand gesture recognition based on finger-earth mover’s distance with a commodity depth camera.  In Proceedings of the 19th ACM international conference on Multimedia, MM ’11, pages 1093–1096, New York, NY, USA, Nov. 2011. Association for Computing Machinery.
  * [18] K. Riesen and H. Bunke.  IAM Graph Database Repository for Graph Based Pattern Recognition and Machine Learning.  In Structural, Syntactic, and Statistical Pattern Recognition, volume 5342, pages 287–297. Springer, 2008.
  * [19] Y. Rubner, C. Tomasi, and L. J. Guibas.  The Earth Mover’s Distance as a Metric for Image Retrieval.  International Journal of Computer Vision, 40(2):99–121, Nov. 2000\.
  * [20] A. Sanfeliu and K.-S. Fu.  A distance measure between attributed relational graphs for pattern recognition.  IEEE Transactions on Systems, Man, and Cybernetics, SMC-13(3):353–362, May 1983.
  * [21] P. Willett.  Similarity Searching in Databases of Three-Dimensional Chemical Structures.  In H.-H. Bock, W. Lenski, and M. M. Richter, editors, Information Systems and Data Analysis, pages 280–293. Springer, 1994.


# An Image is Worth Multiple Words:
Learning Object Level Concepts using
Multi-Concept Prompt Learning

Chen Jin1, Ryutaro Tanno2, Amrutha Saseendran1, Tom Diethe1, Philip Teare1
1Centre for AI, DS&AI, AstraZeneca, UK, 2Google DeepMind, UK
{chen.jin, amrutha.saseendran, tom.diethe, philip.teare}
@astrazeneca.com<EMAIL_ADDRESS>
###### Abstract

Textural Inversion, a prompt learning method, learns a singular embedding for
a new “word” to represent image style and appearance, allowing it to be
integrated into natural language sentences to generate novel synthesised
images. However, identifying and integrating multiple object-level concepts
within one scene poses significant challenges even when embeddings for
individual concepts are attainable. This is further confirmed by our empirical
tests. To address this challenge, we introduce a framework for Multi-Concept
Prompt Learning (MCPL), where multiple new “words” are simultaneously learned
from a single sentence-image pair. To enhance the accuracy of word-concept
correlation, we propose three regularisation techniques: Attention Masking
(AttnMask) to concentrate learning on relevant areas; Prompts Contrastive Loss
(PromptCL) to separate the embeddings of different concepts; and Bind
adjective (Bind adj.) to associate new “words” with known words. We evaluate
via image generation, editing, and attention visualisation with diverse
images. Extensive quantitative comparisons demonstrate that our method can
learn more semantically disentangled concepts with enhanced word-concept
correlation. Additionally, we introduce a novel dataset and evaluation
protocol tailored for this new task of learning object-level concepts.

Figure 1: Multi-concepts learning and composition with previous vs. our
approach. Textural Inversion (left) can only learn a single concept from each
image and fails at composing multiple ones. In contrast, our method (right)
can learn, compose, and edit multiple concepts simultaneously. The learning
input consists of image(s) accompanied by descriptive sentences with learnable
prompts, represented as coloured pseudo words. The average cross-attentions
and the corresponding mask of the learned prompts denote a disentangled and
precise prompt-concept correlation.

##  1 Introduction

In nurseries, toddlers are shown pictures to learn new things. Teachers talk
about each picture using sentences with new ideas, like sentences with
unfamiliar words. In the Figure 1 right (ours) example, the describing
sentence for the images is: “a photo of brown * on a rolling & at time
square”. Here, “* (teddy bear)” and “& (skateboard)” are the unfamiliar
concepts to be learned. This way of learning with simple hints is more
economical and preferred over the current method of teaching machines using
detailed contours and masks.

Recent research (Gal et al. (2022); Ruiz et al. (2022)) shows that the
appearance and style of an image can be encapsulated as a cohesive concept via
a learned prompt (“word”). The textural embedding of this new prompt is
optimised in the frozen embedding space of a pre-trained text-to-image
diffusion model to reconstruct several example input images. The concept
conveyed by the learned prompt can then be composed into natural language
sentences to generate or edit various novel scenes. Despite the significant
interest in object-level image editing, (Wu et al., 2020; Meng et al., 2021;
Hertz et al., 2022), Gal et al. (2022) points out that recent prompt learning
methods struggle with learning and composing multiple prompts within multi-
object scenes (Figure 1 left).

In this work, we start with a motivational study to investigate the
capabilities and limitations of existing prompt learning methods in multi-
concept settings. Our findings confirm that while applying careful sampling
such as manual masking or cropping yields distinct embeddings, object-level
learning and editing without manual intervention remains challenging.
Motivated by this finding, we introduce Multi-Concept Prompt Learning (MCPL)
framework Figure 2 (top) for simultaneous learning of multiple prompts from
one scene.

However, without further assumptions on the embedding relationships, jointly
learning multiple prompts is problematic. The model may disregard the semantic
associations and instead prioritise optimising multiple embedding vectors for
optimal image-level reconstruction. To enhance the accuracy of prompt-object
level correlation, we propose the following regularisation techniques: 1) To
ensure a concentrated correlation between each prompt-concept pair, we propose
Attention Masking (AttnMask), restricting prompt learning to relevant regions
defined by a cross-attention-guided mask. 2) Recognising that multiple objects
within a scene are semantically distinct, we introduce Prompts Contrastive
Loss (PromptCL) to facilitate the disentanglement of prompt embeddings
associated with multiple concepts. 3) To further enable accurate control of
each learned embedding, we bind each learnable prompt with a related
descriptive adjective word, referred to as Bind adj., that we empirically
observe has a strong regional correlation. The middle and bottom row of Figure
2 illustrates the proposed regularisation techniques.

In this work we implement our proposed method based on Textural Inversion by
Gal et al. (2022), but the method can be adapted to other prompt learning
methods such as Dreambooth by Ruiz et al. (2022). To our knowledge, our
technique is the first to address the novel and challenging problem of
learning and composing multiple concepts within multi-object scenes. To
evaluate this task, we assembled datasets of multi-concept images featuring a
total of 16 categories of object-level concepts. These datasets include both
natural images, familiar to the pre-trained model, and out-of-distribution
biomedical images, each equipped with object-level masks. We evaluate and
demonstrate that our framework enables enhanced precision in object-level
concept learning, synthesis, editing, quantification, and understanding of
relationships between multiple objects, as exemplified in Figure 1 (right) and
further illustrated in Figure 9. Through extensive quantitative analysis of
approximately 4000 learned object-level embeddings, using both t-SNE and four
robust, pre-trained text/image embedding spaces, we validate that our method
excels in discerning semantically distinct object-level concepts, ensuring
enhanced prompt-to-concept correlation.

Figure 2:  Method overview. MCPL takes a sentence (top-left) and a sample
image (top-right) as input, feeding them into a pre-trained text-guided
diffusion model comprising a text encoder $c_{\phi}$ and a denoising network
$\epsilon_{\theta}$. The string’s multiple prompts are encoded into a sequence
of embeddings which guide the network to generate images $\tilde{X}_{0}$ close
to the target one $X_{0}$. MCPL focuses on learning multiple learnable prompts
(coloured texts), updating only the embeddings $\\{v^{*}\\}$ and
$\\{v^{\&}\\}$ of the learnable prompts while keeping $c_{\phi}$ and
$\epsilon_{\theta}$ frozen. We introduce Prompts Contrastive Loss (PromptCL)
to help separate multiple concepts within learnable embeddings. We also apply
Attention Masking (AttnMask), using masks based on the average cross-attention
of prompts, to refine prompt learning on images. Optionally we associate each
learnable prompt with an adjective (e.g., “brown” and “rolling”) to improve
control over each learned concept, referred to as Bind adj.

##  2 Related Works

#### Prompt learning for image concept inversion.

Prompt tuning, first proposed by Lester et al. (2021), has been utilised to
expedite the tuning of large language models for downstream tasks. Jia et al.
(2022); Zhou et al. (2022) further extended this approach to vision-language
models such as CLIP (Radford et al. (2021)). In the context of text-guided
image synthesising, prompt learning would enable connecting the appearance and
style of an unseen image to a learnable prompt and transferring to newly
generated images, as demonstrated by Textural Inversion Gal et al. (2022) and
DreamBooth Ruiz et al. (2022). To better composite multiple concepts Kumari et
al. (2023) proposed to fine-tune a subset of cross-attention layers. However,
this approach learns multiple concepts separately from carefully sampled
images rather than from the same scene.

#### Mask and text-driven local image editing.

In the context of diffusion mode, Meng et al. (2021) first proposed SDEdit for
mask-guided image-to-image style translation. Lugmayr et al. (2022) developed
RePaint to enable mask-guided local image editing. Avrahami et al. (2022)
further conditioned local editing with text condition. These methods use
manual masks prior to guide local image editing. A set of recent works showed
that text-guided local object-level editing can be achieved without using a
mask prior but instead the attention-derived masks (Hertz et al. (2022);
Tumanyan et al. (2023); Patashnik et al. (2023)). The success of these
approaches heavily relies on the accurate text-concept semantic correlation in
the pre-trained model and is limited to in-distribution concepts.

#### Disentangled per-concept image editing.

Interpretable and disentangled per-concept image manipulation has garnered
significant interest in the literature on Generative Adversarial Networks
(GANs). Traditional approaches often focus on layer-wise or channel-wise
control within a pre-trained generator network. The goal is to identify and
modify a subset of parameters responsible for specific concepts (Brock et al.,
2018; Karras et al., 2020; Wu et al., 2020). Although our work is not centred
on GAN-based approaches, we emphasise that we directly optimise multiple
embeddings rather than network parameters. This methodology has been shown to
better adapt to unseen and novel concepts by Gal et al. (2022).

##  3 Methods

In this section, we outline the preliminaries in Section 3.1 and present a
motivational study in Section 3.2. These tests investigate the challenges of
applying existing image-level prompt learning methods in identifying and
integrating multiple object-level concepts within one scene. Inspired by these
results, we introduce the Multi-Concept Prompt Learning (MCPL) framework for
simultaneous learning of multiple prompts from one scene. To address the
complexity of optimising multiple object-level prompts in tandem with a single
image-level reconstruction goal, we propose several regularisation techniques
in Section 3.4. The code will be released here upon publication.

###  3.1 Preliminaries: prompt learning in text-to-image diffusion model

Text-guided diffusion models are probabilistic generative models trained to
approximate the training data distribution through a process of incremental
denoising from Gaussian random noise, conditioned on text embeddings.
Specifically, a denoising network ${\epsilon}_{\theta}$ is trained to map an
initial noise map ${\bm{\epsilon}}\sim\mathcal{N}(\mathbf{0},\textbf{I})$ and
conditional textual embedding $v=c_{\phi}(p)$ to generate images $\tilde{x}$
close to the target one $x$. Here $c_{\phi}$ is the text encoder and $p$ is
the text prompt. To enable sequential denoising, $c_{\phi}$ and
$\epsilon_{\theta}$ are jointly optimised to minimize the loss:

$L_{DM}=L_{DM}(x,\tilde{x}):=E_{x_{0},{\epsilon}\sim
N(0,I),t\sim\text{Uniform}(1,T)}\|{\epsilon}-{\epsilon}_{\theta}(x_{t},t,c_{\phi}(p))\|^{2},$
(1)

where $x_{t}$ is obtained by adding noise to the initial image $x_{0}$ at a
given time step $t$ in the set $T$. Intuitively, the objective here is to
correctly remove the noise added to an image conditioned to a given text
prompt. During inference, the pre-trained model iteratively eliminates noise
from a new random noise map to generate a fresh image. Our work builds on
Latent Diffusion Models (LDMs) (Rombach et al., 2022), which encode images $x$
with an encoder $\mathcal{E}$ to get latent representation $z=\mathcal{E}(x)$,
prior to the diffusion process and decode after generation to conserve
computation.

The prompt learning method by (Gal et al. (2022)) is aimed at identifying the
text embedding $v^{*}$ for a new prompt $p^{*}$ in a pre-trained text-guided
diffusion model. Given a few (3-5) example images representing a specific
subject or concept, the method optimises $v^{*}$ in the frozen latent space of
a pre-trained text encoder $c_{\phi}$. The objective is to generate an image
via the denoising network ${\epsilon}_{\theta}$ that closely resembles the
example images when conditioned on $v^{*}$. The optimisation is guided by the
diffusion model loss defined in equation 1, updating only $v^{*}$ while
keeping $c_{\phi}$ and $\epsilon_{\theta}$ frozen. The Textural Inversion is
also trained with random sentences to generalise the learning, refer to
Appendix A.6 for the full detailed algorithm.

Cross-attention layers play a pivotal role in directing the text-guided
diffusion process. Within the denoising network, ${\epsilon}_{\theta}$, the
textual embedding, $v=c_{\phi}(p)$, interacts with the image embedding,
$z=\mathcal{E}(x)$, via the cross-attention layer. Here, $Q=f_{Q}(z)$,
$K=f_{K}(v)$, and $V=f_{V}(v)$ are acquired using learned linear layers
$f_{Q},f_{K},f_{V}$. As Hertz et al. (2022) highlighted, the per-prompt cross-
attention maps, $M=\text{Softmax}(QK^{T}/\sqrt{d})$, correlate to the
similarity between $Q$ and $K$. Therefore the average of the cross-attention
maps over all time steps reflects the crucial regions corresponding to each
prompt word, as depicted in Figure 2. In this study, the per-prompt attention
map serves as one of the primary matrices to evaluate the correlation between
prompt and concept. Our results will demonstrate that without proper
constraints, the attention maps of newly learned prompts are not consistently
disentangled and may lack accurate prompt-concept correlation.

###  3.2 Motivational study: is image-level prompt learning sufficient for
object-level multi-concept learning?

#### Do multiple distinct embeddings arise from the same image?

To understand the challenges in learning and composing multiple concepts, we
explored whether Textural Inversion can discern semantically distinct concepts
from processed images, each highlighting a single concept. Following Wu et al.
(2020), we used images with manual masks to isolate concepts, as seen in
Figure 3 (left). We applied Textural Inversion to these images to learn
embeddings for the unmasked or masked images. Our findings indicate that when
focusing on isolated concepts, Textural Inversion can successfully learn
distinct embeddings, as validated by the generated representations of each
concept.

#### Is separate learning of concepts sufficient for multi-object image
generation?

While human-guided, separate learning of each concept in a multi-object scene
deviates from our objective, it is valuable to evaluate its effectiveness.
Specifically, we use Textural Inversion to separately learn concepts like
“ball” and “box” from carefully cropped images, as shown in Figure 3 (second
column). We then attempt to compose images using strings that combine these
concepts, such as ”a photo of a green ball on orange box.” Our results
indicate that the accurate composition of multi-object images remains
challenging, even when individual concepts are well-learned.

Figure 3:  Motivational Study with “Watch Face-Band” and “Ball-Box” Images.
Left: Embeddings are learned using Textural Inversion on both multi-concept
(unmasked) and single-concept (masked) images. Right: Concepts of “ball” and
“box” are learned and composed using different methods: Textural Inversion
(T.I.), which crops and learns each concept separately; MCPL-one, learning
both concepts jointly from uncropped examples with a single string; and MCPL-
diverse accounting for per-image specific relationships. Refer to Appendix
Section A.4 for more results.

###  3.3 Multi-Concept Prompt Learning (MCPL)

Our motivational study confirm that: 1) multiple unique embeddings can be
derived from a single multi-concept image, albeit with human intervention, and
2) despite having well-learned individual concepts, synthesizing them into a
unified multi-concept scene remains challenging. To address these issues, we
introduce the Multi-Concept Prompt Learning (MCPL) framework. MCPL modifies
Textural Inversion to enable simultaneous learning of multiple prompts within
the same string. In specific, MCPL learn a list of multiple embeddings
$\mathcal{V}=[v^{*},\ldots,v^{\&}]$ corresponds to multiple new prompts
$\mathcal{P}=[p^{*},\ldots,{p}^{\&}]$. The optimisation is still guided by the
image-level $L_{DM}$, but now updating $\\{v^{*},\ldots,v^{\&}\\}$ while
keeping $c_{\phi}$ and $\epsilon_{\theta}$ frozen. The MCPL algorithm is
outlined in Appendix A.6, Algorithm 2. Recognising the complexity of learning
multiple embeddings with a single image-generation goal, we propose three
training strategies: 1) MCPL-all, a naive approach that learns embeddings for
all prompts in the string (including adjectives, prepositions and nouns.
etc.); 2) MCPL-one, which simplifies the objective by learning single prompt
(nouns) per concept; 3) MCPL-diverse, where different strings are learned per
image to observe variances among examples. Preliminary evaluations of MCPL-one
and MCPL-diverse methods on the “ball” and “box” multi-concept task are shown
in Figure 3. Our findings indicate that MCPL-one enhance the joint learning of
multiple concepts within the same scene over separate learning. Meanwhile,
MCPL-diverse goes further by facilitating the learning of intricate
relationships between multiple concepts.

#### Limitations of plain MCPL.

Our primary aim is to facilitate accurate interpretation and modification of
multi-concept scenes. To evaluate object-level prompt-concept correlation, we
visualise the average cross-attention maps for each prompt. As depicted in
Figure 4, both MCPL-one and MCPL-all inadequately capture this correlation,
especially for the target concept. These results suggest that naively
extending image-level prompt learning techniques (Gal et al., 2022) to object-
level multi-concept learning poses optimisation challenges, notwithstanding
the problem reformulation efforts discussed in Section 3.3. Specifically,
optimising multiple object-level prompts based on a single image-level
objective proves to be non-trivial. Given the image generation loss equation
1, prompt embeddings may converge to trivial solutions that prioritize image-
level reconstruction at the expense of semantic prompt-object correlations,
thereby contradicting our objectives. In the next section, we introduce
multiple regularisation terms to overcome this challenge.

###  3.4 Regularising the multiple object-level prompts learning

#### Encouraging focused prompt-concept correlation with Attention Masking
(AttnMask).

Previous results show plain MCPL may learn prompts focused on irrelevant
areas. To correct this, we apply masks to both generated and target images
over all the denoising steps (Figure 2, middle-right). These masks, derived
from the average cross-attention of learnable prompts (Figure 2, bottom-row),
constrain the image generation loss (equation 1) to focus on pertinent areas,
thereby improving prompt-concept correlation. To calculate the mask, we
compute for each learnable prompt $p\in\mathcal{P}$ the average attention map
over all time steps $\overline{M}^{p}=1/T\sum_{t=1}^{T}M_{t}^{p}$. We then
apply a threshold to produce binary maps for each learnable prompt, where
$B(M^{p}):=\\{1\text{ if }M^{p}>k,0\text{ otherwise}\\}$ and $k=0.5$
throughout all our experiments. For multiple prompt learning objectives, the
final mask $\mathcal{M}$ is a union of multiple binary masks of all learnable
prompts $\mathcal{M}=\bigcup_{p\in\mathcal{P}}B(M^{p})$. We compute the
Hadamard product of $\mathcal{M}$ with $x$ and $\tilde{x}$ to derive our
masked loss $L^{\textit{AttnMask}}_{DM}$ as equation 2. Our AttnMask is
inspired by Hertz et al. (2022), but a reverse of the same idea, where the
AttnMask is applied over the pixel-level loss equation 1 to constrain the
prompt learning to only related regions.

$L^{\textit{AttnMask}}_{DM}=L_{DM}(\mathcal{M}\odot
x,\mathcal{M}\odot\tilde{x}),$ (2)

#### Encouraging semantically disentangled multi-concepts with Prompts
Contrastive Loss (PromptCL).

AttnMask focuses the learning of multiple prompts on the joint area of target
objects, eliminating the influence of irrelevant regions like the background.
However, it doesn’t inherently promote separation between the embeddings of
different target concepts. Leveraging the mutual exclusivity of multiple
objects in a scene, we introduce a contrastive loss in the latent space where
embeddings are optimised. Specifically, we employ an InfoNCE loss Oord et al.
(2018), a standard in contrastive and representation learning, to encourage
disentanglement between groups of embeddings corresponding to distinct
learnable concepts (Figure 2, middle-left).

Concretely, at each learning step as described in Algorithm 2, a mini-batch
$B$ minor augmented (e.g. with random flip) example images are sampled, with
$N$ learnable prompts/concepts for each image, yields a set of $BN$
embeddings, $\\{v_{b}^{n}\\}_{b=1}^{B},_{n=1}^{N}$. Then, the similarity
between every pair $v_{i}$ and $v_{j}$ of the $BN$ samples is computed using
cosine similarity, i.e. $sim(v_{i},v_{j})=v_{i}^{T}.v_{j}/||v_{i}||||v_{j}||$.
Given our goal is to differentiate the embeddings corresponding to each
prompt, we consider the embeddings of the same concept as positive samples
while the others as negative. Next, the contrastive loss $l^{\eta}_{i,j\in B}$
for a positive pair $v^{\eta}_{i}$ and $v^{\eta}_{j}$ of each concept $\eta\in
N$ (two augmented views of the example image) is shown in the equation 3,
where $\tau$ is a temperature parameter following Chen et al. (2020). The
contrastive loss is computed for $BN$ views of each of the $N$ learnable
concepts. The total contrastive loss $L_{PromptCL}$ is shown in equation 4
(left).

$l^{\eta}_{i,j\in
B}=-log(\frac{exp(sim(v^{\eta}_{i},v^{\eta}_{j}))/\tau}{\sum_{\eta=1}^{N}\sum_{j=1,j\neq{i}}^{B}exp(sim(v^{\eta}_{i},v^{\eta}_{j})/\tau)})$
(3)
$\begin{gathered}L_{PromptCL}=\frac{1}{N}\frac{1}{B}\sum_{\eta=1}^{N}\sum_{i=1}^{B}{l^{\eta}_{i,j\in
B}},\qquad
L_{PromptCL}^{adj}=\frac{1}{NM}\frac{1}{B}\sum_{\eta=1}^{NM}\sum_{i=1}^{B}{l^{\eta}_{i,j\in
B}}\end{gathered}$ (4)

#### Enhance prompt-concept correlation by binding learnable prompt with the
adjective word (Bind adj.).

An additional observation from the misaligned results in Figure 4 reveals that
adjective words often correlate strongly with specific regions. This suggests
that the pre-trained model is already adept at recognising descriptive
concepts like colour or the term ”fluffy.” To leverage this innate
understanding, we propose to optionally associate one adjective word for each
learnable prompt as one positive group during the contrastive loss
calculation. In particular, consider $M$ adjective words associated with $N$
learnable prompts. Then the positive pair $v^{\eta}_{i}$ and $v^{\eta}_{j}$ of
each concept is sampled from $\eta\in MN$ instead of $N$. Therefore The
contrastive loss is now computed for $BNM$ views of each of the $N$ learnable
concepts. The resulting total contrastive loss $L_{PromptCL}^{adj}$ is
detailed in equation 4 (right). We scale $L_{PromptCL}^{adj}$ with a scaling
term $\gamma$ and add with $L^{\textit{AttnMask}}_{DM}$ (equation 2), for them
to have comparable magnitudes, resulting our final loss in equation 5.

$L=L^{\textit{AttnMask}}_{DM}+\gamma L_{PromptCL}^{adj},$ (5)

#### Implementation details.

Unless otherwise noted, we retain the original hyper-parameter choices of LDM
(Rombach et al., 2022). All learnable embeddings were initialised ‘randomly’
with the embeddings of a single-word “photo”. Our experiments were conducted
using a single V100 GPU with a batch size of 4. The base learning rate was set
to $0.005$. Following LDM, we further scale the base learning rate by the
number of GPUs and the batch size, for an effective rate of $0.02$. On
calculating $L_{PromptCL}$, we apply the temperature and scaling term
$(\tau,\gamma)$ of $(0.2,0.0005)$ when AttnMask is not applied, and
$(0.3,0.00075)$ when AttnMask is applied. All results were produced using
$6100$ optimisation steps. We find that these parameters work well for most
cases.

##  4 Results

In this section, we start with qualitatively verifying our proposed
regularisation terms in Section 4.1 We further quantitatively assess the
efficacy of our MCPL method in Section 4.2. Finally, we scrutinise multi-
concept learning and composing capabilities in Section 4.3 across various
tasks, such as image synthesis, editing, and multi-concept separation with
visualisation of attention.

###  4.1 Assessing regularisation terms with cross-attention

We start with assessing our proposed regularisation terms on improving the
accuracy of semantic correlations between prompts and concepts. We visualise
the cross-attention and segmentation masks, as shown in Figure 4. Our visual
results suggest that incorporating all of the proposed regularisation terms
enhances concept disentanglement, whereas applying them in isolation yields
suboptimal outcomes (refer to full ablation results in Appendix A.5).
Moreover, the results demonstrate that MCPL-one is a more effective learning
strategy than MCPL-all, highlighting the importance of excluding irrelevant
prompts to maintain a focused learning objective.

Figure 4: Enhancing object-level prompt-concept correlation in MCPL using the
proposed AttnMask, PromptCL and Bind adj. regularisation techniques. We
compare our best results of MCPL-one applying all regularisation terms against
the plain MCPL-one, using a “Ball and Box” example (left) and the plain MCPL-
all, using a “Hamster and Watermelon” example (right). We use the average
cross-attention maps and the AttnMask to assess the accuracy of correlation.

###  4.2 Quantitative evaluations

We collect both in-distribution natural images and out-of-distribution
biomedical images over 16 object-level concepts, with all images containing
multiple concepts and object-level masks. To approximate the unknown “ground
truth” for disentangled embeddings per concept, we use masks in conjunction
with Textural Inversion Gal et al. (2022). However, we note that these
estimated embeddings serve as our best guess for an unknown true value. Our
evaluation involves four variations of our approach and compares them against
three baseline methods.

#### Dataset.

For the in-distribution natural images dataset, we generate variations of
target objects using local text-driven editing, as proposed by Patashnik et
al. (2023). This minimizes the influence of irrelevant elements like
background. This approach also produces per-text local masks based on
attention maps, assisting us in getting our best approximation for the “ground
truth” of disentangled embeddings. We generate five sets of natural images
containing 10 object-level concepts. For the out-of-distribution bio-medical
image dataset, we assemble three sets of radiological images featuring six
organ/lesion concepts. These images are sourced from three public MRI
segmentation datasets: heart myocardial infarction (Lalande et al., 2020),
prostate segmentation (Antonelli et al., 2022), and Brain Tumor Segmentation
(BraTS) (Menze et al., 2014). Each dataset includes per-concept masks. For
both natural and biomedical datasets, we collected 40 images for each concept.
Figure 5 gives some examples of the prepared datasets.

Figure 5: Quantitative evaluation dataset examples. We prepared five sets of
in-distribution natural images and three sets of out-of-distribution
biomedical images, each containing two concepts resulting in a total of 16
concepts. Visualisation of full sets is available in the Appendix A.7.

#### Baselines and experiments.

We evaluate the effectiveness of four learning methods: 1) Textural Inversion
applied to unmasked multi-concept images, 2) Textural Inversion applied to
each masked object serving as our best estimate for a “ground truth”, 3) MCPL-
all as our naive adaptation of the Textural Inversion method to achieve the
multi-concepts learning goal, which acts as the “state-of-the-art (SoTA)”
given the absence of prior multi-concept learning methods, 4) MCPL-one as our
proposed method. For our method, we additionally examine four variations to
scrutinise the impact of the regularisation terms discussed in Section 3.4.
It’s important to note that, despite the use of a “ground truth” all learning
is performed on unmasked images. To assess the robustness of each learning
method, we randomly sample four images to learn an embedding, leading to 10
learned embeddings per concept. The experiments were executed on a single V100
GPU, with each run taking approximately one hour, resulting in a total
computational cost of around 1940 GPU-hours (or 80 days on a single GPU). We
employed various metrics to evaluate the four methods.

#### Investigate the disentanglement of learned embeddings with t-SNE.

To assess disentanglement, we begin by visualising the t-SNE projection of the
learned features Van der Maaten & Hinton (2008). The results, depicted in
Figure 7, encompass both natural and biomedical datasets. They illustrate that
our MCPL-one combined with all regularisation terms can effectively
distinguish all learned concepts compared to the “SoTA”. It’s noteworthy that,
despite our best efforts to estimate the “ground truth” with masking, their
learned embeddings exhibit less disentanglement when compared to ours. This
performance discrepancy suggests that applying an image-level learning method
for object-level concepts with focused examples is less beneficial, as it
excludes inter-object relationships within the scene. This finding confirms
the necessity of our proposed method.

Figure 6: Visualisation of generated concepts with the “SoTA” and our method.
Masks are derived from cross-attentions. Full ablation results are presented
in the Appendix A.3 Figure 7: The t-SNE projection of the learned embeddings.
Our method can effectively distinguish all learned concepts compared to the
“SoTA” (full results in Appendix A.1).

#### Embedding similarity comparing to the estimated “ground truth”.

To assess the preservation of per-concept semantic and textural details, we
calculate both prompt and image fidelity. This evaluation follows prior
research by Gal et al. (2022) and Ruiz et al. (2022), but differently, we
perform the calculations at the object level. In specific, we compare the
masked “ground truth” (as in Figure 5) and the generated image masked by its
own AttnMask (as in Figure 6). We generated 20 masked images per concept,
resulting in 320 generated images. Prompt fidelity is determined by measuring
the average pairwise cosine similarity between the embeddings learned from the
estimated “ground truth” and the generated masked images, in the pre-trained
embedding space of BERT (Devlin et al., 2018). For image fidelity, we compare
the average pairwise cosine similarity in the pre-trained embedding spaces of
CLIP Radford et al. (2021), DINOv1 (Caron et al., 2021) and DINOv2 (Oquab et
al., 2023), all based on the ViT-S backbone. The results in Figure 8 show our
method combined with all the proposed regularisation terms can improve both
prompt and image fidelity, which consistently outperforms all baselines across
both in-/out-of-distribution concepts and over all four embeddings spaces.

Figure 8: Embedding similarity of learned object-level concept comparing to
the estimated “ground truth”. We measure in both pre-trained text (BERT) and
image encoder (CLIP, DINOv1 and DINOv2) spaces, each bar is an average of
40,000 pairwise cosine similarities. For the text encoder, we use the BERT. We
also present a full object-level comparison in Appendix Section A.2.

###  4.3 Applications: image editing over disentangled concepts.

Finally, we demonstrate our ability to capture object-level embeddings which
enables more accurate object-level synthesis, editing (with methods by e.g.
Hertz et al. (2022)) and quantification (Figure 9 top-left). The framework
also has the flexibility to handle per-image specified string to learn the
subtle differences or new object-level concepts within each example image, as
shown in the top-right example of Figure 9. Furthermore, our method can also
learn unknown concepts from challenging out-of-distribution images (Figure 9
bottom-left and right), opening an avenue of knowledge mining from pairs of
textbook figures and captions, which are abundantly available on the internet.

|
---|---

|
---|---

Figure 9: MCPL learning and composing capabilities. (top-left) learning and
editing multiple concepts with a single string; (top-right) learning per-image
different concepts with per-image specified string; (bottom-left) learning to
disentangle multiple unseen concepts from cardiac MRI images; (bottom-right)
learning to disentangle multiple unseen concepts from chest X-ray images.

##  5 Conclusions

We present MCPL, an innovative approach to prompt learning, designed to tackle
the challenge of managing multiple concepts in scenes with multiple objects.
This enables improved synthesis, editing, quantification, and understanding of
multi-object relationships with greater precision at the object level. We
empirically validated the constraints of utilising the preceding single-
concept learning method in a multi-concept environment, which inspired the
development of our MCPL framework. Importantly, we illustrate that plain MCPL
is ineffective in learning multiple prompt-concept correlations accurately. To
overcome this challenge and prevent trivial solutions, we introduce multiple
regularisation terms during learning. We introduce a novel dataset and
evaluation protocol for this new task. Our approach demonstrates robust and
consistent learning of semantically disentangled concepts over extensive
experiments. Notably, our method can be seamlessly integrated with existing
prompt learning techniques without requiring architectural modifications.

## References

  * Antonelli et al. (2022) Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, Olaf Ronneberger, Ronald M Summers, et al.  The medical segmentation decathlon.  _Nature communications_ , 13(1):4128, 2022.
  * Avrahami et al. (2022) Omri Avrahami, Dani Lischinski, and Ohad Fried.  Blended diffusion for text-driven editing of natural images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 18208–18218, 2022.
  * Brock et al. (2018) Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  _arXiv preprint arXiv:1809.11096_ , 2018.
  * Caron et al. (2021) Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers, 2021.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pp. 1597–1607. PMLR, 2020.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Gal et al. (2022) Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion, 2022.  URL https://arxiv.org/abs/2208.01618.
  * Hertz et al. (2022) Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  2022\.
  * Jia et al. (2022) Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning.  In _European Conference on Computer Vision_ , pp. 709–727. Springer, 2022.
  * Karras et al. (2020) Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 8110–8119, 2020.
  * Kumari et al. (2023) Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu.  Multi-concept customization of text-to-image diffusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 1931–1941, 2023.
  * Lalande et al. (2020) Alain Lalande, Zhihao Chen, Thomas Decourselle, Abdul Qayyum, Thibaut Pommier, Luc Lorgis, Ezequiel de La Rosa, Alexandre Cochet, Yves Cottin, Dominique Ginhac, et al.  Emidec: a database usable for the automatic evaluation of myocardial infarction from delayed-enhancement cardiac mri.  _Data_ , 5(4):89, 2020.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning.  _arXiv preprint arXiv:2104.08691_ , 2021.
  * Lugmayr et al. (2022) Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool.  Repaint: Inpainting using denoising diffusion probabilistic models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 11461–11471, 2022.
  * Meng et al. (2021) Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  Sdedit: Image synthesis and editing with stochastic differential equations.  _arXiv preprint arXiv:2108.01073_ , 2021.
  * Menze et al. (2014) Bjoern H Menze, Andras Jakab, Stefan Bauer, Jayashree Kalpathy-Cramer, Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, et al.  The multimodal brain tumor image segmentation benchmark (brats).  _IEEE transactions on medical imaging_ , 34(10):1993–2024, 2014.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ , 2018.
  * Oquab et al. (2023) Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.  Dinov2: Learning robust visual features without supervision.  _arXiv preprint arXiv:2304.07193_ , 2023.
  * Patashnik et al. (2023) Or Patashnik, Daniel Garibi, Idan Azuri, Hadar Averbuch-Elor, and Daniel Cohen-Or.  Localizing object-level shape variations with text-to-image diffusion models, 2023.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision, 2021.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 10684–10695, 2022.
  * Ruiz et al. (2022) Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  2022\.
  * Tumanyan et al. (2023) Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel.  Plug-and-play diffusion features for text-driven image-to-image translation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 1921–1930, 2023.
  * Van der Maaten & Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  _Journal of machine learning research_ , 9(11), 2008.
  * Wu et al. (2020) Zongze Wu, Dani Lischinski, and Eli Shechtman.  Stylespace analysis: Disentangled controls for stylegan image generation, 2020.
  * Zhou et al. (2022) Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  _International Journal of Computer Vision_ , 130(9):2337–2348, 2022.

##  Appendix A Appendix

Here we provide additional results and various ablation studies and
implementation details that have not been presented in the main paper.

###### Contents

  1. 1 Introduction
  2. 2 Related Works
  3. 3 Methods
    1. 3.1 Preliminaries: prompt learning in text-to-image diffusion model
    2. 3.2 Motivational study: is image-level prompt learning sufficient for object-level multi-concept learning?
    3. 3.3 Multi-Concept Prompt Learning (MCPL)
    4. 3.4 Regularising the multiple object-level prompts learning
  4. 4 Results
    1. 4.1 Assessing regularisation terms with cross-attention
    2. 4.2 Quantitative evaluations
    3. 4.3 Applications: image editing over disentangled concepts.
  5. 5 Conclusions
  6. A Appendix
    1. A.1 Full t-SNE results
    2. A.2 Per-object embedding similarity.
    3. A.3 More qualitative results
    4. A.4 Full motivational experiment results
    5. A.5 Full ablation results of assessing regularisation terms with cross-attention
    6. A.6 Algorithms of Textural-Inversion and MCPL
    7. A.7 Visualisation of quantitative evaluation datasets

###  A.1 Full t-SNE results

Expanding on the results shown in Figure 7, we display the t-SNE results for
all four variations of our approach along with the three baseline methods in
Section 4.2. This includes both in-distribution natural images (Figure 10) and
out-of-distribution biomedical images (Figure 11) across 16 object-level
concepts. Our conclusions remain consistent: each proposed regularisation
improves performance, but employing all regularisation terms with our MCPL-one
yields the most effective disentanglement of all concepts.

Figure 10: The t-SNE visualisations of learned prompt-concept features
(comparing all variants) on the in-distribution natural dataset. We use the
features learned with Textural Inversion Gal et al. (2022) on per-concept
masked images to approximate the unknown ’Ground truth’. We compare two
versions of our proposed MCPL framework MCPL-all (learn all prompts in a
string and MCPL-ong (learn one noun prompt per concept. We also compare two
variants one each version of our method, where we add the proposed
regularisation of AttnMask or AttnMask + PromptCL + Bind adj. over the multi-
concepts learning goal. Figure 11: The t-SNE visualisations of learned prompt-
concept features (comparing all variants) on the out-distribution medical
dataset. We use the features learned with Textural Inversion Gal et al. (2022)
on per-concept masked images to approximate the unknown ’Ground truth’. We
compare two versions of our proposed MCPL framework MCPL-all (learn all
prompts in a string and MCPL-ong (learn one noun prompt per concept. We also
compare two variants one each version of our method, where we add the proposed
regularisation of AttnMask or AttnMask + PromptCL + Bind adj. over the multi-
concepts learning goal.

###  A.2 Per-object embedding similarity.

Building upon the results in Figure 8, we present more extensive object-level
comparisons of the learned concepts against the estimated ’ground truth’. This
encompasses both in-distribution natural images and out-of-distribution
biomedical images, covering 16 object-level concepts. Our conclusions are
consistent: each added regularisation enhances performance, with the
combination of all regularisation terms providing the most effective
disentanglement of concepts. We do notice occasional variations at the object
level, particularly more pronounced in out-of-distribution medical images — a
foreseen challenge. However, these variations are statistically minor and do
not alter the overall conclusion.

Figure 12: Per-object (natural images) embedding similarity (BERT) of learned
concept comparing to the estimated “ground truth”.  Figure 13: Per-object
(natural images) embedding similarity (CLIP) of learned concept comparing to
the estimated “ground truth”.  Figure 14: Per-object (natural images)
embedding similarity (DINOv1) of learned concept comparing to the estimated
“ground truth”.  Figure 15: Per-object (natural images) embedding similarity
(DINOv2) of learned concept comparing to the estimated “ground truth”.  Figure
16: Per-object (medical images) embedding similarity (BERT) of learned concept
comparing to the estimated “ground truth”.  Figure 17: Per-object (medical
images) embedding similarity (CLIP) of learned concept comparing to the
estimated “ground truth”.  Figure 18: Per-object (medical images) embedding
similarity (DINOv1) of learned concept comparing to the estimated “ground
truth”.  Figure 19: Per-object (medical images) embedding similarity (DINOv2)
of learned concept comparing to the estimated “ground truth”.

###  A.3 More qualitative results

Building on the results in Figure 6, we present comprehensive visual results
for all ablation variations of the “SoTA” and our method. This encompasses
both in-distribution natural images and out-of-distribution biomedical images
across 16 object-level concepts. We showcase generated images, attention-
guided masks, and full average attention maps for every word in the sentence
for all methods. These extensive visual results corroborate our quantitative
findings and substantiate our method.

Figure 20: Visualisation of generated in-distribution natural concepts @
(“cactus”) and * (“ball”) with ours and all baseline methods.  Figure 21:
Visualisation of generated in-distribution natural concepts @ (“hamster”) and
* (“watermelon”) with ours and all baseline methods.  Figure 22: Visualisation
of generated in-distribution natural concepts @ (“chair”) and * (“dog”) with
ours and all baseline methods.  Figure 23: Visualisation of generated in-
distribution natural concepts @ (“teddybear”) and * (“skateboard”) with ours
and all baseline methods.  Figure 24: Visualisation of generated in-
distribution natural concepts @ (“basket”) and * (“bananas”) with ours and all
baseline methods.  Figure 25: Visualisation of generated out-of-distribution
medical concepts @ (“cavity”) and * (“myocardium”) with ours and all baseline
methods.  Figure 26: Visualisation of generated out-of-distribution medical
concepts @ (“tumour”) and * (“edema”) with ours and all baseline methods.
Figure 27: Visualisation of generated out-of-distribution medical concepts @
(“transition”) and * (“peripheral”) with ours and all baseline methods.

###  A.4 Full motivational experiment results

In this section, we expand Figure 3 and show full motivational results with
“Watch Face-Band” and “Ball-Box” Images.

Figure 28: Motivational study with watch images. We learn embeddings using
Textural Inversion on both unmasked multi-concept images (“watch face” and
“watch band”) and masked single-concept images (“watch face” or “watch band”).
Figure 29: Learning and Composing “ball” and “box”. We learned the concepts of
“ball” and “box” using different methods (top row) and composed them into
unified scenes (bottom row). We compare three learning methods: Textural
Inversion (left), MCPL-one (middle) and MCPL-diverse (right).

###  A.5 Full ablation results of assessing regularisation terms with cross-
attention

We present in this section the full results of assessing our proposed
regularisation terms in Section 3.4. The results presented in Figure 30
indicate that our plain MCPL algorithm can learn complex multi-object
relationships, yet it may not accurately capture semantic correlations between
prompts and objects. To address this, we introduce several regularisation
terms: AttnMask, PromptCL, and Bind adj.. We assess the efficacy of these
terms in disentangling learned concepts by visualising attention and
segmentation masks, as shown in Figure 30. Although the primary aim of this
work is not segmentation accuracy, we generate segmentation masks of target
concepts to provide a visual quantification of disentanglement. Our visual
results suggest that incrementally incorporating the proposed regularisation
terms enhances concept disentanglement, whereas applying them in isolation
yields suboptimal outcomes. Moreover, the results demonstrate that MCPL-one is
a more effective learning strategy than MCPL-all, highlighting the importance
of excluding irrelevant prompts to maintain a focused learning objective.

Figure 30: Enhancing object-level prompt-concept correlation in MCPL using
proposed AttnMask, PromptCL and Bind adj. regularisation techniques. We
conduct ablation studies to assess the impact of our proposed regularisation
methods. We apply these techniques to the MCPL-one framework, using a “Ball
and Box” example (left) and compare the performance of MCPL-one against MCPL-
all in a “Hamster and Watermelon” example (right). We use average cross-
attention maps to quantify the correlation of each prompt with its
corresponding object-level concept. Additionally, we construct attention-based
masks from multiple selected prompts for the concepts of interest.

###  A.6 Algorithms of Textural-Inversion and MCPL

In this section, we summarise the algorithm of both Textural-Inversion and our
MCPL approach. It is worth noting, during training, to introduce diversity,
the generation is conditioned on phrases constructed from randomly selected
text template $y$ derived from CLIP ImageNet (Radford et al., 2021) and the
new prompt $p^{*}$, such as “A photo of $p^{*}$”, “A sketch of $p^{*}$”, etc.

Input: a small set of images $x_{0}$, pre-trained text-encoder $c_{\theta}$
and denoising network ${\epsilon}_{\theta}$.
Output: an embedding $v^{*}$ corresponds to new prompt $p^{*}$.
random initialise $v^{*}=c_{\theta}(p^{*})$
# optimising $v^{*}$ with $L_{DM}$
for  _$step=1,\ldots,S$_ do

randomly sample neutral texts $y$ to make string $[y,p^{*}]$
for  _$t=T,T-1,\ldots,1$_ do

$v^{*}:=\operatorname*{arg\,min}_{v}E_{x_{0},{\epsilon}\sim N(0,I)}\\\
\|{\epsilon}-{\epsilon}_{\theta}(x_{t},t,[c_{\theta}(y),v^{*}]\|^{2}$

end for

end for

Return $(p^{*},v^{*})$

Algorithm 1 Textural-Inversion

Input: a small set of images $x_{0}$, pre-trained text-encoder $c_{\theta}$
and denoising network ${\epsilon}_{\theta}$.
Output: a list of multiple embeddings $\mathcal{V}=[v^{*},\ldots,v^{\&}]$
corresponds to multiple new prompts $\mathcal{P}=[p^{*},\ldots,{p}^{\&}]$.
random initialise
$[v^{*},\ldots,v^{\&}]=[c_{\theta}(p^{*}),\ldots,c_{\theta}(p^{\&})]$
# optimising $\\{v^{*},\ldots,v^{\&}\\}$ with $L_{DM}$
for  _$step=1,\ldots,S$_ do

randomly sample neutral texts $y$ to make string $[y,p^{*},\ldots,p^{\&}]$
for  _$t=T,T-1,\ldots,1$_ do

$[v^{*},\ldots,v^{\&}]:=\operatorname*{arg\,min}_{\mathcal{V}}E_{x_{0},{\epsilon}\sim
N(0,I)}\\\
\|{\epsilon}-{\epsilon}_{\theta}(x_{t},t,[c_{\theta}(y),v^{*},\ldots,v^{\&}]\|^{2}$

end for

end for

Return $(\mathcal{P},\mathcal{V})$

Algorithm 2 MCPL

###  A.7 Visualisation of quantitative evaluation datasets

Finally, we present the novel dataset we have generated and collected for this
new task of learning object-level concepts. The detailed description of the
dataset is provided in the earlier Section 4.2 \- Dataset.

Figure 31: Visualisation of the full sets of generated and masked hamster-
watermelon images. Figure 32: Visualisation of the full sets of generated and
masked bananas-basket images. Figure 33: Visualisation of the full sets of
generated and masked cactus-ball images. Figure 34: Visualisation of the full
sets of generated and masked teddybear-skateboard images. Figure 35:
Visualisation of the full sets of generated and masked chair-dog images.
Figure 36: Visualisation of the full sets of generated and masked cavity-
myocardium images. Figure 37: Visualisation of the full sets of generated and
masked transition-peripheral images. Figure 38: Visualisation of the full sets
of generated and masked tumour-edema images.


# Holographic Approach to Neutron Stars

Tinglong Feng https://arendelle-ftl.github.io/<EMAIL_ADDRESS>Xi’an
Jiaotong University

###### Abstract

In this article we explore the holographic approach to neutron stars in the
realm of Quantum Field Theory (QFT). We delve into the structures of neutron
stars, emphasizing the application of the AdS/CFT duality in modeling them. We
discuss both ”bottom-up” and ”top-down” holographic models, comparing their
predictions with astrophysical observations. Finally, we demonstrate the
potential broader applications of the holography method in areas like
superconductivity, highlighting the methodological significance of string
theory and QFT in astrophysics.

Key words

Holography, neutron stars, quantum chromodynamics

##  I Introduction

After a massive star experiences a supernova explosion and a subsequent
gravitational collapse, a compact star emerges as a remnant. If the mass of
the original hydrongen-burning star is not high enough to result in a direct
collapse into a black hole, the remnant will be a neutron star, in which the
Fermi and interaction repulsion should be able to withstand the gravitational
collapse. A general method to deal with the corresponding strong nuclear force
is Quantum Chromodynamics(QCD), which are often approached through weak-
coupling expansion. Nevertheless, at neutron-star cores, QCD remains a strong
coupled theory such that the perturbative methods fail, and we need
nonperturbative method. [1]

Holography is an impressive nonperturbative method based on famous AdS/CFT
duality, which is a strong/weak duality in the sense that the gravitational
theory is weakly curved when the effective coupling of the gauge is large,
which means this method is suitable for describing some strongly interacting
matter, such as neutron stars.

In this article we will give a brief review of some novel holographic models
for building a neutron star. In the next section, we will recall the basics
equations for describing stars and neutron stars. In section III we will give
a brief review of QFT as necessary building blocks that enable studying
holographic method. In section IV we will introduce AdS/CFT duality and
discuss two holographic models, together with the predictions they give.
Section V is the summary of this article.

##  II From Stars to Neutron Stars

###  II.1 Stars

First, let’s consider a star[2]. Suppose it is in equilibrium and is
spherically symmetric, then a thin spherical shell feels a gravitational force

$F_{1}=-G\frac{4\uppi r^{2}\rho(r)\mathrm{d}rM(r)}{r^{2}}=-4\uppi
G\rho(r)M(r)\mathrm{d}r$ (1)

where $M(r)$ is the total mass interior to $r$ satisfying

$\frac{\mathrm{d}M(r)}{\mathrm{d}r}=4\uppi r^{2}\rho(r).$ (2)

The buoyant force provided by pressure is

$F_{2}=4\uppi r^{2}[p(r)-p(r+\mathrm{d}r)]=-4\uppi
r^{2}p^{\prime}(r)\mathrm{d}r.$ (3)

For equilibrium condition we have $F_{1}=-F_{2}$, hence

$\frac{\mathrm{d}p(r)}{\mathrm{d}r}=-\frac{GM(r)\rho(r)}{r^{2}}.$ (4)

Equation (2) and (4) are hydrostatic equilibrium condition for stars.

###  II.2 Neutron Stars

Now let’s turn to neutron stars. Obviously in a neutron star nuclear reactions
have already ended so that its temperature tends to zero. Recall neutrons are
Fermions, we could treat neutron stars as the ideal degenerate Fermi gas at
$T=0$, a model with which we have been familiar when learning statistical
mechanics in kindergarten. According to Fermi statistics the mass density
could be denoted by

$\rho(r)=\frac{8\uppi
m_{\mathrm{n}}}{h^{3}}\int_{0}^{k_{\mathrm{F}}(r)}k^{2}\mathrm{d}k=\frac{8\uppi
m_{\mathrm{n}}k^{3}_{\mathrm{F}}}{3h^{3}}$ (5)

where $m_{\mathrm{n}}$ is the neutron mass and $k_{\mathrm{F}}$ is the Fermi
momentum, the maximum momentum of the filled neutron levels. In a similar way
we could obtain pressure

$p(r)=\frac{8\uppi
c^{2}}{3h^{3}}\int_{0}^{k_{\mathrm{F}}(r)}\frac{k^{4}}{\sqrt{k^{2}c^{2}+m^{2}_{\mathrm{n}}c^{4}}}\mathrm{d}k.$
(6)

For a given central density $\rho(0)$, we could use (5) together with
hydrostatic equilibrium (2),(4) to obtain $\rho(r)$, and then mass $M$ and
radius $R$. Now we try to find the maximum mass of neutron stars. Notice that
there is a critical density $\rho_{c}$ where $k_{\mathrm{F}}=m_{\mathrm{n}}c$

$\rho_{c}=\frac{8\uppi m^{4}_{\mathrm{n}}c^{3}}{3h^{3}}=6.11\times
10^{15}\mathrm{g/cm^{3}}$ (7)

and the mean separation between neutrons

$\left(\frac{\rho}{m_{\mathrm{n}}}\right)^{-1/3}=\left(\frac{\rho_{c}}{\rho}\right)^{1/3}\times
0.52\times 10^{-13}\mathrm{cm}.$ (8)

When we consider $\rho\geq\rho_{c}$, a problem arises that the neutron
velocities are similar to $c$ such that we need to use general gravity to
figure out the structure of neutron stars, calculations corresponding to which
by Oppenheimer and Volkoff[3] suggested the maximum mass is $0.7M_{\odot}$
(where $M_{\odot}$ is solar mass). Nevertheless this calculations are also
inaccurate, for if $\rho\geq\rho_{c}$, (8) shows that the seperation of
neutrons is less than the range of nuclear force such that we can no longer
treat neutrons as ideal Fermi gas. Actually we have found neutron stars with
$2.1M_{\odot}$[4] or even $2.5M_{\odot}$[5] masses, according to recent
observations. Hence we must use QCD to treat this situation. Nevertheless, as
we have said in section I, perturbative QCD can not be used in this situations
as well, so we must turn to nonperturbative method, for example, holography.

##  III A Brief Review of QFT

Before entering holographic method, we shall first give a short review[6] of
QFT to provide a necessary foundation.

###  III.1 Canonical Formalism

The canonical quantization of a field obeys following procedure

  1. 1.

writing down the Lagrangian density $\mathcal{L}(\phi,\partial_{\mu}\phi)$ of
the field, using the Euler-Lagrange equation” to obtain field equation

  2. 2.

figuring out the canonical momentum density
$\Pi^{\mu}=\partial\mathcal{L}/\partial(\partial_{\mu}\phi)$ and Hamiltonian
density $\mathcal{H}=\Pi^{0}\partial_{0}\phi-\mathcal{L}$

  3. 3.

Transferring the field and momentum density to operators satisfying canonical
commutation relations

  4. 4.

Using creation and annihilation operators to express field operator

For example, let’s quantize a free real scalar field (we will use natural
units below). The Lagrangian density is

$\mathcal{L}=\frac{1}{2}\partial_{\mu}\phi(x)\partial^{\mu}\phi(x)-\frac{1}{2}m^{2}\phi(x)^{2}$

canonical momentum density

$\Pi^{0}(x)=\partial^{0}\phi(x)$

Hamiltonian

$\mathcal{H}=\frac{1}{2}[\partial_{0}\phi(x)]^{2}+\frac{1}{2}[\bm{\nabla}\phi(x)]^{2}+\frac{1}{2}m^{2}\phi^{2}(x)$

Transfer field and momentum density to operators

$\phi(x)\to\widehat{\phi}(x),\quad\Pi^{0}(x)\to\widehat{\Pi}^{0}(x)$

satisfying

$[\widehat{\phi}(t,\bm{x}),\widehat{\Pi}^{0}(t,\bm{y})]=\mathrm{i}\bm{\delta}(\bm{x}-\bm{y}),$
$[\widehat{\phi}(\bm{x}),\widehat{\phi}(\bm{y})]=[\widehat{\Pi}^{0}(\bm{x}),\widehat{\Pi}^{0}(\bm{y})]=0$

Use creation and annihilation operators to express field operator

$\widehat{\phi}(\bm{x})=\int\frac{\mathrm{d}^{3}p}{\sqrt{(2\uppi)^{3}2E_{\bm{p}}}}\left(a_{\bm{p}}\mathrm{e}^{-\mathrm{i}(E_{\bm{p}}t-\bm{p}\cdot\bm{x})}+a^{\dagger}_{\bm{p}}\mathrm{e}^{\mathrm{i}(E_{\bm{p}}t-\bm{p}\cdot\bm{x})}\right)$

where

$E_{\bm{p}}=\sqrt{\bm{p}^{2}+m^{2}}$

###  III.2 Path-Integral Methods

There’s another method besides canonical formalism, i.e. the path-integral
method, which will be useful in AdS/CFT. In quantum mechanics we have sum over
path[7]

$K(a,b)=\int_{a}^{b}\mathcal{D}x(t)\mathrm{e}^{(\mathrm{i}/\hbar)S[b,a]}$ (9)

Similarly, in QFT we have generating functional

$Z[J]=\int\mathcal{D}[\phi(x)]\exp\left\\{\mathrm{i}\left[S+\int
J(x)\phi(x)\right]\right\\}$ (10)

where $\phi(x)$ is the field, $S$ is the action, $J(x)$ is a source. From the
generating function we can obtain information the quantum field, such as
Green’s function.

##  IV Holographic Neutron Stars

###  IV.1 AdS/CFT duality

Holography, or AdS/CFT duality, provides the generating fucntional in
$D$-dimensional Minkowski space by using classical action
$S[\phi^{\mathrm{AdS}}]$ of a field on (at least) $D+1$-dimensional Anti-
deSitter Space[8].

Figure 1:  AdS space, taken from[9]

Fig 1 shows AdS/CFT, where the right blue plane representing the Minkowski
space the QFT lives in, often called the boundary. Leaving away from the
boundary is another dimension, labelled with $r$. The larger space usually
called the bulk, inside which a theory of gravity lives. Holography claims
that gravity in the bulk and QFT on the boundry are equivalent[9]. The
fundamental formula[10] of hologrophy is

$Z_{\mathrm{QFT}}[\phi_{0}]=Z_{\mathrm{QG}}[\phi]|_{\phi\to\phi_{0}}$ (11)

where QG denotes quantum gravity, and $Z_{\mathrm{QFT}}[\phi_{0}]$ is indeed
from (10). The most impressive feature is, when the QFT has a large number of
degrees of freedom and is strongly coupled, the dual description reduces to
classical gravity[11], hence

$Z_{\mathrm{QFT}}[\phi_{0}]\simeq\mathrm{e}^{\mathrm{i}S_{\mathrm{bulk}}}|_{\phi\to\phi_{0}}.$
(12)

(12) tells us that if the classical action $S_{\mathrm{bulk}}$ is given, we
can calculate its dual description in the boundry, i.e. , a strongly coupled
QFT, such as strongly coupled QCD.

###  IV.2 Building a Realistic Neutron Stars from Holography

Generally speaking, there are two kinds of holographic models for QCD: the
”bottom-up” models and the ”top-down” models. The former are more
phenomenological while the latter are rigorously based on string theory. Both
of them can be used to build a realistic neutron stars which provide
meaningful predictions. We will discuss two novel models, one from each kind,
and review their predictions.

The first one[12, 13] employs a ”top-down” approach, the Saki-sugimoto model,
to build a neutron star, which has two parameters, the ’t Hooft coupling
$\lambda$ and the Kaluza-Klein mass $M_{\mathrm{KK}}$. Adding leptons, and
constructing a holographic crust in order to account for realistic neutron
stars, the model figures out the scales of mass and radius

$M_{0}\simeq
1.445\lambda_{0}^{-3/2}\left(\frac{M_{\mathrm{KK}}}{\mathrm{GeV}}\right)^{-2}M_{\odot}$
(13) $r_{0}\simeq
2.135\lambda_{0}^{-3/2}\left(\frac{M_{\mathrm{KK}}}{\mathrm{GeV}}\right)^{-2}\mathrm{km}$
(14)

Figure 2:  Mass-radius curves for different parameter sets, taken from[12]

Figure 2 shows the Mass-radius curves obtained in this model. Besides that,
this model also obtains tidal deformability using numerical method.

The second one is a ”bottom-up” model[14, 15], inspired by holographic
electric superconductors. The action is

$S_{\mathrm{Bulk}}+\int\mathrm{d}^{d+1}x\sqrt{-g}\left\\{\mathcal{R}+\frac{d(d-1)}{L^{2}}-\frac{1}{4}F^{2}\right\\}.$
(15)

The equation of state can be derived from this model

$p=\varepsilon-\sqrt{a\varepsilon}\mu_{c}$ (16)

where $p$ is pressure, $\varepsilon$ is energy density and $a,\mu_{c}$ are
parameters. Together with TOV equations the $M(R)$ can be plotted as

Figure 3:  Plot of $M(R)$, taken from[14]

###  IV.3 Prediction

Both models mentioned above give predictions. The first one forecasts the
maximal mass, radius and tidal deformability of corresponding neutron stars,
as in Table 1. Among this, the upper limit of maximal mass, $2.46M\odot$, and
the lower limit of tidal deformability of a $1.4M\odot$ neutron star, 277, are
novel and independent prediction which could be verified according to future
obsevational results.

Table 1: Constraints obtained by combining the holographic results with astrophysical data[16, 17, 18, 19] for maximal mass as well as radius and tidal deformability $\Lambda$ in the ”top-down” model  | lower bound | upper bound
---|---|---
$M_{\max}[M\odot]$ | 2.1 | 2.46
$R_{1.4}[\mathrm{km}]$ | 11.9 | 14.3
$R_{2.1}[\mathrm{km}]$ | 11.4 | 13.7
$\Lambda_{1.4}$ | 277 | 580
$\Lambda_{2.1}$ | 9.13 | 29.3

The second model confirms the possibility of the existence of a neutron star
with $(M,R)\sim(2M\odot,10\mathrm{km})$. Furthermore, for the model is
inspired by holographic electric superconductors, it is suitable to
investigate whether a neutron star with a color superconducting (CSC) core.
According to [15], this case is impossible for the instability of
corresponding stars.

##  V conclusion

In this article we give a brief review of holographic method for building
neutron stars, display some novel models and corresponding predictions.
Holographic neutron star is just a simple application of the impressive
AdS/CFT duality, which is also very useful in other areas, such as
superconductivity[20, 21, 22, 23]. The fantastic power of AdS/CFT is due to
its Strong/Weak duality, which could transfer strongly coupled field problems
to almost classical gravity problems, and the latter is much easier to solve.
This method displays methodological value of string theory, or more generally,
QFT. Even if we have not yet formulated a widely accepted, consistent theory
that combines general relativity and quantum mechanics, the current progress
in QFT still provides us with valuable methods in various application areas.

## References

  * Hoyos  _et al._ [2022] C. Hoyos, N. Jokela, and A. Vuorinen, Holographic approach to compact stars and their binary mergers, Progress in Particle and Nuclear Physics 126, 103972 (2022).
  * Weinberg [2019] S. Weinberg,  _Lectures on Astrophysics_ (Cambridge University Press, 2019).
  * Oppenheimer and Volkoff [1939] J. R. Oppenheimer and G. M. Volkoff, On massive neutron cores, Phys. Rev. 55, 374 (1939).
  * Cromartie  _et al._ [2020] H. T. Cromartie, E. Fonseca, and S. M. Ransom, Relativistic shapiro delay measurements of an extremely massive millisecond pulsar, Nature Astronomy 4, 72 (2020).
  * Abbott  _et al._ [2020] R. Abbott, T. D. Abbott, S. Abraham, L. S. Collaboration, and V. Collaboration, Gw190814: Gravitational waves from the coalescence of a 23 solar mass black hole with a 2.6 solar mass compact object, The Astrophysical Journal Letters 896, L44 (2020).
  * Weinberg [1995] S. Weinberg,  _The Quantum Theory of Fields_ (Cambridge University Press, 1995).
  * Feynman [1965] R. P. Feynman,  _Quantum mechanics and path integrals_ (1965).
  * Rehren [2004] K.-H. Rehren, Qft lectures on ads-cft, (2004).
  * Tong [2013] D. Tong, Holographic conductivity, Acta Physica Polonica B 44, 2579 (2013).
  * Witten [1998] E. Witten, Anti-de sitter space, thermal phase transition, and confinement in gauge theories, Advances in Theoretical and Mathematical Physics 2 (1998).
  * Maldacena [1997] J. Maldacena, The large n limit of superconformal field theories and supergravity, Adv. Theor. Math. Phys. 2 (1997).
  * Kovensky  _et al._ [2022] N. Kovensky, A. Poole, and A. Schmitt, Building a realistic neutron star from holography, Phys. Rev. D 105, 034022 (2022).
  * Kovensky  _et al._ [2021] N. Kovensky, A. R. Poole, and A. Schmitt, Predictions for neutron stars from holographic nuclear matter, SciPost Physics Proceedings (2021).
  * Ghoroku  _et al._ [2021] K. Ghoroku, K. Kashiwa, Y. Nakano, M. Tachibana, and F. Toyoda, Stiff equation of state for a holographic nuclear matter as instanton gas, Phys. Rev. D 104, 126002 (2021).
  * Ghoroku  _et al._ [2023] K. Ghoroku, K. Kashiwa, Y. Nakano, M. Tachibana, and F. Toyoda, Instability of holographic cold compact stars with a color superconducting core, Phys. Rev. D 107, 126007 (2023).
  * Abbott  _et al._ [2019] B. P. Abbott, R. Abbott, and T. D. Abbott (LIGO Scientific Collaboration and Virgo Collaboration), Properties of the binary neutron star merger gw170817, Phys. Rev. X 9, 011001 (2019).
  * Miller  _et al._ [2021] M. C. Miller, F. K. Lamb, A. J. Dittmann, and S. Bogdanov, The radius of psr j0740+6620 from nicer and xmm-newton data, The Astrophysical Journal Letters 918, L28 (2021).
  * Miller  _et al._ [2019] M. C. Miller, F. K. Lamb, and A. J. Dittmann, Psr j0030+0451 mass and radius from nicer data and implications for the properties of neutron star matter, The Astrophysical Journal Letters 887, L24 (2019).
  * Riley  _et al._ [2019] T. E. Riley, A. L. Watts, and S. Bogdanov, A nicer view of psr j0030+0451: Millisecond pulsar parameter estimation, The Astrophysical Journal Letters 887, L21 (2019).
  * Hartnoll  _et al._ [2008a] S. A. Hartnoll, C. P. Herzog, and G. T. Horowitz, Holographic superconductors, Journal of High Energy Physics 2008, 015 (2008a).
  * Hartnoll  _et al._ [2008b] S. A. Hartnoll, C. P. Herzog, and G. T. Horowitz, Building a holographic superconductor, Phys. Rev. Lett. 101, 031601 (2008b).
  * Benini [2012] F. Benini, Holography and condensed matter, Fortschritte der Physik 60, 810 (2012).
  * Salvio [2013] A. Salvio, Superconductivity, superfluidity and holography, Journal of Physics: Conference Series 442, 012040 (2013).