# A family of weighted Erdős-Mordell inequality and applications

Tran Quang Hung  High School for Gifted Students, Hanoi, Vietnam
<EMAIL_ADDRESS>
###### Abstract.

We establish some new generalizations of Erdős-Mordell inequality by adding
weights to its terms. Using these generalizations, we derived strengthened
versions of the original Erdős-Mordell inequality. We also found two other
variations of Erdős-Mordell inequality to be generalizable in the same way.

###### Key words and phrases:

Geometric inequality, Erdős-Mordell inequality, weighted inequality

###### 2010 Mathematics Subject Classification:

51M04, 51M16

##  1\. Introduction

Throughout this article we denote the distance between two points $X$ and $Y$
simply by $XY$. Let $P$ be an interior point of the triangle $ABC$. Then

$PA+PB+PC\geq 2(d_{a}+d_{b}+d_{c}),$

where $d_{a}$, $d_{b}$, and $d_{c}$ are the distances from $P$ to the sides
$BC$, $CA$, $AB$, respectively, and equality holds if and only if triangle
$ABC$ is equilateral and $P$ is its center.

Figure 1. Erdős-Mordell inequality.

This is the well-known and elegant Erdős–Mordell inequality. It was named
after Erdős, who conjectured it in 1935 [1], and Mordell, who was the first to
prove it in 1937 [2]. Throughout the history of elementary geometry, there has
been great deals of proofs, developments and generalizations to this classical
result, see [2, 3, 13, 14, 15, 16, 17, 19, 21, 22, 23, 20].

In this article, following the ideas of generalizing Erdős–Mordell inequality
using weights from Dar and Gueron [3] and Liu [23], we developed a new
generalization using a different type of weight. As a result, we discovered
new weighted generalizations together with strengthened versions of the
original Erdős–Mordell inequality. In addition, the new weighting method is
able to generalize two other variations of the original Erdős–Mordell
inequality. The first one is Barrow’s inequality [2]:

$PA+PB+PC\geq 2(l_{a}+l_{b}+l_{c}),$

where $l_{a}$, $l_{b}$, and $l_{c}$ are the lengths of the bisectors of
$\angle BPC$, $\angle CPA$, and $\angle APB$, respectively, and equality holds
if and only if triangle $ABC$ is equilateral and $P$ is its center. The second
variation is from Dao, Nguyen, and Pham [17]:

$R_{A}+R_{B}+R_{C}\geq 2(d_{a}+d_{b}+d_{c}),$

where $R_{A}$, $R_{B}$, and $R_{C}$ are the distances of $P$ from the tangents
to circumcircle of triangle $ABC$ at $A$, $B$, and $C$, respectively, and
equality holds if and only if triangle $ABC$ is equilateral and $P$ is its
center.

We present the new weighted generalizations with proofs in Section 2 and the
resulting some applications and strengthened inequalities will be given in
Section 3.

##  2\. Main theorems and proofs

We begin with a new weighted version of Erdős-Mordell inequality:

######  Theorem 1 (A new weighted version of Erdős-Mordell inequality).

Let $x$, $y$, $z$, $u$, $v$, and $w$ be positive real numbers such that
$xyz=uvw=1$. Let $P$ be an interior point of the triangle $ABC$. Denote by
$d_{a}$, $d_{b}$, and $d_{c}$ the distances of $P$ from the sides $AB$, $BC$,
and $CA$, respectively. Then

(1) $x(PA+u^{3}d_{a})+y(PB+v^{3}d_{b})+z(PC+w^{3}d_{c})\geq
3(ud_{a}+vd_{b}+wd_{c}).$

Equality holds if and only if $x=y=z=u=v=w=1$, triangle $ABC$ is equilateral
and $P$ is its center.

###### Proof.

With standard notations $BC=a$, $CA=b$, and $AB=c$, we recall a result in [3]:

(2) $PA\geq\frac{b\cdot d_{c}+c\cdot d_{b}}{a}.$

Equality in (2) holds if and only if $P$ lies on the line connecting $A$ to
the circumcenter of $ABC$. Since $x$ is a positive real number,

(3) $x(PA+u^{3}d_{a})\geq x\left(\frac{b\cdot d_{c}+c\cdot
d_{b}}{a}+u^{3}d_{a}\right).$

Similarly, with positive real numbers $y$ and $z$, we get

(4) $y(PB+v^{3}d_{b})\geq y\left(\frac{c\cdot d_{a}+a\cdot
d_{c}}{b}+v^{3}d_{b}\right)$

and

(5) $z(PC+w^{3}d_{c})\geq z\left(\frac{a\cdot d_{b}+b\cdot
d_{a}}{c}+w^{3}d_{c}\right)$

with analogous conditions for equality. Now summing the inequalities (3), (4),
and (5), we obtain

(6)
$x(PA+u^{3}d_{a})+y(PB+v^{3}d_{b})+z(PC+w^{3}d_{c})\geq\left(y\cdot\frac{c}{b}+z\cdot\frac{b}{c}+xu^{3}\right)d_{a}+\\\
+\left(z\cdot\frac{b}{a}+x\cdot\frac{c}{a}+yv^{3}\right)d_{b}+\left(x\cdot\frac{a}{b}+y\cdot\frac{a}{b}+zw^{3}\right)d_{c}.$

and equality holds if and only if $P$ is circumcenter of $\triangle ABC$.

Using the arithmetic mean-geometric mean inequality for three positive
numbers, with notice that $xyz=1$, we have

(7) $y\cdot\frac{c}{b}+z\cdot\frac{b}{c}+xu^{3}\geq
3\sqrt[3]{y\cdot\frac{c}{b}\cdot z\cdot\frac{b}{c}\cdot xu^{3}}=3u.$

with equality holds if and only if

(8) $y\cdot\frac{c}{b}=z\cdot\frac{b}{c}=xu^{3},$

which is equivalent to

(9) $u^{3}\cdot\frac{x}{y}=\frac{c}{b}\quad\text{and}\quad
u^{3}\cdot\frac{x}{z}=\frac{b}{c}.$

Similarly,

(10) $z\cdot\frac{b}{a}+x\cdot\frac{c}{a}+yv^{3}\geq 3v,$

and

(11) $x\cdot\frac{a}{b}+y\cdot\frac{a}{b}+zw^{3}\geq 3w$

Since $uvw=1$, the equalities hold if and only if

(12) $x=y=z=u=v=w=1\quad\text{and}\quad a=b=c.$

Finally, combining (6), (7), (10), and (11), we get

(13) $x(PA+u^{3}d_{a})+y(PB+v^{3}d_{b})+z(PC+w^{3}d_{c})\geq
3(ud_{a}+vd_{b}+wd_{c})$

with equality holds if and only if

(14) $x=y=z=u=v=w=1\quad\text{and}\quad a=b=c.$

This completes the proof of Theorem 1. ∎

######  Remark.

In Theorem 1, where $u=v=w=1$, we obtain

(15) $x(PA+d_{a})+y(PB+d_{b})+z(PC+d_{c})\geq 3(d_{a}+d_{b}+d_{c}).$

In Theorem 1, where $x=y=z=1$, we obtain

(16) $PA+PB+PC\geq(3u-u^{3})d_{a}+(3v-v^{3})d_{b}+(3w-w^{3})d_{c}.$

Theses inequalities can be considered as other weighted versions of Erdős-
Mordell inequality.

Similarly, we get a new weighted Dao-Nguyen-Pham’s inequality in [17].

######  Theorem 2 (A new weighted version of Dao-Nguyen-Pham’s inequality).

Let $x$, $y$, $z$, $u$, $v$, and $w$ be positive real numbers such that
$xyz=uvw=1$. Let $P$ be an interior point of the triangle $ABC$ with
circumcircle $(\omega)$. Denote by $R_{A}$, $R_{B}$, and $R_{C}$ the distances
of $P$ from the tangents to $(\omega)$ at $A$, $B$, $C$ respectively, $d_{a}$,
$d_{b}$, and $d_{c}$ the distances of $P$ from the sides $AB$, $BC$, and $CA$,
respectively. Then

(17) $x(R_{A}+u^{3}d_{a})+y(R_{B}+v^{3}d_{b})+z(R_{C}+w^{3}d_{c})\geq
3(ud_{a}+vd_{b}+wd_{c}).$

Equality holds if and only if $x=y=z=u=v=w=1$, triangle $ABC$ is equilateral
and $P$ is its center.

###### Proof.

With standard notations $BC=a$, $CA=b$, and $AB=c$, we recall a result of the
second proof in [17]:

(18) $R_{A}=\frac{b\cdot d_{c}+c\cdot d_{b}}{a}.$

Since $x$ is a positive real number,

(19) $x(R_{A}+u^{3}d_{a})=x\left(\frac{b\cdot d_{c}+c\cdot
d_{b}}{a}+u^{3}d_{a}\right).$

Similarly, with positive real numbers $y$ and $z$, we get

(20) $y(R_{B}+v^{3}d_{b})=y\left(\frac{c\cdot d_{a}+a\cdot
d_{c}}{b}+v^{3}d_{b}\right)$

and

(21) $z(R_{C}+w^{3}d_{c})=z\left(\frac{a\cdot d_{b}+b\cdot
d_{a}}{c}+w^{3}d_{c}\right)$

Now summing the inequalities (19), (20), and (21), we obtain the identity

(22)
$x(R_{A}+u^{3}d_{a})+y(R_{B}+v^{3}d_{b})+z(R_{C}+w^{3}d_{c})=\left(y\cdot\frac{c}{b}+z\cdot\frac{b}{c}+xu^{3}\right)d_{a}+\\\
+\left(z\cdot\frac{b}{a}+x\cdot\frac{c}{a}+yv^{3}\right)d_{b}+\left(x\cdot\frac{a}{b}+y\cdot\frac{a}{b}+zw^{3}\right)d_{c}.$

Using the arithmetic mean-geometric mean inequality for three positive
numbers, with notice that $xyz=1$, we have

(23) $y\cdot\frac{c}{b}+z\cdot\frac{b}{c}+xu^{3}\geq
3\sqrt[3]{y\cdot\frac{c}{b}\cdot z\cdot\frac{b}{c}\cdot xu^{3}}=3u.$

with equality holds if and only if

(24) $y\cdot\frac{c}{b}=z\cdot\frac{b}{c}=xu^{3},$

which is equivalent to

(25) $u^{3}\cdot\frac{x}{y}=\frac{c}{b}\quad\text{and}\quad
u^{3}\cdot\frac{x}{z}=\frac{b}{c}.$

Similarly,

(26) $z\cdot\frac{b}{a}+x\cdot\frac{c}{a}+yv^{3}\geq 3v,$

and

(27) $x\cdot\frac{a}{b}+y\cdot\frac{a}{b}+zw^{3}\geq 3w$

Since $uvw=1$, the equalities hold if and only if

(28) $x=y=z=u=v=w=1\quad\text{and}\quad a=b=c.$

Finally, combining (22), (23), (26), and (27), we get

(29) $x(R_{A}+u^{3}d_{a})+y(R_{B}+v^{3}d_{b})+z(R_{C}+w^{3}d_{c})\geq
3(ud_{a}+vd_{b}+wd_{c})$

with equality holds if and only if

(30) $x=y=z=u=v=w=1\quad\text{and}\quad a=b=c.$

This completes the proof of Theorem 2. ∎

######  Remark.

Since $R_{A}\leq PA$, $R_{b}\leq PB$, and $R_{c}\leq PC$, Theorem 2 can be
considered as a strengthened version of Theorem 1. In Theorem 2, let $u=v=w=1$
or $x=y=z=1$, we obtain two weighted inequalities

(31) $x(R_{A}+d_{a})+y(R_{B}+d_{b})+z(R_{C}+d_{c})\geq 3(d_{a}+d_{b}+d_{c}),$

and

(32) $R_{A}+R_{B}+R_{C}\geq(3u-u^{3})d_{a}+(3v-v^{3})d_{b}+(3w-w^{3})d_{c}.$

They are considered as other weighted versions of Dao-Nguyen-Pham’s
inequality.

By the same way, we have a generalization for Barrow’s inequality. Similar to
Barow’s proof, we will use Wolstenholme’s inequality [4] to prove this
generalization.

######  Theorem 3 (A new weighted version of Barrow’s inequality).

Let $x$, $y$, $z$, $u$, $v$, and $w$ be positive real numbers such that
$xyz=uvw=1$. Let $P$ be an interior point of the triangle $ABC$. Denote by
$l_{a}$, $l_{b}$, and $l_{c}$ the lengths of the bisectors of $\angle BPC$,
$\angle CPA$, and $\angle APB$, respectively. Then

(33)
$x(PA+u^{3}l_{a})+y(PB+v^{3}l_{b})+z(PC+w^{3}l_{c})\geq(3ul_{a}+vl_{b}+wl_{c}).$

Equality holds if and only if $x=y=z=u=v=w=1$, triangle $ABC$ is equilateral
and $P$ is its center.

###### Proof.

Let $\angle BPC=\alpha$, $\angle CPA=\beta$, angle $APB=\gamma$, using formula
of bisector, we have

(34) $l_{a}=\frac{PB\cdot PC\cos\frac{\alpha}{2}}{PB+PC},$

this is equivalent to

(35) $2\sqrt{PB\cdot
PC}\cos\frac{\alpha}{2}=\left(\sqrt{\frac{PB}{PC}}+\sqrt{\frac{PC}{PB}}\right)l_{a}.$

Multiplying both sides of (35) with $\sqrt{yz}$, we have

(36) $2\sqrt{yPB\cdot
zPC}\cos\frac{\alpha}{2}=\sqrt{yz}\left(\sqrt{\frac{PB}{PC}}+\sqrt{\frac{PC}{PB}}\right)l_{a},$

and similarly,

(37) $2\sqrt{zPC\cdot
xPA}\cos\frac{\beta}{2}=\sqrt{zx}\left(\sqrt{\frac{PC}{PA}}+\sqrt{\frac{PA}{PC}}\right)l_{b},$
(38) $2\sqrt{xPA\cdot
yPB}\cos\frac{\gamma}{2}=\sqrt{xy}\left(\sqrt{\frac{PA}{PB}}+\sqrt{\frac{PB}{PA}}\right)l_{c}.$

Since $\frac{\alpha}{2}+\frac{\beta}{2}+\frac{\gamma}{2}=\pi$, using
Wolstenholme’s inequality [4], we get

(39) $2\sqrt{PB\cdot PC}\cos\frac{\alpha}{2}+2\sqrt{zPC\cdot
xPA}\cos\frac{\beta}{2}+\\\ +2\sqrt{xPA\cdot yPB}\cos\frac{\gamma}{2}\leq
xPA+yPB+zPC.$

From (36), (37), (38), and (39), we obtain

(40)
$xPA+yPB+zPC\geq\sqrt{yz}\left(\sqrt{\frac{PB}{PC}}+\sqrt{\frac{PC}{PB}}\right)l_{a}+\\\
+\sqrt{zx}\left(\sqrt{\frac{PC}{PA}}+\sqrt{\frac{PA}{PC}}\right)l_{b}+\sqrt{xy}\left(\sqrt{\frac{PA}{PB}}+\sqrt{\frac{PB}{PA}}\right)l_{c}.$

Adding two sides of (40) with $xu^{3}l_{a}+yv^{3}l_{b}+zw^{3}l_{c}$, we get

(41) $x(PA+u^{3}l_{a})+y(PB+v^{3}l_{b})+z(PC+w^{3}l_{c})\geq\\\
\left(\sqrt{yz}\sqrt{\frac{PB}{PC}}+\sqrt{yz}\sqrt{\frac{PC}{PB}}+xu^{3}\right)l_{a}+\left(\sqrt{zx}\sqrt{\frac{PC}{PA}}+\sqrt{zx}\sqrt{\frac{PA}{PC}}+yv^{3}\right)l_{b}+\\\
+\left(\sqrt{xy}\sqrt{\frac{PA}{PB}}+\sqrt{xy}\sqrt{\frac{PB}{PA}}+zw^{3}\right)l_{c}.$

Since $xyz=1$, using arithmetic mean-geometric mean inequality for three
positive numbers, we have

(42) $\sqrt{yz}\sqrt{\frac{PB}{PC}}+\sqrt{yz}\sqrt{\frac{PC}{PB}}+xu^{3}\geq
3\sqrt[3]{\sqrt{yz}\sqrt{\frac{PB}{PC}}\cdot\sqrt{yz}\sqrt{\frac{PC}{PB}}\cdot
xu^{3}}=3u,$

and similarly,

(43) $\sqrt{zx}\sqrt{\frac{PC}{PA}}+\sqrt{zx}\sqrt{\frac{PA}{PC}}+y\geq 3v,$
(44) $\sqrt{xy}\sqrt{\frac{PA}{PB}}+\sqrt{xy}\sqrt{\frac{PB}{PA}}+z\geq 3w.$

Combining (41), (42), (43), and (44), we have

(45) $x(PA+u^{3}l_{a})+y(PB+v^{3}l_{b})+z(PC+w^{3}l_{c})\geq
3(ul_{a}+vl_{b}+wl_{c}),$

with equality holds if and only if

(46) $x=y=z=u=v=w=1,$

triangle $ABC$ is equilateral and $P$ is its center. This completes the proof
of Theorem 3. ∎

######  Remark.

Similarly, we have two particular cases as new weighted versions of Barrow’s
inequality. In Theorem 3, let $u=v=w=1$ then

(47) $x(PA+l_{a})+y(PB+l_{b})+z(PC+l_{c})\geq 3(l_{a}+l_{b}+l_{c}),$

let $x=y=z=1$ then

(48) $PA+PB+PC\geq(3u-u^{3})l_{a}+(3v-v^{3})l_{b}+(3w-w^{3})l_{c}.$

##  3\. Some applications

In this section, we apply Theorem 1, Theorem 2, and Theorem 3 to get some
strengthened versions of Erdős-Mordell inequalities and its variations.

######  Theorem 4.

Let $u$, $v$, and $w$ be positive real numbers such that $uvw=1$. Let $P$ be
an interior point of the triangle $ABC$. Denote by $d_{a}$, $d_{b}$, and
$d_{c}$ the distances of $P$ from sides $AB$, $BC$, and $CA$, respectively.
Then

(49)
$(PA+u^{3}d_{a})(PB+v^{3}d_{b})(PC+w^{3}d_{c})\geq(ud_{a}+vd_{b}+wd_{c})^{3}.$

Equality holds if and only if $u=v=w=1$, triangle $ABC$ is equilateral and $P$
is its center.

###### Proof.

From Theorem 1, let

(50)
$x=\frac{\sqrt[3]{(PA+u^{3}d_{a})(PB+v^{3}d_{b})(PC+2^{3}d_{c})}}{PA+u^{3}d_{a}},$
(51)
$y=\frac{\sqrt[3]{(PA+u^{3}d_{a})(PB+v^{3}d_{b})(PC+w^{3}d_{c})}}{PB+v^{3}d_{b}},$

and

(52)
$z=\frac{\sqrt[3]{(PA+u^{3}d_{a})(PB+v^{3}d_{b})(PC+w^{3}d_{c})}}{PC+w^{3}d_{c}}.$

It is easily seen that $x$, $y$, and $z$ are positive numbers satisfied the
condition $xyz=1$, we get

(53) $3\sqrt[3]{(PA+u^{3}d_{a})(PB+v^{3}d_{b})(PC+w^{3}d_{c})}\geq
3(ud_{a}+vd_{b}+wd_{c})$

or

(54)
$(PA+u^{3}d_{a})(PB+v^{3}d_{b})(PC+w^{3}d_{c})\geq(ud_{a}+vd_{b}+wd_{c})^{3}.$

Equality holds if and only if $u=v=w=1$, triangle $ABC$ is equilateral and $P$
is its center. This completes the proof of Theorem 4. ∎

######  Remark.

From Theorem 4, let $u=v=w=1$, we get a strengthened version of Erdős-Mordell
inequality

(55) $(PA+d_{a})(PB+d_{b})(PC+d_{c})\geq(d_{a}+d_{b}+d_{c})^{3}$

because

(56) $(d_{a}+d_{b}+d_{c})^{3}\leq(PA+d_{a})(PB+d_{b})(PC+d_{c})\leq\\\
\leq\left(\frac{PA+d_{a}+PB+d_{b}+PC+d_{c}}{3}\right)^{3}.$

which is equivalent to

(57) $3(d_{a}+d_{b}+d_{c})\leq PA+d_{a}+PB+d_{b}+PC+d_{c}$

or

(58) $2(d_{a}+d_{b}+d_{c})\leq PA+PB+PC.$

Notice that the inequality (55) has appeared in [22] but with a different
proof. Theorem 4 is a generalization of inequality (55).

Likewise, we have the similar application of Theorem 2 and obtain a
strengthened version of Dao-Nguyen-Pham’s inequality.

######  Theorem 5.

Let $u$, $v$, and $w$ be positive real numbers such that $uvw=1$. Let $P$ be
an interior point of the triangle $ABC$ with circumcircle $(\omega)$. Denote
by $R_{A}$, $R_{B}$, and $R_{C}$ the distances of $P$ from the tangents to
$(\omega)$ at $A$, $B$, $C$ respectively, $d_{a}$, $d_{b}$, and $d_{c}$ the
distances of $P$ from the sides $AB$, $BC$, and $CA$, respectively. Then

(59)
$(R_{A}+u^{3}d_{a})(R_{B}+v^{3}d_{b})(R_{C}+w^{3}d_{c})\geq(ud_{a}+vd_{b}+wd_{c})^{3}.$

Equality holds if and only if $u=v=w=1$, $P$ coincides one of the vertices of
the triangle $ABC$.

From Theorem 5, let $u=v=w=1$, we get a strengthened version of Dao-Nguyen-
Pham’s inequality also it is strengthened version of inequality (55)

(60) $(R_{A}+d_{a})(R_{B}+d_{b})(R_{C}+d_{c})\geq(d_{a}+d_{b}+d_{c})^{3}.$

With the same idea, we get the following theorem as a particular case of
Theorem 3

######  Theorem 6.

Let $u$, $v$, and $w$ be positive real numbers such that $uvw=1$. Let $P$ be
an interior point of the triangle $ABC$. Denote by $l_{a}$, $l_{b}$, and
$l_{c}$ the lengths of the bisectors of $\angle BPC$, $\angle CPA$, and
$\angle APB$, respectively. Then

(61)
$(PA+u^{3}l_{a})(PB+v^{3}l_{b})(PC+w^{3}l_{c})\geq(ul_{a}+vl_{b}+wl_{c})^{3}.$

Equality holds if and only if $u=v=w=1$, triangle $ABC$ is equilateral and $P$
is its center.

From Theorem 6, let $u=v=w=1$, we obtain a strengthened version of Barrow’s
inequality

(62) $(PA+l_{a})(PB+l_{b})(PC+l_{c})\geq(l_{a}+l_{b}+l_{c})^{3}.$

Notice that we also have a strengthened version of the weighted Barrow’s
inequality from (40) by rewriting it as follows

######  Theorem 7 (Another strengthened version of weighted Barrow’s
inequality).

Let $x$, $y$, and $z$ be any three positive real numbers. Let $P$ be an
interior point of the triangle $ABC$. Denote by $l_{a}$, $l_{b}$, and $l_{c}$
be the lengths of the bisectors of $\angle BPC$, $\angle CPA$, and $\angle
APB$, respectively. Then

(63)
$xPA+yPB+zPC\geq\sqrt{yz}\left(\sqrt{\frac{PB}{PC}}+\sqrt{\frac{PC}{PB}}\right)l_{a}+\\\
+\sqrt{zx}\left(\sqrt{\frac{PC}{PA}}+\sqrt{\frac{PA}{PC}}\right)l_{b}+\sqrt{xy}\left(\sqrt{\frac{PA}{PB}}+\sqrt{\frac{PB}{PA}}\right)l_{c}.$

Equality holds if and only if $x=y=z$, triangle $ABC$ is equilateral and $P$
is its center.

We notice that, a form without weights of Theorem 7 has appeared in [18].
Using Theorem 7, we get another strengthened versions of Barrow’s inequality
as follows

######  Theorem 8 (Another strengthened version of Barrow’s inequality).

Let $P$ be an interior point of the triangle $ABC$. Denote by $l_{a}$,
$l_{b}$, and $l_{c}$ the lengths of the bisectors of $\angle BPC$, $\angle
CPA$, and $\angle APB$, respectively. Then

(64) $\displaystyle\frac{PA+PB+PC}{2}$ (65)
$\displaystyle\geq\frac{PA^{2}(PB+PC)^{2}+PB^{2}(PC+PA)^{2}+PC^{2}(PA+PB)^{2}}{(PB+PC)(PC+PA)(PA+PB)}$
(66) $\displaystyle\geq l_{a}+l_{b}+l_{c}.$

Equality at (65) holds if and only if $PA=PB=PC$; equality at (66) holds if
and only if triangle $ABC$ is equilateral and $P$ is its center.

###### Proof.

Let $PA=p$, $PB=q$, and $PC=r$ then the first inequality comes from the
following algebraic identity and inequality

(67) $(p+q+r)(p+q)(q+r)(r+p)-2p^{2}(q+r)^{2}-2q^{2}(r+p)^{2}-\\\
-2r^{2}(p+q)^{2}=pq(p-q)^{2}+qr(q-r)^{2}+rp(r-p)^{2}\geq 0.$

Equality at (67) holds if and only if $p=q=r$ or $PA=PB=PC$.

For the second inequality, from Theorem 7, let

(68) $x=\left(\sqrt{\frac{PB}{PC}}+\sqrt{\frac{PC}{PB}}\right)^{2},\
y=\left(\sqrt{\frac{PC}{PA}}+\sqrt{\frac{PA}{PC}}\right)^{2},$

and

(69) $z=\left(\sqrt{\frac{PA}{PB}}+\sqrt{\frac{PB}{PA}}\right)^{2}.$

We obtain

(70) $\frac{(PB+PC)^{2}}{PB\cdot PC}PA+\frac{(PC+PA)^{2}}{PC\cdot
PA}PB+\frac{(PA+PB)^{2}}{PA\cdot PB}PC\geq\\\
\left(\sqrt{\frac{PB}{PC}}+\sqrt{\frac{PC}{PB}}\right)\left(\sqrt{\frac{PC}{PA}}+\sqrt{\frac{PA}{PC}}\right)\left(\sqrt{\frac{PA}{PB}}+\sqrt{\frac{PB}{PA}}\right)\left(l_{a}+l_{b}+l_{c}\right),$

which is equivalent to

(71)
$\frac{PA^{2}(PB+PC)^{2}+PB^{2}(PC+PA)^{2}+PC^{2}(PA+PB)^{2}}{(PB+PC)(PC+PA)(PA+PB)}\geq
l_{a}+l_{b}+l_{c}.$

Equality holds if and only if triangle $ABC$ is equilateral and $P$ is its
center. This completes the proof of Theorem 8. ∎

Finally, we note that Theorem 1 can be considered as generalization of
”weighted Erdős-Mordell inequality” which is given by Dar and Gueron [3] by
setting $x=\frac{1}{u^{2}}$, $y=\frac{1}{v^{2}}$, and $z=\frac{1}{w^{2}}$
($xyz=1$ since $uvw=1$), we obtain

######  Theorem 9 (An equivalent form of weighted Erdős-Mordell inequality in
[3]).

Let $u$, $v$, and $w$ be positive real numbers such that $uvw=1$. Let $P$ be
an interior point of the triangle $ABC$. Denote by $d_{a}$, $d_{b}$, and
$d_{c}$ the distances of $P$ from the sides $AB$, $BC$, and $CA$,
respectively. Then

(72) $\frac{PA}{u^{2}}+\frac{PB}{v^{2}}+\frac{PC}{w^{2}}\geq
2(ud_{a}+vd_{b}+wd_{c}).$

Equality holds if and only if $u=v=w=1$, triangle $ABC$ is equilateral and $P$
is its center.

##  4\. Conclusion

In this paper, we give a new weighted version of Erdős-Mordell inequality, and
also new weighted versions for Erdős-Mordell type inequalities with proofs and
some applications as the strengthened version of Erdős-Mordell inequality and
Erdős-Mordell type inequalities.

## References

  * [1] Erdős, P., Problem 3740, Amer Math. Monthly, 42(1935), pp. 396.
  * [2] Mordell, L. J. and Barrow, D. F., Solution 3740, Amer. Math. Monthly, 44(1937), pp. 252–254.
  * [3] Dar, S. and Gueron, S., A Weighted Erdős-Mordell Inequality, Amer. Math. Monthly, 108(2001), pp. 165–168.
  * [4] Wolstenholme J., A Book of Mathematical Problems, Cambridge, London, 1867.
  * [5] Bottema, O., Geometric inequalities, Groningen: Wolters-Noordhoff, The Netherlands, 1969.
  * [6] Mitrinović, D. S., Pecarić, J. E., Voloneć, V., Recent Advances in Geometric Inequalities, Dordrecht-Boston-London: Kluwer Academic Publishers, 1989
  * [7] Avez, A., A short proof of a theorem of Erdős and Mordell, Amer. Math. Monthly, 100(1993), pp. 60–62.
  * [8] Abi-Khuzam, F. F., A trigonometric inequality and its geometric applications. Math. Inequal. Appl., 3(2000), pp. 437–442.
  * [9] Bankoff, L., An elementary proof of the Erdős-Mordell theorem. Amer. Math. Monthly, 65(1958), pp. 521.
  * [10] Kazarinoff, D. K., A simple proof of the Erdős-Mordell inequality for triangles, Mich. Math. J., 4(1957), pp. 97–98.
  * [11] Komornik, V., A short proof of the Erdős-Mordell theorem, Amer. Math. Monthly, 104(1997), pp. 57–60.
  * [12] Ozeki, N., On Paul Erdős-Mordell inequality for the triangle, J. Coll. Arts Sci. Chiba Univ. A, 2(1957), pp. 247–250.
  * [13] Oppenheim, A., The Erdős-Mordell inequality and other inequalities for a triangle, Amer. Math. Monthly, 68(1961), pp. 226–230.
  * [14] Oppenheim, A., New inequalities for a triangle and an internal point, Ann. Mat. Pura Appl., 53(1961), pp. 157–163.
  * [15] Pech, P., Erdős-Mordell inequality for space n-gons, Math. Pannon., 5(1994), pp. 3–6.
  * [16] Alsina, C., Nelsen, R.B., A visual proof of the Erdős-Mordell inequality, Forum Geom., 7(2007), pp. 99–102.
  * [17] Dao, T. O., Nguyen, T. D., Pham, N. M., A strengthened version of the Erdős-Mordell inequality, Forum Geom., 16(2016), pp. 317–321.
  * [18] Malešević, B., and Petrović, M., Barrow’s inequality and signed angle bisectors, J. Math. Inequal., 8(2014), pp. 537–544.
  * [19] Liu J., A Geometric inequality with one parameter for a point in the plane of a triangles, J. Math. Inequal., 1(2014), pp. 91–106.
  * [20] Liu J., Sharpened versions of the Erdős-Mordell inequality, J. Inequal. Appl., 206(2015), 12.pp.
  * [21] Liu J., Refinements of the Erdős-Mordell inequality, Barrow’s inequality, and Oppenheim’s inequality, J. Inequal. Appl., 9(2016), 18.pp.
  * [22] Liu J., New refinements of the Erdős-Mordell inequality, J. Math. Inequal., 12(2018), pp. 63–75.
  * [23] Liu J., Two New Weighted Erdős–Mordell Type Inequalities, Discrete Comput. Geom., 59(2018), pp. 707–724.


#  A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image
Generation

Eyal Segalis
Google Research
<EMAIL_ADDRESS>Dani Valevski
Google Research
<EMAIL_ADDRESS>Danny Lumen
Google Research
<EMAIL_ADDRESS>Yossi Matias
Google Research
<EMAIL_ADDRESS>Yaniv Leviathan
Google Research
<EMAIL_ADDRESS>
(September 2023)

Figure 1: Examples of non-cherrypicked generations from the base Stable
Diffusion model (left) and our model trained on a recaptioned dataset (right),
on the same set of random seeds.

##  1 Abstract

Text-to-image diffusion models achieved a remarkable leap in capabilities over
the last few years, enabling high-quality and diverse synthesis of images from
a textual prompt. However, even the most advanced models often struggle to
precisely follow all of the directions in their prompts. The vast majority of
these models are trained on datasets consisting of (image, caption) pairs
where the images often come from the web, and the captions are their HTML
alternate text. A notable example is the LAION dataset, used by Stable
Diffusion and other models. In this work we observe that these captions are
often of low quality, and argue that this significantly affects the model’s
capability to understand nuanced semantics in the textual prompts. We show
that by relabeling the corpus with a specialized automatic captioning model
and training a text-to-image model on the recaptioned dataset, the model
benefits substantially across the board. First, in overall image quality: e.g.
FID 14.84 vs. the baseline of 17.87, and 64.3% improvement in faithful image
generation according to human evaluation. Second, in semantic alignment, e.g.
semantic object accuracy 84.34 vs. 78.90, counting alignment errors 1.32 vs.
1.44 and positional alignment 62.42 vs. 57.60. We analyze various ways to
relabel the corpus and provide evidence that this technique, which we call
RECAP, both reduces the train-inference discrepancy and provides the model
with more information per example, increasing sample efficiency and allowing
the model to better understand the relations between captions and images.

##  2 Introduction

In recent years, text-to-image (T2I) generation models such as Imagen [1],
Muse [2], Dall-E [3], Dall-E 2 [4], Parti [5], and Stable Diffusion [6] have
undergone significant advancements. This progress has enabled the generation
of remarkably high-quality and diverse images by conditioning on textual
inputs. However, while revolutionary, even modern state-of-the-art text-to-
image models may fail to generate images that fully convey the semantics and
nuances from the given textual prompts. Failure modes include: missing one or
more subjects from the input prompt [5, 7]; incorrect binding of entities and
modifiers [5, 8, 7]; and incorrect placement and spatial composition of
entities [5, 9, 10].

In this work we first observe that open-web datasets used to train open text-
to-image models suffer from significant issues. For example, the captions in
the LAION [11] dataset, used to train Stable Diffusion, come from alt HTML
tags (Alttext). According to W3C’s web content accessibility
guidelines111https://www.w3.org/TR/2016/NOTE-WCAG20-TECHS-20161007/H37, the
alt attribute is used to convey the meaning and intent of the image, and not
necessarily being a literal description of the image itself. Indeed, we
observe, that often the Alttext describes only a narrow aspect of the image,
neglecting significant visual details. For example, an image of a person can
have as Alttext the name of the person and the name of the photographer, but
not a description of their appearance, their clothes, their position, or the
background. Also, sometimes Alttext tags contain inaccuracies, mistakes and
out of context information. See Fig. 4 for examples.

We further observe that while trained mainly on similar datasets of open
(image, caption) pairs, recent automatic captioning systems, such as PaLI
[12], produce highly accurate captions. See examples in Fig. 3. This may be
due to the fact that the inverse problem of image-to-text (I2T) is easier, or
to the fact that these captioning models are larger, are trained longer than
the T2I models, or leverage large pre-trained language components.

With these observations, we suggest a new method for horizontally improving
T2I models by training them on improved captions, auto-generated by a custom
I2T model. We call our method RECAP, and show that applying it to Stable
Diffusion results in a model that is better than the baseline across the
board, with a battery of standard metrics substantially improving, e.g. FID
$17.87{\rightarrow}14.84$, as well as in human evaluation of successful image
generation $29.25\%\rightarrow 48.06\%$ (see Section 5).

In Sec. 4 we provide the details of our method, RECAP. Sec. 5 discusses our
results and shows the horizontal improvements both in image quality as well as
in semantic fidelity to the prompts. In Sec. 6 we analyze the issues with the
original captions, demonstrate that the improvements are indeed due to the new
captions, and that they arise as a result of both minimizing the train-test
skew as well as increasing sample efficiency.

To summarize, our main contributions are:

  * •

A new method we call RECAP, that leverages automatic captioning to improve the
quality of a text-to-image model in a substantial way horizontally, both in
fidelity and semantics, measured on a set of 7 standard metrics as well as
with human evaluations.

  * •

An analysis showing how Alttext captions used by current training methods
suffer from train-inference skew and lack in semantic details, resulting in
text-to-image models that often fail in fidelity and semantic alignment, and
how different captions mitigate both issues.

Figure 2: Schematic diagram of our method RECAP. In steps (1a) and (1b) we
fine-tune an image-to-text captioning model on a small set of detailed human
captions. In step (2) we use this fine-tuned model to recaption the images in
the training dataset of a text-to-image model, and with this dataset, in step
(3) we train an image generation model with the recaptioned dataset.

##  3 Related Work

Text-to-Image Models. Deep generative models for image generation from text
have shown notable progress in recent years, transitioning from using GAN-
based methods [13] to using autoregressive transformers [3, 5, 2] and
diffusion models [4, 14, 1]. An important area of enhancement is in improving
a model’s capability to align with the input text effectively. Methods
condition the image on the output of a pre-trained text embedder, usually CLIP
[15]. Imagen [1] shows that using a strong T5 [16] encoder significantly
improves text-image alignment. Parti [5] uses a SimVLM model [17] to annotate
some of the training set images.

Image captioning. Image captioning is a fundamental problem in computer
graphics. Recent models [18, 19, 20, 12, 21] have made significant progress on
this task, thanks to high scale training data and utilizing pre-trained image
and text models, allowing them to address a variety of multi-modal tasks. In
this work we fine-tune PaLI [12] to recaption the training dataset of a text-
to-image model.

Synthetic multimodal datasets. Some concurrent works augment multimodal
dataset via automatic means to improve the capabilities of image captioning
and embedding models. Nguyen et al. [22] show that training CLIP on data with
generated captions improves its performance. Li et al. [23] show how to
improve the BLIP model by iteratively removing captions with high loss,
replacing them with captions from the previous epoch or generating a new image
for them using Stable Diffusion. Ma et al. [24] generate a synthetic text-to-
image model using Stable Diffusion and then uses it to train a captioning
algorithm.

Improving text-alignment of diffusion models. An important line of work
attempts to rectify text-to-image alignment issues in diffusion models. This
is usually done by lexical analysis of the input prompt modifying the
attention maps throughout sampling. Chefer et al. [7] try to prevent the
diffusion models from ignoring certain tokens by guiding the sample process to
reweigh the token attention maps. Rassin et al. [8] analyze the prompt to
identify modifiers, and then uses a sample-time guidance to bind their
attention maps to those of their entities. Wu et al. [9] isolates phrases that
describe individual entities and attends to them. Phung et al. [10] allows
placing objects in certain region by boosting their attention in those
regions.

Concurrently with our work, Dall-E 3 [25] proposes to use an automatic
captioning system to regenerate the captions used to train a T2I model. Our
work uses an open model (Stable Diffusion) and we provide more details and
focus more on analysis and evaluation, but otherwise the main ideas are very
similar.

##  4 Method

Our method, RECAP consists of 3 steps: (1) fine-tune an automatic captioning
system to produce desirable labels; (2) relabel the images from our text-to-
image training set with this automatic recaptioning system; and (3) train the
text-to-image model on the dataset consisting of the images and new captions.
Fig. 2 visualizes the overall method.

###  4.1 Training Dataset

We selected a subset of 10M photos from _LAION-2B-en improved Aesthetics_
dataset. We followed the data filters used to train Stable Diffusion versions
1.2-1.4222https://huggingface.co/CompVis/stable-diffusion, as follows:
$aesthetics\ score\geq 5.0,\ pwatermark<0.5,\ nsfw$ is $UNLIKELY$ and both
$height$ & $width\geq 512$. We note that this filtering operation might
amplify biases in the dataset [26]. We further excluded an arbitrary subset of
10K photos from the training set to be used as an internal validation set.

###  4.2 Captioning Model

We used a pre-trained I2T captioning model (PaLI [12]). As the model outputs
are relatively terse and lack in detail, we first collected a small set of 100
manual captions from human raters and fine-tuned the captioning model on that
set.

In order to experiment with the effect of different captioning distributions,
the raters were asked to provide two types of captions. First, a detailed
caption for each image, with these instructions: “Describe what you see in
each image using 1-2 detailed sentences”. Note that the instructions limited
the length to only 1-2 detailed sentences at most, due to CLIP’s (the
downstream text encoder) context size of only 77 tokens, which longer captions
exceeded. Second, we collected a short and less detailed caption for each
image with this instruction: “Describe what you see in each image using a
single short sentence”. We did not iterate on the quality of these manual
captions. Fig. 3 provides example captions given by the human raters, as well
as those produced by the non fine-tuned PaLI model.

With this small dataset, we fine-tuned PaLI for 300 steps, using a learning
rate of 4e-5, dropout rate of 0.1 and a batch size of 64, mixing 50% short
captions and 50% long captions for multiple copies of the 100 images, using a
different fixed conditioning prefix for the short vs. long captions. When
generating captions from the fine-tuned model, we use the terms _RECAP Short_
and _RECAP Long_ to refer to generations conditioned on the short and long
prefixes respectively. Example outputs, as well as a comparison to the
original Alttext captions can be found in Fig. 4. See Appendix C for
additional dozen random examples of the generated RECAP captions.

Overall, the captions produced by this bespoke model improve both of the
issues above - their distribution better matches inference time prompts and
they contain much more detail to improve sample efficiency (see Sec. 6.1.1).

Figure 3: Examples of captions given by human raters, and the automatically
generated caption from the non fine-tuned PaLI model. Photos taken from LAION.

Figure 4: Examples of captions generated by the RECAP model conditioned on the
short or long prefixes, the original PaLI model, and the original Alttext
captions. Photos taken from LAION.

###  4.3 Image Generation Model

Next, we fine-tuned Stable Diffusion v1.4 for an additional 250k333When
further fine-tuning the model for 1M steps, we observe better visual results
along with diminishing returns in the auto computed metrics. All results in
the paper are for 250K steps except Fig. 1 and the human evaluation, where we
used a model fine-tuned for 1M steps. steps with a learning rate of 1e-5,
batch size of 512, and prompt dropout rate of 0.1444Prompts with more than 77
tokens (CLIP’s upper limit) were dropped. This happened for ¡1% of the data in
each training set.. We fine-tuned the model training both UNet and CLIP
weights and used a 50%-50% mixture of RECAP Short and RECAP Long captions
(RECAP Mix) as this performed best. The results in the main text are all from
this configuration.

RECAP is independent of the sampling method, so we can use any sampling method
with it. That said, for all of the experiments in this work, we used DDIM
sampling with 50 inference steps and a guidance scale of 7.5.

##  5 Results

We compare the RECAP model to two models: Baseline and Alttext. Baseline is
the Stable Diffusion v1.4 model. Alttext is the baseline model fine-tuned for
the same number of steps and on the same set of images as RECAP, but with the
original captions (Alttext) instead of the RECAP captions. The Alttext model
resolves contamination concerns, as it includes the exact set of images. In
all the comparisons we used the same random seeds across models.

We compare the models using a variety of automated metrics, human evaluation
and a qualitative evaluation of examples. We observe improvements in all
metrics (see Tabs. 1 and 2).

###  5.1 Automated metrics

We evaluated the performance and semantic capabilities of the RECAP model
using a battery of metrics suggested by [27] (using their publicly available
code) on the MS-COCO validation dataset. Tab. 1 contains a summary of the
results.

To assess overall generation quality we use the standard FID metric and
observe that images generated with the RECAP model have a significantly better
score ($17.87\rightarrow 14.84$).

In addition, we assess the semantic capabilities of our model: to check that
the model generates faithfully the requested objects we use Semantic Object
Accuracy ($80.80\rightarrow 86.17$), to check the number of generated objects
we use Counting Alignment errors ($1.44\rightarrow 1.32$), to check that the
locations of objects are correct we use Positional Alignment
($57.60\rightarrow 62.42$), and finally to check the overall adherence to the
prompt we measure Clip score ($92.78\rightarrow 93.80$).

In all the metrics, we see no improvement in the Alttext model compared to the
baseline, proving the improvements stem from the captions themselves and not
from the additional training.

Note that following [27], we omit the IS and O-IS metrics, as all Stable
Diffusion models yield a higher score than real images. As noted by [27], IS
is better suited for datasets with a single object and does not perform well
for the MS-COCO dataset, containing multiple objects per each image.

Additional details on the automated metrics are in Secs. A.1, A.2, A.3 and
A.4.

###  5.2 Human evaluation

For complementary evaluation of model performance, we used human raters.
Results are summarized in Tab. 2.

Raters were asked to select images generated from each model, only if they
successfully follow a given prompt. We evaluated once on 200 random prompts
from the MS-COCO validation set, and separately on the challenging DrawBench
dataset [1]. We presented four images (from different seeds) for each prompt,
using the same seeds across models.

We calculated two metrics: percentage of successful image generation across
all prompts and seeds (i.e. given a prompt and a seed, the chance of a
successful image generation); and percentage of at least one successful image
generation for a given prompt, out of four seeds (i.e. given a prompt, the
chance of successfully generating an image for it, in four attempts). We see a
relative 64.3% improvement in successful image generation on MS-COCO, and a
41.7% improvement on DrawBench. We also see a relative improvement of 42.1% in
successful prompt generation on MS-COCO and 37.5% improvement on DrawBench.
The Alttext model showed minor improvement on the MS-COCO dataset (12%-13%)
and did not improve the DrawBench dataset. Further details can be found in
Sec. A.5.

| FID$\downarrow$ | O-FID$\downarrow$ | SOA-C$\uparrow$ | SOA-I$\uparrow$ | CA$\downarrow$ | PA$\uparrow$ | RP$\uparrow$
---|---|---|---|---|---|---|---
Baseline | 17.87 | 8.19 | 78.90 | 80.80 | 1.44 | 57.60 | 92.78
Alttext | 17.53 | 8.90 | 78.99 | 80.85 | 1.47 | 57.40 | 91.32
RECAP | 14.84 | 6.23 | 84.34 | 86.17 | 1.32 | 62.42 | 93.80
Real Images | 2.62 | 0.00 | 90.02 | 91.19 | 1.05 | 100.0 | 83.54
Table 1: Results for the automated metrics for RECAP model vs. baseline and Alttext models. RECAP model improves across all metrics. | MS-COCO | MS-COCO | DrawBench | DrawBench
---|---|---|---|---
| Successful Images | Successful Prompts | Successful Images | Successful Prompts
Baseline | 29.3% | 53.5% | 15.6% | 33.1%
Alttext | 33.4% | 60.0% | 13.2% | 33.8%
RECAP | 48.1% | 76.0% | 22.1% | 45.5%
Table 2: Human evaluation results comparing RECAP, Alttext and Baseline
models, on two benchmarks (MS-COCO and DrawBench) for two metrics: percentage
of images generated that fully follow the prompt, and percentage of prompts
with at least one generated image (out of four seeds) that fully followed the
prompt.

###  5.3 Qualitative Results

Fig. 1 provides representative examples where RECAP outperforms the base
Stable Diffusion model (and sometimes also larger models, see Appendix B).

Generally, we observe that RECAP can better interpret relations between
entities. Prepositions like "near", "through", "made off" are often ignored by
the base Stable Diffusion model. The RECAP model applies the prepositions
correctly, while the base model often resorts to the most common relation
between the entities (based on the training data distribution). For example,
in the prompt "_A pizza near a pineapple_ ", the base model places the
pineapple on the pizza, as the probable relation between the two, while RECAP
generates a pineapple near it, as requested.

RECAP also better handles cases where different modifiers are applied to
multiple entities (e.g. "_A red bench and a yellow clock_ "). The base model
will treat the sentence as a bag of words, applying all modifiers to all
entities or ignore some of them. RECAP is also able to interpret complex
modifiers like anaphors. For examples, in the prompt "_Two flowers, one is
blue and the other one is green_ " it understands that "one" refers to a
flower.

##  6 Analysis

We hypothesize that the underlying improvement in image generation quality, as
measured in the results above, stems from two improvements in the training
captions: (1) reducing the discrepancy between the train and inference
prompts, and (2) giving the model more textual information per image, thus
improving the training sample efficiency. Below we provide an ablation
analysis showing that RECAP captions achieve both properties, and that the
resulting model benefits from both.

###  6.1 Comparing Different Caption Types

####  6.1.1 Generated Captions

Tab. 3 compares the generated captions in the training set, to the caption in
the MS-COCO validation set, on a variety of language metrics. We used MS-COCO
to represent the prompts we expect to see in inference time. We observe that
our generated captions are closer in distribution to MS-COCO in several
senses.

First, we compare them by using standard language metrics from the _textstat_
python package555https://pypi.org/project/textstat calculated over the 10M
examples in each train dataset. _Flesch Reading Ease_ score is a standard
measure for how difficult it is to read a text in a given language. We observe
that RECAP is able to generate easy to read sentences, while the original
Alttext is often difficult to read. Similarly, the _text_standard_ score is a
consensus score based on a battery of metrics to estimate the smallest grade
of a native speaking reader to be able to read the text. RECAP generates texts
which a 4th grader (the lowest possible) can read, while the original Alttext
is estimated to require 8th graders to fully understand.

To measure more directly the reduction in train-inference skew, we compare the
distribution of the embedded texts. Since Stable Diffusion is using CLIP to
encode the text, we calculate the Fréchet distance between the CLIP embeddings
of the various datasets666Similarly to the FID metric used to compare between
image distributions, we take the Fréchet distance of the CLIP embedding
distributions, modeled as a Gaussian.. Results are summarized in Tab. 3. As
suspected, RECAP generated captions are closer in distribution to the MS-COCO
captions than the Alttext captions. Furthermore, the RECAP Short captions are
closer to MS-COCO than RECAP Long captions. These results are in line with the
improvement in FID of the images from models trained with the corresponding
datasets, as detailed in Sec. 6.1.2.

Next, we measured how well the automatically generated captions describe the
images, using a human evaluation of 100 random images, asking raters to score
each caption considering both faithfulness to the image, and completeness in
describing the image, on a scale of 1-5. RECAP generated captions were rated
as more faithful and complete (with an average score of 3.58 for RECAP Short
and 4.3 for RECAP Long) than Alttext captions (scoring on average 2.9). This
supports our hypothesis that automated captions make training more efficient
by providing more textual information.

| Alttext | RECAP Long | RECAP Short | MS-COCO
---|---|---|---|---
# sentences | 1.15 | 2.15 | 1.00 | 1.00
# words | 11.38 | 21.29 | 5.89 | 10.45
# letters | 61.08 | 87.96 | 24.39 | 41.67
Flesch Reading Ease score | 45.71 (= Difficult) | 88.35 (= Easy) | 86.61 (= Easy) | 86.93 (= Easy)
_text_standard_ score | 8.78 (8th grader) | 4.50 (4th grader) | 3.11 (4th grader) | 4.59 (4th grader)
Distance to MS-COCO | 0.45 | 0.24 | 0.18 | 0.00
Table 3: Comparison of average statistics on the captions generated by RECAP
vs. the original Alttext over the 10m examples in the training set, and the
10k examples in the MS-COCO validation set. Distance to MS-COCO is the Fréchet
distance of the CLIP embeddings of the RECAP caption sets (calculated on 250k
samples) vs. the MS-COCO validation set. Note that since the embeddings are
normalized, the distance in between 0-1.

####  6.1.2 Generated Images

Next, we compared the results of fine-tuning Stable Diffusion on our two
caption types, RECAP Long and RECAP Short. The results are summarized in Tab.
4. We observe that training on RECAP Short captions achieves better FID
scores, and faster, but with little semantic improvement, while the RECAP Long
captions exhibit significant semantic improvement (see representative metrics
in Fig. 5). Mixing the caption sets (RECAP Mix) provides the best of both
worlds.

Note that the improvement in FID on the MS-COCO validation set, is correlated
to the improvement in the Fréchet distance of the generated captions, as
detailed in Sec. 6.1.1. This indicates that the semantic improvement in the
RECAP Long model stems from improved training sample efficiency, and not only
from reducing the train-inference skew.

| FID$\downarrow$ | O-FID$\downarrow$ | SOA-C$\uparrow$ | SOA-I$\uparrow$ | CA$\downarrow$ | PA$\uparrow$ | RP$\uparrow$
---|---|---|---|---|---|---|---
Baseline | 17.87 | 8.19 | 78.90 | 80.80 | 1.44 | 57.60 | 92.78
Alttext | 17.53 | 8.90 | 78.99 | 80.85 | 1.47 | 57.40 | 91.32
RECAP Short | 14.85 | 5.81 | 79.81 | 81.95 | 1.43 | 60.00 | 93.30
RECAP Long | 15.61 | 7.48 | 84.59 | 86.16 | 1.32 | 63.27 | 93.26
RECAP Mix | 14.84 | 6.23 | 84.34 | 86.17 | 1.32 | 62.42 | 93.80
Real Images | 2.62 | 0.00 | 90.02 | 91.19 | 1.05 | 100.0 | 83.54
Table 4: Results for the automated metrics for RECAP models vs. baseline and
Alttext models, comparing models that trained on different sets of captions.

Figure 5: FID (left) and SOA (right) scores for various checkpoints of the
Alttext model vs. RECAP models, comparing models that trained on different
sets of captions. 0 fine-tuning steps is vanilla Stable Diffusion 1.4. Lower
FID is better. We see that RECAP Short achieves better FID and faster, but
with no semantic improvement. RECAP Long achieves some FID improvement with
significant semantic improvement, and RECAP Mix achieves both.

###  6.2 Training Different Model Weights

To further explore the contribution of the improved examples to each part of
the Stable Diffusion model, we compared training only the UNet weights, vs.
the CLIP weights, vs. both, using the same training and evaluation procedures.
For simplicity, we only report the results for RECAP Mix vs. Alttext and
baseline models.

The results are summarized in Tab. 5. Overall, as expected, training more
weights achieves better performance. Interestingly, training only CLIP weights
(which are $\backsim 12\%$ of the total weights) achieves better FID, with
less training steps, but with little semantic improvement. Training both CLIP
and UNet weights results in significantly higher improvement to semantic
scores than training only on one of them. See Sec. A.6 for more details.

We believe that training CLIP weights mainly reduces the skew in the
distribution of texts between the training set and the evaluation set, while
training the UNet weights mostly improves the alignment of the text to the
image.

| FID$\downarrow$ | O-FID$\downarrow$ | SOA-C$\uparrow$ | SOA-I$\uparrow$ | CA$\downarrow$ | PA$\uparrow$ | RP$\uparrow$
---|---|---|---|---|---|---|---
Baseline | 17.87 | 8.19 | 78.90 | 80.80 | 1.44 | 57.60 | 92.78
Alttext UNet | 16.83 | 8.20 | 76.90 | 78.72 | 1.50 | 56.05 | 91.89
Alttext CLIP | 17.04 | 8.32 | 79.08 | 81.10 | 1.52 | 58.72 | 91.34
Alttext UNet+CLIP | 17.53 | 8.90 | 78.99 | 80.85 | 1.47 | 57.40 | 91.32
RECAP Mix UNet | 15.49 | 6.47 | 82.15 | 84.04 | 1.40 | 59.77 | 92.76
RECAP Mix CLIP | 14.60 | 6.09 | 80.19 | 82.04 | 1.37 | 60.44 | 92.48
RECAP Mix UNet+CLIP | 14.84 | 6.23 | 84.34 | 86.17 | 1.32 | 62.42 | 93.80
Real Images | 2.62 | 0.00 | 90.02 | 91.19 | 1.05 | 100.0 | 83.54
Table 5: Results for the automated metrics for RECAP models vs. baseline and
Alttext models, comparing models that trained different set of weights.

##  7 Summary and Discussion

In this paper, we show how text-to-image models can be improved across the
board by training on synthetically generated captions. We performed an in-
depth analysis demonstrating that short descriptions which narrow the train-
inference gap are helpful, as are long and detailed descriptions that improve
sample efficiency despite being different from the inference set. We further
demonstrated that mixing these descriptions in the training set improves all
fronts simultaneously.

There are several interesting directions for future work. It would be
interesting to check if by tuning the captioning model to produce ample detail
in narrow areas, the same recipe can be used to improve semantic capabilities
in new domains (for example, could we create models that can accurately
generate hair styles, room designs, facial expressions, clothing, etc. based
on detailed descriptions?). Similarly, it is possible to use RECAP to train
T2I models in domains that lack textual captions altogether (e.g., a personal
photo album or screencaps of TV shows). We conducted initial experiments here,
and they show a lot of promise.

We experimented with fine-tuning a model with the RECAP captions, but it would
be interesting to compare that to a model that was pre-trained on the RECAP
captions from scratch. Relatedly, it would be interesting to further
experiment with different mixtures of the three caption types we have (RECAP
Short, RECAP Long, and Alttext). Even more generally, we could imagine
creating and mixing together several more flavors of recaptioning models. This
could allow us to circumvent the token limit (by training on the same image
multiple times with a different subset of the caption each time). Regardless
of token limit, it could be interesting to explore training on several shorter
captions per image instead of a single long one. It would also be interesting
to explore the effects of RECAP on larger models trained on larger datasets.

Finally, RECAP shows the importance of high quality datasets, and that it can
be improved with synthetic data, we hope that this provides yet another
encouragement to apply such techniques even beyond the T2I domain.

##  8 Acknowledgments

We would like to extend our gratitude to Eyal Molad, Matan Kalman, Jason
Baldridge, and the Theta Labs team at Google, for great reviews, suggestions,
and support to this paper.

## References

  * [1] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding, 2022.
  * [2] Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan.  Muse: Text-to-image generation via masked generative transformers, 2023\.
  * [3] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation, 2021.
  * [4] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents, 2022\.
  * [5] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu.  Scaling autoregressive models for content-rich text-to-image generation, 2022.
  * [6] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models, 2022.
  * [7] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or.  Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models, 2023.
  * [8] Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoav Goldberg, and Gal Chechik.  Linguistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment, 2023.
  * [9] Qiucheng Wu, Yujian Liu, Handong Zhao, Trung Bui, Zhe Lin, Yang Zhang, and Shiyu Chang.  Harnessing the spatial-temporal attention of diffusion models for high-fidelity text-to-image synthesis, 2023.
  * [10] Quynh Phung, Songwei Ge, and Jia-Bin Huang.  Grounded text-to-image synthesis with attention refocusing, 2023.
  * [11] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev.  Laion-5b: An open large-scale dataset for training next generation image-text models, 2022.
  * [12] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut.  Pali: A jointly-scaled multilingual language-image model, 2023.
  * [13] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks, 2014.
  * [14] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models, 2021.
  * [15] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision, 2021.
  * [16] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  Journal of Machine Learning Research, 21(140):1–67, 2020.
  * [17] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao.  Simvlm: Simple visual language model pretraining with weak supervision, 2022.
  * [18] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023.
  * [19] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan.  Flamingo: a visual language model for few-shot learning, 2022.
  * [20] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  Coca: Contrastive captioners are image-text foundation models, 2022.
  * [21] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning, 2023.
  * [22] Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco, Sewoong Oh, and Ludwig Schmidt.  Improving multimodal datasets with image captioning, 2023.
  * [23] Wenyan Li, Jonas F. Lotz, Chen Qiu, and Desmond Elliott.  Data curation for image captioning with text-to-image generative models, 2023.
  * [24] Feipeng Ma, Yizhou Zhou, Fengyun Rao, Yueyi Zhang, and Xiaoyan Sun.  Text-only image captioning with multi-context data generation, 2023.
  * [25] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Lia, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, Wesam Manassra, Prafulla Dhariwal, Casey Chu, Yunxin Jiao, and Aditya Ramesh.  Improving image generation with better captions.  https://cdn.openai.com/papers/dall-e-3.pdf, 2023.
  * [26] Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe.  Multimodal datasets: misogyny, pornography, and malignant stereotypes, 2021.
  * [27] Tan M. Dinh, Rang Nguyen, and Binh-Son Hua.  Tise: Bag of metrics for text-to-image synthesis evaluation, 2022.
  * [28] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  * [29] Tobias Hinz, Stefan Heinrich, and Stefan Wermter.  Semantic object accuracy for generative text-to-image synthesis.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(3):1552–1565, mar 2022.
  * [30] T. Xu, P. Zhang, Q. Huang, H. Zhang, Z. Gan, X. Huang, and X. He.  Attngan: Fine-grained text to image generation with attentional generative adversarial networks.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1316–1324, Los Alamitos, CA, USA, jun 2018. IEEE Computer Society.

##  Appendix A Detailed Results

The following sub-sections explain in more detail the metrics used to evaluate
our model, as well as provides plots of several metrics over the number of
training steps.

###  A.1 Image Realism

FID [28] is a measure of how close two image datasets are in terms of the
distribution of the semantic content across a large set of photos in each
dataset. It is commonly used for evaluation of text-to-image models, by taking
a paired set of texts and images, and comparing the set of original images to
a set of newly generated images from the texts. A lower score means the
distributions are more alike.
O-FID [27] is a variant of FID where off-the-shelf object detector crops out
objects from the images first, and FID is calculated on the cropped image
sets, giving somewhat more granular measurement of the distribution of
objects.

Fig. 6 shows the calculated FID and O-FID scores for several checkpoints
throughout training. It is clear that the RECAP model improves significantly
the FID and O-FID scores over the MS-COCO dataset, while fine-tuning the same
amount of steps on the same set of images, but with the original Alttext
captions, produces only a minor improvement.

###  A.2 Semantic Object Accuracy

The SOA metric was suggested in [29] and is used to evaluate the accuracy of a
text-to-image generative model, by measuring how well it follows the
instructions to generate specific objects as part of a prompt. To do so, it
uses off-the-shelf specialized object detectors of 80 different classes (e.g.
a motorcycle or a keyboard) on labeled data. There are two variants to this
metric, one that averages across images (SOA-I) and another that averages
across classes (SOA-C).

Results can be found in Fig. 7. We see a very significant improvement for the
RECAP model vs. the baseline, while the Alttext model does not show any
improvement.

###  A.3 Counting and Positional Alignments

Generative text to image models are known to struggle counting objects, i.e.
if a prompt specifies a specific number of objects (e.g. “3 birds”) the model
often generates a different number of objects. In the Counting Alignment (CA)
metric [27], MS-COCO was filtered to prompts which contain a specific
instruction to generate a known number of objects. An off-the-shelf counting
model was used to count the number of expected objects in the generated image
(per object type). The lower the score the better.

Similarly, generative text to image models often struggle to follow positional
cues, e.g. “a girl in front of a boy” or “a plate of avocado under the table”.
In the Positional Alignment (PA) metric [27], MS-COCO was filtered to prompts
with specific positional cues. Each image generated by such prompt gets a CLIP
score vs. the original prompt, and also vs. each replacement of the positional
cue in the original prompt with a different (wrong) one (e.g. "under" instead
of "in front of").

Results for both metrics can be found in Fig. 8. Once again, the RECAP model
improves the baseline while the Alttext model does not.

###  A.4 Text Alignment

R-Precision metric (RP), also known as CLIP score, [30] is a popular measure
to how close a prompt is to an image generated from it, by using the CLIP
embeddings distance between each prompt and the image generated from it.
However, CLIP is also used by Stable Diffusion as the text encoder, producing
a bias towards images generated by Stable Diffusion model, and in particular
yielding higher RP score for the generated images vs. the original real
images. Still, we see that relative to the base model (scoring $92.78$), the
Alttext model achieves lower score ($91.31$, probably due to the small dataset
size which reduces the training data variance), while the RECAP model improves
it ($93.8$, despite the small dataset size).

###  A.5 Human Evaluation

We sent samples from the base, Alttext and RECAP models for human evaluation.
Raters were presented with four images for each model along with the prompt
used to generate the image. The instructions were to only select images that
strictly followed the prompt and did not contain any major deformities (minor
deformities are fairly common with Stable Diffusion 1.4). Presented results
are averaged across raters.

We evaluate all of these models on 200 randomly sampled prompts from MS-COCO
and the DrawBench dataset [1]7772 prompts which were ¿77 CLIP tokens were
dropped, which Stable Diffusion 1.4 is known to have difficulty with. The
RECAP model generates 64.3% ($29.25\%\rightarrow 48.06\%$) more valid images,
and is 42.1% ($53.5\%\rightarrow 76\%$) more likely to be able to generate at
least one valid image (out of four seeds) over the base model. This indicates
that RECAP is both better at generating images as well as being capable of
following more difficult prompts (Tab. 2).

###  A.6 Model Weights

We provide additional plots for the various models trained with different
subset of unfrozen weights, as described in Sec. 6.2, in Figs. 9 and 10.

Figure 6: FID (left) and O-FID (right) scores for various checkpoints of the
Alttext model vs. the RECAP model. 0 fine-tuning steps is vanilla Stable
Diffusion 1.4. Lower score is better.

Figure 7: Semantic Object Accuracy (SOA) scores for various checkpoints of the
Alttext model vs. the RECAP model. 0 fine-tuning steps is vanilla Stable
Diffusion 1.4. SOA-I (left) averages across images, while SOA-C (right)
averages across classes.

Figure 8: Counting Alignment (CA) (left) and Positional Alignment (PA) (right)
scores for various checkpoints of the Alttext model vs. the RECAP model. 0
fine-tuning steps is vanilla Stable Diffusion 1.4. Lower CA score is better.

Figure 9: FID (left) and O-FID (right) scores for various checkpoints of the
Alttext model vs. RECAP models, comparing models that trained on different set
of weights. 0 fine-tuning steps is vanilla Stable Diffusion 1.4. Lower is
better.

Figure 10: Semantic Object Accuracy (SOA) scores for various checkpoints of
the Alttext model vs. RECAP models, comparing models that trained on different
set of weights. 0 fine-tuning steps is vanilla Stable Diffusion 1.4. SOA-I
(left) averages across images, while SOA-C (right) averages across classes.

##  Appendix B Other Image Generation Models

Fig. 11 shows example images generated by SDXL and Midjourney for the prompts
in the top figure.

Figure 11: Examples of images generated by SDXL 1.0 and Midjourney 5.1 for the
prompts in the top figure.

##  Appendix C Additional Captioning Examples

Fig. 12 and Fig. 13 show additional example captions generated by RECAP versus
the original Alttext.

Figure 12: Examples of captions generated by RECAP variants vs. the original
Alttext. Photos taken from LAION.

Figure 13: More examples of captions generated by RECAP variants vs. the
original Alttext. Photos taken from LAION.