11institutetext:  Department of Physics, Faculty of Natural Sciences,
Norwegian University of Science and Technology, HÃgskoleringen 5, Trondheim,
N-7491, Norway,
Niels Bohr International Academy, Blegdamsvej 17, DK-2100 Copenhagen, Denmark

# Phases and condensates in zero-temperature QCD at finite $\mu_{I}$ and
$\mu_{S}$

Jens O Andersen<EMAIL_ADDRESS>
###### Abstract

I discuss pion and koan condensation and the the properties of the phases of
QCD at finite isospin chemical potential $\mu_{I}$ and strangeness chemical
potential $\mu_{S}$ at zero temperature using three-flavor chiral perturbation
theory. Electromagnetic effects are included in the calculation of the phase
diagram, which implies that the charged meson condensed phases become
superconducting phases of QCD with a massive photon via the Higgs mechanism.
Without electromagnetic effects, we show results for the light quark
condensate and the pion condensate as functions of $\mu_{I}$ at next-to-
leading (NLO) order in the low-energy expansion. The results are compared with
recent lattice simulations and by including the NLO corrections, one obtains
very good agreement.

##  1 Introduction

In this talk, I would like to discuss various aspects of the phases of QCD at
zero temperature, but finite isospin and strangeness density. However, before
I do that, I would like to briefly comment on the QCD phase diagram as it is
normally presented, namely in the $\mu_{B}$–$T$ plane. It is shown in Fig. 1,
borrowed from Ref. hatsuda . Few of the results for the phase diagram are
rigorous in the sense that they are obtained from first principles, rather
they are obtained by model calculations. For asymptotically high temperatures
and zero baryon chemical potential, we know that QCD is in a quark-gluon
plasma phase consisting of weakly interacting deconfined quarks and gluons.
Similarly, we know that at asymptotically high baryon density and zero
temperature, QCD is in the color-flavor locked phase arising from an
attracting channel of one-gluon exchange and the resulting instability of the
Fermi surface. From lattice simulations, we know that there is a cross-over
transition for $\mu_{B}=0$ at a temperature of around 155 MeV. For low
temperatures and large chemical potentials, the infamous sign problem,
prohibits the use of standard Monte Carlo techniques to study the properties
of QCD. One must therefore resort to low-energy models such as the NJL model
and the quark-meson model. Over the past two decades, a huge amount of work
has been done to map out the phase diagram.

The situation is even more complex than this since, instead of using a common
quark chemical potential for all quarks, one can introduce a separate chemical
potential $\mu_{f}$ for each flavor. For three flavors, we use either
$\mu_{u}$, $\mu_{d}$, and $\mu_{s}$ or the baryon chemical potential
$\mu_{B}$, the isospin chemical potential $\mu_{I}$, and strangeness chemical
potential $\mu_{S}$ defined as

$\displaystyle\mu_{B}={3\over 2}(\mu_{u}+\mu_{d})\;,\hskip
28.45274pt\mu_{I}={1\over 2}(\mu_{u}-\mu_{d})\;,\hskip
28.45274pt\mu_{S}={1\over 2}(\mu_{u}+\mu_{d}-2\mu_{s})\;.$ (1)

For $\mu_{B}=\mu_{S}=0$ but nonzero $\mu_{I}$, one can carry out Monte Carlo
simulations using standard techniques since the fermion determinant is real in
this case and consequently there is no sign problem. This opens up the
possibility to study charged pion condensation on the lattice and confront it
with results from low-energy effective theories. In this talk, I will discuss
pion condensation for two and three flavors using chiral perturbation theory
($\chi$PT) as a low-energy effective theory for QCD and show results for the
light quark and pion condensates as a function of $\mu_{I}$ with
$\mu_{B}=\mu_{S}=0$. The results will be compared to recent high-precision
lattice simulations gergo ; gergo1 ; gergo2 ; gergo3 . I will also discuss the
phase diagram and meson condensation for three flavors in the
$\mu_{I}$–$\mu_{S}$ plane at zero temperature.

Figure 1: Phase diagram of QCD in the $\mu_{B}$–$T$ plane. Figure from Ref.
hatsuda .

In Fig. 2, we sketch the phase diagram of two-flavor QCD in the the
$T$–$\mu_{I}$ plane. In the lower left region, we have the hadronic phase
where chiral symmetry is broken and quarks are confined. As the temperature
increases, one enters the quark-gluon plasma phase. Along the $\mu_{I}$ axis,
there is a transition from the hadronic phase to a Bose-condensed phase of
charged pions. In this phase, the $U(1)_{I_{3}}$ symmetry is broken giving
rise to a massless Goldstone boson, which is a mixture of $\pi^{+}$ and
$\pi^{-}$. For large isospin chemical and low temperature, one expects that
quarks are the relevant degrees of freedom rather than pions son  . The Fermi
surface that exists when the interactions are turned off, is rendered unstable
once they are turned on, since they are attractive. The system is then
described in terms of loosely bound Cooper pairs instead of tightly bound
pions. Since the symmetry breaking pattern is the same, there is a cross-over
transition rather than a true phase transition between the BEC and the BCS
phases son .

Figure 2: Phase diagram of QCD in the $\mu_{I}$–$T$ plane.

##  2 $\chi$PT at finite isospin chemical potential $\mu_{I}$

We will be using chiral perturbation theory to describe the pion-condensed
phase of QCD at finite $\mu_{I}$ and zero temperature. $\chi$PT is a low-
energy effective theory for QCD based on the (global) symmetries and degrees
of freedom. It provides a correct model-independent description as long as one
is inside its domain of validity weinberg ; gasser1 ; gasser2 . The effective
Lagrangian that describes the low-energy degrees of freedom of QCD (pions,
kaons, eta) can be written in a low-energy expansion,

$\displaystyle{\cal L}$ $\displaystyle=$ $\displaystyle{\cal L}_{2}+{\cal
L}_{4}+...\;,$ (2)

where the subscript $n$ denoted the order in the expansion. The expansion
parameter can be written as ${M\over 4\pi f}$ with $M$ being a relevant mass
or momentum scale and $f$ is the pion-decay constant. The leading-order
Lagrangian is the nonlinear sigma model, which for two flavors reads

$\displaystyle{\cal L}_{2}$ $\displaystyle=$ $\displaystyle{1\over
4}f^{2}\langle\nabla^{\mu}\Sigma^{\dagger}\nabla_{\mu}\Sigma\rangle+{1\over
4}f^{2}\langle\chi^{\dagger}\Sigma+\Sigma^{\dagger}\chi\rangle\;,$ (3)

where $\langle\rangle$ means trace in flavor space and

$\displaystyle\Sigma=e^{i{\phi_{a}\tau_{a}\over f}}\;,\hskip
56.9055pt\chi=2B_{0}{\rm diag}(m_{u},m_{d})\;,$ (4)

with $\phi_{a}$ being the Goldstone bosons fields, $m_{u,d}$ are the quark
masses, and $B_{0}$ is the related to the quark condensate in the vacuum via
$\langle\bar{\psi}\psi\rangle=-f^{2}B_{0}$. The covariant derivative is

$\displaystyle\nabla_{\mu}\Sigma\equiv\partial_{\mu}\Sigma-i\left[v_{\mu},\Sigma\right]\;,$
(5)

with $v_{\mu}=\delta_{\mu 0}(\mbox{$1\over 3$}\mu_{B}\mathbb{I}+\mbox{$1\over
2$}\mu_{I}\tau_{3})$. Note that the results will be independent of $\mu_{B}$
since the unit matrix $\mathbb{I}$ commutes with everything, this reflects
that the mesons have zero baryon number.

The most general ansatz for the (normalized) ground state can after some
simplifcations be written as son

$\displaystyle\Sigma_{\alpha}$ $\displaystyle=$
$\displaystyle\mathbb{I}\cos\alpha+i\tau_{2}\sin\alpha=e^{i\alpha\tau_{2}}\;,$
(6)

which simply is a rotation in flavor space of the vacuum state $\mathbb{I}$ by
an angle $\alpha$. The leading-order thermodynamic potential is

$\displaystyle\Omega_{0}$ $\displaystyle=$
$\displaystyle-f^{2}B_{0}(m_{u}+m_{d})\cos\alpha-{1\over
2}f^{2}\mu_{I}^{2}\sin^{2}\alpha\;.$ (7)

We see that there is a competition between the two terms in Eq. (7), the first
term prefers the vacuum state $\alpha=0$, while the second term prefers
$\alpha={1\over 2}{\pi}$. The optimum is found by balancing these two terms in
the thermodynamic potential,

$\displaystyle\cos\alpha$ $\displaystyle=$
$\displaystyle{m_{\pi,0}^{2}\over\mu_{I}^{2}}\;,\hskip
28.45274pt\mu_{I}^{2}\geq m_{\pi,0}^{2}\;,$ (8) $\displaystyle\alpha$
$\displaystyle=$ $\displaystyle 0\;,\hskip
28.45274pt\mu_{I}^{2}<m_{\pi,0}^{2}\;,$ (9)

where $m_{\pi,0}^{2}=B_{0}(m_{u}+m_{d})$ is the tree-level pion mass. Thus
there is phase transition from the vacuum to a pion-condensed phase at a
critical $\mu_{I}$, $\mu_{I}^{c}=\pm m_{\pi,0}$. To determine the order of the
transition, one can construct a Ginzburg-Landau energy functional by expanding
the thermodynamic potential $\Omega_{0}$ around $\alpha=0$,

$\displaystyle\Omega_{0}$ $\displaystyle=$
$\displaystyle-f^{2}m_{\pi,0}^{2}+{1\over
2}f^{2}[m_{\pi,0}^{2}-\mu_{I}^{2}]\alpha^{2}-{1\over
24}f^{2}\left[m_{\pi,0}^{2}-4\mu_{I}^{2}\right]\alpha^{4}+{\cal
O}(\alpha^{6})\;.$ (10)

We define a critical chemical isospin potential $\mu_{I}^{c}$ when the
order-$\alpha^{2}$ term vanishes, i.e. $\mu_{I}^{c}=\pm m_{\pi,0}$. Since the
coefficient of the order-$\alpha^{4}$ term is positive for
$\mu_{I}=\mu_{I}^{c}$, the transition is second order. Note that all
thermodynamic quantities are independent of $\mu_{I}$ for all
$\mu_{I}^{2}<m_{\pi,0}^{2}$ implying that e.g. the isospin density vanishes in
the same region and not only for $\mu_{I}=0$. This is an example of the
Silver-Blaze property cohen0 .

##  3 Phase diagram for three flavors

We next consider the phase diagram for three-flavor QCD at finite $\mu_{I}$
and $\mu_{S}$ including electromagnetic effects. If we couple $\chi$PT to
dynamical photons, the Lagrangian contains a few extra terms at leading order
in the low-energy expansion gasser89 ,

$\displaystyle{\cal L}_{2}$ $\displaystyle=$ $\displaystyle-{1\over
4}F_{\mu\nu}F^{\mu\nu}+{1\over
4}f^{2}\langle\nabla_{\mu}\Sigma\nabla^{\mu}\Sigma^{\dagger}\rangle+{1\over
4}f^{2}\langle\chi^{\dagger}\Sigma+\Sigma^{\dagger}\chi\rangle+C\langle
Q\Sigma Q\Sigma^{\dagger}\rangle$ (11) $\displaystyle+{\cal L}_{\rm gf}+{\cal
L}_{\rm ghost}\;,$

with $\chi=2B_{0}{\rm diag}(m_{u},m_{d},m_{s})$ and
$\Sigma=e^{i{\phi_{a}\lambda_{a}\over f}}$. The new term $C\langle Q\Sigma
Q\Sigma^{\dagger}\rangle$ is responsible for the tree-level mass splitting
between the charged and neutral pions, and it also contributes to the tree-
level mass splitting between the charged and neutral kaons. The inclusion of
electromagnetic effects also implies that the phases with charged meson
condensates are superconductors and that the massless degree of freedom (the
Goldstone bosons) is eaten up by the now massive photon via the Higgs
mechanism. The term $v_{\mu}$ in the covariant derivative Eq. (5) is replaced
by

$\displaystyle v_{0}$ $\displaystyle=$ $\displaystyle\mbox{$1\over
3$}(\mu_{B}-\mu_{S})\mathbb{I}+\mbox{$1\over
2$}\mu_{K^{\pm}}\lambda_{Q}+{1\over 2}\mu_{K^{0}}\lambda_{K}\;,\hskip
28.45274ptv_{i}=0\;,$ (12)

where

$\displaystyle\mu_{K^{\pm}}$ $\displaystyle=$ $\displaystyle{1\over
2}\mu_{I}+\mu_{S}\;,\hskip 56.9055pt\mu_{K^{0}}=-{1\over 2}\mu_{I}+\mu_{S}\;,$
(13) $\displaystyle\lambda_{Q}$ $\displaystyle=$
$\displaystyle\lambda_{3}+{1\over\sqrt{3}}\lambda_{8}\;,\hskip
51.21504pt\lambda_{K}=-\lambda_{3}+{1\over\sqrt{3}}\lambda_{8}\;.$ (14)

In analogy with the two-flavor case, we expect onset of charged kaon
condensation when $\mu_{K^{\pm}}^{2}=m_{K^{\pm}}^{2}$ and neutral kaon
condensation when $\mu_{K^{0}}^{2}=m_{K^{0}}^{2}$. The corresponding ansätze
for the ground states are 111One could imagine multiple condensates in parts
of the $\mu_{I}$–$\mu_{S}$ plane, but that is ruled out by actual
calculations. The angles $\beta$ and $\gamma$ are the rotation angles of the
quark condensate into a charged or neutral kaon condensate, respectively.

$\displaystyle\Sigma_{\beta}=e^{i\beta\lambda_{5}}\;,\hskip
85.35826pt\Sigma_{\gamma}=e^{i\gamma\lambda_{7}}\;.$ (15)

The thermodynamic potential in the different phases can then be computed as
functions of the chemical potentials. For example, in the charged kaon
condensed phase, the thermodynamic potential is

$\displaystyle\Omega_{0}$ $\displaystyle=$
$\displaystyle-f^{2}B_{0}(m_{u}+m_{s})\cos\alpha-{1\over
2}f^{2}\left[\mu_{K^{\pm}}^{2}-\Delta m^{2}_{\rm EM}\right]\sin^{2}\alpha\;,$

where $\Delta m^{2}_{\rm EM}={2Ce^{2}\over f^{2}}$ is the splitting between
the charged and neutral kaons due to electromagnetism. It follows that the
transition takes place exactly at
$\mu_{K^{\pm}}^{2}=m_{K^{\pm}}^{2}=B_{0}(m_{u}+m_{d})+{2Ce^{2}\over f^{2}}$ as
expected.

For each value of $(\mu_{I},\mu_{S})$, we find the phase with the lowest value
of $\Omega_{0}$ (or largest pressure). This phase wins and we can map out the
phase diagram in this manner. The result is shown in the left panel of Fig. 3.
The black lines are the transition lines without electromagnetic interactions
and the red lines are with electromagnetic interactions. The former was first
obtained by Kogut and Toublan kogut33  in the isospin limit. The phases with
charged meson condensation become Higgs phases upon including electromagnetic
effects. The tree-level mass of the photon is $m_{A}=ef\sin\alpha$. The
transitions from the normal phase to a meson-condensed phase is always second
order with mean-field exponents in the $O(2)$ universality class. The
transitions between the various condensed phases are always first order and
involve the competition between the order parameters of the different phases.
As we cross the transition lines, the order parameters as well as the isospin
and strangeness densities, $n_{I}$ and $n_{S}$ jump discontinuously. The small
offset of the dashed vertical lines is due to the mass difference between the
charged and neutral kaons, which is both due to $\Delta m_{\text{EM}}\neq 0$,
and $m_{u}\neq m_{d}$. These contributions, however, pull in opposite
directions, as we see in the phase diagram. The contribution due to the
difference in quark masses adds to the mass of the $K^{0}/\bar{K}^{0}$ meson,
which is why the black transition line between the kaon condensate is to the
left of the $\mu_{I}=0$ line, while the electromagnetic contribution adds to
the mass of the charged kaon, which is why the red line is between these two
lines. The partition function in the normal phase is independent of the two
chemical potentials $\mu_{I}$ and $\mu_{S}$, which again is the Silver Blaze
property cohen0 . In the right panels, we have zoomed in on the triple points.
Upper panel shows the intersection of the normal, neutral kaon condensed and
charged kaon condensed phases, while the lower shows the intersection of the
normal, pion condensed, and charged kaon condensed phases.

Figure 3: Left panel shows the phase diagram as predicted by $\chi$PT in the
$\mu_{I}$–$\mu_{S}$-plane. In the right panel, we have zoomed in on the triple
points. See main text for details. Fig. from Ref. johnsrud .

##  4 ${\cal O}(p^{4})$ calculation of thermodynamic potential

I will next sketch the NLO calculation of the thermodynamic potential. For
simplicity I consider two flavors in the isospin limit, $m_{u}=m_{d}=m$. The
thermodynamic potential can be calculated in a low-energy expansion,

$\displaystyle\Omega$ $\displaystyle=$
$\displaystyle\Omega_{0}+\Omega_{1}+...,$

where $\Omega_{n}$ is the order-${\cal O}(p^{2n+2})$ contribution. The term
$\Omega_{1}$ receives contributions from the one-loop graphs of ${\cal
L}_{2}^{\rm quadratic}$ and counterterms coming from ${\cal L}_{4}^{\rm
static}$. The relevant terms are

$\displaystyle\mathcal{L}_{2}^{\text{quadratic}}$ $\displaystyle=$
$\displaystyle\mbox{$1\over
2$}(\partial_{\mu}\phi_{a})(\partial^{\mu}\phi_{a})+\mu_{I}\cos\alpha(\phi_{1}\partial_{0}\phi_{2}-\phi_{2}\partial_{0}\phi_{1})$
(16) $\displaystyle-\mbox{$1\over
2$}\left[(m^{2}_{\pi,0}\cos\alpha-\mu_{I}^{2}\cos
2\alpha)\phi_{1}^{2}\right.+(m^{2}_{\pi,0}\cos\alpha-\mu_{I}^{2}\cos^{2}\alpha)\phi_{2}^{2}$
$\displaystyle\left.+(m^{2}_{\pi,0}\cos\alpha+\mu_{I}^{2}\sin^{2}\alpha)\phi_{3}^{2}\right]\;,$
$\displaystyle\mathcal{L}_{4}^{\text{static}}$ $\displaystyle=$
$\displaystyle(l_{1}+l_{2})\mu_{I}^{4}\sin^{4}\alpha+l_{4}m^{2}_{\pi,0}\mu_{I}^{2}\cos\alpha\sin^{2}\alpha+(l_{3}+l_{4})m^{4}_{\pi,0}\cos^{2}\alpha+h_{1}m_{\pi,0}^{4}\;,$
(17) $\displaystyle\;,$

where $l_{1}$–$l_{4}$ and $h_{1}$ are bare couplings. They are related to the
renormalized couplings ${l_{i}^{r}}$ and $h_{i}^{r}$ via
$l_{i}=l_{i}^{r}(\Lambda)+{\gamma_{i}\Lambda^{-2\epsilon}\over
2(4\pi)^{2}}\left[{1\over\epsilon}+1\right]$ and
$h_{i}=h_{i}^{r}(\Lambda)+{\delta_{i}\Lambda^{-2\epsilon}\over
2(4\pi)^{2}}\left[{1\over\epsilon}+1\right]$, where $\gamma_{i}$, $\delta_{i}$
are coefficients and $\Lambda$ is the renormalization scale in the
$\overline{\rm MS}$ scheme. Since $\delta_{1}=0$, $h_{1}=h_{1}^{r}$ and does
not run. Performing the Gaussian integral over the quantum fields $\phi_{a}$
in dimensionsal regularization using Eq. (16), we obtain a divergent
contribution to $\Omega_{1}$. The divergences are cancelled by adding the
quartic terms from Eq. (17) and renormalizing the couplings by making
replacing the bare couplings with the renormalized ones. The final result is

$\displaystyle\Omega_{0}+\Omega_{1}$ $\displaystyle=$
$\displaystyle-f^{2}m^{2}_{\pi,0}\cos\alpha-\frac{1}{2}f^{2}\mu_{I}^{2}\sin^{2}\alpha$
(18)
$\displaystyle-\frac{1}{4(4\pi)^{2}}\left[\frac{3}{2}-\bar{l}_{3}+4\bar{l}_{4}+\log\left({m_{\pi,0}^{2}\over\tilde{m}_{2}^{2}}\right)+2\log\left({m_{\pi,0}^{2}\over
m_{3}^{2}}\right)\right]m_{\pi,0}^{4}\cos^{2}\alpha$
$\displaystyle-\frac{1}{(4\pi)^{2}}\left[{1\over
2}+\bar{l}_{4}+\log\left({m_{\pi,0}^{2}\over
m_{3}^{2}}\right)\right]m^{2}_{\pi,0}\mu_{I}^{2}\cos\alpha\sin^{2}\alpha$
$\displaystyle-\frac{1}{4(4\pi)^{2}}\left[1+\frac{2}{3}\bar{l}_{1}+\frac{4}{3}\bar{l}_{2}+2\log\left({m^{2}_{\pi,0}\over
m_{3}^{2}}\right)\right]\mu_{I}^{4}\sin^{4}\alpha$
$\displaystyle-{1\over(4\pi)^{2}}\bar{h}_{1}m_{\pi,0}^{4}+V_{{1},\pi^{+}}^{\text{fin}}+V_{{1},\pi^{-}}^{\text{fin}}\;,$

where $\tilde{m}_{2}^{2}=m_{\pi,0}^{2}\cos\alpha$,
$m_{3}^{2}=m_{\pi,0}^{2}+\mu_{I}^{2}\sin^{2}\alpha$, and
$V_{{1},\pi^{+}}^{\text{fin}}+V_{{1},\pi^{-}}^{\text{fin}}$ are two
complicated finite terms that must be evaluated numerically. Finally,
$\bar{l}_{i}$ and $\bar{h}_{i}$ are, up to a prefactor, equal to $l_{i}^{r}$
and $h_{i}^{r}$ at the scale $\Lambda=m_{\pi,0}$. Using Eq. (18) one can show
that the phase transition takes place at $\mu_{I}^{c}=m_{\pi}$, where the
physical pion mass now includes radiative corrections gasser1 , see Eq. (19)
below. The parameters $\bar{l}_{i}$ are determined by experiment and
$\bar{h}_{1}$ estimated by model calculations. and the parameters
$m_{\pi,0}^{2}=2B_{0}m$ and $f$ can be found by inverting the one-loop
relations using the experimental values for the pion mass and the pion decay
constant,

$\displaystyle m_{\pi}^{2}=m^{2}_{\pi,0}\left[1-{m^{2}_{\pi,0}\over
2(4\pi)^{2}f^{2}}\bar{l}_{3}\right]\;,\hskip
28.45274ptf_{\pi}^{2}=f^{2}\left[1+{2m^{2}_{\pi,0}\over(4\pi)^{2}f^{2}}\bar{l}_{4}\right]\;.$
(19)

##  5 Condensates

In order to obtain the light quark and pion condensates, we need to calculate
the thermodynamic potential $\Omega$ with sources $m$ and $j$, where the
latter is a pionic source. The former has already been included in the
calculations I have shown and it is also straightforward to include a pionic
source $j$ in the calculations. For example, in the two-flavor expression for
the thermodynamic potential, one simply makes the replacements
$m\cos\alpha\rightarrow m\cos\alpha+j\sin\alpha$ and
$\bar{h}_{1}m_{\pi,0}^{4}=\bar{h}_{1}(2B_{0}m)^{2}\rightarrow\bar{h}_{1}[(2B_{0}m)^{2}+(2B_{0}j)^{2}]$
martin . Once these replacements are made, the condensates are given by

$\displaystyle\langle\bar{\psi}\psi\rangle_{\mu_{I}}={1\over
2}{\partial\Omega\over\partial m}=-f^{2}B_{0}\cos\alpha+...\;,\hskip
28.45274pt\langle\pi^{+}\rangle_{\mu_{I}}={1\over
2}{\partial\Omega\over\partial j}=-f^{2}B_{0}\sin\alpha+...\;,$ (20)

where I on the the right-hand side have written explicitly the tree-level
contributions. The subscript $\mu_{I}$ on the expectation values indicate that
they depend on the chemical potential. Instead of plotting the condensates
directly, we define the normalized deviations as

$\displaystyle\Sigma_{\bar{\psi}\psi}=-\frac{2m}{m_{\pi}^{2}f_{\pi}^{2}}\left[\langle\bar{\psi}\psi\rangle_{\mu_{I}}-\langle\bar{\psi}\psi\rangle_{0}^{j=0}\right]+1\;,\hskip
28.45274pt\Sigma_{\pi}=-\frac{2m}{m_{\pi}^{2}f_{\pi}^{2}}\langle\pi^{+}\rangle_{\mu_{I}}\;.$
(21)

At tree level, Eq. (20) shows the rotation of the quark condensate into a pion
condensate. Equivalently, from Eq. (21), the deviations at tree level satisfy
$\Sigma_{\bar{\psi}\psi,\rm tree}+\Sigma^{2}_{\pi,\rm tree}=1$. This
interpretation no longer holds beyond ${\cal O}(p^{2})$. In the left panel of
Fig. 4, we show $\Sigma_{\bar{\psi}\psi}$ as a function of ${\mu_{I}\over
m_{\pi}}$ at leading order (red line) 222The leading order result is the same
for two and three flavors. and next-to-leading order for two flavors (blue
line) and three flavor (green line). The data points are from the lattice
simulations of Ref. gergo ; gergo1 ; gergo2 ; gergo3 . In the right panel, we
show $\Sigma_{\pi}$ in the same approximations. We note that the difference
between $\Sigma_{\bar{\psi}\psi}$ in the various approximations is very small
and they all agree very well with the lattice data points. Regarding
$\Sigma_{\pi}$, we notice that it is nonzero for $\mu_{I}<m_{\pi}$, which
simply reflects that the curves shown are for nonzero pion source,
$j=0.00517054m_{\pi}$ The $U(1)_{I_{3}}$-symmetry is therefore broken
explicitly for all values of $\mu_{I}$. Comparing the various approximations
and lattice data, it is evident that including the ${\cal O}(p^{4})$
corrections results in a substantially better agreement between $\chi$PT and
simulations. All the numerical results have been obtained by using the same
physical meson masses as well as $f_{\pi}$ as in the lattice simulations. This
requires that we invert the relations Eq. (19) to obtain the values for the
bare mass $m$ and the bare pion decay constant $f$ using the experimental
values of $\bar{l}_{i}$.

Figure 4: $\Sigma_{\bar{\psi}\psi}$ (left panel) and $\Sigma_{\pi}$ (right
panel) as functions of $\mu_{I}/m_{\pi}$ at zero temperature and finite source
$j=0.00517054m_{\pi}$. Fig. taken from Ref. martin .

##  6 Acknowledgements

I would like to thank Prabal Adhikari, Martin Mojahed, and Martin Kjøllesdal
Johnsrud for collaboration. I would also like to thank the organizers of the
XVth Quark confinement and the Hadron spectrum conference for a very
interesting meeting.

## References

  * (1) T. Hatsuda, K. Fukushima, Rept. Prog. Phys 74, 014001 (2011)
  * (2) B.B. Brandt, G. Endrődi, PoS LATTICE p. 039 (2016)
  * (3) B.B. Brandt, G. Endrődi, S. Schmalzbauer, EPJ Web Conf. 175, 07020 (2018)
  * (4) B.B. Brandt, G. Endrődi, S. Schmalzbauer, Phys. Rev. D 97, 054514 (2018)
  * (5) B.B. Brandt, G. Endrődi, Phys. Rev. D 99, 014518 (2019)
  * (6) D.T. Son, M. Stephanov, Phys. Rev. Lett 86, 592 (2001)
  * (7) S. Weinberg, Physica A 96 (1979)
  * (8) J. Gasser, H. Leutwyler, Ann. Phys. 158, 142 (1984)
  * (9) J. Gasser, H. Leutwyler, Nucl. Phys. 250, 465 (1985)
  * (10) T.D. Cohen, Phys. Rev. Lett. 91, 222001 (2003)
  * (11) G. Ecker, J. Gasser, A. Pich, E. de Rafael, Phys. Lett. B 321 (1989)
  * (12) J. Kogut, D. Toublan, Phys. Rev. D 64, 034007 (2001)
  * (13) J.O. Andersen, M.K. Johnsrund (2022), 2206.04291
  * (14) P. Adhikari, J.O. Andersen, M. Mojahed, Eur. J. Phys. C 81, 449 (2021)


11institutetext: Simon Fraser University
11email: {shitaot, jca348, fuyangz<EMAIL_ADDRESS>22institutetext: Meta
Reality Labs
22email: {wdilin, chengzhout, ycfan, vchandra<EMAIL_ADDRESS>
# MVDiffusion++: A Dense High-resolution Multi-view Diffusion Model for Single
or Sparse-view 3D Object Reconstruction

Shitao Tang 1*1* Jiacheng Chen 1*1* Dilin Wang 2*2* Chengzhou Tang 22
Fuyang Zhang 11 Yuchen Fan 22 Vikas Chandra 22
Yasutaka Furukawa 1†1† Rakesh Ranjan 2†2†

###### Abstract

This paper presents a neural architecture MVDiffusion++ for 3D object
reconstruction that synthesizes dense and high-resolution views of an object
given one or a few images without camera poses. MVDiffusion++ achieves
superior flexibility and scalability with two surprisingly simple ideas: 1) A
“pose-free architecture” where standard self-attention among 2D latent
features learns 3D consistency across an arbitrary number of conditional and
generation views without explicitly using camera pose information; and 2) A
“view dropout strategy” that discards a substantial number of output views
during training, which reduces the training-time memory footprint and enables
dense and high-resolution view synthesis at test time. We use the Objaverse
for training and the Google Scanned Objects for evaluation with standard novel
view synthesis and 3D reconstruction metrics, where MVDiffusion++
significantly outperforms the current state of the arts. We also demonstrate a
text-to-3D application example by combining MVDiffusion++ with a text-to-image
generative model. The project page is at https://mvdiffusion-
plusplus.github.io.

††*Equal contribution. †Joint last author. Work done during an internship with
Meta.

##  1 Introduction

Figure 1: MVDiffusion++ generates dense(32) and high-
resolution(512$\times$512) images of an object from a single or multiple
unposed images. The input images of the three examples are from a latent
diffusion model, OmniObject3D[39], and Google Scanned Objects[4],
respectively. Please see our project website for more results:
https://mvdiffusion-plusplus.github.io.

Human vision demonstrates remarkable flexibility. Look at the images of
objects at the left in Figure 1. While unable to create millimeter-accurate 3D
models, our visual system can combine information from a few images to form a
coherent 3D representation in our minds, including intricate facial features
of a tiger or the arrangement of blocks forming a toy train, even parts that
are fully obscured.

3D reconstruction technology has evolved over the last fifteen years in a
fundamentally different way. Unlike the human ability to infer 3D shapes from
a few images, the technology takes hundreds of images of an object, estimates
their precise camera parameters, and reconstructs high-fidelity 3D geometry at
a sub-millimeter accuracy.

This paper explores a new paradigm of 3D reconstruction that combines the
high-fidelity of computational methods and the flexibility of human visual
systems. Our inspiration comes from exciting recent developments in multi-view
image generative models [32, 30, 17, 18, 29, 14]. MVDiffusion [32] is an early
attempt to extend pre-trained image diffusion models [26] to a multi-view
generative system, when pixel correspondences across views are available
(e.g., generating perspective images to form a panorama). MVDream [30] and
Wonder3D [18] further extend to more general settings where generated images
yield 3D reconstruction via techniques such as NeRF [20] or NeuS [35].

This paper pushes the frontier of multi-view diffusion models towards flexible
and high-fidelity 3D reconstruction systems. Concretely, the paper presents
MVDiffusion++, a novel approach to generate dense (32) and high-resolution
(512$\times$512) images of an object, conditioned with single or sparse input
views without camera poses, whose reliable estimation is difficult due to
minimal or no visual overlaps. Standard 3D reconstruction techniques turn
generated images into a 3D model. Two simple ideas are at the heart of our
method. First, we leverage a latent diffusion inpainting model with
conditional and generation branches, where self-attention among 2D features
learns 3D consistency without using camera poses or image projection formula.
Second, we introduce “view dropout” training strategy, which randomly excludes
generation views in each batch, enabling the use of high-resolution images
during training. During testing, this simple approach surprisingly generates
high-quality, dense views for all the images simultaneously.

MVDiffusion++ achieves state-of-the-art performance on the task of novel view
synthesis, single-view reconstruction, and sparse-view reconstruction. For
single-view reconstruction, our method achieves 0.6973 IoU and 0.0165 Chamfer
distance on the Google Scanned Objects dataset, higher than SyncDreamer [17]
by 0.1552 in terms of Vol. IOU. For novel view synthesis in sparse view
setting, MVDiffusion++ improves the PSNR by 8.19 compared with a recent pose-
free view synthesis method, LEAP [10]. Lastly, we demonstrate applications in
text-to-3D by combining MVDiffusion++ with a text-to-image generative model.

##  2 Related work

This paper presents a multi-view image generative model for object
reconstruction, given one or a few condition images. The section reviews
related work on multi-view image generation and single to sparse-view 3D
reconstruction techniques.

Multi-view image generation. The evolution of text-to-image diffusion models
has paved the way for multi-view image generation. MVDiffusion [32] introduces
an innovative multi-branch Unet architecture for denoising multi-view images
simultaneously. This approach, however, is constrained to cases with one-to-
one image correspondences. Syncdreamer [17] uses 3D volumes and depth-wise
attention for maintaining multi-view consistency. MVDream [30] takes a
different path, incorporating 3D self-attention to extend the work to more
general cases. Similarly, Wonder3D [18] and Zero123++ [29] apply 3D self-
attention to single-image conditioned multi-view image generation. These
methods, while innovative, tend to produce sparse, low-resolution images due
to the computational intensity of the attention mechanism. In contrast, our
framework represents a more versatile solution capable of generating dense,
high-resolution multi-view images conditioned on an arbitrary number of
images.

Single view reconstruction. Single View Image Reconstruction is an active
research area [17, 40, 18, 21, 37], driven by the advancements of generative
models [17, 18, 37, 21]. Large reconstruction model [9] predicts triplanes
from a single image, but the 3D volume limits its resolutions. The other
method, Syncdreamer [17] generates multi-view images with a latent diffusion
model by constructing a cost volume. These images are then used to recover 3D
structures using conventional reconstruction methods like Neus. However, this
process requires substantial GPU memory, limiting it to low resolutions.
Similarly, Wonder3D faces challenges due to the computational demands of self-
attention, leading to similar restrictions. In contrast, our approach
introduces a ”view dropout” technique, which randomly samples a limited number
of views for training in each iteration. This enables our model to generate a
variable number of high-resolution images while employing full 3D self-
attention, effectively addressing the limitations faced by existing methods.

Sparse view reconstruction. Sparse View Image Reconstruction (SVIR) [10, 41]
is a challenging task where only a limited number of images, typically two to
ten, are given. Traditional 3D reconstruction methods estimate camera poses
first, then perform dense reconstruction using techniques such as multi-view
stereo [31, 42] or NeRF [35]. However, camera pose estimation is difficult for
SVIR, where visual overlaps are none to minimal. To address this, FvOR [41]
optimizes camera poses and shapes jointly. LEAP [10] along with PF-LRM [36]
highlight the issues of noisy camera poses and suggest a pose-free approach.
However, they are not based on generative models, lacking generative priors,
and suffer from low-resolution outputs due to the use of volume rendering. In
contrast, our method employs a diffusion model to generate high-resolution
multi-view images directly, then a reconstruction system Neus [35] to recover
a mesh model.

##  3 Preliminary: Multi-view latent diffusion models

MVDiffusion [32] is a multi-view latent diffusion model [32, 30, 17, 29],
generating multiple images given a text or an image, when pixel-wise
correspondences are available across views. MVDiffusion is the foundation of
the proposed approach, where the section reviews its architecture and
introduces notations (See Figure 3).

For generating eight perspective views forming a panorama, eight latent
diffusion models (LDM) denoise eight noisy latent images
$\\{Z_{1}(t),Z_{2}(t),\cdots Z_{8}(t)\\}$ simultaneously. A UNet is the core
of a LDM model, consisting of a sequence of blocks through the four levels of
the feature pyramid.

Let $U^{i}_{b}$ denote the feature image of $i$-th image at $b$-th block. A
CNN initializes an input $U^{0}_{i}$ from $Z_{i}(t)$ at the first block. Each
UNet block has four network modules. The first is a novel correspondence-aware
attention (CAA), enforcing consistency across views with visual overlaps: The
left/right neighboring images ($U^{b}_{i-1},U^{b}_{i+1}$) for panorama. The
remaining three modules are from the original: 1) Self-attention (SA) layers;
2) Cross-attention (CA) layers from the condition with the CLIP embedding; and
3) CNN layers with the pixel-wise concatenation of a positional encoding of
time $\tau(t)$. At test time, a standard DDPM sampler [8] updates all noisy
latents with the predicted noise from the last CNN layer. The training
objective is defined as follows by omitting the conditions for notation
simplicity, where ${\boldsymbol{\epsilon}}^{i}$ is a Gaussian and
$\epsilon_{\theta}$ denotes the UNet output.

$\displaystyle
L_{MVLDM}:=\mathbb{E}_{\left\\{Z_{i}(0)\right\\}_{i=1}^{N},\left\\{{\boldsymbol{\epsilon}}^{i}\sim\mathcal{N}(0,I)\right\\}_{i=1}^{N},t}\Big{[}\sum_{i=1}^{N}\|{\boldsymbol{\epsilon}}^{i}-\epsilon_{\theta}^{i}(\left\\{Z_{i}(t)\right\\},\tau(t))\|_{2}^{2}\Big{]}.$
(1)

##  4 MVDiffusion++

MVDiffusion++ pushes the frontier of multi-view diffusion models for 3D
modeling in their flexibility and scalability by generating denser and higher-
resolution images given an arbitrary number of un-posed condition views. With
the prevalence of Transformer models [33], high-fidelity 3D modeling would
require large-scale attention over dense and high-resolution image features,
potentially with volumetric ones. Furthermore, 3D consistency learning is at
the heart of the task, which would usually require precise image projection
models and/or camera parameters. Our surprising discovery is that self-
attention among 2D latent image features is all we need for 3D learning
without projection models or camera parameters, and a simple training strategy
would further achieve dense and high-resolution multi-view image generation.
The section defines the task (i.e., input condition and output target images),
then explains the two key ideas: 1) pose-free multi-view conditional diffusion
model for flexibility and 2) view dropout training strategy for scalability.
§5 provides the remaining system details.

###  4.1 Task: Input condition images and output target images

Figure 2: The input and output specification (§4.1) of MVDiffusion++. The 32
target images are defined in eight azimuths and four elevation levels. During
training, our view dropout strategy (§4.3) randomly drops a substantial number
of views (dashed blue) and trains the model to denoise the remaining views
(red).  Figure 3: The denoising architectures for MVDiffusion and
MVDiffusion++ for sampling multi-view images. The order of the MVDiffusion
network modules is rearranged to highlight the differences (in orange) with
MVDiffusion++.  Figure 4: Illustration of the pose-free multi-view conditional
diffusion model of MVDiffusion++. The model takes any number of input images
and generates images at fixed viewpoints. The condition branch and generation
branch have different input configurations but share the same structure and
weights. We use superscript $C$ and $G$ to denote conditional and generation
branches.

The generation target is a set of dense (32) and high-resolution
(512$\times$512) images, positioned at uniform 2D grid points on a sphere
along the azimuth and elevation angles for 3D object reconstruction (See
Figure 2). Specifically, there are eight azimuth angles (every 45∘) and four
elevation angles (every 30∘ in the range $[-30^{\circ},60^{\circ}]$). Camera
up-vectors are aligned with gravity, and their optical axes pass through the
sphere center. Our input condition is one or a few images without camera
poses, where visual overlaps are too minimal or possibly none for Structure
from Motion algorithms to work reliably. The number of condition images is up
to a pre-determined number, which is $10$ in our experiments but can easily
change. The input image resolution is 512$\times$512\. The horizontal and
vertical field-of-view of both the input and output views is 60∘.

We use synthetic rendered images from 3D object databases for training and
evaluations, then real-world images as the input condition for further
qualitative evaluations. The task settings vary slightly between datasets,
with details provided in §5. Here, we explain one preprocessing step that
removes ambiguity in the training task. 3D object databases such as Objaverse
[3] and Google Scanned Object [4] align the Z-axis with the object up-vectors.
However, the azimuth of the ground-truth object pose is ambiguous without
camera poses of the condition images. Therefore, we rotate the output views to
align the azimuth of the first condition and the first output image as shown
by the right of Figure 2.

###  4.2 Pose-free multi-view conditional diffusion model

MVDiffusion++ is a multi-view latent diffusion model as defined in §3,
comprising of a condition branch for single or sparse-view input images and a
generation branch for output images (See Figure 3 and Figure 4). Note that the
condition branch shares the same architecture and is tasked to generate the
condition images that are also given as guidance (i.e., a trivial task).

Diffusion process. The forward diffusion process is the same as MVDiffusion,
except for the image resolution and the pre-trained VAE. Concretely, it 1)
converts all $512\times 512\times 3$ input/output image ($I_{i}$) with
foreground masks to $64\times 64\times 4$ latent images ($Z_{i}$) by a fine-
tuned latent diffusion VAE (denoted as MVAE, see §5 for the fine-tuning
process); and 2) adds a Gaussian noise with a linear schedule, as suggested by
zero-123++ [29] to each feature of $Z_{i}$.

Denoising process. The denoising process is highlighted in Figure 3, where a
latent diffusion UNet with a few modifications processes a noisy latent
$Z_{i}(t)$ at each denoising step $t$. The UNet consists of $9$ blocks of
network modules over the four levels of feature pyramids on either side of the
encoder/decoder. The details are explained as follows.

[At first block] The UNet feature $U^{0}_{i}$ at the first block is
initialized with the concatenation of 1) the noisy latents $Z_{i}$; 2) a
constant binary mask of either 1 or 0, denoted by $\mathfrak{M}_{pos}$ or
$\mathfrak{M}_{neg}$ to indicate the branch type (condition or generation);
and 3) the condition latents ($\text{MVAE}(I_{i},M_{i})$) where we use the
conditonal VAE from latent diffusion to encode the condition image ($I_{i}$)
with its segmentation mask ($M_{i}$). Note that this concatenation has
$9=(4+4+1)$ channels, and a $1\times 1$ final convolution layer reduces the
channel dimension to $4$. For a generation branch, we pass a white image as
$I_{i}$ and a binary image of 1 (i.e., $\mathfrak{M}_{pos}$) as $M_{i}$. For
Objaverse and Google Scaned Object datasets, we use the masks provided by the
datasets. Otherwise, we run segmentation to generate the masks.

[For each block] Three network modules are at the heart of the processing: 1)
Global self-attention mechanism among the UNet features across all the images,
learning 3D consistency; 2) Cross-attention mechanism, injecting the CLIP
embedding of the condition images to all the other images through the CLIP
embedding; and 3) CNN layers, process per-image features while injecting the
timestep frequency encoding $\tau(t)$ and the learnable embedding of an image
index $V_{i}$. For the self-attention module, we copy the network architecture
and model weights and apply it across all the views. This module is inspired
by MVDream [30], while the key differences in our work are 1) Scalability
deployment via the view-drop training strategy in §4.3; and 2) Handling of
multiple condition images without camera poses via the network design.
$42=(32+10)$ learnable embedding vectors $\\{V_{i}\\}$ are trained for 32
generation and 10 condition images, each of which is multiplied with a zero-
initialized trainable scale $s$ to avoid model disruption at the start of
training.

[At last block] The output of the last UNet block yields the noise estimation,
and a standard DDPM sampler [8] takes it to produce the noisy latent of the
next timestep $Z_{i}(t-1)$ for each sampling step. The loss function is the
same as MVDiffusion. Note that the model is first trained with
$\epsilon$-prediction and then with v-prediction (See §5), where Equation 1 is
the loss function for the $\epsilon$-prediction model. The velocity [28],
${\boldsymbol{v}}^{i}(t)=\alpha_{t}{\boldsymbol{\epsilon}}^{i}-\gamma_{t}Z_{i}(0)$,
becomes the prediction target for the v-prediction model, while $\alpha_{t}$
and $\gamma_{t}$ are predefined angular parameters.

###  4.3 View dropout training strategy

MVDiffusion++ training would face a scalability challenge. $42(=32+10)$ copies
of UNet features yield more than 130k tokens, where the global self-attention
mechanism becomes infeasible even with the latest memory efficient
transformers for large language models [2, 1]. We propose a simple yet
surprisingly effective view dropout training strategy, which completely
discards a set of views across all layers during training. Specifically, we
randomly drop 24 out of 32 views for each object at each training iteration,
significantly reducing memory consumption at training. At test time, we run
the entire architecture and generate 32 views.

##  5 Remaining system details

This section explains the remaining system details on the data preparations,
the mesh extraction process, the MVAE pre-fine-tuning, and the three-stage
training strategy.

###  5.1 Training data preparation

Out of 800k 3D object models from Objaverse [3], we use 180k models whose
aesthetic scores [22] are at least 5 for training. For each object 3D model,
we translate the bounding box center to the origin and apply uniform scaling
so that the longest dimension matches $[-1,1]$. The output camera centers are
placed at a distance of 1.5 from the origin. Input condition views are chosen
in a similar way as Zero-123 [16]. Concretely, an azimuth angle is randomly
chosen from one of the eight discrete angles of the output cameras (also see
§4.1). The elevation angle is set randomly from [-10∘, 45∘]. The distance of
the camera center from the origin is set randomly from [1.5, 2.2]. We use
Blender to render images.

###  5.2 Testing data preparation

Single-view cases. Google Scanned Object (GSO) [4] is our testing dataset,
where we borrow the rendered images and the evaluation pipeline from
SyncDreamer [17]. Concretely, the test set consists of 30 objects. Each object
has 16 images with a fixed elevation of 30∘ and every 22.5∘ for azimuth.
SyncDreamer selected condition images by “visual plausibility”, which we copy.
Since the azimuth angles in our training setting are every 45∘, eight images
(starting from and including the condition image) are used for evaluation. The
resolution of the rendered images is 256x256, while the image resolution of
our architecture is 512x512. We upscale the condition images to 512x512 for
our system inputs. The ground-truth images are 256x256 and we downscale our
generated images to 256x256 for evaluation, while 512x512 images are used for
the mesh reconstruction. The Chamfer Distances (CD) and volume IoU between the
ground-truth and reconstructed shapes are reported for single-view 3D
reconstruction. The PSNR, SSIM [38], and LPIPS [44] are reported for novel
view synthesis (NVS) by averaging over the eight images.

Sparse-view cases. Sparse-view un-posed condition is a new setup (except the
work of LEAP [10] and PF-LRM [36] to our knowledge). We use a process similar
to the single-view setting to render images. Concretely, we first render 10
condition images for each of the 30 GSO objects. The azimuth and the elevation
angles are chosen randomly from [0, 360) and [-10, 45] respectively. We render
32 ground-truth target images while aligning the azimuth of the first target
view and the first input view (See §4.1 and Fig. 2). The same evaluation
metrics are used, while we vary the number of condition images to be 1, 2, 4,
8, and 10.

###  5.3 Mesh extraction from generated images

After generating 32 images (all target views in Fig. 2), a neural implicit
reconstruction method recovers a mesh model, similar to SyncDreamer [17] and
Wonder3D [18]. Specifically, we use grid-based NeuS [6, 12], where the
foreground masks are decoded from the latent images $\\{Z_{i}(0)\\}$ by MVAE.
Since our generated images have high resolution and quality, we directly run
the monocular normal estimator released by Omnidata [5] to obtain additional
normal supervisions for NeuS without a normal generation module like Wonder3D.
We borrow the NeuS implementation from Wonder3D’s official codebase but do not
use their ranking-based loss. With a single Nvidia 2080 Ti, it takes around 3
minutes to reconstruct a textured mesh model. The mesh could directly use the
exported vertex color or be re-textured with the generated images.

###  5.4 Mask-aware VAE pre-fine-tuning

We copy the network architecture and model weights of the default VAE [27] and
add additional input and output channels to handle the mask. We found that
fine-tuning M-VAE only with object images improves performance. Concretely, we
use approximately 3 million RGBA images rendered from Objaverse to fine-tune
M-VAE as a pre-processing. We follow the original VAE hyperparameters with a
base learning rate of 4.5e-6 and a batch size of 64. The training runs for
60,000 iterations. The binary cross entropy loss is used for the mask channel.
The process improves PSNR from 36.6 to 41.2.

###  5.5 Three-stage training strategy

After initializing the UNet model weights by a pre-trained latent diffusion
inpainting model, we train the proposed system in three stages. First, we
train as an $\epsilon$-prediction model only with single-view conditioning
cases, because our pre-trained model was trained as $\epsilon$-prediction.
Second, we fine-tune as a v-prediction model [28] still with single-view
conditioning cases. Third, we fine-tune as a v-prediction model with both
single and sparse-view conditioning cases. Half the samples are single-view
conditioning, and the other half are sparse-view conditioning, where the
number of condition images is uniformly sampled between 2 and 10.

##  6 Experiments

We train the model with a batch size of 1024 using 128 Nvidia H100 GPUs for
about a week. We employ an AdamW optimizer with a learning rate of 7e-5 and
use a cosine learning rate scheduler similar to Zero123++. We also incorporate
the Zero-SNR fix [13] commonly adopted in video generation models. At test
time, we use DDPM [8] sampler with 75 steps to sample the multi-view images,
and it takes our model 30s, 77s, 123s, and 181s to generate 8, 16, 24, and 32
images, respectively. The section presents the single view experiments in
§6.1, the sparse view experiments in §6.2, and text-to-3D application
experiments in §6.3.

###  6.1 Single-view object modeling

Table 1:  Single-view object modeling results, evaluating reconstructed meshes (left) and generated images (right). The ground-truth meshes and images are prepared by SyncDreamer [17] based on the Google Scanned Object [4] dataset. ICP is necessary to align reconstructed meshes for methods marked with ${}^{*}.$ Task $\rightarrow$ | 3D reconstruction | Novel view synthesis
---|---|---
Method | Chamfer Dist.$\downarrow$ | Vol. IoU$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
Realfusion [19] | 0.0819 | 0.2741 | 15.26 | 0.722 | 0.283
Magic123 [25] | 0.0516 | 0.4528 | - | - | -
One-2-3-45 [15] | 0.0629 | 0.4086 | - | - | -
Point-E [23] | 0.0426 | 0.2875 | - | - | -
Shap-E [11] | 0.0436 | 0.3584 | - | - | -
Zero123 [16] | 0.0339 | 0.5035 | 18.93 | 0.779 | 0.166
SyncDreamer [17] | 0.0261 | 0.5421 | 20.05 | 0.798 | 0.146
Wonder3D [18]∗ | 0.0329 | 0.5768 | - | - | -
Open-LRM [7]∗ | 0.0285 | 0.5945 | - | - | -
Ours | 0.0165 | 0.6973 | 21.45 | 0.844 | 0.129
Figure 5: Single-view object modeling results of generated images. The input
image and the generated images by Wonder3D and SyncDreamer are in
256$\times$256\. Our rendered images are in 512$\times$512, showing higher
fidelity and richer details.

Three state-of-the-art single-view object modeling methods are our main
baselines: SyncDreamer [17], Wonder3D [18], and Open-LRM [7]. Since the
evaluation pipeline is the same as SyncDreamer, we copy numbers of other
baselines in their paper for comparison, which includes Zero123 [16],
RealFusion [19], Magic123 [25], One-2-3-45 [15], Point-E [23], and Shap-E
[11]. The following introduces the three main baselines and how we reproduce
their systems:

$\bullet$ SyncDreamer generates 16 images from fixed viewpoints given a single
input image. The image resolution is 256x256. Their denoising network
$\epsilon_{\theta}$ initializes from Zero123 and leverages 3D feature volumes
and depth-wise attention to learn multi-view consistency. It requires users to
provide the elevation of the input image. We run the official codebase on the
GSO input images to get the results, and evaluate them with the provided
testing scripts.

$\bullet$ Wonder3D takes a single input image as the canonical view and
generates 6 images as well as the normal maps. The image resolution is
256$\times$256\. Multi-view self-attention and an extra cross-domain attention
ensure the consistency of generation results, while the views are sparser than
ours. We run the official codebase on the GSO input images to get the results.
However, the released model assumes orthographic cameras and we cannot use the
same test set to evaluate the NVS performance. ICP aligns the reconstructed
mesh with the ground truth before computing the metrics.

$\bullet$ Open-LRM is an open-source implementation of Large Reconstruction
Model (LRM) [9], a generalized reconstruction model that predicts a triplane
NeRF from a single input image using a feed-forward transformer-based network.
ICP aligns the reconstructed mesh with the ground truth before computing the
CD and volume IoU.

Figure 6:  Single-view object modeling results of reconstructed mesh models.
Our meshes are exported from dense (32) and high-resolution (512$\times$512)
generated images, demonstrating finer details.

Results. Table 1 presents the quantitative evaluations of the reconstructed 3D
meshes and the generated images. MVDiffusion++ consistently outperforms all
the competing methods with clear margins. Note that the evaluation is not
completely fair for Wonder3D that assumes orthographic camera projections,
where perspective images are used in the experiments. However, we believe the
clear performance gaps suffice to demonstrate the strength of our method.

Figure 5 and Figure 6 show generated images and reconstructed mesh models. In
Figure 5, our method clearly shows the number on the clock (row 3), while
others exhibit blurry numbers. Another example (row 5) showcases two perfectly
symmetrical windows generated by our method, contrasting with Wonder3D’s
failure to maintain symmetry or clarity. In Figure 6, our method can recover a
plausible and detailed shape of the turtle example (row 1), while Wonder3D and
OpenLRM fail to recognize it as a turtle and exhibit significant artifacts.

###  6.2 Sparse-view object modeling

Table 2:  Sparse-view object modeling results, evaluating reconstructed meshes
(left) and generated images (right), based on the GSO [4] dataset.

Method | Views | Chamfer Dist.$\downarrow$ | Vol. IoU$\uparrow$
---|---|---|---
Sync- Dreamer | 1 | 0.0318 | 0.5610
NeuS[35] (G.T. pose) | 1 | 0.0536 | 0.4400
2 | 0.0307 | 0.5884
4 | 0.0158 | 0.7323
6 | 0.0121 | 0.7768
8 | 0.0105 | 0.7982
10 | 0.0096 | 0.8092
Ours | 1 | 0.0208 | 0.6689
2 | 0.0158 | 0.7260
4 | 0.0122 | 0.7737
6 | 0.0109 | 0.7925
8 | 0.0101 | 0.8033
10 | 0.0101 | 0.8046

Method | Views | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
---|---|---|---|---
Sync- Dreamer | 1 | 19.46 | 0.847 | 0.188
LEAP[10] | 1 | 14.66 | 0.47 | 0.43
2 | 16.22 | 0.59 | 0.36
4 | 16.54 | 0.61 | 0.35
6 | 16.65 | 0.62 | 0.34
8 | 16.79 | 0.63 | 0.34
10 | 16.84 | 0.64 | 0.34
Ours | 1 | 20.25 | 0.862 | 0.157
2 | 21.73 | 0.872 | 0.137
4 | 23.44 | 0.886 | 0.117
6 | 24.54 | 0.894 | 0.107
8 | 24.92 | 0.899 | 0.102
10 | 25.03 | 0.899 | 0.102

Figure 7: Novel view synthesis and 3D reconstruction with sparse-view input
images. Left: a qualitative example of novel view synthesis, comparing LEAP
[10] and MVDiffusion++ with different numbers of unposed input images. Right:
qualitative comparison of reconstructed meshes between NeuS [35] with ground-
truth relative poses and our pose-free MVDiffusion++.

Views | CD $\downarrow$ | IoU$\uparrow$
---|---|---
8 | 0.0262 | 0.6020
16 | 0.0216 | 0.6532
24 | 0.0208 | 0.6657
32 | 0.0208 | 0.6689

Figure 8: Ablation study on different numbers of generated views. Left: a
qualitative comparison shows that denser views produce mesh with better
quality. Right: quantitative results of 3D reconstruction using different
numbers of generated views, evaluated on 30 GSO [4] objects. Figure 9: Text-
to-3D application examples. (Top) A text-to-image model generates an image
given a text-prompt. (Bottom) MVDiffusion++ turns the generated image into a
3D model. Please see our project website for more results:
https://mvdiffusion-plusplus.github.io. Figure 10: Failure examples from the
Text-to-3D application.

Sparse-view un-posed input images is a challenging setting, where we are aware
of only a few existing approaches such as LEAP [10] and PF-LRM [36], a sparse-
view pose-free extension of LRM [9]. There is no public implementation of PF-
LRM, and we pick LEAP as the first baseline. The literature on multi-view 3D
reconstruction is extensive. It would be valuable to contrast our approach,
even though they require camera poses as input. As a compromise, we have
selected NeuS [35] as our second benchmark by providing the ground-truth
camera poses as their input.

$\bullet$ LEAP leverages a transformer to predict neural volumes of radiance
fields from a sparse number of views and is also pose-free. LEAP employs
DINOv2 [24] as the feature extractor and has reasonable generalization
capacity.

$\bullet$ NeuS is a 3D reconstruction method, where we provide the ground-
truth camera poses of the condition images as well as surface normals
estimated by Omnidata’s monocular normal estimator [5]. We use the public
grid-based NeuS implementation [6]. This baseline is similar to MonoSDF [43]
or NeurIS [34] equipped with ground-truth foreground masks and camera poses,
thus sets a performance upper bound for methods without generative priors.

Results. Table 2 and Figure 7 present the quantitative and qualitative
comparison results, respectively. Compared to LEAP, MVDiffusion++ generates
images with much better quality. LEAP and our method both exploit multi-view
self-attention to establish global 3D consistency. Therefore, we attribute our
better performance to the strong image priors inherited from the pre-trained
latent diffusion models. Our reconstructed meshes outperform NeuS in most
settings, a notable achievement considering that NeuS uses ground-truth camera
poses. This comparison highlights the practicality of our method, enabling
users to achieve high-quality 3D models from just a few object snapshots.

Ablation study on the number of generated views. Figure 8 varies the number of
generated images. The 32 views are divided into four elevation-based groups
with the order 30∘, 0∘, -30∘, 60∘. The 8-view setting only generates the first
group, and the 16-view setting generates the first two groups, and so on. The
results suggest that fewer views cannot cover the entire object, leading to
worse reconstruction quality. While the metric scores are almost the same
between 24 views and 32 views, the mesh reconstructions from 32 views look
smoother without artifacts.

###  6.3 Text-to-3D application

MVDiffusion++shows consistent performance with minimal errors on the GSO
dataset. Note that our training data solely comes from Objaverse dataset [3],
and MVDiffusion++already achieves remarkable generalization capabilities. To
further challenge the system, we demonstrate a text-to-3D application, where a
text-to-image model prepares an input condition image. MVDiffusion++ turns the
condition image into a 3D model. Figure 9 has four examples demonstrating the
power of our approach.

##  7 Limitations and future challenges

This paper presents a pose-free technique for reconstructing objects using an
arbitrary number of images. Central to this approach is a sophisticated multi-
branch, multi-view diffusion model. This model processes any number of
conditional images to produce dense, consistent views from fixed perspectives.
This capability significantly enhances the performance of existing
reconstruction algorithms, enabling them to generate high-quality 3D models.
Our results show that MVDiffusion++ sets a new standard in performance for
both single-view and sparse-view object reconstruction.

Figure 10 presents typical failure modes and the limitations of our approach.
Our method struggles with thin structures as in the leftmost example, which
fails to reconstruct a cable. Our method occasionally generates implausible
images for views occluded in the input, a notable instance being the depiction
of a cat with two tails. These shortcomings are predominantly attributed to
the lack of training data, where one future work will expand the framework to
incorporate videos, which offer richer contextual and spatial information,
potentially enabling dynamic video generation.

## References

  * [1] Dao, T.: Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691 (2023)
  * [2] Dao, T., Fu, D., Ermon, S., Rudra, A., Ré, C.: Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems 35, 16344–16359 (2022)
  * [3] Deitke, M., Schwenk, D., Salvador, J., Weihs, L., Michel, O., VanderBilt, E., Schmidt, L., Ehsani, K., Kembhavi, A., Farhadi, A.: Objaverse: A universe of annotated 3d objects. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13142–13153 (2023)
  * [4] Downs, L., Francis, A., Koenig, N., Kinman, B., Hickman, R., Reymann, K., McHugh, T.B., Vanhoucke, V.: Google scanned objects: A high-quality dataset of 3d scanned household items. In: ICRA (2022)
  * [5] Eftekhar, A., Sax, A., Malik, J., Zamir, A.: Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10786–10796 (2021)
  * [6] Guo, Y.C.: Instant neural surface reconstruction (2022), https://github.com/bennyguo/instant-nsr-pl
  * [7] He, Z., Wang, T.: Openlrm: Open-source large reconstruction models. https://github.com/3DTopia/OpenLRM (2023)
  * [8] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems 33, 6840–6851 (2020)
  * [9] Hong, Y., Zhang, K., Gu, J., Bi, S., Zhou, Y., Liu, D., Liu, F., Sunkavalli, K., Bui, T., Tan, H.: Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.04400 (2023)
  * [10] Jiang, H., Jiang, Z., Zhao, Y., Huang, Q.: Leap: Liberate sparse-view 3d modeling from camera poses. arXiv preprint arXiv:2310.01410 (2023)
  * [11] Jun, H., Nichol, A.: Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02463 (2023)
  * [12] Li, Z., Müller, T., Evans, A., Taylor, R.H., Unberath, M., Liu, M.Y., Lin, C.H.: Neuralangelo: High-fidelity neural surface reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8456–8465 (2023)
  * [13] Lin, S., Liu, B., Li, J., Yang, X.: Common diffusion noise schedules and sample steps are flawed. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 5404–5411 (2024)
  * [14] Liu, M., Shi, R., Chen, L., Zhang, Z., Xu, C., Wei, X., Chen, H., Zeng, C., Gu, J., Su, H.: One-2-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. arXiv preprint arXiv:2311.07885 (2023)
  * [15] Liu, M., Xu, C., Jin, H., Chen, L., Xu, Z., Su, H.: One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. arXiv preprint arXiv:2306.16928 (2023)
  * [16] Liu, R., Wu, R., Van Hoorick, B., Tokmakov, P., Zakharov, S., Vondrick, C.: Zero-1-to-3: Zero-shot one image to 3d object. In: ICCV (2023)
  * [17] Liu, Y., Lin, C., Zeng, Z., Long, X., Liu, L., Komura, T., Wang, W.: Syncdreamer: Generating multiview-consistent images from a single-view image. arXiv preprint arXiv:2309.03453 (2023)
  * [18] Long, X., Guo, Y.C., Lin, C., Liu, Y., Dou, Z., Liu, L., Ma, Y., Zhang, S.H., Habermann, M., Theobalt, C., et al.: Wonder3d: Single image to 3d using cross-domain diffusion. arXiv preprint arXiv:2310.15008 (2023)
  * [19] Melas-Kyriazi, L., Laina, I., Rupprecht, C., Vedaldi, A.: Realfusion: 360deg reconstruction of any object from a single image. In: CVPR (2023)
  * [20] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. In: ECCV (2020)
  * [21] Mittal, P., Cheng, Y.C., Singh, M., Tulsiani, S.: Autosdf: Shape priors for 3d completion, reconstruction and generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 306–315 (2022)
  * [22] Murray, N., Marchesotti, L., Perronnin, F.: Ava: A large-scale database for aesthetic visual analysis. In: 2012 IEEE conference on computer vision and pattern recognition. pp. 2408–2415. IEEE (2012)
  * [23] Nichol, A., Jun, H., Dhariwal, P., Mishkin, P., Chen, M.: Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.08751 (2022)
  * [24] Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)
  * [25] Qian, G., Mai, J., Hamdi, A., Ren, J., Siarohin, A., Li, B., Lee, H.Y., Skorokhodov, I., Wonka, P., Tulyakov, S., et al.: Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors. arXiv preprint arXiv:2306.17843 (2023)
  * [26] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10684–10695 (2022)
  * [27] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)
  * [28] Salimans, T., Ho, J.: Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512 (2022)
  * [29] Shi, R., Chen, H., Zhang, Z., Liu, M., Xu, C., Wei, X., Chen, L., Zeng, C., Su, H.: Zero123++: a single image to consistent multi-view diffusion base model. arXiv preprint arXiv:2310.15110 (2023)
  * [30] Shi, Y., Wang, P., Ye, J., Long, M., Li, K., Yang, X.: Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308.16512 (2023)
  * [31] Stereopsis, R.M.: Accurate, dense, and robust multiview stereopsis. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 32(8) (2010)
  * [32] Tang, S., Zhang, F., Chen, J., Wang, P., Furukawa, Y.: Mvdiffusion: Enabling holistic multi-view image generation with correspondence-aware diffusion. arXiv preprint arXiv:2307.01097 (2023)
  * [33] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [34] Wang, J., Wang, P., Long, X., Theobalt, C., Komura, T., Liu, L., Wang, W.: Neuris: Neural reconstruction of indoor scenes using normal priors. In: European Conference on Computer Vision. pp. 139–155. Springer (2022)
  * [35] Wang, P., Liu, L., Liu, Y., Theobalt, C., Komura, T., Wang, W.: Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In: NeurIPS (2021)
  * [36] Wang, P., Tan, H., Bi, S., Xu, Y., Luan, F., Sunkavalli, K., Wang, W., Xu, Z., Zhang, K.: Pf-lrm: Pose-free large reconstruction model for joint pose and shape prediction. arXiv preprint arXiv:2311.12024 (2023)
  * [37] Wang, Y., Lira, W., Wang, W., Mahdavi-Amiri, A., Zhang, H.: Slice3d: Multi-slice, occlusion-revealing, single view 3d reconstruction. arXiv preprint arXiv:2312.02221 (2023)
  * [38] Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. TIP (2004)
  * [39] Wu, T., Zhang, J., Fu, X., Wang, Y., Ren, J., Pan, L., Wu, W., Yang, L., Wang, J., Qian, C., et al.: Omniobject3d: Large-vocabulary 3d object dataset for realistic perception, reconstruction and generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 803–814 (2023)
  * [40] Yan, X., Yang, J., Yumer, E., Guo, Y., Lee, H.: Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision. Advances in neural information processing systems 29 (2016)
  * [41] Yang, Z., Ren, Z., Bautista, M.A., Zhang, Z., Shan, Q., Huang, Q.: Fvor: Robust joint shape and pose optimization for few-view object reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2497–2507 (2022)
  * [42] Yao, Y., Luo, Z., Li, S., Fang, T., Quan, L.: Mvsnet: Depth inference for unstructured multi-view stereo. In: Proceedings of the European conference on computer vision (ECCV). pp. 767–783 (2018)
  * [43] Yu, Z., Peng, S., Niemeyer, M., Sattler, T., Geiger, A.: Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. Advances in neural information processing systems 35, 25018–25032 (2022)
  * [44] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR (2018)