# Nematicity Liquid in a Trimerized-Kagome Antiferromagnet

Ichiro Tanaka and Hirokazu Tsunetsugu  The Institute for Solid State Physics
The Institute for Solid State Physics The University of Tokyo The University
of Tokyo Kashiwanoha 5-1-5 Kashiwanoha 5-1-5 Chiba 277-8581 Chiba 277-8581
Japan Japan

###### Abstract

We theoretically study low-temperature properties of the antiferromagnetic
Heisenberg model with half-integer spin $S$ on the kagome lattice with large
trimerization. We have derived a low-energy effective model for general $S$ in
terms of spin and nematicity operators in triangular units, and studied their
low-temperature correlations for $S$=$3/2$ by classical Monte Carlo
simulations. The previous study for the $S$=$1/2$ case [M. Ferrero et al.,
Phys. Rev. B 68, 214431 (2003) ] reported a spin liquid state at low
temperatures driven by a glassy behavior of isolated dimers and trimers of
nematicities. The results for $S$=$3/2$ show that nematicity dimers and
trimers are connected by weak links to form a defective planar network. At
very low temperatures, nematicities show glassy slow dynamics, and cluster
spins continue to fluctuate in a nonperiodically frozen background of
nematicities. The characteristic time of glassy dynamics scales with
temperature following a power law.

Heisenberg antiferromagnets on the kagome lattice is the representative
frustrated spin system, and the possibility of quantum spin liquid ground
state has been intensively studied for many decades both theoretically and
experimentally, but its conclusion remains under debate.[1, 2] For its better
understanding, it is useful to study a related system with frustration more
tamed, and we study in this letter the model on a trimerized kagome lattice,
which is also called breathing kagome. This system is also interesting in
itself due to a few instances of its realization. The vanadate
(NH4)2[C7H14N][V7O6F18] (DQVOF) has a trimerized kagome network of V4+ ions
with spin $S$=1/2, and a spin liquid behavior is observed.[3, 4] There is also
a proposal of an optical lattice with trimerized kagome geometry.[5] Recently,
Ishii et al. synthesized Li2Cr3SbO8, in which Cr3+ ions with $S$=3/2 form a
trimerized kagome network, and observed a $\sqrt{3}$$\times$$\sqrt{3}$
magnetic order and plateaus in its magnetization curve.[6, 7] A few
theoretical studies discussed that its ground state is a spin liquid.[8, 5, 9,
10]

One of the authors studied the Heisenberg antiferromagnets on a breathing
pyrochlore lattice.[11, 12] An important difference from kagome lattices lies
in the parity of the number of spins in unit cell. Since the pyrochlore
lattice has a tetrahedron unit, the parity is even. Its ground states have
degeneracy $2S$$+$$1$, and they are all spin-singlet irrespective of spin $S$.
By contrast, the parity is odd for the kagome lattice. For integer $S$, each
triangular unit has a unique ground state. Therefore the ground state of the
whole system is a valence bond crystal, and no symmetry is broken. The case of
half-integer $S$ is interesting and the ground state in each unit has
degeneracy 4$=$2$\times$2, i.e. two spin doublets.[14] This additional
nonmagnetic 2-fold degrees of freedom is related to the point group symmetry
and let us call it nematicity, which will be explained more explicitly later.
The issue to study is to see how these cluster spins and nematicities in the
triangular units develop correlations at low temperatures. Ferrero et al.
investigated this problem for the $S$=1/2 case.[8] Performing classical Monte
Carlo (MC) simulations for its low-energy Hamiltonian, they observed a glassy
behavior of nematicity at very low temperatures. Note that they used the term
“chirality”, which is synonymous with our nematicity. In this letter, we will
study how the low temperature physics changes for higher $S$. We will
construct an effective Hamiltonian for general half integer $S$ and perform
classical MC simulations.

The model to study is the trimerized-kagome antiferromagnetic Heisenberg
(TKAFH) Hamiltonian with half-integer spin $S$

$\tilde{H}=J_{0}\sum_{\bm{R}}\sum_{1\leq i<j\leq
3}\\!\\!\\!\\!\bm{S}_{i}(\bm{R})\cdot\bm{S}_{j}(\bm{R})+J_{1}\sum_{\bm{R}}\sum_{j=1}^{3}\bm{S}_{j}(\bm{R})\cdot\bm{S}_{j+1}(\mbox{$\bm{R}$+$\bm{a}_{j}$}),$
(1)

where $J_{0}$ and $J_{1}$ denote the nearest-neighbor exchanges in triangles
pointing up and down respectively as shown in Fig. 1(a). We will consider the
case of strong trimerization $0$$<$$J_{1}$$\ll$$J_{0}$ in this letter. In the
following, let us call up-pointing triangles clusters. The sum on $\bm{R}$ is
taken over clusters, which constitute a triangular lattice with the lattice
vectors $\bm{a}_{1}$ and $\bm{a}_{2}$. $\bm{S}_{j}(\bm{R})$ denotes the spin
on the sublattice $j$ shown in Fig. 1(b) with the convention
$\bm{S}_{j+3}(\bm{R})$=$\bm{S}_{j}(\bm{R})$, and define the cluster spin by
$\bm{S}(\bm{R})$$\equiv$$\bm{S}_{1}(\bm{R})$+$\bm{S}_{2}(\bm{R})$+$\bm{S}_{3}(\bm{R})$.

Figure 1: (Color online) (a) Trimerized kagome lattice, aka breathing kagome.
With lattice constant $a$, $\bm{a}_{1}$=$a(1/2,-\\!\sqrt{3}/2)$,
$\bm{a}_{2}$=$a(1/2,\\!\sqrt{3}/2)$, and
$\bm{a}_{3}$=$-$$\bm{a}_{1}$$-$$\bm{a}_{2}$. (b) Sublattice indices in unit
cell. (c) Brillouin zone.

In the decoupled limit $J$=0, the ground state is a direct product of cluster
ground states. Each cluster has 4-fold degenerate ground states
$|\Phi^{p}(M)\rangle$, ($p$=$\pm$, $M$=$\pm 1/2$), for any half-integer spin
$S$. They are spin doublet with $S^{z}$=$M$, and eigenstates of
$(\mbox{$\bm{S}_{1}$$+$$\bm{S}_{3}$})^{2}$ with eigenvalue
$(\mbox{$S$+$p/2$})(\mbox{$S$+$p/2$+1})$. This doublet $|\Phi^{\pm}\rangle$
constitutes a basis of E-irreducible representation of the C3v point group of
the triangular unit, and therefore describes breaking of the lattice symmetry.
For later use, it is convenient to use the chiral basis:
$|\Phi_{L,\,R}(M)\rangle$=
$2^{-1/2}\bigl{[}|\Phi^{+}(M)\rangle$$\,\pm\,$$i\,|\Phi^{-}(M)\rangle\bigr{]}$.
After lengthy calculations of Clebsh-Gordan coefficients and 6-$j$
symbols,[13] we can represent the sublattice spin operators in terms of this
chiral basis within the ground state manifold as

${\bm{S}}_{j}(\bm{R})\rightarrow\zeta\,\bm{S}(\bm{R})\,\bigl{[}\mbox{$\gamma$$+$$\bm{d}_{j}$$\cdot$$\bm{\tau}(\bm{R})$}\bigr{]},\
\ \bm{d}_{j}=(\cos\omega_{j},\sin\omega_{j}),$ (2)

where $3\omega_{3}$=$-3\omega_{1}$=$\omega_{2}$=$\pi$,
$\zeta$=$(\mbox{$2S$$+$$1$})/3$ and $\gamma$=$1/(\mbox{$2S$$+$$1$})$$<$$1$.
Note that while $\bm{S}_{j}(\bm{R})$ are spin-$S$ operators, the cluster ones
$\bm{S}(\bm{R})$ are spin-1/2. $\bm{\tau}$=$(\tau_{1},\tau_{2})$ are Pauli
matrices operating in the chiral space $(\Phi_{L},\Phi_{R})$ at each unit
$\bm{R}$. These operators constitute a basis set of E-irreducible
representation of the triangular point group, and describe breaking of this
symmetry. Therefore, we will call them nematicity in this letter. Note that
the chirality is represented by the other operator
$\tau_{3}$=$i\tau_{2}\tau_{1}$.[11]

The low-energy effective Hamiltonian of this system is derived by projecting
$\tilde{H}$ into the macroscopically degenerate ground states of the $J_{0}$
term. Dropping a constant term, it is the $J_{1}$ term represented with the
projected operators (2)

$\displaystyle H=J\sum_{\bm{R}}\sum_{j=1}^{3}\
\bm{S}(\bm{R})\cdot\bm{S}(\mbox{$\bm{R}$+$\bm{a}_{j}$})\
\Gamma(\bm{R},\mbox{$\bm{R}$+$\bm{a}_{j}$}),\ \ \
(\mbox{$J$=$\zeta^{2}$$J_{1}$})$ (3)
$\displaystyle\Gamma(\bm{R},\mbox{$\bm{R}$+$\bm{a}_{j}$})\equiv\bigl{[}\mbox{$\gamma$$+$$\bm{d}_{j}$$\,\cdot\,$$\bm{\tau}(\bm{R})$}\bigr{]}\bigl{[}\mbox{$\gamma$$+$$\bm{d}_{j+1}$$\cdot\,$$\bm{\tau}($}\mbox{$\bm{R}$+$\bm{a}_{j}$})\bigr{]}.$
(4)

This effective Hamiltonian was previously obtained for the $S$=$1/2$ case by
Mila.[14] It is important that the Hamiltonian does not change its form for
higher $S$’s,[15] but the value of $\gamma$ varies in addition to the
renormalization of the energy unit $J$. We will show later that the change of
$\gamma$ qualitatively modifies the correlations of $\bm{\tau}$’s from the
$S$=1/2 case. This approach is based on the first order perturbation, and
justified as far as the renormalized energy scale $J$ is small enough compared
with the energy gap $J_{0}$ in the decoupling limit.

Figure 2: (Color online) (a) Internal energy and (b) specific heat of the
effective model $H$ for $S$=3/2. Values are normalized per site, i.e. three
spins in $\tilde{H}$. Error bars are estimated by the bootstrap method with
95% percentile.

In order to examine how their correlations evolve with lowering temperature
$T$, we have investigated the effective model $H$ by classical MC simulations.
Namely, spins $\\{\bm{S}(\bm{R})\\}$ and nematicities
$\\{\bm{\tau}(\bm{R})\\}$ are treated as classical unit vectors with 3 and 2
components, respectively. In the following, we use $J$ as the units of energy
and set $J$=1. The previous study for $S$=$1/2$ used a similar approximation
but treated $\\{\bm{\tau}(\bm{R})\\}$ as 3-component classical vectors.[8]
However, this difference does not change results qualitatively, since
nematicities hardly point to the third direction at low $T$. Simulated systems
are rhombus parts of the triangular lattice with edges parallel to
$\bm{a}_{1}$ and $\bm{a}_{2}$, and periodic boundary conditions are imposed.
The number of sites $N$ is up to 36$\times$36, and we use the standard
Metropolis algorithm with local updates. Physical quantities are measured
typically for $10^{6}$ Monte Carlo steps (MCS) following $10^{6}$ MCS of
annealing for each temperature, and they are averaged over 128 different
ensembles.

We first calculated thermodynamic quantities to check a possibility of phase
transition. Since the spin part of $H$ is isotropic, we do not expect a
finite-temperature phase transition in the spin space. However, since the
nematicity interactions are anisotropic, a possibility of phase transition
remains. Figures 2(a) and (b) plot internal energy $E(T)$ and specific heat
$C(T)$, respectively, of the effective Hamiltonian (3). The $C(T)$ curve gives
no indication of diverging peak down to the very low temperature
$T$$\sim$$10^{-6}$. The internal energy provides no indication of jump either.
These results imply that no phase transition occurs, neither first-order or
continuous one, and the lowest-temperature state remains disordered. In
contrast to the results in Ref. 8, $C(T)$ exhibits no peak, but it is common
that the nearly flat part has a value $\sim$1.5, although their $\bm{\tau}$
has 3 components. We will discuss this point later.

Figure 3: (Color online) (a) Spin correlations $C_{S}(\bm{k})$ and (b)
nematicity correlations $C_{\tau}(\bm{k})$ of the effective model $H$ for
$S$=$3/2$ and $N$=36$\times$36\. For comparison, $C_{S}(\bm{k})$ for $S$=$1/2$
is also calculated and shown in (a).

We next investigate correlations of spins and nematicities. Two-dimensional
systems with continuous internal symmetry often undergo a phase transition at
$T$=0, and in such a case a related correlation function shows a divergent
temperature dependence.[16] Define spin and nematicity correlation functions
as $C^{\mu\nu}_{O}(\bm{k})$$\equiv$$N^{-1}\sum_{\bm{R},\bm{R}^{\prime}}\langle
O_{\mu}(\bm{R})O_{\nu}(\bm{R}^{\prime})\rangle$
$e^{-i\bm{k}\cdot(\bm{R}-\bm{R}^{\prime})}$, ($O$=$S$, $\tau$) and let
$C_{S}(\bm{k})$=$\sum_{\mu}C^{\mu\mu}_{S}(\bm{k})$ and $C_{\tau}(\bm{k})$
denote the larger eigenvalue of the latter one. Note that measured $\langle
S_{\mu}(\bm{R})S_{\nu}(\bm{R}^{\prime})\rangle$ is proportional to the
identity matrix within statistical errors. We have calculated $C_{S}(\bm{k})$
for all $\bm{k}$’s, and plot the results along the high symmetry axes in Fig.
3(a). It has a broad peak at the Brillouin zone corners (K and K’) at all
$T$’s, which indicates antiparallel or a 120∘-type spin correlation between
nearest neighbor sites. The peak does not grow or narrow with lowering
temperature. This sharply contrasts with the behavior of the growing 120∘
structure in the antiferromagnetic Heisenberg model on the triangular
lattice.[16, 17]

Figure 4: (Color online) (a) Snapshot of nematicity correlations
$\Gamma(\bm{R},\bm{R}^{\prime})$ for the $S$=$3/2$ case at $T$=$10^{-5}$. The
system size is $N$=36$\times$36\. (b) Pair distribution
$P(\theta_{0},\theta_{1})$ calculated at $T$=$10^{-5}$ with averaging over
6144 ensembles. (c) Distribution function of bond and site energies for the
snapshot (a). Each peak is broadened by a Gaussian with width $0.002$ and
$0.005$ for site and bond energy, respectively.

Nematicity correlation function $C_{\tau}(\bm{k})$ is plotted in the same way
in Fig. 3 (b). Compared to the spin correlations, the peak at K point is more
prominent. However, it remains broad and its width and height barely change at
low temperatures. These results clearly demonstrate that the effective
Hamiltonian (3) exhibits no tendency of long-range order in not only spin but
also nematicity spaces. In this sense, the system remains in the liquid phase
of spin and nematicity.

The previous study for the $S$=$1/2$ case pointed out that this low-
temperature state has a unique character in their local nematicity
correlations.[8] To quantify this, let us examine for each bond
$\Gamma(\bm{R},\mbox{$\bm{R}$+$\bm{a}_{j}$})$ defined in Eq. (4), which is
bounded as
$\Gamma_{-}$$\leq$$\Gamma(\bm{R},\bm{R}^{\prime})$$\leq$$\Gamma_{+}$ with
$\Gamma_{-}$=$-(\mbox{1$-$$\gamma^{2}$})$$<0$ and
$\Gamma_{+}$=$(\mbox{$1$+$\gamma$})^{2}$. Each site is connected to 6 bonds,
and MC snapshots for the $S$=$1/2$ case showed a unique imbalance in $\Gamma$
among these 6 bonds.[8] At majority of sites, one and only one bond has a
nonvanishing value $\approx\Gamma_{+}$, and those bonds were called dimers. At
most of the remaining sites, $\tau$’s form trimers, which are called triangles
in the previous study[8]. Three sites are connected by three bonds with
$\Gamma_{+}$ to form a triangular unit, while $\Gamma$=0 for the remaining 12
bonds.

For the $S$=$3/2$ case, a MC snapshot of nematicity correlations
$\Gamma(\bm{R},\bm{R}^{\prime})$ is plotted in Fig. 4(a). It exhibits
similarities and important differences compared with the $S$=$1/2$ case. Bonds
with large $\Gamma$ value are shown by thick blue line, and the most
noticeable similarity is that most of the sites belong to a dimer or a trimer.
An important difference is that many of the remaining bonds have a sizable
$\Gamma$, and let us call them weak links. Thus, dimers and trimers are not
isolated but connected by these weak links to form a defective planar network
with modulated bond strengths mixed with small disconnected blobs such as
unglued dimers and hexamers.

Spins fluctuate on this network with modulated effective exchange couplings
$\\{\Gamma(\bm{R},\bm{R}^{\prime})\\}$, which are predominantly
antiferromagnetic. Since the network extends over the whole system, spins
establish long-range correlations, and this is manifested by
$C_{S}(\bm{k})$$\sim$$|\bm{k}|$ around $\Gamma$ point as shown in Fig. 3(a).
This nonanalyticity indicates a power-law scaling of long-range asymptotics in
envelope
$|\langle\mbox{$\bm{S}(\bm{R})$$\cdot$$\bm{S}(\bm{0})$}\rangle|$$\sim$$\Lambda^{1/2}R^{-5/2}$,
where $\Lambda$ is the wavenumber cutoff. By contrast, the $S$=1/2 result is
smooth, $C_{S}(\bm{k})$$\sim$$\bm{k}^{2}$, and this is a consequence of short-
range spin correlations confined within isolated dimers and trimers.

Let us investigate the origin of these local nematicity correlations at low
$T$. Local energy (3) is a product of $\bm{S}$$\cdot$$\bm{S}^{\prime}$ and
$\Gamma$. Thus lowering energy requires either
$\bm{S}$$\cdot$$\bm{S}^{\prime}$$\sim$$+1$ and $\Gamma$$\sim$$\Gamma_{-}$, or
$\bm{S}$$\cdot$$\bm{S}^{\prime}$$\sim$$\,-1$ and $\Gamma$$\sim$$\Gamma_{+}$,
and the latter is more effective since $|\Gamma_{-}|$$<$$\Gamma_{+}$.
Therefore, many bonds realize
$\Gamma(\bm{R},{\mbox{$\bm{R}$$+$$\bm{a}_{j}$}})$$\sim$$\Gamma_{+}$ and this
requires $\bm{\tau}(\bm{R})$$\approx$$\bm{d}_{j}$ and
$\bm{\tau}({\mbox{$\bm{R}$$+$$\bm{a}_{j}$}})$$\approx$$\bm{d}_{j+1}$. This
constrains $\Gamma$ on the other bonds connected to the site $\bm{R}$ or
$\bm{R}$$+$$\bm{a}_{j}$. This is because
$\bm{\tau}(\bm{R})$$\cdot$$\bm{d}_{j\pm 1}$$\approx$$-$$1/2$ and this enforces
$\Gamma(\bm{R},{\mbox{$\bm{R}$$+$$\bm{a}_{j\pm
1}$}})$$\sim$$($$\gamma$$-$1/2$)^{2}$, i.e. 0 for $S$=$1/2$. This explains the
configuration of isolated nematicity dimers in the $S$=$1/2$ case.[8] The same
argument predicts for our $S$=$3/2$ case that nematicity dimers are weakly
connected by all the other bonds with $\Gamma$$\sim$$1/16$, but the observed
snapshot exhibits different correlations. This is clearly seen in the
distribution function Fig. 4(b) of angle pairs $P(\theta_{0},\theta_{1})$,
where $\theta_{0}$ and $\theta_{1}$ are the angle of $\bm{\tau}(\bm{R})$ and
$\bm{\tau}({\mbox{$\bm{R}$$+$$\bm{a}_{1}$}})$, respectively. The above
argument predicts a sharp peak at
$(\theta_{0},\theta_{1})$=$(\omega_{1},\omega_{2})$=$(-\pi/3,\pi)$. However,
the calculated results show that the peak splits with a finite deviation. We
will continue analysis to clarify this point.

Figure 5: (Color online) Nematicity correlations in some degenerate ground
states: $(\delta\theta_{0},\delta\theta_{1})$= (a)($+$$\Delta$, $-$$\Delta$),
(b)($+$$\Delta$, $+$$\Delta$), (c)($-$$\Delta$, $-$$\Delta$), (d)($-$$\Delta$,
$+$$\Delta$). (e) Mixed with a trimer. (f) Three-sublattice structure. In
(a)-(e), strong bonds with $\Gamma_{s}$ are shown by blue solid line, while
weak links with $\Gamma_{w}$ by green dotted line. While the configurations
(a)-(c) are periodic, (d) is a local realization including bonds with
$\Gamma$=$-(3/4)^{2}$ shown in red.

To obtain the lowest-energy configuration of spins and nematicities for the
$S$=$3/2$ case, we have performed numerical annealing using a rhombus part
with $N$=6$\times$6\. The initial configuration is the one where nematicity
dimers are periodically aligned with
$(\theta_{0}^{(0)},\theta_{1}^{(0)})$=$(-\pi/3,\pi)$ and a random
configuration is set for spins. The annealed result is again a periodic
pattern with
$(\theta_{0}^{*},\theta_{1}^{*})$=$(\theta_{0}^{(0)},\theta_{1}^{(0)})$+$(\delta\theta_{0},\delta\theta_{1})$
shown in Fig. 5(a), but the bonds shown by thin line have $\Gamma$=0 within
numerical accuracy. This means that
$\delta\theta_{0}$=$-$$\delta\theta_{1}$=$\Delta$ with
$\Delta$=$\cos^{-1}\\!\gamma$$-$$\pi/3$$\approx$0.0862$\pi$, and this leads to
$\Gamma_{s}$= $(3\phi/4)^{2}$$\approx$1.4726 for dimer and $\Gamma_{w}$=
$(3/4\phi)^{2}$$\approx$0.2149 for weak link, where
$\phi$$\equiv$$(\mbox{$\sqrt{5}$$+$1})/2$ is the golden ratio. Spin
configuration is Néel like along each chain of coupled dimers, but has no
correlation between different chains. The corresponding energy is
$E_{0}^{*}$=$-27/32$=$-0.84375$ per site. The lowest-energy configuration is
degenerate, even within periodic ones. Figure 5 (b) and (c) show some of them
with a doubled unit cell, and the weak links are now along $\bm{a}_{2}$ and
$\bm{a}_{3}$.

These straight chains of weakly connected dimers do not exhaust the degeneracy
of the lowest-energy configuration. First, these chains can bend with no
energy cost. Another variant is the one shown in Fig. 5(d) in which weak links
connect differently with $\Gamma$$<$0\. Secondly, one can mix a trimer and it
changes the direction of dimers as shown in Fig. 5 (e). Thirdly, there also
exist two-dimensional configurations, and a typical example is a 120∘ state
shown in Fig. 5(f). This is a state with three sublattices A-C, where cluster
spins and nematicities exhibit an ordinary 120∘ order with
$\theta_{A}$$-$$\theta_{B}$$\equiv$$\theta_{B}$$-$$\theta_{C}$$\equiv$$2\pi/3$
modulo $2\pi$, and this state has a continuous degeneracy in both spaces.

Thus, the lowest-energy configuration has very large degrees of freedom,
supposedly a macroscopic entropy as for the $S$=1/2 case,[14] but we leave its
estimate for a future study. The snapshot in Fig. 4(a) is one example. We plot
in Fig. 4(c) the integrated distribution of bond and site energies in this
snapshot. Site energy is defined as half of the sum of the related 6 bond
energies. Bond energies have a large variance and range from $-1.5601$ to
$0.3883$. Many of them are around zero, which means energetically disconnected
bonds, i.e., missing bonds in the defective network. In contract, site
energies have only a small variance, and its maximum and minimum differ by
only 0.0125. Its average value is $\bar{E}_{\rm site}$=$-0.84374$, almost
identical to $E_{0}^{*}$ considering the effect of nonzero temperature.

Figure 6: (Color online) Semi-log plot of glass susceptibility of
nematicities. (a) System size dependence. (b) Dependence on $M_{t}$. Data for
$M_{t}$=$10^{6}$ are a different set from that in (a). (c) Scaling of the
freezing temperature $T_{f}$ defined by the midpoint of each slope in (b).

Now, let us come back to the discussion of the low-$T$ value of $C$. Since
$\bm{S}$ and $\bm{\tau}$ are both classical unit vectors, Dulong-Petit law
predicts that each of their angles contributes 1/2 to $C$, and this explains
the value $\sim$1.5 in Fig. 2(b). By contrast, for $S$=1/2, $C$=3/2 was found
in Ref. 8 using 3-component $\bm{\tau}$, and our calculation using 2-component
$\bm{\tau}$ found $C$=1. In both cases, 1/2 is missing and this is because
half the spin degrees of freedom are zero modes in dimers and trimers.[8] The
value in Fig. 2(b) is slightly smaller than 3/2, and this reduction is also
due to spin zero modes in disconnected blobs mixed in the defective network.
Precise counting of zero modes requires analysis of density and shapes of
isolated blobs together with analysis of local spin structure in each blob.
This is another challenge to be studied in future.

Lastly, let us discuss the glassy behavior of nematicities, which was reported
for the $S$=1/2 case,[8] while spin glass behaviors in undistorted kagome has
been discussed much longer.[18] At very low $T$, a crossover takes place into
a state where nematicities are frozen and only spin degrees of freedom
fluctuate subject to the frozen pattern of effective exchange couplings.[8,
19, 20] This crossover is signaled by the nematicity glass susceptibility

$\chi_{g}\equiv\frac{1}{N}\sum_{\bm{R},\bm{R}^{\prime}}\left[\frac{1}{M_{t}}\sum_{t=1}^{M_{t}}\bm{\tau}_{t}(\bm{R})\cdot\bm{\tau}_{t}(\bm{R}^{\prime})\right]^{2}$
(5)

where $\bm{\tau}_{t}(\bm{R})$ is the variable at Monte-Carlo step $t$. Note
that $\chi_{g}$ takes a value of O$(N)$ in states where nematicities do not
fluctuate with “time” $t$, which include both regularly ordered and glassy
states. Since the possibility of regularly ordered state is already excluded
by the absence of sharp peak in $C_{\tau}(\bm{k})$, a nonvanishing value of
$\chi_{g}/N$ means a glassy behavior of nematicity. Figure 6 shows its
temperature dependence for various values of system size and $M_{t}$. Panel
(a) shows that $\chi_{g}/N$ grows noticeably at $T$$\lesssim$$10^{-3}$, and
this is an evidence of freezing to a glassy state. The system size dependence
is very small except in the region of lowest temperatures, and the value
increases with $N$ there. This indicates that this glassy behavior survives in
the $N$=$\infty$ limit, but it is premature to conclude this.

Figure 6(b) shows the results with varying measurement time $M_{t}$. With
increasing $M_{t}$, freezing starts at a lower temperature, and this agrees
with the previous study for the $S$=1/2 case.[8] Each time multiplying $M_{t}$
by the factor 10, the whole curve moves with a nearly constant shift towards
the lower-temperature side in this semi-log plot. This implies the
characteristic freezing temperature scales as $T_{f}$$\sim$$M_{t}^{-\eta}$,
and the exponent is estimated in Fig. 6(c) as $\eta$$\approx$0.483. Therefore,
the defective network of nematicity has slow Monte Carlo dynamics, and its
time scale $t_{f}$ grows with lowering temperature as
$t_{f}$$\sim$$T^{-1/\eta}$. We have also calculated the results for the
$S$=1/2 case. The difference from the $S$=3/2 case is only quantitative and
the exponent is smaller $\eta$$\approx$0.184, while the temperature dependence
was claimed activation type in the previous study.[8] This nonuniversal
exponent may imply that it is only a transient behavior but this scaling holds
well down to the lowest temperatures in our simulation.

In this letter, we have theoretically studied the trimerized kagome
antiferromagnetic Heisenberg model with half integer spin $S$ in the region of
large trimerization. For general $S$, we constructed its low-energy effective
Hamiltonian in terms of spin and nematicity of each triangular unit. We then
performed its classical Monte Carlo simulations for the $S$=3/2 case and
studied its low temperature properties. We found that nematicities form local
dimers and trimers as for the $S$=1/2 case, but they are weakly coupled to
form a defective planar network mixed with disconnected small blobs.
Nematicities also show glassy slow Monte Carlo dynamics, and its
characteristic time scale grows with temperature following a power law.
Concerning spin interactions, they are modulated in space according to the
pattern of defective network of nearly frozen nematicities, and many bonds are
effectively broken. Spin correlations grow at low temperatures but exhibit no
periodic pattern detected by a sharp peak in $C_{S}(\bm{k})$. Therefore, the
low-energy state is a strongly correlated classical spin liquid on a defective
network of nematicities with glassy slow dynamics. Since nematicity represents
breaking of the triangular symmetry of each unit, it couples to local lattice
distortion. It is worthwhile to note that one can detect these nematicities in
principle by measuring E-mode of local lattice distortion.

The major part of the numerical calculations were performed at the
Supercomputer Center, ISSP, at the University of Tokyo.

## References

  * [1] G. Misguich and C. Lhuillier, in “Frustrated Spin Systems”, (ed.) H. T. Diep, (World Scientific, 2004).
  * [2] P. Mendels and A. S. Wills, in “Introduction to Frustrated Magnetism”, (ed.) C. Lacroix, P. Mendels, and F. Mila, (Springer, 2010) and references therein.
  * [3] L. Clark, L.-C. Orian, F. Bert, M.A. De Vries, F.H. Aidoudi, R.E. Morris, P. Lightfoot, J.S. Lord, M.T.F. Telling, P. Bonville, J.P. Attfield, P. Mendels, and A. Harrison, Phys. Rev. Lett. 110, 207208 (2013).
  * [4] L.-C. Orain, B. Bernu, P. Mendels, L. Clark, F.H. Aidoudi, P. Lightfoot, R.E. Morris, and F. Bert, Phys. Rev. Lett. 118, 237203 (2017).
  * [5] L. Santos, M. A. Baranov, J. I. Cirac, H.-U. Everts, H.Fehrmann, and M. Lewenstein, Phys. Rev. Lett. 93, 030601 (2004).
  * [6] K. Iida, J-PARC MLF Experimental Report, Project No. 2017A0143 (2017).
  * [7] Yuto Ishii, Doctoral Thesis submitted to Hokkaido University, 2020.
  * [8] M. Ferrero, F. Becca, and F. Mila, Phys. Rev. B 68, 214431 (2003). Note that the direction of $\bm{d}$ vectors differ from ours since they chose different wave function bases depending on the cluster position.
  * [9] R. Schaffer, Y. Huh, K. Hwang, and Y.B. Kim, Phys. Rev. B 95, 054410 (2017).
  * [10] M. Iqbal, D. Poilblanc, and N. Schuch, Phys. Rev. B 101, 155141 (2020).
  * [11] H. Tsunetsugu, J. Phys. Soc. Jpn. 70, 640 (2001); H. Tsunetsugu, Phys. Rev. B 65, 024415 (2001).
  * [12] H. Tsunetsugu, Prog. Theor. Exp. Phys. 2017, 033101 (2017).
  * [13] A. Messiah, Quantum Mechanics, Vol. II, Appendix C, (North-Holland, 1962).
  * [14] F. Mila, Phys. Rev. Lett. 81, 2356 (1998).
  * [15] Reference 8 remarked that the same type of effective Hamiltonian was derived for general half integer $S$ by M. Mambrini, but we have found no related publication.
  * [16] H. Kawamura and S. Miyashita, J. Phys. Soc. Jpn. 53, 4138 (1984).
  * [17] H. Kawamura, A. Yamamoto, and T. Okubo, J. Phys. Soc. Jpn. 79, 023701 (2010).
  * [18] P. Chandra, P. Coleman, and I. Ritchey, J. de Phys. I 3, 591 (1993).
  * [19] F. Mila and D. Dean, Eur. Phys. J. B 26,301 (2002).
  * [20] O. Cépas and B. Canals, Phys. Rev. B 86, 024434 (2012).


# Who Responded to Whom: The Joint Effects of Latent Topics and Discourse in
Conversation Structure

First Author
Affiliation / Address line 1
Affiliation / Address line 2
Affiliation / Address line 3
email@domain
Second Author
Affiliation / Address line 1
Affiliation / Address line 2
Affiliation / Address line 3
email@domain
Lu Ji1, Jing Li2, Zhongyu Wei3, Qi Zhang1, Xuanjing Huang1
1School of Computer Science, Fudan University, Shanghai, China
2Department of Computing, The Hong Kong Polytechnic University, Hong Kong,
China
3School of Data Science, Fudan University, China
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Numerous online conversations are produced on a daily basis, resulting in a
pressing need to conversation understanding. As a basis to structure a
discussion, we identify the responding relations in the conversation
discourse, which link response utterances to their initiations. To figure out
who responded to whom, here we explore how the consistency of topic contents
and dependency of discourse roles indicate such interactions, whereas most
prior work ignore the effects of latent factors underlying word occurrences.
We propose a model to learn latent topics and discourse in word distributions,
and predict pairwise initiation-response links via exploiting topic
consistency and discourse dependency. Experimental results on both English and
Chinese conversations show that our model significantly outperforms the
previous state of the arts, such as $79$ vs. $73$ MRR on Chinese customer
service dialogues. We further probe into our outputs and shed light on how
topics and discourse indicate conversational user interactions.

##  1 Introduction

The growing popularity of online platforms have resulted in the revolution of
interpersonal communications. Individuals now engage in diverse forms of
online conversations to exchange viewpoints and share ideas. It allows users
to access an abundance of fresh materials, whereas the explosive growth of
online texts — essentially conversational and usually in multiple threads Wang
and Rosé (2010) — has also hindered human capability to find the information
needed. There consequently presents a pressing need to develop conversation
understanding methods to digest massive texts and complex interactions
therein. To that end, it is crucial to capture the interactions of who
responded to whom — the base to build and understand the conversation
structure, as pointed out in many previous studies Wang and Rosé (2010). By
reflecting how participants interact with each other, such structure has shown
useful to predict users’ online social activities Zeng et al. (2019b),
summarize key discussion topics Li et al. (2018), measure argument
persuasiveness Ji et al. (2018), and so forth.

[$C_{1}$] I am aware that you can thank them in private argument but what does
that matter?
---
[$C_{2}$] The most important part of my argument is that it hurts literally
nobody.
[$C_{3}$] All they are doing is trying to be polite.
[$C_{4}$] Some people gild comments anonymously and do not respond to the
private messages, so the gildee never knows who gave them gold.
[$C_{5}$] Note: for the purposes of my argument, assume I am talking about
comments edited in such a way as to say thanks for the gold!
[$R$] We are all aware that you can do that, but sometimes people like to
express gratitude publicly.
Figure 1:  A Reddit conversation snippet. $C_{1}$ and $R$ is an initiation-
response pair while $C_{2}$ to $C_{5}$ are the other four candidates. Topic
words reflecting the discussion points “public gratitude expression” are in
bold. The blue and italic “that” occurring in both $C_{1}$ and $R$ imply $R$’s
possible intention to answer $C_{1}$’s question.

To date, despite of the extensive efforts on user interaction modeling, many
of them employ user-annotated in-reply-to signals, such as @-mention on
Twitter Li et al. (2018); Zeng et al. (2019b). Nonetheless, such labels are
usually unavailable or unreliable Du et al. (2017); He et al. (2019),
especially for online conversations in informal styles. Other studies assume
utterances only respond to their chronological neighbors Jiao et al. (2018);
Zhao et al. (2018), largely ignoring the long-distance interactions prominent
in online conversations Wang and Rosé (2010). All these concerns lay down our
objective to investigate who responded to whom in conversation contexts.

Following previous practice Schegloff (2007), we define our task to predict
pairwise initiation utterances and their responses in an online conversation
(henceforth initiation-response pairs), where an initiation sets up an
expectation earlier and its response later react to it in process of a
discussion. To illustrate our task, Figure 1 shows an example response $R$ and
the other five utterances $C_{1}$ to $C_{5}$ from $R$’s previous post in a
Reddit conversation. Our goal is to identify which utterance from $C_{1}$ to
$C_{5}$ is $R$’s initiation. As can be seen, $R$ is most likely to respond to
$C_{1}$ for two possible reasons: First, they both focus on the topic of
public gratitude expression (as topic words “thank”, “public”, “gratitude” are
mentioned); Second, $C_{1}$ raises a question (signaled by “what” and the
question mark “?”) that can be well answered by $R$ (via echoing the pronoun
“that”).

Here, we examine two latent factors that implicitly link an initiation and its
response — the consistency of the topics they center around (henceforth topic
consistency) and the dependency of their discourse roles (henceforth discourse
dependency).111Discourse roles refer to utterance-level dialogue acts, such as
asking a question and making an argument Ritter et al. (2010).  Our intuition
is that responses tend to follow the points pushed forward in their
initiations (such as public gratitude expression in Figure 1) and their
discourse roles are likely to exhibit dependency in interactions, such as an
answer responding to an initiated question (like $R$ answering $C_{1}$ in
Figure 1) and an argument followed by another argument in a back-and-forth
debate. To the best of our knowledge, _we are the first to analyze the effects
of topics and discourse in conversational responding behavior_ , while
previous work predict initiation-response pairs without modeling such latent
factors embedded in the relations Du et al. (2017).

To learn topics and discourse, we separate two word distributions for
representing each of them. The latent variables are inferred with a neural
architecture in an unsupervised manner Zeng et al. (2019a), which enables
topic and discourse inference without either manually annotated data Zhao et
al. (2017) or expertise involvements to customize model inference Li et al.
(2018). Afterwards, two neural modules are employed, one to capture topic
consistency and the other discourse dependency, both aim to explore the
implicit links of a response and a candidate initiation. The learned
representations are hence coupled to predict how likely the two utterances
form an initiation-response pair.

In an empirical study, we conduct extensive experiments on two conversation
datasets, one contains English argumentative discussions on Reddit (from the
ChangeMyView subreddit), and the other Chinese customer service dialogues from
e-commerce platform Wangwang. Both of them will be released upon publication
as part of our work. The experimental results show that our model
significantly outperforms state-of-the-art methods from previous work. For
example, we achieve $79.02$ MRR on the Wangwang dialogues compared with
$72.69$ produced by He et al. (2019). In extensive analyses on latent topics
and discourse, we find that meaningful representations can be learned by our
model and both topics and discourse may contribute to indicate initiation-
response pairs. Lastly, we show that our learned representation to indicate
initiation-response relations can benefit to identify persuasive arguments in
social media debates.

##  2 Study Design

###  2.1 Task Formulation

We define initiation-response pairs following Schegloff (2007) and refer both
initiations and responses to conversation utterances from different
participants. In a discussion flow, responses appear and react to the points
raised earlier in their initiations and hence hold responding relations with
them. Here, we formulate our task into pairwise ranking following previous
settings Wang and Rosé (2010) and will experiment on two types of initiation-
response pairs: quotation-reply pairs in forum discussions and question-answer
pairs in customer service dialogues. Specifically, given a response utterance
$r$, we rank a set of candidate utterances with one positive initiation
$q^{+}$ and $u$ negative ones $q_{1}^{-}\sim q_{u}^{-}$. In practice, we
measure a matching score $S(q,r)$ to indicate the likelihood of $q$ as $r$’s
initiation and the one with the highest score will be considered as $r$’s
predicted initiation.

###  2.2 Data Collection and Analysis

Two conversation datasets are collected for experiments — a CMV dataset in
English from the CMV subreddit and the other in Chinese named as CS from
Wangwang customer service platform. CS dialogues are synchronous while CMV
asynchronous.

#### Data Collection.

CMV gathers social media arguments, whose raw data is released by Tan et al.
(2016). For each discussion, we examine the context of an opinion holder
(OH)’s post and a challenger’s comment to explore the quotation-reply
relations therein. In challenger’s comments, we form a quotation and the
utterance right after it to be an initiation-response pair. The rest
utterances in the quoted post (from OH) are used as the negative instances,
and the samples are randomly selected with a cap at $4$ to avoid unbalanced
labels.

CS is released with He et al. (2019). In a dialogue thread, customers may
raise multiple questions sequentially and the seller’s answers may appear in
the following turns. Our goal is to pair a question from the customer’s
utterances and an answer from the seller’s. The newest $4$ consecutive
utterances from the customer (skipping the positive initiation) before a
seller’s response serve as the negative instances. Here the candidate number
is also capped at $4$ for comparable results with CMV.

| CMV Dataset | CS Dataset
---|---|---
# of utt. per conv | 21.2$\pm$15.6 | 9.6$\pm$2.8
# of words per conv | 403.1$\pm$292.5 | 130.8$\pm$73.1
# of convs | 7,937 | 4,277
# of words per $r$ | 19.7$\pm$6.0 | 15.0$\pm$20.8
# of words per $q^{+}$ | 20.6$\pm$6.2 | 6.5$\pm$4.3
# of words per $q^{-}$ | 16.5$\pm$5.0 | 11.2$\pm$18.7
max # of pairs | 14 | 7
avg. # of pairs | 1.1$\pm$0.3 | 1.7$\pm$1.1
Table 1:  Data statistics. Means and standard deviations appear before and
after $\pm$. $r$ refers to response, while $q^{+}$ and $q^{-}$ for positive
and negative initiation. # of pairs represents the number of initiation-
response pairs per conversation.

#### Data Analysis.

Table 1 shows the data statistics, where the two datasets exhibit different
characteristics. We further analyze the relative positions of initiations and
responses and show the distribution of their intermediate utterance number in
Figure 2. As can be seen, large proportion of responses do not interact with
the closest utterance, though CS sellers do respond more to newer questions,
probably because of recency effects in in synchronous dialogues — people’s
attention tends to be drawn by new information. However, in asynchronous forum
discussions, CMV challengers are more likely to quote the opening points in
OH’s post. Another possible reason is that most key arguments are located at
the beginning of a post.

Figure 2: The distribution over relative positions of initiations and
responses. X-axis: initiations’ utterance order counted from responses (only
considering customer’s or OH’s turns). Y-axis: proportions.

##  3 Learning Topics and Discourse Effects for Initiation-Response Prediction

Figure 3:  (a) Our model architecture to predict initiation-response pairs. We
first learn latent topics and discourse factors for both response $r$ and the
candidate initiation $q$ in award of their contexts $c_{r}$ and $c_{q}$ and
show the detailed learning process in (b) ($x$ denotes $q$ or $r$.) Then,
utterance matching is conducted to measure topic consistency and discourse
dependency. Lastly, we predict $S(q,r)$ — the likelihood of $r$ responding to
$q$.

The overall architecture of our model is shown in Figure 3 (a). It takes an
initiation candidate $q$, a response $r$, and their corresponding contexts
$c_{q}$ and $c_{r}$ as inputs. The outputs are matching scores indicating how
likely $r$ responds to $q$. The model mainly consists of two modules — one
jointly models latent topics and discourse (to be described in Section 3.1) in
both $q$ and $r$ (both denoted by utterance $x$ and its context by $c$) and
the other predict initiation-response relations coupling topic consistency and
discourse dependency (to be covered in Section 3.2). The learning objectives
will be presented at last in Section 3.3.

###  3.1 Latent Topics and Discourse Modeling

Inspired by previous efforts in neural topic models Miao et al. (2017), we
adopt variational autoencoder (VAE) Kingma and Welling (2013) to learn latent
topics and discourse. It allows their associated word distributions to be
learned in neural architecture and end-to-end training with other components
in a deep learning framework. The corresponding networks are illustrated in
Figure 3 (b). In below, we first describe how we model the topics, followed by
the process to learn discourse.

#### Latent Topics.

We first assume there are $K$ latent topics in the corpus, each represented by
a word distribution $\Phi_{k}^{T}(k=1,2,...,K)$ over the vocabulary $V$. The
latent topics of each utterance is defined as $z$ and generated from the topic
composition of its context $c$. Here we learn utterance-level topics in its
conversation context assuming that utterances in a discussion excerpt tend to
focus on similar topics. It allows the modeling of rich patterns of word
statistics for topic inference.

The following process presents how to generate an utterance $x$ in context of
$c$. Here, we adopt the bag-of-words assumption of most latent topic models
Blei et al. (2002); Miao et al. (2017) and generate $x$ in its bag of words
(BoW) form $x^{BoW}$.

  * •

Draw the latent topic $z\sim N(\mu,\sigma^{2})$

  * •

$c$’s topic mixture $\theta=softmax(f_{\theta}(z))$

  * •

For the $n$-th word in $x$:

    * –

$\beta_{n}=softmax(f_{\Phi^{T}}(\theta))$

    * –

Draw the word $w_{n}\sim Multi(\beta_{n})$

where $f_{*}(\cdot)$ is a neural perceptron. The weight matrix of
$f_{\Phi^{T}}(\cdot)$ (after the softmax normalization) is viewed as the
topic-word distributions $\Phi^{T}$.

The prior parameters $\mu$ and $\sigma$ are estimated from conversation $c$’s
bag of words $c^{BoW}$:

$\mu=f_{\mu}(f_{e}(c^{BoW})),\,\log\sigma=f_{\sigma}(f_{e}(c^{BoW}))$ (1)

$f_{\mu}$, $f_{e}$ and $f_{\sigma}$ are neural perceptron defined above.

As can be seen, the entire topic modeling process follows a VAE fashion — for
each utterance $x$, we first encode its latent topic $z$ from the conversation
context $c$ (in BoW form $c^{BoW}$) and then reconstruct its BoW ($x^{BoW}$)
via decoding.

#### Latent Discourse.

Similar to latent topics, we represent latent discourse with word
distributions $\Phi_{d}^{D}$ ($d=1,2,...,D$) and $D$ denotes the number of
discourse roles observed from the corpus.

Following Ritter et al. (2010), we assume each utterance $x$ reflects only one
discourse role $d$ (to signal its dialogue act). It is hence represented by a
$D$-dimensional one-hot vector over the discourse inventory (the high bit
indicates $x$’s discourse role). To learn latent discourse, we adopt the
similar VAE-based process as topic modeling with both the input and output as
utterance $x$’s BoW ($x^{BoW}$). First, $x^{BoW}$ is encoded into its latent
discourse role $d$ with the following formula:

$\pi=gs(f_{\pi}(x^{BoW})),\,d=Multi(\pi)$ (2)

where $gs$ refers to Gumbel softmax function Lu et al. (2017) to encode the
discrete nature of latent discourse $d$ and $f_{\pi}$ is another neural
perceptron. Afterwards, the decoding process reconstructs $x^{BoW}$
conditioned on $d$ with another fully connected layer:

$x^{BoW}=f_{\Phi^{D}}(d)$ (3)

Here similar to latent topics, we utilize $f_{\Phi^{D}}$’s weights to compute
discourse-word distributions.

###  3.2 Initiation-Response Pair Prediction

Given topic and discourse representations of a response $r$ ($z_{r}$ and
$d_{r}$) and those of its candidate initiation $q$ ($z_{q}$ and $d_{q}$), we
further predict how likely they form an initiation-response pair with an
utterance matching process. Here we measure the effects of topic consistency
and discourse dependency to indicate initiation-response relations.

For topic consistency, we capture how similar the topics of $q$ and $r$ is
with the following score:

$S_{topic}(q,r)=z_{r}^{T}W_{t}z_{q}$ (4)

where $W_{t}$ is a weight matrix learned to indicate the importance of each
topic factor.

Likewise, $q$ and $r$’s discourse-level matching score is denoted as
$S_{discourse}$ and defined below:

$S_{discourse}(q,r)=d_{r}^{T}W_{d}d_{q}$ (5)

where the trainable weight matrix $W_{d}$ is employed to capture the
transition probabilities from $q$’s discourse role to $r$’s
($Pr(d_{r}\,|\,d_{q})$).

Further, to yield the final matching score $S(q,r)$ to estimate how likely $r$
responding to $q$, we leverage $S_{topic}(q,r)$ and $S_{discourse}(q,r)$ to
couple both topic and discourse effects with the weighted sum:

$S(q,r)=\gamma S_{topic}(q,r)+(1-\gamma)S_{discourse}(q,r)$ (6)

where $\gamma\in[0,1]$ is the parameter balancing the relative contributions
of topic and discourse.

###  3.3 Learning Objectives

Here we describe the learning objectives of the entire framework to jointly
explore latent topic and discourse, and their combined effects to predict
initiation-response pairs.

#### Latent Topics and Discourse Modeling Loss.

We employ neural variational inference to approximate the posterior
distributions over the latent topic $z$ and the latent discourse $d$.

Encoding Topics and Discourse. To examine how to learn topics and discourse,
the cross entropy loss is used to reflect the estimation of $z$ and $d$ from
encoding process:

$L_{t}=E_{q(z\,|\,c)}[\log p(c\,|\,z)]-KL(q(z\,|\,c)\,||\,p(z))$ (7)
$L_{d}=E_{q(d\,|\,x)}[\log p(x\,|\,d)]-KL(q(d\,|\,x)\,||\,p(d))$ (8)

$KL$ cost term is added to avoid posterior collapse. For space limitation, we
leave out the derivation details and refer the readers to Zhao et al. (2018).

Reconstructing Utterances. For the reconstruction loss to reflect how an
utterance can be inferred from $z$ and $d$, we define the loss $L_{x}$ as:

$L_{x}=E_{q(z\,|\,x)q(d\,|\,c)}[\log p(x\,|\,z,d)]$ (9)

Distinguishing Topics and Discourse. As discussed above, topics and discourse
are modeled in different granularity (discourse in utterance only while topics
in richer contexts). To further distinguish their respective word
distributions, we follow Zeng et al. (2019a) to employ the mutual information
to define the mutual dependency of latent topics and discourse:

$E_{q(z)q(d)}[\log\frac{p(z,d)}{p(z)p(d)}]$ (10)

The mutual information loss is adopted to separate the semantic space of
topics and discourse:222All distributions in Eq. 10 and 11 are conditional
distribution given utterance $x$ and its conversation context $c$ and the
conditions are omitted for simplicity.

$L_{MI}=E_{q(z)q(d)}[{KL}(p(d\,|\,z)\,||\,p(d))]$ (11)

#### Initiation-Response Pair Prediction Loss.

To allow positive pairs to obtain higher scores than negative, we use hinge
loss in training:

$L_{m}=\sum_{i=1}^{u}max(0,\lambda-S(q^{+},r)+S(q_{i}^{-},r))$ (12)

where $u$ is the number of negative initiations for each response. $\lambda$
is a margin parameter and $S(q^{+},r)$ and $S(q_{i}^{-},r)$ are the matching
scores of a response and its positive and negative initiations.

#### The Final Objective.

Finally, we combine all the effects above and define the overall objective of
the entire model as:

$L=L_{t}+L_{d}+L_{x}+L_{m}-L_{MI}$ (13)

In the training process, the optimization of final objective $L$ enables the
end-to-end exploration of topic and discourse representation and their joint
effects to signal pairwise initiation-response relations in conversation
structure.

##  4 Experimental Setup

#### Data Preprocessing.

For CMV dataset, the raw data was preprocessed by Tan et al. (2016). We first
filter out tokens occurring less than $15$ maintain a vocabulary with $15,182$
tokens. Then, we remove utterances with word length less than $7$ or over
$45$. Next, to form context for quotations and replies ($c_{q}$ and $c_{r}$),
we consider all utterances in the original post (from OH) as $c_{q}$ and those
in the challenger’s comment as $c_{r}$. Lastly, the training and test data is
separated following Tan et al. (2016), where $6,839$ pairs are used for
training and and $1,098$ for test.

For CS dataset, we don’t remove words and the vocabulary size is $15,407$,
with the scale similar to CMV. Short utterances with less than $5$ words are
removed. The Chinese word segmentation and the separation of training and test
set has been done by He et al. (2019), with $3,701$ and $576$ instances for
training and test. Here all utterances in the dialogue thread are used to form
both $c_{q}$ and $c_{r}$ due to the synchronous nature of CS conversations.

For both datasets, $10\%$ data is further sampled from the training set for
validation.

#### Model Settings.

The hyperparameters are tuned on validation set. For the number of topics
($K$) and discourse roles ($D$), we set $K=50,D=5$ for CMV dataset and
$K=10,D=3$ for CS. Max margin weight $\lambda$ is set to $10$ (Eq. 12) and
$\gamma=0.5$ for balancing topic consistency and discourse dependency (Eq. 6).
In model training, we set the batch size to $32$, dropout probability to
$0.5$, and the maximum epoch number to $200$ (with early stop). The trainable
parameters are optimized via stochastic gradient descent with learning rate
decay, whose initial learning rate is set to $0.1$.

#### Comparison Models.

We first consider three non-neural baselines that rank initiations based on:
1) Position, where earlier utterances are ranked higher for CMV while later is
higher for CS based on the findings from Figure 2; 2) Embedding_Sim — the
cosine similarity between a response and an initiation utterance measured by
the average word embeddings from Glove; 3) LDA_Disc — using cross entropy to
discriminate initiation’s and response’s topic distributions inferred by
latent Dirichlet allocation (LDA) Du et al. (2017).

We also compare with the following neural models proposed by previous work: 1)
MaLSTM Mueller and Thyagarajan (2016) designed for sentence-level semantic
matching (LSTM for utterance encoding and Manhattan distance for matching); 2)
CoAttention Ji et al. (2018) proposed for pairwise argument quality
evaluation, where a co-attention network learns alignment representations and
a BiGRU layer computes similarity for matching. 3) RPN He et al. (2019), the
state-of-the-art model for question-answer pairing in dialogues that ranks
initiations by recurrent pointer networks (RPN).

In addition, we consider the following models with a fully connected layer to
score initiation-response pairs and the following encoders for utterance-level
representation learning: RNN (henceforth Match_RNN), autoencoder (henceforth
Match_AE), variational autoencoder (henceforth Match_VAE), and discrete
variational autoencoder Zhao et al. (2018) (henceforth Match_DVAE).

Further, to study the relative contributions of topic consistency and
discourse dependency, we compare with our two ablations, one only explores the
topic effects (henceforth Topic_Only) and the other discourse (henceforth
Discourse_Only).

##  5 Results and Discussions

We first discuss the main comparison results in Section 5.1, followed by an
analysis of topic and discourse effects in Section 5.2. Lastly, in Section
5.3, we probe into our outputs and present a parameter analysis, case study,
and model extension results on argument persuasiveness prediction.

Models | CMV Dataset | CS Dataset
---|---|---
Hits@1 |  Hits@2 |  MRR |  Hits@1 |  Hits@2 |  MRR
Non-Neural Models |  |  |  |  |  |
Position |  24.68 |  24.68 |  24.68 |  49.13 |  49.13 |  49.13
Embedding_Sim |  22.77 |  45.00 |  48.66 |  17.01 |  39.06 |  44.04
LDA_Disc |  24.68 |  42.99 |  47.77 |  26.39 |  49.65 |  52.40
Neural Models |  |  |  |  |  |
MaLSTM |  29.87 |  42.99 |  50.91 |  43.58 |  72.40 |  65.80
CoAttention |  47.72 |  68.31 |  67.26 |  51.56 |  79.17 |  71.77
RPN |  46.45 |  67.21 |  66.22 |  52.95 |  80.21 |  72.69
Match_RNN |  49.45 |  71.58 |  68.79 |  50.00 |  80.38 |  71.13
Match_AE |  51.82 |  74.77 |  70.50 |  52.78 |  82.12 |  72.88
Match_VAE |  53.19 |  73.95 |  71.11 |  52.60 |  81.42 |  72.70
Match_DVAE |  47.45 |  69.95 |  67.34 |  53.82 |  82.81 |  73.65
Ablations |  |  |  |  |  |
Topic_Only |  58.20 |  76.14 |  73.78 |  42.53 |  69.10 |  64.11
Discourse_Only |  41.44 |  63.02 |  62.20 |  48.96 |  76.04 |  69.76
Our model |  59.74 |  76.23 |  74.41 |  64.93 |  84.20 |  79.23
Table 2: Comparison results on two datasets and our model achieves the best
results under all settings. Our model significantly outperforms the comparison
model (Wilcoxon signed rank test, _p_ $<$0.05).

###  5.1 Main Comparison Results

The overall results are shown in Table 2. Two widely-used information
retrieval metrics Hits@N, $N=1,2$ and Mean Reciprocal Rank (MRR) are used for
evaluation metrics. Several interesting observations can be drawn.

$\bullet$ All models yield generally better performance on CS than CMV. It
shows that initiation-response links are more difficult to be identified on
dialogues in argumentative than everyday styles.

$\bullet$ Neural networks perform better than non-neural baselines. Shallow
features from position, word embeddings, and LDA-based latent topics can’t
obtain good performance. Neural models explore deeper semantic features and
provide better results.

$\bullet$ Autoencoders learn useful representations. It is observed that
models based on autoencoders perform generally better than other models. This
shows that autoencoders are effective in encoding utterances compared with
other alternatives.

$\bullet$ Topics contribute more on CMV while discourse is more useful in CS.
Topic_Only performs much better than Discourse_Only on CMV, while the opposite
is observed on CS. It is probably because of the richer context in CMV to
learn latent topics (with more words per conversation as shown in Table 1),
while the synchronous CS dialogues exhibits richer discourse word patterns
from back and forth interactions between participants and hence allow better
discourse modeling.

$\bullet$ Our model significantly outperforms all comparisons. This shows that
the joint effects of topics and discourse can usefully indicate the relations
of initiations and responses in conversation context.

Figure 4: The distribution of topic similarity in the CMV dataset (a) and CS
(b). X-axis shows cosine similarity intervals and y-axis indicates
proportions. For each interval, positive pair results are displayed on the
left (in blue) and negative on the right (in orange).

###  5.2 Effects of Topics and Discourse

#### Topic Effects.

We first analyze the effects of topic consistency and compute the cosine
similarity of the latent topics we learn for responses ($z_{r}$) and candidate
initiations ($z_{q}$). The distributions over positive and negative pairs are
shown in Figure 4. For both datasets, our model generally assigns higher topic
similarity for positive pairs than negative, probably because responses tend
to follow the concern of initiations and are hence likely to contain similar
topic words. We also observe a proportion drop in very similar positive pairs
($sim>0.8$), indicating that most responses do not echo what were said in
initiations, though their topics might be similar. Nevertheless, negative
pairs exhibit different distributions compared with the positive ones. Our
model is able to capture such features in topic consistency modeling (Eq. 4),
which might help in distinguishing positive and negative initiations for a
response.

Figure 5: The transition distributions of discourse roles from initiations to
responses, CMV in (a) and CS in (b). Only the top $5$ transitions observed in
positive (on the left in blue) and negative pairs (on the right in orange) are
displayed. X-axis: initiation-response discourse roles ($d_{q}\rightarrow
d_{r}$); Y-axis: proportions.

#### Discourse Effects.

We discuss how discourse dependency affects the prediction of initiation-
response pairs. The transition distributions of discourse roles from
initiations to responses ($d_{q}\rightarrow d_{r}$) are shown in Figure 5. As
can be seen, the discourse transition distributions in CS dataset are diverse
for positive and negative pairs. It may help explain why discourse can better
signal initiation-response pairs on CS compared with CMV (observed from
Discourse_Only’s performance in Table 2). For CMV, there are slightly
different distributions for positive and negative pairs. For this reason,
topic factors may contribute more than discourse (seen via comparing
Discourse_Only and Topic_Only on CMV). This also indicates that discourse
modeling for argumentative dialogues is challenging, which may require the
learning of more complex features other than word statistics and is beyond the
capacity of our model.

###  5.3 Further Discussions

#### Parameter Analysis.

Here we investigate the impact of two hyper-parameters on our model, namely,
the number of topics ($K$) and discourse ($D$).

Varying Topic Number. Figure 6 (a) shows how Hits@1 scores change over varying
number of topics ($K$). For comparison, we also display Match_DVAE’s results,
the best comparison model in Table 2. For relatively large $K$, our model
performs consistently better than Match_DVAE. We also find that our trend on
both datasets are not monotonic, where the best performance is attained at
$K=50$ for CMV and $K=10$ for CS. This implies that the topics in customer
service dialogues are limited while participants may discuss wide range of
topics in social media debates.

Varying Discourse Number. The results for varying discourse number ($D$) are
displayed in Figure 6 (b). Similar to $K$, our model exhibits better results
than Match_DVAE for $D>1$. It’s also observed that CS is more sensitive to $D$
compared with CMV, indicating that discourse factors largely affect the
initiation-response prediction results on $CS$.

Figure 6: Hits@1 over varying number of topics and discourse X-axis: topic
number ($K$ in (a)) and discourse number ($D$ in (b)). Y-axis: Hits@1 score.
Blue and red curves: our model on CMV and CS. Purple and black lines:
Match_DVAE on CMV and CS.

#### Case Study.

To examine what we learn to represent topics and discourse, we take the CMV
conversation snippet in Figure 1 as an example to analyze the topic and
discourse words assigned by our model. Recall that $R$ answers $C_{1}$’s
question suggested by the shared pronoun “that” and the similar topics they
concern. Figure 7 shows the visualization results and displays topic words in
red and discourse in blue. It is observed that our model is able to separate
topic words (e.g., “thank”, “private”, and “public) from discourse (e.g.,
“that”, “what”, and “?”), which may result in coherent topic and discourse
distributions and indicative representations to signal initiations-response
relations. Interestingly, discourse words are mostly stop words and
punctuation. Their meaningful clusters exhibiting different statistic patterns
might usefully indicate varying discourse behaviors in conversations, which is
consistent with the findings from previous studies Li et al. (2018); Zeng et
al. (2019a).

Figure 7: Visualization of the topic and discourse word assignment for the CMV
conversation snippet in Figure 1. The blue words are prone to indicate
discourse ($p(w\,|\,d)>p(w\,|\,z)$) while red topic. Darker colors indicate
higher confidence.

#### Downstream Task.

In Introduction, we mentioned that the detection of initiation-response pairs
may contribute to a better understanding of conversation structure and hence
benefit downstream applications. We take the prediction of argument
persuasiveness as an example to discuss whether the representations learned by
our model can advance the state-of-the-art performance on this task. Table 3
shows the performance of the non-neural baseline Tan et al. (2016), the state-
of-the-art model Ji et al. (2018), and Ji et al. (2018) incorporating the
topic and discourse representations we learn ($z$ and $d$). The dataset is
collected from CMV and argument quality is labeled by $\Delta$ (given by OH to
indicate the successful persuasion). It is seen that the latent topics and
discourse learned to signal initiation-response relations can help to predict
argument quality, suggesting that the persuasiveness of arguments are closely
related to the structure of who respond to whom in argumentation processes.

##  6 Related Work

Our work is in the line with prior efforts to detect initiation-response
pairs. Wang and Rosé (2010) explore how topic features discovered via latent
semantic analysis work in this task, ignoring the effects of discourse roles.
However, our study shows that both topics and discourse are helpful to
identify who respond to whom in conversation structure. Other work Jamison and
Gurevych (2014); Du et al. (2017); Chen et al. (2017) focus on the design of
hand-crafted features (e.g., text similarity, pre-detected dialogue acts,
etc.). Compared with them, our model enables learning of deep semantic
features for topics consistency and discourse dependency, which can be
captured without the labor-intensive feature engineering process. Recently,
there exists an attention over how neural framework perform to identify
replying relations in conversation discourse Guo et al. (2018); He et al.
(2019). However, they ignore the effects of latent topics and discourse to
structure a conversation, which are studied here and shown useful to indicate
initiation-response relations in experiments.

We are also inspired by the previous approaches to discover latent topics and
discourse in conversations contexts. Many of them employ probabilistic
graphical models in LDA-fashion to explore word statistics Ritter et al.
(2010); Li et al. (2018); Zeng et al. (2018). These models use expertise
involvements to customize inference algorithms, whereas our architecture
allows end-to-end training of topic and discourse model together with other
components. We take the advantage of the recent progress to explore
conversation representations via variational autoencoders Miao et al. (2017);
Zhao et al. (2018); Zeng et al. (2019a), allowing to capture topic and
discourse factors in an unsupervised manner. However, their effects to signal
user interactions in conversation structure have never been studied before,
which is a gap our work fills in.

Models | Pairwise accuracy
---|---
Tan et al. (2016) (baseline) | 65.70
Ji et al. (2018) (SOTA) | 70.45
Ji et al. (2018)+Our model | 74.12
Table 3: The pairwise accuracy to predict argument persuasiveness. The results
in the first two rows were reported in their original paper. Our
representations help advance the state of the art (SOTA).

##  7 Conclusion

This work explores the effects of latent topics and discourse roles to signal
initiation-response relations. We first employ a VAE-based neural model to
capture topic and discourse representations in an unsupervised manner. Then,
topic consistency and discourse dependency are further exploited to predict
how likely an utterance responds to an initiation. Extensive experiments on
two datasets show that our model significantly outperform the previous state-
of-the-art models. Further analyses show that both topics and discourse are
useful to signal who respond to whom in conversation structure.

## References

  * Blei et al. (2002) David M Blei, Andrew Y Ng, and Michael I Jordan. 2002.  Latent dirichlet allocation.  In _Advances in neural information processing systems_ , pages 601–608.
  * Chen et al. (2017) Jun Chen, Chaokun Wang, Heran Lin, Weiping Wang, Zhipeng Cai, and Jianmin Wang. 2017\.  Learning the structures of online asynchronous conversations.  In _International Conference on Database Systems for Advanced Applications_ , pages 19–34. Springer.
  * Du et al. (2017) Wenchao Du, Pascal Poupart, and Wei Xu. 2017.  Discovering conversational dependencies between messages in dialogs.  In _Thirty-First AAAI Conference on Artificial Intelligence_.
  * Guo et al. (2018) Gaoyang Guo, Chaokun Wang, Jun Chen, and Pengcheng Ge. 2018.  Who is answering to whom? finding "reply-to" relations in group chats with long short-term memory networks.  In _Proceedings of the 7th International Conference on Emerging Databases_ , pages 161–171. Springer.
  * He et al. (2019) Shizhu He, Kang Liu, and Weiting An. 2019.  Learning to align question and answer utterances in customer service conversation with recurrent pointer networks.  In _Thirty-Third AAAI Conference on Artificial Intelligence_.
  * Jamison and Gurevych (2014) Emily Jamison and Iryna Gurevych. 2014.  Adjacency pair recognition in wikipedia discussions using lexical pairs.  In _Proceedings of the 28th Pacific Asia Conference on Language, Information and Computing_.
  * Ji et al. (2018) Lu Ji, Zhongyu Wei, Xiangkun Hu, Yang Liu, Qi Zhang, and Xuanjing Huang. 2018.  Incorporating argument-level interactions for persuasion comments evaluation using co-attention model.  In _Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018_ , pages 3703–3714.
  * Jiao et al. (2018) Yunhao Jiao, Cheng Li, Fei Wu, and Qiaozhu Mei. 2018.  Find the conversation killers: A predictive study of thread-ending posts.  In _Proceedings of the 2018 World Wide Web Conference on World Wide Web, WWW 2018, Lyon, France, April 23-27, 2018_ , pages 1145–1154.
  * Kingma and Welling (2013) Diederik P Kingma and Max Welling. 2013.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_.
  * Li et al. (2018) Jing Li, Yan Song, Zhongyu Wei, and Kam-Fai Wong. 2018.  A joint model of conversational discourse and latent topics on microblogs.  _Computational Linguistics_ , 44(4).
  * Lu et al. (2017) Jiasen Lu, Anitha Kannan, Jianwei Yang, Devi Parikh, and Dhruv Batra. 2017.  Best of both worlds: Transferring knowledge from discriminative learning to a generative visual dialog model.  In _Advances in Neural Information Processing Systems_ , pages 314–324.
  * Miao et al. (2017) Yishu Miao, Edward Grefenstette, and Phil Blunsom. 2017.  Discovering discrete latent topics with neural variational inference.  In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , pages 2410–2419. JMLR. org.
  * Mueller and Thyagarajan (2016) Jonas Mueller and Aditya Thyagarajan. 2016.  Siamese recurrent architectures for learning sentence similarity.  In _Thirtieth AAAI Conference on Artificial Intelligence_.
  * Ritter et al. (2010) Alan Ritter, Colin Cherry, and Bill Dolan. 2010.  Unsupervised modeling of twitter conversations.  In _Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 172–180. Association for Computational Linguistics.
  * Schegloff (2007) Emanuel A Schegloff. 2007.  _Sequence organization in interaction: A primer in conversation analysis I_ , volume 1.  Cambridge University Press.
  * Tan et al. (2016) Chenhao Tan, Vlad Niculae, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. 2016\.  Winning arguments: Interaction dynamics and persuasion strategies in good-faith online discussions.  In _Proceedings of the 25th international conference on world wide web_ , pages 613–624. International World Wide Web Conferences Steering Committee.
  * Wang and Rosé (2010) Yi-Chia Wang and Carolyn P Rosé. 2010.  Making conversational structure explicit: identification of initiation-response pairs within online discussions.  In _Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 673–676. Association for Computational Linguistics.
  * Zeng et al. (2019a) Jichuan Zeng, Jing Li, Yulan He, Cuiyun Gao, Michael R Lyu, and Irwin King. 2019a.  What you say and how you say it: Joint modeling of topics and discourse in microblog conversations.  _Transactions of the Association for Computational Linguistics_ , 7:267–281.
  * Zeng et al. (2018) Xingshan Zeng, Jing Li, Lu Wang, Nicholas Beauchamp, Sarah Shugars, and Kam-Fai Wong. 2018.  Microblog conversation recommendation via joint modeling of topics and discourse.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 375–385.
  * Zeng et al. (2019b) Xingshan Zeng, Jing Li, Lu Wang, and Kam-Fai Wong. 2019b.  Neural conversation recommendation with online interaction modeling.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 4632–4642.
  * Zhao et al. (2018) Tiancheng Zhao, Kyusong Lee, and Maxine Eskenazi. 2018.  Unsupervised discrete sentence representation learning for interpretable neural dialog generation.  _arXiv preprint arXiv:1804.08069_.
  * Zhao et al. (2017) Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017.  Learning discourse-level diversity for neural dialog models using conditional variational autoencoders.  _arXiv preprint arXiv:1703.10960_.


# SOFT-INTROVAE FOR CONTINUOUS LATENT SPACE IMAGE SUPER-RESOLUTION

###### Abstract

Continuous image super-resolution (SR) recently receives a lot of attention
from researchers, for its practical and flexible image scaling for various
displays. Local implicit image representation is one of the methods that can
map the coordinates and 2D features for latent space interpolation. Inspired
by Variational AutoEncoder, we propose a Soft-introVAE for continuous latent
space image super-resolution (SVAE-SR). A novel latent space adversarial
training is achieved for photo-realistic image restoration. To further improve
the quality, a positional encoding scheme is used to extend the original pixel
coordinates by aggregating frequency information over the pixel areas. We show
the effectiveness of the proposed SVAE-SR through quantitative and qualitative
comparisons, and further, illustrate its generalization in denoising and real-
image super-resolution.

Index Terms—  Introspective Variational AutoEncoder, super-resolution, latent
space

##  1 Introduction

Image super-resolution (SR) aims to enlarge the low-resolution (LR) images to
the larger desired high-resolution (HR) images. It is widely used in digital
display, broadcast and data compression/restoration. With various display
devices and data resolutions, a flexible arbitrary image SR model can adjust
different needs to produce image/videos with best visual experiences. Most
existing state-of-the-art SR methods [1, 2, 3, 4, 5, 6] either focus on fixed
super-resolution solutions (one model for one dedicated upsampling scenario),
or integral upsampling scales (2$\times$, 4$\times$ or 8$\times$). They result
in costly training efforts and imperfect image resolutions.

Instead, continuous image super-resolution [7, 8] provides arbitrary
image/video scaling with photo-realistic visual quality. The goal is to
discover the hidden latent space where the missing pixels can be estimated by
the continuous feature representation. The advantage is that it can adjust
size-varied display devices and reduce many training efforts when applying
out-of-distribution super-resolution tasks.

However, continuous image super-resolution tends to generate over-smooth
images and is sensitive to noise. In order to produce clean photo-realistic
super-resolution images, we propose Soft-introVAE for continuous latent space
image super-resolution (SVAE-SR). The novelty is to use an autoencoder to
discover the continuous image distribution space, where we condition the
continuous LR features for supervision. The reused encoder works adversarially
against the decoder for discriminating between real and generated samples. A
soft threshold function is utilized to replace the hard margin in the evidence
lower bound (ELBO). Furthermore, the positional encoding is embedded as the
frequency expansion to introduce more pixels for prediction. To sum up, our
key claims are 1) Soft-introVAE for arbitrary image super-resolution that can
measure the adversarial conditional distribution of SR and HR images for
reconstruction, and 2) a positional encoding scheme is modified to involve
more neighborhood pixels for estimation.

##  2 RELATED WORKS

Implicit neural representation. The idea of implicit neural representation is
to use multi-layer perceptron (MLP) to learn pixels or other signals from
coordinates. It has been widely used in 3D shape modeling [9, 10], surface
reconstruction [11, 12], novel view rendering [13, 14, 15, 16] and so on. For
instance, Mildenhall et al. [13] propose to map the camera poses to the pixel
values via a multi-layer MLP network. They use multiple-view images to
optimize the network for implicit feature representation. Instead of using a
voxel or point cloud, the implicit neural representation can 1) capture the
fine details of scenes for photo-realistic reconstruction, and 2) also emit
complex 3D representation as a small number of differentiable network
parameters.

Fig. 1: SVAE-SR. (a) At training, it uses an encoder (E), a decoder (D) and
the feature extractor (G) to learn the continuous latent space distribution.
Given the reference image as the conditional feature, the encoder works as a
discriminator to identify whether the SR distribution is close to the HR
distribution. The decoder learns the local implicit image function to have
arbitrary scales of upsampling. (b) At test time, SVAE-SR samples a random
vector in the trained latent space and combines it with the reference features
for high-frequency signal estimation. A final LIIF is trained to map the
continuous features to the estimated RGB pixels.

Image super-resolution. The goal of image super-resolution is to enlarge LR
images to the desirable resolutions with high visual quality. Depending on the
metrics of evaluation, it can categorize into distortion-based SR [1, 2, 3, 4,
17, 18] and perception-based SR [5, 6, 19]. For the former one, using deeper
neural networks with sophisticated designs usually leads to lower pixel
distortions. For example, EDSR [2] proposes a network using more convolutional
kernels and layers for optimization. RDN [4] proposes a residual dense network
to allow feature sharing. Most recently, attention [20] is also used in image
SR [17, 18] to involve more neighborhood pixels for estimation. For
perception-based SR, GAN [21] and VAE [22] are two major architectures used
for photo-realistic reconstruction. ESRGAN [5] proposes a patch based GAN
network to supervise the perceptual quality via a binary discriminator. SRVAE
[6] achieves the goal using a conditional VAE to minimize the distribution
divergence. However, all the methods are for fixed upsampling factors, like
2$\times$ and 4$\times$. To have an arbitrary super-resolution solution, LIIF
[7] proposes to use the Local Implicit Image Function (LIIF) to explore
continuous feature representation for SR. MetaSR [8] uses a Meta-Upscale
Module to weight the upsampling filters for prediction dynamically.

##  3 METHOD

Here, we describe the proposed Soft-Intro Variational AutoEncoder for
continuous latent space image Super-Resolution (SVAE-SR). SVAE-SR performs
continuous super-resolution by projecting image features into the latent
space, and it conditions the LR features with random coordinate space sampling
and transfers the new features to the decoder for reconstruction. The encoder
is reused as a discriminator to distinguish the distributions of HR (real) and
SR (fake) images.

### Overview.

To learn the ground truth HR images $\mathbf{Y}$, SVAE-SR is a feed-forward
network (shown in Figure 1) that reconstructs SR images $\mathbf{Y}^{\prime}$
from LR images $\mathbf{X}$ and reference images $\mathbf{R}$. It consists of
an encoder (E), a decoder (D) and a feature extractor (G). The encoder learns
the latent vector of the joint HR-Reference distribution. The decoder uses the
local implicit image function (LIIF) to transform the 2D features into pixel-
coordinate pairs for 2D grid based interpolation. The proposed positional
encoding scheme process the 2D grid by high-frequency coding (HFC) and low-
frequency coding (LFC) to split the frequency bands. The LFC is with the LR
features for low-frequency component reconstruction and the HFC is with the
latent features for high-frequency reconstruction.

### Soft-IntroVAE.

Overall, SVAE-SR works as a soft-IntroVAE to explicitly extract the latent
distribution for pixel reconstruction. Different from existing CNN/GAN based
SR methods [1, 5], using VAE [6] has been proven to be useful for robust
photo-realistic quality restoration for real image SR. Soft-IntroVAE [23]
demonstrates its outstanding performance in image generation. By combining the
advantages of VAE and GAN, it adversarially optimizes the encoder as an
introspective discriminator. It fits the task of image SR that it can model
the pixel correlations as multivariate Gaussian distributions $z\sim
Q_{\omega}(z|\mathbf{X})=\mathit{N}(z;\mu_{i},\sigma^{2}_{i}\mathit{I})$. We
further modify it to a conditional Soft-IntroVAE model that it uses a
reference image as the condition to learn the missing high-frequency signal.
Formally, we give the mathematical description of the proposed conditional
Soft-IntroVAE as,

$\begin{matrix}\begin{split}&L_{E_{\phi}}(\mathbf{Y}|\mathbf{R},z)=ELBO(\mathbf{Y}|\mathbf{R})-\frac{1}{\alpha}exp\left(\alpha
ELBO(D_{\theta}(z))\right)\\\
&L_{D_{\theta}}(\mathbf{Y}|\mathbf{R},z)=ELBO(\mathbf{Y}|\mathbf{R})+\gamma
ELBO(D_{\theta}(z))\\\ &where\quad
ELBO(\mathbf{Y}|\mathbf{R})=\mathbb{E}_{z\sim
Q(z|\mathbf{Y},\mathbf{R})}[logP_{\theta}(\mathbf{Y},\mathbf{R}|z)]\\\
&\quad\quad\quad-D_{KL}[Q_{\phi}(z|\mathbf{Y},\mathbf{R})||P(z)]\leq
logP_{\theta}(\mathbf{Y}|\mathbf{R})\end{split}\end{matrix}$ (1)

In Eq (1), it can be seen that the process includes two steps: 1) fix the
decoder and optimize the encoder to distinguish through the ELBO value,
between HR images (high ELBO) and SR images (low ELBO) and 2) fix the encoder
and optimize the decoder to “fool” the encoder with photo-realistic SR images.
In such a way, we can push the distribution of the SR data close to the HR
data. Note that the natural advantage of using Soft-IntroVAE is that we use a
soft exponential function over the ELBO to improve the training stability.
Meanwhile, using a reference image as the condition to learn the joint
$p(\mathbf{Y}|\mathbf{R})$ probability so that the matched features from the
reference images can be extracted for aiding reconstruction.

For clarification, the detailed structure of the encoder and decoder is shown
in Figure 2. For the encoder, it takes the reference, HR/SR image and 2D grid
as inputs. A to pixel sampler converts them to pixel-coordinate 1D vectors.
The self-attention and cross-attention is applied to learn the latent
distribution. The decoder combines the HF signal and LF signal together to
form the final SR image.

### Positional Encoding for frequency band splitting.

Another key component of the proposed SVAE-SR is positional encoding (PE).
Inspired by [13], we expand the 2D grid map to much wider frequency bands. It
can better fit the data with a high-frequency variation. It is specifically
useful for image SR because the missing details around the edges and textures
are high-frequency signals. Neural networks are biased to low-frequency
reconstruction, which leads to over-smooth visual quality. Using PE can
explicitly lift the feature for high-frequency mapping. Mathematically, given
signal $o$,

$\lambda(o)=\left(sin(2^{0}\pi o),cos(2^{0}\pi o),...,sin(2^{L-1}\pi
o),cos(2^{L-1}\pi o)\right)$ (2)

Eq (2) maps the signal to an L-degree frequency band. We further split it into
two parts, the first $L//2$ frequency bands for the low-frequency coding (LFC)
and the rest for the high-frequency coding (HFC).

Fig. 2: The detailed structure of the encoder and decoder. The encoder takes
the HR/SR image and reference image as inputs to computer their features
correlations for joint distributions. The high-frequency positional Encoding
(HFC) ensures the estimated features expand to the high-frequency bands. The
decoder combines the LR features and sampled high-frequency features and pass
them to the LIIF model for pixel estimation.

### Training Loss.

We train SVAE-SR using the $l_{1}$ loss between SR and HR images and KL
divergence as follows:

$\mathit{L}=\mathit{L}_{1}+\lambda||Y^{\prime}-Y||^{1}+\beta
KL[Q_{\phi}(z|\mathbf{Y}|\mathbf{R})||\mathit{N}(0,1)],$ (3)

where $\lambda$ and $\beta$ are the weighting parameters to balance pixel
distortion and KL losses.

##  4 EXPERIMENTS

### Implementation details.

We train our model on DIV2K [24] and Flickr2K [2] datasets. They both contain
images with resolutions larger than 1000$\times$1000\. Same as [7], we first
extract LR patches with the size of 48$\times$48\. Then we randomly sample
upsampling scales $\alpha$ in uniform distribution $U(1,4)$. The corresponding
HR patches 48$\alpha\times$48$\alpha$. are then used to sample $48^{2}$ pixels
to form coordinate-RGB pairs. For the reference image, we randomly choose one
from Wikiart [25] that is widely used in style transfer. The testing datasets
include Set5 [26], Set14 [27], Urban100 [28] and DIV2K validation [24]. We
train with Adam optimizer with a learning rate of $10^{-4}$ and a batch size
of 32 for 100k iterations (8hrs on two NVIDIA V100 GPUs).

Table 1: PSNR comparison between ours and other state-of-the-art methods in
various upsampling scales. RDN trains different models for different scales.
MetaSR, LIIF, and ours use one model for all scales and are trained with
continuous random scales uniformly sampled in $\times 1\sim\times$4\. We also
test on out-of-distribution scenarios in $\times$6 and $\times$8.

Dataset | Method | In-distribution | Out-of-distribution
---|---|---|---
$\times$2 |  $\times$3 |  $\times$4 |  $\times$6 |  $\times$8
Set5 | RDN | 38.24 | 34.71 | 32.47 | - | -
RDN-MetaSR | 38.22 | 34.63 | 32.38 | 29.04 | 26.96
RDN-LIIF | 38.17 | 34.68 | 32.50 | 29.15 | 27.14
SVAE-SR (ours) | 38.23 | 34.72 | 32.60 | 29.23 | 27.24
Set14 | RDN | 34.01 | 30.57 | 28.81 | - | -
RDN-MetaSR | 33.98 | 30.54 | 28.78 | 26.51 | 24.97
RDN-LIIF | 33.97 | 30.53 | 28.80 | 26.64 | 25.15
SVAE-SR (ours) | 34.01 | 30.58 | 28.86 | 26.72 | 25.23
B100 | RDN | 32.34 | 29.26 | 27.72 | - | -
RDN-MetaSR | 32.33 | 29.26 | 27.71 | 25.90 | 24.83
RDN-LIIF | 32.32 | 29.26 | 27.74 | 25.98 | 24.91
SVAE-SR (ours) | 32.36 | 29.32 | 27.80 | 26.06 | 5.00
Urban100 | RDN | 32.89 | 28.80 | 26.61 | - | -
RDN-MetaSR | 32.92 | 28.82 | 26.55 | 23.99 | 22.59
RDN-LIIF | 32.87 | 28.82 | 26.68 | 24.20 | 22.79
SVAE-SR (ours) | 32.92 | 28.88 | 26.73 | 24.29 | 22.87

### General image super-resolution.

SVAE-SR performs efficient super-resolution with high reconstruction quality.
To show its effectiveness, we compare it to four state-of-the-art methods:
Bicubic, EDSR [2], RDN [4], LIIF [7] and MetaSR [8] in Table 1. For
demonstration, we use continuous upsampling scales $\alpha$ in uniform
distribution $U(1,4)$, and we also test on out-of-training-distribution
scenarios, where larger unseen upsampling scales, namely 6$\times\sim
8\times$, are evaluated on unknown images. We can find that using our proposed
method can achieve arbitrary image enlargement with superior performance
across different datasets. Especially on out-of-distribution scenarios, ours
performs even better with $+0.08\sim+0.09$ dB in PSNR.

We show visual comparisons in Figure 3. It can be seen that our method can
restore the fine textures with better visual quality, like the windows in
78004, floor strides in 148024, lighting rays in 0828 and the handrails in
0851.

Fig. 3: Visual comparison to the state of the arts. We show 4$\times$ super-
resolution on BSD100 (78004, 148026) and DIV2K-validation (0828,0851)
datasets. We enlarge the regions in red boxes for better visualization.  Table
2: Abalation studies on our methods with or without Soft-IntroVAE and PE
structures for image SR. The tests are done on DIV2K validation in PSNR(dB).

Method | In-distribution | Out-of-distribution
---|---|---
$\times$2 |  $\times$3 |  $\times$4 |  $\times$6 |  $\times$12 |  $\times$18
Bicubic | 31.01 | 28.22 | 26.66 | 24.82 | 22.27 | 21.00
LIIF | 34.99 | 31.26 | 29.27 | 26.99 | 23.89 | 22.34
LIIF-VAE | 34.89 | 31.20 | 29.16 | 26.90 | 23.81 | 22.29
LIIF-GAN | 34.78 | 31.12 | 29.03 | 26.58 | 23.53 | 22.07
LIIF-PE | 35.01 | 31.29 | 29.30 | 27.02 | 23.93 | 22.37
SVAE-SR w/o PE | 35.02 | 31.30 | 29.31 | 27.04 | 24.01 | 22.42
SVAE-SR w/ PE | 35.05 | 31.33 | 29.35 | 27.08 | 24.06 | 22.47

### Ablation studies.

Effect of Soft-IntroVAE for SR. To show the effectiveness of the proposed
SVAE-SR, we compare it with LIIF (baseline with pure MLP structure), LIIF-VAE
(LIFF with ordinary VAE structure), LIIF-GAN (LIIF with ordinary GAN
structure) and proposed SVAE-SR. From rows 2 to 4 and SVAE-SR w/o PE in Table
2, we can see that with the same LIIF network, using Soft-IntroVAE (SVAE-SR
w/o PE) achieve $+0.03\sim+0.12dB$ in PSNR. On the other hand, using LIIF-GAN
actually obtains lower PSNR, about $-2\sim-4dB$.

Fig. 4: Visualization on 1) $2\times\sim 16\times$ continuous upsampling, and
2) image super-resolution on the noisy image with noise level $\tau=1.5$. We
show the residual image between SR and HR images to enhance the differences.

### Effect of positional encoding for SR.

To illustrate the effect of the positional encoding for SR. From Table 2, we
conduct the comparisons among LIIF (baseline with pure MLP structure) (row 2),
LIIF-PE (baseline with positional encoding) (row 5) and ours with or without
PE (row 6 and 7). We can see that using PE can improve the reconstruction
quality by about $+0.03\sim 0.05dB$ in PSNR.

In Figure 4, we show two more our results on continuous image super-resolution
between $2\times\sim 16\times$. We can see that the window patterns are well
restored across different scenarios. We also mentioned that the proposed SVAE-
SR has the ability to overcome the noise for real image SR. We show one
example using LR image with Gaussian random noise of intensity 1.5. We can see
that our method achieves better results compared to other methods.

### Computational cost.

Our method achieves relative real-time inference in a Nvidia V100 GPU,
approximately 0.3s in 4$\times$ upsampling and 1.1s in 16$\times$ upsampling.

##  5 CONCLUSION

In this paper, we propose a Soft-IntroVAE for continuous image super-
resolution (SVAE-SR). It combines the advantages of VAE and LIIF to explore
the continuous latent space interpolation, which results in arbitrary
upsampling with photo-realistic visual quality. In the meantime, the proposed
positional encoding scheme expands the signal to much wider frequency bands,
which avoids the network bias in the low-frequency domain. Experimental
results on qualitative and quantitative comparisons show that our proposed
SVAE-SR achieves outstanding performance and points in a promising direction
in robust real-image super-resolution.

## References

  * [1] Zhi-Song Liu, Li-Wen Wang, Chu-Tak Li, Wan-Chi Siu, and Yui-Lam Chan,  “Image super-resolution via attention based back projection networks,”  in Proc. ICCV-Workshops, 2019, pp. 3517–3525.
  * [2] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee,  “Enhanced deep residual networks for single image super-resolution,”  in Proc. CVPR-Workshops, 2017, pp. 1132–1140.
  * [3] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu,  “Image super-resolution using very deep residual channel attention networks,”  Proc. ECCV, 2018.
  * [4] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu,  “Residual dense network for image restoration,”  Proc. CVPR, vol. 43, no. 7, pp. 2480–2495, 2021.
  * [5] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy,  “Esrgan: Enhanced super-resolution generative adversarial networks,”  in Proc. ECCV-Workshops, September 2018.
  * [6] Wan-Chi Siu Zhi-Song Liu, Li-Wen Wang, Chu-Tak Li, Marie-Paule Cani, and Yui-Lam Chan,  “Unsupervised real image super-resolution via generative variational autoencoder,”  in Proc. CVPR-Workshops, June 2020.
  * [7] Yinbo Chen, Sifei Liu, and Xiaolong Wang,  “Learning continuous image representation with local implicit image function,”  in Proc. CVPR, 2021, pp. 8624–8634.
  * [8] Xuecai Hu, Haoyuan Mu, Xiangyu Zhang, Zilei Wang, Tieniu Tan, and Jian Sun,  “Meta-sr: A magnification-arbitrary network for super-resolution,”  in Proc. CVPR, 2019, pp. 1575–1584.
  * [9] Zhiqin Chen and Hao Zhang,  “Learning implicit fields for generative shape modeling,”  in Proc. CVPR, 2019, pp. 5932–5941.
  * [10] Mateusz Michalkiewicz, Jhony Kaesemodel Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders Eriksson,  “Implicit surface representations as layers in neural networks,”  in Proc. ICCV, 2019, pp. 4742–4751.
  * [11] Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, and Thomas Funkhouser,  “Local implicit grid representations for 3d scenes,”  in Proc. CVPR, 2020, pp. 6000–6009.
  * [12] Peng Songyou, Niemeyer Michael, Mescheder Lars, Pollefeys Marc, and Geiger Andreas,  “Convolutional occupancy networks,”  in Proc. ECCV, 2020.
  * [13] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng,  “Nerf: Representing scenes as neural radiance fields for view synthesis,”  in Proc. ECCV, 2020.
  * [14] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger,  “Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision,”  in Proc. CVPR, 2020, pp. 3501–3512.
  * [15] Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger,  “Texture fields: Learning texture representations in function space,”  in Proc. ICCV, 2019, pp. 4530–4539.
  * [16] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman,  “Mip-nerf 360: Unbounded anti-aliased neural radiance fields,”  in Proc. CVPR, 2022, pp. 5460–5469.
  * [17] Ben Niu, Weilei Wen, Wenqi Ren, Xiangde Zhang, Lianping Yang, Shuzhen Wang, Kaihao Zhang, Xiaochun Cao, and Haifeng Shen,  “Single image super-resolution via a holistic attention network,”  in Proc. ECCV, 2020, p. 191–207.
  * [18] Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang,  “Second-order attention network for single image super-resolution,”  in Proc. CVPR, 2019, pp. 11065–11074.
  * [19] Wan-Chi Siu Zhi-Song Liu and Li-Wen Wang,  “Reference based image super-resolution via variational autoencoder,”  in Proc. CVPR-Workshops, June 2021.
  * [20] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  in NeurIPS. 2017, vol. 30, Curran Associates, Inc.
  * [21] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio,  “Generative adversarial nets,”  in NeurIPS, Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, Eds., pp. 2672–2680. Curran Associates, Inc., 2014.
  * [22] Diederik P Kingma and Max Welling,  “Auto-encoding variational bayes,” 2014.
  * [23] Tal Daniel and Aviv Tamar,  “Soft-introvae: Analyzing and improving the introspective variational autoencoder,”  in Proc. CVPR, June 2021, pp. 4391–4400.
  * [24] Radu Timofte and et al.,  “Ntire 2017 challenge on single image super-resolution: Methods and results,”  in Proc. CVPR. 8 2017, IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 1110–1121, IEEE Computer Society.
  * [25] “K. nichol. painter by numbers, wikiart,” https://www.kaggle.com/c/painter-by-numbers, 2016.
  * [26] Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie line Alberi Morel,  “Low-complexity single-image super-resolution based on nonnegative neighbor embedding,”  in Proceedings of the British Machine Vision Conference. 2012, pp. 135.1–135.10, BMVA Press.
  * [27] Roman Zeyde, Michael Elad, and Matan Protter,  “On single image scale-up using sparse-representations,”  06 2010, vol. 6920, pp. 711–730.
  * [28] J. Huang, A. Singh, and N. Ahuja,  “Single image super-resolution from transformed self-exemplars,”  in Proc. CVPR, June 2015, pp. 5197–5206.


###### Proof.

We use induction on $h$. The base case $h=3$ has been proved in Theorem 58.
Assume now $h\geq 4$ and the claim is true for $h-1$. Let $1\leq\ell\leq
n_{h}$. Consider $S((\operatorname{{TEP}},h),\ell)$.

  1. (i)

$1\leq\ell\leq\sum_{i=2}^{h-1}n_{i}$. When $1\leq\ell\leq n_{h-1}$, choose
$A=(\emptyset,A_{L},\emptyset)$. Apply the induction hypothesis and Lemma 56,

$S((\operatorname{{TEP}},h)_{A})=S((\operatorname{{TEP}},h-1)_{A_{L}})\leq
k^{h-1}.$

When $n_{h-1}<\ell\leq\sum_{i=2}^{h-1}n_{i}$, choose $A=(\emptyset,L,A_{R})$.
Then, $1\leq|A_{R}|\leq\sum_{i=2}^{h-2}n_{i}$. Apply the induction hypothesis
and Lemma 56,

$S((\operatorname{{TEP}},h)_{A})=S((\operatorname{{TEP}},h-1)_{L})\cdot
S((\operatorname{{TEP}},h-1)_{A_{R}})\leq k\cdot k^{h-2}=k^{h-1}.$

  2. (ii)

$\sum_{i=2}^{h-1}n_{i}<\ell<\sum_{i=2}^{h-1}n_{i}+k^{2}$. Choose
$A=(\emptyset,L,A_{R})$. Then,
$\sum_{i=2}^{h-2}n_{i}<|A_{R}|<\sum_{i=2}^{h-2}n_{i}+k^{2}$. Apply the
induction hypothesis and Lemma 56,

$S((\operatorname{{TEP}},h)_{A})=S((\operatorname{{TEP}},h-1)_{L})\cdot
S((\operatorname{{TEP}},h-1)_{A_{R}})\leq k\cdot
k^{h-3}(k^{2}-k+2)=k^{h-2}(k^{2}-k+2).$

  3. (iii)

$\sum_{i=2}^{h-1}n_{i}+k^{2}\leq\ell\leq 2n_{h-1}$. Choose
$A=(\emptyset,L,A_{R})$. Then, $\sum_{i=2}^{h-2}n_{i}+k^{2}\leq|A_{R}|\leq
n_{h-1}$. Apply the induction hypothesis and Lemma 56,

$\displaystyle S((\operatorname{{TEP}},h)_{A})$
$\displaystyle=S((\operatorname{{TEP}},h-1)_{L})\cdot
S((\operatorname{{TEP}},h-1)_{A_{R}})$ $\displaystyle\leq
k\cdot\max\\{k^{h-3}(k+2),k^{2}+k+1\\}=k^{h-2}(k+2),$

where we used $h\geq 4$ in the last step.

  4. (iv)

$2n_{h-1}<\ell\leq n_{h}$. This can be proved in the same way as (i) in the
proof of Theorem 58. ∎