# Enhanced superconducting properties of double-chain based superconductor
Pr2Ba4Cu7O15-δ synthesized by citrate pyrolysis technique

Kazuma Honnami  Michiaki Matsukawa<EMAIL_ADDRESS>Tatsuya Senzaki
Tomoki Toyama  Haruka Taniguchi  Koichi Ui  Faculty of Science and
Engineering, Iwate University, Morioka 020-8551, Japan  Takahiko Sasaki  Kohki
Takahashi  Institute for Materials Research, Tohoku University, Sendai
980-8577, Japan  Makoto Hagiwara  Kyoto Institute of Technology, Kyoto
606-8585, Japan  Fumihiko Ishikawa  Department of Physics, Niigata University,
Niigata 950-2181, Japan

###### Abstract

We report the enhanced superconducting properties of double-chain based
superconductor Pr2Ba4Cu7O15-δ synthesized by the citrate pyrolysis technique.
The reduction heat treatment in vacuum results in the appearance of
superconducting state with $T_{\mathrm{c}}$=22-24 K, accompanied by the higher
residual resistivity ratios. The superconducting volume fractions are
estimated from the ZFC data to be 50$\sim 55\%$, indicating the bulk
superconductivity. We evaluate from the magneto-transport data the temperature
dependence of the superconducting critical field, to establish the
superconducting phase diagram. The upper critical magnetic field is estimated
to be about 35 T at low temperatures from the resistive transition data using
the Werthamer-Helfand-Hohenberg formula. The Hall coefficient $R_{H}$ of the
48-h-reduced superconducting sample is determined to be -0.5$\times 10^{-3}$
cm3/C at 30 K, suggesting higher electron concentration. These findings have a
close relationship with homogeneous distributions of the superconducting
grains and improved weak links between their superconducting grains in the
present synthesis process.

###### pacs:

74.25.Ha,74.25.F-,74.90.+n

Fig. 1: (color online)(a) X-ray diffraction patterns of as-sintered
polycrystalline Pr2Ba4Cu7O15-δ. The (004) peak corresponds to one of typical
Miller indexes of Pr247. The calculated curve is obtained using the lattice
parameters in the text. Inset shows the crystal structure of Pr247 with CuO
single-chain and double-chain blocks along the $b$-axis. Here, S and D denote
CuO single-chain and double-chain blocks along the $b$-axis. (b)X-ray
diffraction pattern for polycrystalline film prepared by an electrophoretic
deposition technique under 10 T. The magnetic field was applied parallel to
the direction of applied electric field between anode and cathode electrodes.
Fig. 2: (color online) SEM image for polycrystalline film prepared by an
electrophoretic deposition technique under 10 T.

##  I INTRODUCTION

Since the discovery of high-$T_{\mathrm{c}}$ copper-oxide superconductors,
researches have focused on the unconventional superconductivity on the two-
dimensional CuO2 planes in some cuprates. In quasi one-dimensional (1D) ladder
system without CuO2 planes, it is well known that the superconductivity at
$T_{\mathrm{c}}$= 12 K appears only under the application of high
pressure.UE96  In a previous study of Pr-based cuprates with metallic CuO
double chains and insulating CuO2 planes , Pr2Ba4Cu7O15-δ is found to be a new
superconductor with a higher $T_{\mathrm{c}}$ (15 K) after a reduction
treatment.MA04  A nuclear quadrupole resonance (NQR) study has revealed that
the newly discovered superconductivity is realized at the CuO double-chain
block. WA05

Structurally, the Pr-based cuprates, PrBa2Cu3O7-δ (Pr123) and PrBa2Cu4O8
(Pr124), are identical to their corresponding Y-based high-$T_{\mathrm{c}}$
superconductors, YBa2Cu3O7-δ (Y123) and YBa2Cu4O8 (Y124). Pr123 and Pr124
compounds have insulating CuO2 planes and are non-superconductive. SO87 ; HO98
The suppression of superconductivity in the Pr substitutes has been explained
by the hybridization of Pr-4$f$ and O-2$p$ orbitals.FE93  The crystal
structure of Pr124 with CuO double chains differs from that of Pr123 with CuO
single chains. It is well known that CuO single chains in Pr123 and CuO double
chains in Pr124 show semiconducting and metallic behaviors, respectively.MI00
The carrier concentration of doped double chains of Pr124 is difficult to
vary, because it is thermally stable up to high temperatures.

The compound Pr2Ba4Cu7O15-δ (Pr247) is an intermediate between Pr123 and
Pr124. In this compound, CuO single-chain and double-chain blocks are
alternately stacked along the $c$-axis such as {-D-S-D-S-} sequence BO88 ;
YA94  (see Fig.1). Here, S and D denote CuO single-chain and double-chain
blocks along the $b$-axis, respectively. The physical properties of the
metallic CuO double chains can be examined by controlling the oxygen content
along the semiconducting CuO single chains. Anisotropic resistivity
measurements of single-crystal Pr124 have revealed that metallic transport
arises by the conduction along the CuO double chains.HO00  In oxygen removed
polycrystalline Pr2Ba4Cu7O15-δ, the superconductivity appears at an onset
temperature $T_{\mathrm{c,on}}$ of $\sim$15 K. MA04  Hall coefficient
measurements of superconducting Pr247 with $T_{\mathrm{c,on}}=15$ K have
revealed that at intermediate temperatures below 120 K, the main carriers
change from holes to electrons, as the temperature decreases. Accordingly,
this compound is an electron-doped superconductor.MA07  In our previous study,
we examined the effect of magnetic fields on the superconducting phase of
Pr247.CH13  Despite of the resistive drop associated with the superconducting
transition, we found that the diamagnetic signal was strongly suppressed as
expected in the 1D superconductivity of CuO double chains. We also reported
the effect of pressure on magneto-transport properties in the superconducting
and normal phases of the metallic double chain compound Pr2Ba4Cu7O15-δ. KU16 .
The model of slightly warped Fermi surfaces explains not only the magneto-
resistance (MR) effect of the non-superconducting sample, but also the
pressure-induced MR phenomena of the superconducting sample.

In this paper, we demonstrate the magneto-transport properties of
Pr2Ba4Cu7O15-δ with $T_{\mathrm{c,on}}$ (22-24 K) accompanied by the strongly
metallic conduction, for our understanding of the effect of magnetic field on
the superconducting properties of electron-doped metallic double-chain
compound. In the next section, the experimental outline is described. In Sect.
3, we show the outstanding data including higher residual resistivity ratios,
larger superconducting volume fractions, enhanced zero-resistance states under
magnetic field, and higher carrier concentrations. The final section is
devoted to the summary.

##  II EXPERIMENT

Polycrystalline samples of Pr2Ba4Cu7O15-δ were synthesized by using the
citrate pyrolysis technique.KO91 ; HA06  In the first step, stoichiometric
mixtures of high purity Pr6O11, Ba(NO3)2, and CuO were dissolved in a nitric
acid solution at50-60 ∘C. After adding citric acid and neutralizing it by
aqueous ammonia, we then obtained the porous products through the self-
ignition process using halogen lamp stirrer. In the next step, the precursors
were ground and resultant fine powders were annealed under ambient oxygen
pressure at 890-891 ∘C for an extended period over 110-120h. In the present
citrate pyrolysis synthesis procedure, we adopted the electric tube furnace
with three zone temperature controlled system, to achieve the temperature
uniformity within 1 ∘C.

For scanning electron microscope (SEM) measurements, we prepared the
polycrystalline film sample, to reveal microstructures of single-phase
Pr2Ba4Cu7O15 powders synthesized by the citrate pyrolysis technique. The
polycrystalline film on Ag substrate was fabricated from the as-sintered
powders by an electrophoretic deposition technique. The electrophoretic
deposition was conducted in the acetone and iodine bath under the application
of electric voltage up to 300 V for 120 s. KA01  We set Pt and Ag plates as
anode and cathode electrodes, respectively. Furthermore, we performed the
magnetic field assisted electrophoretic deposition process for the fabrication
of $c$-axis aligned Pr2Ba4Cu7O15 polycrystalline film.KA02  Here, the magnetic
anisotropy of rare earth ion Pr including Pr2Ba4Cu7O15 shows that the magnetic
susceptibility of $\chi_{H\parallel c}$ is larger than that of $\chi_{H\perp
c}$. XU10  A magnetic field up to 10 T was applied to the colloid bath
including the suspended Pr247 powders using a superconducting magnet with a
100 mm diameter bore at room temperature at the High Field Laboratory for
Superconducting Materials, Institute for Materials Research (IMR), Tohoku
University. The magnetic field was applied parallel to the direction of
applied electric field for the electrophoretic deposition.

We performed X-ray diffraction measurements on the produced samples at room
temperature with an Ultima IV diffractometer (Rigaku) using Cu-K$\alpha$
radiation. The lattice parameters were estimated from the x-ray diffraction
data using the least-squares fits.

The oxygen in the as-sintered sample was removed by reduction treatment in a
vacuum at 500 ∘C, yielding a superconducting material. Typical dimensions of
the pelletized rectangular sample were $4.2\times 3.4\times 1.3$ mm3. X-ray
diffraction data revealed that the as-sintered polycrystalline samples are an
almost single phase with an orthorhombic structure ($Ammm$), as shown in
Fig.1($a$).

The lattice parameters of the as-sintered sample are $a=3.89293$ Å,
$b=3.91394$ Å, and $c=50.79270$ Å, which are in fair agreement with those
obtained by a previous studyYA05 . The oxygen deficiencies in the 48 h and 72
h reduced samples in a vacuum were estimated from gravimetric analysis to be
$\delta=0.43$ and 0.54, respectively. As a function of the oxygen deficiency,
the $T_{\mathrm{c,on}}$ rises rapidly at $\delta\geq\sim 0.2$, then
monotonically increases with increasing $\delta$, and finally saturates around
26-27 K at $\delta\geq\sim 0.6$.HA08  Accordingly, we expect that the carriers
in the present sample are concentrated around the optimally doped region.

The electric resistivity in zero magnetic field was measured by the $dc$ four-
terminal method. The magneto-transport up to 9 T was measured by the $ac$
four-probe method using a physical property measuring system (PPMS, Quantum
Design), increasing the zero-field-cooling (ZFC) temperatures from 4 K to 40
K. The high field resistivity (up to 14 T) was measured in a superconducting
magnet at IMR, Tohoku University. The electric current $I$ was applied
longitudinally to the sample ; consequently, the applied magnetic field $H$
was transverse to the sample (because $H\perp I$). We performed Hall
coefficient measurements on the as-sintered and 48 h reduced samples with the
five-probe technique using PPMS and the 15T-superconducting magnet. TA13  The
$dc$ magnetization was performed under ZFC in a commercial superconducting
quantum interference device magnetometer (Quantum Design, MPMS).

Table 1: Physical and superconducting properties for Pr2Ba4Cu7O15-δ. In details, see the corresponding text and references. $\delta$ and RRR denote oxygen deficiency and residual resistivity ratio $\rho$(300K)/ $\rho$(30K), respectively. $f_{\mathrm{SC}}$: the superconducting volume fraction estimated from the ZFC magnetization data at low temperatures. $H_{\mathrm{c}}^{*}$ is defined from the field sweep data at fixed temperatures as the critical field where the zero-resistance state is violated with increasing the field.  Synthesis |  | Reduced time |  | $\delta$ |  | RRR |  | $f_{\mathrm{SC}}$ |  | $T_{\mathrm{c,on}}$ |  | $T_{\mathrm{c,zero}}$ |  | $H_{\mathrm{c}}^{*}$ |  |  $R_{H}$(30K)
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
|  |  |  |  |  |  |  | ($\%$) |  | (K) |  | (K) |  | (T) |  | ($10^{-3}$ cm3/C)
citrate pyrolysis |  | 48h |  | 0.43 |  | 10 |  | 55 |  | 22 |  | 18 |  | 10.6(4.2K) |  | -0.5
citrate pyrolysis |  | 72h |  | 0.54 |  | 12 |  | 50 |  | 24.3 |  | 18 |  | - |  | -
citrate pyrolysis |  | 48h |  | 0.52a |  | 4a |  | 30a |  | 26.5a |  | 15a |  |  $\sim$2(5K) |  | -1.1b
sol-gelTO12  |  | 72h |  | 0.94 |  | 2 |  |  $\sim$10 |  | 30.5 |  |  $\sim$10 |  | - |  | -
high pressureMA07  |  | 48h |  | 0.5 |  | - |  | 30 |  | 16 |  |  $\sim$10 |  | - |  | -1.5
a see ref. CH13 , bref.TA13  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

Fig. 3: (color online)(a) Temperature dependences of electric resistivities of
the as-sintered, 48-h-reduced and 72-h-reduced Pr2Ba4Cu7O15-δ compounds. (b)
low-temperature dependences of magnetic susceptibilities $\chi$ of the
48-h-reduced and 72-h-reduced superconducting samples measured at 5 mT under
ZFC scan. In the inset, the magnified data are plotted to clarify the
definition of $T_{c,on}$.

Fig. 4: (color online) (a) Low-temperature dependences of electric
resistivities of the 48-h-reduced superconducting Pr2Ba4Cu7O15-δ compound
measured under 0T, 10T, and 14T. Inset plots the magneto-resistance data of
the 48-h-reduced sample at 4.2 K and 30 K. (b) Low-temperature dependences of
electric resistivities of the 72-h-reduced superconducting Pr2Ba4Cu7O15-δ
compound measured at the several applied fields ($H$ = 0, 1, 3, 6 and 9 T).
The inset displays the enlarged data of the 72-h-reduced sample around
$T_{c}$.  Fig. 5: (color online) Temperature dependence of the superconducting
(SC) critical field of the reduced samples of Pr2Ba4Cu7O15-δ. The onset
$T_{\mathrm{c}}$ ( $T_{\mathrm{c,on}}$ ) is determined from the resistivity
data as described in the text. For comparison, the previous data of the
reduced Pr247 are referred (open symbol). The solid curve separating the
superconducting and normal phases was the best fit to the experimental data
and the upper critical field of $\sim$35 T. Here, the corresponding
$H_{\mathrm{c2}}$(0) is estimated using the WHH formula in the text.  Fig. 6:
(color online) Temperature dependences of the Hall coefficients $R_{H}$ for
the as-sintered non-superconducting and 48-h-reduced superconducting samples
of Pr2Ba4Cu7O15-δ.

##  III RESULTS AND DISCUSSION

Figure1($a$) shows X-ray diffraction (XRD) patterns of as-sintered
polycrystalline Pr2Ba4Cu7O15-δ(Pr247). The (004) peak corresponds to one of
typical Miller indexes of Pr247. The calculated curve is obtained using the
lattice parameters. The inset of Fig. 1($a$) displays the crystal structure of
Pr247 with CuO single-chain and double-chain blocks along the $b$-axis. The
unknown impurity peaks near 2$\theta\sim$30 ∘ are identified as the BaCuO2
phase.HA06

For comparison, X-ray diffraction pattern of polycrystalline film prepared by
an electrophoretic deposition technique under 10 T is shown in in Fig. 1($b$).
We note that the XRD pattern of the Pr247 film fabricated by the
electrophoretic deposition process without the applied magnetic field (not
shown here) was the same as that of the Pr247 powders. (00$l$) peaks did not
appear except for the (004) and (00$\underline{26}$) peaks. When the magnetic
field was applied parallel to the direction of applied electric field, the
peak intensities of (00$l$) were strongly enhanced as shown in Fig. 1($b$),
indicating the c-axis alignment of polycrystalline grains. The SEM image of
the Pr247 film in Fig. 2 reveals that plate-like grains with sub micron size
are homogeneously dispersed.

Now, the temperature dependences of electric resistivities of the as-sintered,
48-h-reduced and 72-h-reduced Pr2Ba4Cu7O15-δ compounds are shown in Fig.
3($a$). The reduction heat treatment on the as-sintered sample in vacuum
results in the appearance of superconducting state with $T_{\mathrm{c}}$=22-24
K, accompanied by the strongly metallic properties over a wide range of
temperature. Here, we define the residual resistivity ratio (RRR) as
$\rho$(300K)/ $\rho$(30K), For the present samples reduced in vacuum, we
obtained the higher RRR values ($10\sim 12.5$), in comparison with the
previous data. CH13

Furthermore, to check the bulk superconductivity, we performed to measure low-
temperature dependences of magnetic susceptibilities $\chi$ of the 48-h and
72-h-reduced superconducting samples measured at 5 mT under ZFC scan. Figure
3($b$) exhibits diamagnetic signals below $T_{\mathrm{c,on}}$=22.5 K and 24.0
K for the 48-h-reduced and 72-h-reduced samples, respectively. In addition,
the superconducting volume fractions due to the shielding effect are estimated
to be $50\sim 55\%$ from the ZFC values at 5 K, which are much higher than the
previous data ($\sim 30\%$). CH13  In the inset of Fig. 3($b$) , the magnified
data are plotted to clarify the definition of $T_{\mathrm{c,on}}$. The
characteristic values for the present and previous samples are listed in Table
I.

Next, we try to measure the magneto-transport properties of the reduced
samples, to examine the magnetic effect on the superconducting phase of the
Pr247 compound. Figure 4($a$) shows the low-temperature dependences of
electric resistivities of the 48-h-reduced superconducting Pr2Ba4Cu7O15-δ
compound measured under zero field and applied fields of 10 and 14T.

In zero field, the resistivity starts to decrease around $T_{\mathrm{c,on}}$ =
22 K, then follows a rapid drop, and finally achieves a zero-resistance state
at $T_{\mathrm{c,zero}}$ = $\sim$18 K. The observed transition width
$T_{\mathrm{c,on}}$ = 22 K is relatively sharp in comparison with the
published data. Inset plots the magneto-resistance data of the 48-h-reduced
sample at 4.2 and 30 K. At the high field of 10 T, we observed
$T_{\mathrm{c,on}}$ = 18 K and $T_{\mathrm{c,zero}}$ = $\sim$5 K. Upon
increasing the applied field up to 14 T, the superconductive critical field at
4.2K is determined to be 10.6 T. Even at the maximum field of 14 T, we judge
that the zero-resistance state is almost realized over the whole sample since
the magnitudes of $\rho$ are negligibly small with those of the normal state
at 30 K. These findings strongly suggest that for the present sample the
superconducting properties under relatively high fields are considerably
improved. For the previous Pr247 samples, it has been reported that the
critical field is as low as a few tesla at low temperatures.

In Fig. 4($b$) , we show the low-temperature dependences of electric
resistivities of the 72-h-reduced superconducting Pr2Ba4Cu7O15-δ compound
measured at the several applied fields ($H$ = 0, 1, 3, 6 and 9 T). The inset
displays the enlarged data of the 72-h-reduced sample around
$T_{\mathrm{c,on}}=24.3$ K. In a similar to the data of the 48-h-reduced
sample, we observed a zero-resistance state below $T_{\mathrm{c,zero}}$ =
$\sim$18 K and 5 K, at 0 T and at 9 T, respectively. Here, we evaluate from
the magneto-transport data the temperature dependence of the superconductive
critical field, to establish the superconducting phase diagram. The onset
$T_{\mathrm{c}}$ ($T_{\mathrm{c,on}}$) is determined from the intersection
between the two lines extrapolated from the normal and superconducting
transition resistivity data, just above and bellow $T_{\mathrm{c}}$,
respectively. (see Fig.4). The zero-point $T_{\mathrm{c}}$
($T_{\mathrm{c,zero}}$) denotes the value of the critical temperature reaching
the zero-resistance state. The upper critical field at 0 K is estimated to be
about 35 T from the data of the onset $T_{\mathrm{c}}$ using the Werthamer-
Helfand-Hohenberg (WHH) formula in the dirty limit,
$\mu_{0}H_{\mathrm{c2}}(0)=-0.69T_{\mathrm{c}}(dH_{\mathrm{c2}}(T)/dT)|_{T=T_{c}}$.WE66

There are quite differences in the critical fields determined from
$T_{\mathrm{c,zero}}$ between the present and previous samples, strongly
suggesting the enhanced superconducting properties in the presence of magnetic
field as represented by a right arrow in Fig. 5.

Furthermore, in Fig.6, we show the temperature dependences of the Hall
coefficients $R_{H}$ for the as-sintered non-superconducting and 48-h-reduced
superconducting samples of Pr2Ba4Cu7O15-δ. The $R_{H}$ data of the as-sintered
sample are very similar, in the both magnitude and temperature dependence, to
those of the previous sample. For the 48-h-reduced sample, the $R_{H}$ data
exhibit negative values in the limited temperature range between 30 and 80 K,
accompanied by electron doping due to the reduced heat treatment in vacuum.
Moreover, we estimate $R_{H}=-0.5\times 10^{-3}$ cm3/C at 30 K, which is as
about half as the published data. TA13 , indicating higher carrier
concentration. These results have a close relationship with the enhanced
superconducting properties, leading to the superconducting phase diagram of
the present samples.

Finally, the physical and superconducting properties for Pr2Ba4Cu7O15-δ
compounds are summarized in Table I. In the present citrate pyrolysis
synthesis procedure, we adopted the electric tube furnace with three zone
temperature controlled system, to achieve the temperature uniformity within 1
K. In spite of limited heat-treatment conditions , we obtained homogeneous
distributions of the superconducting grains and improved weak links between
their superconducting grains, leading to the enhanced superconducting
properties in the present samples. The former is closely related to the
narrower transition width of superconductivity in zero magnetic field. The
latter improvement is mainly responsible for the considerable enhancement of
superconducting properties in high field. In addition, we give some comments
on the high-$T_{\mathrm{c}}$ samples synthesized by a sol-gel technique.TO12
The transmission electron microscopy image of the high-$T_{\mathrm{c}}$ sample
with $T_{\mathrm{c,on}}$ =30.5 K revealed that there exists an irregular long-
period stacking structure along the $c$-axis such as {-D-D-D-S-S-S-D-D-S-S}
sequences.CH13  Here, S and D denote CuO single-chain and double-chain blocks
along the $b$-axis. This finding is close to higher concentration, resulting
in higher $T_{\mathrm{c}}$ , in comparison to the regular stacking case of
{-S-D-S-D-S-D}. The sol-gel sample showed highly inhomogeneous features such
as the irregular stacking structures and the coexistence of the nominal Pr247
phase with the Pr123 and Pr124 phases. For the corresponding inhomogeneous
sample, we also observed a broad width between the onset and zero transition
temperatures as listed in Table I.

##  IV SUMMARY

We have demonstrated the enhanced superconducting properties of double-chain
based superconductor Pr2Ba4Cu7O15-δ synthesized by the citrate pyrolysis
technique using the 3-zone controlled electric furnace. The outstanding
experimental data include the higher residual resistivity ratios, the larger
superconducting volume fractions, the enhanced zero-resistance state under
magnetic field, and the higher carrier concentrations.

The SEM image of the Pr247 film revealed that plate-like grains with sub
micron size are homogeneously dispersed. For the polycrystalline bulk samples,
we obtained the higher residual resistivity ratio ranging from 10 to 12 for
the 48-72 h reduction. The reduction heat treatment on the as-sintered sample
in vacuum results in the appearance of superconducting state with
$T_{\mathrm{c}}$=22-24 K, accompanied by the metallic properties over a wide
range of temperature. The superconducting volume fractions are estimated from
the ZFC data to be 50$\sim 55\%$, indicating the bulk superconductivity. We
evaluated from the magneto-transport data the temperature dependence of the
superconductive critical field, to establish the superconducting phase
diagram. The upper critical magnetic field was estimated to be about 35 T from
the resistive transition data using the WHH formula. For the 48-h-reduced
sample, the $R_{H}$ data exhibit negative values in the limited temperature
range between 30 and 80 K, accompanied by electron doping due to the reduced
heat treatment in vacuum. Moreover, we estimated $R_{H}=-0.5\times 10^{-3}$
cm3/C at 30 K, which is as about half as the published data, indicative of
higher carrier concentration. In spite of limited heat-treatment conditions
for the Pr247 phase , we attained homogeneous distributions of the
superconducting grains and improved weak links between their superconducting
grains, leading to the enhanced superconducting properties in the present
samples.

##  V Acknowledge

This work was supported in part by MEXT Grands-in-Aid for Scientific Research
(JPSJ KAKENHI Grants No. JP19K04995). We thank Mr. K. Sasaki for the SEM
measurement and M. Nakamura for his assistance in the PPMS experiments.

## References

  * (1) M. Uehara, T. Nagata, J. Akimitsu, H. Takahashi, N. Mori, and K. Kinoshita, J. Phys. Soc. Jpn. 65 (1996) 2764.
  * (2) M. Matsukawa, Y. Yamada, M. Chiba, H. Ogasawara, T. Shibata, A. Matsushita, and Y. Takano, Physica C 411 (2004) 101.
  * (3) S. Watanabe, Y. Yamada and S. Sasaki, Physica C 426-431 (2005) 473.
  * (4) L. Soderholm, K. Zhang, D. G. Hinks, M. A. Beno, J. D. Jorgensen,C. U. Segre, I. K. Schuller, Nature 328 (1987) 604.
  * (5) S. Horii, Y. Yamada, H. Ikuta, N. Yamada, Y. Kodama, S. Katano,Y. Funahashi, S. Morii, A. Matsushita, T. Matsumoto, I. Hirabayashi, and U. Mizutani, Physica C 302 (1998) 10.
  * (6) R. Fehrenbacher and T. M. Rice, Phys. Rev. Lett. 70 (1993) 3471.
  * (7) T. Mizokawa, C. Kim, Z. -X. Shen, A. Ino, T. Yoshida, A. Fujimori, M. Goto, H. Eisaki, S. Uchida, M. Tagami, K. Yoshida, A. I. Rykov, Y. Siohara, K. Tomimoto, S. Tajima, Yuh Yamada, S. Horii, N. Yamada, Yasuji Yamada, and I. Hirabayashi, Phys. Rev. Lett. 85 (2000) 4779.
  * (8) P. Bordet, C. Chaillout, J. Chenavas, J. L. Hodeau, M. Marezio,J. Karpinski, and E. Kaldis, Nature 334 (1988) 596.
  * (9) Y. Yamada, S. Horii, N. Yamada, Z. Guo, Y. Kodama, K. Kawamoto,U. Mizutani, and I. Hirabayashi, Physica C 231(1994)131.
  * (10) S. Horii, U. Mizutani, H. Ikuta, Y. Yamada, J. H. Ye, A. Matsushita, N. E. Hussey, H. Takagi, and I. Hirabayashi, Phys. Rev. B61 (2000) 6327.
  * (11) A. Matsushita, K. Fukuda, Y. Yamada, F. Ishikawa, S. Sekiya,M. Hedo, and T. Naka, Science and Technology of Advanced Materials 8 (2007) 477.
  * (12) T. Chiba, M. Matsukawa, J. Tada, S. Kobayashi, M. Hagiwara, T. Miyazaki, K. Sano, Y. Ono, T. Sasaki, and J.Echigoya, J. Phys. Soc. Jpn. 82 (2013) 074706.
  * (13) M. Kuwabara, M. Matsukawa, K. Sugawara,H. Taniguchi, A. Matsushita, M. Hagiwara, K. Sano, Y. Ono, and T. Sasaki, J. Phys. Soc. Jpn. 85 (2016) 124704.
  * (14) K. Koyama, A. Junod, T. Graf, G. Triscone, and J. Muller, Physica C185-189 (1991)66.
  * (15) M. Hagiwara, T. Shima, T. Sugano, K. Koyama, and M. Matsuura, Physica C 445-448 (2006) 111.
  * (16) M. Kawachi, N. Sato, E. Suzuki, S. Ogawa, K. Noto, and M. Yoshizawa, Physica C 357-360 (2001)1023.
  * (17) M. Kawachi, N. Sato, K. Noto and M. Yoshizawa, Physica C 802-805 (2002)802.
  * (18) X. Xu, A. Carrington, A. I. Coldea, A. Enayati-Rad, A. Narduzzo, S. Horii, and N. E. Hussey, Phys. Rev. B 81(2010)224435.
  * (19) Y. Yamada and A. Matsushita, Physica C 426-431 (2005) 213.
  * (20) M. Hagiwara, S. Tanaka, T. Shima, K. Gotoh, S. Kanda, T. Saito, and K. Koyama, Physica C 468 (2008) 1217.
  * (21) J. Tada, M. Matsukawa, T. Konno, S. Kobayashi, M. Hagiwara, T. Miyazaki, K. Sano, Y. Ono, and A. Matsushita, J. Phys. Soc. Jpn. 82 (2013) 105003.
  * (22) S. Toshima, M. Matsukawa, T. Chiba, S. Kobayashi, S. Nimori, and M. Hagiwara, Physica C 480(2012)1.
  * (23) N. R. Werthamer, E. Helfand, and P. C. Hohenberg, Phys. Rev. 147, (1966)295.


# DePA: Improving Non-autoregressive Machine Translation with Dependency-Aware
Decoder

Jiaao Zhan1, Qian Chen, Boxing Chen, Wen Wang, Yu Bai1, Yang Gao1
1School of Computer Science and Technology,
Beijing Institute of Technology, Beijing, China
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
Corresponding Author

###### Abstract

Non-autoregressive machine translation (NAT) models have lower translation
quality than autoregressive translation (AT) models because NAT decoders do
not depend on previous target tokens in the decoder input. We propose a novel
and general Dependency-Aware Decoder (DePA) to enhance target dependency
modeling in the decoder of fully NAT models from two perspectives: decoder
self-attention and decoder input. First, we propose an autoregressive forward-
backward pre-training phase before NAT training, which enables the NAT decoder
to gradually learn bidirectional target dependencies for the final NAT
training. Second, we transform the decoder input from the source language
representation space to the target language representation space through a
novel attentive transformation process, which enables the decoder to better
capture target dependencies. DePA can be applied to any fully NAT models.
Extensive experiments show that DePA consistently improves highly competitive
and state-of-the-art fully NAT models on widely used WMT and IWSLT benchmarks
by up to 1.88 BLEU gain, while maintaining the inference latency comparable to
other fully NAT models.111We released our code at:
https://github.com/zhanjiaao/NAT_DePA.

DePA: Improving Non-autoregressive Machine Translation with Dependency-Aware
Decoder

Jiaao Zhan1, Qian Chen, Boxing Chen, Wen Wang, Yu Bai1, Yang Gao1††thanks:
Corresponding Author 1School of Computer Science and Technology, Beijing
Institute of Technology, Beijing, China<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Autoregressive translation (AT) systems achieve state-of-the-art (SOTA)
performance for neural machine translation (NMT) and Transformer (Vaswani et
al., 2017) encoder-decoder is the prevalent architecture. In AT systems, each
generation step depends on previously generated tokens, resulting in high
inference latency when output is long. Non-autoregressive translation (NAT)
models (Gu et al., 2018) significantly accelerate inference by generating all
target tokens independently and simultaneously. However, this independence
assumption leads to degradation in accuracy compared to AT models, as NAT
models cannot properly learn target dependencies. Dependency in prior works
and our work takes its standard definition in NLP, i.e., syntactic relations
between words in a sentence.

The mainstream NAT models fall into two categories: iterative NAT models and
fully NAT models. Iterative NAT models (Gu et al., 2019; Ghazvininejad et al.,
2019; Lee et al., 2018) improve translation accuracy by iteratively refining
translations at the expense of slower decoding speed. In contrast, fully NAT
models (Gu et al., 2018; Bao et al., 2022) have great latency advantage over
AT models by making parallel predictions with a single decoding round, but
they suffer from lower translation accuracy. In this paper, we aim at
improving the translation accuracy of fully NAT models while preserving their
latency advantage.

| Under-Translation | Over-Translation | Wrong Lexical Choice
---|---|---|---
Source | Wir haben es weltweit in 300 Gemeinden gemacht. | Einige leute wollten ihn einfach König nennen | Woher komme ich ? Wer bin ich ?
Target Reference | We ’ve done it in 300 communities around the world. | Some people just wanted to call him King . | Where am I from ? Who am I ?
F-NAT | We did it the world in 300 communities. | Some people just wanted to call him him king. | How do I come from ? Who am I ?
FB-NAT | We ’ve done it in 300 communities around the world. | Some people just wanted to call him king. | Where do I come from? Who am I ?

Table 1: Case studies of our proposed FBD approach on the highly competitive
fully NAT model GLAT (Qian et al., 2021) for alleviating three types of multi-
modality errors on the IWSLT16 DE-EN validation set. Repetitive tokens are in
red. Source words that are not semantically translated are in bold and
underlined. Wrong lexical choices (for polysemous words) and redundant words
are in blue. F-NAT denotes only modeling forward dependencies while FB-NAT
denotes modeling both forward and backward dependencies, the same as the
models in Table 5. Case studies of our proposed IT approach are in Appendix.

Previous research (Gu and Kong, 2021) argues that reducing dependencies is
crucial for training a fully NAT model effectively, as it allows the model to
more easily capture target dependencies. However, dependency reduction limits
the performance upper bound of fully NAT models, since models may struggle to
generate complex sentences. Previous studies show that multi-modality (Ran et
al., 2020) is the main problem that NAT models suffer from (Huang et al.,
2021; Bao et al., 2022), i.e., the target tokens may be generated based on
different possible translations, often causing over-translation (token
repetitions), under-translation (source words not translated), and wrong
lexical choice for polysemous words. Table 1 Row3 shows all three multi-
modality error types from the highly competitive fully NAT model GLAT (Qian et
al., 2021) with modeling only forward dependency (F-NAT) in our experiments.
We observe that lack of complete dependency modeling could cause multi-
modality errors. For example, for the source text (in German) “Woher komme
ich?” in the last column of Table 1, “Woher” means both “where” and “how”. The
NAT model modeling only forward dependency (F-NAT) incorrectly translates
“woher” into “how” and outputs “How do I come from?”; whereas the model
modeling both forward and backward dependency (FB-NAT) translates it correctly
into “Where do I come from?”. Therefore, instead of dependency reduction, we
propose a novel and general Dependency-Aware Decoder (DePA), which enhances
the learning capacity of fully NAT models and enables them to learn _complete_
and _complex_ forward and backward target dependencies in order to alleviate
the multi-modality issue.

Firstly, we enhance the NAT decoder to learn complete target dependencies by
exploring decoder self-attention. We believe that previous works (Guo et al.,
2020a) incorporating only forward dependency modeled by AT models into NAT
models are inadequate to address multi-modality. Therefore, we propose an
effective forward-backward dependency modeling approach, denoted by FBD, as an
auto-aggressive forward-backward pre-training phase before NAT training, using
curriculum learning. The FBD approach implements triangular attention masks
and takes different decoder inputs and targets in a unified framework to train
the model to attend to previous or future tokens and learn both forward or
backward dependencies.

Secondly, we enhance target dependency modeling within the NAT decoder from
the perspective of the decoder input. Most prior NAT models (Gu et al., 2018;
Wang et al., 2019; Wei et al., 2019) use a copy of the source text embedding
as the decoder input, which is independent from the target representation
space and hence makes target dependency modeling difficult. We transform the
initial decoder input from the source language representation space to the
target language representation space through a novel attentive transformation
process, denoted by IT. Previous works on transforming the decoder input
cannot guarantee that the decoder input is in the exact target representation
space, resulting in differences from the true target-side distribution. Our
proposed IT ensures that the decoder input is in the exact target
representation space hence enables the model to better capture target
dependencies.

Our contributions can be summarized as follows: (1) We propose a novel and
general Dependency-Aware Decoder (DePA) for fully NAT models. For DePA, we
propose a novel approach FBD for learning both forward and backward
dependencies in NAT decoder, through which the target dependencies can be
better modeled. To the best of our knowledge, our work is the first to
successfully model both forward and backward target-side dependencies
explicitly for fully NAT models. We also propose a novel decoder input
transformation approach (IT). IT could ease target-side dependency modeling
and enhance the effectiveness of FBD. DePA is model-agnostic and can be
applied to any fully NAT models. (2) Extensive experiments on WMT and IWSLT
benchmarks demonstrate that our DePA consistently improves the representative
vanilla NAT model (Gu et al., 2018), the highly competitive fully NAT model
GLAT (Qian et al., 2021) and the current SOTA of fully NAT models, CTC w/ DSLP
& Mixed Training (denoted by CTC-DSLP-MT) (Huang et al., 2021) (DSLP denotes
Deep Supervision and Layer-wise Prediction), by up to +0.85 BLEU on the SOTA
CTC-DSLP-MT, +1.88 BLEU on GLAT, and +2.2 BLEU on vanilla NAT, while reserving
inference latency as other fully NAT models, about $\mathbf{{15}\times}$
speed-up over AT models. Experiments show that DePA achieves greater BLEU
gains with less speed-up loss than DSLP when applied to various fully NAT
models.

##  2 Related Work

#### Forward and Backward Dependencies

Prior works explore bidirectional decoding to improve modeling of both forward
and backward dependencies in phrase-based statistical MT (Finch and Sumita,
2009) and RNN-based MT (Zhang et al., 2018). For NAT, Guo et al. (2020a) and
Wei et al. (2019) use forward auto-regressive models to guide NAT training.
Liu et al. (2020) introduces an intermediate semi-autoregressive translation
task to smooth the shift from AT training to NAT training. However, backward
dependencies are rarely investigated in NAT.

#### Decoder Input of Fully NAT Models

The decoder input of AT models consists of previously generated tokens.
However, selecting appropriate decoder input for fully NAT models could be
challenging. Most prior NAT models (Gu et al., 2018; Wang et al., 2019; Wei et
al., 2019) use uniform copy (Gu et al., 2018) or soft copy (Wei et al., 2019)
of the source text embedding as the decoder input, which is independent of the
target representation space hence hinders target dependency modeling. Methods
such as GLAT (Qian et al., 2021) and (Guo et al., 2020a, b) attempt to make
the NAT decoder input similar to the target representation space by
substituting certain positions in the decoder input with the corresponding
target embedding. However, this creates a mismatch between training and
inference. Guo et al. (2019) uses phrase-table lookup and linear mapping to
make the decoder input closer to the target embedding, but this method still
causes difference between the decoder input and the real target-side
distribution.

#### Fully NAT Models

To address multi-modality for fully NAT models, various approaches are
proposed. Gu et al. (2018) uses knowledge distillation (KD) (Kim and Rush,
2016) to reduce dataset complexity. Libovickỳ and Helcl (2018) and Saharia et
al. (2020) use connectionist temporal classification (CTC) (Graves et al.,
2006) for latent alignment. Sun et al. (2019) utilizes CRFs to model target
positional contexts. Kaiser et al. (2018), Ma et al. (2019) and Shu et al.
(2020) incorporate latent variables to guide generation, similar to VAEs
(Kingma and Welling, 2013). Guo et al. (2020c) initializes NAT decoders with
pretrained language models. Huang et al. (2021) proposes CTC with Deep
Supervision and Layer-wise Prediction and Mixed Training (CTC-DSLP-MT),
setting new SOTA for fully NAT models on WMT benchmarks. DA-Transformer (Huang
et al., 2022) represents hidden states in a directed acyclic graph to capture
dependencies between tokens and generate multiple possible translations. In
contrast, our DePA utilizes forward-backward pre-training and a novel
attentive transformation of decoder input to enhance target dependency
modeling. Under same settings and with KD, DA-Transformer performs only
comparably to CTC-DSLP-MT; however, performance of DA-Transformer benefits
notably from Transformer-big for KD while CTC-DSLP-MT uses Transformer-base
for KD. DDRS w/ NMLA (Shao and Feng, 2022) benefits greatly from using diverse
KD references while CTC-DSLP-MT uses only a single KD reference. Hence, CTC-
DSLP-MT is still the current SOTA for fully NAT models on WMT benchmarks.

#### Non-autoregressive Models

Besides fully NAT models, iterative NAT models are proposed such as iterative
refinement of target sentences (Lee et al., 2018), masking and repredicting
words with low probabilities (Ghazvininejad et al., 2019), edit-based methods
to iteratively modify decoder output (Stern et al., 2019; Gu et al., 2019),
and parallel refinement of every token (Kasai et al., 2020). Iterative NAT
models improve translation accuracy at the cost of slower speed. Non-
autoregressive models are practically important due to high efficiency. Other
than MT, they are applied to various tasks such as image captioning (Gao et
al., 2019), automatic speech recognition (Chen et al., 2019), and text-to-
speech synthesis (Oord et al., 2018).

##  3 Methodology

###  3.1 Problem Formulation

NMT can be formulated as a sequence-to-sequence generation problem. Given a
sequence $X=\\{x_{1},...,x_{N}\\}$ in the source language, a sequence
$Y=\\{y_{1},...,y_{T}\\}$ in the target language is generated following the
conditional probability $P(Y|X)$. NAT models are proposed to speed up
generation by decoding all the target tokens in parallel, using conditional
independent factorization as:

$P_{NA}(Y|X)=P_{L}(T|x_{1:N})\cdot\prod^{T}_{t=1}P(y_{t}|x_{1:N};\theta)$ (1)

where the target sequence length $T$ is modeled by the conditional
distribution $P_{L}$, and dependence on previous target tokens is removed.
Compared to AT models, NAT models speed up inference significantly at the
expense of translation quality, because the conditional independence
assumption in Eq.1 enables parallel processing but lacks explicit modeling of
dependency between target tokens. To enhance target dependency modeling, we
propose two innovations as incorporating both forward and backward dependency
modeling into the training process (Section 3.2) and transforming the decoder
input into the target representation space (Section 3.3).

Figure 1: The proposed forward-backward dependency modeling (FBD) with
triangular attention masks in a unified framework. The red dashed lines
indicate the attention masks. We use different colors to highlight the
difference of inputs and targets in each phase.

###  3.2 Target Dependency Modeling with Curriculum Learning (FBD)

Prior work (Guo et al., 2020a) utilizes forward dependency in AT models to
initialize model parameters for NAT. However, as discussed in Section 1, for
fully NAT models, only modeling forward dependency is inadequate for
addressing the multi-modality problem (Finch and Sumita, 2009; Zhang et al.,
2018) (the Row for F-NAT in Table 1). Our innovations include incorporating
both forward and backward dependency modeling into NAT models, via triangular
attention masks in a unified framework through curriculum learning (Figure 1),
and investigating efficacy of different curricula. In Figure 1, the NAT
decoder phase denotes standard NAT training of any NAT decoder $Dec$. The
Forward Dependency and Backward Dependency phases serve pre-training for NAT
training, learning left-to-right and right-to-left dependencies to initialize
NAT models with better dependencies. Forward Dependency and Backward
Dependency training phases apply the same upper triangle attention mask on
$Dec$. We use KD data from AT models for each phase but the inputs and the
targets are different. The Forward Dependency training phase uses $y_{1}$ to
predict $y_{2}$ and so on. The Backward Dependency training phase reverses the
target sequence and uses $y_{2}$ to predict $y_{1}$ and so on. The NAT
Training phase uses features of each word to predict the word itself. We make
the following hypotheses: (1) Considering the nature of languages, learning
forward dependency in Phase 1 is easier for the model for language generation.
(2) Modeling backward dependency relies on learned forward dependency
knowledge, hence it should be in the second phase. In fact, we observe the
interesting finding that the best curriculum remains forward-backward-forward-
NAT (FBF-NAT) for both left-branching and right-branching languages, proving
our hypotheses. We speculate that NAT training may benefit from another
forward dependency modeling in Phase 3 because the order of left-to-right is
more consistent with characteristics of natural languages, hence adding the
second forward dependency modeling after FB (i.e., FBF) smooths the transition
to the final NAT training. Detailed discussions are in Section 4.3.

###  3.3 Decoder Input Transformation (IT) for Target Dependency Modeling

Figure 2: The proposed Decoder Input Transformation (IT) from $z$ to
$z^{\prime}$, where $z\in\mathbb{R}^{T\times d}$ denotes the initial decoder
input copied from the source text embedding $x_{emb}$, $T$ and $d$ denote the
length of the target text $y$ and the size of hidden states, respectively.
$Emb\in\mathbb{R}^{d\times v}$ denotes the output embedding matrix of the
decoder (the target representation space), where $v$ denotes the size of the
target vocabulary.

Given the initial decoder input $z$ as a copy of source text embedding, we
propose to directly select relevant representations from target embedding to
form a new decoder input $z^{\prime}$ (Figure 2). $z$ is used as the query and
the selection is implemented as a learnable attention module. The learnable
parameters bridge the gap between training and inference while the selection
guarantees consistency between the decoder input matrix and the target
representation space (i.e., the output embedding matrix of the decoder). This
way, the decoder input is in the exact target-side embedding space and more
conducive to modeling target dependencies for NAT models than previous
approaches using source text embedding or transformed decoder input.

#### Decoder Input Transformation

To transform $z$ into the target representation space, we apply attention
mechanism between $z$ and the output embedding matrix
$Emb\in\mathbb{R}^{d\times v}$, where $d$ and $v$ denote sizes of hidden
states and the target vocabulary. Since NAT models usually have embedding
matrix $Emb$ including both source and target vocabularies, first, we conduct
a filtering process to remove source vocabulary (mostly not used by the
decoder) from the decoder output embedding matrix (the linear layer before
decoder softmax). We build a dictionary that contains only target-side tokens
in the training set. We then use this dictionary to filter $Emb$ and obtain
the new output embedding matrix of the decoder
$Emb^{\prime}\in\mathbb{R}^{d\times v^{\prime}}$, where $v^{\prime}$ denotes
size of the filtered vocabulary. This filtering process guarantees that
$Emb^{\prime}$ is strictly from the target representation space. The attention
process starts with a linear transformation:

$z^{l}=W_{q}\cdot z$ (2)

Next, the dot-product attention is performed on $z^{l}$ (as query) and
$Emb^{\prime}$ (as key and value):

$Sim={\rm softmax}(z^{l}\cdot Emb^{\prime})$ (3)

$Sim$ represents similarity between each $z^{l}_{i}$ and each embedding in the
target vocabulary. Finally, we compute a weighted sum $z^{\prime}$ of target
embedding based on their similarity values:

$z^{\prime}=Sim\cdot Emb^{\prime T}$ (4)

Since $z^{\prime}$ is a linear combination of $Emb^{\prime}$ which is strictly
in the target representation space, $z^{\prime}$ is also strictly in the
target representation space, hence using $z^{\prime}$ as the decoder input
provides a more solid basis for target dependency modeling.

#### Target-side Embedding Compression

To reduce the computational cost of IT, we propose a target-side embedding
compression approach to compress the large target embedding matrix. We process
$Emb^{\prime}$ through a linear layer to obtain a new target embedding
$Emb^{*}\in\mathbb{R}^{d\times v^{*}}$:

$Emb^{*}=(W_{c}\cdot Emb^{\prime T})^{T}$ (5)

where $W_{c}\in\mathbb{R}^{v^{*}\times v^{\prime}}$ is trainable and the size
of compressed vocabulary $v^{*}$ is set manually. The result $Emb^{*}$ is
still in the target representation space. Since we can manually set $v^{*}$ as
a relatively small number (e.g., 1000, 2000), the computational cost of the
attention mechanism can be greatly reduced. We hypothesize that target-side
embedding compression may also alleviate over-fitting on small datasets and
confirm this hypothesis in Section 4.3.

##  4 Experiments

###  4.1 Experimental Setup

#### Datasets

We compare our methods with prior works on widely used MT benchmarks for
evaluating NAT models: WMT14 EN$\leftrightarrow$DE (4.5M pairs), WMT16
EN$\leftrightarrow$RO (610K pairs). Also, we use IWSLT16 DE-EN (196K pairs),
IWSLT14 DE-EN (153K pairs), and SP EN-
JA222https://github.com/odashi/small_parallel_enja (50K pairs) for further
analysis. For WMT16 EN$\leftrightarrow$RO and IWSLT16 DE-EN, we adopt the
processed data from Lee et al. (2018). For WMT14 EN$\leftrightarrow$DE, we
apply the same preprocessing and learn subwords as Gu and Kong (2021). For
IWSLT14 DE-EN, we follow preprocessing in Guo et al. (2019). For SP EN-JA, we
use sentencepiece333https://github.com/google/sentencepiece to tokenize the
text into subword units following Chousa et al. (2019). Following prior works,
we share the source and target vocabulary and embeddings in each language pair
in $Emb$, except EN-JA. Also following prior works (Gu et al., 2018; Qian et
al., 2021), all NAT models in our experiments are trained on data generated
from pre-trained AT Transformer-base with _sequence-level knowledge
distillation (KD)_ for all datasets except EN-JA.

#### Baselines and Training

We implement the baseline models based on their released codebases. We
implement the representative vanilla NAT (Gu et al., 2018; Qian et al., 2021;
Huang et al.,
2021)444https://github.com/facebookresearch/fairseq/tree/main/examples/nonautoregressive_translation,
the highly competitive fully NAT model GLAT (Qian et al.,
2021)555https://github.com/FLC777/GLAT, and current fully NAT SOTA CTC w/ DSLP
& Mixed Training (CTC-DSLP-MT) (Huang et al.,
2021)666https://github.com/chenyangh/DSLP and apply our methods to them.
Following Qian et al. (2021), we use base-Transformer ($d_{model}$=512,
$n_{head}$=8, $n_{layer}$=6) for WMT datasets and small-Transformer
($d_{model}$=256, $n_{head}$=4, $n_{layer}$=5) for IWSLT and SP EN-JA
datasets. We use the same training setup for training the three models,
Vanilla NAT , GLAT, and CTC-DSLP-MT as in their original papers cited above.
We train models with batches of 64K tokens for WMT datasets, and 8K tokens for
IWSLT and SP EN-JA datasets, using NVIDIA V100 GPUs. For GLAT, we use Adam
optimizer (Kingma and Ba, 2015) with $\beta$ = (0.9, 0.999) and set dropout
rate to 0.1. For Vanilla NAT and CTC-DSLP-MT, we use Adam optimizer (Kingma
and Ba, 2015) with $\beta$ = (0.9, 0.98). For WMT datasets, the learning rate
warms up to 5$e$-4 in 4K steps and gradually decays according to inverse
square root schedule (Vaswani et al., 2017). As for IWSLT and SP EN-JA
datasets, we adopt linear annealing (from 3$e$-4 to 1$e$-5 ) as in Lee et al.
(2018). We choose the model with the best performance on the validation set as
the final model and evaluate the final model on the test sets. For experiments
using our method FBD (Section 3.2), we use the FBF-NAT configuration (as in
Section 4.3) and train the same number of steps at each phase (including NAT
training phase), with 300K steps for each phase for WMT datasets and 100K
steps for each phase for IWSLT datasets and SP EN-JA. IT by default is without
Target-side Embedding Compression (Section 3.3).

#### Evaluation

To evaluate the translation accuracy, we use SacreBLEU (Post, 2018) for all
experiments and ChrF (Popovic, 2015) (also using the SacreBLEU tool)
additionally for ablation study on IWSLT benchmark. To evaluate the inference
latency, following Gu and Kong (2021), we measure the wall-clock time for
translating the entire WMT14 EN-DE test set with batch_size=1 on a single
NVIDIA V100 GPU, then compute the average time per sentence. We report Speed-
up based on the inference latency of Transformer-base AT (teacher) and fully
NAT models.

Row# | Models | Speed-up $\uparrow$ | WMT’14 | WMT’16
---|---|---|---|---
EN-DE | DE-EN | EN-RO | RO-EN
1 | Transformer-base (teacher) | 1.0$\times$ | 27.48 | 31.39 | 33.70 | 34.05
2 |  \+ KD | 2.5$\times$ | 27.34 | 30.95 | 33.52 | 34.01
3 | Vanilla NAT | 15.6$\times$ | 20.36 | 24.81 | 28.47 | 29.43
4 |  w/ DSLP∗ | 14.8$\times$ | 22.72 | 25.83 | 30.48 | 31.46
5 |  w/ DePA (Ours) | 15.4$\times$ | 23.15 | 26.59 | 30.78 | 31.89
6 | GLAT | 15.3$\times$ | 25.21 | 29.84 | 31.19 | 32.04
7 |  w/ DSLP∗ | 14.9$\times$ | 25.69 | 29.90 | 32.36 | 33.06
8 |  w/ DePA (Ours) | 15.1$\times$ | 26.43 | 30.42 | 33.07 | 33.82
10 | CTC∗ | 15.5$\times$ | 25.72 | 29.89 | 32.89 | 33.79
11 |  w/ DSLP∗ | 14.8$\times$ | 26.85 | 31.16 | 33.85 | 34.24
12 |  w/ DSLP & Mixed Training | 14.8$\times$ | 27.02 | 31.61 | $33.99$ | $34.42$
13 |  w/ DSLP & Mixed Training & w/ DePA (Ours) | 14.7$\times$ | 27.51 | 31.96 | 34.48 | 34.77
14 | Average improvement from DSLP | - | 1.32 | 0.78 | 1.38 | 1.17
15 | Average improvement from DePA (Ours) | - | 1.50 | 0.90 | 1.56 | 1.53

Table 2: BLEU and Speed-up from our DePA and existing methods on WMT benchmark
test sets. Speed-up is measured on WMT14 EN-DE test set. BLEUs without
rescoring are reported, with the best BLEU scores in bold for each group. ∗
denotes the results are copied from previous work (Huang et al., 2021), other
results are obtained by our implementation. Average improvements of DSLP are
re-calculated using our results, which are slightly different from Table 1 in
Huang et al. (2021).

###  4.2 Main Results

Table 2 shows the main results on the WMT benchmarks. For
EN$\leftrightarrow$RO, we report the mean of BLEU from $3$ runs with different
random seeds for Row 12-13, all with quite small standard deviations ($\leq
0.16$) 777WMT14 EN$\leftrightarrow$DE is much larger than WMT16
EN$\leftrightarrow$RO. Since standard deviations of BLEU from multiple runs
with different random seeds on WMT14 EN$\leftrightarrow$DE are very small,
$\leq 0.08$ (Huang et al., 2022), following prior works, we report single-run
BLEU on WMT14 EN$\leftrightarrow$DE to save energy.. We apply our proposed
DePA, which includes IT and FBD, to vanilla NAT, GLAT, and the current fully
NAT SOTA CTC-DSLP-MT, on WMT, IWSLT, and EN-JA benchmarks. We use the same
hyperparameters and random seeds to fairly compare two models. It is crucial
to point out that accuracies of vanilla NAT, GLAT, and CTC-DSLP-MT models have
plateaued out after 300K training steps on WMT datasets hence original papers
of these three models set max training steps to 300K. We verify this
observation in our own experiments as we also see no gains on these models
after 300K training steps on the WMT datasets. Hence, although our DePA trains
$300K\times 4=1200K$ steps on WMT datasets due to FBF pre-training as in
Section 4.3, all comparisons between baselines w/ DePA and w/o DePA are fair
comparisons. Table 2 shows that DePA consistently improves the translation
accuracy for both vanilla NAT and GLAT on each benchmark, achieving mean=+1.37
and max=+1.88 BLEU gain on GLAT and mean=+2.34 and max=+2.46 BLEU gain on
vanilla NAT. DePA also improves the SOTA CTC-DSLP-MT by mean=+0.42 and
max=+0.49 BLEU gain on the WMT test sets (Table 2), +0.85 BLEU gain on the
IWSLT16 DE-EN validation set and +1.43 BLEU gain on the EN-JA test set (Table
3). All gains from DePA on vanilla NAT, GLAT, and CTC-DSLP-MT are
statistically significant ($p<0.05$) based on a paired bootstrap resampling
test conducted using 1K resampling trials and the SacreBLEU tool.

Table 2 also shows that on each benchmark, the average improvement from DePA
on three models (vanilla NAT, GLAT, and CTC-DSLP-MT) is within [0.90,1.56]
(Row15), always larger than the average improvement from w/DSLP on them,
[0.78,1.38] (Row14). DePA brings consistent improvement over SOTA CTC-DSLP-MT
on all benchmarks (Table 2 Row13-over-Row12, Table 3), hence we expect DePA to
also improve DA-Transformer (Huang et al., 2022) and DDRS w/ NMLA (Shao and
Feng, 2022) and will verify this w/ and w/o KD in future work. Applying DePA
to fully NAT models retains the inference speed-up advantages of fully NAT
models. Applying DePA to vanilla NAT, GLAT, and SOTA CTC-DSLP-MT obtain
$\mathbf{{15.4}\times}$, $\mathbf{{15.1}\times}$, and $\mathbf{{14.7}\times}$
speed-up over the autoregressive Transformer-base (teacher) (Row1). Overall
Table 2 shows that DePA achieves greater BLEU gains with less speed-up loss
than DSLP on all baselines. These results demonstrate superiority of DePA over
DSLP on improving other fully NAT models.

Models | IWSLT16 | WMT’14 | WMT’16
---|---|---|---
DE-EN | EN-DE | DE-EN | EN-RO | RO-EN
BLEU | ChrF | BLEU | BLEU
CTC-DSLP-MT | 31.04 | 56.7 | 27.02 | 31.61 | 34.17 | 34.60
CTC-DSLP-MT w/ IT | 31.29 | 57.1 | 27.21 | 31.78 | 34.32 | 34.71
CTC-DSLP-MT w/ FBD | 31.73 | 57.5 | 27.44 | 31.90 | 34.60 | 34.92
CTC-DSLP-MT w/ IT+FBD | 31.89 | 57.8 | 27.51 | 31.96 | 34.68 | 34.98
Models | IWSLT16 | EN-JA
DE-EN |
BLEU | ChrF | BLEU
GLAT | 29.61 | 51.8 | 27.67
GLAT (400K step) | 29.68 | 52.1 | –
GLAT w/ IT | 29.95 | 52.8 | 27.95
GLAT w/ FBD | 30.87 | 53.3 | 28.87
GLAT w/ IT+FBD | 31.01 | 54.5 | 29.10

Table 3: Effect of IT and FBD and IT+FBD (i.e., DePA) on the IWSLT16 DE-EN
validation set, the WMT and SP EN-JA test sets. We report mean of BLEU/ChrF
from $3$ runs with different random seeds. BLEU gains from DePA on SOTA CTC-
DSLP-MT on each set, [0.85, 0.49, 0.51], are larger than std ($\leq 0.17$).

###  4.3 Analysis

Models | BLEU
---|---
Vanilla NAT (Guo et al., 2019) | 22.95
Vanilla NAT w/ Linear Mapping (Guo et al., 2019) | 24.13
Vanilla NAT (our implementation) | 23.26
Vanilla NAT w/ IT | 26.44

Table 4: Compare IT and Linear Mapping (Guo et al., 2019) on vanilla NT on the
IWSLT14 DE-EN test set.

#### Ablation Study

We analyze the respective efficacy of IT and FBD in DePA on the IWSLT16 DE-EN
validation and the WMT and SP EN-JA test sets. Table 3 shows that FBD and IT
improve GLAT by +1.26 BLEU/+1.5 ChrF and +0.34 BLEU/+1.0 ChrF on IWSLT16 DE-EN
validation set, respectively. Considering that GLAT w/FBD has more training
steps than GLAT, we also train GLAT (400K steps) which has the same training
steps as GLAT w/FBD for fair comparison. Similar to findings on WMT datasets,
we observe plateaus of accuracy on IWSLT and EN-JA datasets from more training
steps than the original 100K. Just training more steps hardly improves the
baseline (only +0.07 BLEU gain) on IWSLT16 DE-EN, whereas GLAT w/FBD brings
+1.19 BLEU/+1.2 ChrF gains over GLAT (400K steps).

Table 4 shows our IT outperforms Linear Mapping (Guo et al., 2019) by +2.31
BLEU gain on IWSLT14 DE-EN test set. IT has the same number of extra
parameters as Linear Mapping. Hence, the large gain proves that improvements
from IT are not just from additional layers. The number of extra parameters of
IT, as from $W_{q}$ in Eq.2, is quite small: 512*512=262144 for Transformer-
base on WMT datasets and 256*256=65536 for Transformer-small on IWSLT
datasets. The large BLEU gain +3.18 from applying IT to vanilla NAT proves
vanilla transformer decoder cannot achieve similar transformation
effectiveness as IT. Table 3 shows that for language pairs with different
levels of source-target vocabulary sharing, such as WMT EN-DE and DE-EN, IWSLT
DE-EN, EN-RO, and EN-JA, our IT method can achieve consistent improvements
over GLAT and CTC-DSLP-MT. Applying IT consistently improves GLAT and CTC-
DSLP-MT although these gains are smaller than gain on vanilla NAT. This is
because decoder input of vanilla NAT only replicates source embedding, whereas
GLAT and CTC-DSLP-MT already transform decoder input by replacing selected
positions in decoder input with target embedding, hence reducing improvements
of IT. Still, gains from w/IT+FBD over w/FBD confirms our hypothesis that IT
can enhance effectiveness of FBD. On GLAT, IT+FBD yields +1.4 BLEU/+2.7 ChrF
gains on IWSLT16 DE-EN and +1.43 BLEU on EN-JA and on SOTA CTC-DSLP-MT, +0.85
BLEU/+1.1 ChrF gain on IWSLT16 DE-EN.

To further analyze IT, we compare cosine similarity between the target
embedding against the original decoder input and the transformed decoder
input, respectively. For each sample in the IWSLT16 DE-EN validation set, we
average all its token embeddings as the decoder input representation and the
same for the target representation and then compute cosine similarity. We
average similarities of all samples as the final similarity. We find that IT
significantly improves similarity between the decoder input and the target
representation, $0.04951\rightarrow 0.14521$ for GLAT and $0.04837\rightarrow
0.14314$ for vanilla NAT.

#### Impact of Different Dependency Curricula in FBD

Table 5 presents results from applying different forward-backward dependency
modeling curricula (Figure 1) on GLAT on the IWSLT16 DE-EN validation and the
SP EN-JA test sets. Compared with modeling backward dependency in Phase 1
(B-NAT and BF-NAT), modeling forward dependency in Phase 1 (F-NAT , FB-NAT,
and FBF-NAT) performs notably better. FB-NAT outperforms BF-NAT by +3.04 BLEU
on IWSLT16 DE-EN and +2.08 BLEU on EN-JA. It seems that forward dependency
modeling achieves good initialization for subsequent training phases, while
backward dependency modeling cannot. We observe the best curriculum as FBF-
NAT, i.e., first learn forward dependency, next learn backward dependency,
then another round of forward dependency training before NAT training. Table 5
shows the same trend of curricula on SP EN-JA as on IWSLT16 DE-EN, with FBF-
NAT performing best, demonstrating that this trend of forward-backward
dependency modeling curricula is consistent for both right-branching (English)
and left-branching (Japanese) target languages. All these observations confirm
our hypotheses in Section 3.2. Our FBF-NAT consistently outperforms baseline
GLAT (denoted by NAT in Table 5) by +1.58 on IWSLT16 DE-EN and +1.56 on SP EN-
JA and outperforms prior works modeling forward dependency only (Guo et al.,
2020a) on GLAT (denoted by F-NAT in Table 5) by +1.15 on DE-EN and +1.08 on
EN-JA.

#### DePA on Raw Data

We evaluate DePA on raw data by training models on the original training set
without KD (Section 4.1). DePA improves GLAT on the IWSLT16 DE-EN validation
set by +1.57 BLEU (26.57 $\rightarrow$ 28.14), proving that DePA effectively
enhances the dependency modeling ability of fully NAT models hence reduces
dependence of NAT training on AT models.

IWSLT16 DE-EN validation set | SP EN-JA test set
---|---
Models | BLEU | Models | BLEU | Models | BLEU | Models | BLEU
NAT | 29.61 | BF-NAT | 27.83 | NAT | 27.67 | BF-NAT | 26.79
F-NAT | 30.04 | FB-NAT | 30.87 | F-NAT | 28.15 | FB-NAT | 28.87
B-NAT | 27.05 | FBF-NAT | 31.19 | B-NAT | 25.83 | FBF-NAT | 29.23

Table 5: BLEU from different dependency modeling curricula on GLAT. Best
results for each set are in bold. NAT denotes GLAT baseline. F and B denote
forward dependency and backward dependency phase respectively (Figure 1). For
example, F-NAT denotes forward dependency training then NAT training.

(a) NAT

(b) F-NAT

(c) B-NAT

(d) FB-NAT

Figure 3: Visualization of the decoder self-attention distribution in NAT
models on IWSLT16 DE-EN validation set. Definitions of model names are the
same as in Table 5.

#### Effectiveness of Target-side Embedding Compression

We propose a linear compression module to reduce the selection candidates of
the target embedding for IT (Section 3.3). We use the dichotomy to determine
the compression dimension interval $[1000,2000]$ and evaluate GLAT w/ IT using
different dimensions with step size 200 in this interval for IT on the IWSLT16
DE-EN validation set. As shown in Table 6, applying IT on GLAT improves BLEU
up to +0.78 (29.61$\rightarrow$ 30.39) with compressed dimension 1800. We also
experiment with target-side embedding compression on a larger model on WMT16
EN-RO but find no gains. We assume that for relatively small models and data,
this approach helps filter out some redundant target information, hence
refines the target representation space and improves the translation accuracy.

Compressed | w/o IT | 1000 | 1200 | 1400 | 1600 | 1800 | 2000
---|---|---|---|---|---|---|---
Dimension
BLEU | 29.61 | 29.45 | 29.56 | 29.77 | 29.85 | 30.39 | 29.14

Table 6: BLEU from GLAT w/ IT on the IWSLT16 DE-EN validation set with Target-
side Embedding Compression described in Section 3.3.

###  4.4 Case Study and Visualization

Table 1 presents case studies of GLAT w/ FBD (“FB-NAT”) and with only forward
modeling (“F-NAT”) on IWLST16 DE-EN validation set. Some typical multi-
modality errors in F-NAT predictions are corrected by incorporating both
forward and backward dependency modeling through FBD.

For a more intuitive analysis of FBD, we present a visualization of the
decoder self-attention distribution of different NAT models in Figure 3. All
models are based on GLAT and model names conform to those in Table 5. In the
baseline GLAT (Figure 3(a)), the self-attention distribution of each position
is scattered in adjacent positions, indicating that the NAT model lacks
dependency and has high confusion during decoding, causing multi-modality
errors. In F-NAT and B-NAT models, significant forward and backward
dependencies can be observed in Figure 3(b) and 3(c), indicating that these
two models can better use information in previous or future positions.
Encouragingly, forward and backward dependencies are fused in the FB-NAT model
(Figure 3(d)), which can focus on future information while modeling forward
dependency, capable of alleviating problems shown in Table 1.

##  5 Conclusion

We propose a novel and general Dependency-Aware Decoder (DePA) to enhance
target dependency modeling for fully NAT models, with forward-backward
dependency modeling and decoder input transformation. Extensive experiments
show that DePA improves the translation accuracy of highly competitive and
SOTA fully NAT models while preserving their inference latency. In future
work, we will evaluate DePA on iterative NAT models such as Imputer, CMLM, and
Levenshtein Transformer and incorporate ranking approaches into DePA.

##  6 Limitations

Apart from all the advantages that our work achieves, some limitations still
exist. Firstly, in this work, we investigate the efficacy of applying our
proposed DePA approach on the representative vanilla NAT, the highly
competitive fully NAT model GLAT and current SOTA CTC-DSLP-MT for fully NAT
models, but we have yet to apply DePA to iterative NAT models, such as Imputer
Saharia et al. (2020), CMLM Ghazvininejad et al. (2019), and Levenshtein
Transformer Gu et al. (2019). Hence, the effectiveness of DePA on iterative
NAT models still needs to be verified. Secondly, we have not yet incorporated
re-ranking approaches such as Noisy Parallel Decoding (NPD) Gu et al. (2018)
into DePA. Thirdly, our proposed method FBD requires multiple additional
training phases before NAT training, resulting in longer training time and
using more GPU resources. Reducing the computational cost of FBD training is
one future work that will be beneficial for energy saving. Last but not least,
NAT models have limitations on handling long text. They suffer from worse
translation quality when translating relatively long text. We plan to
investigate all these topics in future work.

## References

  * Bao et al. (2022) Yu Bao, Hao Zhou, Shujian Huang, Dongqi Wang, Lihua Qian, Xinyu Dai, Jiajun Chen, and Lei Li. 2022.  GLAT: glancing at latent variables for parallel text generation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 8398–8409. Association for Computational Linguistics.
  * Chen et al. (2019) Nanxin Chen, Shinji Watanabe, Jesús Villalba, and Najim Dehak. 2019.  Listen and fill in the missing letters: Non-autoregressive transformer for speech recognition.  _arXiv preprint arXiv:1911.04908_.
  * Chousa et al. (2019) Katsuki Chousa, Katsuhito Sudoh, and Satoshi Nakamura. 2019.  Simultaneous neural machine translation using connectionist temporal classification.  _arXiv preprint arXiv:1911.11933_.
  * Finch and Sumita (2009) Andrew M. Finch and Eiichiro Sumita. 2009.  Bidirectional phrase-based statistical machine translation.  In _Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, EMNLP 2009, 6-7 August 2009, Singapore, A meeting of SIGDAT, a Special Interest Group of the ACL_ , pages 1124–1132. ACL.
  * Gao et al. (2019) Junlong Gao, Xi Meng, Shiqi Wang, Xia Li, Shanshe Wang, Siwei Ma, and Wen Gao. 2019\.  Masked non-autoregressive image captioning.  _arXiv preprint arXiv:1906.00717_.
  * Ghazvininejad et al. (2019) Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019.  Mask-predict: Parallel decoding of conditional masked language models.  _arXiv preprint arXiv:1904.09324_.
  * Graves et al. (2006) Alex Graves, Santiago Fernández, Faustino J. Gomez, and Jürgen Schmidhuber. 2006.  Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks.  In _Machine Learning, Proceedings of the Twenty-Third International Conference (ICML 2006), Pittsburgh, Pennsylvania, USA, June 25-29, 2006_ , volume 148 of _ACM International Conference Proceeding Series_ , pages 369–376. ACM.
  * Gu et al. (2018) Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. 2018\.  Non-autoregressive neural machine translation.  In _International Conference on Learning Representations_.
  * Gu and Kong (2021) Jiatao Gu and Xiang Kong. 2021.  Fully non-autoregressive neural machine translation: Tricks of the trade.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 120–133, Online. Association for Computational Linguistics.
  * Gu et al. (2019) Jiatao Gu, Changhan Wang, and Jake Zhao. 2019.  Levenshtein transformer.  _arXiv preprint arXiv:1905.11006_.
  * Guo et al. (2019) Junliang Guo, Xu Tan, Di He, Tao Qin, Linli Xu, and Tie-Yan Liu. 2019.  Non-autoregressive neural machine translation with enhanced decoder input.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 3723–3730.
  * Guo et al. (2020a) Junliang Guo, Xu Tan, Linli Xu, Tao Qin, Enhong Chen, and Tie-Yan Liu. 2020a.  Fine-tuning by curriculum learning for non-autoregressive neural machine translation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 7839–7846.
  * Guo et al. (2020b) Junliang Guo, Linli Xu, and Enhong Chen. 2020b.  Jointly masked sequence-to-sequence model for non-autoregressive neural machine translation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 376–385.
  * Guo et al. (2020c) Junliang Guo, Zhirui Zhang, Linli Xu, Hao-Ran Wei, Boxing Chen, and Enhong Chen. 2020c.  Incorporating bert into parallel sequence decoding with adapters.  In _NeurIPS_.
  * Huang et al. (2021) Chenyang Huang, Hao Zhou, Osmar R Zaïane, Lili Mou, and Lei Li. 2021.  Non-autoregressive translation with layer-wise prediction and deep supervision.  _arXiv preprint arXiv:2110.07515_.
  * Huang et al. (2022) Fei Huang, Hao Zhou, Yang Liu, Hang Li, and Minlie Huang. 2022.  Directed acyclic transformer for non-autoregressive machine translation.  _CoRR_ , abs/2205.07459.
  * Kaiser et al. (2018) Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, and Noam Shazeer. 2018.  Fast decoding in sequence models using discrete latent variables.  In _International Conference on Machine Learning_ , pages 2390–2399. PMLR.
  * Kasai et al. (2020) Jungo Kasai, James Cross, Marjan Ghazvininejad, and Jiatao Gu. 2020.  Non-autoregressive machine translation with disentangled context transformer.  In _International Conference on Machine Learning_ , pages 5144–5155. PMLR.
  * Kim and Rush (2016) Yoon Kim and Alexander M Rush. 2016.  Sequence-level knowledge distillation.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1317–1327.
  * Kingma and Ba (2015) Diederik P Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _ICLR (Poster)_.
  * Kingma and Welling (2013) Diederik P Kingma and Max Welling. 2013.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_.
  * Lee et al. (2018) Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018.  Deterministic non-autoregressive neural sequence modeling by iterative refinement.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1173–1182.
  * Libovickỳ and Helcl (2018) Jindřich Libovickỳ and Jindřich Helcl. 2018.  End-to-end non-autoregressive neural machine translation with connectionist temporal classification.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3016–3021.
  * Liu et al. (2020) Jinglin Liu, Yi Ren, Xu Tan, Chen Zhang, Tao Qin, Zhou Zhao, and Tie-Yan Liu. 2020\.  Task-level curriculum learning for non-autoregressive neural machine translation.  In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020_ , pages 3861–3867. ijcai.org.
  * Ma et al. (2019) Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig, and Eduard Hovy. 2019.  Flowseq: Non-autoregressive conditional sequence generation with generative flow.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4282–4292.
  * Oord et al. (2018) Aaron Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg, et al. 2018.  Parallel wavenet: Fast high-fidelity speech synthesis.  In _International conference on machine learning_ , pages 3918–3926. PMLR.
  * Popovic (2015) Maja Popovic. 2015.  chrf: character n-gram f-score for automatic MT evaluation.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation, WMT@EMNLP 2015, 17-18 September 2015, Lisbon, Portugal_ , pages 392–395. The Association for Computer Linguistics.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers, WMT 2018, Belgium, Brussels, October 31 - November 1, 2018_ , pages 186–191. Association for Computational Linguistics.
  * Qian et al. (2021) Lihua Qian, Hao Zhou, Yu Bao, Mingxuan Wang, Lin Qiu, Weinan Zhang, Yong Yu, and Lei Li. 2021.  Glancing transformer for non-autoregressive neural machine translation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1993–2003, Online. Association for Computational Linguistics.
  * Ran et al. (2020) Qiu Ran, Yankai Lin, Peng Li, and Jie Zhou. 2020.  Learning to recover from multi-modality errors for non-autoregressive neural machine translation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3059–3069.
  * Saharia et al. (2020) Chitwan Saharia, William Chan, Saurabh Saxena, and Mohammad Norouzi. 2020.  Non-autoregressive machine translation with latent alignments.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1098–1108.
  * Shao and Feng (2022) Chenze Shao and Yang Feng. 2022.  Non-monotonic latent alignments for ctc-based non-autoregressive machine translation.  _arXiv preprint arXiv:2210.03953_.
  * Shu et al. (2020) Raphael Shu, Jason Lee, Hideki Nakayama, and Kyunghyun Cho. 2020.  Latent-variable non-autoregressive neural machine translation with deterministic inference using a delta posterior.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 8846–8853.
  * Stern et al. (2019) Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. 2019.  Insertion transformer: Flexible sequence generation via insertion operations.  In _International Conference on Machine Learning_ , pages 5976–5985. PMLR.
  * Sun et al. (2019) Zhiqing Sun, Zhuohan Li, Haoqing Wang, Di He, Zi Lin, and Zhi-Hong Deng. 2019.  Fast structured decoding for sequence models.  In _NeurIPS_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Wang et al. (2019) Yiren Wang, Fei Tian, Di He, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu. 2019.  Non-autoregressive machine translation with auxiliary regularization.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 5377–5384.
  * Wei et al. (2019) Bingzhen Wei, Mingxuan Wang, Hao Zhou, Junyang Lin, and Xu Sun. 2019.  Imitation learning for non-autoregressive neural machine translation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1304–1312.
  * Zhang et al. (2018) Xiangwen Zhang, Jinsong Su, Yue Qin, Yang Liu, Rongrong Ji, and Hongji Wang. 2018\.  Asynchronous bidirectional decoding for neural machine translation.  In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018_ , pages 5698–5705. AAAI Press.

| Case #1 | Case #2
---|---|---
Source | obwohl sie erwischt wurden , wurden sie schließlich | das ist ein Bauplan für Länder wie China und den Iran .
freigelassen aufgrund immensen internationalen Drucks .
Target Reference | even though they were caught , they were eventually | this is a blueprint for countries like China and Iran .
released after heavy international pressure .
Vanilla NAT | although they were caught , they were released released | this is a blueprint plan for countries like China and and Iran .
because because of huge drug .
Vanilla NAT w/ IT | although they were caught , they were finally released | this is a blueprint for countries like China and Iran .
because huge international pressure .
GLAT | although they were caught , they finally were released | this is a blueprint plan for countries like China and Iran .
because of of international printing .
GLAT w/ IT | although they were caught , they were finally | this is a blueprint for countries like China and Iran .
released after huge international pressure .

Table 7: Case studies of our method IT on the IWSLT16 DE-EN validation set by
comparing the translations from the two baseline models Vanilla NAT and GLAT
and from them after applying IT (models in bold). Repetitive tokens are in
red. Source words that are not semantically translated are marked in bold and
underlined (under-translation). Wrong lexical choice (incorrect translations
caused by polysemy) and redundant words are in blue.

#### Case study for the proposed IT

NAT models generally suffer from the multi-modality problem, which shows as
_over-translation_ (repetition), _under-translation_ (missing information),
and _wrong lexical choice_ (incorrect translations caused by polysemy) Ran et
al. (2020). As shown in Table 7, Vanilla NAT and GLAT tend to generate
repetitive tokens which are highlighted in red (over-translation).
Additionally, Vanilla NAT omits the translation of “schließlich” which is in
bold and underlined (under-translation). By applying our method IT, the
decoder input is closer to the target representation space and the model has a
better perception for the target-side information, so that the repetition and
under-translation problems can be effectively alleviated. As for incorrect
translations caused by polysemy, as shown in Case #1 in Table 7, “Drucks”
means both “printing” and “pressure” in German. GLAT mistakenly translates
“Drucks” into “printing”, but our method can help the model correctly
translate it into “pressure”. Particularly, in Case #2, “Bauplan” means
“blueprint” in German. Although both baseline models Vanilla NAT and GLAT
generate the correct words, they also generate the redundant word “plan” which
is also a subword of “Bauplan”. These examples demonstrate that the baseline
models may confuse the source representation space with the target
representation space during generation, but our method IT effectively remedies
this problem.