# Scintillation detectors with silicon photomultiplier readout in a dilution
refrigerator at temperatures down to 0.2 K

J. Zhang  D. Goeldi  R. Iwai  M. Sakurai  and A. Soter, 11footnotetext:
Corresponding author.

###### Abstract

We are developing a novel high-brightness atomic beam, comprised of a two-body
exotic atom called muonium (M $=\mu^{+}+e^{-}$), for next-generation atomic
physics and gravitational interaction measurements. This M source originates
from a thin sheet of superfluid helium (SFHe), hence diagnostics and later
measurements require a detection system which is operational in a dilution
cryostat at temperatures below $1\text{\,}\K$. In this paper, we describe the
operation and characterization of silicon photomultipliers (SiPMs) at ultra-
low temperatures in SFHe targets. We show the temperature dependence of the
signal shape, breakdown voltage, and single photon detection efficiency,
concluding that single photon detection with SiPMs below $0.85\text{\,}\K$ is
feasible. Furthermore, we show the development of segmented scintillation
detectors, where 16 channels at $1.7\text{\,}\K$ and one channel at
$170\text{\,}\mathrm{m}\K$ were commissioned using a muon beam.

##  1 Introduction

The operation of silicon photomultipliers (SiPMs) at very low temperatures has
the potential to improve detector systems for various experiments conducted at
cryogenic temperatures that require single photon detection or sensitivity in
the vacuum ultraviolet (VUV) range. In rare event searches, SiPMs are
promising candidates to replace photomultiplier tubes (PMTs) to detect VUV
scintillation photons produced by particle interactions with liquid xenon and
argon. Examples are the dark matter experiments DARWIN [1] or DarkSide-20k
[2], nEXO [3] looking for neutrino-less double beta decay, or MEGII [4]
searching for the charged lepton flavour violating decay of muons. The
characterization of SiPMs has thus been done at room temperature [5] and down
to liquid-nitrogen temperatures ($77\text{\,}\mathrm{K}$) [6, 7].

The detection of scintillation light at even lower temperatures is of interest
for several experiments conducted at the Paul Scherrer Institute (PSI), like
the muCool experiment, where positive muons are moderated in helium gas
targets at $\sim$ $4\text{\,}\mathrm{K}$ to $10\text{\,}\mathrm{K}$
temperatures [8, 9]. In a new experiment, we are developing a high-quality
atomic muonium (M $=\mu^{+}+e^{-}$) source, produced in a
$150\text{\,}\mathrm{mK}$ to $200\text{\,}\mathrm{mK}$ superfluid helium
(SFHe) sheet for a direct measurement of the gravitational interaction of
second-generation, leptonic antimatter [10, 11]. These experiments utilize
scintillators at cryogenic temperatures. However, the scintillation light is
typically transported to a room-temperature environment using plastic fibres
before being detected by photodetectors. The application of cryogenic SiPMs in
the cryogenic environment could reduce the complexity of such experiments and
allow for finer detector geometries [12]. Furthermore, single photon detection
capabilities at ultra-low temperatures is of increasing interest in a variety
of applications in quantum information science, where SiPMs have been mainly
used at room temperature [13].

In this work we show that SiPMs can be operated to detect single photons
inside a dilution cryostat with limited cooling power at temperatures below
$1\text{\,}\mathrm{K}$, a behavior which was not expected due to the
overwhelming afterpulsing measured below the advertised lowest operational
ranges ($\sim$$70\text{\,}\mathrm{K}$). The temperature dependence of the
signal shape, breakdown voltage, and photon detection efficiency are presented
as well as the operating range of the SiPM for single photon detection.
Finally, we show how the SiPM can be used in a scintillation detector system
with multiple channels, capable of operating in a dilution cryostat with the
coldest detector operating at $170\text{\,}\mathrm{mK}$.

##  2 Experimental setup

The device studied in this work was the S13370 (VUV4) SiPM from Hamamatsu,
which was originally developed for the detection of VUV photons from liquid
xenon and argon targets. Specific features are the absence of the protection
layer to improve VUV sensitivity and the utilization of metallic quenching
resistors making the resistance less temperature dependent. To study the
performance of SiPMs at ultra-low temperatures a custom-made dilution cryostat
was used [14]. A sketch of the experimental setup is shown in figure 1. A SiPM
was thermalized in an oxygen-free high-thermal-conductivity copper support in
contact with the cold plate (position A), which is protected from thermal
radiation by three layers of aluminium heat shields. Two temperature sensors
(LakeShore CX-1010) were placed on the copper support close to the SiPM to
monitor the temperature. Additional detectors situated at higher temperatures
could be placed on acrylic supports on the first heat shield (position B),
where the temperature was also monitored using the same type of sensor.

Figure 1:  Schematic view of the experimental setup and the data acquisition
system. For the temperature dependent characterization, a SiPM was mounted
within a copper structure (position A) in contact with the cold plate. Up to
16 additional cryogenic detectors can be positioned on the first heat shield
(position B). The signal from the SiPM was then either amplified and read out
using an oscilloscope (a), or directly fed to the WaveDREAM board (b).

Due to the limited cooling power of the cryostat, the heat load from the
signal cables needed to be minimized. We installed 18 channels of
$\sim$$5\text{\,}\mathrm{m}$ long micro coaxial cable (50 $\Omega$, 38 AWG
tinned copper alloy, $0.4\text{\,}\mathrm{mm}$ outer sleeve diameter), which
we chose as a compromise between heat load and signal quality. Each channel
was thermalized at three temperature stages in the cryostat and terminated at
$2.4\text{\,}\mathrm{K}$ in a row of MMCX connectors. From there, 16 channels
were connected with the same wires to the SiPMs at position B, while a semi-
rigid coaxial cable (50 $\Omega$ impedance) led to the SiPM at position A.

The SiPM characterization at cryogenic temperatures was done using a sensor
with 3 $\times$ 3 $\mathrm{m}\mathrm{m}^{2}$ active area and
$75\text{\,}\mathrm{\SIUnitSymbolMicro m}$ pixel pitch, placed at position A.
The sensor faced the polished end of a wavelength shifting fibre (Kuraray
Y-11) with peak emission wavelength at $\lambda$ = 476 nm, which at the other
end was coupled to a LED (Thorlabs TO-18, 450 nm, 7nW) placed in the vacuum
chamber of the cryostat at room temperature. A signal generator (Tektronix AFG
3251) was used to bias the LED with a square pulse ($100\text{\,}\mathrm{kHz}$
rate, $10\text{\,}\mathrm{ns}$ width, $-2.7\text{\,}\mathrm{V}$ amplitude) and
to simultaneously trigger the data acquisition system.

We used two different setups for the measurement. In setup (a), a source
measurement unit (Keithley 2450 SMU) was used to provide the bias voltage and
to measure the resulting current. The signal from the SiPM was amplified and
then read out by a 20 GS/s oscilloscope (Tektronix MSO6B). Amplification is
done using a MAR-amplifier, developed at PSI for fast photo-diode
applications, with a voltage gain of 40 dB (37 dB) at 100 kHz (750 MHz) and
input impedance of 50 $\Omega$. In setup (b), the raw signal of the SiPM was
read out using the WaveDREAM board, based on DRS4 digitizers developed at PSI,
which provided the bias voltage, amplified the input signal, and digitized the
waveforms [15].

##  3 SiPM characteristics at cryogenic temperatures

###  3.1 Pulse shape

In SiPMs, the pulse shape changes with temperature due to the temperature-
dependent quenching resistance of the cell, which determines the pulse
recovery time. Multiple signals under low-light conditions at various
temperatures were recorded with the oscilloscope. In figure 2, the average
over 1000 waveforms is shown. It is visible that the fast rising edge of the
pulse is not affected by the temperature change, where as the slower falling
edge becomes faster at lower temperatures because the resistance of the
metallic quenching resistor decreases with decreasing temperature. In the
range from $80\text{\,}\mathrm{K}$ to $300\text{\,}\mathrm{K}$ the metallic
quenching resistor of this SiPM has a good stability against temperature
variations resulting in a stable pulse shape. Below $7\text{\,}\mathrm{K}$,
the pulse recovery time decreases by a factor of two but the overall pulse
shape is stable.

Figure 2:  Averaged waveform at various temperatures with a constant
overvoltage $\Delta V$ = 1.75 V. The fast rising edge is temperature-
independent, while the slower falling edge becomes faster with increasing
temperature. However, the overall pulse shape is stable over the whole
temperature range. The overshoot is due to an impedance mismatch of the used
micro coax cable at low temperatures.

###  3.2 IV characteristics and breakdown voltage

The reverse bias current-voltage (IV) curves of the SiPM were measured at
temperatures ranging from $0.85\text{\,}\mathrm{K}$ to
$310\text{\,}\mathrm{K}$. At very low temperatures, both the leakage and
avalanche current of the SiPM was in the range of the minimum instrument
sensitivity. For this reason, the measurement of the IV curve was conducted
under low-light condition, which allowed a reliable extraction of the
breakdown voltage $V_{\mathrm{bd}}$. In figure 3, left, a typical IV curve
recorded at $T=$ $60\text{\,}\mathrm{K}$ is shown. The shape of the curve is
similar to one at room temperature: up to $V_{\mathrm{bd}}$ the reverse
current increases gradually due to the dark current followed by a sudden
increase at $V_{\mathrm{bd}}$ [16]. The breakdown voltage is extracted from
the reverse IV curve as the voltage where
$\frac{\mathrm{d}}{\mathrm{d}V}\mathrm{ln}I$ is maximal, corresponding to the
inflection point of the IV curve as described in [17]. The maximum of
$\frac{\mathrm{d}}{\mathrm{d}V}\mathrm{ln}I$ was obtained by fitting a Landau
distribution, which is used due to the skewed shape of the curve.

The breakdown voltage measured at various temperatures is shown in figure 3,
right. Between $150\text{\,}\mathrm{K}$ and $300\text{\,}\mathrm{K}$, a linear
decrease with temperature is visible, as reported in [18, 7]. The dashed red
line matches the linear temperature coefficient of 54 mV/K around room
temperature specified by Hamamatsu [19]. A departure from the linear
dependence is visible at lower temperatures, which can be explained by the
temperature-dependent avalanche formation of charge carriers in silicon [20].

Figure 3:  Left: Breakdown voltage determination from the IV curve. The top
shows the IV curve measured at $60\text{\,}\mathrm{K}$, where two distinct
regions corresponding to the pre (red) and post (green) breakdown are visible.
A Landau peak is fitted on $\frac{\mathrm{d}}{\mathrm{d}V}\mathrm{ln}I$ to
obtain the inflection point of the IV curve, which corresponds to the
breakdown voltage. The discontinuity around 43 V is caused by the change of
range of the measurement device. It lies outside of the fit region and did not
influence the obtained result. Right: Breakdown voltage as a function of
temperature. At lower temperatures a deviation from the linear dependence
given by the datasheet is visible.

###  3.3 Operating range

The operating range of the SiPM for single photoelectron (p.e.) detection was
studied at various temperatures by monitoring the reverse current as a
function of the bias voltage $V_{\mathrm{bias}}$. At cryogenic temperatures,
dark current is limited, but delayed afterpulses are present due to the
temperature-dependent trapping and release of charge carriers [21]. With
higher $V_{\mathrm{bias}}$ the probability for afterpulsing increases [5],
which can mask the single photon signal. Further, this increases the current
drawn by the SiPM resulting in heat dissipation. We defined an upper limit of
a useful operational range by a maximum dark current $I_{\rm
max}=$1\text{\,}\mathrm{\SIUnitSymbolMicro A}$$ and the corresponding bias
voltage $V_{I>$1\text{\,}\mathrm{\SIUnitSymbolMicro A}$}$, at which the heat
dissipation was tolerable in our dilution cryostat. We defined the lower limit
of the SiPM operating range by the breakdown voltage. Figure 4 shows this
above defined operating range for different temperatures between
$0.8\text{\,}\mathrm{K}$ and $100\text{\,}\mathrm{K}$ as the unshaded region.
This operating range becomes narrow when the temperature approaches
$40\text{\,}\mathrm{K}$ due the increasing afterpulsing probability. In the
temperature range between $20\text{\,}\mathrm{K}$ and $40\text{\,}\mathrm{K}$,
the increased afterpulsing probability caused self-sustaining afterpulsing
processes, which narrowed the operational range to below 0.5 V. Below
$15\text{\,}\mathrm{K}$ however, $V_{I>$1\text{\,}\mathrm{\SIUnitSymbolMicro
A}$}$ was observed to increase again. An explanation to this may be the
increasing trapping time of the charge carriers with the decreasing
temperature, a similar phenomenon already reported [22] at slightly higher
($5\text{\,}\mathrm{K}$) temperatures. With the resulting large delays in
afterpulsing, the useful operational range thus allowed to increase again, and
the operation of the SiPM becomes possible with limited a overvoltage $\Delta
V=V_{\mathrm{bias}}-V_{\mathrm{bd}}\sim$ $2\text{\,}\mathrm{V}$ for $T<15$ K.

Figure 4:  The useful SiPM operating range for single p.e. detection. The
unshaded area represents the region where the SiPM can be operated in Geiger-
mode within our dilution cryostat. The lower limit of $V_{\mathrm{bias}}$ is
given by $V_{\mathrm{bd}}$ (blue), while an upper limit (orange) is defined by
the voltage $V_{I>$1\text{\,}\mathrm{\SIUnitSymbolMicro A}$}$ where the
maximum tolerable dark current $I_{\max}>$1\text{\,}\mathrm{\SIUnitSymbolMicro
A}$$ was measured. Between $20\text{\,}\mathrm{K}$ and $40\text{\,}\mathrm{K}$
the SiPM cannot be operated reliably, while at lower temperatures operation
with limited $V_{\mathrm{bias}}$ becomes possible again.

###  3.4 Photon detection efficiency

Single photon detection efficiency (PDE) at cryogenic temperatures at various
$\Delta V$ was studied by illuminating the SiPM with light from the pulsed
LED, resulting in a few photons reaching the sensitive area simultaneously.
For each light pulse, the corresponding signal waveform was acquired and
evaluated using the WaveDREAM board. The baseline was calculated from the
first $60\text{\,}\mathrm{ns}$ of the waveform, and then subtracted from the
subsequent raw waveform data. The charge spectrum was obtained by calculating
the integral of the waveform within a time window of $20\text{\,}\mathrm{ns}$.
The first peak of the charge spectrum corresponds to the pedestal, where no
photons are detected, and the other peaks from 1 p.e., 2 p.e., etc. signals.
The peaks were fit with a Gaussian function to obtain the center of each peak.
The distribution of the integrated entries under each peak was then fit by a
Poisson function to obtain the average number of detected p.e.. Figure 5,
left, shows the charge spectrum obtained a $0.85\text{\,}\mathrm{K}$ with
$\Delta V$ = 1.75 V and the Poisson fit used to obtain the average number of
detected photoelectrons.

Figure 5:  Left: The top shows the SiPM charge spectrum measured at
$0.85\text{\,}\mathrm{K}$ and $\Delta V=$1.75\text{\,}\mathrm{V}$$, and the
Gaussian fits on each p.e. peak. The bottom plot shows the entries under each
peak, which is fitted by a Poisson function to obtain an average of 2.70 $\pm$
0.02 p.e.. Right: Relative PDE compared to room temperature for various
overvoltages. At $0.85\text{\,}\mathrm{K}$ the PDE is 0.666 $\pm$ 0.005 of the
PDE at room temperature.

The absolute PDE could not be determined since the absolute number of photons
reaching the SiPM per LED pulse is not known. The relative PDE compared to
room temperature was obtained by comparing the average number of detected p.e.
from the same source at cryogenic temperature to the one at room temperature.
Figure 5, right, shows the relative PDE at various temperatures for different
$\Delta V$. Variations of the PDE are visible at temperatures above
$80\text{\,}\mathrm{K}$, while below $80\text{\,}\mathrm{K}$ the PDE is
dropping. The PDE is given by the product of three factors: the geometric fill
factor (ratio of the sensitive area to the SiPM area), the photo-conversion
efficiency, and the avalanche triggering probability. As in [23], two
counteracting mechanisms are expected to change the PDE with decreasing
temperature. On one hand, the avalanche triggering probability is increasing
with decreasing temperature due to the enhanced mobility of charge carriers,
on the other hand, the energy gap is wider at lower temperatures, which
decreases the photo-conversion efficiency. These two effects could explain the
PDE variations above $80\text{\,}\mathrm{K}$. The enhancement of the PDE at
low $T$ is also observed [24] for single avalanche photo diodes (APDs).
Furthermore, at low temperatures charge carriers freeze out, resulting in
carrier losses, which could explain the drop of PDE below
$80\text{\,}\mathrm{K}$. The PDE variation with temperature is also happening
at other wavelengths as reported in [25]. At very low temperatures below
$1\text{\,}\mathrm{K}$, the PDE is a factor of 0.666 $\pm$ 0.005 smaller
compared to room temperature, where the PDE is specified by the manufacturer
to be at 40 % for $500\text{\,}\mathrm{nm}$ light and $\Delta V$ =
$4\text{\,}\mathrm{V}$ [19]. Although, the SiPM at ultra low temperatures
cannot be operated at such high gain conditions, single photon detection is
nevertheless feasible at temperatures below $1\text{\,}\mathrm{K}$.

##  4 Commissioning of cryogenic scintillation detectors

In the previous section we have shown that the Hamamatsu S13370 SiPM can be
operated at temperatures below $1\text{\,}\mathrm{K}$. To observe the
formation of a M beam from a SFHe target, charged particle detection with
finely segmented detectors operating at these ultra low temperatures is
required. We developed cryogenic scintillation detectors based on the SiPMs
characterized in Section 3. The SiPMs were directly coupled to plastic
scintillator bars (Eljen EJ-204) which were wrapped with PTFE to improve light
collection and reduce optical crosstalk between the detector channels. Since
this type of SiPM is not covered with a protective layer, special care was
required not to damage the wire bonds and silicon substrate. Coupling using
optical cement damaged multiple detectors during the cooling-warming cycles
such that we decided to use 3D-printed acrylic sleeves to fix the
scintillators to the SiPMs.

In total, we constructed 16 detectors consisting of plastic scintillator bars
of the sizes 1.5 $\times$ 3 $\times$ 20 $\mathrm{m}\mathrm{m}^{3}$, 2 $\times$
3 $\times$ 20 $\mathrm{m}\mathrm{m}^{3}$, and 4 $\times$ 3 $\times$ 20
$\mathrm{m}\mathrm{m}^{3}$, coupled to SiPMs with 3 $\times$ 3
$\mathrm{m}\mathrm{m}^{2}$ active area. The 16 detectors were placed on the
first heat shield of the cryostat (position B in figure 1) at a temperature of
$1.7\text{\,}\mathrm{K}$. A scintillator pill of the size of 5 $\times$ 6
$\times$ 6 $\mathrm{m}\mathrm{m}^{3}$ was coupled to a SiPM with 6 $\times$ 6
$\mathrm{m}\mathrm{m}^{2}$ active area and was placed within the target
chamber at $170\text{\,}\mathrm{mK}$ (position A in figure 1).

The detectors were commissioned using the muon beam from the $\pi$E1 area of
the Paul Scherrer Institute. All detectors were capable of detecting positrons
resulting from muon decays at the given temperatures. We did not observe a
degradation in gain or change of breakdown voltage with multiple heat cycles
between room temperature and the base temperature. The detector operated
reliably and reproducibly over a data taking period of three weeks. Typical
energy spectra of one of the 16 detectors at room temperature and at
$1.7\text{\,}\mathrm{K}$ are shown in figure 6. The detected positrons have an
initial energy distributed below $50\text{\,}\mathrm{MeV}$ and deposit part of
their energy in a thin layer of scintillator material described by a Landau
distribution. Based on Monte Carlo simulations, the Landau peak is situated at
$0.5\text{\,}\mathrm{MeV}$ for muon-decay positrons traversing scintillator
material with geometries described above. At the lower temperature, the
measured Landau peak is situated at a lower ADC count number, indicating a
lower gain of the detector. Nevertheless, the full peak is visible, showing
the charged particle detection feasibility of the detector at ultra low
temperatures.

Figure 6:  Typical energy spectrum of positrons passing a thin plastic
scintillator at two operation temperatures (Left): $296\text{\,}\mathrm{K}$,
(Right): $1.7\text{\,}\mathrm{K}$. In both cases, the full Landau peak,
corresponding to the energy deposition of charged particles in a thin layer,
is visible.

##  5 Conclusion

We have shown that SiPMs of the S13370 series from Hamamatsu can be operated
in a dilution cryostat reaching temperatures below $200\text{\,}\mathrm{mK}$.
We measured the temperature dependence of various SiPM characteristics down to
$0.85\text{\,}\mathrm{K}$. The pulse shape has been shown to be stable with a
decrease of the pulse recovery time at temperatures below
$7\text{\,}\mathrm{K}$. The breakdown voltage was determined using the reverse
IV curves and shows a departure from the linear dependence with temperature
given by the data sheet. The operating range was determined, where we have
seen that in the temperature range from $20\text{\,}\mathrm{K}$ to
$40\text{\,}\mathrm{K}$ the SiPM cannot be operated because afterpulsing
effects cause the SiPM to draw too much current. We measured the temperature
dependence of the relative PDE and determined the PDE at
$0.85\text{\,}\mathrm{K}$ to be a factor of 0.666 $\pm$ 0.005 smaller than the
one at room temperature, demonstrating the single photon detection capability
at temperatures below $1\text{\,}\mathrm{K}$ with the studied SiPM. We
developed finely segmented scintillation detectors using plastic scintillators
and SiPMs, which we commissioned with a muon beam. In total, 16 channels of
detectors at $1.7\text{\,}\mathrm{K}$ and one channel at
$170\text{\,}\mathrm{mK}$ were operated reliably within a dilution cryostat
and were capable of detecting charged particles.

## Acknowledgments

The authors would like to thank the HIPA facility at PSI for the stable beam
and technical support. Special thanks to the PSI detector group, M.
Hildebrandt, A. Stoykov, and F. Barchetti and U. Greuter and U. Hartmann from
the electronics group. This work was supported by the SNF Ambizione Grant No.
PZ00P2 185975.

## References

  * [1] J. Aalbers, F. Agostini, M. Alfonsi, F. D. Amaro, C. Amsler, E. Aprile et al., _DARWIN: Towards the ultimate dark matter detector_ , _J. Cosmol. Astropart. Phys._ 2016 (2016) 017 [1606.07001].
  * [2] C. E. Aalseth, F. Acerbi, P. Agnes, I. F. M. Albuquerque, T. Alexander, A. Alici et al., _DarkSide-20k: A 20 tonne two-phase LAr TPC for direct dark matter detection at LNGS_ , _The European Physical Journal Plus_ 133 (2018) 131.
  * [3] J. B. Albert, G. Anton, I. J. Arnquist, I. Badhrees, P. Barbeau, D. Beck et al., _Sensitivity and discovery potential of the proposed nEXO experiment to neutrinoless double- $\beta$ decay_, _Phys. Rev. C_ 97 (2018) 065503 [1710.05075].
  * [4] S. Ogawa, _Liquid xenon calorimeter for MEG II experiment with VUV-sensitive MPPCs_ , _Nucl. Instruments Methods Phys. Res. Sect. A Accel. Spectrometers, Detect. Assoc. Equip._ 845 (2017) 528.
  * [5] G. Gallina, P. Giampa, F. Retière, J. Kroeger, G. Zhang, M. Ward et al., _Characterization of the Hamamatsu VUV4 MPPCs for nEXO_ , _Nucl. Instruments Methods Phys. Res. Sect. A Accel. Spectrometers, Detect. Assoc. Equip._ 940 (2019) 371 [1903.03663].
  * [6] A. Falcone, R. Bertoni, F. Boffelli, M. Bonesini, T. Cervi, A. Menegolli et al., _Vacuum ultra-violet and ultra-violet scintillation light detection by means of silicon photomultipliers at cryogenic temperature_ , _Nucl. Instruments Methods Phys. Res. Sect. A Accel. Spectrometers, Detect. Assoc. Equip._ 787 (2015) 216.
  * [7] R. Iwai, M. Sakurai, A. Antognini, I. Belosevic, M. Hildebrandt, K. Kirch et al., _Characterization of Cryogenic SiPM Down to 6.5 K_ , in _Proceedings of the 5th International Workshop on New Photon-Detectors (PD18)_.
  * [8] A. Antognini and D. Taqqu, _Review of Particle Physics at PSI_ , _SciPost Phys. Proc_ 5 (2021) 30.
  * [9] A. Antognini, N. J. Ayres, I. Belosevic, V. Bondar, A. Eggenberger, M. Hildebrandt et al., _Demonstration of Muon-Beam Transverse Phase-Space Compression_ , _Phys. Rev. Lett._ 125 (2020) 164802.
  * [10] A. Soter and A. Knecht, _Development of a cold atomic muonium beam for next generation atomic physics and gravity experiments_ , _SciPost Phys. Proc_ 5 (2021) 31.
  * [11] K. Kirch and K. S. Khaw, _Testing antimatter gravity with muonium_ , _Int. J. Mod. Phys. Conf. Ser._ 30 (2014) 1460258 [1509.02918].
  * [12] A. Sótér, K. Todoroki, T. Kobayashi, D. Barna, D. Horváth and M. Hori, _Segmented scintillation detectors with silicon photomultiplier readout for measuring antiproton annihilations_ , _Review of Scientific Instruments_ 85 (2014) 023302.
  * [13] M. D. Eisaman, J. Fan, A. Migdall and S. V. Polyakov, _Invited Review Article: Single-photon sources and detectors_ , _Rev. Sci. Instrum._ 82 (2011) 071101.
  * [14] B. van den Brandt, J. A. Konter and S. Mango, _A compact versatile dilution refrigerator with top-loading capability_ , _Nucl. Instruments Methods Phys. Res. Sect. A Accel. Spectrometers, Detect. Assoc. Equip._ 289 (1990) 526.
  * [15] L. Galli et al., _WaveDAQ: An highly integrated trigger and data acquisition system_ , _Nucl. Instrum. Meth. A_ 936 (2019) 399.
  * [16] V. Chmill, E. Garutti, R. Klanner, M. Nitschke and J. Schwandt, _Study of the breakdown voltage of SiPMs_ , _Nucl. Instruments Methods Phys. Res. Sect. A Accel. Spectrometers, Detect. Assoc. Equip._ 845 (2017) 56 [1605.01692].
  * [17] A. N. Otte, D. Garcia, T. Nguyen and D. Purushotham, _Characterization of three high efficiency and blue sensitive silicon photomultipliers_ , _Nucl. Instruments Methods Phys. Res. Sect. A Accel. Spectrometers, Detect. Assoc. Equip._ 846 (2017) 106 [1606.05186].
  * [18] L. Baudis, M. Galloway, A. Kish, C. Marentini and J. Wulf, _Characterisation of Silicon Photomultipliers for Liquid Xenon Detectors_ , _J. Instrum._ 13 (2018)  [1808.06827].
  * [19] Hamamatsu, _Datasheet_ , .
  * [20] C. R. Crowell and S. M. Sze, _Temperature dependence of avalanche multiplication in semiconductors_ , _Appl. Phys. Lett._ 9 (1966) 242.
  * [21] B. K. Ridley, _Lucky-drift mechanism for impact ionisation in semiconductors_ , _J. Phys. C Solid State Phys._ 16 (1983) 3373.
  * [22] A. Cardini, V. Fanti and A. Lai, _Operation of multi-pixel photon counters down to liquid helium temperatures_ , _IEEE Trans. Nucl. Sci._ 61 (2014) 2672.
  * [23] G. Collazuol, M. G. Bisogni, S. Marcatili, C. Piemonte and A. Del Guerra, _Studies of silicon photomultipliers at cryogenic temperatures_ , _Nucl. Instruments Methods Phys. Res. Sect. A Accel. Spectrometers, Detect. Assoc. Equip._ 628 (2011) 389.
  * [24] I. Rech, I. Labanca, G. Armellini, A. Gulinatti, M. Ghioni and S. Cova, _Operation of silicon single photon avalanche diodes at cryogenic temperature_ , _Review of Scientific Instruments_ 78 (2007) 063105 [https://doi.org/10.1063/1.2743167].
  * [25] N. Anfimov, D. Fedoseev, A. Rybnikov, A. Selyunin, S. Sokolov and A. Sotnikov, _Study of silicon photomultiplier performance at different temperatures_ , _Nucl. Instruments Methods Phys. Res. Sect. A Accel. Spectrometers, Detect. Assoc. Equip._ 997 (2021) 165162.


Xin Jing1, Yi Chang3, Zijiang Yang1, Jiangjian Xie2,
Andreas Triantafyllopoulos1, Björn W. Schuller1,3

# U-DiT TTS: U-Diffusion Vision Transformer for Text-to-Speech

###### Abstract

Deep learning has led to considerable advances in text-to-speech synthesis.
Most recently, the adoption of Score-based Generative Models (SGMs), also
known as Diffusion Probabilistic Models (DPMs), has gained traction due to
their ability to produce high-quality synthesized neural speech in neural
speech synthesis systems. In SGMs, the U-Net architecture and its variants
have long dominated as the backbone since its first successful adoption. In
this research, we mainly focus on the neural network in diffusion-model-based
Text-to-Speech (TTS) systems and propose the U-DiT architecture, exploring the
potential of vision transformer architecture as the core component of the
diffusion models in a TTS system. The modular design of the U-DiT
architecture, inherited from the best parts of U-Net and ViT, allows for great
scalability and versatility across different data scales. The proposed U-DiT
TTS system is a mel spectrogram-based acoustic model and utilizes a pretrained
HiFi-GAN as the vocoder. The objective (i. e., Frechet distance) and MOS
results show that our DiT-TTS system achieves state-of-art performance on the
single speaker dataset LJSpeech. Our demos are publicly available at:
https://eihw.github.io/u-dit-tts/

##  1 Introduction

Text-to-speech (TTS) or speech synthesis systems are computer-based systems
designed to convert text into spoken language [1], which have been widely used
in human–computer interaction (HCI), most recently, for example, to power AI
voice assistants. In recent years, dramatic breakthroughs in deep learning
technology have resulted in the synthesis of more natural-sounding synthetic
speech [1, 2, 3, 4, 5, 6].

Modern Deep Neural Network (DNN)-based TTS systems typically consist of two
main components: an acoustic model and a vocoder [1, 5] – thus incorporating
the two initial steps of a standard pipeline into one single model. The
acoustic model converts the input text into time-frequency domain acoustic
features, then, the vocoder synthesizes the raw waveforms conditioned on these
acoustic features. Since the introduction of Tacotron 2 [1], the mel
spectrogram has been the dominant acoustic feature in most modern TTS systems
[3, 2, 4].

Instead of directly generating mel spectrograms frame by frame in a sequence-
to-sequence (seq2seq) architecture [7], generative models aim to learn the
latent distribution of speech signals and sample from it to synthesize new
ones. Variational Autoencoders (VAEs) [7], Generative Adversarial Networks
(GANs) [8], and Flow-based generative models [3, 9] have produced more
natural, coherent and faster speech synthesis utilizing their ability to
capture the underlying relation between input text and speech.

Recently, Score-based Generative Models (SGMs) [10, 11, 12, 13], also known as
Diffusion Probabilistic Models (DPMs) have achieved state-of-the-art
performance in many research fields, such as image generation [12, 13], 3D
shape creation [14], natural speech synthesis [4, 15], and music fragment
generation [16]. SGMs are based on the simple but effective idea that a
complex data distribution can be gradually turned into a simple one (usually a
normal distribution $N\sim(\sigma,I)$) by iteratively adding scheduled noise.
Meanwhile, a neural network can be trained to invert this procedure by
following the trajectories of the reverse time forward process [17]. In these
SGMs, a modified U-Net architecture [12] has proven to be well-suited as this
diffusion model. Its ability to capture both local and global features with
additional spatial self-attention blocks has lead to its widespread adoption
in research fields. As a further improvement of the U-Net architecture,
Diffusion Visual Transformer (DiT) [18] introduced the Visual Transformer
(ViT) as the backbone of the diffusion model and achieved state-of-the-art
image generation performance on the class-conditional ImageNet benchmarks.

In this work, we propose a U-DiT architecture, thus exploring the possibility
of ViT transforms as the core component in the backbone of a neutral
diffusion-based TTS system. By combining the properties of the U-Net and ViT,
both our objective results and MOS results show that U-DiT TTS generates
higher-quality speech and more natural prosody compared to the state-of-the-
art diffusion-based TTS system.

##  2 Score-based Generative Models

Score-based generative models (SGMs) is a unified framework proposed by Song
et al. [11] which adopts a stochastic differential equation (SDE) formulation
[19]. They are a class of generative diffusion models that have the ability to
learn the gradients of log probability density functions ($logPDF$ or score
functions) on a large number of noise-perturbed data distributions, and then
generate samples with an iterative procedure called Langevin dynamics [20],
which is a class of Markov Chain Monte Carlo (MCMC) algorithms that generate
samples from a probability distribution.

###  2.1 SGMs with stochastic differential equations

Instead of a finite number of noise distributions, the SGMs perturb the
training data with a continuum of distributions that evolve over time
according to a diffusion process, and the forward process $\\{x_{t}\\}_{t}$
can be described by an Itô SDE [21]:

$dx_{t}=f(x_{t},t)dt+g(t)dw,$ (1)

where $f$ is $drift$ coefficient of $x(t)$, $g$ is the $diffusion$ coefficient
of $x(t)$, and $w$ is a standard Wiener process (or Brownian motion). Please
note that the $t$ herein refers to the process step, and it has no relation to
the time axis of the audio or its transformed version.

According to Anderson [22], every SDE in the form of Equation 1 should have a
corresponding reverse SDE, which is also a diffusion process:

$dx_{t}=\left[f(x_{t},t)-g(t)^{2}\nabla_{x_{t}}logp_{t}(x_{t})\right]dt+g(t)d\bar{w},$
(2)

where $dt$ is an infinitesimal negative timestep and $\bar{w}$ is a standard
Wiener process when $t$ reverses from $T$ to $0$.
$\nabla_{x_{t}}logp_{t}(x_{t})$ is the score function of a prior distribution
$p(x)$ which can be approximated by a learned time-dependent score-based model
$s_{\theta}$, such that
$s_{\theta}(x_{t},t)\approx\nabla_{x_{t}}logp_{t}(x_{t})$.

###  2.2 Diffusion Process in TTS systems

Forward Process: As mentioned above, a forward SDE diffusion process should be
defined first to corrupt the input data to the prior distribution. In this
work, we follow the forward diffusion process from the Grad-TTS [4], in which
the forward SDE is defined as:

$dX_{t}=\frac{1}{2}\Lambda^{-1}(\mu-
X_{t})\beta_{t}dt+\sqrt{\beta_{t}}dW_{t},\quad t\in[0,T],$ (3)

where $\beta_{t}$ is the non-negative-valued noise schedule function, $\mu$ is
vector, and $\Lambda$ is a diagonal matrix with positive elements.

Let $I$ be the identity matrix and the prior distribution can be defined as:

$p_{0t}\\{X_{t}|X_{0},\mu\\}=\mathcal{N}(X_{t};\rho(X_{0},\Lambda,\mu,t),\delta(\Lambda,t)^{2}I)$
(4)

It is an Ornstein-Uhlenbeck SDE [23] and its mean $\rho(*)$ and variance
$\delta(*)^{2}$ can be derived in a close form, according to the Øksendal
[24], as:

$\displaystyle\rho(X_{0},\Lambda,\mu,t)=(I$
$\displaystyle-e^{-\frac{1}{2}\Lambda^{-1}\int^{t}_{0}\beta_{s}ds})\mu$ (5)
$\displaystyle+e^{-\frac{1}{2}\Lambda^{-1}\int^{t}_{0}\beta_{s}ds}X_{0}$

$\delta(\Lambda,t)I=\Lambda\left(I-e^{\Lambda^{-1}\int^{t}_{0}\beta_{s}ds}\right)$
(6)

Reverse Process: From Equation 2, we can get the reverse SDE for the reverse
diffusion process. However, to simplify and speed up the reverse procedure,
Song [11] provides a probability flow ordinary differential equation (ODE),
which leads to the following reverse equation:

$dX_{t}=\left(\frac{1}{2}\Lambda^{-1}(\mu-X_{t})-\nabla
logp_{t}(X_{t})\right)\beta_{t}dt$ (7)

Figure 1: Overview of the U-DiT architecture. The downsampling part will
transform the input into the latent space and the process step $t$, which is
randomly sampled between $[0,1]$ and encoded by the sinusoidal position
embedding, will be applied as extra conditional information of the DiT blocks.
At last, the upsampling part will convert the embeddings to the original size

##  3 System Overview

In this work, we propose the U-DiT architecture, which contains several
diffusion visual transformer blocks, one of the ViT variants designed for
diffusion process. This new architecture is designed to effectively capture
both high-level semantic features and fine-grained details of mel
spectrograms, leading to improved performance on speech synthesis tasks.

###  3.1 DiT blocks

The Figure 1 (right) demonstrates the detailed design of the proposed DiT
block. The DiT block has retained the best practice of the ViT block design,
while some minor but important modifications have been made to the standard
ViT block.

Patchify: DiT will convert the input into a sequence of tokens like ViT. Then,
the standard frequency-based positional embedding will be applied to all
tokens. We set the patch size to $[20,64]$ in our model.

Layer Normalization: Normalizing a set of data points transforms them to a
similar scale, which can improve generalization. In particular, adaptive layer
normalization [25] (adaLN) enables smoother gradients and better
generalization accuracy by normalizing the distributions of intermediate
layers. It has been widely utilized by GAN architectures [26] and in the
standard diffusion U-Net backbone [27]. Additionally, the diffusion U-Net
backbone also employs a zero-initialization strategy for its final
convolutional layers before residual connections, which has also been found to
accelerate large-scale training on different tasks. The combination of the two
techniques, adaLN-Zero, has proven to be a powerful design for further
improving DiT performance and generalization on image synthesis [18].

Embedding MLP: Since the diffusion process is very strongly time- and label-
dependent (if a label exists), a 4-layer multilayer perceptron (MLP) has been
applied to regress the shift parameter $\gamma$ and $\beta$ from the sum of
the time embedding vector $t_{e}$ and label embedding vector $l_{e}$.
Furthermore, the dimension-wise scaling parameters $\alpha$, which are applied
prior to any residual connections within the DiT block, are also regressed by
the same MLP block.

###  3.2 Latent space for U-

In our experiments, we initially apply the DiT architecture as the backbone of
the diffusion models in the high-resolution pixel space. However, this lead to
a generated speech plagued by noise and lacking coherence. To address this
issue, we first utilized the downsampling and upsampling components of the
U-Net architecture from the DDPM [12] to transform the input spectrograms into
a latent space representation, which then served as the input of the DiT
blocks. As shown in Figure 1 (left), each layer in the downsampling part
consists of multiple residual blocks with group normalization [28] to extract
the low-level features from the input, which are then processed by a following
self-attention layer. Subsequently, the latent space features are partitioned
into small patches and the spatial information of these patches is encoded by
the sinusoidal position embedding. Finally, the upsampling part, which has a
symmetrical structure to the downsampling part, restores the latent space
features to their original size.

###  3.3 Model Architechure

We build our TTS system based on the structure from the Glow-TTS [3] and Grad-
TTS [4]. Therefore, the U-DiT TTS system comprises three interrelated
components: a text encoder, a duration predictor, and a decoder.

Text Encoder: The text encoder is composed of a pre-net, 6 transformer blocks
with multi-head self-attention, and a final linear projection layer. The pre-
net consists of 3 convolutional layers followed by a fully connected layer.
This architecture is designed to effectively capture the meaningful features
of the input text that can be used by the rest of the TTS system.

Duration predictor: In this work, we follow the Glow-TTS [3] to apply an
iterative approach called Monotonic Alignment Search (MAS) function to find an
optimal monotonic surjective alignment between text and the corresponding
speech. We also apply the duration predictor network from Fastspeech 2 [2],
which is responsible for predicting the duration of each element of the input
text.

Decoder: As shown in Figure 1, we employ the 2-layer U-Net downsampling and
upsampling components in the decoder. We also test with different numbers of
DiT blocks, including 2, 4, and 8. We observed that larger model sizes
resulted in worse performance, potentially due to the limited amount of
available data. Therefore, we settle on using 2 DiT blocks in our system. For
the vocoder, we employ the HiFi-GAN [29] to generate the final speech from the
reconstructed mel spectrogram.

##  4 Training Objectives

Training objectives are crucial to guide the model's training towards the
desired outcome for a given input. According to our model architecture, there
are 3 different loss functions to jointly optimise our TTS system performance.

Encoder loss: in this study, we presume that the noise is generated from a
Gaussian distribution with a diagonal covariance, which has proven to be
effective in several TTS systems [3, 4, 30] by simplifying several
mathematical derivations. Therefore, we can set the text encoder output as an
independent normal distribution
$\bm{\tilde{\mu}}_{A_{(i)}}\sim\mathcal{N}(\bm{\tilde{\mu}}_{A_{(i)}},I)$,
which leads to a negative log-likelihood encoder loss:

$\displaystyle\mathcal{L}_{enc}=$
$\displaystyle-\sum^{N}_{i}log\varphi(y_{i};\bm{\tilde{\mu}}_{A_{(i)}},I),$
(8)

where $\bm{\tilde{\mu}}$ is the text encoder output, and $\varphi(*,\mu,I)$ is
the $PDF$ of normal distribution $\mathcal{N}(\mu,I)$. To further improve
effectiveness, we set the $\mathcal{L}_{enc}$ in the logarithmic domain, which
makes the $\mathcal{L}_{enc}$ be the mean square error (MSE) loss.

Duration loss: we also train the Duration Predictor loss [3] $DP$ with the MSE
loss in the logarithmic domain:

$\displaystyle d_{i}=\,log\sum^{N_{freq}}_{j=1}1_{A^{*}(j)=i},\qquad
i=1,..,N_{freq}$ (9)
$\displaystyle\mathcal{L}_{DP}=MSE(DP(sg[\bm{\tilde{\mu}}],d),$

where $sg[*]$ is the stop gradient operator used to prevent $\mathcal{L}_{DP}$
affecting the text encoder parameters. With the stop gradient operation, it
seems that the text encoder with duration predictor could be trained
independently without a decoder. However, our results show that this only
leads to an ineffective training process with a failure in alignment.

Diffusion loss: as mentioned in subsection 2.2, our aim is to train a score
model $s_{\theta}(x_{t},t)$ to estimate gradients of log-density of the noisy
data $X_{t}$.

$\mathcal{L}_{t}(X_{0})=\mathbb{E}_{\epsilon_{t}}\left[\left\|s_{\theta}(X_{t},t)+\lambda(\Lambda,t)^{-1}\epsilon_{t}\right\|_{2}^{2}\right],$
(10)

where $X_{0}$ is the target spectrogram sampled from the training data at time
$t\in[0,T]$, which is sampled from a uniform distribution
$\xi_{t}\sim\mathcal{N}(0,I)$ and according to Equation 4, the noisy data
$X_{t}$ can be sampled from:

$X_{t}=\rho(X_{0},I,\mu,t)+\sqrt{\lambda_{t}}\xi_{t}.$ (11)

In Equation 8, we already set the $\Lambda=I$, leading to a simplified noise
distribution. The covariance matrix of the noise distribution is an identity
matrix $I$ multiplied by a scalar:

$\lambda_{t}=1-e^{\int^{t}_{0}\beta_{s}ds}.$ (12)

Finally, we have our diffusion loss function as follows:

$\mathcal{L}_{diff}=\mathbb{E}_{X_{0},t}\left[\lambda_{t}\mathbb{E}_{\xi_{t}}\left[\left\|s_{\theta}(X_{t},\mu,t)+\frac{\xi_{t}}{\sqrt{\lambda_{t}}}\right\|^{2}_{2}\right]\right].$
(13)

##  5 Experiments

###  5.1 Dataset

In this work, we used the standard LJSpeech dataset [31] with the official
training and test set. The dataset consists of 13,100 audio files with a total
duration of approximately 24 hours and is accompanied by transcriptions of the
audio in the form of plain text files. The audio is encoded in the WAV format
at a sampling rate of 22.05 kHz.

###  5.2 Experiment settings

For audio feature generation, we used 80-dimensional mel spectrograms with a
1024 window length and a 256 hop length. The input text was converted to
phonemes through CMU pronouncing dictionary. The output of the text encoder
was concatenated with the mel spectrograms. Then, we randomly took 256 frame
lengths on the time axis of speech representation as the final input.

For the diffusion process, we used the same noise schedule as the Grad-TTS [4]
$\beta_{t}=0.05+(20-0.05)t,\,t\in[0,1]$. During the training phase, we used
the Adam optimizer with a learning rate of $1e-4$ and the batch size is 32. We
also applied gradient clipping to prevent exploding gradients. All of our
models were trained and tested in a Python 3.9.12 and PyTorch 1.12.1
environment using an Nvidia RTX 3080 GPU.

The objective evaluation metrics we apply in this work are Frechet Distance
(FD), Log-spectral Distance (LSD), and Kullback–Leibler Divergence (KLD). The
FD [32] measures the similarity between generated speech samples and the
target speech samples. The log-spectral distance (LSD) [33], also referred to
as log-spectral distortion, is a distance measure between two spectra. The KLD
[34] measures dissimilarity between probability distributions. All the
objective metrics are from
AudioLDM111https://github.com/haoheliu/audioldm_eval [35] and all the speech
samples are resampled to 16KHz as the evaluation system requested.

To further evaluate the performance of the proposed TTS system, Mean Opinion
Score (MOS) tests were conducted. In the MOS study, $14$ samples were selected
from each of the $4$ systems listed in Table 1 and were annotated by $16$
individuals. Specifically, for each source text, the subjects were asked to
rate the overall quality (e. g., naturalness, intelligibility) of the $5$
corresponding audios shuffled in a five-point Likert score ($5$: Excellent,
$4$: Good, $3$: Fair, $2$: Poor, $1$: Bad). The subset of the samples used in
the MOS study is available on the demo page.

###  5.3 Inference

During inference, we segmented the input text to address the input size
restrictions of the transformer architecture. We adjusted the text phonemes
segmentation length between 22 and 25 since we find the output length of the
text encoder was slightly less than 256 when the total phonemes count is in
this range. Meanwhile, instead of sampling $x_{T}$ from the $N(\mu,I)$, we
applied a hyperparameter $\tau$ and started to sample from the distribution
$N(\mu,I/\tau)$. In our following experiments, we opted for the
$reverse\\_step=80$ and temperature $\tau=1.5$.

###  5.4 Results

We compared the performance of our models on the objective evaluation with 1)
Grad-TTS [4] with the official implementation as our baseline system, 2) GTmel
(mel + HiFi-GAN), where we first converted the ground truth speech into mel
spectrogram and then used HiFi-GAN to convert it back.

Table 1: Performance comparison on LJSpeech test set Metric | FD$\downarrow$ | LSD$\downarrow$ | KLD$\downarrow$ | MOS
---|---|---|---|---
GT | - | - | - | 4.10
GTmel | 0.6985 | 1.8202 | 0.0109 | 3.90
Grad-TTS [4] | 3.0046 | 2.1765 | 0.0647 | 3.62
U-DiT TTS | 0.8960 | 2.1745 | 0.0316 | 3.91

Table 1 shows that the proposed U-DiT TTS systems outperform the baseline
Grad-TTS system. The FD metric, which indicates speech quality, is lower than
Grad-TTS by a large margin while close to the GTmel, which is also the target
in the training phase. KLD and LSD also show that our systems have a better
similarity to the ground truth on mel spectrogram. All the results on the
objective evaluation indicate that the U-DiT is better capable of capturing
the spatial relationships between different parts of the input data when
compared to the U-Net. The MOS scores also demonstrate the U-DiT TTS
outperforms the Grad-TTS by 0.29, which means that it can synthesize more
natural speech. Furthermore, the MOS score of U-DiT TTS indicates a
competitive synthesis quality compared to the GTmel but is still marginal to
the GT audio. We observe that the pre-trained HiFi-GAN model is unable to
fully restore the original audio and U-DiT TTS produces the mel spectrogram
with less background noise and system noise. In the subjective MOS study,
participants found the synthesized audio to be slightly clearer and easier to
understand, resulting in a higher MOS score.

###  5.5 Ablation study

To find the optimal settings for the best synthesis quality, we performed a
parameter ablation on the reverse step and temperature in the reverse process.
We used FD and KLD as the primary evaluation metrics in the ablation study
since they share the same embeddings extracted by CNN14 from PANNs [36].

The reverse steps of the SGMs can have a great impact on the final generation
quality [12]. In Figure 2 left, we show the FD and KLD metrics as a function
of the reverse steps. Our results show that a good tradeoff between high
quality and efficiency be achieved with 80 to 150 steps. Our experiments also
show that a reverse step of 30 achieves acceptable speech quality while
maintaining a shorter synthesis time.

While the larger step size $1/N$ would negatively impact the quality of the
output mel spectrograms, adjusting the temperature $\tau$ can help mitigate
this effect and improve the generation quality without affecting the inference
time. With the reverse step fixed to 80, we explore the effect of temperature
$\tau$ on our system. According to Figure 2 right, our experiments demonstrate
that the optimal range for $tau$ is between 1.1 and 2. Within this range, we
observed a consistent improvement in the quality of the generated speech, as
indicated by the continuous decrease in FD and a marginal growth on KLD.

Figure 2: Synthesis Performance comparison on different parameters. Synthesis
quality in FD and LSD metric as a function of the reverse steps and
temperature $\tau$ in the reverse SDE

##  6 Conclusion

In this work, we proposed U-DiT architecture exploring the possibility of ViT
transforms as the core component in the backbone of the neutral diffusion-
based TTS system. The results show that the U-DiT TTS system can effectively
generate higher-quality natural speech compared to the state-of-the-art
diffusion-based TTS system. However, there are still some limitations, such as
the fixed input size, and the high requirement for the quality of training
data. In future work, we plan to address the limitations and further optimize
the system to achieve better performance.

Overall, the U-DiT architecture provides a promising solution for the TTS
task, with the potential to significantly advance the state of the art in this
field.

##  7 Acknowledgement

This work was funded by the China Scholarship Council (CSC), Grant #
202006290013, and by the DFG (German Research Foundation), Reinhart Koselleck-
Project AUDI0NOMOUS (Grant No. 442218748).

## References

  * [1] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerrv-Ryan _et al._ , ``Natural tts synthesis by conditioning wavenet on mel spectrogram predictions,'' in _Proceedings of International conference on acoustics, speech and signal processing_ , Calgary, Canada, 2018, pp. 4779–4783.
  * [2] Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu, ``Fastspeech 2: Fast and high-quality end-to-end text to speech,'' in _Proceedings of International Conference on Learning Representations_ , virtual, 2021.
  * [3] J. Kim, S. Kim, J. Kong, and S. Yoon, ``Glow-tts: A generative flow for text-to-speech via monotonic alignment search,'' in _Proceedings of Advances in Neural Information Processing Systems_ , vol. 33, virtual, 2020, pp. 8067–8077.
  * [4] V. Popov, I. Vovk, V. Gogoryan, T. Sadekova, and M. Kudinov, ``Grad-tts: A diffusion probabilistic model for text-to-speech,'' in _Proceedings of International Conference on Machine Learning_ , virtual, 2021, pp. 8599–8608.
  * [5] A. Triantafyllopoulos, B. W. Schuller, G. İymen, M. Sezgin, X. He, Z. Yang, P. Tzirakis, S. Liu, S. Mertes, E. André _et al._ , ``An overview of affective speech synthesis and conversion in the deep learning era,'' _Proceedings of the IEEE (to appear; arXiv preprint arXiv:2210.03538)_ , 2022\.
  * [6] C. Wang, S. Chen, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li _et al._ , ``Neural codec language models are zero-shot text to speech synthesizers,'' _arXiv preprint arXiv:2301.02111_ , 2023.
  * [7] Y. Yasuda, X. Wang, and J. Yamagishd, ``End-to-end text-to-speech using latent duration based on vq-vae,'' in _Proceedings of International Conference on Acoustics, Speech and Signal Processing_ , virtual, 2021, pp. 5694–5698.
  * [8] M. Bińkowski, J. Donahue, S. Dieleman, A. Clark, E. Elsen, N. Casagrande, L. C. Cobo, and K. Simonyan, ``High fidelity speech synthesis with adversarial networks,'' in _Proceedings of International Conference on Learning Representations_ , Addis Ababa, Ethiopia, 2020.
  * [9] R. Valle, K. J. Shih, R. Prenger, and B. Catanzaro, ``Flowtron: an autoregressive flow-based generative network for text-to-speech synthesis,'' in _Proceedings of International Conference on Learning Representations_ , virtual, 2021.
  * [10] Y. Song and S. Ermon, ``Generative modeling by estimating gradients of the data distribution,'' in _Proceedings of Conference on Neural Information Processing Systems_ , Vancouver, Canada, 2019, p. 13 pages.
  * [11] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, ``Score-based generative modeling through stochastic differential equations,'' in _Proceedings of The International Conference on Learning Representations_ , Vienna, Austria, 2021.
  * [12] J. Ho, A. Jain, and P. Abbeel, ``Denoising diffusion probabilistic models,'' _Advances in Neural Information Processing Systems_ , vol. 33, pp. 6840–6851, 2020.
  * [13] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, ``High-resolution image synthesis with latent diffusion models,'' in _Proceedings of the Conference on Computer Vision and Pattern Recognition_ , Louisiana, USA, 2022, pp. 10 684–10 695.
  * [14] L. Zhou, Y. Du, and J. Wu, ``3d shape generation and completion through point-voxel diffusion,'' in _Proceedings of the International Conference on Computer Vision_ , virtual, 2021, pp. 5826–5835.
  * [15] Z. Chen, Y. Wu, Y. Leng, J. Chen, H. Liu, X. Tan, Y. Cui, K. Wang, L. He, S. Zhao _et al._ , ``Resgrad: Residual denoising diffusion probabilistic models for text to speech,'' _arXiv preprint arXiv:2212.14518_ , 2022.
  * [16] G. Mittal, J. Engel, C. Hawthorne, and I. Simon, ``Symbolic music generation with diffusion models,'' in _Proceedings of International Society for Music Information Retrieval Conference_ , virtual, 2021.
  * [17] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, ``Deep unsupervised learning using nonequilibrium thermodynamics,'' in _Proceedings of International Conference on Machine Learning_ , Lille, France, 2015, pp. 2256–2265.
  * [18] W. Peebles and S. Xie, ``Scalable diffusion models with transformers,'' _arXiv preprint arXiv:2212.09748_ , 2022.
  * [19] P. E. Kloeden, E. Platen, P. E. Kloeden, and E. Platen, _Stochastic differential equations_. Springer, 1992\.
  * [20] G. Parisi, ``Correlation functions and computer simulations,'' _Nuclear Physics B_ , vol. 180, no. 3, pp. 378–384, 1981.
  * [21] J. M. Steele, _Stochastic calculus and financial applications_. Springer, 2001, vol. 1.
  * [22] B. D. Anderson, ``Reverse-time diffusion equation models,'' _Stochastic Processes and their Applications_ , vol. 12, no. 3, pp. 313–326, 1982.
  * [23] R. A. Maller, G. Müller, and A. Szimayer, ``Ornstein–uhlenbeck processes and extensions,'' _Handbook of financial time series_ , pp. 421–437, 2009\.
  * [24] B. Øksendal, _Stochastic differential equations_. Springer, 2003.
  * [25] E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville, ``Film: Visual reasoning with a general conditioning layer,'' in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 32, no. 1, 2018.
  * [26] A. Brock, J. Donahue, and K. Simonyan, ``Large scale GAN training for high fidelity natural image synthesis,'' in _Proceedings of International Conference on Learning Representations_ , New Orleans, USA, 2019.
  * [27] P. Dhariwal and A. Nichol, ``Diffusion models beat gans on image synthesis,'' _Advances in Neural Information Processing Systems_ , vol. 34, pp. 8780–8794, 2021.
  * [28] Y. Wu and K. He, ``Group normalization,'' in _Proceedings of the European conference on computer vision_ , Munich, Germany, 2018, pp. 3–19.
  * [29] J. Kong, J. Kim, and J. Bae, ``Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,'' _Advances in Neural Information Processing Systems_ , vol. 33, pp. 17 022–17 033, 2020.
  * [30] H. Kim, S. Kim, and S. Yoon, ``Guided-tts: A diffusion model for text-to-speech via classifier guidance,'' in _Proceedings of International Conference on Machine Learning_ , Baltimore, USA, 2022, pp. 11 119–11 133.
  * [31] K. Ito and L. Johnson, ``The lj speech dataset,'' https://keithito.com/LJ-Speech-Dataset/, 2017.
  * [32] M. Fréchet, ``Sur quelques points du calcul fonctionne,'' _Rendiconti del Circolo Matematico di Palermo_ , vol. 22, 1906.
  * [33] A. Gray and J. Markel, ``Distance measures for speech processing,'' _IEEE Transactions on Acoustics, Speech, and Signal Processing_ , vol. 24, no. 5, pp. 380–391, 1976.
  * [34] S. Kullback and R. A. Leibler, ``On information and sufficiency,'' _The annals of mathematical statistics_ , vol. 22, no. 1, pp. 79–86, 1951.
  * [35] H. Liu, Z. Chen, Y. Yuan, X. Mei, X. Liu, D. Mandic, W. Wang, and M. D. Plumbley, ``Audioldm: Text-to-audio generation with latent diffusion models,'' 2023. [Online]. Available: https://arxiv.org/abs/2301.12503
  * [36] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, ``Panns: Large-scale pretrained audio neural networks for audio pattern recognition,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 28, pp. 2880–2894, 2020.