# Experimental Evaluation of All-Optical Up- and Down-Conversion of 3GPP 5G NR
Signals using an Optomechanical Crystal Cavity Frequency Comb

Vicente Fito*, Raúl Ortiz*, Maria Morant, Laura Mercadé, Roberto Llorente, and
Alejandro Martínez Manuscript received December 18, 2023. Reviewed February
14, April 16 and May 29, 2024.This work was supported in part by Generalitat
Valenciana CIAICO/2021/201 TERAFLEX project, IDIFEDER/2021/061 and Spanish
Ministry of Science and Innovation under grants ALLEGRO (PID2021-124618NB-C21)
and MUSICIAN (CHISTERA IV Cofund 2021) and the European Commission under grant
MAGNIFIC (HORIZON-CL4-2022-RESILIENCE-01-10 101091968). *V. Fito and R. Ortiz
contributed equally to this work, supported by PAID-01-20 UPV and GVA
CIACIF/2021/006 grants, respectively. M. Morant work supported by Spain R&D
project PID2019-106163RJ-I00/AEI/10.13039/501100011033 MULTICORE+.Authors are
with Valencia Nanophotonics Technology Center, Universitat Politècnica de
València, 46022 Valencia, Spain. (e-mail: {vfitest, rortfer, mmorant,
laumermo, rllorent, amartinez}@ntc.upv.es).

###### Abstract

Optomechanical crystal cavities (OMCCs) allow the interaction between
localized optical and mechanical modes through the radiation-pressure force.
Driving such cavities with blue-detuned lasers relative to the optical
resonance can induce a phonon lasing regime where the OMCC supports self-
sustained mechanical oscillations. This dynamic state results in a narrow and
stable microwave tone that modulates the laser at integer multiples of the
mechanical resonance frequency, ultimately creating an optomechanical (OM)
frequency comb suitable for microwave photonics applications. OMCCs enable
compact, low-cost power-efficient all-photonic processing of multiple
microwave signals, crucial for current 5G and future beyond-5G systems, whilst
being compatible with silicon integrated photonic circuits. This work reports
the experimental demonstration of all-optical multi-frequency up- and down-
conversion of 3GPP 5G new-radio (NR) signals from the low- to mid- and
extended-mid bands using the first and second harmonics of the frequency comb
generated in a silicon OMCC. The OM comb generates up to 6 harmonics in the
K-band, which is suitable for microwave photonic applications. The
experimental demonstration also evaluates the impact of the phase-noise and
the signal-to-noise ratio (SNR) in the frequency-converted 5G NR signals when
the first and second OMCC harmonics are employed for frequency conversion.

###### Index Terms:

Microwave photonics, Silicon photonics, opto-mechanical crystal cavity,
frequency comb, all-optical frequency conversion, radio-over-fiber, 5G New-
Radio.

##  I Introduction

Cavity optomechanics explore the interaction between light and mechanical
waves when confined in small cavities, leading to a range of interesting
physical phenomena [1, 2, 3]. When an optomechanical crystal cavity (OMCC) is
driven by a laser whose frequency is larger than the optical resonance (i.e.
blue-detuned), the cavity can enter into a dynamics in which the optical mode
interacts with the mechanical motion reducing its damping rate, which
ultimately leads to a regime of self-sustained oscillations which effectively
generates mechanical phonon-based lasing [4, 5].
The OMCC operates in a highly non-linear regime where a very narrow tone at
the mechanical resonance frequency modulates the driving laser [6, 7].
Interestingly, this new optical signal is also modulated by the amplified
mechanical motion [1], resulting in a cascaded process that resembles the
formation of an optical frequency comb [8, 9]. Ad-hoc design of the cavity
permits obtaining a mechanical resonance at GHz frequencies suitable for
microwave photonic applications, where a set of very narrow and highly stable
microwave tones (at integer multiples of the mechanical frequency) are
generated after photodetection.
Frequency conversion using photonic techniques have been recently introduced
due to their potential advantages to provide large bandwidth and full-band
operation enabling the operation at extended radio-frequency (RF) bands.
Optoelectronic oscillator (OEO) systems have been also proposed for frequency
conversion, as they enable a power-efficient frequency conversion [10].
Silicon photonic integrated OEOs have been demonstrated with phase noise
levels of $-80$ dBc/Hz at 10 and 100 kHz frequency offsets [11]. Also
microwave photonic mixers can implement frequency conversion using two I/Q
modulators as proposed in [12]. Although photonic solutions using modulators
have a larger footprint, the phase noise level can be reduced to $-90$ dBc/Hz
using two cascaded electro-absorption modulators (EAMs) [13]. In this work, we
also evaluate the phase noise of the first and second harmonics generated with
the proposed OMCC, achieving levels smaller than $-97$ dBc/Hz and $-91$ dBc/Hz
at 100 kHz for the first and second harmonics, respectively, while being more
a compact solution than other implementations reported in the literature [14].
The compactness of the solution is of special importance for large antenna
arrays employed in massive MIMO wireless communication systems, where the
integration of numerous antenna elements demands compact, low driving power
and efficient processing chains. As large antenna arrays are becoming a
reality in 5G and beyond-5G radio networks [15], the associated up/down-
conversion stages in the processing chain remain a challenging technology
where bulky and expensive electronic solutions are often employed [16]. For
these reasons, the herein proposed OMCC offers great potential for microwave
photonic applications, as it enables the optical frequency conversion of
antenna signals with a compact, power-efficient and reduced phase-noise device
that can be further integrated with other photonic and electronic systems
employing silicon photonics micro- and nano-fabrication processes.

Figure 1: (a) SEM image of the silicon OMCC composed by 10 mirror cells
($N_{M}$), 6 transition cells ($N_{T}$) and a defect cell ($N_{D}$). Inset:
Cavity unit cell with its dimensions. (b) PSD of the thermally transduced
mechanical mode with a Lorentzian fit. (c) Optical mode ($\lambda_{o}$)
measured with an oscilloscope and a Lorentzian fit curve. (d) RF spectrum with
six harmonics of the OM frequency comb in the lasing regime (measured with
RBW=10 MHz).

OMCCs have also been proposed for microwave-to-optical conversion of single
photons as quantum transducers [17, 18]. This could facilitate the transfer of
quantum states between distant superconducting quantum processors. However,
OMCCs have a role beyond quantum transduction [3] as they can be used for
frequency conversion of telecommunication signals [14, 19]. In previous works,
we demonstrated the frequency conversion using the first harmonic generated by
a silicon OMCC considering orthogonal frequency division multiplexing (OFDM)
signals following the IEEE 802.16e WiMAX wireless standard with up to 7-MHz
bandwidth (BW) [19] and for SATCOM applications [20]. In this work, we extend
the experimental evaluation of all-optical multi-frequency conversion using
the first and the second harmonics via cavity optomechanics in the phonon
lasing regime with higher-BW wireless signals following the $3^{rd}$
Generation Partnership Project (3GPP) 5G New-Radio (NR) specification with up
to 100-MHz BW per channel. The novelty of this work resides in using the on-
chip silicon OMCC to perform frequency up- and down-conversion within the low
($<1$ GHz), mid (1-6 GHz) and extended mid-bands ($8.5$ GHz) of the 5G
spectrum. The 7 to 8.5 GHz spectrum range was recently (in December 2023)
proposed by the ITU for next-generation 6G to provide immersive, reliable, and
low-latency communication (e.g., industrial applications, telemedicine…) as
well as enhanced ubiquitous connectivity to bridge the digital divide in rural
and remote areas. We evaluate all the 3GPP 5G standard channel BWs ranging
from 5 to 100 MHz and analyze experimentally the impact of the OMCC frequency
offset and the resulting signal-to-noise ratio (SNR) levels of the frequency-
converted 5G NR signals, comparing the received error vector magnitude (EVM)
with the 3GPP recommendation.
This paper is structured as follows: In Section II, the designed and
fabricated silicon OMCC is presented and characterized experimentally
including the short- and long-term evaluation of the mechanical frequency
offset for the first and second harmonics of the OMCC, together with phase
noise measurements to identify the possible noise sources in the oscillator.
In Section III, the performance of the all-optical frequency conversion of 5G
NR signals using the first and second OMCC harmonics is evaluated in both up-
and down-conversion processes, taking into account the oscillator frequency
offset and the SNR level of the frequency-converted 5G NR signals. Finally, in
Section IV, the main conclusions of this work are highlighted. Our results
open the door toward the use of OMCCs integrated in silicon chips as
appropriate building blocks for 5G and beyond-5G distribution networks.

##  II Silicon OMCC Design and Characterization

The cavity used to conduct the experimental demonstration is a suspended
silicon OMCC [8]. The silicon OMCC is fabricated using standard silicon
fabrication tools [21]. Its architecture features periodic photonic crystal
mirror cells along its sides, establishing a photonic bandgap for TE-like
modes at $\lambda_{o}=1523.2$ nm. The OMCC structure composed by 10 mirror
cells ($N_{M}$), 6 transition cells ($N_{T}$), and a defect cell ($N_{D}$) is
shown in the scanning electron microscope (SEM) image included in Fig. 1(a)
within a foot-print of $\approx 10\;\mu m^{2}$. A complete phononic bandgap is
achieved at the mirror cells at $f_{m}=3.8$ GHz [8]. The mirror cell sizes
are: $l_{M}=500$ nm, $d_{M}=250$ nm, $r_{M}=150$ nm, $a_{M}=500$ nm. To
increase the interaction between the optical and mechanical modes, the unit
cell dimensions adiabatically vary from the cavity sides to its center so that
the optical and mechanical modes are confined at the center of the cavity. The
defect cell size is $l_{D}=250$ nm, $d_{D}=163$ nm, $r_{D}=98$ nm, $a_{D}=325$
nm. The waveguide width and thickness are $w=570$ nm and $t=220$ nm.
It is possible to calculate this interaction with the vacuum OM coupling rate
($g_{0}$), which has two contributions: the photoelastic (PE) and the moving
interfaces (MI) effects [22]. Our cavity supports a set of GHz mechanical
resonances [23], each of them having a different vacuum OM coupling rate with
the optical mode. One of them will show the highest value of $g_{0}$
($g_{0}\approx 2\pi\cdot 200$ kHz, as reported in [8]) and this will be the
mode used in this experimental demonstration.

In order to characterize the optical and the mechanical response of the
cavity, light is coupled in and out of the system by means of a tapered fiber
[24] via a fiber loop [21]. The resulting optical resonance was observed in
transmission on an oscilloscope, and the mechanical response was evaluated
through reflection using a circulator and monitored using a spectrum analyzer.
Fig. 1(b)-(c) show the thermally transduced mechanical mode at $f_{m}=3.84$
GHz and the normalized transmission of the measured optical resonance of the
cavity centered at $\lambda_{o}=1523$ nm. The mechanical and optical quality
factors are: $Q_{m}=6.7\cdot 10^{3}$ and $Q_{o}=3.0\cdot 10^{3}$. This leads
to an optical decay rate $\kappa=65$ GHz, which means that we are in the
unresolved sideband regime ($\kappa>>2\pi f_{m}$). In this given cavity limit,
an optomechanical frequency comb can arise [9] as it can be seen
experimentally in Fig. 1(d), showing the first six harmonics of the OM
frequency comb in the so-called phonon lasing regime spaced by the mechanical
frequency at $kf_{m}$ (with $k=1,2,...6$). With the harmonics provided by the
OM frequency comb, the microwave photonics RF spectrum up to the K-band can be
covered. Next, a detailed examination of the Voigt-like shape characterizing
the first and second harmonics in the lasing regime mode is presented in Fig.
2(a). The maximum amplitude span and frequency drift of these mechanical
peaks, measured over a twenty-minute interval (long-term evaluation), are
illustrated in Fig. 2(b). Notably, the frequency offset of the second harmonic
($k=2$) is approximately twice that of the first harmonic ($k=1$), yet both
exhibit frequency offsets smaller than 250 kHz. These values are well below
the specification defined by 3GPP in TS 38.141-1 (Release 15)[25], where the
maximum frequency offset is set for both non-contiguous and in-band adjacent
channel configurations. In particular, for non-contiguous channels, a maximum
frequency offset of 2.5 MHz is allowed for 5-MHz BW 5G signals. This frequency
offset can increase up to 30 MHz for 100-MHz BW 5G signals. The selectivity is
more restrictive for in-band adjacent channels where a maximum frequency
offset of ±2.5025 MHz (i.e. a maximum of 5.005 MHz) is allowed for 5-MHz BW 5G
signals and up to ±9.4675 MHz (i.e. maximum of 18.935 MHz) for 100-MHz BW 5G
signals [25].

Figure 2: (a) Measured RF power of the first two harmonics of the OM frequency
comb generated with the OMCC in the lasing regime at $kf_{m}$. (b) Mechanical
frequency drift measured with max hold during 20 minutes (long-term
evaluation) for the first and second harmonics. (c) Frequency offset of the
first and second OMCC harmonic measured over 10 s (short-term evaluation). (d,
e) Phase noise measurements for the first and second harmonics of the GHz
lasing regime for $k=1$ and 2. The fitted slopes correspond to the noise
sources in the oscillator. (f) Allan deviation results calculated from the
phase noise contributions for the first and second harmonics. Blue and red
continuous lines show the Allan deviation, and dashed lines depict their power
low fits.

In order to comprehensively evaluate the system’s stability, we conducted
measurements for phase noise and the temporal evolution of the center
frequency at short periods. An examination of the system’s temporal evolution
of the detected frequency was conducted covering a time span of 10 seconds
(short-term evaluation). The results of this analysis are presented in Fig.
2(c), where the frequency drift of both harmonics is assessed over the
specified duration. Notably, the frequency deviation of $kf_{m}$ for the
second harmonic is observed to be approximately twice that obtained for the
first harmonic, aligning with the anticipated behavior of Fig. 2(b) at longer
time measurements. Figures 2(d)-(e) depict the phase noise of the first and
second harmonics, along with their dependencies based on Lesson’s model [26].
The phase noise exhibits characteristic dependencies, including $f^{-4}$
(random walk of frequency), $f^{-3}$ (flicker frequency noise), $f^{-2}$
(white frequency noise or random walk), and $f^{0}$ (white phase noise). The
lower white phase noise observed in Fig. 2(e) can be attributed to the lower
peak power of the signal. Further analysis involves retrieving the mechanical
linewidth of the harmonics from the white frequency noise dependence
($f^{-2}$), resulting in an effective linewidth of $\Gamma_{eff}/2\pi=74$ Hz
and lower values of the phase noise of -97.1 dBc/Hz at 100 kHz for the first
harmonic. This finding emphasizes the system’s highly competitive, stable
free-running oscillator [27]. The jitter of the system [28] for the two given
harmonics was also calculated providing values as low as $J_{RMS}=1.1$ ps at
10 kHz - 100 kHz for both comb lines, resulting in a thirtyfold reduction in
jitter compared to other free running oscillators [29]. The stability of the
system is further explored through its representation in the Allan deviation
included in Fig. 2(f) for both comb peaks. A noteworthy observation is the
convergence of the random walk of frequency ($\tau^{1}-f^{-4}$ contribution)
for both harmonics. This convergence indicates that the random processes
contributing to this phenomenon are comparable for both harmonics. This
similarity in behavior can be attributed to various factors, such as thermal
noise or other stochastic processes influencing the system [26].

##  III All-optical frequency conversion of 5G NR signals using the OM
Frequency Comb

Figure 3 shows the experimental setup used for the evaluation of all-optical
frequency conversion performance of 5G NR signals using the proposed OMCC. The
primary objective of this experimental demonstration is the frequency
conversion of fully standard 3GPP 5G NR signals from the 5G low band (below 1
GHz) –characterized by broader coverage yet reduced speeds– to both the 5G
mid-band (ranging from 1 to 6 GHz) and the extended mid-band (up to 8.5 GHz)
also proposed by ITU for 6G. This is accomplished by leveraging the OM
frequency comb generated by the OMCC when operating in the phonon lasing
regime. When the continuous wave (CW) light interacts with the OMCC,
transmitted and reflected components are obtained. The transmitted beam is
used for monitoring the mechanical mode of the OMCC, while the reflected beam
is amplified with an Erbium-doped fiber amplifier (EDFA) and modulated using a
Mach-Zehnder (MZ). The MZ is fed with standard 3GPP 5G NR signals using
frequency division duplex (FDD) in the frequency range FR1, generated with a
vector signal generator (R&S SMW200A), centered at $f_{0}$ with electrical BWs
up to 100 MHz. Optical double sideband modulation is achieved with the MZ
operating at the quadrature bias (QB) point, generating replicas of the
original 5G signal at linear combinations of the OMCC harmonic frequencies
(obtained at center $kf_{m}$ with $k=1$ and $k=2$) and the 5G signal original
frequency ($f_{0}$). Both upconversion and downconversion frequency processes
are evaluated considering the same frequency ranges for 5G low-, mid-, and
extended mid-bands. Thus, the original 5G signal center frequency $f_{0}$ is
changed accordingly.
At the receiver, a 12-GHz BW photodetector (PD) is used, and the resulting up-
and down-converted 5G NR signals are demodulated with R&S FSW43 signal and
spectrum analyzer. The error vector magnitude (EVM) and received
constellations of the 5G physical downlink shared channel (PDSCH) are analyzed
and compared with the 3GPP standard EVM recommendation for 5G PDSCH modulated
with quadrature phase shift keying (QPSK) of 18.5%[25].

Figure 3: Experimental setup for all-optical frequency conversion using OMCC.
The transmission path is used for phonon lasing monitoring, and the reflection
path is used for 5G NR transmission implementing up- or down-conversion.

###  III-A Frequency upconversion performance

The resulting RF spectrum for upconversion applications is represented in Fig.
4(a), showing the original low-band 5G NR signal centered at $f_{0}=1$ GHz and
the first OMCC harmonic ($k=1$) located at $f_{m}=3.84$ GHz, with double-
sideband replicas corresponding to the 5G mid-band centered at
$f_{m}-f_{0}=2.84$ GHz and $f_{m}+f_{0}=4.84$ GHz. Next to this, the second
OMCC harmonic ($k=2$) is found at $2f_{m}=7.68$ GHz, and the upconverted
signals appear at $2f_{m}-f_{0}=6.68$ GHz and $2f_{m}+f_{0}=8.68$ GHz
(corresponding to the 5G extended mid-band). In these experiments, the worst-
case scenario is considered by measuring the EVM of the higher sidebands
located at $f_{m}+f_{0}=4.84$ GHz (upconverted using the first harmonic with
$k=1$) and at $2f_{m}+f_{0}=8.68$ GHz (upconverted using the second harmonic
with $k=2$).

Figure 4(b) presents the received EVM for each upconverted frequency and for
different electrical bandwidths following the 3GPP 5G NR standard ranging from
5-MHz to 100-MHz BW and adjusting the electrical power of the generated
signals to obtain the optimum EVM (within the range of the 5G NR generator).
The figure includes examples of received constellations of the 5G physical
downlink shared channel (PDSCH). A horizontal dashed line in the EVM graphs
represents the 3GPP standard EVM recommendation for 5G PDSCH modulated with
QPSK at 18.5%[25]. The experimental results confirm that the upconverted
signals to the 5G mid-band using the first OMCC harmonic meet the 3GPP EVM
recommendation for the maximum 5G BW of 100 MHz. However, the upconversion to
the extended mid-band using the second harmonic is limited to 5G NR signals
with up to 10-MHz BW. The EVM degradation is more pronounced for the signals
upconverted using the second harmonic due to its reduced SNR. As it can be
observed in the resulting spectrum included in Fig. 4(a), the SNR decreases
from $35.6$ dB for the original 5G signal ($f_{0}$) to $26.6$ dB for the
upconverted signal with the first harmonic ($f_{m}+f_{0}$), further
diminishing to $8.6$ dB for the upconverted signal with the second harmonic
($2f_{m}+f_{0}$). This difference in SNR between the signal upconverted with
the first and second harmonics, together with the higher frequency offset
present in the second harmonic, as seen in Fig. 4(b), imposes limitations on
upconversion using the second harmonic to 5G bandwidths up to 10 MHz.
Achieving higher bandwidths with the second harmonic will require a higher
SNR, as evaluated in the next subsections of this study.

Figure 4: (a) Received RF spectrum of the upconversion process of a 100-MHz BW
5G NR signal originally at $f_{0}=1\;$GHz (measured with RBW = 10 MHz) and
optical power received by the PD $=-2$ dBm. (b) Measured EVM and
constellations of the PDSCH upconverted signals at $kf_{m}+f_{0}$ using the
first ($k=1$) and second ($k=2$) OMCC harmonics vs. BW, compared with the
original 5G NR signal at $f_{0}=1$ GHz. The 3GPP EVM recommendation is
included as a horizontal dashed line.

###  III-B Frequency downconversion performance

The downconversion process is also evaluated using the same experimental setup
depicted in Fig. 3. In order to compare the same frequency ranges, in this
case, the original 5G NR signal is generated at $f_{0}=8.68\;$GHz. Thus, the
downconverted signal using the first OMC harmonic appears at
$f_{0}-f_{m}=4.84\;$GHz as shown in the received spectrum reported in Fig.
5(a). The results of Fig. 5(b) confirm the EVM compliance of 5G NR signals
with up to 100-MHz BW downconverted employing the first harmonic.

If we compare the up- and down-conversion processes, we can observe that the
received EVM escalates from 7.36% for a 5-MHz BW to 8.63% for a 100-MHz BW for
signals upconverted with the first harmonic. For downconverted signals also
using the first harmonic, the received EVM rises from 9.6% for a 5-MHz BW to
15.86% for a 100-MHz BW. It is worth noting that the SNR of the upconverted
signal with the first harmonic is measured to be $26.6$ dB, in contrast with
the downconverted signal at the same center frequency, which records an SNR of
$24.6$ dB. Consequently, the EVM of the upconverted signals exhibits a
marginally better performance than the downconverted signals over the same
frequencies. The impact of the frequency offset and received SNR in the EVM
will be further evaluated in the next subsections.

Figure 5: (a) Received RF spectrum of the downconversion process with original
100-MHz BW 5G NR signal centered at $f_{0}=8.6\;$GHz and downconverted to
$4.8\;$GHz (measured with RBW=10 MHz) with optical power received at the
PD$=-2\;$dBm. (b) Received EVM and example constellations of the PDSCH of the
original 5G NR and downconverted signals using the first OMCC harmonic for
different electrical BWs. 3GPP EVM recommendation included as a horizontal
dashed line.

###  III-C Impact of frequency offset

In Fig. 6(a), we examine the impact of frequency offset on the received 5G NR
constellations with the R&S FSW43 analyzer. As previously elucidated in Fig.
2(b)-(c), the central frequencies of the OMCC harmonics display some
variability, with this frequency offset escalating with the harmonic order.
For simplification, the performance of our 3GPP 5G NR receiver is evaluated
using a 100-MHz BW 5G NR signal centered at $f_{0}=1\;$GHz, with the
demodulator’s center frequency adjusted to $f_{0}+f$offset. Figure 6(a)
confirms that frequency offsets below 8 kHz exert negligible impact on the
received constellation, allowing clear identification of two orange points
corresponding to binary phase shift keying (BPSK) pilots and four blue points
of the PDSCH data channel. It should be noted that for offsets higher than 100
Hz the analyzer should be configured to demodulate QPSK data signals, as for
offsets between 100 Hz and 8 kHz, it tends to miscalculate the quality of the
signal by considering some constellation points as higher-order modulation
(e.g. 256QAM). For offsets higher than 8 kHz, a phase mismatch is intensified
in the R&S FSW43 (which is more sensitive than 3GPP 5G NR recommendation[25]),
showing an outer ring similar to an amplitude phase shift keying (APSK)
signal. This phenomenon was also observed in Fig. 4(b) when upconverting using
the second harmonic of the OM frequency comb (with $k=2$).

Figure 6: (a) Received 5G NR constellations when transmitted at $f_{0}=1\;$GHz
and received at $f_{0}+f$offset to evaluate the analyzer sensitivity. (b)
Received PDSCH constellations and EVM of the upconverted 5G NR signals using
the first harmonic ($k=1$ to $4.8\;$GHz) and the second harmonic ($k=2$ to
$8.6\;$GHz) for different BWs and constant SNR = 25 dB. (c) Comparison of
PDSCH QPSK and 16QAM up-converted 5G NR signals with $k=1$.

Fig. 6(b) shows the received EVM and constellations of two upconverted signals
utilizing different harmonics while maintaining a constant SNR$=25\;$dB. It
can be observed that the received signal upconverted using the second harmonic
has a higher EVM compared with using the first harmonic, although taken with
similar SNR. As represented in Fig. 2, higher-order harmonics possess elevated
phase noise and greater frequency drift. Also, higher-order PDSCH modulations
were evaluated experimentally with quadrature amplitude modulation (QAM). As
it can be observed in Fig. 6(c), a successful upconversion of 16QAM 5G NR
signals is achieved with the first harmonic of the OMCC, ensuring an EVM
better than the 3GPP recommendation of 12.5% for 16QAM modulation[25].
Remarkably, a slightly higher SNR is required compared with QPSK to achieve a
similar EVM. As part of ongoing work, the implementation of a closed loop,
wherein the output of the OMCC feeds back into the input, could be explored to
reduce the frequency drift and enhance the system stability. Refining the
oscillator has the additional benefit of increasing the peak power of the
lasing OMCC mode, potentially increasing the power of each harmonic and the
resulting SNR, also enabling higher-order modulations.

###  III-D Impact of signal-to-noise ratio

Next, we studied the influence of the received SNR on the resulting EVM of the
frequency-converted 5G signals. For simplicity, we evaluated the performance
of the upconverted signals using the first harmonic of the OM frequency comb
($k=1$), which appears at center frequency $f_{m}+f_{0}=4.84$ GHz. In Fig.
7(a), two curves depicting BW versus EVM are presented: one curve maintains
constant electrical power (set at PEL$=-6$ dBm), while the other represents a
scenario with constant SNR=19.5 dB. Notably, the curve of constant SNR
exhibits a marginal variation in EVM, ranging from 16.28% for 5-MHz BW signals
to 16.98% with 100-MHz BW. In contrast, the curve with constant generated
power displays a more pronounced variation, escalating from 16.28% at 5 MHz to
67.33% at 100 MHz, only satisfying the 3GPP 5G PDSCH recommendation for 5-MHz
BW, which corresponds to an SNR of 19.5 dB. This observation underscores that
the deterioration in EVM is correlated with a reduction in SNR for the same
electrical power over different BWs. Further clarification is provided in Fig.
7(b), where the electrical spectra of the upconverted signals evaluated in
Fig. 7(a) are presented. In this case, we compare the spectrum of upconverted
20-MHz BW 5G NR signals when setting the SNR to 19.5 dBm and when configuring
the generator with constant power at PEL$=-6$ dBm. The difference in SNR
between both configurations is evident.

Figure 7: (a) Received EVM of a 5G NR signal upconverted to $4.8\;$GHz with
constant electrical power of the signal generator $P$EL$=-6$ dBm and with
constant SNR$=19.5$ dB. (b) Electrical spectrum corresponding to 20-MHz BW 5G
upconverted signals for SNR=19.5 dB vs. $P$EL$=-6$ dBm.

In Fig. 8(a), the EVM evaluation of frequency upconversion using the first
harmonic is evaluated with constant SNR. The experimental results confirm
that, for SNR values above 19.5 dB, all the BWs meet the EVM criteria. In Fig.
8(b), two sets of data are presented to further evaluate the relationship
between SNR and EVM, each one corresponding to the received upconverted 5G NR
signal using the first and second harmonics provided by the OMCC. Within each
dataset, each symbol on the graph represents the mean values of EVM and SNR
calculated from three measurements taken in a 10-minute interval with
identical BWs. The EVM behaviour is consistent with [30, 31]: EVM $\approx$
SNR-1/2. Two fit lines have been introduced in Fig. 8(b) with the
approximation of $y=a\cdot x^{-1/2}+b$ with a coefficient of determination
$(R^{2})$ higher than 0.96 for the first harmonic ($k=1$) and higher than 0.99
for the second harmonic ($k=2$). A discernible difference in EVM can be
observed by comparing the signals upconverted with the first and with the
second harmonic. This arises from the frequency offset and phase noise
associated with the second harmonic, as depicted in Fig. 2(b) and (c).
Consequently, the signals upconverted with the second harmonic show a higher
EVM than those upconverted with the first harmonic, even for similar received
SNR values. Within the SNR range from 15 dB to 25 dB, this difference in EVM
is approximately 6%. This degradation implies that, whilst applications using
the first harmonic require a received SNR of 19.5 dB, using the second
harmonic to achieve higher frequency bands requires SNR values above 23 dB.

Figure 8: (a) Received EVM of the upconverted 5G NR signal with the first
harmonic ($k=1$) measured at $f_{m}+f_{0}=4.8$ GHz for different SNR and BWs.
(b) Received mean EVM calculated from 3 measurements taken in a 10-minute
interval with identical BWs for the upconverted 5G NR signal using the first
and second harmonics for different SNRs. 3GPP EVM recommendation included as a
horizontal dashed line. Constellation examples included as insets marked with
circles in the graphs.

##  IV Conclusions

This paper proposes and demonstrates experimentally the all-optical frequency
conversion of 3GPP 5G NR signals using an on-chip OMCC that enables the up-
and down-frequency conversion of microwave signals in the GHz range using the
generated OM frequency comb. OMCCs main advantages includes the reduced
footprint (ideal for massive integration on silicon chips) and minimal driving
power (ideally below 1 mW), making it well-suited for low-power applications.
These features significantly bolster the practicality and cost-effectiveness
of our proposed all-optical frequency conversion method. The first and second
harmonics generated by the OM frequency comb in the lasing regime mode were
evaluated in detail in terms of amplitude span, frequency offset, and phase
noise. A maximum frequency deviation of 12 kHz for the first harmonic and 28
kHz for the second harmonic have been measured over 10 s (short-term
evaluation). As expected, the stability of the mechanical resonance has a
higher impact on higher-order harmonics of the generated OM frequency comb.
Thus, the frequency offset of the second harmonic was measured to be
approximately twice that of the first harmonic. The stability of the OMCC was
measured for a 20-min interval (long-term evaluation), obtaining a frequency
drift smaller than 250 kHz for both harmonics. These frequency offset values
meet the 3GPP specification for 5G NR signals with the maximum BW of the
standard of 100 MHz. The given OMCC results in low phase noise values of -97.1
dBc/Hz and -91.95 dBc/Hz at 100 kHz for the first and second harmonics,
respectively. This phase noise value is smaller than other Silicon photonic
integrated solutions [11].
Frequency upconversion from the 5G low-band at 1 GHz to the 5G mid-band at 4.8
GHz using the first harmonic is demonstrated with signals up to 100-MHz BW,
with received EVM smaller than 9.84%. Frequency upconversion using the second
harmonic to the 5G extended mid-band and next-generation 6G at 8.6 GHz is also
achievable but with smaller BW, due to the reduced SNR using the second
harmonic. Successful upconversion with the second harmonic can be achieved
with up to 10-MHz BW 5G NR signals with EVM smaller than 17.68%. The
downconversion process from the same frequencies (from 8.6 GHz back down to
4.8 GHz) is also demonstrated with up to 100-MHz BW 5G NR signals with EVM
smaller than 15.87%, confirming the all-optical multi-frequency conversion
using the OM frequency comb.
Comparing the performance using the first and second harmonics, an EVM
degradation of 6% is observed for higher-order harmonics due to the frequency
offset and phase noise difference between the harmonics generated by the OMCC.
When using the first harmonic an SNR of 19.5 dB is recommended, while using
the second harmonic requires a SNR above 23 dB. The reduction of SNR for
higher-order harmonics may hinder the application of our technique at
frequencies above 8 GHz (roughly, the second harmonic of our cavity). A
possible solution would be the use of more complex, two-dimensional OMCCs
supporting mechanical resonances at higher frequencies [32, 33]. In principle,
such cavities should show similar phase noise levels, though no experimental
results have been yet provided. As ongoing work, a closed loop in which the
output of the OMCC feeds back into the input could be implemented to reduce
the frequency offset and improve the stability of higher-order harmonics.

## References

  * [1] M. Aspelmeyer et al., “Cavity Optomechanics,” Reviews of Modern Physics 86 (4), 1391-1452 (2014).
  * [2] J. Chan et al., “Laser cooling of a nanomechanical oscillator into its quantum ground state,” Nature 478, 89–92 (2011).
  * [3] E. Verhagen et al., “Quantum-coherent coupling of a mechanical oscillator to an optical cavity mode,” Nature 482, 63–67 (2012).
  * [4] T. J. Kippenberg et al., “Cavity Optomechanics: Back-Action at the Mesoscale,” Science 321, 1172-1176 (2008).
  * [5] D. Navarro-Urrios et al., “Nonlinear dynamics and chaos in an optomechanical beam,” Nature Communications 8, 14965 (2017).
  * [6] I. S. Grudinin et al., “Phonon laser action in a tunable two-level system,” Phys. Rev. Lett. 104 (8), 083901 (2010).
  * [7] D. Navarro-Urrios et al., “A self-stabilized coherent phonon source driven by optical forces,” Scientific 5, 15733 (2015).
  * [8] L. Mercadé et al., “Microwave oscillator and frequency comb in a silicon optomechanical cavity with a full phononic bandgap,” Nanophotonics 9, 3535–3544 (2020).
  * [9] M. Miri et al., “Optomechanical frequency combs,” New J. Phys. 20, 043013 (2018).
  * [10] J. Capmany et al., “Microwave photonics combines two worlds,” Nature Photonics 1 (6), 319–330 (2007).
  * [11] W. Zhang et al., “A silicon photonic integrated frequency-tunable optoelectronic oscillator,” 2017 International Topical Meeting on Microwave Photonics, 1-4 (2017).
  * [12] Y. Zheng et al., “Full-band direct-conversion receiver using microwave photonic I/Q mixer,” 2016 International Conference on Photonics in Switching, 1-3 (2016).
  * [13] J.Y. Lee et al., “Photonic frequency down-converter based on a frequency-doubling OEO using two cascaded EAMs,” IEEE Photonics Technology Letters 29 (18), 1529-1532 (2017).
  * [14] M. Hossein-Zadeh et al., “Photonic RF down-converter based on optomechanical oscillation,” IEEE Photonics Technology Letters 20 (4), 234–236 (2008).
  * [15] T. Lin et al., “Beamforming design for large-scale antenna arrays using deep learning,” IEEE Wireless Communications Letters, 9 (1), 103-107, (2020)
  * [16] T. Gong et al.,“RF chain reduction for MIMO systems: a hardware prototype,” IEEE Systems Journal, 14 (4), 5296-5307 (2020).
  * [17] J. Bochmann et al., “Nanomechanical coupling between microwave and optical photons,” Nature physics 9, 712–716 (2013).
  * [18] N. Lauk et al., “Perspectives on quantum transduction,” QST 5 (2), (2020).
  * [19] L. Mercadé et al., “Photonic frequency conversion of OFDM microwave signals in a wavelength-scale optomechanical cavity,” Laser Photonics Rev. 15 (11), 2100175 (2021).
  * [20] L. Mercadé et al., “Testing optomechanical microwave oscillators for SATCOM application,” J. Lightwave Technol. 40 (14), 4539-4547 (2022).
  * [21] J. Gomis-Bresco et al., “A one-dimensional optomechanical crystal with a complete phononic band gap,” Nat. Commun. 5, 4452 (2014).
  * [22] K. C. Balram et al., “Moving boundary and photoelastic coupling in GaAs optomechanical resonators,” Optica 1, 414-420 (2014).
  * [23] L. Mercadé et al., “Engineering multiple GHz mechanical modes in optomechanical crystal cavities,” Phys. Rev. Applied 19 (1), 014043 (2023).
  * [24] J. T. Hill et al., “Coherent optical wavelength conversion via cavity optomechanics,” Nat. Commun. 3, 1196 (2012).
  * [25] 3GPP TS 38.141-1 version 15.15.0 Release 15, 5G NR Base Station (BS) conformance testing. Part 1: Conducted conformance testing (2023).
  * [26] E. Rubiola, “Phase noise and frequency stability in oscillators” The Cambridge RF and Microwave Engineering Series. Cambridge University Press (2008).
  * [27] R. Horváth et al., “Sub-Hz closed-loop electro-optomechanical oscillator with Gallium Phosphide photonic crystal integrated on SoI circuitry,” ACS Photonics 10 (8), 2540–2548 (2023).
  * [28] X. Luan et al., “An integrated low phase noise radiation-pressure-driven optomechanical oscillator chipset,” Sci. Rep. 4, 6842 (2014).
  * [29] J. Zheng et al., “Feedback and harmonic locking of slot-type optomechanical oscillators to external low-noise reference clocks,” Appl. Phys. Lett. 102 (14), 141117 (2013).
  * [30] D. Nguyen et al., “M-QAM transmission over hybrid microwave photonic links at the K-band”. Optics Express. 27 (23), 33745-33756 (2019).
  * [31] R. A. Shafik, et al., “On the extended relationships among EVM, BER and SNR as performance metrics,” 2006 International Conference on Electrical and Computer Engineering, 408-411 (2006).
  * [32] G. Madiot et al., “Multimode optomechanics with a two-dimensional optomechanical crystal,” APL Photonics 8 (11), 116107 (2023).
  * [33] H. Amir et al., “Two-Dimensional Phononic-Photonic Band Gap Optomechanical Crystal Cavity,” Physical Review Letters 112 (15), 153603 (2014).


# Analysis of the BUT Diarization System for VoxConverse Challenge

###### Abstract

This paper describes the system developed by the BUT team for the fourth track
of the VoxCeleb Speaker Recognition Challenge, focusing on diarization on the
VoxConverse dataset. The system consists of signal pre-processing, voice
activity detection, speaker embedding extraction, an initial agglomerative
hierarchical clustering followed by diarization using a Bayesian hidden Markov
model, a reclustering step based on per-speaker global embeddings and
overlapped speech detection and handling. We provide comparisons for each of
the steps and share the implementation of the most relevant modules of our
system. Our system scored second in the challenge in terms of the primary
metric (diarization error rate) and first according to the secondary metric
(Jaccard error rate).

Index Terms—  Speaker Diarization, Variational Bayes, HMM, VoxConverse, VoxSRC
Challenge

##  1 Introduction

Speaker diarization applied to broadcast data has been of interest for decades
in part due to the potential applications such as speech collection of a
speaker of interest, speech search, segmentation or automatic transcription.
In order to address the lower performance of systems on data from radio,
television or web videos, during the last decade, new datasets allowing
diarization on such types of data were released: ETAPE [1], MGB-1 [2],
Albayzin [3] and to a lesser extent DIHARD [4]. Although participants of the
challenges related to the datasets, can access the data without fees, any
other party interested in evaluating their system on such corpora needs to pay
a fee.

VoxConverse [5] is a dataset of videos ‘in the wild’ collected from YouTube
consisting in talk shows, news broadcasts, celebrity interviews, home vlogs,
etc. which is released publicly and for
free111http://www.robots.ox.ac.uk/~vgg/data/voxconverse. The organizers of the
VoxSRC Challenge, famous for presenting a benchmarking dataset for speaker
recognition on VoxCeleb [6] proposed a fourth track in VoxSrc Challenge 2020
focused on audio diarization on VoxConverse.

In this paper we present the system devised by Brno University of Technology
for the challenge with the corresponding analyses of results on the
development set of the corpus. Furthermore, we identify the main challenges
with the dataset and where the focus should be put on to improve the
performance of diarization systems.

Although the proposed system is built on existing technologies, we try for the
first time in our pipeline a voice activity detection (VAD) module combining
different systems. However, our main contribution is the analysis of
performance of each of the modules that compose our best system. Such system
was tuned (and not trained) on the development set of VoxConverse and reached
a diarization error rate (DER) of 4% and a Jaccard error rate (JER) of 19.8%.
Such DER represents almost half of the error reported with the baseline system
[5] when both audio and visual cues were used for performing diarization. The
submission of this system for the evaluation set obtained the second position
in the challenge in terms of the primary evaluation metric (8.12% DER) and the
first position according to the secondary metric (18.35% JER). Together with
this publication we make available the most relevant modules of our system
[7].

##  2 SYSTEM OVERVIEW

Our diarization system comprises the following steps:

  1. 1.

Signal pre-processing

  2. 2.

Voice activity detection

  3. 3.

Speaker embedding extraction

  4. 4.

Initial clustering

  5. 5.

Variational Bayes hidden Markov model (HMM) clustering

  6. 6.

Global speaker embedding reclustering

  7. 7.

Overlap speech detection and handling

In the following subsections, we describe each of the steps and present
comparative results to understand the improvement provided by these steps. The
diarization performance will be presented in terms of DER, its three
components: missed speech, false alarm speech, and speaker error, and JER.
Throughout the paper all diarization results are obtained with 0.25s
forgiveness collar for DER [8] following the challenge protocol while for JER
[4] there is no collar by definition. VAD and overlapped speech detection
(OVD) systems are evaluated without forgiveness collar as well.

###  2.1 Signal pre-processing

We considered two methods for signal preprocessing: the speech enhancement
method based on a long short-term memory (LSTM) network trained on simulated
data [9] (also used in the baseline) and the weighted prediction error (WPE)
[10, 11] as it had proved to be useful in the Second DIHARD challenge [12]. In
our experiments, we saw that using the LSTM-based speech enhancer was
beneficial while the WPE method was actually harmful. Since the latter removes
late reverberation, and many of the recordings in VoxConverse are captured
with studio microphones in close-distance, it is not surprising that this
method does not help with this type of recordings. Therefore, LSTM based
enhancement was applied in all experiments shown below.

###  2.2 Voice activity detection

One key aspect of the challenge is that there are no ground truth VAD labels
to use. In our pipeline, this means that labels have to be produced
automatically before extracting embeddings. For this purpose, we evaluated
three systems:

  * •

an energy-based VAD.

  * •

a deep neural network (DNN) based system with three feed-forward layers
receiving as input $\pm$5 stacked frames and trained to output 10ms frame
decisions (silence / speech) [12]. It was trained on part of the second DIHARD
development set (the rest was used for validation while training), the train
set of the “full-corpus” partition of
AMI222http://groups.inf.ed.ac.uk/ami/corpus/datasets.shtml [13] (the test and
development sets were used for validation while training), ICSI [14] and ISL
[15] meetings.

  * •

an automatic speech recognition (ASR) based system. The frame-level phoneme
labels were generated using the official Kaldi [16] Tedlium speech recognition
recipe (s5_r3) based on the TED-LIUM 3 dataset [17]. Phoneme classes
corresponding to silence and noise were considered silence for the purpose of
VAD and the rest of the classes were considered speech.

For the energy and DNN based systems a median filter was applied to “smooth”
the outputs. For both models the detection threshold and span of the median
filter window were optimized (individually) so that the sum of false alarm
(FA) and miss, comprising the total error, were minimized on VoxConverse
development set.

When analyzing the outputs given by the systems we found out that in many
cases speech segments were separated by short periods of silence (especially
on the ASR based one). We therefore labeled segments of silence shorter than a
certain length as speech in order to decrease the total error. Note that this
is not equivalent to using a more tolerant threshold as the post-processing
only affects short pauses.

In order to leverage the performance of individual systems, we used the
outputs of the three models before removing short segments of silence in a
majority voting system. Then, we removed silences shorter than 0.6s to improve
the performance further. Table 4 presents the performance of each method in
terms of different metrics. We see that all the methods evaluated surpass the
performance of the baseline VAD. Surprisingly, the energy-based system
performs quite well in comparison with more sophisticated methods. Still, it
should be noted that the DNN-based method was trained mostly on meeting-like
recordings. Even if DIHARD contains a more diverse set of recordings, they are
not necessarily mostly broadcast like VoxConverse. Most likely, by making use
of annotated multi-media recordings like the ones used in the challenge, the
performance could still be improved. Analogously, using an ASR system trained
on matched data could attain better performance.

Finally, taking advantage of the high precision of the ASR based system, we
improved the overall performance by marking as silence any segment appearing
more than 0.8s afar from speech detected by the ASR system. We used this VAD
for the following results.

Table 1: Frame accuracy, precision, recall, miss, false alarm, and total error for different VADs on the development set. WebRTC was the method used in the baseline [5] (VAD annotations taken from the diarization output444https://github.com/a-nagrani/VoxSRC2020/blob/master/data/diar/baseline.rttm). VAD | Acc. | Prec. | Reca. | Miss | FA | Error
---|---|---|---|---|---|---
WebRTC | $87.95$ | $0.952$ | $0.914$ | $7.81$ | $4.24$ | $12.05$
Energy | $94.56$ | $0.953$ | $0.989$ | $1.01$ | $4.44$ | $5.45$
\- sil $<$ 0.3s | $94.66$ | $0.952$ | $0.992$ | $0.77$ | $4.57$ | $5.34$
DNN | $96.24$ | $0.978$ | $0.981$ | $1.71$ | $2.05$ | $3.76$
\- sil $<$ 0.7s | $96.37$ | $0.974$ | $0.986$ | $1.24$ | $2.39$ | $3.63$
ASR | $83.30$ | $0.989$ | $0.826$ | $15.84$ | $0.86$ | $16.70$
\- sil $<$ 1.1s | $96.28$ | $0.973$ | $0.987$ | $1.21$ | $2.51$ | $3.72$
Majority voting | $96.62$ | $0.978$ | $0.985$ | $1.37$ | $2.01$ | $3.38$
\- sil $<$ 0.6s | $96.82$ | $0.975$ | $0.990$ | $0.87$ | $2.31$ | $3.18$
\- dist. ASR $<$ 0.8s | $97.02$ | $0.978$ | $0.989$ | $0.97$ | $2.01$ | $2.98$

###  2.3 Speaker embeddings

As explained above, a key element in the diarization pipeline is the
extraction of x-vectors. In our system, 256 dimensional speaker embeddings
were extracted from a 152-layer ResNet [18] DNN. The network was trained on
the development part of the VoxCeleb 2 dataset [6] (5994 speakers in 145k
sessions), cut into 2-second chunks and augmented with noise, as described in
[19] and available as part of the Kaldi-recipe collection [16]. As input, we
used 64-dimensional filter-banks extracted from the original 16 kHz audio with
a window size of 25 ms and a 10 ms shift.

The loss function used for training the DNN was CosFace [20], with scaling
parameter $s$ set to 32 and margin parameter $m$ linearly increased from
$0.05$ to $0.3$ throughout the whole period of training. We ran 1 epoch (i.e.,
passing all training data once) of stochastic gradient descent optimization,
throughout which we exponentially decreased the learning rate from $10^{-1}$
to $10^{-6}$. Note that we scaled the learning rate by the number of parallel
jobs to compensate for the dynamic range of the accumulated gradients, in our
case by 3. The momentum and weight decay were kept constant at $0.9$ and
$5\cdot 10^{-4}$, respectively. The batch size was set to 128, however,
training on 3 GPUs in parallel virtually tripled the batch size. Also, due to
large memory requirements, the gradients were computed over 2 “micro-batches”
of size 64 after which the update step was taken. Note that care needs to be
taken when using this approach in connection with any model that uses batch-
normalization (as our ResNet model does) as batch-norm statistics may get
biased with decreasing batch size.

###  2.4 Initial clustering

As the Bayesian HMM requires an initial assignment of frames to speakers, one
possibility is to assign them randomly to a set of speakers. However, the
model benefits from using a more sensible initial assignment. As in previous
work [12], the x-vectors extracted from an input recording are clustered by
means of agglomerative hierarchical clustering (AHC) with similarity metric
based on probabilistic linear discriminant analysis (PLDA) [21] log-likelihood
ratio scores, as used for speaker verification. The PLDA model for this
purpose was trained on x-vectors extracted from concatenated speech segments
from VoxCeleb 2 [6] which are mean-centered, whitened to have identity
covariance matrix and length-normalized [22]. Unlike in the baseline,
x-vectors are extracted on 1.5 s seconds segments but with overlap of 1.25 s
(instead of 0.75 s) as this proved to be beneficial [23]. When applying
principal component analysis (PCA) per recording to project the embeddings to
few dimensions, we found that keeping 55% of the variability provided better
performance than when using only 30% as in [23] or when using 10% as in the
baseline. This could be explained by the fact that we have better embeddings
than before which encode more relevant information for the task.

The AHC clustering threshold was tuned on the development set to perform well
in tandem with the next step as slightly underclustering allows for better
performance when combined with variational Bayes (VB) HMM diarization. A
comparison of the performance when using the threshold to undercluster and the
one that minimizes DER is presented in Table 2 rows 2 and 4 respectively. It
is clear that a great difference in performance with regard to the baseline
comes from an improved VAD; however, improved speaker embeddings and more
frequent segmentation explain the rest of the difference reaching more than
76% relative improvement in terms of DER and more than 58% in terms of JER.

###  2.5 Bayesian HMM for x-vector clustering

Some of the problems of employing AHC for performing diarization are that it
heavily depends on correctly tuning a clustering threshold for having good
performance and that, it does not make use of the temporal nature of the
embeddings when assigning them to speakers. A more principled way of
addressing diarization is to model the problem with a Bayesian HMM where the
complexity control of the Bayesian learning allows to infer the number of
speakers and where the HMM transitions naturally model time dependencies. In
our Bayesian HMM model, the HMM states represent speakers, the transition
between states represent the speaker turns and the state distributions are
derived from a PLDA model pre-trained on labeled x-vectors in order to
facilitate discrimination between speaker voices. More details on this model
can be found in [23].

The configuration parameters of VB-HMM on x-vectors (VBx) were tuned on
VoxConverse development set so that $F_{A}=0.3$, $F_{B}=16$ and
$P_{loop}=0.9$. Table 2 presents the comparison of results for AHC, AHC with
undercluster threshold, and VBx initialized with the latter AHC labels (row
6). Note the more than 63% relative improvement of VBx over the AHC result in
terms of speaker error (miss and FA are defined by the VAD) and more than 8%
relative improvement in terms of JER. Table 3 presents the amount of
recordings where the estimated number of speakers was less, equal or greater
than the correct number. We see that VBx finds the correct number of speakers
in more than two thirds of the files while AHC only does so in little more
than one half.

Table 2: Diarization performance on the development set. System | DER | Miss | FA | Spk. | JER
---|---|---|---|---|---
Baseline (AHC)555Note that these numbers differ from [5], as the authors scored with a different tool. All the numbers in this paper were produced with https://github.com/nryant/dscore, which was also the official scoring tool for the challenge. | $24.57$ | $11.21$ | $2.26$ | $11.10$ | $51.71$
AHC | $5.73$ | $3.10$ | $0.47$ | $2.16$ | $21.56$
\+ reclustering | $5.47$ | $3.10$ | $0.47$ | $1.90$ | $21.40$
AHC (u-cluster) | $7.23$ | $3.10$ | $0.47$ | $3.66$ | $20.20$
\+ reclustering | $6.19$ | $3.10$ | $0.47$ | $2.62$ | $19.53$
VBx | $4.36$ | $3.10$ | $0.47$ | $0.79$ | $19.78$
\+ reclustering | $4.30$ | $3.10$ | $0.47$ | $0.73$ | $19.81$

###  2.6 Reclustering

One of the main disadvantages of the proposed approach for diarization is that
the embeddings are computed over short segments of speech. This is in part
necessary due to the dynamic nature of conversational speech. However, if
embeddings were extracted on longer segments where we have the belief that the
whole segment belongs to the same speaker, then the embeddings might be more
robust, leading to better clustering.

In this direction is that we propose a “reclustering” step where all segments
of a speaker in the recording, given a previous diarization run, are
concatenated to extract a new embedding. Then, the per speaker global
x-vectors are clustered with AHC to join speakers if necessary.

The application of the reclustering on different diarization outputs (obtained
with AHC, AHC with the underclustering threshold and with VBx) are presented
in Table 2. In all cases we applied PCA to rotate the space but keeping all
dimensions. We see that for AHC, reclustering improves the speaker error by
12% relative while for VBx it is around 7.5% relative. It should be noted that
the speaker error for VBx is already quite low before reclustering, possibly
leaving less room for improvement in comparison with AHC.

Table 3: Number of recordings where the amount of found speakers is greater, equal, and less than the correct amount (CA) and mean across recordings of the difference between the correct amount of speakers and the amount of found speakers. System |  #spk $>$ CA |  #spk $=$ CA |  #spk $<$ CA | mean
---|---|---|---|---
Baseline (AHC) | $82$ | $46$ | $88$ | $0.44$
AHC | $67$ | $110$ | $39$ | $-0.50$
\+ reclustering | $67$ | $110$ | $39$ | $-0.42$
AHC (u-cluster) | $143$ | $55$ | $18$ | $-3.09$
\+ reclustering | $140$ | $56$ | $20$ | $-2.63$
VBx | $12$ | $147$ | $57$ | $0.34$
\+ reclustering | $11$ | $145$ | $60$ | $0.37$

###  2.7 Overlapped speech handling

One common scenario in conversations with several participants is overlapped
speech. In particular, in the VoxCeleb development set an average of 2.9% of
speech has two or more speakers speaking simultaneously [5]. To illustrate the
effect of overlapped speech on DER, if we had perfect VAD and correctly
labeled all segments of speech with only one speaker (in segments with more
than one we only mark one of the correct ones) we would still obtain 2.3%
missed speech. If we also labeled correctly the second speaker where at least
two speakers speak simultaneously, we would obtain 0.08% missed speech. Doing
so for three speakers means already less than 0.01% missed speech. In terms of
JER, the results would be 5.38, 0.26 and 0.02 respectively.

Note that with a system that does not label segments for more than one
speaker, even with perfect VAD and no speaker error, the DER can in the best
case be 2.3%. However, if a second speaker is correctly handled, that error
can be decreased substantially. The advantage of modelling three or more
speakers is negligible on these data and for this reason we focus only on
labeling up to two speakers.

Since the pipeline up to this point outputs only one speaker per frame, a
post-processing step is needed to add second labels. This requires doing OVD
first, and then assigning a second speaker on the found segments.

For overlap detection we used the model trained on AMI available in pyannote
[24] but we tuned the detection threshold on VoxConverse development in order
to maximize the precision. The performance on the development set is shown in
Table 4.

Table 4: OVD performance on the development set in terms of frame accuracy, precision, recall, miss, and false alarm. OVD | Acc. | Prec. | Reca. | Miss | FA | Error
---|---|---|---|---|---|---
AMI-trained | $97.11$ | $0.712$ | $0.286$ | $1.02$ | $0.40$ | $1.42$

We evaluated two approaches for assigning a second speaker. An heuristic that
considers the two closest speakers in time [25] and, based on [26], an
approach where the second most-likely speaker of the output of VB-HMM
diarization is used to provide the second label, but applied using x-vectors
as input frames instead of mel-frequency cepstral coefficients. Given the
current pipeline, obtaining the second label is quite straightforward as we
simply need to output the two most likely speakers for each frame.

Results comparing the OVD system and the oracle OVD are presented in Table 5,
together with a comparison of the methods for selecting the second speaker:
with VBx, with the heuristic or directly using the oracle labels. While the
second speaker from VBx provides a convenient and principled mechanism for
selecting the second speaker in overlap segments, we see that the performance
is similar or slightly worse than choosing the closest speaker in time.
However, choosing the second speaker according to the oracle labels would
still provide a notable gain as compared to any of these methods, showing that
there is room for improvement in terms of overlap handling.

Nevertheless, oracle OVD proves to be significantly better than the OVD
system. In this sense, there is far much more room for improvement on the
detection of overlap rather than on the handling. Still, it should be noted
that overlapped speech labels on the development set, like in any other
diarization dataset, are not perfect, as the labeling of these regions is a
very challenging task even for human annotators, and the precision that can be
achieved is hard to define.

Table 5: Diarization performance on the development set for VBx + reclustering (without overlapped speech handling) and after doing OVD and handling. Performance is shown for the OVD system and the oracle OVD and for each case, the two handling approaches and the choice of oracle speaker are evaluated. System | DER | Miss | FA | Spk. | JER
---|---|---|---|---|---
No handling | $4.30$ | $3.10$ | $0.47$ | $0.73$ | $19.81$
System + VBx 2nd | $4.02$ |  $2.36$666Note that if when reclustering the first and second most likely speakers from VBx or the first and second by oracle are merged, then there is no second speaker to select leading to an increase in miss. | $0.75$ | $0.91$ | $19.96$
System + heuristic | $4.00$ | $2.34$ | $0.75$ | $0.91$ | $19.80$
System + oracle | $3.56$ |  $2.36$666Note that if when reclustering the first and second most likely speakers from VBx or the first and second by oracle are merged, then there is no second speaker to select leading to an increase in miss. | $0.47$ | $0.73$ | $18.78$
Oracle + VBx 2nd | $2.80$ |  $0.98$666Note that if when reclustering the first and second most likely speakers from VBx or the first and second by oracle are merged, then there is no second speaker to select leading to an increase in miss. | $0.47$ | $1.35$ | $19.10$
Oracle + heuristic | $2.70$ | $0.92$ | $0.47$ | $1.31$ | $18.52$
Oracle + oracle | $2.20$ |  $1.00$666Note that if when reclustering the first and second most likely speakers from VBx or the first and second by oracle are merged, then there is no second speaker to select leading to an increase in miss. | $0.47$ | $0.73$ | $16.17$

###  2.8 Ablation study with oracle labels

Given that the VAD and OVD play a big role in the final performance of our
system, we decided to further study these two components.

Considering the VAD, we compare performance of our system using the full VAD
as described in section 2.2, the baseline VAD, the simple energy based VAD
with short pauses removed and the oracle VAD. Results are presented in Table
6. Here we can directly see how the VAD error (see table 4) influences DER
performance. The difference on VAD performance between the baseline and the
simple energy VAD (6.71 error) translates in almost 12% DER difference. The
extra improvement using the full VAD model (2.36 error reduction) translates
into only 1.39% DER improvement. Finally the oracle VAD (virtually 2.98 better
VAD total error) only brings 1.21% DER improvement. It is worth noting that
the boundaries of speech segments in the oracle labels are not perfect (for
this reason 0.25 s forgiveness collar is used in the challenge for computing
DER) and the performance gains given by better VAD reduce in terms of DER as
the VAD error approaches to zero.

Next, regarding the OVD we compare the OVD system and the oracle OVD and the
speaker label assignment when using also oracle VAD. Results show how the OVD
system provides only a small DER improvement while oracle OVD brings a large
gain, around 50% decrease in DER. The oracle assignment brings only an extra
15% decrease in error showing again that the main room for improvement is in
the overlap detection.

Table 6: VAD error and diarization performance on the development set for different combinations of VAD (‘O’ refers to oracle) and OVD (‘S’ refers to the system and ‘O’ refers to the oracle) with the heuristic. In all cases, VBx + reclustering system is used. System |  | VAD
---
error
DER | Miss | FA | Spk. | JER
Full VAD | $2.98$ | $4.30$ | $3.10$ | $0.47$ | $0.73$ | $19.81$
Baseline VAD | $12.05$ | $17.58$ | $11.22$ | $2.26$ | $4.10$ | $39.73$
Energy VAD | $5.34$ | $5.69$ | $2.99$ | $1.78$ | $0.92$ | $21.52$
O VAD | $0$ | $3.09$ | $2.30$ | $0$ | $0.79$ | $17.26$
O VAD + S OVD h | $0$ | $2.79$ | $1.55$ | $0.28$ | $0.96$ | $17.29$
O VAD + O OVD h | $0$ | $1.50$ | $0.08$ | $0$ | $1.42$ | $16.12$
O VAD + O OVD O | $0$ | $1.01$ | $0.22$ | $0$ | $0.79$ | $13.80$

##  3 CONCLUSIONS

In this paper we have proposed a system for performing diarization on the new
VoxConverse diarization corpus comprised by broadcast recordings. The pipeline
consists of voice activity detection, embedding extraction, VBx using AHC as
initialization, reclustering using per recording global embeddings and
overlapped speech detection and handling. We have analyzed the effect of each
step in the final performance and identified the aspects that need to be
addressed in the future to improve the performance further.

Our best submission on the challenge, corresponding to DER 4% and JER 19.80 on
the development set, obtained on the evaluation set DER 8.12% and JER 18.35
allowing us to obtain the second position in terms of DER and the first in
terms of JER. Analyzing the reasons for such differences in results between
development and evaluation remains for a post-evaluation stage once the
evaluation labels are released. Together with this publication, the most
relevant modules of our system are released to the public in a branch of [7].

## References

  * [1] G. Gravier, G. Adda, N. Paulson, M. Carré, A. Giraudel, and O. Galibert,  “The etape corpus for the evaluation of speech-based tv content processing in the french language,”  in Proceedings of LREC 2012, 2012.
  * [2] P. Bell, M. J. Gales, T. Hain, J. Kilgour, P. Lanchantin, X. Liu, A. McParland, S. Renals, O. Saz, M. Wester, et al.,  “The mgb challenge: Evaluating multi-genre broadcast media recognition,”  in 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU). IEEE, 2015, pp. 687–693.
  * [3] E. Lleida, A. Ortega, A. Miguel, V. Bazán-Gil, C. Pérez, M. Gómez, and A. de Prada,  “Albayzin 2018 evaluation: the iberspeech-rtve challenge on speech technologies for spanish broadcast media,”  Applied Sciences, vol. 9, no. 24, pp. 5412, 2019.
  * [4] N. Ryant, K. Church, C. Cieri, A. Cristia, J. Du, S. Ganapathy, and M. Liberman,  “The second dihard diarization challenge: Dataset, task, and baselines,”  arXiv preprint arXiv:1906.07839, 2019.
  * [5] J. S. Chung, J. Huh, A. Nagrani, T. Afouras, and A. Zisserman,  “Spot the conversation: speaker diarisation in the wild,”  arXiv preprint arXiv:2007.01216, 2020.
  * [6] J. S. Chung, A. Nagrani, and A. Zisserman,  “Voxceleb2: Deep speaker recognition,”  arXiv preprint arXiv:1806.05622, 2018.
  * [7] “VBHMM x-vectors Diarization (aka VBx),” https://github.com/BUTSpeechFIT/VBx/tree/v1.1_VoxConverse2020.
  * [8] “The NIST Rich Transcription 2009 (RT’09) evaluation,” http://www.itl.nist.gov/iad/mig/tests/rt/2009/docs/rt09-meeting-eval-plan-v2.pdf, 2009\.
  * [9] L. Sun, J. Du, C. Jiang, X. Zhang, S. He, B. Yin, and C.-H. Lee,  “Speaker diarization with enhancing speech for the first dihard challenge.,”  in Interspeech, 2018, pp. 2793–2797.
  * [10] T. Nakatani, T. Yoshioka, K. Kinoshita, M. Miyoshi, and B.-H. Juang,  “Speech dereverberation based on variance-normalized delayed linear prediction,”  IEEE Transactions on Audio, Speech, and Language Processing, vol. 18, no. 7, pp. 1717–1731, 2010.
  * [11] L. Drude, J. Heymann, C. Boeddeker, and R. Haeb-Umbach,  “Nara-wpe: A python package for weighted prediction error dereverberation in numpy and tensorflow for online and offline processing,”  in Speech Communication; 13th ITG-Symposium. VDE, 2018, pp. 1–5.
  * [12] F. Landini, S. Wang, M. Diez, L. Burget, P. Matějka, K. Žmolíková, L. Mošner, A. Silnova, O. Plchot, O. Novotnỳ, et al.,  “But system for the second dihard speech diarization challenge,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 6529–6533.
  * [13] J. Carletta, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, V. Karaiskos, W. Kraaij, M. Kronenthal, et al.,  “The ami meeting corpus: A pre-announcement,”  in International workshop on machine learning for multimodal interaction. Springer, 2005, pp. 28–39.
  * [14] A. Janin, D. Baron, J. Edwards, D. Ellis, D. Gelbart, N. Morgan, B. Peskin, T. Pfau, E. Shriberg, A. Stolcke, et al.,  “The ICSI meeting corpus,”  in 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP’03). IEEE, 2003, vol. 1, pp. I–I.
  * [15] S. Burger, V. MacLaren, and H. Yu,  “The ISL meeting corpus: The impact of meeting type on speech style,”  in Seventh International Conference on Spoken Language Processing, 2002.
  * [16] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, et al.,  “The kaldi speech recognition toolkit,”  in IEEE 2011 workshop on automatic speech recognition and understanding. IEEE Signal Processing Society, 2011.
  * [17] F. Hernandez, V. Nguyen, S. Ghannay, N. Tomashenko, and Y. Estève,  “TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation,”  in International Conference on Speech and Computer. Springer, 2018, pp. 198–208.
  * [18] K. He, X. Zhang, S. Ren, and J. Sun,  “Deep residual learning for image recognition,”  in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778.
  * [19] D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur,  “X-vectors: Robust dnn embeddings for speaker recognition,”  in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018, pp. 5329–5333.
  * [20] H. Wang, Y. Wang, Z. Zhou, X. Ji, D. Gong, J. Zhou, Z. Li, and W. Liu,  “Cosface: Large margin cosine loss for deep face recognition,”  in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 5265–5274.
  * [21] P. Kenny,  “Bayesian Speaker Verification with Heavy-Tailed Priors,”  in in Proceedings of Odyssey, June 2010.
  * [22] D. Garcia-Romero and C. Y. Espy-Wilson,  “Analysis of i-vector length normalization in speaker recognition systems,”  in Proceedings of Interspeech 2011, 2011.
  * [23] M. Diez, L. Burget, F. Landini, S. Wang, and H. Černockỳ,  “Optimizing bayesian hmm based x-vector clustering for the second dihard speech diarization challenge,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 6519–6523.
  * [24] H. Bredin, R. Yin, J. M. Coria, G. Gelly, P. Korshunov, M. Lavechin, D. Fustes, H. Titeux, W. Bouaziz, and M.-P. Gill,  “pyannote.audio: neural building blocks for speaker diarization,”  in ICASSP 2020, IEEE International Conference on Acoustics, Speech, and Signal Processing, Barcelona, Spain, May 2020.
  * [25] S. Otterson and M. Ostendorf,  “Efficient use of overlap information in speaker diarization,”  in 2007 IEEE Workshop on Automatic Speech Recognition & Understanding (ASRU). IEEE, 2007, pp. 683–686.
  * [26] L. Bullock, H. Bredin, and L. P. Garcia-Perera,  “Overlap-aware diarization: resegmentation using neural end-to-end overlapped speech detection,”  in ICASSP 2020, IEEE International Conference on Acoustics, Speech, and Signal Processing, Barcelona, Spain, May 2020.