# Target Speech Diarization with Multimodal Prompts

Yidi Jiang, , Ruijie Tao, , Zhengyang Chen, ,
Yanmin Qian, and Haizhou Li Yidi Jiang and Ruijie Tao are with the Department
of Electrical and Computer Engineering, National University of Singapore,
Singapore 117583<EMAIL_ADDRESS>ruijie@nus.edu.sg).
(Corresponding author: Ruijie Tao.) Zhengyang Chen and Yanmin Qian are with
the Shanghai Jiao Tong University, Shanghai 200240, China (e-mail:
<EMAIL_ADDRESS>yanminqian@sjtu.edu.cn). Haizhou Li is with the
Shenzhen Research Institute of Big Data, School of Data Science, The Chinese
University of Hong Kong, Shenzhen 518172, China, and also with the Kriston AI,
Xiamen 361026, China (e-mail: haizhouli@cuhk.edu.cn).

###### Abstract

Traditional speaker diarization seeks to detect “who spoke when” according to
speaker characteristics. Extending to target speech diarization, we detect
“when target event occurs” according to the semantic characteristics of
speech. We propose a novel Multimodal Target Speech Diarization (MM-TSD)
framework, which accommodates diverse and multi-modal prompts to specify
target events in a flexible and user-friendly manner, including semantic
language description, pre-enrolled speech, pre-registered face image, and
audio-language logical prompts. We further propose a voice-face aligner module
to project human voice and face representation into a shared space. We develop
a multi-modal dataset based on VoxCeleb2 for MM-TSD training and evaluation.
Additionally, we conduct comparative analysis and ablation studies for each
category of prompts to validate the efficacy of each component in the proposed
framework. Furthermore, our framework demonstrates versatility in performing
various signal processing tasks, including speaker diarization and overlap
speech detection, using task-specific prompts. MM-TSD achieves robust and
comparable performance as a unified system compared to specialized models.
Moreover, MM-TSD shows capability to handle complex conversations for real-
world dataset.

###### Index Terms:

Target speech diarization, speaker diarization, natural language processing,
voice-face alignment.

##  I Introduction

Humans have the ability to selectively attend to a specific sound source in a
complex acoustic environment, that is commonly referred to as the cocktail
party effect [1]. Benefit from this remarkable auditory attention mechanism,
human can effectively focus on a particular speaker of interest [2, 3] since
each speaker has the unique voice characteristic. The speaker diarization task
aims to segment multi-talker speech based on speaker identities and determines
“who spoke when” [4, 5, 6], which serves as a front-end for various downstream
speech-related tasks.

In addition to speaker identity [7, 8], there is interest in other semantic
aspects of human speech (referred as “target events”) [9, 10, 11], such as
male/female speech, multi-talker speech mixture, or the speech of a keynote
speaker who speaks the most in a meeting. This indicates the need for
determining target speech in a comprehensive and multi-dimensional manner.
Therefore, we proposed a new paradigm termed “target speech diarization”,
which aims to identify “when target event occurred” guided by prompts
specifying target events. Our previous work [12] provided a feasible method
that leverages prompt vectors to offer conditional information for specific
target events. By switching between different prompt vectors, the framework
can identify the corresponding event regions within an audio signal.

Figure 1: The illustration of four types of prompts supported by our
Multimodal Target Speech Diarization (MM-TSD) framework. The unified target
speech diarization model can accommodate multi-modal and diverse prompts,
including semantic language description, pre-enrolled speech and pre-
registered face of the target speaker and the audio-text logical controller.
Our framework then detects the activity regions of the target speech specified
by the prompt.

In real-world scenarios, the utilization of prompt vectors is constrained and
not user-friendly enough. Humans perceive and interact with the world through
multiple modalities, including linguistic, auditory, and visual cues. For
instance, in a meeting scenario, we rely on language instructions, spoken
speech, facial expressions and gestures to communicate. Similarly, prompts
specifying target events can exist in various multi-modal formats such as
language instructions, pre-enrolled speech and pre-registered faces, or even
the combination of them. However, it remains a challenge how to integrate
multi-modal prompts for diverse scenarios into a system. We hypothesize that
it’s feasible to project multi-modal prompts describing the semantic
characteristics of speech into a shared semantic space. This projection would
facilitate the processing of multi-modal information within a unified model
and interaction between speech input and multi-modal prompts. Therefore, our
objective is to develop a unified framework that allows flexible user
interaction through a range of multi-modal prompts, allowing users to identify
target events across multi-dimensional semantic aspects according to their
requirements and preferences.

In “target speech diarization” task, one uses pre-enrolled speech as a prompt
to specify a desired target event, just like the pre-enrolled speech for a
target speaker in target speech voice activity detection (TS-VAD) [5, 4],
where speech regions that correspond to a reference speaker are detected.

However, it is challenging to describe complex concepts with pre-enrolled
speech, such as overlapping speech or the most talkative speaker. Moreover,
pre-enrolled speech is not always available. Natural language, as the most
natural way of human communication, is commonly used to express complex
concepts. Therefore, it serves as a natural choice of human prompt. Prior
works have focused on language-queried audio source separation [13, 14] and
text-guided target speaker extraction [15], which utilize the natural language
to achieve conditional separation and extraction functions. Nevertheless,
building a target speech diarization system that can effectively handle the
complex and various natural language expressions poses challenges in modeling
intricate audio-text interactions. For instance, the same speech event can be
described through various text prompts, such as “male speech” and “the voice
of the man”. The system must be able to correlate these diverse text prompts
with the same speech event and subsequently identify the corresponding regions
within the audio input.

Apart from using natural language to specify a particular speech event, recent
studies [16, 17, 18] have demonstrated the effectiveness of incorporating lip
movements to detect speech of a particular speaker through audio-visual
synchronization. While these methods are always limited to scenarios with the
high frame rate video. Furthermore, many works [19, 20, 21, 22] have also
explored the usage of still face images and verified the association between
voice representations and face appearance due to shared latent factors, such
as age, gender and nationality/accent. In the field of audio-visual speech
processing, FaceFilter [23] has explored audio-visual speech separation
conditioned on a still face image of a target speaker. In this work, we make
the first attempt for face-based target speech diarization using static face
image as prompt to specify the desired speaker.

As previously discussed, while single-modal prompts can identify specific
speech event in audio signals, real-world communication often involves
combining multiple modalities to convey concepts. We also explore the
interactive multi-modal prompts for complex logical operations. For example,
in certain scenarios, there is a need to filter out the target speaker’s
voice. By inputting the command “Exclude the pre-enrolled speaker’s speech”
along with a reference speech, we can identify segments excluding the enrolled
speaker. In this study, we explore the interaction between audio and text
prompts, where text commands serve as a logical controller to determine
whether detect or exclude the pre-enrolled speaker’s voice. With the
integration of “exclusion” related commands, our framework functions as a “NOT
Gate”, allowing precise exclusion of the target speaker’s voice when
necessary.

As shown in Figure 1, in this work, our primary objective is to establish a
unified target speech diarization model capable of accommodating multi-modal
prompts to specify various target events. These prompts include semantic
language description, pre-enrolled speech, pre-registered face image, and
audio-language logical prompts. We propose Multimodal Target Speech
Diarization (MM-TSD) framework, which contains the modality-specific prompt
encoders and modality-agnostic Transformer encoder-decoder for handling the
diverse application scenarios.

This work is an extension of our previous study, which was presented at ICASSP
[12]. Our contributions in this work are as follows:

  * •

To the best of our knowledge, the proposed MM-TSD is the first attempt for
end-to-end target speech diarization, supporting audio, visual, textual and
audio-text multi-modal prompts to specify the target speech events. This work
sets a reference benchmark and provides valuable insights into multi-modal
prompt-guided target speech processing.

  * •

We introduced the use of static facial cues in diarization-related tasks and
proposed a voice-face aligner module to establish correspondence between human
face and voice biometrics.

  * •

We evaluated MM-TSD framework across various modalities of prompts and
semantic attributes on both simulated and real-world datasets to show its
effectiveness in detecting the prompt-specified target events. The evaluations
further confirm our hypothesis that MM-TSD can project the multi-modal prompts
and speech input into a shared space within a unified framework.

##  II Related Work

###  II-A Speaker Diarization

Speaker diarization seeks to delimit the boundaries of speaker turns in a
multi-talker speech according to speaker characteristics, i.e. voiceprint.
Taking advantage of speaker-specific information [17], target speaker voice
activity detection (TS-VAD) [5] employs speaker embeddings of speakers during
the diarization process.

To obtain the voiceprint of all speakers in the conversation, TS-VAD system
employs an additional clustering-based diarization system to identify the
single-speaker segments for each individual. Subsequently, a pre-trained
speaker embedding extractor is utilized to obtain speaker embeddings. TS-VAD
applies these speaker embeddings as the references to guide the diarization
process and detect the speaking status of each speaker [4, 24]. Inspired by
the success of TS-VAD, our framework utilizes the pre-enrolled speech
utterance as the prompt to detect the speaking activities of the target event.

###  II-B Language-Queried Audio Processing

In recent years, audio-language processing has emerged as a novel research
domain. Introducing text modality into speech tasks provides precise
descriptions and guidance, offering significant value across various
application scenarios in a user-friendly manner. For instance, [13, 14]
propose language-guided audio source separation, which aims to isolate
specific sources from audio mixtures using natural language queries.
Furthermore, language-based audio retrieval [25, 26, 27] has proven efficacy
for multimedia content retrieval and sound analysis. In the context of our
proposed target speech diarization, the text modality can clearly define the
target speech event for flexible user interaction. Motivated by that, we would
like to construct a TSD system that can respond to diverse language-based
queries.

###  II-C Voice-Face Biometric Matching

In addition to text and audio, humans also rely on visual cues to perceive and
interact with the world, which motivates the incorporation of visual cues into
the TSD process. A straightforward approach is to employ lip movements of a
particular speaker as visual cues [16, 28]. While such audio-visual models
have demonstrated remarkable outcomes, they often require high-quality video
data and high computational resource, which may not always be available in
real-world application.

Recognizing this limitation, employing a single static face image as a visual
cue for speech processing is an alternative [23]. Each speaker has distinct
voice characteristics, including the vocal tract shape, pitch, and prosody
variation, as well as unique facial landmarks. Recent research demonstrated
that voice representations can exhibit correlations with face appearance due
to shared latent factors such as age, gender, and ethnicity/accent [29, 30].
This connection can be leveraged in target speech diarization task to specify
the target event, i.e., the speech corresponding to a given static face image.
While this connection may not be robust among speakers of the same gender,
nationality or age range, the exploration for face-based speech processing
still remains meaningful. From a practical standpoint, users’ profile images
are always accessible on various mobile devices, social networks, and company
groupware, enhancing the accessibility of such audio-visual solutions. Our
research represents the first investigation to utilize the static face image
as the visual prompt in target speech diarization, leveraging the inherent
voice-face correlation.

##  III Task Formulation: Target Speech Diarization

In this section, we outline the formulation of our proposed target speech
diarization task. First, we introduced two key concepts for our task: semantic
attribute and semantic value [12]. Semantic attributes contain a set of speech
properties such as speaker identity and gender, which represent the criteria
of demarcating speech segments. Each semantic attribute takes on one or
multiple semantic values associated with specific events. For examples, in
speaker diarization task, speaker identity is the semantic attribute. The
specific speaker ID is semantic value and his/her speaking region is its
aligned speech event.

In the target speech diarization system, it simultaneously takes audio and the
prompt that specify the target speech event as inputs and outputs the
corresponding target event regions. For example, when provided with pre-
enrolled speech (prompt) of “Speaker A” (semantic value), the framework will
output the speaking regions of “Speaker A” (target event).

The core of a TSD system lies in the prompt, which specifies the target event
and guides the TSD process. As depicted in Figure 1, prompts are available in
various formats for different application scenarios.

Figure 2: Our MM-TSD framework takes an audio signal and a switchable multi-
modal prompt as inputs, to output frame-wise binary classification of the
prompt-specified speech event. It accommodates diverse prompt types such as
semantic language descriptions, pre-enrolled speech, pre-registered face
images or the combination of audio-text logical prompts to specify the target
event. This framework comprises a speech encoder, three modality-specific
prompt encoders and a Transformer encoder-decoder structure. Then a dot
product $\bigotimes$ is applied between the encoder and decoder outputs,
followed by a sigmoid operation $\sigma$ to calculate the target event
occurrence probability at each frame.

####  III-1 TSD with Semantic Description

Humans perceive audio signals based on the distinguishing semantic
characteristics, such as female speech or non-overlap speech in the audio
signal. This scenario enables users to incorporate such perceptual cues as
text-based semantic descriptions to guide target speech diarization. For
example, when semantic attribute is gender, there are two semantic values:
female and male. The prompt format is natural language, for example, “please
detect the female speech regions”. Then the target event is the female speech
segments.

####  III-2 TSD with Pre-enrolled Speech

In this scenario, the semantic attribute is speaker identity, the semantic
value is a specific speaker ID, the prompt format is the pre-enrolled speech
of the target speaker, and the target event is the speaking regions of the
target speaker. The system aims to detect the speaking activity of each
prompt-specified speaker.

####  III-3 TSD with Pre-registered Face

Similarly, in this scenario, the semantic attribute is face identity, and the
semantic value is still a specific speaker ID. The prompt format is the pre-
registered face to specify the target speaker, and the target event is the
speaking regions of the specified speaker. This scenario offers users the
capability to identify speaking regions of interest by providing the system
with the pre-registered face of the target speaker.

####  III-4 TSD with Audio-Language Logical Controller

In this scenario, there are two related attributes: “included identity” and
“excluded identity”. The prompts consist of natural language serving as a
logical controller and pre-enrolled speech to specify the target speaker. For
instance, when the semantic attribute and value are “excluded identity” and
“Speaker A”, respectively, the target event is the regions where the voice of
“Speaker A” doesn’t occur. This scenario offers users the flexibility to
decide whether to detect or exclude the pre-enrolled speaker’s active regions
within the audio mixture.

##  IV Multimodal Target Speech Diarization

As illustrated in Figure 2, the proposed MM-TSD system consists of a speech
encoder, multi-modal prompt encoders and a Transformer encoder-decoder
structure. Our framework is designed to flexibly switch and accommodate one or
multiple prompts simultaneously, outputting the associated target event(s)
regions accordingly.

Firstly, the speech encoder is employed to extract the audio feature sequence
from the speech input, denoted as $F^{a}$. Concurrently, a prompt which lies
in various modalities is used to specify the target event. To process input
from different modalities, we employ three modality-specific prompt encoders:
a text prompt encoder, an audio prompt encoder and a visual prompt encoder.
These encoders convert multi-modal prompts (text command, pre-enrolled speech
or pre-registered face) into corresponding prompt embeddings $E^{T}$, $E^{A}$
and $E^{V}$, respectively. Each prompt embedding is characterized by a
dimension $D$. To ensure reliable modality representations, we apply pre-
training techniques for each modality encoder.

Then, to align the prompt-specified event with the input speech, the
Transformer encoder-decoder takes $F^{a}$ and prompt embedding $E$ as inputs
and outputs the prediction sequence $\hat{\mathrm{Y}}\in(0,1)^{1\times T}$,
where $T$ represents the number of frames. The values of $\hat{\mathrm{Y}}$
denote the target event occurrence probability at each frame. Specifically,
the Transformer encoder receives $F^{a}$ and outputs the frame-level speech
representation
$F^{e}=[F^{e}_{1},F^{e}_{2},\ldots,F^{e}_{T}]\in\mathbb{R}^{T\times D}$. The
Transformer decoder takes prompt embeddings $E$ and $F^{e}$ as inputs, and
outputs $F^{d}$ with dimension $D$. Finally, we performed a dot product
operation between the decoder output $F^{d}$ and the encoder output $F^{e}$
and applied a sigmoid operation to get the prediction sequence
$\hat{\mathrm{Y}}$.

###  IV-A Speech Encoder

To obtain the robust representation of the input speech, in our framework, we
employ a pre-trained WavLM encoder [31] as the speech encoder to obtain speech
representations $F^{a}$. The WavLM encoder was designed to learn universal
speech representations from vast amounts of unlabeled speech data, ensuring
the universality and robustness of the frame-level audio representations. With
consideration for the trade-off between computational efficiency and speech
information, we utilized its convolutional feature encoder and the first three
layers of the Transformer encoder, freezing them during our training process.

###  IV-B Text Prompt Encoder with LoRA

The goal of the text prompt encoder is to ensure that our framework can
accommodate diverse textual descriptions for each target speech event.
Specially, the text encoder is designed to map various sentence descriptions,
which specify the same event, into a similar embedding space. To achieve this,
we utilize a Pre-trained Language Model (PLM) as the text prompt encoder to
extract prompt embedding. Additionally, we explore a lightweight fine-tuning
approach to achieve the training efficiency and adaptation.

As decipted in Figure 3, the text prompt is firstly tokenized using the BERT
[32] Tokenizer, converting the textual command to tokens. The tokens are then
fed into the PLM text encoder. To optimize training efficiency, we employ the
DistilBERT model [33] as our PLM text encoder. DistilBERT is a fast, cost-
effective, and lightweight Transformer model derived from the distillation
process of the BERT base model, which use offers fewer parameters but preserve
over 95% of BERT’s performance
111https://huggingface.co/docs/transformers/en/model_doc/distilbert.

Figure 3: Text prompt encoding. The textual prompt is first processed by a
tokenizer to generate word tokens, including a “[CLS]” token at the beginning.
We utilize a pre-trained DistilBERT encoder with Low-Rank Adaptation (LoRA) to
derive the sentence embedding. The feature of the “[CLS]” token is then used
as the prompt embedding $E$.

The DistilBERT model begins with an embedding layer that transforms tokens
into the token embeddings, incorporating position embeddings, and proceeds
through six Transformer encoder layers to generate the sentence’s hidden state
embedding. After that, we regard the “[CLS]” token embedding as a pooled
embedding with condensed semantic information, and pass it through a linear
layer followed by ReLU and dropout. Then, we use another linear layer to adapt
the embedding to the dimension $D$, serving as our text prompt embedding
$E^{T}$.

Furthermore, to adapt the PLM to our textual prompt semantic space without
conducting full fine-tuning on the PLM text encoder, we adopt the parameter-
efficient Low-Rank Adaptation (LoRA) technique [34]. This approach involves
incorporating trainable rank decomposition matrices into each layer of the
Transformer architecture. Consequently, PLM adaptation can be achieved with a
reduced number of trainable parameters. Specially, the LoRA structure is
incorporated into query and value linear layers in each multi-head attention
of every Transformer layer.

###  IV-C Audio Prompt Encoder

The audio prompt is the pre-enrolled speech of the target speaker. To
guarantee robust performance, we leverage a pre-trained ECAPA-TDNN [7] speaker
recognition model to obtain the target speaker embedding as audio prompt
embedding $E^{A}$. The ECAPA-TDNN model has demonstrated reliable performance
in speaker recognition tasks.

The ECAPA-TDNN model employs emphasized channel attention to selectively focus
on critical parts of the speech signal, propagating that information through
the network and aggregating it to make a final decision. From the variable
lengths of input utterances, the output speaker embedding has the fixed
dimension $D$. We freeze the parameters of this module in our framework, since
our purpose is to obtain robust embedding for the target speaker.

###  IV-D Visual Prompt Encoder with Voice-Face Aligner

Previous works have demonstrated a correlation between the facial appearance
and voice characteristics [19, 20, 21, 22]. Building on this cross-modal
association, we investigate using pre-registered face images of the target
individual as prompts to identify their speech regions. In MM-TSD, we employ a
pre-trained ResNet50 model [35] as the face prompt encoder, known for its
robust face recognition performance trained on large-scale face datasets.
However, the face embeddings extracted from the pre-trained model may exist in
a mismatched space with input audio representations.

To address this discrepancy and leverage the intrinsic associations between
human face and voice biometrics, we incorporate a novel voice-face aligner
module with the additional “Aligner Training Stage”, as shown in Figure 4. The
voice-face aligner is designed to learn associations between audio and visual
inputs, encompassing general identity features (such as gender, age, and
ethnicity) and appearance features (such as prominent facial attributes like
big nose, chubby cheeks, or double chin). Its goal is to establish
correspondence between voice-face identity pairs and bridge the modality gap
between voice and face embeddings.

Figure 4: Voice-face alignment involves inputs from speech segments and face
images belonging to the same individual, which are denoted within dashed
boxes. We utilize pre-trained ECAPA-TDNN as the speaker encoder and ResNet50
as the face encoder to extract respective embeddings from the speech segment
and face image. Following this, a voice-face aligner is employed to match face
identity with voice characteristics in a shared embedding space. During the
aligner training phase, the voice-face aligner is trained using Mean Squared
Error (MSE) loss. In the subsequent MM-TSD training phase, the visual prompt
encoder and voice-face aligner are both frozen to derive the visual prompt
embedding $E$.

The aligner training pipeline is illustrated in Figure 4. Each training data
sample consists of the facial image and the corresponding reference speech
from the same individual, which offer biometric information from diverse
perspectives. Voice embeddings are obtained using a pre-trained speaker
encoder, ECAPA-TDNN, similar to the audio prompt encoder introduced in Section
IV-C. The face embedding is extracted from a pre-trained face encoder and a
trainable voice-face aligner. This voice-face aligner is trained using Mean
Squared Error (MSE) loss, which quantifies the probability that the voice and
face embeddings belong to the same person.

Finally, after the voice-face “Aligner Training Stage”, we fix the ResNet50
face encoder and voice-face aligner for the entire “MM-TSD Training Stage”.
These two modules cooperate with each other to generate aligned face
embeddings, serving as our visual prompt embeddings $E^{V}$.

Figure 5: The encoder receives the speech embedding $F^{a}$, which is
extracted from the speech encoder, and produces a frame-level speech
representation $F^{e}$. The decoder utilizes the prompt embedding $E$ as the
query within a cross-attention mechanism, with $F^{e}$ serving as both the key
and value. This setup enables precise alignment and interaction between the
speech embedding $F^{a}$ and prompt embedding $E$ to detect the prompt-
specified target event activities within the speech signal. $\bigotimes$
denotes the dot product operation between transformer encoder and decoder
ouputs.

###  IV-E Transformer Encoder-Decoder

After obtaining the speech representation $F^{a}$ from the speech encoder and
the prompt embedding $E$ from the prompt encoders, we feed them into a
Transformer [36] encoder-decoder architecture to predict the MM-TSD output
sequence. This design leverages the self-attention and cross-attention
mechanisms to capture intricate temporal patterns in the audio data and align
relevant information with the prompt embeddings which serve as query.

Within the Transformer encoder, self-attention enables interaction among the
learnt speech representations, enhancing the overall quality of frame-level
speech representations. As shown in Figure 5, the Transformer encoder produces
encoder memory $F^{e}$ as frame-level speech representation. Both the prompt
embedding $E\in\mathbb{R}^{1\times D}$ and the Transformer encoder memory
$F^{e}\in\mathbb{R}^{T\times D}$ are then fed into the Transformer decoder.
Here, the prompt embedding $E$ serves as the query in the cross-attention
structure, while the encoder memory $F^{e}$ serves as the key and value. The
cross-attention module within the Transformer decoder enables prompt
embeddings to attend to all frame-level speech representations, ensuring that
the resulting decoder outputs capture the most relevant information about the
prompt-specified target event.

With the Transformer encoder memory and decoder output, we can calculate the
posterior probability that each frame belongs to the prompt-specified event
through a simple dot product operation:

$\hat{Y}=\sigma(F^{d}{F^{e}}^{\top})\in(0,1)^{1\times T}$ (1)

where the $\sigma$ symbol corresponds to the element-wise sigmoid function.
MM-TSD benefits from this Transformer encoder-decoder to accurately identify
and detect target event regions based on the prompts, achieving a robust and
adaptable solution for our task.

###  IV-F Loss Function

The learning targets of our framework are frame-wise binary ground truth
labels $Y\in\\{0,1\\}^{1\times T}$ of the target event. We utilized binary
cross-entropy loss to train our model, as defined in Equation 2. $\hat{y}_{t}$
and $y_{t}$ represent the predicted and ground-truth labels of the specific
target event for the $t^{th}$ audio frame, where $t\in[1,T]$. The loss
function is designed to minimize the difference between predicted and ground-
truth labels, encouraging our model to accurately detect target event
activities.

$\mathcal{L}=-\frac{1}{T}\sum_{t=1}^{T}[(y_{t}\cdot
log\hat{y}_{t})+(1-y_{t})\cdot log(1-\hat{y}_{t})]$ (2)

##  V MM-TSD Benchmark

In this section, we establish a comprehensive benchmark for the proposed
target speech diarization task. It includes the dataset corpus design, text
prompt formulation, and evaluation metrics. This benchmark serves as the basis
for our experiments and provides a standard reference for future studies to
ensure fair comparisons.

###  V-A Data Corpus

####  V-A1 Simulation Dataset for MM-TSD Training

Since real-world speech datasets cannot meet all the required ground-truth
labels according to application scenarios introduced in Section III, we
followed the recipe222https://github.com/BUTSpeechFIT/EEND_dataprep/ proposed
in [37] to generate simulation datasets for MM-TSD training from VoxCeleb2
[38] which is an audio-visual dataset derived from YouTube interviews. In
[37], the authors proposed a method that leverages the statistics from real
recordings to guide the synthesis of simulated data. To create datasets that
closely resemble real-world conversations, we utilized conversation statistics
from the DIHARD II development set [39] to generate 1000 hours of audio for
each MM-2spk, MM-3spk, and MM-4spk dataset to simulate the condition with
different number of speakers as details shown in Table I.

To showcase the generalization capabilities of our system, we have devised
both “Seen-Heard” and “Unseen-Unheard” test sets. The “Seen-Heard” set
comprises speakers present in the training data, while the “Unseen-unheard”
set consists of entirely new speakers. After filtering the heavily noisy and
unavailable videos, we select 5,702 speakers in VoxCeleb2 for training
purposes. Additionally, we reserve 49 speakers for unseen validation and
another 65 speakers for the unseen test set. This setup allows us to assess
our system’s performance in scenarios where it encounters entirely new
speakers.

####  V-A2 Simulation Dataset for MM-TSD with Audio Prompt Analysis

To demonstrate the effectiveness of MM-TSD with audio prompts, we conduct
comparison experiments with state-of-the-art speaker diarization systems. It
follows the traditional two-stage speaker diarization training process: pre-
training stage with the simulated dataset and adaptation stage with the real-
world dataset. To ensure a fair comparison, we followed the simulation
configuration and method outlined in [6] to generate two subsets: Audio-2spk
and Audio-3spk, with the statistics from Part1 of the CALLHOME dataset. As
shown in Table I, Audio-2spk and Audio-3spk has 2481 hours and 4226 hours
training data, respectively, featuring utterances with 2 and 3 speakers each.
The utterances in each subset have a fixed number of speakers.

TABLE I: Simulated dataset configuration. # Spk and # Utt represent the number of speakers and utterances, respectively. Ovl.(%) corresponds to the overlap ratio. Dataset |  Real-world Data Statistic |  Split |  # Spk |  # Utt |  Duration (hrs)
---|---|---|---|---|---
MM-2spk |  DIHARD II dev |  Train |  2 |  14,361 |  1,000
|  |  Test |  2 |  145 |  11
MM-3spk |  DIHARD II dev |  Train |  3 |  9,752 |  1,000
|  |  Test |  3 |  96 |  11
MM-4spk |  DIHARD II dev |  Train |  4 |  7,472 |  1,000
|  |  Test |  4 |  71 |  10
Audio-2spk |  CALLHOME (Part1 2spk) |  Train |  2 |  24,343 |  2,481
|  |  Test |  2 |  118 |  12
Audio-3spk |  CALLHOME (Part1 3spk) |  Train |  3 |  29,297 |  4,226
|  |  Test |  3 |  86 |  12

####  V-A3 Real-world Dataset

The datasets with real recordings used in our experiments are presented in
Table II. We employed the CALLHOME [40] dataset for analyzing MM-TSD with
audio prompts and the DIHARD II [39] dataset for analyzing MM-TSD with text
prompts.

The CALLHOME dataset is divided into two parts according to the kaldi recipe
333https://github.com/kaldi-asr/kaldi/tree/master/egs/callhome_diarization/v2.
Part 1 is used for model adaptation, while Part 2 is used for evaluation. We
selected the best-performing models trained on the Audio-2spk and Audio-3spk
Train sets, which were evaluated on the corresponding Test sets. Then we
employed the model for finetuning adaption on CALLHOME Part 1 subsets for 2
speaker and 3 speakers, respectively. For the DIHARD II dataset, we conduct
model adaptation on the Dev part and evaluate on the Test part. The CALLHOME
dataset comprises 8kHZ telephone-channel recordings, and the DIHARD II dataset
contains 16kHZ recordings from a diverse range of sources. To simplify the
training setup, we upsampled the CALLHOME dataset to 16kHZ in our experiments.

TABLE II: Real dataset configuration. # Spk and # Utt represent the number of speakers and utterances, respectively. Ovl.(%) corresponds to the overlap ratio. The numbers in the Duration column represents the minimum duration/maximum duration/average duration of each utterance.  Dataset |  Split |  # Spk |  # Utt |  Ovl. (%) |  Duration (hrs)
---|---|---|---|---|---
CALLHOME-2spk [40] |  Part 1 |  2 |  155 |  14.0 |  0.86/2.21/1.23
|  Part 2 |  2 |  148 |  13.1 |  0.88/2.23/1.20
CALLHOME-3spk [40] |  Part 1 |  3 |  61 |  19.6 |  0.95/6.35/2.07
|  Part 2 |  3 |  74 |  17.0 |  0.77/8.21/2.42
DIHARD II [39] |  Dev |  1-10 |  192 |  9.8 |  0.45/11.62/7.44
|  Test |  1-9 |  194 |  8.9 |  0.63/13.50/6.96

###  V-B Text Prompt Generation

In the first scenario “TSD with Semantic Description” referred to Section
III-1, we take three semantic attributes as examples in our work, denoted as
Gender, Speaker counter and Keynote speaker. The gender attribute contains two
values, female and male, which can guide the system to output the gender-
specific event regions. Speaker counter attribute identifies the number of
concurrent speakers at each frame and contains three event values: non-speech,
single-speaker speech, and overlapped speech. Keynote speaker attribute
focuses on identifying the keynote speaker. It contains one event value to
represent the person who talks most.

In practice, human’s descriptions of an target event speech are often diverse.
To mimic the real-world scenario, for each target event, we first prepare a
single command template. Then each template will undergo rephrasing and
expansion through ChatGPT-4, resulting in the generation of 50 distinct text
commands. To be precise, we utilize an 80%/10%/10% partitioning method that
ensures non-overlapped training/validation/test sets. The text commands
associated with each target event in the testing set remain unseen for those
of the training process to ensure the generalization ability of our text
encoder and our framework.

We show the ChatGPT command for “male speech event” as an example.

ChatGPT command:  You are asked to come up with 50 diverse instructions
rephrased and expanded from the template “Please detect the regions that male
speech occurs in the audio.” Here are the requirements: 1\. These instructions
should be to instruct someone to identify the target event regions. 2\. Try
not to repeat the verb for each instruction to maximize diversity. 3\. The
type of instructions should be diverse. 4\. The instructions should be oral
English. List of instructions:

Our generation method yields 50 instructions and some examples are shown.

Male Speech:
1\. I need you to identify areas in this recording where male voices are
present. 2\. Your task is to find and label the instances of male dialogue in
this recording. 3\. Your objective is to identify the segments where men are
speaking in this audio. 4\. Can you mark out the sections where men’s voices
appear in this audio track? 5\. Please trace the intervals in this sound clip
featuring speech from a male.

###  V-C Evaluation Metrics

The output of the target speech diarization system focuses on identifying each
target event regions rather than all speakers’ activities like traditional
diarization systems. Therefore, as a new task, it’s not appropriate to use the
traditional speaker diarization metric. Thus, we primarily employed three
metrics: accurate precision (AP), area under the receiver operating
characteristic (AUC), and equal error rate (EER) based on the implementation
from scikit-learn package.

Moreover, when the pre-enrolled speech of all speakers are provided, our
system is functioned as the traditional speaker diarization. In this scenario,
we report the diarization error rate (DER) to show our effectiveness compared
with other SOTA speaker diarization systems.

##  VI Experimental Setup

###  VI-A Implementation Details

####  VI-A1 Training and Inference Details

The proposed MM-STD framework was implemented using PyTorch and optimized with
the Adam optimizer. We set the initial learning rate to $10^{-4}$ and decrease
it by 5% for each epoch.

To achieve a multi-task unified model, our study utilized the parallel
characteristic of the Transformer decoder structure and adopted a multi-task
training strategy. To fully explore all prompt-aligned input-label pairs, we
provided all events’ prompts for each utterance. This allowed our multi-task
training model to accommodate a wide range of prompts during the evaluation
phase. It is worth noting that to achieve unified yet independent multi-task
learning, we applied an attention mask for Transformer decoder. This ensured
that diverse prompt embeddings remain independent, with the exception of
prompts requiring text and audio interaction, such as prompts under “TSD with
Audio-language Logical Controller” scenario as introduced in Section III-4.

####  VI-A2 Model Details

The Pre-trained Language Model (PLM) consists of 6 Transformer encoder blocks,
each with 12 attention heads and 768 hidden dimensions. Both the dimension $D$
of audio feature $F_{a}$ and prompt embeddings $E$ were set to 192. The pre-
trained audio prompt encoder ECAPA-TDNN is trained on VoxCeleb2 dataset. The
pre-trained face encoder ResNet50 is pre-trained on the Glint360K dataset
[41]. The voice-face aligner module is composed of a 4-layer multi-layer
perceptron (MLP). Each layer consists of a linear layer followed by a Gaussian
error linear unit (GeLU). The output dimensions of each layer are 1024, 1024,
256, and 512, respectively. For both Transformer encoder and decoder
structure, 4-layer Transformer with 8 attention heads was applied.

###  VI-B Data Augmentation

During training, we perform speech and face augmentation for audio and visual
prompts, to improve the diversity of training samples, thus the robustness of
audio and visual prompt embedding.

####  VI-B1 Speech Augmentation

We apply an online augmentation strategy with two datasets: the RIR dataset
[42] and the MUSAN dataset [43]. The RIR dataset contains room impulse
responses that can be used to simulate the reverberation effects via
convolution. These effects occur due to signal reflections bouncing off
surfaces such as walls, floor, and other objects within an acoustic enclosure.
Meanwhile, the MUSAN dataset [43] contains a variety of ambient sounds,
including nature noises (such as the sounds from train, thunder, rain),
background music (instrument or singing) and babble (multi-speaker talking
simultaneously).

####  VI-B2 Face Augmentation

Facial images are usually distracted by non-identity information, such as
colour, background, and image layout. A well-designed face augmentation
approach can assist the encoder in capturing distinctive facial features more
effectively. Firstly, we align all the faces with the detected landmarks
during pre-processing [44] since the unaligned faces in the training set make
recognition harder [45]. We then reshape the face image into
3$\times$112$\times$112, and apply the random horizontal flip with probability
0.5. Finally, we apply Gaussian blur techniques with kernel size 5$\sim$9 and
sigma 0.1$\sim$5, and randomly convert image to gray scale with a probability
of 0.2.

##  VII Results and Analysis

TABLE III: The results of MM-TSD trained on MM-2spk dataset and tested on both Seen-heard and Unseen-unheard test sets. The performance showcases the effectiveness of MM-TSD in detecting the target events guided by different types of prompts across diverse semantic attributes. Prompt Modality | Attribute | Seen-Heard | Unseen-Unheard
---|---|---|---
Text | Audio | Visual | AP (%)$\uparrow$ | AUC (%)$\uparrow$ | EER (%)$\downarrow$ | AP (%)$\uparrow$ | AUC (%)$\uparrow$ | EER (%)$\downarrow$
✓ |  |  | gender | 99.26 | 99.38 | 2.48 | 99.83 | 99.84 | 1.33
|  | counter | 99.16 | 99.45 | 2.87 | 99.72 | 99.86 | 1.71
|  | keynote | 99.88 | 99.52 | 3.34 | 99.88 | 99.49 | 3.32
| ✓ |  | speaker id. | 97.72 | 98.09 | 7.11 | 95.46 | 95.91 | 11.99
|  | ✓ | face id. | 92.46 | 93.99 | 13.51 | 85.74 | 88.28 | 20.82
✓ | ✓ |  | included id. | 97.67 | 98.06 | 7.18 | 95.37 | 95.84 | 11.91
| excluded id. | 96.07 | 96.03 | 8.86 | 96.56 | 95.81 | 11.67

In this section, we begin with an overview evaluation and analysis of our
proposed MM-TSD framework across various modalities and attributes, showcasing
the effectiveness of our approach in target speech diarization. Additionally,
we conduct a comparative analysis and ablation studies for each modality
prompts to further demonstrate the efficacy of each component of our approach.
Moreover, it’s worth noting that the applicability of our framework extends
beyond target speech diarization. With attribute-aligned prompts, it can be
utilized for tasks such as traditional speaker diarization, overlap speech
detection, and gender diarization. Remarkably, the performance of our
framework is comparable to specialist models dedicated to these individual
tasks. These investigations highlight the robustness and versatility of our
system.

###  VII-A Overall Evaluation of MM-TSD

During the training phase of our framework, we trained the MM-TSD model on the
MM-2spk dataset using audio-visual-text prompts. The performance results,
including AP, AUC, and EER, across diverse prompt modalities and attributes,
are detailed in Table III.

Indeed, the performance of our system in the text modality is particularly
impressive. It excels on both the “Seen-Heard” and “Unseen-Unheard” datasets,
achieving AP and AUC values that consistently surpass 99%, with EER values
remaining under 4%. These outcomes serve as compelling evidence of our MM-TSD
framework’s capability to accurately identify desired event regions, guided by
any provided text commands.

Also, the model’s specialization in attributes related to speaker identity
during training could lead to over-fitting on seen speakers, resulting in
better performance on the “Seen-Heard” set. When the unified model primarily
focuses on semantic attributes related to speaker identity, it might allocate
more resources to learning speaker-specific patterns, potentially at the
expense of other semantic concepts. This could explain the slightly lower
performance on attributes not directly related to speaker identity on the
“Seen-Heard” set.

In the case of audio prompts and audio-text prompts, as depicted in the fourth
and last two rows of Table III, our system continues to shine by effectively
detecting or excluding the target speaker based on the provided text as
logical controller and pre-enrolled audio. The AP and AUC values exceed 96%,
and EER values are under 12%, even for unseen speakers. This demonstrates the
cross-modal interaction and generalization ability of our system, which
remains robust in various scenarios.

In the challenging task of face-based target speaker detection, our system
achieves AP and AUC values exceeding 90% for “Seen-Heard” speakers. However,
it faces greater difficulty with unseen speakers, resulting in slightly lower
performance due to the limited information provided by a single face image.
Nevertheless, these results confirm our system’s capacity to discern the
relationship between a target speaker’s voice and face, albeit with a greater
challenge when dealing with unseen individuals.

As a conclusion, our MM-TSD system can support diverse prompts to detect
different type of desired speech events.

###  VII-B Text Modality Analysis

####  VII-B1 Parameter-efficient Exploration

For the text prompt, we employ parameter-efficient tuning methods for Pre-
trained Language Model (PLM) as our text prompt encoder. The pre-trained text
encoder DistilBERT can be fine-tuned and adapted to our tasks and language
commands by adding only a few trainable parameters. Specially, we study the
effect of various parameter-efficient techniques, such as bottleneck (bn)
adapter [46] and LoRA [34] on the MM-2spk dataset. Table IV shows a
performance comparison on the MM-2spk dataset between a frozen DistilBERT
(denoted as “DistilBERT”), a frozen DistilBERT with bottleneck adapter tuning
(denoted as “+ bn adapter”) and a frozen DistilBERT with LoRA tuning (denoted
as “+ LoRA”). The results indicate that a frozen DistilBERT with parameter-
effecient tuning outperforms the original fixed DistilBERT model, and LoRA
shows slightly better performance than bottleneck adapter. Therefore, we adopt
LoRA as our chosen tuning technique for DistilBERT as our text prompt encoder.

TABLE IV: Prameter-efficient exploration for three different text prompt encoders of MM-TSD trained on MM-2spk. ‘DistilBERT’ means the text encoder is the frozen DistilBERT only, “+ bn adapter” and “+ LoRA” represent the frozen DistilBERT with additional bottleneck adapter and LoRA tuning, respectively. Text Encoder |  Attribute |  ACC (%)$\uparrow$ |  AP (%)$\uparrow$ |  AUC (%)$\uparrow$ |  EER (%)$\downarrow$
---|---|---|---|---|---
DistilBERT |  gender |  97.78 |  99.53 |  99.56 |  2.26
|  counter |  94.48 |  97.19 |  97.93 |  6.36
|  keynote |  98.28 |  99.95 |  99.81 |  2.10
\+ bn adapter |  gender |  98.09 |  99.67 |  99.71 |  1.96
|  counter |  98.54 |  99.61 |  99.74 |  1.75
|  keynote |  98.37 |  99.96 |  99.85 |  1.89
\+ LoRA |  gender |  98.23 |  99.63 |  99.70 |  1.81
|  counter |  98.76 |  99.78 |  99.87 |  1.36
|  keynote |  98.40 |  99.96 |  99.83 |  1.96

####  VII-B2 Variable Number of Speakers

We further evaluate the performance of our system on MM-3spk and MM-4spk
datasets, each featuring 3 and 4 speakers, respectively. The results are
summarized in Table V. For these challenging conditions with more speakers,
both the AP and AUC of MM-TSD still surpass 99 %, which demonstrate its
ability to accurately detect the prompt-specified target event in conversation
scenarios involving multiple speakers.

TABLE V: The results of MM-TSD trained on MM-3spk and MM-4spk datasets with text prompts. Dataset | Attribute | AP (%)$\uparrow$ | AUC (%)$\uparrow$ | EER (%)$\downarrow$
---|---|---|---|---
MM-3spk | gender | 99.20 | 99.32 | 2.42
counter | 99.75 | 99.87 | 1.56
keynote | 99.88 | 99.60 | 3.18
MM-4spk | gender | 99.27 | 99.37 | 2.54
counter | 99.53 | 99.69 | 2.17
keynote | 99.88 | 99.62 | 3.07

####  VII-B3 MM-TSD v.s. OSD

Figure 6: The precision and recall results of MM-TSD when it achieves the
overlap speech detection task on the DIHARD II evaluation set, and results of
other specialized methods are also shown for comparison.

MM-TSD can also be functioned as a three-class speaker counter, capable of
estimating the number of concurrent speakers at each frame when we provide
prompts for non-speech, single speaker speech and overlapped speech
simultaneously.

We evaluate the performance of MM-TSD in the overlapped speech detection (OSD)
task [47] on DIHARD II evaluation set. We provide the text prompts aligned
with the overlapped speech event, which was initially trained on the MM-4spk
training set and further fine-tuned on the DIHARD II development set. Figure 6
compares the results of MM-TSD with two previous OSD models, which are
proposed by Bullock et al [48] and Jung et al [49] respectively. MM-TSD
achieves significantly better precision at $68.3\%$ and recall at $41.9\%$
compared to the specialized overlapped speech detection models on DIHARD II
evaluation set.

####  VII-B4 Structure Ablation

We then investigated the Transformer encoder-decoder architecture within MM-
TSD using text prompts. We implemented two baseline approaches that remove the
Transformer decoder part and treated each semantic event detection as a
supervised frame-wise classification problem. The first baseline still use the
Transformer encoder to obtain speech representation named as “Transformer
Encoder”. For the second baseline, we replaced the Transformer encoder with
ECAPA-TDNN encoder, named as “TDNN Encoder”. The encoder was followed by a
linear layer for each event. In our MM-TSD approach, prompts were used to
specify the target events, and a Transformer decoder decoded the event
occurrence probability from Transformer encoder representations as key and
value, and the prompt embedding as query, as illustrated in Figure 5.

The results reported in Table VI were trained on the MM-4spk training set and
validated on MM-4spk test set. The good performance of “Transformer Encoder”
as a baseline demonstrates that Transformer encoder representations contain
sufficient information about gender, concurrent number of speakers, and
speaker duration. Our MM-TSD achieves comparable performance with Transformer
encoder-only baseline. MM-TSD utilizes prompt embeddings as queries inputted
into the Transformer decoder to accurately retrieve the target event related
information from Transformer encoder representations. Additionally, our
framework can produce more flexible prompt-driven outputs with the Transformer
decoder structure.

TABLE VI: Structure ablation comparison between encode-only methods and prompt-driven encoder-decoder structure. The results are evaluated on MM-4spk test set.  Method |  Attribute |  ACC (%)$\uparrow$ |  AP (%)$\uparrow$ |  AUC (%)$\uparrow$ |  EER (%)$\downarrow$
---|---|---|---|---|---
TDNN Encoder |  gender |  97.09 |  98.95 |  99.17 |  2.94
|  counter |  97.97 |  99.54 |  99.77 |  2.13
|  keynote |  94.50 |  99.46 |  98.41 |  6.72
Transformer Encoder |  gender |  97.40 |  99.19 |  99.32 |  2.69
|  counter |  98.31 |  99.72 |  99.86 |  1.78
|  keynote |  97.03 |  99.79 |  99.36 |  3.39
MM-TSD |  gender |  97.51 |  99.27 |  99.37 |  2.54
|  counter |  98.02 |  99.53 |  99.69 |  2.17
|  keynote |  97.28 |  99.88 |  99.62 |  3.07

###  VII-C Audio Modality Analysis

In the context of audio prompts, when pre-enrolled speeches from all speakers
are available, our framework can function as a traditional speaker diarization
system. To demonstrate the effectiveness of MM-TSD with audio prompts, we
conducted a comparative analysis with traditional speaker diarization systems.

The main difference lies in the utilization of pre-enrolled speech as
auxiliary information in MM-TSD with audio prompts from all speakers. Inspired
by TS-VAD [5], we employed a clustering-based system [50] as a frontend to
obtain reference speech for each speaker roughly, achieving 15.60% and 21.25%
DER performance on Audio-2spk and Audio-3spk datasets, respectively.
Subsequently, our MM-TSD system with estimated reference speech as audio
prompt was applied to detect the activity of each speaker with the frontend
pre-enrollment.

In Table VII, we provide the DER performance on the CALLHOME Part 2 2-spk and
3-spk subsets. When oracle pre-enrolled speeches are provided for each
speaker, our MM-TSD achieves impressive DER values of 6.80% and 8.51% for 2
and 3 speakers, respectively. Even after adopting a clustering-based system to
obtain the estimated pre-enrolled speech for each speaker as audio prompts,
our system obtains DER values of 8.17% and 11.76% for 2 and 3 speakers,
respectively, and maintains comparable performance with various state-of-the-
art speaker diarization systems. Moreover, our MM-TSD is a more general
framework, and the others are all specialized systems.

TABLE VII: DER (%) results comparison on the CALLHOME Part 2 2-spk and 3-spk
subsets. Lower is better. ‘MM-TSD ($\mathcal{A}$)-oracle’ represents MM-TSD
with the ground-truth enrolled speech as audio prompts for speaker
diarization, ‘MM-TSD ($\mathcal{A}$)-clust.’ denotes that the audio prompts
come from the estimated target speech by clustering front-end.

Method |  CALLHOME (2spk) |  CALLHOME (3spk)
---|---|---
x-vector clustering [51] |  11.53 |  19.01
clutering frontend [50] |  15.60 |  21.25
BLSTM-EEND [52] |  26.03 |  -
SA-EEND [51, 52] |  9.54 |  14.00
SC-EEND [53] |  8.86 |  -
EEND-EDA [51] |  8.07 |  13.92
EEND-EDA † |  8.32 |  17.07
TS-VAD [5] |  9.51 |  -
MTFAD [5] |  7.82 |  -
AED-EEND [6] |  7.75 |  12.87
MM-TSD (oracle) |  6.80 |  8.51
MM-TSD (clustering) |  8.17 |  11.76

  * •

†: our implementation.

###  VII-D Visual Modality Analysis

####  VII-D1 Voice-face Alignment

MM-TSD applies the face prompt for face-based target speech diarization. To
study the robustness of our proposed voice-face alignment module, we presented
comparative results with and without voice-face alignment for different gender
pairs, showcasing the significant impact on performance for both the “Seen-
Heard” and “Unseen-Unheard” speakers, as detailed in Table IX.

In the “Seen-Heard” test set, the target speakers are already present during
training, making it relatively straightforward to match the speaker’s facial
features with its speaking regions. As a result, it is natural that the
performance is consistently better than that on the ‘Unseen-Unheard’ set.
Furthermore, we observe that the results exhibit slight improvements after the
incorporation of audio-visual alignment.

The “Unseen-Unheard” test sets consist of samples that were not encountered
during the training of both the voice-face alignment and face-based
diarization tasks. Although the discrimination performance of unseen people is
slightly lower than that of seen pairs, it’s worth noting that the performance
remains significantly superior to the “w/o align” baseline.

Interestingly, results for different gender pairs, namely F-M and M-F sets,
consistently demonstrate good performance. This is attributed to the distinct
gender characteristics, which make it easier to select the target speech using
the provided images. Conversely, mixtures with the same gender, i.e., M-M and
F-F sets, pose a greater challenge in terms of differentiation based solely on
the given images. However, the performance improvements following voice-face
alignment are notably more pronounced in these cases. The results demonstrate
that our voice-face alignment module indeed learn the intrinsic relationship
of cross-modal biometrics, that will help face-based diarization performance.
To the best of our knowledge, our paper is the first work to propose the face-
based diarization and it can be used for the diverse scenarios such as meeting
discussion and human-robot interaction.

TABLE VIII: Comparative study between the voice-face aligner and previous systems for cross-modal speaker verification. Results are performed on the different datasets. Method | EER (%) $\downarrow$ | AUC (%) $\uparrow$ | Dataset
---|---|---|---
DIMNet [54] | 24.56 | NA | VoxCeleb, VGGFace
SSNet [55] | 29.50 | 78.8 | VoxCeleb
Pins [21] | 29.60 | 78.5 | VoxCeleb
VF Aligner (Ours) | 24.40 | 83.0 | VoxCeleb2

####  VII-D2 Cross-modal Verification

Face-based diarization relies on the ability of cross-modal speaker
verification, which determines if a given voice segment and face image belong
to the same person [56, 57]. So we further evaluated the performance of our
Voice-Face Aligner (VF Aligner) on VoxCeleb2 dataset for this task and
compared it with some of the existing systems. The results are presented in
Table VIII, with Equal Error Rate (EER) and Area Under the Curve (AUC) as
performance metrics.

We observe that our audio-visual module with voice-face aligner effectively
performs for cross-modal speaker verification. After alignment, our audio-
visual module can perform comparably to the existing systems for cross-modal
verification. This highlights the effectiveness of our voice-face aligner
module in capturing the general higher-order information between voice
characteristic and face appearance, including but not limited to age,
nationality and gender.

TABLE IX: The results of MM-TSD for face-based diarization, different gender combinations with or without voice-face alignment are studied. Each test sample contains two speakers, ‘M’ denotes male and ‘F’ denotes the female. Visual Encoder | Gender | Seen-heard Speaker | Unseen-unheard Speaker
---|---|---|---
AP (%)$\uparrow$ | AUC (%)$\uparrow$ | EER (%)$\downarrow$ | AP (%)$\uparrow$ | AUC (%)$\uparrow$ | EER (%)$\downarrow$
w/o align | M-M | 97.63 | 97.84 | 7.73 | 77.11 | 79.73 | 29.09
M-F | 99.80 | 99.82 | 1.98 | 99.90 | 99.91 | 1.52
F-M | 99.84 | 99.84 | 1.74 | 96.37 | 97.15 | 7.03
F-F | 94.75 | 95.63 | 10.78 | 57.54 | 62.59 | 40.20
with align | M-M | 98.05 | 98.27 | 6.59 | 84.00 | 85.81 | 22.59
M-F | 99.84 | 99.85 | 1.85 | 99.90 | 99.91 | 1.39
F-M | 99.89 | 99.89 | 1.28 | 96.71 | 96.95 | 6.72
F-F | 96.60 | 97.04 | 9.49 | 74.52 | 77.69 | 27.68

###  VII-E Visualization of Results on Real-World Data

Figure 7: Visualization of results on real world samples selected from AVDIAR
dataset. Each row represents an utterance id in AVDIAR dataset, followed by
the prompt modality, semantic attribute, target event, prompt example, and
visualization comparison of prediction results versus groundtruth. The green
solid lines depict groundtruth labels of the target event, while the blue
dashed lines represent the probabilities of the target event’s occurrence at
each frame. The blue solid lines indicate the model’s prediction output of the
target event.

In previous experimental sessions, we showcased the effectiveness of each
modality module within MM-TSD and evaluated them separately on real datasets.
To further highlight the unified capability of all modalities to handle
complex conversations in real-world data, we conducted an assessment on the
AVDIAR (Audio-Visual Diarization) dataset [58]. This dataset is dedicated to
the audio-visual analysis of real-world conversational scenes. Due to the
absence of labels for all potential target events that MM-TSD can support, we
manually labeled some samples and tested MM-TSD trained on MM-2spk dataset
without further fine-tuning. Additionally, for an intuitive understanding of
our framework’s functionality, we provide visualizations of some prompt
examples and their corresponding prediction results from our proposed MM-TSD
framework in Figure 7.

As our prompt examples show, when we provide a text command like “Can you
distinguish the parts of this recording where male voices are present?”, our
framework will output the male speech regions within the input utterance audio
(the second row). In the scenario where the desired speaker’s face is
available, our framework can detect the active speaking regions according to
this face identity. It’s reasonable to expect that utterances with different
genders (the first row) will perform better than those with the same gender
(the third row), which is consistent with our previous analysis results.
Moreover, when the target speaker’s pre-enrolled speech is available, our
framework can detect all the speaking regions accordingly. Furthermore, if we
want to “Filter out the moments where the specific speaker is heard”, we can
provide the target speech and text commands so that our framework can achieve
a NOT Gate function and output the moments without the enrolled speaker’s
voice.

###  VII-F Future Work and Discussion

Complementary and composite prompts. Currently, our framework supports
textual, audio, or visual prompts to determine target events. In the scenario
involving audio-text prompts, the text merely functions as the logical
controller. However, the potential of audio-visual-textual prompts as
complementary inputs could be explored in future research. For instance, to
specify female speech, we could provide a text prompt for female speech
regions, along with the female face image and enrollment female speech
simultaneously. To achieving this, multi-modal training requires aligned
audio-visual-textual data collection.

Furthermore, composite prompt-specified events, such as “the female single
speaker speech,” could be explored in the future. Currently, in our framework,
we would need to separate such composite events into single events “female
speech” and “single speaker speech”. Then, using two prompts to output two
event regions, we can compute the composite event region. We aim to extend our
framework to handle such composite prompts directly in our future work.

Broader scope. While our study focuses on three semantic attributes as
examples for natural language descriptions of target speech, the potential
scope of semantic description is much broader. Attributes such as age, pitch,
nationality, and others could provide valuable context for specifying target
speech events. Although our paper introduces a foundational model, it
inherently cannot cover the entire spectrum of attribute scope. However, given
access to relevant data, our proposed framework and training strategy can be
adapted to accommodate any semantic attributes, offering a more comprehensive
and adaptable solution.

Moreover, our current framework operates in a supervised manner, relying on
labeled data for training. Expanding its capability to handle out-of-domain
events, where labeled data may be scarce or unavailable, would significantly
enhance its utility and effectiveness. Developing techniques for unsupervised
or semi-supervised learning within our framework could unlock its potential to
address a wider range of real-world scenarios and applications.

##  VIII Conclusion

Our paper proposed the new task named target speech diarization to detect
“when target event occurs”. To solve this problem, we have introduced MM-TSD,
a foundational model for target speech diarization that supports diverse and
multi-modal prompts. MM-TSD enables users to utilize semantic language
descriptions, pre-enrollment speech, pre-registered face images and audio-
language logical prompts to specify the target event(s). We have conducted a
comprehensive evaluation, including an overview and analysis of each modality,
across various tasks such as traditional speaker diarization and overlap
speech detection. MM-TSD achieved comparable performance with state-of-the-art
specialist models. Furthermore, we made the first attempt at face-based target
speech diarization with voice-face alignment. Our results demonstrate that MM-
TSD is a promising approach for effectively addressing the target speech
diarization problem. Our framework can be applied for human-robot interaction
and meeting analysing scenarios. In future work, we plan to explore
complementary prompts and extend the scope of our framework to further enhance
its comprehensiveness.

## References

  * [1] E. C. Cherry, “Some experiments on the recognition of speech, with one and with two ears,” _The Journal of the acoustical society of America_ , vol. 25, no. 5, pp. 975–979, 1953.
  * [2] J. B. Fritz, M. Elhilali, S. V. David, and S. A. Shamma, “Auditory attention—focusing the searchlight on sound,” _Current opinion in neurobiology_ , vol. 17, no. 4, pp. 437–455, 2007.
  * [3] N. Mesgarani and E. F. Chang, “Selective cortical representation of attended speaker in multi-talker speech perception,” _Nature_ , pp. 233–236, 2012.
  * [4] M. Cheng, W. Wang, Y. Zhang, X. Qin, and M. Li, “Target-speaker voice activity detection via sequence-to-sequence prediction,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2023, pp. 1–5.
  * [5] I. Medennikov, M. Korenevsky, T. Prisyach, Y. Khokhlov, M. Korenevskaya, I. Sorokin, T. Timofeeva, A. Mitrofanov, A. Andrusenko, I. Podluzhny, A. Laptev, and A. Romanenko, “Target-Speaker Voice Activity Detection: A Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenario,” in _Proc. INTERSPEECH_ , 2020, pp. 274–278.
  * [6] Z. Chen, B. Han, S. Wang, and Y. Qian, “Attention-based encoder-decoder end-to-end neural diarization with embedding enhancer,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 32, pp. 1636–1649, 2024.
  * [7] B. Desplanques, J. Thienpondt, and K. Demuynck, “ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification,” in _Proc. INTERSPEECH_ , 2020, pp. 3830–3834.
  * [8] T. Liu, K. A. Lee, Q. Wang, and H. Li, “Golden Gemini is All You Need: Finding the Sweet Spots for Speaker Verification,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , pp. 2324–2337, 2024.
  * [9] Y. Wang, M. Ravanelli, A. Nfissi, and A. Yacoubi, “Speech emotion diarization: Which emotion appears when?” _arXiv preprint arXiv:2306.12991_ , 2023.
  * [10] E. Tzinis, G. Wichern, A. S. Subramanian, P. Smaragdis, and J. Le Roux, “Heterogeneous Target Speech Separation,” in _Proc. INTERSPEECH_ , 2022, pp. 1796–1800.
  * [11] M. Lebourdais, M. Tahon, A. LAURENT, and S. Meignier, “Overlapped speech and gender detection with WavLM pre-trained features,” in _Proc. INTERSPEECH_ , 2022, pp. 5010–5014.
  * [12] Y. Jiang, Z. Chen, R. Tao, L. Deng, Y. Qian, and H. Li, “Prompt-driven target speech diarization,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2024, pp. 11 086–11 090.
  * [13] X. Liu, H. Liu, Q. Kong, X. Mei, J. Zhao, Q. Huang, M. D. Plumbley, and W. Wang, “Separate What You Describe: Language-Queried Audio Source Separation,” in _Proc. INTERSPEECH_ , 2022, pp. 1801–1805.
  * [14] X. Liu, Q. Kong, Y. Zhao, H. Liu, Y. Yuan, Y. Liu, R. Xia, Y. Wang, M. D. Plumbley, and W. Wang, “Separate anything you describe,” _arXiv preprint arXiv:2308.05037_ , 2023.
  * [15] X. Hao, J. Wu, J. Yu, C. Xu, and K. C. Tan, “Typing to listen at the cocktail party: Text-guided target speaker extraction,” _arXiv preprint arXiv:2310.07284_ , 2023.
  * [16] R. Tao, Z. Pan, R. K. Das, X. Qian, M. Z. Shou, and H. Li, “Is someone speaking? exploring long-term temporal features for audio-visual active speaker detection,” in _Proceedings of the 29th ACM International Conference on Multimedia_ , 2021, pp. 3927–3935.
  * [17] Y. Jiang, R. Tao, Z. Pan, and H. Li, “Target Active Speaker Detection with Audio-visual Cues,” in _Proc. INTERSPEECH_ , 2023, pp. 3152–3156.
  * [18] M. Cheng and M. Li, “Multi-input multi-output target-speaker voice activity detection for unified, flexible, and robust audio-visual speaker diarization,” _arXiv preprint arXiv:2401.08052_ , 2024.
  * [19] S.-W. Chung, J. S. Chung, and H.-G. Kang, “Perfect match: Self-supervised embeddings for cross-modal retrieval,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 14, no. 3, pp. 568–576, 2020.
  * [20] A. Nagrani, S. Albanie, and A. Zisserman, “Seeing voices and hearing faces: Cross-modal biometric matching,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 8427–8436.
  * [21] ——, “Learnable pins: Cross-modal embeddings for person identity,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 71–88.
  * [22] T.-H. Oh, T. Dekel, C. Kim, I. Mosseri, W. T. Freeman, M. Rubinstein, and W. Matusik, “Speech2face: Learning the face behind a voice,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 7539–7548.
  * [23] S.-W. Chung, S. Choe, J. S. Chung, and H.-G. Kang, “FaceFilter: Audio-Visual Speech Separation Using Still Images,” in _Proc. INTERSPEECH_ , 2020, pp. 3481–3485.
  * [24] D. Wang, X. Xiao, N. Kanda, T. Yoshioka, and J. Wu, “Target speaker voice activity detection with transformers and its integration with end-to-end neural diarization,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2023, pp. 1–5.
  * [25] X. Mei, X. Liu, H. Liu, J. Sun, M. D. Plumbley, and W. Wang, “Language-based audio retrieval with pre-trained models,” _Detection and Classification of Acoustic Scenes and Events (DCASE) Challenge, Tech. Rep_ , 2022.
  * [26] A. S. Koepke, A.-M. Oncescu, J. Henriques, Z. Akata, and S. Albanie, “Audio retrieval with natural language queries: A benchmark study,” _IEEE Transactions on Multimedia_ , 2022.
  * [27] S. Lou, X. Xu, M. Wu, and K. Yu, “Audio-text retrieval in context,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022, pp. 4793–4797.
  * [28] Z. Pan, R. Tao, C. Xu, and H. Li, “Muse: Multi-modal target speaker extraction with visual cues,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021, pp. 6678–6682.
  * [29] R. Thornhill and A. P. Møller, “Developmental stability, disease and medicine,” _Biological Reviews_ , vol. 72, no. 4, pp. 497–548, 1997.
  * [30] H. Hollien and G. P. Moore, “Measurements of the vocal folds during changes in pitch,” _Journal of Speech and Hearing Research_ , vol. 3, no. 2, pp. 157–165, 1960.
  * [31] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao _et al._ , “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 16, no. 6, pp. 1505–1518, 2022.
  * [32] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [33] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,” _arXiv preprint arXiv:1910.01108_ , 2019.
  * [34] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in _International Conference on Learning Representations_ , 2022.
  * [35] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [36] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [37] F. Landini, M. Diez, A. Lozano-Diez, and L. Burget, “Multi-speaker and wide-band simulated conversations as training data for end-to-end neural diarization,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2023, pp. 1–5.
  * [38] J. S. Chung, A. Nagrani, and A. Zisserman, “VoxCeleb2: Deep Speaker Recognition,” in _Proc. INTERSPEECH_ , 2018, pp. 1086–1090.
  * [39] N. Ryant, K. Church, C. Cieri, A. Cristia, J. Du, S. Ganapathy, and M. Liberman, “The second dihard diarization challenge: Dataset, task, and baselines,” _arXiv preprint arXiv:1906.07839_ , 2019.
  * [40] G. R. Doddington, M. A. Przybocki, A. F. Martin, and D. A. Reynolds, “The nist speaker recognition evaluation–overview, methodology, systems, results, perspective,” _Speech communication_ , vol. 31, no. 2-3, pp. 225–254, 2000.
  * [41] X. An, X. Zhu, Y. Gao, Y. Xiao, Y. Zhao, Z. Feng, L. Wu, B. Qin, M. Zhang, D. Zhang _et al._ , “Partial fc: Training 10 million identities on a single machine,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 1445–1449.
  * [42] T. Ko, V. Peddinti, D. Povey, M. L. Seltzer, and S. Khudanpur, “A study on data augmentation of reverberant speech for robust speech recognition,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2017, pp. 5220–5224.
  * [43] D. Snyder, G. Chen, and D. Povey, “Musan: A music, speech, and noise corpus,” _arXiv preprint arXiv:1510.08484_ , 2015.
  * [44] K. Zhang, Z. Zhang, Z. Li, and Y. Qiao, “Joint face detection and alignment using multitask cascaded convolutional networks,” _IEEE signal processing letters_ , vol. 23, no. 10, pp. 1499–1503, 2016.
  * [45] M. Kowalski, J. Naruniec, and T. Trzcinski, “Deep alignment network: A convolutional neural network for robust face alignment,” in _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 2017, pp. 88–97.
  * [46] A. Bapna, N. Arivazhagan, and O. Firat, “Simple, scalable adaptation for neural machine translation,” _arXiv preprint arXiv:1909.08478_ , 2019.
  * [47] K. Boakye, B. Trueba-Hornero, O. Vinyals, and G. Friedland, “Overlapped speech detection for improved speaker diarization in multiparty meetings,” in _2008 IEEE international conference on acoustics, speech and signal processing_. IEEE, 2008, pp. 4353–4356.
  * [48] L. Bullock, H. Bredin, and L. P. Garcia-Perera, “Overlap-aware diarization: Resegmentation using neural end-to-end overlapped speech detection,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020, pp. 7114–7118.
  * [49] J.-w. Jung, H.-S. Heo, Y. Kwon, J. S. Chung, and B.-J. Lee, “Three-class overlapped speech detection using a convolutional recurrent neural network,” _arXiv preprint arXiv:2104.02878_ , 2021.
  * [50] H. Wang, C. Liang, S. Wang, Z. Chen, B. Zhang, X. Xiang, Y. Deng, and Y. Qian, “Wespeaker: A research and production oriented speaker embedding learning toolkit,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2023, pp. 1–5.
  * [51] S. Horiguchi, Y. Fujita, S. Watanabe, Y. Xue, and K. Nagamatsu, “End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractors,” in _Proc. INTERSPEECH_ , 2020, pp. 269–273.
  * [52] Y. Fujita, N. Kanda, S. Horiguchi, K. Nagamatsu, and S. Watanabe, “End-to-End Neural Speaker Diarization with Permutation-Free Objectives,” in _Proc. INTERSPEECH_ , 2019, pp. 4300–4304.
  * [53] Y. Fujita, S. Watanabe, S. Horiguchi, Y. Xue, J. Shi, and K. Nagamatsu, “Neural speaker diarization with speaker-wise chain rule,” _arXiv preprint arXiv:2006.01796_ , 2020.
  * [54] Y. Wen, M. A. Ismail, W. Liu, B. Raj, and R. Singh, “Disjoint mapping network for cross-modal matching of voices and faces,” _arXiv preprint arXiv:1807.04836_ , 2018.
  * [55] S. Nawaz, M. K. Janjua, I. Gallo, A. Mahmood, and A. Calefati, “Deep latent space learning for cross-modal mapping of audio and visual signals,” in _2019 Digital Image Computing: Techniques and Applications (DICTA)_. IEEE, 2019, pp. 1–7.
  * [56] S. Nawaz, M. S. Saeed, P. Morerio, A. Mahmood, I. Gallo, M. H. Yousaf, and A. Del Bue, “Cross-modal speaker verification and recognition: A multilingual perspective,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 1682–1691.
  * [57] R. Tao, R. K. Das, and H. Li, “Audio-Visual Speaker Recognition with a Cross-Modal Discriminative Network,” in _Proc. INTERSPEECH_ , 2020, pp. 2242–2246.
  * [58] I. D. Gebru, S. Ba, X. Li, and R. Horaud, “Audio-visual speaker diarization based on spatiotemporal Bayesian fusion,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 40, no. 5, pp. 1086–1099, 2018.


# Probing Ring Resonator Sensor Based on Vernier Effect

1st Wenwen Zhang  Department of Electrical and Computer Engineering
University of British Columbia
Vancouver, Canada
<EMAIL_ADDRESS>2nd Hao Zhang  Department of Electrical and Computer
Engineering
University of British Columbia
Vancouver, Canada
<EMAIL_ADDRESS>
###### Abstract

The Vernier effect has seen extensive application in optical structures,
serving to augment the free spectral range (FSR). A substantial FSR is vital
in a myriad of applications including multiplexers, enabling a broad, clear
band comparable to the C-band to accommodate a maximum number of channels.
Nevertheless, a large FSR often conflicts with bending loss, as it
necessitates a smaller resonator radius, thus increase the insertion loss in
the bending portion. To facilitate FSR expansion without amplifying bending
loss, we employed cascaded and parallel racetrack resonators and ring
resonators of varying radius that demonstrate the Vernier effect. In this
study, we designed, fabricated, and tested multiple types of racetrack
resonators to validate the Vernier effect and its FSR extension capabilities.
Our investigations substantiate that the Vernier effect, based on cascaded and
series-coupled micro-ring resonator (MRR) sensors, can efficiently mitigate
intra-channel cross-talk at higher data rates. This is achieved by providing
larger input-to-through suppression, thus paving the way for future
applications.

###### Index Terms:

Free spectral range (FSR), ring resonator, Vernier effect.

##  I Introduction

With the fast development of communication technology, the amount of
information to be transmitted increased rapidly. Photonic devices play an
important role in computing systems for wider bandwidth and more cost-
efficient interconnects. Traditional interconnects are typically made of
metallic materials, which introduces high energy consumption and large latency
[1]. Silicon-on-insulator (SOI) interconnection devices lead to a new field of
communication [2]. But it still demands larger free spectral range (FSR). To
get extended FSR, flat passband response, or increased sensor sensitivity,
various cascaded and parallel structures are designed to explore excellent
spectra characteristics. The cascaded structure is not limited to Mach–Zehnder
interferometers [3] or Fabry–Perot cavities and micro-ring resonators [4]. By
applying the vernier effect to optical waveguide sensors, the sensitivity of
these sensors can be further improved.

Vernier effect based on micro-ring resonators (MRRs) has been validated
effectively in increasing device performance in communication equipment [5].
Adopting the vernier effect in ring/racetrack structures could extend the FSR
which benefits dense wavelength-division multiplexing (DWDM) applications and
increase sensor sensitivity [6]. Sensors with sensitivity-enhanced refractive
index are one of the outstanding applications of micro-fibers [7]. This fiber
sensor achieves sensitivity enhancement of 3301.76 nm/RIU and is implemented
by connecting two microfiber knot resonators in series, which exhibits the
vernier effect. Refractive index sensor based on silicon-on-insulator with
vernier effect and enhanced sensitivity is also realized in [8]. These sensors
perform a significant role in measurements on monitoring the environment and
medical analysis, food detection, etc. Silicon-on-insulator racetrack
resonators with extended FSR are also realized by cascaded rings with exhibit
vernier effect [9]. Adding a straight waveguide in the middle of ring
resonators makes coupling length more controllable. Vernier effect in this
design could broaden the FSR value while not increasing bending losses. Both
through port insertion loss and interstitial peak suppression could be
improved by the addition of contra-directional couplers. In our work, several
types of racetrack resonators are designed and fabricated to validate the
vernier effect and its FSR–extending performance. Parameters (inter-stage
coupling, coupling length $\&$ coupling strength of straight waveguide, field
transmission, etc.) in design are varied to find out their influence on the
performance.

##  II Modeling

###  II-A Cascaded Coupled Resonator

To suppress bending loss, radius of racetrack resonators is always settled as
large value, which contributes to small FSR. To extend FSR, multiple racetrack
resonators are connected to exhibit vernier effect. Two cascaded racetrack
resonators are displayed in Fig.1 a), which show vernier effect.

Figure 1: Coupling scheme: (a) series pattern. (b) parallel pattern.

The drop port transmission function [10] in amplitude forms for through port
and drop port is expressed by Eq. 1:

$(\frac{E_{t1}}{E_{i1}})_{through}=\frac{-t_{1}\kappa_{1}^{2}\alpha_{1}e^{j\theta_{1}}(t_{3}\alpha_{2}e^{j\theta_{2}}-t_{2})}{1-t_{3}t_{2}\alpha_{2}e^{j\theta_{2}}-t_{2}t_{1}\alpha_{1}e^{j\theta_{1}}+t_{3}t_{1}\alpha_{1}e^{j\theta_{1}}e^{j\theta_{2}}}.$
(1)

$(\frac{E_{t1}}{E_{i1}})_{drop}=\frac{\kappa_{3}\kappa_{2}\kappa_{1}\alpha_{1}\alpha_{2}e^{j\frac{\theta_{1}}{2}}e^{j\frac{\theta_{2}}{2}}}{1-t_{3}t_{2}\alpha_{2}e^{j\theta_{2}}-t_{2}t_{1}\alpha_{1}e^{j\theta_{1}}+t_{3}t_{1}\alpha_{1}e^{j\theta_{1}}e^{j\theta_{2}}}.$
(2)

Where $\kappa_{1}$, $\kappa_{2}$, $\kappa_{3}$ and its conjugated complex form
value represents corresponding coupling coefficient, and t1, t2, t3, and its
conjugated complex form represents corresponding field transmission factor.
$\alpha$ (zero loss: $\alpha$=1) is the total field loss coefficient. To
simplify the model, we let $\kappa_{1}$=$\kappa_{3}$.

Series cascaded rings/racetrack shapes with different radii makes it possible
to extend FSR to the least common multiple of single one. This is because the
resonant condition is satisfied only when both single resonators are in
resonance. This leads to the suppression of some transmission peaks of
racetrack resonators. The FSR of cascaded resonators [6] with different radii
is given by Eq. 3 \- Eq. 4:

$FSR=N\times FSR_{1}=M\times FSR_{2}$ (3)

$FSR=|M-N|\frac{FSR_{1}\times FSR_{2}}{FSR_{1}-FSR_{2}}$ (4)

Where N and M are natural and coprime numbers. The group index ng and
effective index neff is defined as:

$n_{g}=n_{eff}-\lambda\frac{\partial n_{eff}}{\partial\lambda}$ (5)

###  II-B Serially Coupled Double Resonator

To realize box-like filter response, parallelly coupled racetrack resonators
are designed and analyzed in Suzuki et al. (1995). The parallel type of
racetrack resonator is displayed in Fig. 1 b).

This kind of configuration can be treated as gratings. Choosing $\Lambda$ to
be odd multiple of a quarter wavelength, constructive interference forms by
reflected light. Let $\lambda_{0}$ to be the center wavelength (1550 $nm$),
$\Lambda$ is expressed as Eq. 6:

$\Lambda=(2m+1)\frac{\lambda_{0}}{4n_{eff}}$ (6)

The parallel coupled racetrack resonators exhibit vernier effect when the
periods of FSR of racetrack resonators and gratings follow Eq. 7 and Eq. 8. In
this case, transmission peaks are suppressed as well, overall FSR is extended.

$\Lambda=N_{Racetrack}FSR_{Racetrack}=M_{Grating}FSR_{Grating}$ (7)

$\Lambda=\frac{M_{grating}n_{Weff}}{N_{Racetrack}n_{Weff}}\pi r$ (8)

Figure 2: Mode files for fundamental TE and TM mode. (a) TE mode: E intensity.
(b) TE mode: H intensity. (c) TM mode: E intensity. (d) TM mode: H intensity.

##  III Results and Discussion

###  III-A Simulation results

To determine the effective size of the cascaded racetrack resonator, a strip
waveguide with 550 nm width and 220 nm height is modeled to explore the field
distribution. The effective index ($\lambda=1550nm$) calculated by Taylor
expansion with the simulation results have a good agreement [11].

To further decide the parameter of the racetrack resonator, the relationship
between inter-coupling length, gap, and coupling coefficient should be
explored. We use FDTD simulation to figure out how coupling coefficients
depend on these parameters. To satisfy the requirement of adequate
interstitial suppression, $M$ or $N$ should be small enough. The length of
straight waveguide parts in cascaded ring resonators determines the coupling
coefficients and eventual FSR. With the unavoidable fabrication error of the
lithography process, more copies with slight variations around the appointed
circuit size are submitted to analyze and predict the effect of the error
(Fig. 5a).

###  III-B Experiment results

Figure 3: Coupling types and devices. (a) Racetrack resonator with a straight
coupling. (b) Parallel ring resonators (5 poles) with arc coupling. (c)
Parallel ring resonators (3 poles) with bend coupling. (d) Parallel ring
resonators (2 poles) with a straight coupling.

To limit insertion loss and ensure TE-polarization with a single mode,
periodic grating couplers with the fixed-length taper are utilized to couple
light from the fiber array to the waveguide. Different coupling types between
the waveguide and ring are investigated. Various coupling length has also been
tested, given the machining accuracy and fabrication error. Finally, the
performance of several parallel rings and racetrack resonators has been
verified. For racetrack resonators, decreasing the coupling gap will increase
the coupling coefficient and decrease insertion loss. This will also lead to
the appearance of parasitic bands and decay of interstitial resonance
suppression as in Fig. 4a. For the parallel ring resonators, we have
fabricated three types of coupling: textbfarc waveguide, textbfbend waveguide,
and textbfstraight waveguide, displayed in Fig. 3. Straight coupling is the
most normal way of coupling, while bend coupling can provide stronger coupling
at the same condition. Arc coupling provides the weakest strength. Radius in
the parallel rings decides the FSR, as Fig. 4b shows, a larger radius presents
a shorter FSR. The insertion loss is smaller as well. To investigate the
influence of fabrication error on the final performance, we fabricated and
measured four copies. In parallel rings, the coupling type doesn’t affect
performance too much, as shown in Fig. 5b. For two rings, a larger radius
brings in smaller FSR, as shown in Fig. 5c, where the ring with $10\mu m$
generates $10nm$ FSR, while the ring with $7\mu m$ generates $13nm$ FSR. When
more rings are loaded between ports, the center frequency moves toward to
lower band, as shown in Fig. 5 d).

Figure 4: Comparison between devices: (a) Experimental results for racetrack
resonators. Gaps between resonators are $0.31\mu m$ (red line) and $0.39\mu m$
(blue line). (b) Comparison between parallel rings (5 poles) with different
radius: $10\mu m$ (red line) and $6.5\mu m$ (blue line).  Figure 5: Comparison
between parallel ring performance. (a) Parallel rings (5 poles) have the same
size but display different performances. (b) Parallel rings (3 poles) with
different coupling types: straight waveguide (blue) and bend waveguide (red).
(c) Parallel rings (2 poles) with different radius : radius=$7\mu m$ (blue
line) and radius=$10\mu m$ (red line). (d) Different ring numbers loaded
between sources: five rings (blue) and two rings (red).

##  IV Conclusion

Our work focused on the vernier effect of cascaded and series-coupled MRRs
sensors with different parameters. From our simulation and experiment results,
it is proven to be effective for low intra-channel crosstalk at higher data
rates by offering larger input-to-through suppression for future applications.

## Acknowledgment

We acknowledge the Natural Sciences and Engineering Research Council of Canada
(NSERC) Silicon Electronic-Photonic Integrated Circuits (SiEPIC) Program, the
NSERC CREATE in Quantum Computing program, and the Canadian Microelectronics
Corporation (CMC). Devices were fabricated at Advanced Micro Foundry (AMF) A
STAR foundry in Singapore.

## References

  * [1] L. Zhang, Y. Li, M. Song, J.-Y. Yang, R. G. Beausoleil, and A. E. Willner, “Silicon microring-based signal modulation for chip-scale optical interconnection,” _Applied Physics A_ , vol. 95, no. 4, pp. 1089–1100, 2009\.
  * [2] F. Xia, M. Rooks, L. Sekaric, and Y. Vlasov, “Ultra-compact high order ring resonator filters using submicron silicon photonic wires for on-chip optical interconnects,” _Optics express_ , vol. 15, no. 19, pp. 11 934–11 941, 2007.
  * [3] M. La Notte and V. M. Passaro, “Ultra high sensitivity chemical photonic sensing by mach–zehnder interferometer enhanced vernier-effect,” _Sensors and Actuators B: Chemical_ , vol. 176, pp. 994–1007, 2013.
  * [4] J. Song, L. Wang, L. Jin, X. Xia, Q. Kou, S. Bouchoule, and J.-J. He, “Intensity-interrogated sensor based on cascaded fabry–perot laser and microring resonator,” _Journal of lightwave technology_ , vol. 30, no. 17, pp. 2901–2906, 2012.
  * [5] G. Griffel, “Vernier effect in asymmetrical ring resonator arrays,” _IEEE Photonics Technology Letters_ , vol. 12, no. 12, pp. 1642–1644, 2000\.
  * [6] C. Chaichuay, P. P. Yupapin, and P. Saeung, “The serially coupled multiple ring resonator filters and vernier effect.” _Optica Applicata_ , vol. 39, no. 1, 2009.
  * [7] Z. Lu and Y. Sun, “Sensitivity-enhanced refractive index sensor based on series structure of two microfiber knot resonators with vernier effect,” in _2019 18th International Conference on Optical Communications and Networks (ICOCN)_. IEEE, 2019, pp. 1–3.
  * [8] T. Claes, W. Bogaerts, and P. Bienstman, “Experimental characterization of a silicon photonic biosensor consisting of two cascaded ring resonators based on the vernier-effect and introduction of a curve fitting method for an improved detection limit,” _Optics express_ , vol. 18, no. 22, pp. 22 747–22 761, 2010.
  * [9] R. Boeck, N. A. Jaeger, N. Rouger, and L. Chrostowski, “Series-coupled silicon racetrack resonators and the vernier effect: theory and measurement,” _Optics express_ , vol. 18, no. 24, pp. 25 151–25 157, 2010.
  * [10] D. G. Rabus, _Integrated ring resonators_. Springer, 2007.
  * [11] L. Chrostowski and M. Hochberg, _Silicon photonics design: from devices to systems_. Cambridge University Press, 2015\.