Speaker anonymization using orthogonal Householder neural network
Speaker anonymization aims to conceal a speaker's identity while preserving
content information in speech. Current mainstream neural-network speaker
anonymization systems disentangle speech into prosody-related, content, and
speaker representations. The speaker representation is then anonymized by a
selection-based speaker anonymizer that uses a mean vector over a set of
randomly selected speaker vectors from an external pool of English speakers.
However, the resulting anonymized vectors are subject to severe privacy leakage
against powerful attackers, reduction in speaker diversity, and language
mismatch problems for unseen-language speaker anonymization. To generate
diverse, language-neutral speaker vectors, this paper proposes an anonymizer
based on an orthogonal Householder neural network (OHNN). Specifically, the
OHNN acts like a rotation to transform the original speaker vectors into
anonymized speaker vectors, which are constrained to follow the distribution
over the original speaker vector space. A basic classification loss is
introduced to ensure that anonymized speaker vectors from different speakers
have unique speaker identities. To further protect speaker identities, an
improved classification loss and similarity loss are used to push
original-anonymized sample pairs away from each other. Experiments on
VoicePrivacy Challenge datasets in English and the \textit{AISHELL-3} dataset
in Mandarin demonstrate the proposed anonymizer's effectiveness.
Xiaoxiao Miao, Xin Wang, Erica Cooper, Junichi Yamagishi, Natalia Tomashenko
# Language-independent speaker anonymization using orthogonal Householder neural network

###### Abstract

Speaker anonymization aims to conceal a speaker's identity while preserving content information in speech. Current mainstream neural-network speaker anonymization systems disentangle speech into prosody-related, content, and speaker representations. The speaker representation is then anonymized by a selection-based speaker anonymizer that uses a mean vector over a set of randomly selected speaker vectors from an external pool of English speakers. However, the resulting anonymized vectors are subject to severe privacy leakage against powerful attackers, reduction in speaker diversity, and language mismatch problems for unseen language speaker anonymization. To generate diverse, language-neutral speaker vectors, this paper proposes an anonymizer based on an orthogonal Householder neural network (OHNN). Specifically, the OHNN acts like a rotation to transform the original speaker vectors into anonymized speaker vectors, which are constrained to follow the distribution over the original speaker vector space. A basic classification loss is introduced to ensure that anonymized speaker vectors from different speakers have unique speaker identities. To further protect speaker identities, an improved classification loss and similarity loss are used to push original-anonymized sample pairs away from each other. Experiments on VoicePrivacy Challenge datasets in English and the _AISHELL-3_ dataset in Mandarin demonstrate the proposed anonymizer's effectiveness.

 Speaker anonymization, selection-based anonymizer, orthogonal Householder neural network anonymizer, weighted additive angular softmax.

## I Introduction

Speech technology enables machines to recognize, analyze, and understand human speech, which facilitates human-machine communication and offers great convenience in our daily lives. Despite its prominent advantages, it suffers from voice privacy leakage, which intrudes on or tampers with a speaker's private information. For instance, by using advanced speaker [1, 2], dialect [3, 4], pathological condition [5, 6], or other types of speech attribute recognition systems, attributes such as a speaker's identity, geographical origin, and health status can easily be captured from speech recordings. Moreover, advanced speech synthesis techniques enable resynthesis, cloning, or conversion of a speaker's identity information to access personal voice-controlled devices [7, 8, 9]. In this paper, we are especially interested in speaker anonymization, which is a user-centric voice privacy solution to conceal a speaker's identity without degrading intelligibility and naturalness [10, 11, 12]. This task was standardized by the VoicePrivacy Challenge (VPC) committee [11, 12, 13], which held challenges in 2020 and 2022, to advance the development of voice privacy preservation techniques.

Several approaches to protect speaker privacy are based on digital signal processing (DSP) methods [11, 12, 14, 15, 16, 17, 18], which modify instantaneous speech characteristics such as the pitch, spectral envelope, and time scaling. State-of-the-art anonymization approaches have borrowed ideas from neural speech conversion and synthesis, mainly focusing on disentangled latent representation learning [19, 20, 21, 22, 23, 24, 25] via two hypotheses. The first is that speech can be explicitly decomposed into content, speaker identity, and prosodic (intonation, stress, and rhythm) representations. Here, the speaker identity is a statistical time-invariant representation throughout an utterance, whereas content and prosodic information vary over time. The second hypothesis is that a speaker's identity representation carries most of his or her private information. Thus, generated speech using original content, prosodic, and anonymized speaker representations can suppress the original identity information (privacy) while maintaining intelligibility and naturalness (utility).

A general framework for disentanglement-based speaker anonymization involves the following components.

_Fine-grained disentangled representation extraction from original speech._ Here, extraction entails three aspects: (i) Content feature extraction. Typically, low-dimensional phonetic bottleneck features are extracted from an intermediate layer of a language-specific automatic speech recognition neural acoustic model (ASR AM) [26, 27]. This type of content encoder requires a large amount of transcribed English training data to obtain accurate linguistic representations, making it impossible to use for a different language. Nevertheless, content encoders based on self-supervised learning (SSL) and trained on unlabeled data can provide general content representations regardless of the language, enabling anonymization of speech data from any language. (ii) Prosody-related feature extraction to obtain the fundamental frequency, i.e., F0. (iii) Speaker embedding extraction. A speaker vector is extracted either from an automatic speaker verification (ASV) system based on a time-delay neural network (TDNN) [28], or from a more effective ASV based on emphasized channel attention,
propagation and aggregation in TDNN (ECAPA-TDNN) [29].

_Speaker representation anonymization_: The core idea of a speaker vector anonymizer is to hide original speaker information while preserving the diversity among different speakers. A widely used selection-based speaker anonymizer [30, 23] replaces an original speaker vector with the mean vector (pseudo-speaker vector) of a set of randomly selected speaker vectors from an external pool of English speakers.

_Anonymized speech synthesis_: An anonymized speaker vector with the original fundamental frequency and content features is passed to a speech waveform generation model to synthesize high-quality anonymized speech. The speech synthesis model can be a traditional text-to-speech pipeline model--a speech synthesis acoustic model (SS AM) and a neural source filter- (NSF-) based vocoder [31]--or a unified HiFi-GAN [32].

Despite confirmation of this approach's effectiveness [11, 12, 33], there remains much room for improvement for different attack scenarios and unseen language anonymization. Previous works [11, 12, 33, 23] have suggested that the most significant performance bottleneck for the current mainstream approach is the selection-based speaker anonymizer, whose performance significantly depends on the distribution of the external pool and how pseudo-speakers are selected from the pool. (i) For English speaker anonymization [11, 12, 13], the performance of speaker verifiability has gradually decreased against more powerful attackers. Additionally, voice distinctiveness is significantly degraded by anonymization. (ii) For unseen language (e.g., Mandarin) speaker anonymization, pseudo-speaker representations are generated from an external English speaker vector pool, and the resulting language mismatch increases the character error rate (CER) [33, 34].

Following this pipeline of disentanglement-based anonymization, with special consideration of the selection-based approach's problems, we propose a novel speaker anonymization system (SAS) based on an orthogonal Householder neural network (OHNN). As shown in the lower part of Fig. 1, the OHNN-based anonymizer generates distinctive anonymized speaker vectors that can protect privacy under all attack scenarios and can successfully be adapted to unseen-language speaker anonymization without severe language mismatch. Specifically, original speaker vectors are rotated to anonymized ones by an OHNN, which is a linear transformation with orthogonality. This module ensures that the anonymized speaker vectors follow the distribution over the original speaker vector space. To prevent anonymized speakers from overlapping with any other speakers, a basic classification loss, namely, additive angular margin softmax (AAM) is used to ensure that the anonymized speaker vectors from different speakers have unique speaker identities. To further protect speaker identities, an improved classification loss, weighted AAM (w-AAM), and the cosine similarity loss are used to push original-anonymized sample pairs away from each other.

The main contributions of this work are as follows:

* We propose an OHNN-based anonymizer that does not require an external pool of speaker vectors but instead transforms original speaker vectors to anonymized ones with carefully designed training constraints. We show empirically that these anonymized speaker vectors are diverse and language-neutral.
* We visualize the cosine similarities between pairs of speaker vectors extracted from the generated speech of users and different attackers, as obtained with the commonly used selection-based anonymizer and our OHNN-based anonymizer. The results show that our proposed method effectively reduces the privacy leakage against different attackers and improves the diversity of anonymized speakers.
* We conduct experiments on VPC English datasets and the _AISHELL-3_ Mandarin datasets. The results show that the proposed model can be successfully adapted for different languages, as it achieved either the best or near-best performance published to date under all attack scenarios in terms of privacy and utility metrics.

## II Related Work

In this section, we introduce the VPC's official design, which provides the setting for this study, including definitions of specific goals, attack models, and objective evaluation metrics. We also overview existing speaker anonymization approaches and their limitations.

### _The VoicePrivacy Challenges_

The VPC formulates the speaker anonymization task as a game between users and attackers, as shown in Fig. 2. A user publishes anonymized data, called _test trials_, after applying an SAS to his or her original private speech. These anonymized data are expected to conceal the speaker's identity from different attackers while keeping other characteristics unchanged to maintain intelligibility and naturalness; this

Fig. 1: Architecture of an SSL-based language-independent SAS, with selection- and OHNN-based anonymizers.
facilitates downstream tasks to be achieved, such as human communication, model training, and so on.

#### Ii-B1 Attack Models and Objective Evaluation Metrics

Privacy metricTo assess the ability to protect a speaker's identity in different scenarios, the ASV performance in terms of the equal error rate (EER) is computed as the primary privacy metric by using language-matched ASV evaluation models. This metric is calculated under the four attack models shown in the lower left of Fig. 2. The attackers are assumed to have access to a few original or anonymized utterances for each speaker, called _enrollment_ utterances, and to have different levels of knowledge about the SAS:

* _Unprotected_: No anonymization is applied, and attackers verify the original test trials against the original enrollment data by using an ASV system trained on the original dataset, denoted \(ASV_{\text{eval}}\).
* _Ignorant_: Attackers are unaware of the anonymization strategy used for the test trial utterances; instead, they use the original enrollment data and \(ASV_{\text{eval}}\) to infer a speaker's identity.
* _Lazy-informed_: Attackers use a similar SAS without accurate parameters to anonymize their enrollment data, and they use \(ASV_{\text{eval}}\) to detect a speaker's identity.
* _Semi-informed_: The only difference from _Lazy-informed_ is that the attackers use \(ASV_{\text{eval}}^{\text{anom}}\), a more powerful version trained on anonymized speech, to reduce the mismatch between the original and anonymized speech and infer the speaker's identity.

Primary utility metricTo assess how well speech content is preserved in anonymized speech, the ASR performance in terms of the word error rate (WER) is computed as a primary utility metric by using language-matched ASR evaluation models. As illustrated in the lower right of Fig. 2, two ASR models are trained in the same way to decode the anonymized data: \(ASR_{\text{eval}}\), trained on the original data, and \(ASR_{\text{eval}}^{\text{anom}}\), trained on the anonymized data. This enables exploration of whether speech content can be maintained better by simply retraining with similarly anonymized data.

Secondary utility metricTo assess and visualize the preservation of voice distinctiveness, the gain of voice distinctiveness metric, \(\text{G}_{\text{VD}}\)[35, 36], is computed. Precisely, \(M=(M(i,j))_{1\leq i\leq N,1\leq j\leq N}\) is a voice similarity matrix for \(N\) speakers, where the similarity value \(M(i,j)\) for speakers \(i\) and \(j\) is formulated as follows:

\[M(i,j)=\text{sigmoid}\left(\frac{1}{n_{i}n_{j}}\sum_{\begin{subarray}{c}1 \leq k\leq n_{i}\text{ and }1\leq l\leq n_{j}\\ k\neq l\,i\,i=j\end{subarray}}\text{LLR}(x_{k}^{(i)},x_{l}^{(j)})\right), \tag{1}\]

Here, \(n_{i}\) and \(n_{j}\) are the numbers of utterances for each speaker; and \(\text{LLR}(x_{k}^{(i)},x_{l}^{(j)})\) is the log-likelihood ratio obtained by comparing the \(k\)-th utterance of the \(i\)-th speaker with the \(l\)-th utterance of the \(j\)-th speaker. These LLR scores are computed by probabilistic linear discriminant analysis (PLDA) of the \(ASV_{\text{eval}}\) model trained on the original data.

Three matrices are constructed from the original (o) and anonymized (a) data: \(M_{\text{oo}}\) from the original data, \(M_{\text{oa}}\) from the original and anonymized data, and \(M_{\text{aa}}\) from the anonymized data. The diagonal dominance \(D_{\text{diag}}(M)\) is computed as the absolute difference between the mean values of diagonal and off-diagonal elements:

\[D_{\text{diag}}(M)=\Bigg{|}\sum_{1\leq i\leq N}\frac{M(i,i)}{N}-\sum_{ \begin{subarray}{c}1\leq j\leq N\text{ and }1\leq k\leq N \\ j\neq k\end{subarray}}\frac{M(j,k)}{N(N-1)}\Bigg{|}. \tag{2}\]

Next, \(G_{\text{VD}}\)[35] is defined as the diagonal dominance ratio of the two matrices:

\[G_{\text{VD}}=10\log_{10}\frac{D_{\text{diag}}(M_{\text{aa}})}{D_{\text{diag }}(M_{\text{oa}})}, \tag{3}\]

Fig. 2: Speaker anonymization task. A user anonymizes original speech to hide his or her identity before publication, and attackers use biometric (ASV) technology and knowledge of the anonymization method to re-identify the original speaker’s identity.
Here, a gain of \(G_{\text{VD}}=0\) dB indicates that voice distinctiveness is preserved on average after anonymization, while a gain above or below 0 dB corresponds respectively to an average increase or decrease in voice distinctiveness.

An ideal anonymization system should achieve high EERs (close to 50%) in the _Ignorant_, _Lazy-informed_, and _Semi-informed_ scenarios to protect the speaker's information. In addition, the WER should be as low as for the original speech, and \(G_{\text{VD}}\) should be close to 0 dB to preserve voice distinctiveness.

### _Existing Speaker Anonymization Approaches_

#### Ii-B1 Digital Signal Processing (DSP) Methods

A simple approach [14] that does not require training data is to change speaker attributes with distortion of the spectral envelope by using McAdams coefficients [37] to randomly shift the positions of formant frequencies. Widening of formant peaks [15] further distorts the spectral envelope. Data-driven formant modification can also be applied by using the formant statistics of desired speakers [16] or time-scale algorithms [18]. Phonetically controllable anonymization [17] modifies a speaker's vocal tract and voice source features, with a focus on F0 trajectories. Although these methods perceptually manipulate the speech signal, previous works have indicated that powerful attackers can effortlessly recover speaker identities [11, 12, 38].

#### Ii-B2 Disentangled Representation Methods

A typical approach based on disentangled representation learning, called x-vector based anonymization, is used as the primary baseline in the VPC [10, 11, 12, 13]. It extracts speaker representations and linguistic features by using a pretrained TDNN-based ASV system [28] and ASR AM based on a factorized time-delay neural network (TDNN-F), respectively. Then, to hide the original speaker's information, a selection-based speaker anonymizer [30] replaces the original x-vector with the mean vector of a set of randomly selected speaker vectors from an external pool of English speakers. Specifically, given a centroid of source speaker vectors from one speaker, the cosine distance is used to find the 200 farthest centroids in an external speaker vector pool, and 100 of those are randomly selected and averaged to obtain an anonymized speaker vector [30]. Finally, an SS AM generates mel-filterbank features from the anonymized pseudo x-vector, F0, and linguistic features, and an NSF-based waveform generator to synthesizes anonymized speech.

Because this disentanglement-based method is more effective at protecting speaker identities than the DSP-based methods discussed in Section II-B1 [38, 12], most speaker anonymization studies have followed a similar framework. Improvements mainly come from two sources:

_Improved speech disentanglement_: Some works [39, 40, 41] have argued that the disentangled linguistic information extracted from the language-specific ASR AM and F0 still contain speaker information. Accordingly, they modify the F0 and linguistic information to remove the residual speaker identity.

_Improved speaker vector anonymization_: Other researchers have modified the original x-vector in ways that increase the privacy protection ability. Perero-Codosero _et al._[42] transformed an original x-vector to an anonymized one by using an autoencoder with an adversarial training strategy to suppress speaker, gender, and accent information. This requires labels for the speaker identity, gender, and nationality. Turner _et al._[43] sampled anonymized x-vectors from a Gaussian mixture model in a space reduced by principal component analysis (PCA) over an external pool of speakers, which preserves the distributional properties of the original x-vectors. More recently, Meyer _et al._[24] used a generative adversarial network to generate artificial speaker embeddings of a vector pool. The anonymization stage requires manual search to find vectors that are dissimilar to the anonymized one. Chen _et al._[44] did the original speaker identity by adding adversarial noise to the original speaker vectors while distorting the speech.

In addition to their own limitations, each of these approaches requires large amounts of transcribed English training data to train the ASR AM model, which prevents direct use for unseen language speaker anonymization. By contrast, the SSL-based, language-independent SAS [33] shown in Fig. 1--comprising a HuBERT-based soft content encoder [45], ECAPA-TDNN speaker encoder [29], F0 extractor and HiFi-GAN decoder [32]--can anonymize speech from any language without relying on text labels or other language-specific resources. However, it suffers from a remaining limitation of selection-based anonymizers according to previous results [11, 12, 13, 33, 34]: the distribution of the external speaker pool significantly affects anonymized speakers, and the averaging of vectors from the speaker pool reduces voice distinctiveness.

## III Proposed OHNN-Based Anonymizer

To mitigate the problems with existing approaches, we propose the OHNN-based anonymizer shown in Fig. 3. Hence, this section formulates speaker anonymization as a constrained optimization problem, describes a general form of the proposed anonymizer, and explains the implementation details.

### _Problem Formulation_

The training set \(\{(\mathbf{x}_{i}^{o},y_{i}^{o})\}_{i=1}^{M}\) comprises \(M\) speaker vector \(\mathbf{x}_{i}^{o}\) and the corresponding speaker label \(y_{i}^{o}\). The speaker vector \(\mathbf{x}_{i}^{o}\in\mathbb{R}^{d}\) is a \(d\)-dimensional segment-level speaker embedding obtained from an ECAPA-TDNN pretrained on the original

Fig. 3: Framework for an OHNN-based anonymizer. The \(\mathbf{x}_{o}\) are original speaker representations extracted from a pretrained ECAPA-TDNN, which then pass through a transfer module \(f(\cdot)\) to produce the corresponding anonymized speaker representations \(\mathbf{x}_{a}\). The \(\mathbf{x}_{o}\) and \(\mathbf{x}_{a}\) are trained as different speakers. After training, \(\mathbf{x}_{a}\) is used as a pseudo-speaker vector to synthesize speech.
audio waveform. \(\mathbf{x}_{i}^{o}\) follows an unknown distribution \(\mathbf{x}_{i}^{o}\sim p_{\mathbf{x}^{o}}\).

Anonymized speaker vectors \(\mathbf{x}_{i}^{a}\in\mathbb{R}^{d}\) are obtained by transforming \(\mathbf{x}_{i}^{o}\) with a function \(f_{\Theta}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}\), written as follows:

\[\mathbf{x}^{a}=f_{\Theta}(\mathbf{x}^{o}). \tag{4}\]

Accordingly, the anonymized speaker vectors follow another distribution \(\mathbf{x}_{i}^{a}\sim p_{\mathbf{x}^{o}}\) or \(\mathbf{x}_{i}^{a}\sim p_{f_{\Theta}(\mathbf{x}^{o})}\).

An ideal speaker anonymization method should meet at least three constraints:

* Speaker privacy protection: \(\mathbf{x}_{i}^{o}\) and \(\mathbf{x}_{i}^{a}\) are dissimilar to hide the original speaker identity.
* Speaker diversity: \(\mathbf{x}_{i}^{a}\) has a unique speaker identity \(y_{i}^{a}\) to maintain the diversity of anonymized speech across different speakers.
* Distribution similarity: \(\mathbf{x}_{i}^{a}\sim p_{\mathbf{x}^{o}}\) satisfies the same distribution as \(\mathbf{x}_{i}^{o}\) to maintain the naturalness of the original speech.

The above constraints can be formulated as an optimization problem:

\[(\Theta,\Psi)^{*} =\arg\min_{\Theta,\Psi}\mathbb{E}_{\{\mathbf{x}^{o},\mathbf{y}^{ o}\}\in D}\Big{[}\lambda\mathcal{L}_{s}\big{(}\mathbf{x}^{o},f_{\Theta}( \mathbf{x}^{o})\big{)} \tag{5}\] \[\quad+\mathcal{L}_{c}\big{(}y^{o},g_{\Psi}(\mathbf{x}^{o});y^{a}, g_{\Psi}(f_{\Theta}(\mathbf{x}^{o}))\big{)}\Big{]},\] \[\quad\text{s.t.}\ \mathcal{D}\big{(}p_{\mathbf{x}^{o}},p_{f_{ \Theta}(\mathbf{x}^{o})}\big{)}<\epsilon, \tag{6}\]

where \(\lambda\) is a hyperparameter to balance the multi-objective function. \(\mathcal{L}_{s}\) is a similarity metric to optimize \(\Theta\) by minimizing the similarity of the original-anonymized pair, thus ensuring that the anonymized speaker vectors are dissimilar to the original ones to protect speaker privacy.

Next, \(g_{\Psi}(\cdot)\) denotes the classifier layer, and \(\mathcal{L}_{c}\) is its classification loss function to optimize \(\Theta\) and \(\Psi\) by minimizing the discrepancy between the sets of desired outputs, \(y^{o},y^{a}\), and predicted outputs, \(g_{\Psi}(\mathbf{x}^{o}),g_{\Psi}(\mathbf{x}^{a})\). The outputs may be defined for a multi-speaker classification task, in which the original and corresponding anonymized speaker vectors are intentionally treated as different target speaker classes. This means that all speaker vectors after anonymization are treated as different classes, as well as different classes from the original speakers to maintain speaker diversity.

Finally, \(\mathcal{D}\big{(}p_{\mathbf{x}^{o}},p_{f_{\Theta}(\mathbf{x}^{o})}\big{)}\) is the divergence between distributions of \(\mathbf{x}\) included in a training database before and after anonymization. This term ensures similarity between the distributions of the anonymized and original speaker vectors, with some tolerance \(\epsilon\). The Kullback-Leibler divergence (KLD) or other types of divergence are applicable.

### _General Form of Proposed Anonymizer_

Direct solution of Eqs. (5) and (6) for an arbitrarily designed DNN-based \(f_{\Theta}\) is difficult. Here, we propose an anonymizer that, with a few assumptions, always satisfies the constraint in Eq. (6) regardless of the value of \(\Theta\). In such a case, \(\Theta\) and \(\Psi\) can be optimized via Eq. (5) and a conventional gradient descent method.

Let \(\mathbf{\mu}_{\mathbf{x}^{o}}\in\mathbb{R}^{d}\) and \(\mathbf{\Sigma}_{\mathbf{x}^{o}}\in\mathbb{R}^{d\times d}\) be the mean and covariance matrix of \(p_{\mathbf{x}^{o}}\), respectively. Our proposed anonymizer \(f_{\Theta}(\cdot)\) can be written as follows:

\[\mathbf{x}^{a}=f_{\Theta}(\mathbf{x}^{o})=\mathbf{L}_{\mathbf{x}^{o}}^{-1} \mathbf{W}\mathbf{L}_{\mathbf{x}^{o}}(\mathbf{x}^{o}-\mathbf{\mu}_{\mathbf{x}^{o} })+\mathbf{\mu}_{\mathbf{x}^{o}}, \tag{7}\]

where \(\mathbf{L}_{\mathbf{x}^{o}}\) is a whitening matrix1 that satisfies \(\mathbf{L}_{\mathbf{x}^{o}}^{-1}\mathbf{L}_{\mathbf{x}^{o}}^{-1}=\mathbf{\Sigma}_{ \mathbf{x}^{o}}\), and \(\mathbf{W}\in\mathbb{R}^{d\times d}\) is an orthogonal matrix that satisfies \(\mathbf{W}\mathbf{W}^{\top}=\mathbf{W}^{\top}\mathbf{W}=\mathbf{I}\). While \(\mathbf{\mu}_{\mathbf{x}^{o}}\) and \(\mathbf{L}_{\mathbf{x}^{o}}\) are determined by the data distribution, the values of \(\mathbf{W}\) are learned via Eq. 5.

Footnote 1: \(\mathbf{L}_{\mathbf{x}^{o}}\) is a whitening matrix. It can be derived from \(\mathbf{\Sigma}_{\mathbf{x}^{o}}\) by a matrix decomposition method used in, e.g., PCA or Cholesky whitening [46].

Before introducing the parameterization and optimization of \(\mathbf{W}\), we show that the proposed anonymizer satisfies \(\mathcal{D}\big{(}p_{\mathbf{x}^{o}},p_{f_{\Theta}(\mathbf{x}^{o})}\big{)}=0\) when \(p_{\mathbf{x}^{o}}\) is a Gaussian distribution \(\mathcal{N}(\mathbf{\mu}_{\mathbf{x}^{o}},\ \mathbf{\Sigma}_{\mathbf{x}^{o}})\). Equation (7) can be decomposed into three steps:

* Centering and whitening: \(\tilde{\mathbf{x}}^{o}=\mathbf{L}_{\mathbf{x}^{o}}(\mathbf{x}^{o}-\mathbf{\mu}_{ \mathbf{x}^{o}})\),
* Rotation: \(\tilde{\mathbf{x}}^{a}=\mathbf{W}\tilde{\mathbf{x}}^{a}\),
* De-whitening and de-centering: \(\mathbf{x}^{a}=\mathbf{L}_{\mathbf{x}^{o}}^{-1}\tilde{\mathbf{x}}^{a}+\mathbf{\mu} _{\mathbf{x}^{o}}\).

The centered and whitened speaker vector \(\tilde{\mathbf{x}}^{o}\) obviously follows a normal distribution \(\tilde{\mathbf{x}}^{o}\sim\mathcal{N}(\mathbf{0},\mathbf{I})\). As \(\mathbf{W}\) is an orthogonal matrix, \(\tilde{\mathbf{x}}^{a}\) also follows a normal distribution \(\mathcal{N}(\mathbf{W}\mathbf{0},\mathbf{W}\mathbf{W}^{\top})=\mathcal{N}( \mathbf{0},\mathbf{I})\). Through the affine transformation in the last step, we know that \(\mathbf{x}^{a}\sim\mathcal{N}(\mathbf{\mu}_{\mathbf{x}^{o}},\mathbf{L}_{\mathbf{x}^{o} }^{-1}\mathbf{L}_{\mathbf{x}^{o}}^{-1})=\mathcal{N}(\mathbf{\mu}_{\mathbf{x}^{o}}, \mathbf{\Sigma}_{\mathbf{x}^{o}})\). Hence, the defined anonymizer does not change the distribution, i.e., \(\mathcal{D}\big{(}p_{\mathbf{x}^{o}},p_{f_{\Theta}(\mathbf{x}^{o})}\big{)}=0\).

The above explanation also reveals the core idea of our proposed anonymizer: while it does not change the overall distribution, each speaker vector is rotated through an orthogonal transformation. The anonymized \(\mathbf{x}^{a}\) is guaranteed to be different from the original \(\mathbf{x}^{o}\) as long as \(\mathbf{W}\neq\mathbf{I}\). While an infinite number of orthogonal matrices can be applied for rotation, the optimal \(\mathbf{W}\) with respect to the criterion in Eq. (5) must be estimated through an optimization process.

In real applications, \(\mathbf{\mu}_{\mathbf{x}^{o}}\) and \(\mathbf{\Sigma}_{\mathbf{x}^{o}}\) of the test set data are unknown. They can be estimated by collecting multiple samples from the test domain if it is possible. Otherwise, we can either use the statistics from the training set or make some simplifications. Through preliminary experiments, we found an effective, simplified form:

\[\mathbf{x}^{a}=f_{\Theta}(\mathbf{x}^{o})=\mathbf{W}(\mathbf{x}^{o}-\mathbf{\mu}_{ \mathbf{x}^{o}}^{\text{train}})+\mathbf{\mu}_{\mathbf{x}^{o}}^{\text{train}}, \tag{8}\]

where \(\mathbf{\mu}_{\mathbf{x}^{o}}^{\text{train}}\) is the mean of the speaker vectors in the training set, and \(\mathbf{\Sigma}_{\mathbf{x}^{o}}\) is assumed to be an identity matrix.

### _Rotation Matrix Using Householder Reflection_

We now need a specific way to parameterize \(\mathbf{W}\) through gradient descent to guarantee that the learned \(\mathbf{W}\) is orthogonal. While many methods can be used, we found that one based on a Householder reflection [47] is efficient for DNNs. Without loss of generality, assume that \(\mathbf{W}\) is a product of multiple orthogonal matrices:

\[\mathbf{W}=\mathbf{W}_{1}\mathbf{W}_{l}...\mathbf{W}_{L}, \tag{9}\]
where each matrix \(\mathbf{W}_{l}\in\mathbb{R}^{d\times d}\) is given by

\[\mathbf{W}_{l}=\mathbf{H}_{qi}\,\mathbf{H}_{qi-1}\,...\,\mathbf{H}_{1},\ \ q_{l}\leq d, \tag{10}\]

Here, each sub-matrix \(\mathbf{H}_{qi}\) is constructed with a Householder reflection [47] given a non-zero vector \(\mathbf{v}_{qi}\in\mathbb{R}^{d}\) as follows:

\[\mathbf{H}_{qi}=\mathbf{I}-\frac{2}{\mathbf{v}_{qi}^{\top}\mathbf{v}_{q_{l}}} \mathbf{v}_{qi}\mathbf{v}_{qi}^{\top}. \tag{11}\]

The resulting \(\mathbf{H}\) is known to be an orthogonal matrix for any non-zero vector \(\mathbf{v}_{qi}\), i.e., \(\mathbf{H}^{\top}\mathbf{H}=\mathbf{H}\mathbf{H}^{\top}=\mathbf{I}\) and \(\mathbf{H}\neq\mathbf{I},\forall\mathbf{v}_{qi}\neq\mathbf{0}\). Accordingly, \(\mathbf{W}_{l}\) and \(\mathbf{W}\) are orthogonal and thus guaranteed not to be the identity matrix.

Equations (9-11) allow us to parameterize \(\mathbf{W}\) as \(\{\cdots,\mathbf{v}_{qi},\cdots\}\). We further propose two implementations, which differ in how they compute \(\mathbf{v}\):

1. _Random orthogonal Householder (ROH) reflection_: \(\mathbf{v}\) is treated as a learnable free parameter, i.e., \(\Theta=\{\cdots,\mathbf{v}_{qi},\cdots\}\), and each \(\mathbf{v}\) is randomly initialized and optimized using Eq. 5. The anonymization process is illustrated in Fig. 4(a).
2. _Learnable orthogonal Householder (LOH) reflection_: Each \(\mathbf{v}\) is transformed from a small NN given the input \(\mathbf{x}^{o}\). In such a case, \(\Theta\) is the set of the trainable weights in a set of small NNs. Fig. 4(b) illustrates an implementation in which each DNN has a single 1D convolution layer with 192 output channels and a kernel size of \(3\).

While both implementations ensure that the transformation matrix \(\mathbf{W}\) is orthogonal, the first approach assumes a global transformation for all the input speaker vectors. In contrast, the latter approach assumes that the transformation matrix varies according to the input.

### _Loss Functions_

Before delving into the details of the loss functions, we describe how to build batch data for an OHNN-based anonymizer. Let \(N\) be the batch size and \(C\) be the number of original speakers. Each mini-batch comprises \(N/2\) original samples: \([\mathbf{x}^{o},y^{o}]=\{(\mathbf{x}^{o}_{i},y^{o}_{i})\}_{i=1}^{N/2}\), where \(y^{o}_{i}\in[1,C]\) and \(N/2\) corresponding anonymized samples. \([\mathbf{x}^{a},y^{a}]=\{(\mathbf{x}^{a}_{i},y^{o}_{i}+C)\}_{i=(N/2)+1}^{N}\). Therefore, the number of speakers is \(2C\) during the training of an OHNN-based anonymizer.

We now explain the loss functions for learning the best values of \(\Theta\) and \(\Psi\) as defined in Eq. (5). For the classification loss \(\mathcal{L}_{c}\), we first consider the widely used AAM softmax loss [48, 49]:

\[\mathcal{L}_{c}=\mathcal{L}_{\text{AAM-softmax}}=-\frac{1}{N}\sum_{i=1}^{N} \log\frac{e^{\|\mathbf{w}_{qi}\|\cdot\|\mathbf{x}_{i}\|\cdot\cos(\theta_{ji},i +m_{1})}}{Z}, \tag{12}\]

where \(Z=e^{|\mathbf{w}_{qi}|\cdot\|\mathbf{x}_{i}\|\cos(\theta_{ji},i)}+\sum_{j\neq i \neq i}^{2C}e^{\|\mathbf{w}_{j}\|\cdot\|\mathbf{x}_{i}\|(\theta_{j,i})}\). \(\mathbf{w}_{j}\) is the \(j\)-th column of the weight in the fully-connected layer before the softmax layer, where \(\mathbf{w}\in\mathbb{R}^{d\times 2C}\); and \(\theta_{y_{i},i}\) is the angle between \(\mathbf{x}_{i}\) and the target class's weight vector \(\mathbf{w}_{y_{i}}\). After fixing the weight \(||\mathbf{w}_{y_{i}}||=1\) by \(\ell_{2}\)-normalization and rescaling \(||\mathbf{x}_{i}||\) to \(s\) to ensure that the gradient is not too small during training, we can write Eq. 12 as

\[\mathcal{L}_{c}=\mathcal{L}_{\text{AAM-softmax}}=-\frac{1}{N}\sum_{i=1}^{N} \log\frac{e^{s(\cos(\theta_{ji},i+m_{1}))}}{Z}, \tag{13}\]

where \(Z=e^{s(\cos(\theta_{ji},i+m_{1}))}+\sum_{j=1,j\neq i}^{2C}e^{s(\cos(\theta_{j,i}))}\).

By applying the effective AAM softmax loss, we add an extra margin penalty \(m_{2}\) to explicitly improve the discrepancy for original-anonymized (or anonymized-original) pair samples. The approach is called weighted additive angular margin (w-AAM) softmax. Let \(i\in[1,N]\) be the index of the original (or anonymized) sample in a mini-batch based on this batch data construction method. The corresponding anonymized (or original) sample is indexed by \((i+N/2)\%N\), where \(\%\) denotes the modulo operation. Then, the proposed w-AAM loss function can be written as follows:

\[\mathcal{L}_{c}=\mathcal{L}_{\text{w-AAM}}=-\frac{1}{N}\sum_{i=1}^{N}\log \frac{e^{s(\cos(\theta_{ji},i+m_{1}))}}{Z}, \tag{14}\]

where the factor \(Z\) is now defined as:

\[\begin{split} Z=& e^{s(\cos(\theta_{ji},i+m_{1}))}+e^ {s(\cos(\theta_{y(i+N/2)\%N,i}-m_{2}))}\\ &+\sum_{j=1,j\neq i,j\neq(i+N/2)\%N}^{2C}e^{s(\cos(\theta_{j,i} ))}.\end{split} \tag{15}\]

In our experiments, we set \(m_{1}=m_{2}=0.2\), \(s=30\) and compared the performance with settings of \(\mathcal{L}_{c}=\mathcal{L}_{\text{AAM}}\) and \(\mathcal{L}_{c}=\mathcal{L}_{\text{w-AAM}}\).

For the similarity metric \(\mathcal{L}_{s}\), we choose the cosine similarity2 given by \(\mathcal{L}_{s}(\mathbf{x}^{o}_{i},\mathbf{x}^{a}_{i})=\max(0,\cos(\mathbf{x} ^{o}_{i},\mathbf{x}^{a}_{i})-margin)\), we set \(margin=0\).

Footnote 2: [https://pytorch.org/docs/stable/generated/torch.nn.CosineEmbeddingLoss.html](https://pytorch.org/docs/stable/generated/torch.nn.CosineEmbeddingLoss.html)

## IV Evaluation

To evaluate the effectiveness of the SSL-based SAS using the proposed OHNN-based anonymizer under all the attack scenarios for English speaker anonymization, we followed the VPC evaluation plan [11, 12, 13] described in Section II. Then, we conducted anonymization experiments on Mandarin data to show that the proposed OHNN-based

Fig. 4: Two types of OHNN-based anonymizers.
anonymizer, which removes the requirement of an English speaker pool, can reduce the language mismatch encoded in anonymized speaker representations. As a result, better speech content preservation is achieved for unseen-language speaker anonymization.

### _Speaker Anonymization Dataset and Experimental Setup_

#### Iv-A1 Dataset

The SSL-based SAS was built using the following VPC standard datasets [11]: an ECAPA-TDNN speaker encoder trained on the _VoxCeleb-2_[50]; a HuBERT-based soft content encoder finetuned from a pretrained HuBERT Base model3 on _LibriTTS-train-clean-100_[51] to capture content representations; and a HiFi-GAN model trained on _LibriTTS-train-clean-100_[51].

Footnote 3: [https://github.com/pytorch/fairseq/tree/main/examples/hubert](https://github.com/pytorch/fairseq/tree/main/examples/hubert)

Unlike the selection-based anonymizer, which relies on an additional multi-speaker English dataset (_LibriTTS-train-other-500_) containing data from 1,160 speakers, the OHNN-based anonymizer's reuse a multi-speaker multi-language dataset (_VoxCeleb-2_), that is used to train the ECAPA-TDNN of the SSL-based SAS [33]. This large-scale dataset contains over 1 million utterances by 5,994 speakers of 145 different nationalities.

English speaker anonymization was evaluated on the official VPC development and test sets [11, 12, 13]. These two sets contain English utterances by several female and male speakers from the _LibriSpeech_ and _VCTK_[52] corpora. For the _Ignorant_ and _Lazy-informed_ conditions, we used the language-matched \(ASV_{\text{eval}}\) system provided by the VPC [11, 12, 13]. It was trained on the original _LibriSpeech-train-clean-360_ English dataset. For the _Semi-informed_ condition, we trained \(ASV_{\text{eval}}^{\text{anom}}\) system in the same way as \(ASV_{\text{eval}}\), but with anonymized speech data. Likewise, \(ASR_{\text{eval}}\) and \(ASR_{\text{eval}}^{\text{anom}}\) were trained with the same original and anonymized speech data, respectively.

The same anonymization systems used for English speaker were directly adopted for Mandarin speaker anonymization without training or fine-tuning on Mandarin data. The evaluation for Mandarin was conducted on a test set sampled from a 20-hour, multi-speaker Mandarin corpus called _AISHELL-3_[53]. The test set contains 4,267 utterances by 44 speakers. We split the utterances into test trial (88 utterances) and enrollment (4,179 utterances) subsets, which were used to produce 10,120 enrollment-test pairs for ASV evaluation, including 2,200 same-speaker and 7,920 different-speaker pairs. The ASV evaluation model \(ASV_{\text{eval}}^{\text{mand}}\) was an ECAPA-TDNN trained on the Mandarin datasets called _CN-Celeb-1 & 2_[54, 55]. The ASR evaluation model \(ASR_{\text{eval}}^{\text{mand}}\) was a publicly available ASR Transformer [56] trained on a 150-hour Mandarin ASR dataset, _AISHELL-1_[57].

#### Iv-A2 Experimental Setup

Table I lists notations for the different speaker anonymization approaches that we examined. **B1.a**, **B1.b**, and **B2** are the baseline systems from VPC 2022 [13]. **S-Select** denotes the SSL-based SAS using a selection-based anonymizer. **S-ROH** denotes a system obtained by replacing the selection-based anonymizer of **S-Select** with a random OH (ROH) anonymizer and keeping other components unchanged. Likewise, **S-LOH** indicates the use of a learnable OH (LOH) anonymizer. Noted that, hereafter, **S-ROH*** and **S-LOH*** refer to models trained with the w-AAM and cosine similarity losses.

For **S-Select**, the YAAPT algorithm [58] is used to extract the F0. The ECAPA-TDNN with 512 channels in the convolution frame layers [29] provides 192-dimensional speaker identity representations. The HuBERT-based soft content encoder [45] takes the CNN encoder and the first sixth transformer layers of the pretrained HuBERT base model as a backbone. It downsamples a raw audio signal into a 768-dimensional continuous representation, which is then mapped to a 200-dimensional vector by one projection layer to predict discrete speech units. These speech units are obtained by discretizing the intermediate 768-dimensional representations via _k_-means clustering4[59, 60]. The training procedures are detailed in [33]. For the selection-based anonymizer, attackers had different random seeds from users when randomly choosing 100 speaker vectors from the 200 farthest ones; thus, the attackers had different pseudo-speaker vectors.

Footnote 4: [https://github.com/pytorch/fairseq/tree/main/examples/textless_nlp/gslm/speech2unit](https://github.com/pytorch/fairseq/tree/main/examples/textless_nlp/gslm/speech2unit)

The OHNN-based anonymizer accepts 192-dimensional speaker representations extracted from a pretrained ECAPA-TDNN, which was the same here as the ECAPA-TDNN of the SSL-based SAS. We followed the VPC evaluation plan, in which attackers in the _Lazy-informed_ and _Semi-informed_ scenarios have partial knowledge of the speaker anonymizer. They are assumed to know the training dataset, structure, loss functions, and other training parameters of the user's OHNN

Fig. 5: Visualization of original and anonymized speaker vectors generated by the **S-Select** and **S-LOH*** anonymizers.
based anonymizer, except for the training seed to initialize the training weights. Specifically, the training seeds were 50 and 1986 for users and attackers, respectively5. Using knowledge of the OHNN-based anonymizer, an attacker trains a new anonymizer to anonymize speech. All the OHNN-based anonymizers were trained with a cyclical learning rate [61], which varied between 1e-8 and 1e-3, and the Adam optimizer [62] by using the SpeechBrain [56] toolkit based on PyTorch [63]. The number of iterations of one cycle was set to 130k. We fixed \(d=192\), \(L=12\) for both the ROH and LOH anonymizers, but we use \(q_{l}=192\) and \(q_{l}=50\) for the ROH and LOH training, respectively. The hyperparameter \(\lambda\) in Eq. 5 was set to 20 6.

Footnote 5: The training seeds can be any values as long as they are different for users and attackers.

Footnote 6: Source code and audio samples are available at [https://github.com/ni-yamagishilab/SSL-SAS](https://github.com/ni-yamagishilab/SSL-SAS)

### _Speaker Anonymization Experiments in English_

For the English experiments, first, we explored the difference between selection- and OHNN-based anonymizers by comparing the performance of **S-Select** and **S-LOH***. Then, we investigated different configurations for the OHNN-based anonymizer, including the losses and whether to explicitly use speaker information to optimize the Householder transformation. Finally, we compared SSL-based speaker anonymization using an OHNN-based anonymizer with other approaches, including the disentanglement- and DSP-based approaches.

#### Iv-B1 Comparison of Selection- and OHNN-Based Anonymizers

In the first experiments, we visualized the original and anonymized speech generated by **S-Select** and **S-LOH***** in terms of speaker embeddings, the cosine similarity of the speech pairs, and voice distinctiveness.

**Original and anonymized speaker embeddings**: To show the difference between the **S-Select** and **S-LOH*** anonymizers, we first applied t-distributed stochastic neighbor embedding (t-SNE) [64] to visualize the original and anonymized embeddings. The results are shown in Fig. 5. The speaker embeddings were extracted from 50 speakers in the _VoxCeleb-2_ training set, which are shown in different colors, and 10 utterances were randomly selected from each speaker. Clearly, the anonymized speaker vectors generated by **S-Select** were heavily dependent on the distribution of an external pool, whereas **S-LOH***** generated distinctive anonymized speaker vectors that followed the distribution of the original speaker vector space.

**Cosine similarity distribution on speech pairs**: Fig. 6 plots the cosine similarities between pairs of speaker vectors extracted from generated speech for all the test sets of _LibriSpeech_ and _VCTK_ on speech pairs provided by [12]. Depending on the attack condition, the speech can be original or anonymized generated by **S-Select** or **S-LOH*****. For the _Unprotected_ condition, shown on the left side of Fig. 6, the positive cosine similarity distributions (green) are close to 1, and the negative distributions (yellow) are close to 0, which indicates that the speaker vectors of the original speech were highly discriminative. To protect speaker privacy, an ideal SAS should push the positive score distributions toward the negative ones regardless of the attacker type.

On the right side of Fig. 6, the top part shows the score distributions for three attacker conditions with **S-Select**. There are much bigger overlaps of the positive and negative distributions for the _Ignorant_ condition than for the _Unprotected_

Fig. 6: Cosine similarities between pairs of the speaker vectors extracted from the generated speech of users and different attackers. Positive: paired utterances from the same speaker. Negative: paired utterances from different speakers.

Fig. 7: Voice similarity matrices for **S-Select** and **S-LOH***** on the female speakers in the LibriSpeech-dev and VCTK-dev datasets. The global matrix \(M\) for each system comprises three submatrices \(M_{\text{o}}\), \(M_{\text{oa}}\), and \(M_{\text{na}}\) defined in Section II-A1 via \(M=\begin{pmatrix}q_{\text{oa}}&\text{MnO}\\ M_{\text{oa}}&M_{\text{na}}\end{pmatrix}\).
condition, which means that **S-Select** achieved reasonable speaker privacy performance under the _Ignorant_ condition. Unfortunately, the overlaps are smaller for the _Lazy-informed_ and _Semi-informed_ conditions. This reveals the reason for the significant speaker privacy leakage under more powerful attack conditions. Moreover, most of the cosine similarity scores are very close to 1, which may pose a risk of reducing the diversity of the anonymized speakers.

The bottom right of Fig. 6 shows the score distributions for three attacker conditions with **S-LOH***. The overlaps of the positive and negative distributions are well magnified under all the attack scenarios. This verifies the effectiveness of our OHNN-based anonymizer in ensuring that the attackers cannot gain significant speaker privacy information from users. Furthermore, most of the cosine similarity scores are far from 1, indicating the diversity of the anonymized speakers.

**Comparison of gain of voice distinctiveness (\(G_{\text{VD}}\))**: Fig. 7 shows voice similarity matrices obtained for **S-Select** and **S-LOH***. The upper-left submatrix of each matrix \(M\) is \(M_{\text{oo}}\), and the distinct diagonal reflects the high voice distinctiveness within the original speech. The upper-right (or lower-left) submatrix \(M_{\text{oa}}\) reflects the voice similarity between the original and the anonymized speech, such that the diagonal disappears when they differ. The lower-right submatrix \(M_{\text{aa}}\) reflects the voice similarity within the anonymized speech, where a dominant diagonal appears if the anonymized speakers remain distinguishable [35]. There is a very weak dominant diagonal in \(M_{\text{aa}}\) for **S-Select**, indicating that voice distinctiveness was lost among the anonymized speakers. In contrast, the matrices for **S-LOH*** exhibit distinct diagonals in \(M_{\text{aa}}\), indicating that voice distinctiveness was preserved after anonymization.

In general, the **S-LOH*** anonymizer met the three constraints described in Section III-A: good privacy protection, voice distinctiveness, and naturalness of the speaker vector space from the above analysis and visualization.

#### Iv-A2 Effects of Various Components for Proposed OHNN-Based Anonymizer

The proposed OHNN-based anonymizer has two novel components: the loss functions and the Householder transformations. Table II summarizes the average EERs and WERs 7 under all attack scenarios with the selection-based anonymizer and two OHNN-based anonymizers with different losses.

Footnote 7: The EER weights and detailed results for each subset are given in Appendix A. Due to limited space, other results are moved to the appendix of the paper on Arxiv.

**Effect of the different losses**: For the proposed OHNN-based anonymizer, w-AAM+cos performed better than AAM+cos in terms of the EER under most attacker conditions. This was because the introduced margin of w-AAM expands the inter-class variance of original-anonymized pairs, thus increasing the dissimilarity.

**Effect of different Householder transformations**: Clearly, the LOH anonymizers generally achieved better EERs than the ROH did. This result supports the view that, instead of using a global transformation for ROH, the LOH is more flexible because it learns from the speaker embeddings and thus brings more discriminative information.

As for the WERs, those computed by \(ASR_{\text{eval}}^{\text{anom}}\) were consistently lower than those of \(ASR_{\text{eval}}\) for all systems. Although there was no obvious improvement in the WERs decoded by \(ASR_{\text{eval}}\) for the OHNN-based anonymizers, the WERs decoded by \(ASR_{\text{eval}}^{\text{anom}}\) were lower than those of the selection-based anonymizer. This implies that such utility degradation due to OHNN-based anonymizers can easily be offset by training ASR evaluation models on similar anonymized data. Meanwhile, all the OHNN-based anonymizers achieved similar WERs with \(ASR_{\text{eval}}\) or \(ASR_{\text{eval}}^{\text{anom}}\), which confirms that the orthogonality of ROH and LOH did not change the distributions of the original and anonymized speaker vectors. Furthermore, as expected, the OHNN-based anonymizers with different configurations yielded substantially higher \(G_{\text{VD}}\) values than the selection-based anonymizer did. Overall, the results in Table II confirm the findings described in Section IV-B1.

#### Iv-A3 Comparison of Various SASs Using Different Anonymizers

**Primary privacy and utility evaluation**: Table III lists the average EER and WER results for various SASs under all scenarios. To anonymize the speaker representations, **B1.a**, **B1.b**, and **S-Select** used the selection-based anonymizer, while **S-ROH*** and **S-LOH*** used the OHNN-based anonymizer.

First, we examine the results with the selection-based anonymizer. The first observation is that **S-Select**, which follows the VPC's **B1.a** and **B1.b** by using the selection-based anonymizer, could protect speaker privacy information almost as well as **B1.a** and **B1.b** could under the _Ignorant_ and _Lazy-informed_ conditions with \(ASV_{\text{eval}}\). At the same time, **S-Select
better preserved speech content information with \(ASR_{\text{eval}}\). This result is consistent with the findings in [33]. The second observation is that, as with **B1.a** and **B1.b**[13], the EERs for **S-Select** decreased from around 30% under the _Lazy-informed_ condition with \(ASR_{\text{eval}}\) to around 9% under the _Semi-informed_ condition with \(ASR_{\text{eval}}^{\text{anon}}\), indicating severe speaker privacy leakage.

Next, we examine the results with the proposed OHNN-based anonymizer integrated into different configurations. First, **S-ROH*** and **S-LOH*** could protect speaker information almost as well as the VPC baselines (**B1.a** and **B1.b**) could when facing the _Ignorant_ attacker. Moreover, for the _Lazy-informed_ and _Semi-informed_ attackers, it comfortably outperformed all the baseline systems, achieving over 40% EER. Second, among all the methods, **S-ROH*** and **S-LOH*** preserved speech content the best with \(ASR_{\text{eval}}^{\text{anon}}\), achieving even lower WERs than for original speech on average.

**Secondary utility evaluation**: The bottom of Table III lists the results for the average gain of voice distinctiveness, \(\text{G}_{\text{VD}}\). They indicate that our proposed **S-ROH*** and **S-LOH***** achieved much better preservation of voice distinctiveness than the SASs using the selection-based anonymizer.

### _Speaker Anonymization Experiments in Mandarin_

Table IV lists the EERs and CERs for the Mandarin test dataset. The trends for the selection-based anonymizers and OHNN-based anonymizers with different losses were remarkably similar to those observed for English. The proposed OHNN-based anonymizers increased the ASV EER to around 33% under the _Ignorant_ scenario and over 34% under the _Lazy-informed_ scenario, suggesting that speaker information was well-protected. Meanwhile, the CER on the anonymized speech decreased to less than 18% with the OHNN-based anonymizers, suggesting improved utility. This was expected, because we previously demonstrated [34] that the increased CER (18.92%) for the baseline **S-Select** is due to language mismatch with pseudo-speaker vectors obtained from an English speaker pool. This mismatch can be mitigated by using OHNN-based anonymizers trained on large-scale, multi-speaker, and multi-language data.

## V Conclusions

This paper has proposed a novel OHNN-based speaker anonymization approach that rotates original speaker vectors into anonymized ones with a distribution following the original speaker vector space. To guarantee good privacy protection and voice distinctiveness, AAM/w-AAM and cosine similarity loss functions were introduced to ensure distinctive anonymized speaker vectors. Experiments on English VPC datasets demonstrated that the proposed model protects speaker privacy while maintaining speech content: it achieved either the best or near-best performance published to date in terms of privacy and utility metrics under all attack scenarios. Comparison of the cosine similarities between pairs of speaker vectors extracted from the generated speech with a commonly used selection-based anonymizer and the OHNN-based anonymizer further verified that our proposed method can effectively reduce privacy leakage when facing different attackers, while improving the diversity of anonymized speakers. Experiments on the Mandarin _AISHELL-3_ datasets additionally confirmed that our OHNN-based anonymizer reduces language mismatch and can be successfully adopted for unseen language anonymization tasks.