Speaker and Style Disentanglement of Speech Based on Contrastive Predictive Coding Supported Factorized Variational Autoencoder
Yuying Xie Michael Kuhlmann Frederik Rautenberg Zheng-Hua Tan Reinhold Haeb-Umbach






Speaker and Style Disentanglement of Speech Based on Contrastive Predictive Coding Supported Factorized Variational Autoencoder

    Yuying Xie1, Michael Kuhlmann2, Frederik Rautenberg2 , Zheng-Hua Tan1, Reinhold Haeb-Umbach2

    1. Department of Electronic Systems, Aalborg University, Denmark 

    2. Department of Communications Engineering, Paderborn University, Germany 



    
===========================================================================================================================================================================================================================================================



§ ABSTRACT 
Speech signals encompass various information across multiple levels including content, speaker, and style. 
Disentanglement of these information, although challenging, is important for applications such as voice conversion.
The contrastive predictive coding supported factorized variational autoencoder achieves unsupervised disentanglement of a speech signal into speaker and content embeddings by assuming speaker info to be temporally more stable than content-induced variations.
However, this assumption may introduce other temporal stable information into the speaker embeddings, like environment or emotion, which we call style.
In this work, we propose a method to further disentangle non-content features into distinct speaker and style features, notably by leveraging readily accessible and well-defined speaker labels without the necessity for style labels.
Experimental results validate the proposed method's effectiveness on extracting disentangled features, thereby facilitating speaker, style, or combined speaker-style conversion. 



disentangled representation learning, voice conversion




§ INTRODUCTION

Disentangled representation learning aims to extract features which can represent different attributes in the observed data. 
In speech signal processing, disentangled representation learning offers solutions extensively <cit.>, including voice conversion.
One essential property of speech is that the speaker's traits should remain consistent within a single utterance, while the content rapidly varies over time. This characteristic then can serve as a prior for developing an end-to-end voice conversion model to extract disentangled content and speaker features separately, as voice conversion aims to change the speaker trait and preserve the content of an utterance.


Ebbers et al. <cit.> assume that the utterance-level information at the current frame is similar to that from the frames 1s before and after in the same utterance, and different between utterances in a batch. 
This assumption is then introduced as inductive bias in the fvae to disentangle speaker from content information via a completely unsupervised fashion, requiring neither speaker labels nor a transcription. 
Specifically, during training, cpc loss <cit.> is applied on the utterance-level embeddings, by regarding feature of current frame as anchor, features from the same utterance as positive pairs and from other utterance in the batch as negative pairs.
Adversarial training is working on the content embedding for independence of extracted utterance-level and content features. 
The extracted utterance-level feature in fvae are used to represent speaker identity only.



Even though utterance-level feature is used for speaker identity representation widely <cit.>, it can also contain other temporally stable information. 
For instance,  channel, emotion, or environment (e.g., room impulse response) information, should be similarly stable as speaker properties in one utterance.
Results in the works <cit.> about speaker embedding extraction also prove this point.
For instance, Xia et al. <cit.> utilize momentum contrastive (MoCo) <cit.> learning on speaker embedding extraction. 
Results on the Voxceleb test set show that extracted features cluster not only on speaker identity, but also on session id.
Besides, Cho et al. <cit.> apply DIstillation with NO labels (DINO) <cit.> to extract utterance-level embeddings, and results show that the extracted embeddings can not only represent speakers, but also emotions. 


Inspired by the fact that not only speaker traits change on the utterance-level, this work
further disentangles the utterance-level features from fvae <cit.> into speaker embedding and other temporally stable embeddings. 
We call the other temporally stable embeddings as `style' embeddings hereafter.  

The contribution of this work is as follows:
First, we posited that the speech signal is generated from three factors: speaker identity, style and content.
Second, this work extracts features representing these three factors in a hierarchical disentanglement manner.
The proposed method only needs speaker labels, which are cheap and easy to access. 
Third, the proposed method is applicable to different definitions of style embeddings.
In this work, acoustic environment and emotion have been chosen as different cases.
For the environment case, the performance of the proposed method is tested on both synthetic datasets and real unseen challenging datasets. 
For emotion, we evaluate the performance on a cross-language dataset.
Finally, we also investigated the application of the proposed method in style and speaker conversion using real-recorded datasets, both in clean and reverberant conditions.  https://yuxi6842.github.io/speaker_style_disen.github.io/Demos showcasing these applications are provided.[https://yuxi6842.github.io/speaker_style_disen.github.io/https://yuxi6842.github.io/speaker_style_disen.github.io/]




§ RELATED WORK


For fine-grained style control in speech synthesis and conversion, style  disentanglement is applied to obtain more expressive embeddings. Li et al. <cit.> propose a Tacotron2-based framework for cross-speaker and cross-emotion speech synthesis. 
Disentanglement is applied to get speaker-irrelavant and emotion-discriminative embeddings for emotion control. 
Although the model proposed in <cit.> can achieve cross-speaker emotion synthesis, only three speakers are contained in total.
Du et al. <cit.> uses disentanglement to obtain speaker, content and emotion embeddings simultaneously for expressive voice conversion.
While the model in <cit.> uses mutual information loss for disentanglement, the evaluation is not sufficiently comprehensive, especially lacking in disentanglement analysis.
Besides, the extracted speaker features perform speaker verification poorly in <cit.>.
Except for emotion, environment information like noise and reverberation has also been chosen as another controllable factor. 
Omran et al. <cit.> shows an information bottleneck (IB) based way to split speech and noise or reverberation. Masking is used on specific dimensions in the latent variable space, while reconstruction loss controls the particular information flow.
However, similar to other IB based methods, the performance depends heavily on bottleneck design in <cit.>. 
Another work in <cit.> uses disentanglement to realize environment conversion, in which only four seen environments are considered in their work.









§ PROPOSED METHOD




 §.§ Theoretical Explanation


Disentanglement aims to find codes to represent underlying causal factors <cit.>.
The original fvae actually assumes the log-mel feature 𝐗=[𝐱_1, …, 𝐱_T] is generated from independent latent variables 𝐒 and 𝐙 via a random generative process:

    𝐗=g_1(𝐒,𝐙),

in which 𝐒 and  𝐙 represent the utterance-level characteristic and content, respectively.



In this work, we further assume latent variable 𝐒 is generated from a random process with

    𝐒=g_2(𝐒^spk, 𝐒^sty),

in which 𝐒^spk and 𝐒^sty are independent latent variables to denote speaker identity and style in one utterance.
Thus eq (<ref>) can also be written as:

    𝐗=g_1(𝐒,𝐙)=g_1(g_2(𝐒^spk, 𝐒^sty),𝐙),

and can be further simplified as:

    𝐗=g(𝐒^spk, 𝐒^sty,𝐙).



Generation process of the proposed method illustrated in fig <ref>  is learned as eq. (<ref>).
Meanwhile, inference process in the proposed method is based on eq (<ref>) following a hierarchical structure as in fig <ref>.
The first step, inherited from fvae, learns to decompose observation 𝐗 into latent variables 𝐒 and 𝐙 according to the data inherent properties.
The second step further disentangles 𝐒 into 𝐒^sty and 𝐒^spk by introducing low-cost speaker labels in training. 
Adversarial learning is applied in both steps to achieve independence.
As self-supervised learning method have been used in the first disentanglement step, the proposed hierarchical structure reduces the labeling cost of disentangling multiple factors.




 §.§ Structure




The structure of the proposed method is shown in <ref>. 
Compared with FVAE, the new component, the proposed disentanglement module is highlighted in blue in <ref>.
The proposed module contains a speaker encoder, a speaker classifier, a style encoder and an adversarial speaker classifier.
Additionally, like FVAE, both vtlp <cit.> and in <cit.> are used to partially remove speaker information at the content encoder's input.
The utterance-level encoder, the content encoder, the adversarial CPC classifier, and the decoder are same with  <cit.>. 

The first disentanglement step, inherited from FVAE, is applied on 𝐗 to get utterance-level feature 𝐒=[𝐬_1, …, 𝐬_T] and content feature 𝐙=[𝐳_1, …, 𝐳_N]. 
The contrastive learning method is firstly applied to 𝐒, to ensure that the extracted utterance-level feature is temporally stable in one utterance.
Regarding embedding 𝐬_t as an anchor, 𝐬_t+τ from the same utterance as the positive pair, and 𝐬̃_t+τ from other utterances in the training batch ℬ as the negative pair, cpc loss is then calculated on 𝐒 as:

    L_cpc = -1/T - τ∑_t=1^T-τexp(𝐬_t+τ^T·𝐬_t)/∑_ℬexp(𝐬̃_t+τ^T·𝐬_t)

in which T is the frame length of input sequence 𝐗, and τ=80 (corresponding to 1s) is the time lag.
Variational autoencoder (VAE) <cit.> is applied on modeling content embedding 𝐙 as an information bottleneck approach for improving disentanglement.
The length of extracted 𝐙 is N. N≤ T as there may be downsampling in content encoder.
Each vector 𝐳_n in 𝐙 is regarded as a stochastic variable with prior p(𝐳_n)=𝒩(0, 𝐈) and posterior q(𝐳_n)=𝒩(μ_n, diag(σ_n^2)). Parameters of q(𝐳_n) are generated from the content encoder and optimized with KL divergence:

    L_kld = 1/N∑_n=1^NKL(q(𝐳_n)p(𝐳_n)).

Besides, an adversarial CPC classifier is applied on 𝐙 for independence of 𝐒, cf.  <cit.>.

The second disentanglement is then operated on the frame-wise utterance-level features 𝐒 via the proposed new module.
Speaker and style encoders have same structure built on one-dimensional CNN layers.
Denote
𝐒^=[𝐬^_1, …, 𝐬^_T] and 
𝐒^=[𝐬^_1, …, 𝐬^_T] as the output from speaker encoder and style encoder.
Speaker classification then works frame-wisely on 𝐬^_t to encourage extracting speaker information alone.
Meanwhile, frame-wise adversarial speaker classification is applied on 𝐬^_t to ensure that the extracted style embedding is less dependent on speaker trait.
Gradient reversal layer <cit.> is applied between the output layer of the style encoder and the adversarial speaker classifier. 
Cross-entropy loss is used here for classification on both 𝐬^_t and 𝐬^_t.
To get aggregated information over time and discard unnecessary details, vectors 𝐬^ and 𝐬^ are obtained by applying gap on 𝐒^ and 𝐒^.

The decoder takes as input a matrix where 𝐬^ and 𝐬^ are repeated N times along the time axis and then concatenated with the content embedding 𝐙.
To get sharper, less-oversmoothed output, reconstruction loss here is XSigmoid function <cit.>:

    L_rec = 1/T∑_t‖|𝐱̂_t - 𝐱_t| (2 σ(𝐱̂_t - 𝐱_t) - 1)‖_1  ,

in which σ(·) is the sigmoid function. 



Denote θ_enc^utt, θ_enc^spk, θ_enc^sty, θ_enc^cont are the parameter sets of utterance encoder, speaker encoder, style encoder and content encoder.
And the union of all encoders' parameter sets as
θ_enc=θ_enc^utt∪θ_enc^spk∪θ_enc^sty∪θ_enc^cont. 
The parameter sets of speaker classifier and adversarial speaker classifier are θ_clf^spk, θ_adv_clf^spk, while the parameter set of decoder is θ_dec
The total loss function is then written as:

    L(θ_enc,
            θ_dec,
            θ^spk_clf, θ^spk_adv_clf)  
        = L_rec(g_𝐗̂(𝐗; θ_enc,θ_dec),𝐗) 
            + λ_s L_cpc(g_𝐒(𝐗; θ_enc^utt))
        + β L_kld(g_𝐙(𝐗;θ_enc^cont))
             -  λ_z L_cpc (R(g_𝐙(𝐗; θ_enc^cont)))
            
       + L_CE(g_𝐏^spk(g_𝐒^spk(𝐗;θ_enc^utt,θ_enc^spk);θ_clf^spk),𝐘)
            
        - L_CE(g_𝐏^sty(R(g_𝐒^sty(𝐗; θ_enc^utt,θ_enc^sty));θ_adv_clf^spk),𝐘),

in which 𝐘 is the true speaker label one-hot matrix, 𝐏^spk and 𝐏^sty are speaker label predictions from speaker classifier and adversarial speaker classifier. 
Mapping g_𝐲(𝐱;θ) denotes that 𝐲 = g_𝐲(𝐱;θ), where 𝐱 is the input and θ is the parameter set in the corresponding neural networks.
For a better illustration of forward and backward behavior,
gradient reversal layer is represented via a 'pseudo-function' R(·) as <cit.>:

    R(𝐱) = 𝐱, dR(𝐱)/d𝐱 = - 𝐈,

where 𝐈 is an identity matrix. 
To balance the terms, λ_s, λ_z and β are coeffients in the total loss function.








§ EXPERIMENTS





 §.§ Datasets




For testing the separation of speaker from environment information, the model is trained and tested with artificially reverberated data. 
Besides, an unseen real-recorded dataset is chosen for testing only.
For the case of separating speaker from emotion information, a cross-language emotion dataset is used for training and testing.


LibriSpeech: 
The clean speech dataset of this work is based on LibriSpeech <cit.>.
The subsets train-clean-360 (921 speakers), train-clean-100 (251 speakers) and test-clean (40 speakers) are used and convolved with real-recorded RIRs.


LibriSpeech + air:
The air dataset <cit.> contains 107 real RIR recordings, and is used for data augumentation on the training dataset.
Each utterance from the LibriSpeech training set is convolved with 4 different randomly selected RIRs from air. 
Thus, the training dataset in the environment case is 4 times larger than the LibriSpeech training set.


VOiCES: 
To evaluate the performance on speaker verification, 
VOiCES <cit.> is chosen in this work for testing only. 
This dataset is recorded in four different rooms, with differently-located microphones and various directions of arrival between microphones and speakers.
In this work, only the speech dataset without noise from VOiCES_devkit is used for evaluation on real challenging acoustic environment. 

Emotion dataset: The emotional speech dataset (ESD) <cit.> contains studio-quality recordings of 10 English and 10 Mandarin speakers in neutral style and four acted emotions (happy, angry, sad, and surprise).
Each speaker utters the same 350 utterances in all five styles, totalling 35,000 utterances.
The subset which contains all 10 English speakers and 4 styles (neutral, happy, angry, and sad) is used for training, validation and evaluation (seen condition) firstly, with split of 85%/10%/5%, respectively.
Besides, the remaining style, surprise, is used for evaluation as an unseen emotion. Moreover, the Mandarin subset is used for evaluation when both speaker and language are unseen.





 §.§ Implementation Details

Following <cit.>, each update of the encoders, speaker classifier and decoder is followed by three exclusive updates of the adversarial CPC and adversarial speaker classifier.
We set β=0.01 and λ_s=λ_z=1.
We use the Adam optimizer and a learning rate of 5×10^-4. Both fvae and proposed method use eq (<ref>) as reconstruction loss for fair.

Environment
Training is based on the 'LibriSpeech + AIR' set with eq (<ref>) as loss function. 
The utterance-level encoder, content encoder and decoder in this work have the same structure as in <cit.>. 
The speaker encoder and style encoder both contain 3 one-dimensional CNN layers with stride equal to 1 and kernel size equal to 5 for all layers. The extracted speaker embedding and style embedding have both a dimension of 128. The speaker classifier contains one fully-connected (FC) layer, while the adversarial speaker classifier contains three FC layers with 128 hidden units each. The other settings are the same as in <cit.>.

Emotion
Training on ESD is performed for 100,000 iterations.
The networks are configured in the same way as in environment case, except for the input of content encoder: Inspired by HuBERT <cit.>, features from a CNN-based waveform extractor are used to replace the mel spectrogram.
The whole model is firstly pre-trained on LibriTTS <cit.>. 
Then fixed the content encoder, and fine-tuned the other modules in the whole model on ESD. 
The loss functions for pretraining and fine-tuning are the same, i.e. eq (<ref>).










 §.§ Evaluation under Environment Case

We firstly use t-SNE plots to visualize the extracted style and speaker embeddings.
Results are shown in fig <ref>.
The example shown is taken from the LibriSpeech test-clean set, convolved with 4 different RIRs from the AIR dataset. 
These figures show that the style and speaker features form well-defined clusters according to their  style and speaker labels, respectively, while speakers are spread across style clusters, and RIR-labels across speaker clusters.
This demonstrates that the proposed second disentanglement can separate speaker and environment information well.





Moreover, we test the extracted embeddings performance via speaker verification tests on the LibriSpeech test-clean set and VOiCES dev-kit dataset. 
Evaluation is based on equal error rate (EER) according to cosine similarity. 
Results are shown in <ref>.
When dealing with Librispeech test set, we found that there is slight recording environment change between some chapters in the original dataset.
Thus, EER is calculated separately for the case when same speaker pairs are selected within the same chapter (WC) and when selected across chapters (AC), with the latter giving larger EERs.
Compared with original fvae, the proposed speaker embeddings after the second disentanglement step result in an improvement in EER by a margin on the clean LibriSpeech dataset, cf. <ref>. 
Even though the proposed speaker embeddings reduce the gap between WC and AC, the exist of the gap may because: compared to the generated data, environmental changes in LibriSpeech can be negligible.  

Besides, the proposed method shows effective disentanglement on VOiCES dev-kit
dataset. The speaker embedding from proposed second disentanglement shows large improvement with an EER of 12.5% compared with 20.6% before the second disentanglement. 
Even though VOiCES is unseen during training, this result illustrates that the second disentanglement can help on extracting more robust speaker embeddings under challenging realistic situation.









 §.§ Evaluation under Emotion Case

Both speaker verification and speech emotion recognition results are shown in <ref>.
To check how the extracted features influenced by emotion, we calculate EER in speaker verification experiment under two different scenarios when positive speaker utterance pairs have the same emotion (within-emotion, WE) or not (across-emotion, AE).
For emotion recognition, we train a 3-layer FC classifier on the embeddings using the English speaker subset with all 5 emotional states.

In speaker verification experiments, when speaker and language are seen in training, speaker embedding performs superior compared with utterance-level features and style embedding.
Meanwhile, the evaluation of unseen speakers and language (Mandarin) subset, reveals that speaker embedding demonstrates stability and is less prone to the impact of emotional factors in speech. This is especially evident when positive pairs in speaker verification involve varying emotions.
For speech emotion recognition, style embedding shows slightly better performance when language and speaker are seen during training, and comparable performance with utterance-level features in Mandarin subset.










§ CONCLUSION




Inspired by the existing results that the utterance-level feature contains not only speaker identity, but other style attributes, this work proposed a hierarchical disentanglement method based on fvae.
Particularly, the utterance-level embedding from fvae is further decomposed into speaker and style features in this work.
Training of this disentanglement framework  needs speaker labels only, which are easy to get.
We evaluated the proposed method under two cases of style: environment and emotion. 
For the environment case, the extracted style embeddings show clusters on RIR labels.
The extracted speaker embeddings perform well for  speaker verification, when tested on a clean speech dataset and a real-recorded reverberation dataset, indicating the effectiveness of the proposed method.
The findings from the emotion dataset demonstrate that the extracted speaker embeddings become more distinctive in terms of speaker identification and less susceptible to emotional variations. 
We hope this work may contribute to future work including multi-factor voice conversion, and conversion under challenging environments.




IEEEbib



