DiffEVC: Any-to-Any Emotion Voice Conversion with Expressive Guidance
Hsing-Hang Chou Yun-Shao Lin Ching-Chin Sung Yu Tsao Chi-Chun Lee



Limited but consistent gains in adversarial robustness by co-training object recognition models with human EEG
    Manshan Guo1,2,40000-0002-5506-6854 Bhavin Choksi 1,20000-0002-6475-4149 Sari Sadiya1,2,30009-0005-7482-3274 

Alessandro T. Gifford40000-0002-8923-9477 Martina G. Vilas1,2,50000-0002-1097-8534 

Radoslaw M. Cichy4jointly directed work0000-0003-4190-6071 Gemma Roig1,2,⋆0000-0002-6439-8076

    
======================================================================================================================================================================================================================================================================================================


§ ABSTRACT 
Emotional Voice Conversion (EVC) modifies speech emotion to enhance communication by amplifying positive cues and reducing negative ones. This complex task involves entangled factors like voice quality, speaker traits, and content. Traditional deep learning models like GANs and autoencoders have achieved some success in EVC by learning mappings or disentangling features but face challenges like instability and voice quality degradation. Diffusion models offer stable training and high-quality generation. We propose a diffusion-based EVC framework that disentangles emotion and speaker identity using mutual information loss and auxiliary models. An expressive guidance mechanism is introduced to improve emotion conversion while maintaining speaker traits. Experimental results demonstrate our approach's effectiveness for unseen speakers and emotions, achieving state-of-the-art performance in EVC tasks.



Emotion Voice Conversion, Diffusion, Disentanglement, Guidance




§ INTRODUCTION















Emotional Voice Conversion (EVC), which focuses on artificially modifying the emotional expression of speech signals, offers potential application scenarios such as enhancing conversational fluidity by amplifying positive emotional cues and attenuating harmful emotions, thereby reducing potential friction between individuals <cit.>. This cutting-edge research challenge required deep control of complex factors, including voice quality, emotional expressiveness, speaker traits, and language contents, that are mutually entangled and produce the final manifestation of human voices. With the effectiveness of deep learning techniques, building the EVC model on specific speakers or small-scaled parallel emotional speech has gained preliminary success in previous works <cit.>. In this work, different from the controlled scenarios, we target in-the-wild any-to-any EVC problems that require a comprehensive evaluation of each mentioned factor.


To solve the EVC problems, most of the deep learning methods models lie in the two realms, which are generative adversarial networks<cit.> and autoencoder-based models <cit.>. GAN-based methods, by leveraging adversarial mechanisms to learn direct mappings between data distributions of different emotional classes, can produce speech with high naturalness without compromising voice quality. However, the lack of robust modeling of fundamental vocal attributes can result in instability during the conversion process and diminish the accuracy of transforming the desired emotional class. Autoencoder-based approaches tackle this issue by introducing a disentanglement mechanism that separates linguistic features and speaker identity features from emotional representations, enabling better control over emotion conversion. Despite this advantage, effective disentanglement remains a challenging task, often leading to a degradation in the quality of the converted voice. The experiments in previous works point out important facts that an effective EVC learning method should jointly design the disentangled mechanism and the generation process. 

The diffusion model, compared to GAN-based methods, has recently garnered significant attention for its more effective generative capabilities, offering stabilized training and the ability to produce highly realistic samples in multiple application scenarios, in image generation<cit.>, voice conversion <cit.> and noise purification <cit.>. EmoConvDiff<cit.>, as the first work to address the in-the-wild EVC, has shown to be effective for learning the EVC problems by using large-scaled un-parallel emotional speech corpora such as MSP-PODCAST<cit.>. By leveraging the diffusion models, we further design disentangled mechanisms in two separate aspects to boost the effectiveness of the EVC model. First, we incorporate the disentangle loss into the diffusion EVC model training process. Second, with the reversed diffusion process, we further design the emotion-speaker disentangled guidance mechanism to enhance the expressiveness of the target emotion and mitigate the distortion of the source speaker trait.






 We propose a diffusion-based any-to-any emotional voice conversion framework for in-the-wild data. It conditions on pretrained speech representations for both emotion and speaker identity, which are further disentangled by proposed disentangled mechanism. Our objective evaluations on any-to-any EVC tasks for both in-the-wild and act-out data show that our disentangled mechanism successively make the conversion result more emotional. Morover, both objective and subjective evaluations demonstrate that the overall framework can generate more natural and high-quality samples while remain emotional comparing to state-of-the-art approaches.





§ METHODOLOGY




 §.§ Proposed Method





This work aims to solve the problem of in-the-wild any-to-any emotion voice conversion. Given a pair of source (X_c, s, e, or simply X_0) and reference (X_c, s,e) speech utterances, which the source X_c, s, e have language content c, speaker identity s and emotion expressiveness e; X_c, s,e have completely different language content, speaker identity, and emotion, our proposed method F aims to perform the conversion process X̂_c, s, e =F(X_c, s, e, X_c, s,e)  that preserve both content and speaker identity while transform emotion from e to e. The in-the-wild setting requires the source and reference speech utterance to be completely unseen in the training set.

Figure 1 shows the overall framework of our proposed method, which composed with a set of encoders, diffusion-based decoder, and expressive guidance method. <cit.>.






  §.§.§ Encoders


Three pre-trained encoders are used to capture the representations of linguistic content, speaker identity, and emotion expression.

Phoneme Encoding: To encode linguistic content X:=ϕ(X_0), we adapt pre-trained transfoer-based encoder ϕ(.) from <cit.> to convert input mel-spectrograms X_0 into speaker and emotion independent "average voice" mel features that replaced each phoneme-level mel feature with the corresponding average phoneme-level mel features. The encoder output  has the same dimension as the source mel-spectrograms X_0∈ℝ^n× T.

Speaker Encoding: To encode the speaker identity z_s:=S(.), we use a pretrained speaker verification model S(.)<cit.>, adapted from <cit.>, that generates a 256-dimension d-vector speaker representation.

Emotion Encoding: To encode emotional information z_e=E(.)∈ R^1024, we use a SSL-based SER system adapted from <cit.> that was built by fine-tuning the Wav2Vec2-Large-Robutst network on the MSP-Podcast(v1.7) dataset.
The E(.) network was built by fine-tuning the Wav2Vec2-LargeRobust network <cit.> on the MSP-Podcast (v1.7) dataset <cit.>.



  §.§.§ Diffusion-based decoder


We employ a diffusion framework based on stochastic differential equations (SDE), conditioned on the given linguistic content X̅, speaker representation z_s, and emotion representation z_c to generate high-quality speech. In this model, the diffusion process gradually transforms the average-voice mel-spectrogram X̅ into real sample X_0 in a forward process, while the reverse process generates X_0 from X̅. 













  §.§.§ Expressive Guidance


To amplify the effectiveness of the diffusion model on the converted speech, we further design the expressive guidance method that aims to manage the reversed diffusion process with positive and negative direction scores. During the inference stage, we modified s_θ with ŝ_θ as follows:

    ŝ_θ   =s_θ,neg   + λ_EG(s_θ,pos
    - s_θ,neg)





λ_EG with the value 1 controls the intensity of this guidance method and pushes the generation process away from the negative condition but toward the positive condition.  
The score  s_θ,condition are derived with different conditional embedding, including speaker (z_s) and emotion(z_e). In order to keep the naturalness of the converted emotional voice, we find the ideal pos condition = [Spk_src, Emo_ref] and the ideal neg condition = [Spk_ref, Emo_src]



  §.§.§ Disentangled Loss



In order to reduced the correlation between different speech representation, specifically emotion information and speaker identity, we minimize MI loss between reprsentations L_MI=Î(z_s, z_e), 
where Î represents the unbiased esitmation for vCLUB as described in <cit.>. The MI loss has shown to be effective to disentangled between different speech representations in several studies <cit.>.

To further preserve speaker identity and emotion information resides in the represenations after disentanglement, we use two  auxiliary supervised models that 1) predict speaker identity from disetnagled speaker representation z_s, and 2) predict emotion label (Neutral, Angry, Happy and Sad) and emotion attributes (Arousal and Valence) from disentangled emotion representation z_e. These models are trained to minimize loss L_style where the negative log likelihood loss is used for categorical prediction task and the concordance correlation coefficient loss is used for regression task.

In addition to L_diff for training diffusion-based decoder, we follow <cit.> to use a mel-spectrogram recontruction loss L_rec that measures the L_1-norm between X_0 and X̂_0, where X̂_0 is the single-step approximation relying on X_t, X̅, s_θ using Tweedie's formula <cit.>. We use λ_rec=(1-t^2) adapt from <cit.> to reduce importance of the loss with X_t containing more Gaussian noise for larger t.
The final objective function for our proposed method is as follows

    ℒ_Total=ℒ_diff+λ_MIℒ_MI+λ_guideℒ_guide+λ_recℒ_rec


where λ_MI and λ_guide are hyparameters to control the importance of respective loss.










§ EXPERIMENTAL SETUP AND RESULTS





 §.§ Experimental Setup
























  §.§.§ Implementation Details


Our proposed methodology is trained on in-the-wild MSP-Podcast corpus version 1.10  <cit.> that contains real podcast recordings (16kHz, 1ch) with naturalistic emotional expressions segmented in utterances. We select 53685 utterances labeled with four emotion and emotion attributes from 1385 labeled speakers. The Adam optimizer with a learning rate of 1×10^-4 is used for for updating all of the model parameters. We adopt pre-trained model parameters from <cit.> and fine-tuned it on MSP-Podcast for 368k iterations with a batch size of 32. We set λ_MI=0.1 and λ_guide=1 during training, and set λ_EG=1.25 for expressive guidance during inference.


  §.§.§ Evaluation Setup



We evalutae our methods on both in-the-wild dataset,  MSP-Podcast (v1.11), with real world scenrio and act-out dataset, ESD<cit.> with high quality recordings. We randomly sample 100 utterances of each emotion categories with unseen speakers from both dataset to conduct following experiments.



First, we perform any-to-any emotion voice conversion that includes all of the transformations between angry, happy, sad and neutral, besides transforming from emotional speech to neutral one. The experiments is conduct on both in-the-wild MSP-Podcast with real world scenrio and high quality act-out dataset (ESD). We compared methods under different training schemes, i.e., using only mathcolL_diff or using matchcolL_Total in <ref>. For evaluating effectiveness of our guidance method, we compared different settings of s_theta,neg by replacing original conversion input with either reference speaker, source emotionas or both. 
Second, We compared our propose method with five baseline models, i.e. 
StarGAN-EVC<cit.>, Seq2Seq-EVC<cit.>, Emovox<cit.> and Prosody2Vec<cit.> following conversion samples presented in Prosody2Vec[https://leyuanqu.github.io/Prosody2Vec/]. All of the comparison models are trained on act-out ESD datasets, while only Prosody2Vec utilized both predominant and in-the-wild dataset. The audio samples are avaliable in our demo page[https://henrychou36.github.io/DiffEVC/].


  §.§.§ Evaluation Metric


For both experiments, we incorporate non-intrusive objective evaluatin, i.e., UTMOS <cit.> for natureliness, DNSMOS <cit.> for speech quality (SIG) and overall signal quality (OVRL). Both methods are designed to predict mean opinion score (MOS) of subjective listenting tests. To access speaker similarity, speaker embedding cosine similarity (SECS) between extracted embeddings of source and gerneated speech based on Resemblyzer<cit.>. For emotion classification accuracy (ECA), we utilized speech emotion recogntion (SER) model for four class emotion classification pre-trained on both MSP-Podcast and ESD datasets based on emotion embedding from <cit.>. For second experiments, in addition to objective evaluation, we conduct subjective Evaluation using mean opinion score (MOS) for speech quality, natureliness (nMOS), emotion similarity (sMOS) between synthesis speech and the target emotional utterances with a 5-point scale ranging from 1 to 5. We also required listener to label primary emotion for ECA. The evaluation of first and second expereiments are presented in Table <ref> and Table <ref>, seperately.




 §.§ Experimental Results


From the result presented in Table <ref>, the proposed method with both disentaglement mechanism and expressive guidance method significantly improve in ECA with about 15% comparing to base model, will sustain roughly the same performance in naturalness and speech quality for both datasets. This shows that our method can help underlying model generate more expressive samples for any-to-any emotion voice conversion. 


Comparing guidance methods under different setting of unwanted representation with only disentanglement mechanism, we find out that by utilizing unwanted emotion, i.e., emotion information from the source utterance, we can further make the generated speech more emotional in terms of ECA with around 7% increments, while damages naturalness, speaker similarity and speech quality. On the otherhand, using unwanted speaker identity, i.e., speaker embedding from the reference speakers, decreases the emotion accurarcy while improving other criteria slightly. The downside of both methods can be alivieate by joint considering unwanted speaker and emotion information. This overall result shows that the guidance method can control the expressiveness of emotion voice conversion in either speaker identity or emotion information. Inspecting confusion matrix in Figure <ref>, we can find out that for both dataset, proposed method can effectively transfer any emotion into sad and happy, whereas conversion results toward angry would sometimes be recognized as happy, especially for act-out dataset. This demonstartes that the proposed method has room to be improved by incooperating act-out data and difference between angry and happy into training scheme.


Comparing source speech and synthesis speech regardless of different training and inference schemes, the synthesis results have overall better speech and audio quality, which shows that noise can be alleviate through speech decomposition and reconsturction. However, the level of noise will still inherently effects the quality and naturaliness of the generated speech.


Based on the result presented in Table <ref>, we can find out that our method has statistical significant improvement in terms of naturalness, speech and audio quality comparing to other models and is closed to the performance of target speech. In terms of expressiveness, we can find out that GAN-based method has highest speaker similarity with unrecognizable emotion, whereas Prosody2Vec generates most emotional speech while suffers from low naturalness and audio quality. Our method, on the other hand, generate emotional speech and preserves the quality without incooperate these data into training scheme.





§ CONCLUSION AND FUTRUE WORK


It is important to develop any-to-any EVC under real-wolrd scenerio. 
In this work, we propose an in-the-wild any-to-any emotion voice conversion that combines a disentanglement mechanism and expressive guidance and provide thorough evaluation with both objective and subjective tests. We show that our proposed method can effectively converts speech into different emotion for both in-the-wild and act-out dataset while having high speech and audio quality. Morover, comparing to other EVC in control environment, our methods generated natural and high quality emotional speech. 

IEEEbib


