Training Universal Vocoders with Feature Smoothing-Based Augmentation Methods for High-Quality TTS Systems
Jeongmin Liu Eunwoo Song


8.710.6


    Marco Mellia
     
================




§ ABSTRACT 
    While universal vocoders have achieved proficient waveform generation across diverse voices, their integration into text-to-speech (TTS) tasks often results in degraded synthetic quality.
    To address this challenge, we present a novel augmentation technique for training universal vocoders.
    Our training scheme randomly applies linear smoothing filters to input acoustic features, facilitating vocoder generalization across a wide range of smoothings.
    It significantly mitigates the training-inference mismatch, enhancing the naturalness of synthetic output even when the acoustic model produces overly smoothed features.
    Notably, our method is applicable to any vocoder without requiring architectural modifications or dependencies on specific acoustic models. 
    The experimental results validate the superiority of our vocoder over conventional methods, achieving 11.99% and 12.05% improvements in mean opinion scores when integrated with Tacotron 2 and FastSpeech 2 TTS acoustic models, respectively.

0 
    While universal vocoders have demonstrated proficient waveform generation across diverse voices, their integration into text-to-speech (TTS) tasks often results in degraded synthetic quality.
    To address this challenge, we present a novel augmentation technique for training universal vocoders.
    Our training scheme randomly applies linear smoothing filters to input acoustic features, facilitating vocoder generalization across a wide range of smoothings.
    It significantly mitigates the training-inference mismatch, enhancing the naturalness of synthetic output even when the TTS acoustic model produces overly smoothed acoustic features.
    Notably, our method is applicable to any neural vocoder without requiring architectural modifications or dependencies on specific acoustic models.
    The experimental results validate the superiority of our vocoder over conventional methods, achieving 11.99% and 12.05% improvements in mean opinion scores when integrated with Tacotron 2 and FastSpeech 2 TTS acoustic models, respectively.


0 
    We present a novel feature smoothing augmentation technique for training universal vocoders, specifically aimed at mitigating the training-inference mismatch in text-to-speech (TTS) systems.
    While universal vocoders have demonstrated proficient waveform generation across diverse voices, their integration into TTS frameworks often results in degraded synthetic quality.
    This degradation mainly arises from the acoustic model's generation of smoothed acoustic features, inadequately covered during the vocoder's training.
    To address this challenge, our proposed training method employs random augmentation of input acoustic features using linear filters.
    This approach facilitates vocoder generalization across a wide range of smoothings, thereby enhancing the naturalness of synthetic output even when the acoustic model produces overly smoothed acoustic features.
    Notably, our method is applicable to any neural vocoder without necessitating architectural modifications or dependencies on specific acoustic models.
    Experimental results validate the superiority of our vocoder over conventional methods, achieving mean opinion scores of 4.11 and 3.44 when integrated with Tacotron 2 and FastSpeech 2 TTS acoustic models, respectively.




§ INTRODUCTION


    Recent advancements in modeling capacity have led to the development of universal vocoders capable of generating high-fidelity speech waveforms.
    Trained on diverse audio samples recorded from various environments, these vocoding models accommodate a wide spectrum of voices, languages, and styles <cit.>. 
    However, integrating them with text-to-speech (TTS) acoustic models remains challenging due to separate training processes, potentially resulting in degraded synthesis quality caused by the acoustic model's tendency to produce overly smoothed features.

    One straightforward solution is to fine-tune the vocoder using acoustic features generated by the corresponding acoustic model <cit.>.
    However, this approach compromises the vocoder's universality, because different acoustic models trained by different speakers or styles necessitate distinct fine-tuned vocoders, requiring substantial deployment resources and time.
    Alternatively, fully end-to-end TTS models have been proposed <cit.>, whereby the joint training of the acoustic and vocoding models can avoid the training-inference mismatch problem.
    However, this approach may lose the flexibility to control the acoustic characteristics of 0 (polisih) the output speech, as the model often relies on implicit latent representations instead of explicit acoustic features.

    To address these limitations, we propose a novel feature augmentation strategy to preserve the vocoder's universality while mitigating quality loss within the TTS framework.
    This method involves applying linear smoothing filters to input acoustic features, approximating their distributions with those generated by the acoustic model (i.e., smoothed along the time and frequency axes). 
    Specifically, the size of the smoothing filters is randomly selected for every training step, exposing the target vocoder to various levels of smoothing.
    Consequently, the vocoder gains enhanced adaptability to the acoustic model's over-smoothing problem without necessitating fine-tuning or architectural modifications.

    Our proposed method offers the advantage of applicability to any type of universal vocoder.
    In particular, we focus on the UnivNet model
    0
    [
        In the experiments, the effectiveness of the proposed method is also verified with the HiFi-GAN vocoder <cit.> as well.
    ] 
    <cit.>, a generative adversarial network (GAN)-based universal neural vocoder. 
    We enhance its performance with the additional harmonic-noise (HN) architecture <cit.> for stable harmonic production and two discriminators, namely, multi-scale short-time Fourier transform (MS-STFT) <cit.> and collaborative multi-band (CoMB) <cit.>, for improving training accuracy.
    The experimental results demonstrate that our universal vocoder trained with the proposed method outperforms the traditional approaches across different TTS systems.

    
    


§ METHODS




 §.§ Neural vocoders



    GAN-based neural vocoders <cit.> are generative models that comprise a generator G and a discriminator D <cit.>.
    As illustrated in Figure <ref>, these models are designed to learn the distribution of time-domain speech waveforms 𝐱 conditioned on their corresponding ground-truth acoustic features 𝐂, with the following objectives: 
    
    min_G 𝔼_𝐳∼𝒩(0, 𝐈), (𝐱,𝐂)[ L_G(𝐳, 𝐱, 𝐂) ], 
       min_D 𝔼_𝐳∼𝒩(0, 𝐈), (𝐱,𝐂)[ L_D(𝐳, 𝐱, 𝐂) ],

    where L_G and L_D are the following losses of the generator and the discriminator, respectively:
    
    L_G(𝐳, 𝐱, 𝐂)
                   = d[ D(G(𝐳 | 𝐂)), 1 ] + λ L_aux(G(𝐳 | 𝐂), 𝐱), 
    
            L_D(𝐳, 𝐱, 𝐂)
                   = d[ D(𝐱), 1 ] + d[ D(G(𝐳 |𝐂)), 0 ],

    where d is a distance function (e.g., L_2 distance), and λ is the weight for an auxiliary loss L_aux.

    

    


 §.§ Proposed smoothing augmentation method



    Previous research has demonstrated the technical feasibility of universal vocoding through the utilization of numerous training samples <cit.>. However, 0 (polish) the TTS systems often struggle to produce high-quality speech due to the exposure bias problem.
    This issue arises because the vocoder, having been trained solely on ground-truth mel-spectrograms, lacks robustness in handling the over-smoothing tendencies of acoustic models.
    
    To address this challenge, we propose a smoothing augmentation method designed to mitigate the exposure bias problem effectively within the TTS framework. 
    As shown in Figure <ref>, the key idea is to train the universal vocoder conditioned on smoothed mel-spectrograms that closely mimic the distribution of those generated by the acoustic models.
    To approximate the target distribution, we simulate the conditional acoustic feature by applying a linear smoothing filter, as follows:
    
    𝐂̃=𝐇*𝐂,

    where * denotes the 2-dimensional convolution operation and 𝐇 represents the smoothing filter designed to cover a range of smoothings from the acoustic models.
    Instead of using 0 (polish) the ground-truth mel-spectrograms, the universal vocoder is conditioned on these smoothed features and optimized with modified objectives, as follows:
    
    
    min_G 𝔼_𝐳, (𝐱, 𝐂̃)[ L_G(𝐳, 𝐱, 𝐂̃) ], 
       min_D 𝔼_𝐳, (𝐱, 𝐂̃)[ L_D(𝐳, 𝐱, 𝐂̃) ].

    This process does not require any fine-tuning procedure or modification of the network architecture, allowing 0 (polish) for compact training without additional deployment resources.
    Furthermore, as our training scheme exposes the vocoder to a diverse range of smoothings, the model becomes more generalized to the acoustic model's smoothing errors in the generation step. Thus, the entire TTS system is empowered to produce more natural speech outputs.

    


    
        


  §.§.§ Filter design



    Among various types of smoothing filters, i.e., 𝐇 in Equation (<ref>), we employ a 2-dimensional triangular[
        In our preliminary experiments, it was noticeable that any type of linear LPF, e.g., rectangular LPF, could be used to compose 𝐇.
    ]
    low-pass filter (LPF), defined as follows:
    
    h_t,f
                =⌈ l_t/2 ⌉ - |  t - ⌈ l_t/2 ⌉|/⌈ l_t/2 ⌉^2 ·⌈ l_f/2 ⌉ - |  f - ⌈ l_f/2 ⌉|/⌈ l_f/2 ⌉^2 ,

    where l_t and l_f denote the filter sizes along 0 (polish) the time frame t and frequency bin f, respectively.

    To enhance the vocoder's robustness across a diverse range of smoothings, it is crucial to randomly vary the filter sizes l_t and l_f for every training step.
    Specifically, these parameters are randomly sampled based on the following distributions:
    
    l_t ∼ p(l; N_t), l_f ∼ p(l; N_f),
   
    where N_t and N_f denote the numbers of possible candidates for l_t and l_f, respectively;
    p(l; N) denotes a distribution that is mostly uniform, except for the non-smoothing case (l=1), as follows: 
    
    p(l; N) =
            
                p_g    for  l=1, 
    
                p_s    for  l ∈{ 3, 5, ⋯, 2N-1 },
            
       where  p_g+(N-1)p_s = 1.

    While p_g and p_s can have the same value, our early experiments indicated that increasing p_g beyond p_s, such as p_g=2/3, yielded improvements in the vocoder's synthetic quality.
    
    Notably, we set the filter sizes l_t and l_f to odd numbers to ensure symmetric filters.
    



  §.§.§ Analysis of smoothing augmentation


    To validate the generalization capacity of our proposed method, we examined the mel-spectral distance (MSD; dB), defined as the L2-norm of mel-spectral frames between the ground-truth mel-spectrogram and those predicted by the acoustic model or simulated by our smoothing filters.    
    In Figure <ref>, the orange dashed lines depict examples of MSD histograms corresponding to distinct sizes of smoothing filters. 
    The random sampling of these filter sizes during training guides the vocoder to accommodate various levels of smoothing.
    We believe that this enables the vocoder to effectively manage the overly smoothed features generated by acoustic models such as Tacotron 2 <cit.>, as indicated by the blue solid line.
    This observation is further supported by the mel-spectrograms presented in Figure <ref>, where the simulated features exhibit tendencies similar to those generated by the Tacotron 2 acoustic model.
    0 (polish) in both low- and high-frequency regions 는 삭제함 

    
    
    


§ EXPERIMENTS



 §.§ Datasets

    The universal vocoder was trained on an internal dataset comprising recordings in five languages (Korean, Japanese, Mandarin Chinese, English, and Spanish) spoken by 73 speakers.
    The dataset contained 219,407 utterances (about 269 hours), with 5% reserved for validation.
    Each waveform was sampled at 24 kHz and quantized by 16 bits.    
    For acoustic model training, we randomly selected four Korean speakers (two female, F1 and F2, and two male, M1 and M2) from the vocoder's training dataset (seen speakers) and two Korean speakers (one female, F3, and one male, M3) not included in the training set (unseen speakers).
    The corpus for seen speakers comprised 4,600 utterances (about 7.7 hours), while the corpus for unseen speakers contained 2,400 utterances (about 3.6 hours).
    Validation and testing utilized 6% and 3% of each corpus, respectively.

    We extracted 100-dimensional mel-spectrograms, covering 0 (polish) from 0 to 12 kHz, with a 256-length frame shift and a 1,024-length Hann window.
    Additionally, log-F0 and voicing flags, extracted using the PYIN algorithm <cit.>, were included to compose 102-dimensional acoustic features.
    Before training, all acoustic features were globally normalized using the mean and variance of the training set.
    
    

    


 §.§ Model details




  §.§.§ Vocoding model


    Despite the availability of many state-of-the-art vocoders, we opted for the UnivNet model <cit.>, thanks to its competitive synthetic quality and fast generation speed.
    Our model followed the overall setup of the original UnivNet-c16 model[
        We used an open-source implementation at the following URL:
 <https://github.com/maum-ai/univnet/>.
    ], but we enhanced it by incorporating the following techniques, as depicted in Figure 4b:
    First, we introduced a harmonic-noise (HN) model into the generator <cit.>.
    The model received three inputs composed of F0-dependent sinusoidal, Gaussian noise, and a sequence of voicing information to enable the generator to efficiently learn the periodic and aperiodic behavior of the target waveform.
    We employed U-Net-style downsampling blocks <cit.> to align the sample-level inputs with the frame-level mel-spectrogram.
    
    Second, the discriminators were replaced with MS-STFT and CoMB discriminators, extending the vocoder's capabilities to capture complex audio features.
    The MS-STFT discriminator <cit.> facilitated analysis in the complex STFT domain, while the CoMB discriminator <cit.> allowed 0 (polish) for the capturing of the frequency band-wise periodic attributes of the target voice.
    These modifications were integrated into the model 0 (polish) , following their official implementations <cit.>. 
    Additionally, adjustments were made to the upsampling ratios of the upsampling blocks from {8,8,4} to {8,8,2,2} 0 (polish), to connect the last two blocks to the CoMB discriminator.
    Last, all activation functions in the generator were replaced with the Snake function <cit.>, known for its effectiveness in handling periodic signals <cit.>.
    We defined our enhanced vocoder as an eUnivNet for the remaining parts of the experiments.

    The generator and discriminators were trained for 600k steps using the AdamW optimizer <cit.>.
    During training, the model was conditioned on the ground-truth acoustic features for the first 450k steps and the features with randomly smoothed mel-spectrograms derived from our proposed method for the remaining 150k steps.  
    When sampling the size of smoothing filters in Equation (<ref>), we used six and three candidates for N_t and N_f, respectively.
    Additional training parameters included a weight decay of 0.01, a batch size of 32, and an exponentially decayed learning rate from 10^-4 with a decay rate of 0.99 per epoch.
    



  §.§.§ Acoustic model

    To generate acoustic features from the text, we employed Tacotron 2 (T2) with a phoneme-alignment approach <cit.> due to its stable generation and competitive synthetic quality.        
    The model received 364-dimensional phoneme-level linguistic features as inputs and predicted the corresponding phoneme duration through a combination of three fully connected layers and one long short-term memory (LSTM) network.
    By utilizing this predicted duration, the linguistic features were upsampled to the frame 0 (polish) - level and transformed into high-level context features via three convolutional layers, followed by a bi-directional LSTM network.
    Consequently, the T2 decoder autoregressively decoded those context features to reconstruct the target acoustic features.
    
    
    
    The model was initialized by Xavier initializer <cit.> and trained by Adam optimizer <cit.>.
    
    

    To faithfully evaluate the generalization capacity of the proposed method, we included a FastSpeech 2 (FS2) acoustic model <cit.> as the baseline.
    The FS2 model was trained similarly to the T2 model, but it used non-autoregressive encoder and decoder.
    More detailed setups for training the T2 and FS2 models were given in 0 (polish) the conventional work <cit.>.
    
    
    
0
    




 §.§ Evaluations

    
    

    We performed naturalness mean opinion score (MOS) tests to evaluate the TTS quality of the proposed method.    
    0
        For each speaker, we randomly selected ten utterances from the test set and synthesized speech samples via TTS systems with different universal vocoders as follows:
        
        
        
  * eUnivNet: Proposed enhanced UnivNet with both the HN-generator and the MS-STFT/Co-MB discriminators
        
  * eUnivNet-G: eUnivNet only with the HN-generator
        
  * eUnivNet-D: eUnivNet only with the MS-STFT/Co-MB discriminators
        
        
  * UnivNet: Baseline vanilla UnivNet-c32[
            We used an open source implementation at the following URL: 
 <https://github.com/maum-ai/univnet/>
        ] model <cit.>
        
        
  * HiFi-GAN: Baseline HiFi-GAN V1 model <cit.>.
        
        
        Twenty native Korean listeners were asked to rate the synthetic quality (in total, 10 utterances × 4 speakers × 20 listeners = 800 hits for each system) using the 5-point responses: 1=Bad, 2=Poor, 3=Fair, 4=Good, and 5=Excellent.
    Twenty native Korean listeners were asked to rate the synthetic quality using 0 (polish) the 5-point responses: 1=Bad, 2=Poor, 3=Fair, 4=Good, and 5=Excellent.
    For each speaker, we randomly selected 10 utterances[
        Generated audio samples are available at the following URL:
 <https://sytronik.github.io/demos/voc_smth_aug>.
    ] from the test set (in total, 10 utterances×4 speakers×20 listeners=800 hits for each system).
    The speech samples were synthesized by the different vocoders, as described in Table <ref>.
    
    Table <ref> shows the evaluation results with respect to various systems, and the analytic results are summarized as follows:
    Compared to the vanilla UnivNet-c32 (S6), our eUnivNet model (S3) performed significantly better despite having half the number of convolutional channels (e.g., 32 vs. 16).
    This highlights the importance of employing the HN-generator (S4) and the MS-STFT/CoMB discriminators (S5) to improve 0 (polish) the synthetic quality.
    Noteworthily, although the overall MOS score of the eUnivNet-HN-G model was lower than that of the vanilla UnivNet-c32, it offered benefits in terms of lower model size (36.0%) and fast inference speed (172.8%), as described in Table <ref>.

    
    Among the eUnivNet-based vocoders (S1, S2, and S3), the conventional training method (S1), i.e., separated from the acoustic model, performed the worst due to the exposure bias problem.
    Fine-tuning the vocoder with the generated features (S2) significantly addressed this limitation, but required time-consuming deployment resources, such as generating all features in the training set and retraining the vocoder speaker-dependently.
    Conversely, a single model trained solely with our smoothing augmentation method (S3) provided competitive synthetic quality without any dependency related to the speaker or acoustic model.
    
    The generalized performance was verified by changing the acoustic model (S7 vs. S8) and the vocoder (S9 vs. S10). The proposed method significantly improved the naturalness of synthesized speech.
    This trend was also observed in the other experiments in Table <ref>, where the proposed method robustly generated unseen speakers' voices compared to conventional methods.

    
    


§ CONCLUSION

    This paper proposes a novel feature smoothing augmentation method for training universal vocoders aimed at mitigating the mismatch between the acoustic model and the vocoder within the TTS framework.
    Our method introduced 0 (polish) the random linear filters to augment acoustic features, thereby approximating their distributions to those generated by acoustic models. 
    This approach enhances the generalization capacity of universal vocoders, enabling the generation of high-quality speech outputs even when the acoustic model produces overly smoothed features.
    The experimental results verified the superiority of our vocoder over 0 (polish) the conventional methods.
    Future research directions should explore extending this framework to other generation tasks, such as singing voice synthesis and music/audio generation.






§ ACKNOWLEDGEMENTS

We would like to thank Min-Jae Hwang, Meta AI, Seattle, WA, USA, for the helpful discussion.
This work was supported by Voice, NAVER Cloud Corp., Seongnam, Korea.

IEEEtran



