HiGNN-TTS: Hierarchical Prosody Modeling with Graph Neural Networks for
  Expressive Long-form TTS
Recent advances in text-to-speech, particularly those based on Graph Neural
Networks (GNNs), have significantly improved the expressiveness of short-form
synthetic speech. However, generating human-parity long-form speech with high
dynamic prosodic variations is still challenging. To address this problem, we
expand the capabilities of GNNs with a hierarchical prosody modeling approach,
named HiGNN-TTS. Specifically, we add a virtual global node in the graph to
strengthen the interconnection of word nodes and introduce a contextual
attention mechanism to broaden the prosody modeling scope of GNNs from
intra-sentence to inter-sentence. Additionally, we perform hierarchical
supervision from acoustic prosody on each node of the graph to capture the
prosodic variations with a high dynamic range. Ablation studies show the
effectiveness of HiGNN-TTS in learning hierarchical prosody. Both objective and
subjective evaluations demonstrate that HiGNN-TTS significantly improves the
naturalness and expressiveness of long-form synthetic speech.
Dake Guo, Xinfa Zhu, Liumeng Xue, Tao Li, Yuanjun Lv, Yuepeng Jiang, Lei Xie
# Hignn-Tts: Hierarchical prosody modeling with graph neural networks for expressive long-form TTS

###### Abstract

Recent advances in text-to-speech, particularly those based on Graph Neural Networks (GNNs), have significantly improved the expressiveness of short-form synthetic speech. However, generating human-parity long-form speech with high dynamic prosodic variations is still challenging. To address this problem, we expand the capabilities of GNNs with a hierarchical prosody modeling approach, named HiGNN-TTS. Specifically, we add a virtual global node in the graph to strengthen the interconnection of word nodes and introduce a contextual attention mechanism to broaden the prosody modeling scope of GNNs from intra-sentence to inter-sentence. Additionally, we perform hierarchical supervision from acoustic prosody on each node of the graph to capture the prosodic variations with a high dynamic range. Ablation studies show the effectiveness of HiGNN-TTS in learning hierarchical prosody. Both objective and subjective evaluations demonstrate that HiGNN-TTS significantly improves the naturalness and expressiveness of long-form synthetic speech1.

Footnote 1: Speech samples: [https://anonymous-asru.github.io/HiGNN-TTS/](https://anonymous-asru.github.io/HiGNN-TTS/)

Dake Guo\({}^{1}\), Xinfa Zhu\({}^{1}\), Liumeng Xue\({}^{2}\), Tao Li\({}^{1}\), Yuanjun Lv\({}^{1}\), Yuepeng Jiang\({}^{1}\), Lei Xie\({}^{1}\)+\({}^{1}\)Audio, Speech and Language Processing Group (ASLP@NPU), School of Computer Science,

Northwestern Polytechnical University, Xi'an, China

\({}^{2}\) School of Data Science, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China Expressive long-form TTS, hierarchical prosody modeling, graph neural network, semantic representation enhancing

Footnote †: 1}\)Speech samples: [https://anonymous-asru.github.io/HiGNN-TTS/](https://anonymous-asru.github.io/HiGNN-TTS/)

## 1 Introduction

Text-to-Speech (TTS), aiming to generate human-like speech from text, has achieved dramatically advanced in naturalness with the proliferation of sequence-to-sequence (seq2seq) based neural approaches [1, 2, 3]. With the growing demand for anthropomorphic human-computer interaction, there has been increasing interest in generating speech with high expressiveness [4, 5, 6, 7], which exhibits high dynamic range in prosodic variation, including pitch and duration, etc. Human speech is expressive in nature, and proper expression rendering affects overall speech perception, which is essential for many TTS applications, such as newsreaders and voice assistants, etc. However, there is still a noticeable gap between synthetic speech and human speech in terms of expressiveness, particularly in long-form speech synthesis scenarios like audiobooks. Conventional TTS usually works on the prosody modeling of a single sentence. However, long-form speech usually contains multiple semantically coherent sentences, where the prosody of each individual sentence is also affected by its corresponding context. Therefore, in long-form speech generation, besides ensuring the coherence of the overall rhythm, it is also necessary to model the local fine-grained and global prosody in each sentence as well as the cross-sentence contextual prosody [8].

To address sentence-level global prosody, several works have brought attention to the extraction of a global style embedding from a given reference speech which has successfully captured the global style features of speech [9, 10]. In order to synthesize expressive speech without the need of auxiliary reference speech during inference, some methods attempt to obtain prosodic variations directly from text, which is more practical. By incorporating text-only predictions, the Text-Predicted Global Style Token (TP-GST) [11] extends the capabilities of GST [10], enabling the generation of style embeddings or style token weights solely based on textual input.

However, only sentence-level global prosody modeling lacks local fine-grained prosody information within a sentence, such as pauses and emphasis. By aligning the extracted prosody embedding sequence to the phoneme sequence, some works manage to model fine-grained prosody [12, 13, 14]. Moreover, the tight interplay of prosody and semantic information within sentences has led to a growing focus on leveraging pre-trained language models (such as BERT [15], XLNET [16]) to enhance fine-grained prosody representation [17, 18]. Further, to fully utilize the tree-structured syntactic information, some works have leveraged graph-based methods to model word-level prosody. Some studies incorporate BERT embeddings and employ Relational Gated Graph Networks (RGGN) to extract semantic information, and consequently enriches prosodic variations and improves expressiveness in the generated speech [19]. Instead of using BERT embedding, other work utilizes the average pooling of hidden layer representations from the acoustic model as