Paper Title: Can LLMs "Reason" in Music? An Evaluation of LLMs' Capability of Music Understanding and Generation.




Methodological Foundations of Modern Causal Inference in Social Science Research
    Guanghui Pan[Department of Sociology, University of Oxford. 41 Park End Street, Oxford, the United Kingdom, OX1 1HH. Email: guanghui.pan@sociology.ox.ac.ukguanghui.pan@sociology.ox.ac.uk. Manuscript in progress, do not cite or redistribute without permission.]
    August 5, 2024
========================================================================================================================================================================================================================================================================



§ ABSTRACT 
Symbolic Music, akin to language, can be encoded in discrete symbols. Recent research has extended the application of large language models (LLMs) such as GPT-4 and Llama2 to the symbolic music domain including understanding and generation. Yet scant research explores the details of how these LLMs perform on advanced music understanding and conditioned generation, especially from the multi-step reasoning perspective, which is a critical aspect in the conditioned, editable, and interactive human-computer co-creation process. 
This study conducts a thorough investigation of LLMs' capability and limitations in symbolic music processing. We identify that current LLMs exhibit poor performance in song-level multi-step music reasoning, and typically fail to leverage learned music knowledge when addressing complex musical tasks. An analysis of LLMs' responses highlights distinctly their pros and cons.
Our findings suggest achieving advanced musical capability is not intrinsically obtained by LLMs, and future research should focus more on bridging the gap between music knowledge and reasoning, to improve the co-creation experience for musicians.




§ INTRODUCTION












Large language models (LLMs), such as GPT-4, harness the power of deep learning to produce human-like text. These models, trained on vast datasets of textual content, have notably propelled advancements in natural language processing (NLP). They excel in complex language understanding and generation tasks including translation, sentiment analysis, question answering, and summarization, showcasing their reasoning capability with sophistication.

Large language models (LLMs), initially pre-trained on extensive textual corpora, can assimilate general linguistic patterns and structures. They are subsequently fine-tuned with domain-specific data, such as code and mathematical symbols, to enhance the adaptation to specific tasks. This refinement allows LLMs' proficiency to more accurately manage domain-specific terminology and complicated challenges like multi-step reasoning. Music Reasoning refers to the ability to estimate the varying harmonies, keys, rhythms, and other musical elements that are not explicitly annotated in a piece of music and are significant for music themes, progression, and styles <cit.>. The analogy between the reasoning process in music and mathematics suggests their structural similarities. Both disciplines fundamentally rely on patterns: music in rhythms, scales, and chord progressions, while mathematics involves sequences, symmetries, and geometric configuration. Moreover, music theory utilizes mathematical concepts to articulate intervals between pitches, chord structures, and the rhythmic temporal division  <cit.>, underscoring the intrinsic reasoning nature of the musical components. 

Music can be represented as sequences of symbols such as MIDI or ABC notation, rendering it suitable for processing by LLMs, which excel in long-context understanding and multi-step reasoning. These models are capable to dissect and generate intricate musical patterns encompassing melodic, harmonic, and rhythmic structures. LLMs also play a pivotal role in enhancing interactive music generation systems, where user inputs tailor the model's output, enriching the composing experience. While previous studies <cit.> have investigated LLMs in music tasks, detailed interpretations of the process remains less explored. This paper conduct an evaluation of four LLMs, GPT-4 <cit.>, Gemma-7B-it <cit.>, Llama2-7B-chat <cit.>, and Qwen-7B-chat <cit.>, assessing their capabilities on tasks related to symbolic music understanding and generation:


    
  * Music Understanding: 1) Music theory exercise; 2) Motif extraction; 3) Musical form extraction.
    
    
    
    
    
    
  * Music Generation: 1) Chord-conditioned music generation; 2) Melody harmonization; 3) Musical-form-and-motif-conditioned music generation
    
    
    
    
    
    


The task of "Musical Form & Motif Conditioned Music Generation" as described in Chatmusician <cit.> involves generating music that adheres to detailed prescribed conditions like form and motif. Figure <ref> illustrates this process: The prompt's green text specifies conditional constraints including the musical form, motif, and some musical elements (key, time signature, etc.). Under the prompt, the left sheet presents the human composer's work. The right sheets show ABC notations from different models alongside the reference. The Gemma-7B-it model merely replicates the provided motif, adding no new elements. Similarly, GPT-4 simply repeats the given condition. Qwen-7B-chat and Llama-7B-chat include correct musical elements and the motif but fail to capture the musical form "AB" and maintain the duration of a measure.














The main contributions of our paper are as follows: (1) we provide multi-step prompt engineering and explore how LLMs exhibit their reasoning capabilities with multi-step instructions in music understanding and generation tasks. (2) we assess four major LLMs on various symbolic music tasks, analyzing their reasoning in ABC sequences through quantitative statistical results and qualitative human assessment, including error analysis. The examples, hand-crafted prompts, and codes of data preprocessing are available at https://github.com/SylviaZiyaZhou/LLMs_music_reasoninggithub.



§ RELATED WORK

In this section, we summarize related works from two perspectives. First, we introduce previous studies on LLMs in the symbolic music domain, explaining their performance and evaluation methods in music understanding and generation tasks. Then, we discuss the application of LLMs in reasoning math problems and controllable creative text generation, highlighting similarities between the reasoning processes in music and math and the conditioned, open-ended nature of both music and text generation.



 §.§ LLMs in Symbolic Music Domain

This subsection reviews the application of LLMs in the symbolic music domain. Previous studies have focused on adapting LLMs for music understanding and generation. 

Chatmusician <cit.> uses continual pre-training and fine-tuning on LLaMA2 to understand and generate ABC notation music, without specialized music structures or tokenizers. SongComposer <cit.> collects a song pretraining dataset including lyrics, melodies and paired lyrics-melodies, employing 10K crafted QA pairs to enable LLMs to perform multiple music-related tasks such as lyric-to-melody conversion and song continuation. MusicAgent <cit.> integrates various music tools into a single system, though it lacks interaction among these tools. 
Most approaches view music creation as a linear process, which diverges from the multi-step approach humans use, limiting their applicability for generating creative works.  To mimic human creative processes,  ByteComposer <cit.> employs a four-step method to replicate the creative workflow of human composers: conception analysis, draft Composition, self-evaluation and modification, and human aesthetic selection. 
And designs an interactive agent system consisting of expert, generator, voter, and memory modules. What's more, they construct supervised fine-tuning data covering tasks of basic music theory conception, control code generation, music score evaluation and next-step planning. Despite being a significant step towards multi-step music creation with LLMs, it lacks a detailed discussion on the limits of LLMs at each stage.



 §.§ Reasoning and Controllable Generation with LLMs

"Reasoning" in NLP involves integrating various knowledge sources or contexts to generate new assertions, events, or actions <cit.>.  This process often breaks complex questions into sequential steps <cit.>.  Techniques such as Chain-of-Thoughts (CoT) <cit.> have shown effectiveness in addressing complex reasoning tasks, particularly in mathematics. The Program-of-Thoughts approach improves upon CoT by using language models to generate text and code, enhancing math problem-solving performance <cit.>. Plan-and-Solve (PS) Prompting, a zero-shot technique, outperforms zero-shot CoT significantly, exceeds Zero-shot Program-of-Thoughts, and matches 8-shot CoT in math reasoning <cit.>.

While music and mathematics share similarities, it is crucial to recognize that music is not as deterministic. In controllable music generation, despite given chords, motifs, and forms, unpredictable elements still significantly affect the quality of the music, similar to controllable text generation. Zhang et al.<cit.>, identify three types of control conditions: semantic, structural, and lexical. Semantic controls refer to content control such as sentiment <cit.> or topic <cit.>, resembling style and emotion in music. Structural control involves shaping the structure of the generated text, such as setting a story's framework or using data from tables or graphs as input, similar to specifying musical forms for generation <cit.>. Lexical controls manage vocabulary usage, ensuring specific keywords appear, akin to using musical chords and motifs as guidelines. LLMs are extensively applied in diverse controllable and creative generation tasks  <cit.>. These systems' abilities in long-context and multi-step generation under predefined conditions are examined, though such analyses are rarely applied in the music domain.  















§ METHODOLOGY



 §.§ Datasets

In this paper, we incorporate six tasks covering from music understanding to generation. The data is collected from MusicPile and MusicBench in ChatMusician <cit.>. The statistics of the dataset we use are shown in Table <ref>. Each model can support the maximum length of tokens of each task. 
 





 §.§ Prompt Engineering

Before examining each LLM's task performance, we conducted preliminary tests to verify their understanding of the relevant musical concepts. These tests confirmed that all models possess foundational knowledge of the six music tasks assessed in this study.

We employed two prompt modes in our experiments of all tasks, Default and Chain-of-Thoughts (CoT). Default mode means forcing the model to respond without any analysis. Additionally, for music theory exercises, to make the model better understand the questions and options, and return the answer in a unified format, we also include the In-Context-Learning (ICL) mode by adding some question-answer pairs as examples shown to the models in the prompt. Taking the task of music theory exercises understanding as an example, three modes of prompts as the prefix of inputs followed by each item in the datasets are shown in Figure <ref>. Different from the music theory exercise, we specifically design prompts to support a multi-round chat conversation with LLMs for the generation tasks. Figure <ref> shows an example of a four-round prompt set of chord-conditioned generation. We invite graduates who majored in music composition to write down their multi-step thoughts when completing the generation tasks involved in this paper. We summarize the common steps of all answers, adapt them to the prompt set, and make sure LLMs can understand or at least intend to follow the instructions. An example of GPT-4's response to the instruction in Figure <ref> is shown on the website[https://github.com/SylviaZiyaZhou/LLMs_music_reasoning/blob/main/
CoT_music_generation_GPT4_response.pdf].

































 §.§ Pre-processing Responses

The responses of models are supposed to have correct ABC notations, but it may have certain syntax or formatting issues, and some outputs may even contain a large amount of natural language.


We select the main features of ABC notation including field names and bar line symbols to help us extract the ABC sequence. 

If the extracted ABC sequence can be rendered into MIDI files using Music21[https://web.mit.edu/music21] successfully and can be later rendering into audio file using midi2audio[https://pypi.org/project/midi2audio], we consider it capable of producing valid ABC notation.



 §.§ Multi-step Reasoning Analysis

In order to compare each model's reasoning capability on both understanding and generation tasks, we first conduct a subjective assessment to evaluate how different models' reasoning processes influence their performances. Participants are all familiar with basic music theory and can understand each task as well as the ABC notation. Secondly, based on the results of the subjective assessment, we further perform an error analysis in detail to show the intermediate answers during the reasoning process of each model.     



  §.§.§ Human Assessment Pipeline

In this section, we will provide a detailed description of our subjective experiments on four popular and open-source LLMs, including Gemma-7B-it, Llama2-7B-Chat, GPT-4, and Qwen-7B-chat. 
We ask the participants to evaluate to what extent the model understands the instructions and correctly answer the questions in the understanding tasks, and to what extent the responses contain the conditions and make creative works in the generation tasks. Specifically, the questions in the human assessment are as follows:






  * For both understanding and generation tasks: 1) To what extent does the model understand and follow the instructions? 

  * Specifically for the understanding tasks: 1) To what extent does the model correctly answer the question? 2) To what extent does the model reason like human beings?

  * Specifically for the generation tasks: 1) AB test: please choose the better one
between a pair of music excerpts by considering their "Musicality"; 2) To what extent does the model contain the conditions?


Except for the AB test in the generation task, each question should be rated in a scoring range from 0 to 10 points. We invited music experts who are familiar with ABC notations as the participants in the human assessment, ensuring that each item was evaluated by at least two experts. 




§ EVALUATION RESULTS

In this section, we provide the evaluation results based on the methodology we discussed in the last section. The quantitative results include the correctly parsing rate of ABC notation in the generation tasks, and the accuracy of music theory exercises. The qualitative results include the statistical analysis of human assessment and the detailed error analysis. Due to space limitation, we provide the examples at https://github.com/SylviaZiyaZhou/LLMs_music_reasoninggithub and the online links of the corresponding files will be attached in the illustration.



 §.§ Quantitative Results

Figure <ref> shows the success rate of rendering valid audio from each LLM's responses under different generation tasks. The pre-processing methodology is introduced in Section <ref>. Except for GPT-4, the other three models all have an audio generation rate of less than 50%, finding it difficult to generate the correct ABC notation format to be converted into audio. 

Table <ref> displays the accuracy of the music theory exercises in three modes. The reason why some models have an accuracy rate below 25% in multiple-choice questions with four options is that most of their responses seek additional information about the questions rather than answering them. Gemma-7B-it has a comparable performance with GPT-4 in the Reason. subset in the Default mode even with a much smaller model size. However, CoT and ICL modes, which significantly improve the GPT-4's performance, show very limited effect or even deficiency in other models. This may inspire us to reconsider the utilization of classical CoT and ICL approaches in solving music tasks.



















































































    
























 §.§ Qualitative Results





For human assessment, Table 3 
shows LLMs on ME and MFE tasks under the CoT mode. We randomly sampled 40 examples of each task. In the instruction following question, GPT-4 demonstrates very good results, while other LLMs more or less can accomplish the tasks, indicating a certain level of capability. However, when it comes to the correctness, even GPT-4 finds it challenging to provide satisfactory answers to the prompts. When testing the logical reasoning of LLMs, the average scores indicate that all LLMs encounter difficulties in applying logical reasoning when answering questions step by step, leading to fundamental errors in music theory or illogical conclusions. This highlights the LLMs' limitation of involving music background knowledge. 




Table 4 
presents the results of human assessment we conducted on generative tasks. In addition to the results shown in the table, we also conducted an AB test based on Musicality. We find that the GPT-4 and Gemma-7B-it achieve comparable results in MFMCG task, while in other tasks GPT-4 always wins. This means Gemma-7B-it has a potential in creating high-quality symbolic music with limited model size.

As depicted in Figure <ref>, on MH and CCG tasks, Qwen-7B-chat and Llama2-7B-chat struggled to effectively output correct ABC sequences to be rendered into audios. Therefore, for MH and CCG tasks, we only include the AB test results for GPT-4 and Gemma-7B-it. Despite GPT-4 achieving relatively better scores in generative tasks, it still falls far away from humans' expectations. Interestingly, beyond the data, LLMs' generative results occasionally exhibit instances of copying motifs provided in the prompt, as well as displaying unstructured harmonic repetitions or completely off-key notes. We believe that although LLMs can adhere to the ABC format condition provided in the prompt, their lack of musical information and knowledge makes it challenging to understand the high-level information within the condition, resulting in less satisfactory generated outcomes.



In terms of the results from subjective experiments, we identified a common issue prevalent in LLMs. Firstly, LLMs, apart from GPT-4, struggle to generate data in the correct ABC format with high probability, despite being able to provide a perfect answer when asked what ABC notation is. This phenomenon led us to speculate that while LLMs are trained extensively and comprehensively, LLMs can hardly understand all the information they have been exposed to and utilize them in different scenarios. Besides, LLMs can generate music in a seemingly appropriate ABC format in generative tasks, but what appears to be a correctly-formatted response is merely copying the prompt without grasping the semantic and structural information in the given condition.



  §.§.§ Multi-step Reasoning Analysis




































































To better illustrate each model's reasoning capability when it is used to complete the music theory exercises, we provide an example of a question in the music theory exercises subset and step-by-step responses of the four models[https://github.com/SylviaZiyaZhou/LLMs_music_reasoning/blob/
main/CoT_music_theory_exercise_all_LLMs.pdf]. The question is about recognizing the interval property of an ABC sequence referring to a compound in a music sheet. From the responses, we can see that GPT-4 is the only model which can actually perform the calculation but still unable to understand the musical notes in the ABC notation. In the GPT-4's responses in the CoT mode, "4", which is mistaken as "a fourth apart", should be a note duration. Accordingly, this mistake influences the whole reasoning process of the calculation of intervals. The response of Llama2-7B-chat also shows its incapability of involving correct music knowledge understanding of notes intervals in the reasoning process. What's more, Qwen-7B-chat even accidentally contains Chinese in the English text and Gemma-7B-it failed to recognize musical notes in the ABC sequence (see in the supplementary materials), although they can return the correct answer if they are merely asked about "the definition of note intervals". 

Besides, the responses of generation tasks such as MFMC generation also have similar problems. In the CoT mode, we find all LLMs except GPT-4, are hard to follow the multi-step instructions and output music in a correct ABC format, so we only provide a GPT-4 response respectively in the raw text[https://github.com/SylviaZiyaZhou/LLMs_music_reasoning/blob
/main/CoT_music_generation_GPT4_response.pdf] and music sheet[https://github.com/SylviaZiyaZhou/LLMs_music_reasoning/blob
/main/Music_Sheet_of_music_generation_GPT4_response.pdf] form given the prompt in Figure <ref>. Although GPT-4 can well understand the instructions in every step, it generates repetitive and simple rhythm without enough progression and variation, compared to the human composer's work in Figure <ref>. 







§ CONCLUSION AND DISCUSSION


In conclusion, our experimental analysis highlights current LLMs' limitations in the realm of music understanding and generation, particularly from the perspective of song-level multi-step reasoning. These findings are crucial as they underline the challenges LLMs face when tasked with generating coherent and contextually rich musical compositions, which often require both complex sequential processing and creative fineness. From the human assessment results and the error analysis, we find that all these models failed to inject correct music theory and knowledge in the music understanding, reasoning and generation process. This knowledge generalization gap is analogous to the reversal curse problem illustrated in  <cit.> where LLMs trained on “A is B” fail to learn “B is A”. Without making sure the fundamental concepts are correctly mentioned in the generated responses, it is hard to alleviate the LLMs' hallucination and guarantee the responses' quality. Therefore, it is significant to implement the knowledge augmentation module in the Supervised Fine-Tuning (SFT) stage to ensure the LLMs can reason based on correct music knowledge by curating more SFT data with enough knowledge-based contexts and practical reasoning processes. 

Specifically, several insights for the multi-step SFT dataset construction can be concluded from the process where professional musicians are asked to create music following the instructions. Firstly, more expert knowledge should be involved in the dataset construction to guarantee its quality. For example, in the chord-conditioned generation task in Chatmusician's dataset, the bass note sequence of the given chords does not conform to the musicians' expectation of the progression generally. 
Secondly, some conditions in the old one-step form are too lengthy and informative with limitations that the human composers feel difficult to follow. For example, when they are given an "AB" structure with two different motives in the MFMC task, all of them find hard to integrate two segments with different motives into a complete piece of music in an "AB" form. Therefore, it might not be reasonable to ask the LLMs to output a completely and well composed music in a one-step approach.

What's more, although four models are all claimed to be able to handle the input size from 4K to 8K tokens, which is much longer than the instructions in the dataset we used, they do not show their long-context processing advantages in the symbolic music domain. Our experimental results show that the widely-used CoT and ICL approaches are not always effective in improving the model's performance. In this way, more step-by-step learning strategies should be specifically developed for instruction-based symbolic music tasks by focusing on correctly answering music theory exercises, explicitly extracting motifs and implicitly extracting musical forms, and consistently following the conditions in the instructions.











§ ACKNOWLEDGEMENT

The research was supported by Early Career Scheme (ECS-HKUST22201322), Theme-based Research Scheme (T45-205/21-N) from Hong Kong RGC, NSFC (No. 62206234), and Generative AI Research and Development Centre from InnoHK. Yinghao Ma is a research student at the UKRI Centre for Doctoral Training in Artificial Intelligence and Music, supported by UK Research and Innovation [grant number EP/S022694/1].

We would like to express our sincere gratitude to Jia Ding and Xiaoduan Li, professional musicians who majored in music composition at Central Conservatory of Music, for their valuable contributions and suggestions throughout the multi-step prompt engineering in conditioned generation tasks. 





































































































 Paper Title: PiCoGen: Generate Piano Covers with a Two-stage Approach.

















  National Taiwan University, KKCompany Technologies
  No. 1, Sec. 4, Roosevelt Rd.
  Taipei
  Taiwan
  
tanchihpin0517@gmail.com



  KKCompany Technologies
  
  Taipei
  Taiwan
drakeguan@kkcompany.com



  National Taiwan University
  No. 1, Sec. 4, Roosevelt Rd.
  Taipei
  Taiwan
yhyangtw@ntu.edu.tw





















































































§ ABSTRACT 
Cover song generation 

stands out as 
a popular way of music making

in the music-creative community. In this study, we introduce Piano Cover Generation (PiCoGen),  a two-stage approach for automatic cover song generation that transcribes the melody line and chord progression of a song given its audio recording, and then uses the resulting lead sheet as the condition to generate a piano cover in the symbolic domain. 
This approach is advantageous in that it does not required paired data of covers and their original songs for training. 
Compared to an existing approach that demands such paired data, our evaluation shows that PiCoGen demonstrates competitive or even superior performance across songs of different musical genres.






































<ccs2012>
   <concept>
       <concept_id>10010405.10010469.10010475</concept_id>
       <concept_desc>Applied computing Sound and music computing</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10002951.10003227.10003251.10003256</concept_id>
       <concept_desc>Information systems Multimedia content creation</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
 </ccs2012>


[500]Applied computing Sound and music computing 
[500]Information systems Multimedia content creation



























PiCoGen: Generate Piano Covers with a Two-stage Approach
    Yi-Hsuan Yang
    August 5, 2024
========================================================





§ INTRODUCTION


Cover song generation, recreating or rearranging the musical elements from an existing piece, is popular within the music-creative community.
Musicians craft a cover song with careful consideration of musical components such as melody, chords, rhythm, and performance techniques, in a style or musical genre that may be different from the original piece (e.g., Pop to Jazz), and with different sets of instruments (e.g., using only the piano or the guitar).

Attempts have also been made to create cover songs automatically,

relying on a supervised approach trained on paired data of songs and their human-made cover versions.
Takamori et al. <cit.> employ a regression model with acoustic features and structural analysis to generate 

piano covers.
Song2Guitar <cit.> uses a hidden Markov model 
to generate guitar covers with fundamental frequency (f0), beat, and chord information.


More recently, Choi et al. present the Pop2Piano model <cit.>, which represents the first attempt that uses 

deep learning 

for cover song generation.
Specifically, it 

leverages the MT3 architecture <cit.>, an encoder-decoder Transformer designed to convert an audio signal into a sequence of discrete tokens, to create piano covers 
of music.
In collecting paired data needed for model training, 
Choi et al. find it important to employ heuristics to improve the data quality,


involving a matching routine to identify covers of songs from the Internet,

and an alignment routine to 
synchronize the songs and their covers. 







To get rid of the reliance on paired data and to supply deep learning models with more training data,

we explore instead in this paper a novel approach for piano cover generation that does not need paired data at all. Our approach draws inspiration from musical domain knowledge that a cover and its original song tend to share a common melody line and chord progression, and therefore uses the symbolic-domain lead sheet <cit.> (i.e., a MIDI-like representation of melody+chord)[The lead sheet is strongly linked to the f0 contours and chroma features of music, both of which play significant roles in cover song identification <cit.>.]

as the intermediate gateway transferring a musical audio involving arbitrary instrumentation into a piano-only rendition. Specifically, as depicted in Figure <ref>, we propose a two-stage model that firstly extracts the lead sheet directly from the song audio, and then generates a piano cover given the extracted lead sheet. We refer to the resulting model as
Piano Cover Generation (PiCoGen).




The previous Pop2Piano model  <cit.> uses a piano transcription-like approach and creates piano covers with a single stage of audio-to-symbolic conversion 
with no explainable intermediate representation, trained using paired data of original music and  piano covers.  
In contrast, PiCoGen bypasses the need of such a paired data by decomposing the conversion task into two stages.

PiCoGen needs other two types of paired data, one for each stage, yet both of which have been available in the research community through efforts on audio-to-symbolic lead sheet transcription <cit.> and symbolic-to-symbolic conditional piano  generation <cit.>.






As its name implies, Pop2Piano is evaluated on Pop music only <cit.>. However, we set forth to investigate how well PiCoGen 
compares to Pop2Piano on generating the piano covers for up to ten different musical genres, including cases where lead sheet transcription can be difficult. This is to study to which extent the domain knowledge (or assumptions) incorporated by PiCoGen is adequate, and to evaluate the generalizability of both Pop2Piano and PiCoGen. 
































We invite readers to our project website[<https://tanchihpin0517.github.io/PiCoGen>] for audio examples of the generated piano covers and the source code of this project.









§ BACKGROUND


Piano cover generation is a special class of cover song generation where the target output is a piano-only rearrangement of a given music piece. It is expected that the generated cover and the original piece can be identified as different versions of the same music composition, and that the cover itself is pleasant to listen to.


Due to the availability of high-quality piano synthesizers, piano cover generation can be approached via generating piano music in a symbolic representation such as Musical Instrument Digital Interface (MIDI), instead of generating audio signals directly. This is the approach taken by Pop2Piano and similarly this work. As such, piano cover generation is related to the following three tasks.

Automatic Music Transcription (AMT) involves converting music signals into symbolic representations such as 

piano rolls.


Automatic piano transcription, for example, aims to generate a musical score-like transcription of all the notes involved in an audio recording of piano performance <cit.>. Lead sheet transcription, as another instance of AMT, transcribes the notes corresponding to the melody line and recognizes the chord names involved in the harmonic progression 
<cit.>.
Pop2Piano adopts MT3 <cit.>, the state-of-the-art of piano transcription, as their model backbone as both piano transcription and piano cover generation convert audio into piano scores. A main difference here is that MT3 assumes its input to be audio recordings of piano-only music, so Pop2Piano needs paired data of {original songs, piano covers} for supervised training to cope with input with arbitrary instrumentation.




Symbolic-domain music generation aims to generate novel pieces of music in a symbolic format, usually by representing a music piece as a sequence of discrete tokens 
<cit.>. The generation process can be either unconditional (i.e., from-scratch generation) or conditional. A key observation of our work is that piano cover generation can be viewed as a conditional generation task—generating token sequences of piano performances conditioned on pre-given lead sheets. Such a symbolic-to-symbolic conditional piano generation problem has been tackled before <cit.>. 




Music style transfer involves the process of altering the style of a music piece to match the style of a provided example, influenced by various attributes such as orchestration, chord progression, and tonality <cit.>. 



Cover song generation can be considered as a specific type of music style transfer in general.










§ 

Given an audio segment A of a song,
cover song generation establishes a model f: A → S that creates an alternative version of the input. For piano cover generation, the output S is pure piano music, which can be represented in the so-called symbolic domain with discrete tokens bearing explicit musical meaning <cit.>.
While many token representations for symbolic music have been proposed in the literature <cit.>,
Pop2Piano adopts the MIDI-like representation <cit.>, which uses tokens that indicate the pitch, onset time, offset time, and velocity (which is related to perceptual loudness) of each  note involved in a piano playing. Pop2Piano can therefore be viewed as a cross-domain sequence-to-sequence model converting a segment of continuous audio waveform to a sequence of discrete tokens.

PiCoGen is different from Pop2Piano mainly in two aspects: the model architecture (two-stage vs. single-stage), and the adopted token representation for symbolic music.
We provide details below.








 §.§ Proposed Two-stage Model


PiCoGen decomposes f into two steps, f=e ∘ g. 
The first step e: A→ L converts the audio input into an intermediate representation L, while the second step g: L → S generates a cover based only on L.
In other words, this two-stage approach assumes A and S are conditional independent given L.
As shown in Figure <ref>, a fundamental assumption of PiCoGen is that the lead sheet can serve as such an intermediate representation.
When A and S share the same underling lead sheet, we assume that they would sound like the same song.
From a style transfer viewpoint, lead sheet is regarded as the “content” to be reserved.

Unlike style transfer in general, the content here is a human-readable lead sheet, facilitating interpretable and controllable music generation.























As shown in fig:overview,
 accordingly contains two sub-models, the Extractor that performs e and outputs the corresponding lead sheet sequence L of the input audio A, and the Performer that performs g and generates piano token sequence S based on the transcribed lead sheet.






















Here e and g can be viewed as two sequence-to-sequence problems. In both cases, the objectives are to maximize the probability of the target sequence given the source sequence.

To better inform the Performer of the temporal correspondence between the lead sheet L and the piano S, we follow the approach of Wu & Yang <cit.> and implement the Performer as a decoder-only Transformer that deals with an interleaved sequence composed of a “bar-wise mix” of L and S, as shown in fig:repr.


Specifically, we segment L and S respectively into multiple sub-sequences L=[L^1, …, L^B] and S=[S^1, …, S^B], where L^k and S^k are the lead sheet sub-sequence and piano sub-sequence for the same, k-th bar (i.e., musical measure) of the input music, and B denotes the number of bars.

We use the bar-wise mix [L^1, S^1, …, L^B, S^B] to train the Performer, such that the generation for the k-th bar of piano S^k would depend on (i.e., can attend to) the current and preceding sub-sequences of lead sheet [L^1, L^2, … L^k] and the preceding piano sub-sequences [S^1, S^2, … S^k-1].












 §.§ Proposed Token Representation



A known weakness of the MIDI-like tokens <cit.> used by Pop2Piano to represent piano music S is the overly large number of tokens needed to represent a musical bar, 

making it hard to learn long-term dependency <cit.>.

We instead adopt a modified version of compound-word (CP) token representation 
<cit.> to make the token sequence  compact.
The main idea of CP is to group related tokens into a “super token.” 
As shown in fig:repr,

tokens belonging to the same super token are combined into a single embedding vector before feeding to the Transformer, which in turn predicts the tokens belonging to the next super token collectively at the same time using different heads.

We refer to <cit.> for details.


Specifically, CP uses the idea of “token classes” to organize the tokens <cit.>. While the original CP considers only two token classes, metric and non-metric, 



we consider four token classes so as to have a unified CP-like representation for not only the piano S but also the lead sheet L.
They are Spec, Bar, Metric and Note.
Spec class includes special tokens, e.g., BOS and EOS (i.e., beginning or ending of a sequence).
Bar class contains bar_src and bar_tgt, which are used to distinguish tokens belongs to lead sheet or piano cover.
Metric class is composed with three sub-classes: Position, Tempo, and Chord.


Note class is composed with three sub-classes describing a musical note: Pitch, Duration and Velocity.[Position indicates the timing shift from the beginning of each bar, in the resolution of 16th note.
Tempo indicate the beat-per-minute (BPM), 
ranging from 32 to 244.
Chord is used to represent the chord condition, composed by {chord root, chord quality} pair. 
Pitch ranges from A0 to C8.
Duration indicates the length of each notes, with the same resolution of Position.
We limit the maximum duration to the length of 8 quarter notes in this work.
Velocity takes 

MIDI value from 0 to 127.
]




 §.§ Implementation Details


To implement the Extractor of PiCoGen,
we employ SheetSage <cit.>, the state-of-the-art for lead sheet transcription.
Authors of SheetSage have kindly released a model checkpoint[<https://github.com/chrisdonahue/sheetsage>] well-trained on the Hook Theory dataset <cit.>, which consists of approximately 40,000 paired data of audio clips and lead sheets.  We simply leverage this checkpoint as our Extractor without fine-tuning.





As for the Performer, we adopt the decoder-only architecture of the CP Transformer <cit.> and train it from scratch.

The model has 8 self-attention layers, 8 heads for multi-head attention, 512 hidden dimensions, a sequence length of 1,024 super tokens, and GeLU as the activation function.
At inference time, the lead sheet L obtained from the Extractor is converted to the proposed CP-like tokens 


and then fed to the Performer to generate the piano cover S.







§ EXPERIMENTAL SETUP





 §.§ Datasets



Training the Performer requires paired data of lead sheets 𝒟_l and piano performances 𝒟_p. For 𝒟_p, we adopt the Pop1k7 dataset <cit.>, which encompasses around 1,700 piano covers of various Japanese anime, Korean popular and Western popular songs.
To construct 𝒟_l, we follow the approach of Wu & Yang <cit.> and employ symbolic-domain music analysis techniques <cit.> to recognize the lead sheet from each piano performance.





For model evaluation, we use the GTZAN dataset 
<cit.>,[<http://marsyas.info/download/data_sets>] a well-known public-domain dataset containing 100 30-second song clips for each of 10 selected genres, including Pop, Jazz, Classical, Rock, etc (see fig:mos for the full list), enabling evaluation beyond Pop music.



Note that SheetSage assumes constant tempo and would report errors for songs that violate this assumption. We therefore discard the 117 songs of GTZAN that has this problem, using the remaining 883 songs in our evaluation. 





 §.§ Baseline and Ablations


We consider Pop2Piano <cit.> as the baseline, as it represents the state-of-the-art for piano cover generation.
Specifically, we use the model checkpoint released by the original authors 
for testing.[<https://github.com/sweetcocoa/pop2piano>]



Moreover, we train and evaluate two ablated versions of PiCoGen.
The first version, denoted as “w/o chord”, uses only the melody as the intermediate representation L, namely dropping chord information entirely. This is to study the effect of chords.

The second version, denoted as “midi-like,” uses the MIDI-like tokens <cit.> to represent the piano music, instead of the more advanced CP-like representation. This is to use the same token representation as Pop2Piano <cit.> so as to examine whether any possible performance difference between Pop2Piano and PiCoGen is due to the two-stage architecture or due to token representations. 

We note that these ablations involve changes only to the intermediate data, and that both ablations adhere to the proposed two-stage strategy. 



 §.§ Evaluation Metrics
 
For objective evaluation, we follow 
Pop2Piano and compute melody chroma accuracy (MCA) <cit.>. The  idea is to compare the top line of the piano MIDI generated by a model with the melody contour of the input audio. 
According to Pop2Piano <cit.>, the melody contour is computed by  separating the vocal  of the input via a source separation model <cit.>, and then using

pYIN <cit.> for f0 estimation from the separated vocal. MCA is ∈ [0,1] and the higher the better.

For subjective evaluation, 

we conduct an online user study involving 30 volunteers consisting of 10 amateurs, 17 pro-ams<cit.>, and 3 pros.


We generate 31 distinct testing sets, distributing them to the human subjects according to their birth dates.
In each testing set, we randomly select 5 genres from the GTZAN dataset and designate one song per genre as the testing target. Consequently, each testing set comprises 5 songs, and we generate 4 results for each song, using Pop2Piano, PiCoGen and the two ablation models, presented to the subject in random order.
Subjects are instructed to score the results with a Likert scale from 1 to 5 (the higher the better) in the following five aspects:

    
  * Similarity (SI): How much the piano cover sounds similar to its original song in general?
    
  * Melody similarity (SI_m): The perceived similarity between the melody lines of the piano cover and the original song.
    
    
  * Chord similarity (SI_c): The similarity in chord progression.
    
    
  * Music fluency (FL): Does the piano cover sound fluent?
    
  * Overall (OVL): How much do you like the piano cover?





























































































§ RESULTS AND DISCUSSION


table:pop displays the average MCA and mean opinion scores (MOS) from the user study for Pop only, the genre targeted by Pop2Piano.

We see that the proposed PiCoGen and its variants show a lower MCA than Pop2Piano, but outperform it greatly in every subjective metric.
Although statistical test does not reveal significant performance difference (due to large stanadard deviation), we view this as an encouraging result since PiCoGen does not use paired data of music and their piano covers at all.

Moreover, although the two ablated variants receive moderately lower MOS than PiCoGen, the result is in general comparable, providing further evidences the effectiveness of the two-stage approach.
We also note that the MOS in OVL reduces from 3.35 to 3.05 when dropping chords from the intermediate representation, suggesting the superiority of using lead sheet instead of melody only in PiCoGen.






table:pop also suggests a conflicting result between MCA and its subjective counterpart SI_m.
PiCoGen underperforms Pop2Piano in MCA, but its MOS in SI_m is higher.

We conjecture that this might due to the fact that MCA only considers vocal melody, neglecting the melody of the leading instrument, as discussed in <cit.>.
We consider the development of better objective metrics as a future work and focus on the result of the user study here.








fig:mos shows the MOS in OVL for all the ten genres in GTZAN, extending our evaluation beyond Pop music.


fig:mos shows that PiCoGen outperforms Pop2Piano not only for Pop but also for genres such as Blues, Disco, and Metal, demonstrating the generalizability of PiCoGen to some extent. 
However, there are also many genres for which Pop2Piano and PiCoGen have similar MOS.

In particular, both Pop2Piano and PiCoGen perform poorly for Hip-hop music.
Upon scrutinizing the attributes of the testing songs, we identify that the quality of the lead sheets extracted by SheetSage is not satisfactory for some of the genres, especially for Hip-hop.
Moreover, 

the MOS of PiCoGen in table:pop and fig:mos ranges from 2.00 to 3.50 in general, leaving considerable room for future improvement given that the scores are from 1 to 5. 

There are many interesting avenues to be investigated in future work.
First, both Pop2Piano and PiCoGen can be potentially improved by collecting a larger training dataset encompassing more genres.
However, we note that Pop2Piano demands both the piano covers and their original songs in diverse genres as the paired data for training. 
In contrast, it is easier to expand the training data of PiCoGen as we only need the piano covers (not the audio recordings of the original songs).
Second, besides data, better result for PiCoGen might be obtained along with advances in lead sheet transcription, and by incorporating intermediate representations of music other than the lead sheet (e.g., rhythmic elements) for musical genres with no dominant melody line.







Finally, it might also be interesting to develop a model that combines the strengths of the music-informed design of PiCoGen and the data-driven design of Pop2Piano, leveraging both unpaired and paired data for training.













§ CONCLUSION

In this paper, we have presented PiCoGen, a  generative model capable of creating a piano cover of an input audio with two distinct steps: extracting a lead sheet from the audio and generating a piano performance based on this lead sheet.
Treating the lead sheet as the common ground between the input music 
and the target piano MIDI, 
PiCoGen bypasses the need of curating paired data of  covers and their original songs for training.
Compared to the existing single-stage model Pop2Piano, PiCoGen exhibits comparable or  superior performance for Pop music and some other  genres.




Directions for future improvement have also been discussed.




















ACM-Reference-Format







































 Paper Title: Iterative Prototype Refinement for Ambiguous Speech Emotion Recognition.



Exploiting Preferences in Loss Functions for Sequential Recommendation via Weak Transitivity
    Hyungwon Choi
    August 5, 2024
============================================================================================


* Corresponding author.

§ ABSTRACT 
    Recognizing emotions from speech is a daunting task due to the subtlety and ambiguity of expressions. Traditional speech emotion recognition (SER) systems, which typically rely on a singular, precise emotion label, struggle with this complexity. Therefore, modeling the inherent ambiguity of emotions is an urgent problem. In this paper, we propose an iterative prototype refinement framework (IPR) for ambiguous SER. IPR comprises two interlinked components: contrastive learning and class prototypes. The former provides an efficient way to obtain high-quality representations of ambiguous samples. The latter are dynamically updated based on ambiguous labels—the similarity of the ambiguous data to all prototypes. These refined embeddings yield precise pseudo labels, thus reinforcing representation quality. Experimental evaluations conducted on the IEMOCAP dataset validate the superior performance of IPR over state-of-the-art methods, thus proving the effectiveness of our proposed method.





§ INTRODUCTION


The advancement of affective computing <cit.> has spurred the development of a wide array of emotion recognition corpora <cit.>. Despite this progress, the intricate and costly process of data annotation, combined with the inherent ambiguity of emotional expressions, poses significant challenges. Traditional speech emotion recognition (SER) systems commonly rely on the assumption that emotions can be clearly categorized, attributing a precise or singular label to each vocal expression. For instance, Sun et al. <cit.> have engaged in representation learning with IEMOCAP <cit.>, while Liu et al. <cit.> have explored metric learning on EmoDB <cit.> and CAISA <cit.>. Nevertheless, these approaches may fall short of capturing the full spectrum of emotional ambiguity. This observation is supported by psychological and statistical research <cit.>, as well as machine learning studies <cit.>, which collectively suggest that the boundaries between different emotional categories are not always distinct. The complexity and nuance of emotional expressions, coupled with the diversity in corpus composition, demand more sophisticated SER methodologies capable of more effectively navigating the intricate emotional landscape. Such advancements would transcend the constraints of singular emotion labeling, thereby enhancing the precision and broad applicability of emotion recognition across a varied range of emotional states.


Recently, researchers have attempted to address the challenges in ambiguous SER by modeling emotion ambiguity. For example, Lotfian et al. <cit.> propose a multitask learning framework to learn the primary and secondary emotions of an utterance, ignoring information about other minor emotions. In contrast, Ando et al. <cit.> propose soft-target label learning – estimating the proportion of all emotions, which increases the complexity of soft label learning. Moreover, Fei et al. <cit.> propose a multi-label emotion classification method to represent the ambiguity of emotions. However, these methods are primarily applicable to corpora retaining expert voting information during annotation. In the absence of annotated records, the generalizability of these methods is challenged. 

To address these challenges, Zhou et al. <cit.> propose a Multi-Classifier Interaction Learning (MCIL) framework. This framework emulates the expert annotation process through multiple classifiers. On the one hand, it represents the annotation process using the classification results of multiple classifiers, facilitating soft label learning. On the other hand, it employs the majority voting mechanism to generate a precise label, enabling the study of traditional SER techniques. It is important to note that the efficacy of MCIL is heavily contingent on the performance of individual classifiers. Poor performance of these classifiers may result in unsatisfactory data labeling quality and emotion recognition performance. Conversely, if a particular classifier exhibits exceptional performance, the effectiveness of interactive learning may be diminished. Furthermore, simply using classifiers to annotate ambiguous samples and retraining the model may compromise the discriminability of the model in this type of data.

Inspired by the work <cit.>, in this paper, we propose an iterative prototype refinement framework (IPR) for ambiguous SER. IPR comprises three key phases: class prototype learning, class prototype updating, and contrastive learning. In the class prototype learning phase, precise samples facilitate the initial acquisition of class prototype embeddings via a warm-up mechanism. Subsequently, during the class prototype updating phase, unlabeled ambiguous samples are introduced to participate in training process. We calculate the distance of the samples from each class prototype to construct ambiguous soft labels and update the class prototypes by the proportion of classes in the soft labels. Finally, contrastive learning is employed to augment the discriminative capability of the model, updating class prototypes when ambiguous samples and their enhanced counterparts points to one prototype. The main contributions are summarized as follows:




(1) We propose a novel iterative prototype refinement framework for ambiguous SER. Contrastive learning aids in the learning of ideal representations, improving the quality of class prototypes. Subsequently, enhanced class prototypes guide the model in generating high-quality representations.


(2) IPR is trained on a small amount of high-quality data, enabling the construction of ambiguous soft labels and single labels for a large amount of data, thereby possessing better generalization capability.

(3) Experimental results on IEMOCAP demonstrate that IPR outperforms the current state-of-the-art methods, achieving an accuracy of 70.75%, with an absolute improvement of 2.00%.



§ PROPOSED METHODS

Illustrated in Fig. <ref>, our proposed framework, IPR, comprises three key phases: class prototype learning, class prototype updating, and contrastive learning. 




 §.§ Class Prototype Learning

First, we initially train the encoder using the precise labeled samples x^q_i. Meanwhile, for each class c ∈{1,2,...,C}, the prototype vectors p_c are initialized based on the precise samples as representative embeddings.

    q_i=   Encoder(x^q_i ; θ),
    
    p_c=   1/n∑_i ∈ c q_i,

where Encoder represents the pre-trained model Wav2vec2.0 <cit.>, which is commonly used as a feature extractor <cit.>. θ represents the trainable parameters. n represents the number of samples belonging to class c.

Since, when the class prototype is not adequately learned, it has a negative impact in guiding the model to learn unlabeled data. Therefore, we set up the warm-up mechanism so that the encoder learns the better representation of the class prototypes on the precise labeled data.



 §.§ Class Prototype Updating

Once we acquire a prototype representation trained on precise labeled samples, the training process of the model will incorporate ambiguous unlabeled samples x^k_i. Then, we compute the similarity between k_i and each class prototype vector p_c to construct ambiguous soft label {s_1,s_2,...,s_C},  where s_c is k^⊤_i p_c. The class prototype vector with the largest similarity to k_i is the pseudo label z_c of that representation.

    z_c = maxk^⊤_i p_c,

where p_c represents the c-th class. max represents maximization operation.

In the subsequent step, following the assignment of pseudo labels to ambiguous unlabeled samples, we dynamically adjust all class prototype vectors based on the ambiguous labels to incorporate the emotional information introduced by these samples. Traditional methods for calculating prototype embeddings require iterative updates, which is time-consuming and computationally intensive. To avoid this problem, we use moving average as the update method for prototype embedding.

    p_c   =Normalize(γ p_c+(1-γ) k_is_c), 
       if  c   =maxMLP(k_i)=maxMLP(k^aug_i),

where Normalize represents the normalization operation. γ represents a tunable hyperparameter. MLP stands for the classifier. 



 §.§ Contrastive learning

To enhance the quality of learned emotion representations in ambiguous samples, we integrate contrastive learning into IPR. For each ambiguous sample x^k_i, we initiate the process by generating an augmented version, denoted as x^aug_i, through a mixing method. This method encompasses various techniques such as adding noise, changing volume, adding reverberation, and changing pitch. Subsequently, we employ class prototype embeddings to assign pseudo labels for both x^k_i and x^aug_i. In a batch of 2N samples, comprising x^k_i and x^aug_i, we differentiate between positive examples, which share the same pseudo label, and negative examples, which possess different pseudo labels. The loss is computed as follows:

    ℒ_con=-logexp((x^k_i·x^aug_i) / τ)/∑_j ∈{k,aug}exp((x^k_i·x^aug_j) / τ),

where τ is a temperature parameter.



 §.§ Training Objective

To train our proposed IPR, three distinct loss functions are employed: cross-entropy loss for precise labeled samples, cross-entropy loss for ambiguous samples, and contrastive learning loss. The overall training loss function is formulated as:

    ℒ=αℒ_c l s(x^q,y)+βℒ_c l s(x^k,ŷ) + μℒ_con,

where α, β, and μ represent the tunable trade-off factors. y and ŷ represent ground-truth and pseudo labels, respectively.



§ EXPERIMENTS



 §.§ Dataset

IEMOCAP <cit.> stands out as a widely used dataset in SER. Developed by the Signal Analysis and Interpretation Laboratory at the University of Southern California, this dataset comprises emotionally charged dialogues enacted by 10 actors representing diverse demographics in terms of gender and age. These dialogues are orchestrated using constraint-based scripts crafted to mimic authentic interaction scenarios. we focus on four emotions: anger, happiness, sadness, and neutral, where excitement class is merged into the happiness class.





 §.§ Experimental Settings

As shown in Table <ref>, following the methodology proposed by <cit.>, we partition the IEMOCAP dataset into three distinct subsets tailored for addressing ambiguous SER scenarios. Specifically, D1 encompasses precise labeled samples exhibiting minimal ambiguity in emotional classification. D2 comprises unlabeled samples with moderate ambiguity in emotion. D3, on the other hand, includes labeled samples characterized by the highest degree of ambiguity in emotion, serving as the evaluation set. The determination of emotion ambiguity is predicated on the level of consensus among experts during the annotation process. For each utterance in IEMOCAP, multiple experts contribute their assessments, and the consistency of their judgments dictates the attribution of the sample. A high level of agreement among annotators results in D1, whereas increased variability in assessments places a sample within D2. Samples demonstrating substantial disagreement among annotators are designated to D3.

In our experimental setup, we establish three baseline systems: a baseline system, a baseline^+ system, and a supervised baseline system. The baseline system is trained exclusively on D1 and subsequently evaluated on D3. In contrast, the baseline^+ system utilizes both D1 and unlabeled D2 during training, which employs model-generated soft labels instead of the prototype-assigned pseudo labels employed in our proposed method. Finally, the supervised baseline system is trained on a combination of labeled D1 and D2 data and tested on D3. This system consists of an encoder and a MLP without the incorporation of class prototype and contrastive learning.

Our proposed framework, IPR, is trained on A100 utilizing PyTorch. We adopt a batch size of 8 and set the maximum training epoch to 50, incorporating a warm-up mechanism for the initial 10 epochs. We employ the AdamW optimizer with an initial learning rate of 10^-5, setting γ to 0.99. The trade-off parameters α and μ are fixed at 1.0 and 0.2, respectively. Initially, β is set to 0.0 and updated according to Eq. 8, gradually increasing based on the current and maximum epoch numbers. The increment of β accelerates with each epoch until reaching 0.5.

    β = min(weight_m×(r^epoch_c/epoch_m / 2-1)/r-1,weight_m),

where the weight parameter weight_m is set to 0.5, denoting its maximum value. Here, epoch_m represents the maximum epoch, while epoch_c denotes the current epoch. The exponential growth factor r governs the rate of increase. Consistent with prior research <cit.>, we adopt accuracy (Acc) as the primary metric for assessing the efficacy of our proposed method. The final Acc is the average results across five seeds.





 §.§ Comparison with State-of-the-Art Methods

To demonstrate the effectiveness of IPR, we compare IPR with many state-of-the-art (SOTA) methods. Table <ref> lists a brief description and accuracy results of these methods, where the first half is SER method, retrained following MCIL training algorithm, and the second half is the classical semi-supervised methods. 

We could find that our method achieves the optimal performance on Acc, reaching 70.75%, which is significantly better than SOTA methods (with an absolute improvement of 2.00%). This performance is mainly attributed to the collaborative effect of class prototype learning and contrastive learning, which reinforce each other, thus allowing the model to capture weak dominant emotions from ambiguous utterances. In comparison with the SER approaches, IPR demonstrates effectiveness and reliability in generating pseudo labels. With the guidance of pseudo labels, IPR achieves a 3.75% improvement on Acc. Furthermore, in the comparison with the semi-supervised methods, we find that IPR is improved by 2.00% on Acc. These semi-supervised methods only utilize a large amount of unlabeled data for the purpose of improving recognition performance. It does not seem to tackle the problem of annotation on large amounts of data by learning on a small number of precise samples. The experimental results provide strong evidence that our proposed IPR could provide ideal performance for ambiguous SER and permit more reliable data annotation.



 §.§ Baseline System Analysis

We report the experimental results of IPR with different baseline systems as shown in Table <ref>. First, we observe that by integrating these unlabeled samples, the performance of IPR improves significantly, by 6.50% over baseline on Acc. This highlights the importance of using unlabeled ambiguous data to optimize model. Second, IPR outperforms baseline^+. This proves that IPR, an iterative prototype refinement framework, outperforms model-generated soft labels in the generation of pseudo labels. Notably, the baseline^+ also outperforms the majority voting method of MCIL in SOTA. Thus, when faced with a corpus without voting information, our method utilizes a small amount of data for training and could achieve a large number of data annotation, both single and ambiguous soft labels. Finally, by comparing IPR and supervised baseline, we could notice that on Acc, the proposed method reduces 1.55%, while MCIL reduces 5.30%. As a result, IPR is closer to the performance of supervised system, while it produces more precise and reliable pseudo labels when annotating unlabeled samples.






 §.§ Class Prototype Analysis

Figure <ref> shows the trend of similarity between class prototype embeddings during the training process. In the initial 10 epochs of training, we observe a transient increase in the similarity between the neutral prototype and other class prototype vectors. This phenomenon arises from the localized occurrence of emotional expression within speech segments. Additionally, given the similarity in activation levels between happiness and neutrality, they exhibit the highest prototype similarity, aligning with prior finding <cit.>. Intuitively, anger and sadness represent contrasting emotions, thus exhibiting a declining trend in the similarity between their prototypes. In subsequent training stages, prototype-assigned pseudo labels are highly inaccurate, resulting in unlabeled data negatively impacting the model and thus reducing the quality of the learned class prototypes. Furthermore, all prototype embeddings are soft updated according to the proportion of each emotion in ambiguous soft labels, which leads to an increase in the similarity of the emotion prototypes. Ultimately, we observe that the class prototypes stabilize at a relatively steady state with a similarity greater than 0, affirming the inherent ambiguity in emotion expression.



 §.§ Pseudo Label Analysis


Figure <ref> shows agreement rate of model-generated pseudo labels and prototype-assigned pseudo labels with ground-truth labels. We observe that the upward trend in the agreement rate between prototype-assigned and ground-truth labels is significantly greater than the agreement rate between model-generated soft labels and ground-truth labels. Additionally, the prototype-assigned pseudo labels exhibit a higher agreement rate with the ground-truth labels. On the one hand, IPR could achieve ambiguous soft label annotation on a large amount of unlabeled data by training on a small amount of precise labeled data. On the other hand, it could assign more precise single labels while maintaining the desired recognition performance. 





§ CONCLUSIONS

In this paper, we propose a novel iterative prototype refinement framework leveraging class prototypes and contrastive learning for ambiguous SER. IPR capitalizes on the clustering effect induced by contrastive learning to generate optimal representations. These representations facilitate the refinement and updating of class prototype embeddings, thereby enhancing the precision of assigned pseudo labels. Consequently, improved pseudo labels contribute to the augmentation of representation quality, establishing a positive feedback loop. Through a comprehensive array of comparative experiments, ablation studies, and visualization analyses conducted on the IEMOCAP benchmark dataset, we substantiate the effectiveness of IPR.





§ ACKNOWLEDGMENTS

This work was supported in part by NSF China (Grant No. 62271270).


IEEEtran




 Paper Title: Interaural time difference loss for binaural target sound extraction.






[
    R. Sasportes^†
    August 5, 2024
==================


§ ABSTRACT 
Binaural target sound extraction (TSE) aims to extract a desired sound from a binaural mixture of arbitrary sounds while preserving the spatial cues of the desired sound. Indeed, for many applications, the target sound signal and its spatial cues carry important information about the sound source. Binaural TSE can be realized with a neural network trained to output only the desired sound given a binaural mixture and an embedding characterizing the desired sound class as inputs. Conventional TSE systems are trained using signal-level losses, which measure the difference between the extracted and reference signals for the left and right channels. In this paper, we propose adding explicit spatial losses to better preserve the spatial cues of the target sound. In particular, we explore losses aiming at preserving the interaural level (ILD), phase (IPD), and time differences (ITD). We show experimentally that adding such spatial losses, particularly our newly proposed ITD loss, helps preserve better spatial cues while maintaining the signal-level metrics.




target sound extraction, binaural, deep learning





§ INTRODUCTION




Target sound extraction (TSE) aims at isolating a sound signal of interest belonging to a desired (or specified) sound class from a mixture of arbitrary sounds <cit.>. For example, TSE could extract the sound of a siren in a recording including also dog barking and car passing. The TSE problem can thus be seen as a generalization of the target speech extraction problem <cit.> to arbitrary sounds. Realizing TSE could have many practical applications, such as hearables or hearing aids that can focus on sounds of interest in everyday environments or smart audio post-production systems. 

A typical TSE system consists of a neural network that accepts a sound mixture and outputs the desired sound signal free from other sounds and noise. The network is conditioned on a clue that is used to identify the target sound in the mixture. Several types of clues can be used, such as a one-hot vector identifying the target sound class <cit.>, audio recording similar to the desired sound <cit.>, video <cit.>, onomatopoeia <cit.>, the region where to focus <cit.> or multiple clues  <cit.>. Here, we focus on TSE conditioned on the target sound class.

Most TSE frameworks focus on single-channel processing <cit.>. However, humans rely on binaural hearing to capture spatial information about the sounds. For example, the interaural level difference (ILD) and interaural time difference (ITD) serve as essential cues to localize sound sources <cit.>. It is thus essential to develop TSE systems that can extract a target sound while preserving its binaural cues. Binaural processing can be achieved by extending a monaural TSE system to accept binaural input and output binaural signals. This idea was first introduced for speech enhancement <cit.> and recently applied to TSE  <cit.>. We can train such systems using a signal-level loss on each output channel, which guides the model to output left and right signals as close as possible to the references. Such a training loss implicitly pushes the system to preserve spatial cues. However, prior works <cit.> did not use any explicit spatial loss, which may limit the ability of TSE to recover spatial cues.



This paper investigates whether adding an explicit spatial loss could further improve spatial cue preservation of binaural TSE. 
It is straightforward to define a loss measuring the error of ILD between the estimated and reference signals <cit.>, as it only involves computing the difference of the ratio of the norm of the left and right signals, which is differentiable. In contrast, computing the ITD involves finding the position of the maximum in the cross-correlation between the left and right signals. This computation requires the “argmax” operation, which is not differentiable, making it challenging to use as a loss. A recent work <cit.> has proposed instead a loss measuring the error of the interaural phase difference (IPD), which is related to the time difference. However, optimizing IPD errors may be challenging because of the phase wrapping problem. 

We propose instead an alternative loss, which measures the errors between the cross-correlation coefficients of the estimated and the reference signals.  By abuse of language, we refer to it as ITD loss. The proposed ITD loss is more directly related to the ITD computation than the IPD loss. We thus hope it will lead to better ITD cue preservation. 

Note that a prior works <cit.> have proposed using ILD and IPD losses for binaural speech enhancement. However, it is important to explore spatial losses for the TSE problem since it deals with a much larger variety of sounds than speech signals <cit.>.
To summarize, the contribution of this paper is twofold. First, we propose a new spatial loss that is directly related to ITD cues. Second, we compare three types of spatial losses (ITD, ILD, and the newly proposed ITD losses) for binaural TSE. 
We perform experiments on TSE of sound mixtures containing three to four sounds from 20 sound classes. We use a binaural TSE system as a baseline <cit.> and confirm experimentally that adding spatial losses can reduce ITD, ILD, and IPD errors while preserving signal-level metrics. Moreover, the newly proposed ITD loss achieves overall superior binaural TSE performance.






§ BINAURAL TARGET SOUND EXTRACTION



In this section, we first introduce the problem statement and describe a baseline binaural TSE model. We then discuss the signal-level and spatial losses we use to train the TSE system. In particular, we introduce our proposed ITD loss in Sec. <ref>.



 §.§ Problem Statement

The objective of TSE is to isolate a sound signal of interest belonging to a desired sound class from an observed signal 𝐱_m consisting of a mixture of sounds defined as:

    𝐱_m = ∑_i=1^I𝐬_i,m,

where 𝐬_i,m∈ℝ^ T is the sound signal of the i-th source at the m-th microphone, T is the number of samples, and I is the total number of sound sources. 
In the following, we consider binaural recordings and define binaural mixture as 𝐱 = [ 𝐱_l, 𝐱_r] ∈ℝ^T × 2 where  l and r represent the index of the left and right microphones, respectively.
Similarly, we denote by 𝐬^tgt= [ 𝐬_l^tgt, 𝐬_r^tgt] ∈ℝ^T× 2 the binaural target signal, which corresponds to the desired sound class. 

Binaural TSE aims to recover the binaural signal of the target by preserving the spatial information of the sound.
Therefore, a TSE model follows the expression:

    𝐬̂^tgt = TSE(𝐱, 𝐨;θ),

where 𝐬̂^tgt represents the extracted target sound signal, TSE(·) is a neural network with parameters θ, and 𝐨 = [0, …, 1, … 0]^⊤ is a 1-hot vector representing the target sound class, i.e., it has a value of 1 for the index of the target class and 0 otherwise.  The 1-hot vector has a dimension O corresponding to the number of classes the models can handle.




 §.§ TSE Model

Our TSE model consists of an encoder-decoder model with a mask estimation network, or extractor, as shown in Fig.<ref>. Unlike conventional single-channel TSE models <cit.>, we use Semantic Hearing <cit.> as the baseline model, which is a binaural extension of the Waveformer model <cit.>. 


First, the input signal is processed by an encoder block:

    𝐘 = Encoder(𝐱),

where Encoder(·) is the encoder layer, 𝐘∈ℝ^T' × D is the encoded representation of the input signal, D is the feature dimension and T' is the number of frames. 
The encoded mixture representation is then passed to an extractor, which predicts a mask 𝐌∈ℝ^T' × D of the target sound to extract:

    𝐌 = Extractor(𝐘, 𝐞^tgt),

where Extractor(·) is an extractor neural network, and 𝐞^tgt∈ℝ^D is an embedding vector characterizing the target sound class. The embedding vector can be simply obtained from an embedding layer, which accepts the one-hot vector characterizing the desired sound class, 𝐨, as input, i.e., 𝐞^tgt = 𝐖𝐨, where 𝐖∈R^D × O is an embedding matrix. 

Finally, the estimate of the target sound is obtained by applying the decoder to the masked features as 

    ŝ^𝐭𝐠𝐭 = Decoder(𝐌⊙𝐘),

where Decoder(·) is the decoder layer, and ⊙ is the Hadamard product.




Training the TSE system requires the triplets of binaural sound mixtures, target sound class, and binaural target sound signals. The optimal model parameters, θ̂ can be obtained by minimizing the training loss:

    θ̂ = _θℒ( 𝐬^tgt,𝐬̂^tgt(θ)).


Conventional TSE systems use only a signal-level training loss. Here, we investigate using a multi-task loss as:

    ℒ = αℒ^signal + βℒ^spatial,

where α and β are weights, and ℒ^signal, ℒ^spatial are the signal-level and spatial losses, respectively, which we define below.




 §.§ Signal-Level Losses

Conventional TSE systems are usually trained using a signal-level loss like the signal-to-noise ratio (SNR), the scale-invariant SNR (SI-SNR)<cit.> or a combination of both <cit.>. We can use these losses for binaural outputs by computing a signal-level loss for each channel as proposed in <cit.>. For example, the SNR loss for binaural outputs is:

    ℒ^SNR(𝐬^tgt,𝐬̂^tgt) = - ( 1/2SNR(𝐬_r^tgt, 𝐬̂_r^tgt) + 1/2SNR(𝐬_l^tgt, 𝐬̂_l^tgt) ),

where SNR(𝐬, 𝐬̂) = 10log_10( ||𝐬||_2^2/||𝐬 - 𝐬̂||_2^2) is the SNR between the reference signal, 𝐬, and the estimated target signal, 𝐬̂. We can define a similar loss for the SI-SNR.

In this paper, following <cit.>, we use a weighted sum of SNR and SI-SNR losses as the signal-level loss:

    ℒ^signal = 0.9ℒ^SNR + 0.1ℒ^SI-SNR.




















 §.§ Spatial Losses

To help the TSE model better preserve spatial cues, we investigate adding binaural metrics to the signal-level losses. For this purpose, we employ ILD and IPD losses, which were previously proposed for speech enhancement<cit.>, and also propose a new ITD-related loss. 





  §.§.§ Interaural Level Difference Loss

ILD aims to measure the level difference between the channels of a signal. 
The ILD of a binaural signal is defined as:

    ILD = 10 log_10( ||𝐬_ l||_2^2/||𝐬_r||_2^2),

where 𝐬_l∈R^T and 𝐬_r∈R^T are left and right signals.





Then, the ILD loss can be expressed as the mean of the absolute difference of the target and predicted ILDs:

    ℒ^ILD =  | ILD^tgt - ILD^tgt|,

where ILD^tgt and ILD^tgt are the ILD of the reference and extracted target sound signals computed with Eq. (<ref>). 
Note that prior work <cit.> defines this loss in the Short-Time Fourier Transform (STFT) domain, whereas we define it in the waveform domain.




  §.§.§ Interaural Phase Difference Loss
 
The time difference of arrival (TDOA) between microphones translates into phase differences between the received sound signals. To preserve the spatial information of the sound source, one approach is to focus on maintaining the IPD of the reference left and right signals in the extracted signals.

The IPD between two signals can be calculated using the STFT of these signals.

However, the phase of a complex number has inherent periodicity (i.e., 0 and 2π are equivalent). This property, known as the circularity problem, makes direct subtraction of phases unreliable. To address this, we compute the IPD between the two channels of a signal as:

    IPD_u,v = atan( Im( S_u,v,l  S_u,v,r^*)/Re( S_u,v,l  S_u,v,r^*)),

where S_u,v,r, S_u,v,l∈ℂ represent the STFT of the right and left signals, u, v are the indexes of the time and frequency bins, ^* is the conjugate operation, Im(·) and Re(·) represent the imaginary and real parts of a complex number, and atan(·) denotes the arctangent function. 

To leverage the IPD as a loss function, we compute the MSE between the IPD of the target and predicted signals:

    ℒ^IPD = 1/U V∑_u=1^U ∑_v=1^V  ( IPD^tgt_u,v - IPD^tgt_u,v)^2,

where IPD^tgt 
 and IPD^tgt are the IPD of the reference and extracted target sound computed with Eq. (<ref>), and U and V are the number of time and frequency bins, respectively.
 
Note that a similar IPD loss was used in <cit.>. They computed the IPD directly as the angle of the ratio S_u,v,l/S_u,v,r[<https://github.com/VikasTokala/BCCTN>]. This is mathematically equivalent to Eq. (<ref>) but handles the phase wrapping differently as Eq. (<ref>) defines the phase difference between -π/2 and π/2, i.e., the smallest possible phase difference. Besides, they used a binary mask to limit the IPD computation to the regions where the source is active. In our experiments, we did not use such a mask as it led to poorer performance, probably because of the difficulty of defining a mask suitable for the diversity of sounds covered by TSE.





  §.§.§ Proposed Interaural Time Difference Loss


Our aim is to develop a TSE system that can preserve the ITD of the target sound since it is an important spatial cue used by humans to localize sounds <cit.>. Therefore, we propose training the TSE model to output signals with an ITD close to that of the target sound.

ITD measures the difference in time of sound arrival between the left and right microphones. It can be obtained by finding the position of the highest peak in the cross-correlation between the left and right signals of the target sound. 

We can compute the ITD as:

    ITD = _t ∈ [-τ,τ]
         c_t,

where c_t is the cross-correlation coefficient between the left and right signals at time step t.
τ is a scalar limiting the predicted delay to be in a given range, which is related to the distance between the microphones. 

Typically, the cross-correlation is computed using the generalized cross-correlation phase transform (GCC-PHAT) algorithm <cit.> as follows:

    𝐜 = ℱ^-1(ℱ(𝐬_l) ⊙ℱ(𝐬_r)^*/| ℱ(𝐬_l) ⊙ℱ(𝐬_r)^* |),

where 𝐜 =[c_t=-T, …, c_t=0, …, c_t=T ] ∈ℝ^2T+1 is the vector of the cross-correlation coefficients, ℱ and ℱ^-1 are the Fourier transform (FT) and inverse FT (IFT), respectively. We use the entire signal (here 6 seconds in both training and evaluation) as the windows length to compute ℱ.

A loss on the ITD should measure the difference between the ITD of the reference target sound signals, 𝐬^tgt, and that of the extracted signals, 𝐬̂^tgt.
However, as seen in Eq. (<ref>), the ITD computation involves the argmax operation, which is not differentiable.
Therefore, we define the ITD loss as the mean squared error (MSE) between the cross-correlation of the reference and extracted signals as:

    ℒ^ITD =  1/2τ + 1∑_t = -au^τ( c^tgt_t - ĉ^tgt_t )^2
    ,

where c_t^tgt and ĉ_t^tgt are the cross-correlation between the reference and extracted signals computed with Eq. (<ref>). Note that the proposed ITD loss is differentiable since all operations required to compute the cross-correlations, including FT and IFT, are differentiable. 




































































§ EXPERIMENTS






























 §.§ Experimental Settings



  §.§.§ Datasets

In our experiments, we used an openly available dataset of binaural sound mixtures <cit.>. The data consists of simulated reverberant mixtures of three to four sound events added to urban background noise. 
This dataset leverages 20 sound classes from FSD50K <cit.> (general-purpose), ESC-50 <cit.> (environmental sounds), MUSDB18 <cit.> and noise files for the DISCO dataset <cit.>. 

The background sounds were taken from TAU Urban Acoustic Scenes 2019 <cit.>. Binaural mixtures were generated by convolving the sound source signals with room impulse responses (RIRs) and head-related transfer functions (HRTFs) as it is done in Semantic Hearing <cit.>. We used HRTFS from 43 subjects from the CIPIC corpus<cit.>, and real and simulated RIRs from three corpora<cit.>. The sampling frequency of all signals was 44.1 kHz.


We mixed the data on the fly with Scaper <cit.>. From each mixture, we only extracted one of the foreground sources (the same one across the experiments) <cit.>. 
The training was done with 6-second mixtures. The number of training, validation, and testing mixtures were 100K, 1K, and 10K, respectively.






























         



        





















































  §.§.§ System Configuration

We employed the same configuration for the TSE system as used in <cit.>.[We used the model variant D = 256 proposed by Semantic Hearing  <https://github.com/vb000/SemanticHearing>]  The system performs online processing with a latency of 20 msec. It has  1.74M trainable parameters in total.

The model consists of a 1-D convolution and 1-D transposed convolution for the encoder and decoder, respectively. The encoder  1-D convolution layer has a stride of L=32 samples and a kernel size of K=3L. 
After the mask is obtained with the extractor, the 1-D transposed convolution layer returns the waveform of the target sound.
The extractor consists of an encoder-decoder architecture conditioned on the target class embedding by multiplying the output of the dilated convolution with the target sound embedding, 𝐞^tgt. The encoder is a stack of 1-D dilated convolution (DCC) layers followed by a Transformer-like layer. The DCC layers have a kernel size of K=3 and dilation factors set to { 2^0, 2^1, ..., 2^9 }. The Transformer-like layer consists of two multi-head attention (MHA) layers with 8 heads. 

We trained all models for 80 epochs using a batch size of 32 and a learning rate of 5e-4. 
For the IPD loss, we computed the complex STFT with a Hann window of 1024, a hop length of 256, and an FFT of 1024 samples. For all models, we used the signal-level loss defined in Eq. (<ref>), and combined it with the spatial losses according to Eq. (<ref>) with fixed weight α=1. 
We selected the hyperparameters, including the loss weights β and the number of epochs, that achieved the best spatial metric among those that did not degrade signal-level metrics on the validation set. Accordingly, the values of β were chosen to 0.1, 1, and 1 for the ILD, IPD, and ITD losses, respectively. For ITD loss, we set τ = 1ms. 



  §.§.§ Evaluation Metrics

Following Semantic Hearing <cit.>, we measured the performance on both signal-level (SI-SNR and SNR), and spatial metrics by computing the difference between the reference and extracted signals, i.e., ΔILD, ΔIPD, and ΔITD.  ΔILD and ΔIPD were computed with Eq. (<ref>) and Eq. (<ref>), respectively. ΔITD was obtained as the absolute difference between the ITD of the reference and extracted signals, where ITD is obtained with Eq. (<ref>). To compute ITD, we used the GCC-PHAT as it is known to be more robust to reverberation. However, we also report results with the simple cross-correlation for comparison with prior work <cit.>. We name these metrics ΔITD-GCC and ΔITD, respectively. 
Note that prior works designed spatial metrics considering human perception <cit.>, by, e.g., computing ITD only on the frequencies below 1.5 kHZ and ILD on the frequencies above 3 kHz. With such evaluation metrics, we observed a small improvement over the baseline of about 5 μs with our proposed ITD loss. However, we expect a larger improvement if we would include a low-pass filter in the loss computation, which will be part of future investigations.





Finally, we also report the failure rate (FR) to provide a rough estimate of how often TSE failed to correctly identify the target sound in the mixture. We defined FR as the percentage of test samples for which SI-SNR improvement is below 1dB <cit.>.  





 §.§ Experimental Results

Table <ref> compares the performance of the proposed losses with that of the unprocessed mixture and a baseline TSE that only uses the signal-level loss of Eq. (<ref>) <cit.>. Note that the ILD, ITD and, IPD values computed on the mixture are only provided as a reference. We observe that the baseline TSE greatly reduces ILD and ITD errors compared to the mixture but not IPD. Actually, most experiments failed to improve IPD errors, which may suggest the difficulty of using IPD metrics for TSE.










Using the spatial losses in addition to the signal-level loss (systems (3)-(5)) outperforms the baseline TSE system (system (2)) for most metrics. These results show that the multi-task loss does not degrade extraction performance in terms of signal-level metrics and FR (it even slightly improves it).

We confirm that adding a spatial loss improves performance on the related spatial metric.
Using the ILD loss (system (3)) performs best in terms of ΔILD but does not consistently improve ΔITD. 
Using the IPD loss (system (4)) greatly reduces ΔIPD, and still slightly improves ΔILD. However, it does not translate into improving ITD errors, suggesting the limit of the IPD loss.
In contrast, using our newly proposed ITD loss (system (5)) improves all spatial metrics compared to the baseline while achieving comparable SI-SNR, SNR, and FR values. The improvement in terms of ΔITD is particularly large, i.e., 26.2μs (more than 1 sample) or a relative improvement of 16 %. These results confirm that our proposed ITD loss contributes to improving spatial cue preservation of TSE.









§ CONCLUSIONS

In this paper, we introduced a new loss function based on ITD, which preserves binaural properties and improves ITD difference between left and right channels. Our proposed ITD loss is superior to a conventional IPD loss as it improves binaural cue recovery of TSE in terms of ILD, IPD, and ITD, while maintaining performance in terms of signal-level metrics. 

The proposed ITD loss is general and could be used for other speech and audio processing tasks, such as binaural speech enhancement or speech separation. Future works will include such investigations.





IEEEbib




 Paper Title: Combining audio control and style transfer using latent diffusion.




[
    Amy Navarathna,1,2 Benjamin J. Carey,1,2 James S. Bennett,3 Soroush Khademi,1,2 and Warwick P. Bowen1,2,4*
    August 5, 2024
==============================================================================================================



§ ABSTRACT 

Deep generative models are now able to synthesize high-quality audio signals, shifting the critical aspect in their development from audio quality to control capabilities. Although text-to-music generation is getting largely adopted by the general public, explicit control and example-based style transfer are more adequate modalities to capture the intents of artists and musicians. 

In this paper, we aim to unify explicit control and style transfer within a single model by separating local and global information to capture musical structure and timbre respectively. To do so, we leverage the capabilities of diffusion autoencoders to extract semantic features, in order to build two representation spaces. We enforce disentanglement between those spaces using an adversarial criterion and a two-stage training strategy. Our resulting model can generate audio matching a timbre target, while specifying structure either with explicit controls or through another audio example. We evaluate our model on one-shot timbre transfer and MIDI-to-audio tasks on instrumental recordings and show that we outperform existing baselines in terms of audio quality and target fidelity. Furthermore, we show that our method can generate cover versions of complete musical pieces by transferring rhythmic and melodic content to the style of a target audio in a different genre. 

 




§ INTRODUCTION





Deep generative models are now particularly successful at synthesising high-quality, realistic audio signals. Hence, the major impediment to their broader use in creative workflows is not their audio quality anymore, but rather how end-users can have complete control over the generation process. Following early works on unconditional generation <cit.>, multiple methods proposed to enable control by conditioning generation on semantic tags or audio-descriptors <cit.>. However, such supervised approaches remain limited to the use of explicit descriptors and are constrained by their reliance on annotated datasets. The recent development of language models and representation learning led to impressive performance in text-conditioned generation, mainly relying on transformers <cit.> or diffusion models <cit.>. However, concepts such as timbre, musical style or genres boundaries are usually elusive and highly subjective. Hence, text descriptions might remain limited to common sounds and insufficient to precisely capture musical intentions. 





A parallel stream of research to alleviate those issues is to guide specific aspects of the generation process by providing audio examples. Most approaches in this audio-to-audio editing are focused on timbre transfer, where the timbre of a given sound is applied on the content of another. While some works can transfer any audio to the timbre of a given training set <cit.>; others achieve many-to-many timbre transfer but only between a small set of predefined instrument classes <cit.>. One-shot timbre transfer between different instrument recordings have been achieved using Variational Autoencoders (VAE) <cit.>, but these models rely on a latent bottleneck to enforce disentanglement between timbre and pitch, which hampers their ability to generate high-quality audio on real-world data. More recently, a text-inversion technique was proposed to perform musical style transfer between arbitrary content and style examples <cit.>, but it relies on a large pretrained text-to-music model and requires optimisation prior to each transfer, resulting in very slow inference. 



In this paper, we aim to unify explicit control through audio descriptors or MIDI sequences and style transfer within a single model. To do so, we separate local, time-varying factors of variations and global information, capturing musical structure and timbre in two separate representation spaces. We slightly abuse the terms structure and timbre: by structure, we designate time-varying features, e.g. melody, loudness; by timbre, we designate global features such as actual timbre, but also style or genre. The principle of our method is depicted in Figure <ref>.
Our approach is based on the recently proposed diffusion autoencoder <cit.>, which trains a semantic encoder to condition a diffusion model, in order to achieve both high-quality generation while being able to extract and control high-level features from the data. We extend this approach by building separate representations for timbre and structure, while enforcing their disentanglement with an adversarial criterion combined with a two-stage training strategy. Our method can generate audio matching a timbre target, while specifying the musical structure either with explicit controls (such as MIDI data input) or an audio example. For computation efficiency, our diffusion model operates in the latent space of pretrained autoencoders, resulting in faster than real-time inference on GPU[Experiments were conducted on a NVIDIA A5000 GPU]. 


First, we benchmark our model on a one-shot timbre transfer tasks and demonstrate that our model improves upon existing baselines in terms of audio quality, timbre similarity as well as note onsets and pitch accuracy. On the same dataset, we show that our model can also generate audio from MIDI input and a target timbre example with performances superior to a state-of-the-art MIDI-to-audio baseline. 
Finally, we show that our method can be applied to complete musical pieces and generate cover versions of a track by transferring its rhythmic and melodic content to the style of a target audio in a different genre. We provide audio examples, additional experiments and source code on a supporting webpage[ <https://nilsdem.github.io/control-transfer-diffusion/>]




§ BACKGROUND










 §.§ Diffusion models



Diffusion models (DMs) are a family of generative models that learn to reverse a stochastic process that gradually adds noise to the input data. These models benefit from high-quality generation, stable training and conditioning abilities, which led to their widespread adoption in image <cit.> and audio generation <cit.>. 

Formally, we define a forward process q(𝐱_1:T|𝐱_0) = ∏_t=1^T q(𝐱_t|𝐱_t-1), which is a Markov chain that increasingly adds noise to the data 𝐱_0 by relying on the conditional distribution 


    q(𝐱_t | 𝐱_t-1) = 𝒩(𝐱_t; √(1 - β_t)𝐱_t-1, β_t 𝐈),


where β_t are hyperparameters defining the noise levels at times t ∈{0,T}, with T ∈ℕ. We are interested in learning the reverse diffusion process p_θ(𝐱_t-1 | 𝐱_t), from which we can iteratively denoise a random sample 𝐱_T ∼𝒩(0, 𝐈) to a data sample 𝐱_0 ∼ p(𝐱_0). In a recenty study, <cit.> made a connection between diffusion and denoising score matching <cit.>, leading to a simplified formulation and improved experimental results. The authors show that the reverse process can be approximated by learning a denoising network ϵ_θ that predicts the noise ϵ∼𝒩(ϵ, 0, 𝐈) used to corrupt the data. This results in a simpler training objective 



    min_θ∈Θ  𝔼_t,𝐱_0,ϵ [ ϵ_θ(√(α_t)𝐱_0 + √(1 - α_t)ϵ, t) - ϵ ] ,


where α_t = ∏_s=1^t (1 - β_s), and ϵ_θ is usually parametrized by a UNet <cit.>. 



 §.§ Diffusion autoencoders


DMs naturally yield a series of latent variables 𝐱_1:T through their forward process. However, these stochastic variables built from increasingly adding noise do not capture much semantic information over the data. Although the more recent proposal of Denoising Diffusion Implicit Models (DDIMs) <cit.> extends the original diffusion formulation to a deterministic process allowing each data input to be mapped to a unique latent code 𝐱_T, it still fails to extract and organise high-level features from of the data. Diffusion autoencoders <cit.> alleviate this issue by employing a learnable encoder that compress the data to a semantic latent code 𝐳_sem = E_ϕ (𝐱_0), which then conditions a diffusion decoder. The semantic encoder and the DDIM decoder are trained jointly, following the objective


    min_θ, ϕ𝔼_t,𝐱_0,ϵ [ ϵ_θ(√(α_t)𝐱_0 + √(1 - α_t)ϵ, E_ϕ (𝐱_0), t) - ϵ ]


On image applications, the authors show that the semantic code captures high-level attributes such as person identity, smile or presence of glasses, and can be used for downstream tasks such as conditional generation and attributes manipulation, while achieving state-of-art reconstruction. This approach was also successfully applied to audio <cit.>, where the authors encode magnitude spectrograms into a semantic latent space, allowing to achieve high quality text-conditioned waveform generation. 




 §.§ Control in audio generation


A straightforward approach to extend unconditional generative models in order to provide instruments befitting artistic control is to introduce conditioning on explicit controls. The DDSP model <cit.> proposes explicit pitch and loudness conditioning, while FaderRave <cit.> extended explicit control to non-differentiable time-varying attributes, but both methods remain limited to explicit descriptors and annotated datasets.  

While recent text-to-music methods like MusicGen <cit.> and Music ControlNet <cit.>  have incorporated melody conditioning capabilities, their expressiveness remains constrained by the need to define subjective timbre properties through text prompts. Li et al. <cit.> proposed to use text-inversion in order to extract pseudo-words that represent timbre directly from audio, but their method is computationally intensive and requires to perform an optimisation for each new timbre target. Timbre conditioning directly from a waveform example was also recently proposed the context of bass accompaniment generation<cit.>.







 §.§ Unsupervised disentanglement in sequential data


Many works proposed to model sequential data as a combination of local (time-variant) and global (time-invariant) factors of variation. Notably, the disentangled sequential autoencoder <cit.> relies on simple architecture biases and parameter tuning to obtain disantangled local and global latent variables. Following this work, multiple methods improved the learned representation by explicitely minimizing the mutual information between the two learned variables <cit.>. It was shown that disentanglement can be further improved with contrastive learning and domain-specific transformations that preserve local or global attributes <cit.>.
























More specifically in audio generation, SS-VAE <cit.> employs a Vector-Quantized VAE to achieve disentanglement through compression on quantized structure latent codes, combined with timbre-preserving data augmentations. Luo et al. <cit.> follow a two-stage training strategy similar to ours, and improve disentanglement by enforcing the consistency of the global and local latent variables in style or content transfer. However, both disentanglement strategies degrades reconstruction accuracy, which on top of spectrogram inversion based synthesis leads in poor audio quality. 



§ METHOD


Our approach is based on the assumption that musical audio samples can be seen as specific instances of a set of latent features that are separated between global features that capture style, and time-varying features that capture the local evolution of the signal. Although diffusion models are capable of high-quality conditional generation, they are computationally expensive when dealing with high-dimensional data. Hence, we employ an invertible audio codec to first compress the audio into a low-dimensional latent space, onto which we can build an efficient generative model. We extend the DiffAE <cit.> architecture to two semantic encoders in order to extract separately timbre and structure features from input samples. To further disentangle the learned features and improve transfer as well as explicit control performances, we employ an adversarial training strategy during training. In this section, we detail our proposed model depicted in Figure <ref>.




 §.§ Audio codec


We build our audio codec as a convolutional autoencoder based on the RAVE model <cit.> architecture, featuring the adversarial discriminator recently proposed in <cit.>. The model compresses audio waveforms 𝐱 into an invertible latent sequence 𝐳∈ℝ^L × D, where D and L are the embedding space and time dimensions respectively. On top of the reconstruction and adversarial training objectives of RAVE, we introduce a penalty on the latent codes f(𝐳) = max(0, |𝐳| - 1) to enforce that most latent codes are distributed between -1 and 1. 



 §.§ Model structure



We extract a timbre representation  𝐯_T ∈ℝ^D_T from an audio target using an encoder E_ϕ^T applied to the latent sequence 𝐳 obtained with our audio codec. For structure, we extract a temporal representation 𝐯_S ∈ℝ^L× D_S from either an audio input or an explicit control signal 𝐜 (such as a MIDI sequence), using an encoder E_ψ^S. In the case of an audio input, it would be natural to also infer it from the latent sequence 𝐳. However, we found experimentally that it is particularly difficult to extract fine structure information from the highly-compressed representation 𝐳. Hence, we instead infer structure form the Constant Q Transform (CQT) <cit.> of the target signal, which has been shown to be a well-suited representation for pitch extraction tasks <cit.>. For explicit control, the sequence 𝐜 is directly used as input for E_ψ^S.

To generate audio, we sample a noise vector 𝐳_T and decode it to a latent code 𝐳_0 through latent diffusion conditioned on representations 𝐯_S and 𝐯_T. As diffusion formulation, we leverage the recent improvements introduced in <cit.> and parameterise our denoiser network D_θ to predict the data 𝐳_0 instead of ϵ. E_ψ^S, E_ϕ^T and D_θ are trained end-to-end to minimise the following loss function


    L_diff = 𝔼_t,𝐳_0,ϵ D_θ(√(α_t)𝐳_0 + √(1 - α_t)ϵ, 𝐯_S, 𝐯_T, t) - 𝐳_0


We parameterise D_θ as a 1D convolutional UNet with residual blocks and self-attention layers. The two encoders share a  similar architecture as the encoding half of the UNet, with the difference that the timbre encoder compresses the input temporally and applies average pooling on the time dimension of the last layer. We condition the UNet architecture on 𝐯_S through concatenation with each block inputs. For the timbre vector 𝐯_T we use Adaptative Group Normalisation (AdaGN) <cit.>. 





 §.§ Style and content disentanglement

Although splitting the semantic content between two vectors that are constrained on their dimensions already encourages disentanglement between timbre and structure information, there is no theoretical guarantee regarding their separation. Furthermore, the appropriate feature dimensions are highly dependent on the task and dataset at hand. Hence, to enforce disentanglement without constraints on the dimensions, we introduce a two-stage training combined with an adversarial strategy. First, we freeze the structure encoder and train the model to build an adequate timbre representation. To avoid 𝐯_T to encode all of the information required to reconstruct the target 𝐳, we extract timbre from a different sample 𝐳̃ coming from the same track, following the assumption that it shares the same timbre as 𝐳 but with a different structure. 

In the next stage, we introduce a discriminator D_ζ that tries to predict 𝐯_T from 𝐯_S and is trained to minimise 


    L_D = 𝔼_𝐯_𝕊, 𝐯_𝕋 [ 𝐯_T - D_ζ(𝐯_S)  ].


We train the discriminator alternatively with the encoders and denoising network, which try to minimize the following objective


    L_total =  L_diff - γ L_D,


where γ is an hyperparameter balancing between the reconstruction objective and disentanglement. Indeed, increasing L_D maximises the confusion of the timbre information in the structure space. This restrains the diffusion model from reconstructing 𝐳 solely from 𝐯_S and enables independant control of structure and timbre at inference. 



§ EXPERIMENTS


We aim to asses the ability of our model to generate high-quality audio samples that match characteristics of structure and timbre targets, with the structure being either taken from an audio example or through an explicit control signal. 

MIDI-to-audio For explicit structure control, we evaluate the capability of our model to generate audio from a MIDI score and a target recording for timbre. We compare it to a state-of-art baseline in MIDI-to-audio generation. 

Timbre transfer We evaluate the efficiency of our disentanglement strategy on a task of one-shot timbre transfer between polyphonic mono-instrument recordings. In this case, we consider that structure designates the notes being played (in terms of onset timing, pitch and loudness), while timbre corresponds to the remaining characteristics of the sound. We evaluate our model by randomly sampling timbre and structure examples and evaluate audio quality, timbre similarity as well as note accuracy. We compare our model with two example-based timbre transfer methods on synthetic and real recordings. 




 §.§ Dataset


Synthetic Data The Synthesized Lakh Dataset (SLAKH) <cit.> was generated from the LAKH MIDI collection using professional-grade sample-based virtual instruments. Synthesis parameters as well as audio effects settings were randomly chosen resulting in a very diverse set of timbres. We retain only the individual stems of non-percussive instruments, resulting in 400 hours of audio. 


Real Data To the best of our knowledge there is no multi-instrumental dataset of real recordings that contain a very large number of hours of audio. Hence, we combined the following three datasets to conduct our experiments :

    
  * MaestroV2 : Maestro <cit.> is a piano dataset recorded on Disklavier pianos, capturing both audio and notes played, resulting in approximately 200hours of annotated piano recordings. 

     
  * GuitarSet : Guitarset <cit.> is a collection of live guitar performances with solos and accompaniment from various genres and play styles, with a total audio duration of 6 hours.

     
  * URMP : The URMP dataset <cit.> is composed of pieces played by a large variety of classical instruments. For each piece we retain the mono-instrumental recordings, resulting in approximately 4 hours of audio.


As the GuitarSet and URMP are low sample-size datasets, we add synthetic data stems from SLAKH to the training set to facilitate learning. Furthermore, as the different datasets are greatly imbalanced in terms of sample size, we apply a sampling strategy to even the model performance on each dataset. Following <cit.>, if n_i is the number of samples in a given dataset, we draw examples from this dataset during training with probability (n_i / Σ_j n_j)^0.3.











 §.§ Evaluation metrics


We aim to evaluate how our method is able to match the timbre and structure targets characteristics, while maintaining high-quality audio. 

Audio quality We rely on the widely used Frechet Audio Distance (FAD) <cit.> to evaluate how the generated audio distribution matches the dataset distribution, both for reconstructed and transferred samples. We use the available reference implementation of FAD[https://github.com/gudgud96/frechet-audio-distance/tree/main] and use VGGish <cit.> embeddings of the samples to compute the distance


Timbre Similarity To evaluate timbre similarity we employ the metric proposed in the SS-VAE baseline <cit.>. It relies on a triplet network trained to predict if samples are played by the same instrument based on Mel-frequency cepstral coefficients 2-13. We use their implementation and train the metric on the mixing-secrets[https://www.cambridge-mt.com/ms/mtk/] dataset.

Structure To evaluate if our model is able to reproduce the notes of the structure target, we employ a transcription model <cit.> and compare its output to the ground-truth MIDI data. As metric, we use Onset F1 score from mir-eval, where two notes are considered identical if they have identical pitch and onsets within ± 50 ms of each other. 


 §.§ Baselines


For the timbre transfer experiments, we compare our method to SS-VAE <cit.> and Music Style Transfer <cit.> presented in Section <ref>. We train both models on the real and synthetic datasets, using the official implementation. For explicit control, we evaluate our method against a MIDI-to-audio model <cit.> that was also trained on the SLAKH dataset. 
We use the small configuration of the publicly available pretrained model, as larger models do not fit on our NVIDIA A5000 GPU. 



 §.§ Training details


We start by training our audio codec for 1M steps before training our diffusion model for 500k steps, with an initial timbre learning stage of 100k steps. The overall training takes one day on NVIDIA A5000 GPU. For all experiments we rely on the AdamW optimizer <cit.> with a constant learning rate of 1e^-4 and a batch size of 48. For inference we use the deterministic sampler proposed in <cit.> with 40 diffusion steps. 



§ RESULTS




 §.§ MIDI-to-audio


First, we evaluate our model performance in MIDI-to-audio generation in terms of audio quality, timbre similarity and Onset F1 score. We detail our results in Table <ref> for reconstruction and transfer setups, where the target timbre corresponds to either the instrument of the MIDI sequence or a different sample. In both cases, we obtain higher similarity, as our dedicated timbre embedding captures timbre much more precisely than simple label conditioning on instrument categories. Interestingly, we also obtain better F1 scores, although we did not design our model specifically for MIDI inputs as opposed to <cit.> where authors employ a dedicated note sequence embedding strategy. 

To assess the benefit of our disentanglement strategy, we experiment with bypassing the structure encoder and directly conditioning the UNet on the MIDI sequence (Ours w/o E_S entry in  Table <ref>). This results in overall better Onset F1 score, but degrades timbre similarity and FAD. This demonstrates that our disentanglement strategy improves the capability of the model to precisely render the timbre of the target recording. As described in Section <ref>, the Onset F1 score characterises the difference between the generated notes and the input MIDI sequence. The lower accuracy obtained with our full model in the transfer setup can be explained by the fact that some MIDI sequence are not plausible scores for some target instruments (such as playing chords with a flute). Through the disentangled structure encoding, our model is capable of adapting the input MIDI sequence to the range and capabilities of the target instrument, which results in more realistic sounding samples. We encourage the reader to listen to the examples on our supporting webpage that support this statement. 



 §.§ Timbre transfer


Synthetic data Here, we first evaluate timbre transfer on synthetic data and display the results in Table <ref>. The two baselines appear to provide low audio quality and timbre similarity, and both methods obtain lower Onset F1 scores indicating that they are not able to adequately control structure and timbre independently. Our method improve upon the baselines on all three evaluated aspects, and is interestingly able to reach a comparable performance as in the explicitly conditioned MIDI-to-audio setup.

We also performed an ablation study, by evaluating the effect of applying an information bottleneck on the structure latent space instead of using our adversarial strategy. As mentioned in Section <ref>, the model is able to transfer timbre when D_S is small but achieves a low F1 score. Increasing the latent dimensions improves structure fidelity at the cost of degrading timbre similarity. Using our disentanglement strategy, we are able to employ a 32-dimensional latent space and achieve higher timbre similarity with a slight decrease in note accuracy. Although multiple definitions of timbre transfer are possible, we argue that the most convincing timbre transfer do not necessarily imply a perfect note structure F1 score. In the case of a transfer between monophonic instruments playing in the same pitch range, we can expect all the notes from one recording to be transferred to the other. However, when performing transfer between recordings with very distinct timbre such as a piano playing in its high range and a bass, an interesting transfer would rather be the bass playing the main melodic line a few octaves lower than the piano, which would result in a low F1 score. The improvement in terms of FAD between the distribution of transferred samples and the original data obtained with our disentanglement strategy supports that our method generates more realistic transfers, as an instrument playing notes outside its usual range would be considered as out-of-distribution. 






Real data. We present our results for transfer between real instrumental recordings in Table <ref>. Our model improves even further on the existing baselines for which real instruments timbre seems particularly challenging. Even without adversarial regularisation, our model obtains better FAD, timbre similarity and note accuracy. Our disentanglement strategy further improves timbre match, although the relative decrease in note accuracy appears to be greater than on synthetic data. We believe this is mainly due to a necessary simplification of note structure when transferring complex piano recordings to the mainly monophonic URMP instruments. The improvement of transfer quality captured by the FAD and timbre similarity supports this interpretation. 

Qualitatively, we found that on top of generating realistic samples with the appropriate structure, the model is also able to add characteristic sound artefacts of the target instrument such as fret or hammer noises, as well as matching precise acoustic features of the original recording such as reverb or background noise. 



§ COMPLETE MUSIC STYLE TRANSFER





Finally, we apply our model to the task of creating cover versions of a song that match the style of an example in a different genre. We rely on an in-house dataset of 200 hours of jazz, dub, lofi hip-hop and rock. We use the same model architecture to extract structure from the original track and timbre from the cover targets, with two minor modifications. First, we introduce temporal compression in the structure encoder to avoid 𝐯_T to capture information that is too precisely located in time. Second, we condition the UNet on a BPM time series to help it generating coherent rhythms. Without those modifications, the rhythmic elements from timbre and structure targets are conflicting with each other, resulting in somewhat chaotic generations. We evaluate our model using the cover detection algorithm proposed in <cit.>, which outputs a cover probability based on melodic and harmonic similarities between tracks. To assess style transfer, we rely on the text and audio joint-embedding model CLAP <cit.> and classify genre based on the cosine similarity between the audio and genre label embeddings. 
We compare our method to MusicGen <cit.>, a text-to-music generation model with audio-based melody conditioning. We derive an input prompt from the target genre and use the structure target for melody. 

Our model without regularisation obtains a better cover identification than MusicGen, and our disentanglement strategy further improves transfer resulting in higher genre accuracy. Qualitatively, MusicGen seems to only extract the main melodic idea from the structure audio, whereas our method is able to capture most of the harmonic and melodic content. Furthermore, as our model extract style directly form audio rather than through a text prompt, it transfers the different structural elements towards the instruments actually present in the timbre target rather than just the typical instruments of the genre.





§ CONCLUSION


We presented a simple method to learn disentangled timbre and structure representations. To the best of our knowledge, this is the first model capable of generating realistic, high-quality audio through transfer and MIDI rendering. We leave for future works improvements on the trade-off between reconstruction and disentanglement and applications to more complex musical datasets. Furthermore, we aim to initiate a reflection on the characterisation of the elusive concept of musical style transfer, which we believe to be an exciting stream of research towards a broader use of deep generative models in artistic work-flows. 





























 Paper Title: Self-Supervised Models in Automatic Whispered Speech Recognition.















Self-Supervised Models in Automatic Whispered Speech Recognition 
    Aref Farhadipour
Department of Computational Linguistics 

University of Zurich

Zurich, Switzerland.

aref.farhadipour@uzh.ch
Homa Asadi
Department of Linguistics

University of Isfahan

Isfahan, Iran.

h.asadi@fgn.ui.ac.ir
Volker Dellwo
Department of Computational Linguistics

University of Zurich

Zurich, Switzerland.

volker.dellwo@uzh.ch
    August 5, 2024
============================================================================================================================================================================================================================================================================================================================================================





§ ABSTRACT 
In automatic speech recognition, any factor that alters the acoustic properties of speech can pose a challenge to the system's performance. This paper presents a novel approach for automatic whispered speech recognition in the Irish dialect using the self-supervised WavLM model. Conventional automatic speech recognition systems often fail to accurately recognise whispered speech due to its distinct acoustic properties and the scarcity of relevant training data. To address this challenge, we utilized a pre-trained WavLM model, fine-tuned with a combination of whispered and normal speech data from the wTIMIT and CHAINS datasets, which include the English language in Singaporean and Irish dialects, respectively. Our baseline evaluation with the OpenAI Whisper model highlighted its limitations, achieving a Word Error Rate (WER) of 18.8% on whispered speech. In contrast, the proposed WavLM-based system significantly improved performance, achieving a WER of 9.22%. These results demonstrate the efficacy of our approach in recognising whispered speech and underscore the importance of tailored acoustic modeling for robust automatic speech recognition systems. This study provides valuable insights into developing effective automatic speech recognition solutions for challenging speech affected by whisper and dialect. The source codes for this paper are freely available [https://github.com/areffarhadi/Whisper_fine_tuning_ASRGitHub Repository for Whisper transcription.][https://github.com/areffarhadi/PaddleSpeechGitHub Repository for WavLM fine-tuning.].








§ INTRODUCTION



Recently, self-supervised models have been extensively utilized in various speech processing fields, including speech recognition, speaker recognition, and speaker diarization <cit.>. These models have notably enhanced the efficiency of such systems, particularly for the English language. The training of these models typically involves large datasets comprising speech from different languages, predominantly normal speech. However, these models exhibit diminished performance when processing abnormal speech types such as whispered, reverberant, and disordered speech, as these variants are not represented in the training data.

Whispered speech is a distinct mode of speech production that most individuals can produce naturally and at their discretion. It is often used for privacy or to avoid disturbing others in environments like libraries. Additionally, whispered speech may be a compulsion for individuals with laryngeal or vocal cord damage. The acoustic properties of whispered speech differ substantially from those of normal speech, presenting significant challenges for conventional speech processing systems. 

Furthermore, the presence of different dialects within a language can adversely impact the performance of automatic speech recognition (ASR) systems trained on standard dialects. Variations in dialect result in significant changes in the acoustic parameters of speech, thereby reducing the accuracy of ASR systems when encountering non-standard dialectal speech. 

In this work, we aim to design an ASR system for whispered speech in the Irish dialect using the powerful self-supervised model, WavLM. Among various self-supervised models, WavLM has demonstrated superior performance in different scenarios, particularly in speaker recognition, where the acoustic characteristics of the speaker are more critical than the speech content. Consequently, WavLM has been selected to address the challenge of recognising whispered speech in the Irish dialect.

To illustrate the limitations of conventional speech recognition systems, we will also employ the Whisper model introduced by OpenAI <cit.>, a widely acclaimed model in normal speech recognition. In this article, the Whisper model developed by OpenAI is italicized to differentiate it easily from the concept of whispered speech. 

Developing a speech recognition system necessitates a robust and comprehensive dataset. However, research on whispered speech is limited primarily due to the scarcity of such datasets. Currently, only a few whispered speech datasets, such as wTIMIT <cit.> and CHAINS <cit.>, are available and primarily cater to the English language. While some processes can effectively address some challenges, such as producing reverberant speech data <cit.> as a data augmentation method, they have not yet proven suitable for whispered speech conversion. Despite some efforts in the literature to develop methods for automatically converting normal speech to whispered speech, these artificially generated whispered speech samples tend to be unnatural and noisy <cit.>.

To mitigate the impact of data scarcity, fine-tuning self-supervised models presents a viable solution compared to end-to-end (E2E) methods based on conformers and transformers <cit.>, as suggested in several studies. This approach leverages the existing knowledge encoded in pre-trained models to enhance performance with limited training data. 

Previous work has explored various methods for recognising whispered speech, considering its unique characteristics and the challenge of limited data availability. As one of the first works in whispered speech recognition, <cit.>, designed a system based on hidden Markov model and mel frequency cepstral coefficient features. Some works were also focused on improving the features <cit.> and adapting the acoustic model <cit.>. However, Several works focused on improving pre-processing <cit.> to enhance system performance for whispered speech. 

Some studies have addressed the challenge of limited data for whispered speech by employing generative methods and speech conversion techniques. These approaches include recurrent neural networks, Gaussian mixture models, sequence-to-sequence models,  and various generative architectures <cit.>. For example, in <cit.>, the authors proposed approaches based on MelGAN and VQ-VAE, adapted explicitly for whispered speech conversion.<cit.> focused on low-delayed whisper-to-speech voice conversion, which can be helpful in real-life communication of people. 

In <cit.>, an E2E method was proposed that included a frequency-weighted SpecAugment policy and a frequency-divided Convolutional Neural Network (CNN) feature extractor to better capture the high-frequency structures of whispered speech. This approach resulted in a 19.8% reduction in Phoneme Error Rate (PER) and 44.4% in Character Error Rate (CER) on the wTIMIT corpus. In <cit.>, speakers were divided into four categories based on accent and speech type, and the E2E method with the ESPnet toolbox <cit.> was used. The study found a WER of 36.3% for whispered speech, using pitch and filterbank features without a language model. 

Gudepu et al., <cit.>, generated a synthetic whisper corpus from normal speech using CycleGAN, which was then used to augment data for ASR systems. This approach achieved a 23% relative reduction in WER on whispered speech compared to the baseline. Ghaffarzadegan et al., <cit.>, introduced two strategies to generate pseudo-whisper utterances using vector Taylor series and denoising autoencoders. These strategies significantly reduced WER for a TIMIT-trained speech recogniser on whisper recognition tasks. 

In <cit.>, a new framework using deep denoising autoencoders and Teager-energy-based cepstral features was proposed. This framework improved whisper recognition accuracy by 31% over traditional methods, achieving a 92.81% word recognition rate in mismatched train/test scenarios.

The remainder of this article is structured as follows: Section <ref> discusses whispered speech and its phonetic differences from normal speech. Section <ref> introduces the datasets and the proposed system. The results and findings are presented in Section <ref>, and the conclusion is provided in Section <ref>.



§ WHISPERED SPEECH


Whispered speech is a natural communication mode often employed in public situations to protect privacy or when one wishes to avoid being overheard. This form of speech differs significantly from normal speech in both production and acoustic properties. The fundamental difference between whispered and normal speech lies in their generation mechanisms. Whispered speech is produced by exhaling air and adjusting the shape of the pharynx so that the vocal cords do not vibrate <cit.>.
While normal speech involves vocal fold vibrations to produce sound, whispered speech relies on turbulent, aperiodic airflow passing through a relatively small, nearly constant aperture formed between the vocal folds <cit.>. 

This results in a breathy quality with less distinct harmonic patterns, making it difficult to identify formants in the spectrogram of whispered speech <cit.>.
In terms of production, physiological measurements using magnetic resonance imaging have shown that during the whispered speech, the supraglottal structures are constricted and shifted downward, attaching to the vocal folds to prevent their vibration <cit.>. As a result, speakers produce sounds without fully engaging their vocal cords, leading to a hushed, breathy quality. The resonance of whispered speech usually occurs at significantly higher frequencies than normally phonated speech <cit.>.


Thus, whispered sounds tend to have a higher frequency range, primarily due to increased airflow turbulence. This turbulence generates a noisier signal with less defined formant structures, which are crucial for identifying vowels and certain consonants in normal speech. Consequently, whispered speech can be less intelligible, especially in noisy environments. Furthermore, whispered speech has a lower energy contour, reduced overall amplitude, and a less distinct formant structure, particularly in the lower frequencies, along with a flatter spectral slope <cit.>.

The articulation of phonemes is also altered in whispered speech <cit.>. Research indicates that in whispered speech, the first and second formants shift to higher frequencies, and vowel duration is extended <cit.>. 

Additionally, studies have shown that the duration of consonants is prolonged compared to normal speech, with the intensity of voiceless consonants remaining unchanged. In contrast, the intensity of voiced consonants decreases <cit.>. 

<cit.> found that producing whispered speech demands greater physiological exertion than normal speech. Maintaining the necessary vocal fold position can increase tension in the laryngeal muscles, potentially leading to vocal fatigue if sustained for extended periods. These characteristics pose challenges for speech recognition systems, necessitating different approaches for analysis and synthesis compared to normal speech. 

Despite these challenges, whispered speech serves important functions in communication, often used for privacy, intimacy, or in situations where normal speech volume is inappropriate. Fig <ref> displays the notable differences in the spectrogram characteristics of the speech signals spoken by the same individual in both neutral and whisper modes in Praat software.


As Fig <ref> shows, in whispered speech, the waveform exhibits lower amplitude and lacks the regular periodicity observed in normal phonation, which indicates the absence of vocal fold vibrations. The frequency content in whispered speech is more diffuse and less defined, characterized by an evenly distributed energy spectrum and a lack of pronounced harmonic structures. The overall intensity is lower, as evidenced by the lighter grayscale in the spectrogram, and the yellow intensity contour line displays lower and less stable intensity levels compared to normal speech. 

Conversely, normal speech is characterized by higher amplitude and regular periodic patterns, with distinct harmonic bands and well-defined speech formants. The spectrogram of normal speech shows higher intensity levels, indicated by darker areas, and the yellow intensity contour line is more stable and reaches higher intensity levels, reflecting concentrated energy at specific frequencies.




§ METHODOLOGY


In this section, the WavLM model as the core of the proposed system and the CHAINS dataset are introduced as the evaluation dataset utilized in this study.



 §.§ WavLM

The WavLM network, a self-supervised learning (SSL) model introduced by Microsoft in 2021, aims to provide a comprehensive solution for various speech processing applications. WavLM was designed and trained for full-stack speech processing differently from other SSL models, achieving superior performance in tasks such as speaker recognition and separation, which demand an efficient understanding of acoustic models. The training process of this model involved masked speech prediction and denoising, which were performed jointly. This dual-task training strategy enables the model to predict masked speech content while learning to denoise, thereby equipping it with the ability to handle multiple speech processing scenarios beyond ASR.


WavLM was pretrained on 94,000 hours of speech data, encompassing a variety of speech types, including read and spontaneous speech, allowing it to model a broad spectrum of speech characteristics. According to benchmarks, WavLM outperforms other SSL models across many scenarios. The model architecture incorporates a gated relative position bias in the transformer self-attention mechanism to enhance its performance. 

As depicted in Fig <ref>, WavLM consists of a CNN feature encoder and a transformer encoder with gated relative position bias. The CNN encoder comprises seven blocks of temporal convolution, followed by normalization and GELU activation layers, with each convolutional block representing 25 milliseconds of the input signal. The transformer section includes position embedding with a kernel size of 128 and 16 groups at its bottom <cit.>. 

By predicting the masked noisy/overlapped parts, the model learns the speaker's acoustic information from the unmasked segments and estimates the masked parts using the speaker's characteristics. This strategy significantly enhances the model's ability to learn and generalize acoustic information. 

In this network, let X = {x_1, x_2, …, x_T} represent a speech utterance with T frames, and let the discovered hidden units be Z = {z_1, z_2, …, z_T}. Let M ⊂ [T] denote the set of indices to be masked for the length-T sequence X. The corrupted version of X, denoted as X̂ = r(X, M), is defined such that x_t is replaced with a mask embedding x̂ if t ∈ M <cit.>.



 §.§ Whisper Dataset

The CHAINS dataset is utilized to evaluate the proposed system. This dataset includes both normal speech, referred to as solo, and whispered speech, known as whsp. The dataset comprises recordings from 36 speakers, dominantly with Irish dialects. To develop a model capable of accurately capturing the acoustic features of the Irish dialect, the normal speech portion of this dataset is used for system training, while the whispered speech portion is employed for evaluation. Additionally, to improve the model's performance on a whispered speech by learning its unique acoustic characteristics, the whispered speech portion of the wTIMIT dataset, which includes Singaporean and US dialects, is also used as auxiliary data in the training process. 

In this evaluation, only the Singaporean dialect whispered speech from wTIMIT is used for training, ensuring that the system encounters different dialects from those used during pre-training, thereby fostering a more general acoustic model. Each speaker's evaluation data involves 33 whispered utterances from the CHAINS dataset. Further details of these datasets are presented in Table <ref>. 




§ EXPERIMENTAL RESULTS


According to the results from E2E systems based on conformers and transformers, which typically require substantial amounts of data, the utilization of pre-trained self-supervised models can be an effective strategy to address the data scarcity challenge in whisper ASR. As previously mentioned, conventional systems are inadequate for recognizing whispered speech. To validate this claim, a baseline system utilizing the OpenAI Whisper model—one of the most powerful speech recognition systems for the English language—has been employed. No fine-tuning was performed on the Whisper model for this work. We evaluated this system using whispered speech from the CHAINS dataset, with transcription performed using the beam search algorithm.

The ASR results using the Large-v2 version of the Whisper model as the baseline system are presented in Table <ref>. This system achieved a WER of 16.69% for normal and 18.8% for whispered speech parts of the CHAINS dataset. The speech texts in the dataset are identical for both speech styles. As shown, the system experienced a drop in performance when dealing with whispered speech. In addition, the performance of the Irish dialect was lower than that of the typical English <cit.>. These results indicate that whispered speech with a different dialect presents substantial modeling complexities for a system trained predominantly on normal speech data, underscoring the need for a dedicated system for whispered speech recognition.

In the subsequent step, the proposed system was trained using the Base+ version of WavLM model. This model comprises 94.7 million parameters and was pr-etrained over one million steps on 94,000 hours of speech data. The proposed system was fine-tuned using 17 hours of wTIMIT whispered and normal speech with a Singaporean dialect, along with 3 hours of normal speech with an Irish dialect from the CHAINS dataset. This training of WavLM was conducted using the PaddleSpeech toolbox <cit.> on an NVIDIA A100 80GB GPU over 25 epochs with a batch size of 6 and an adaptive learning rate schedule. SpecAugment <cit.> with time masking was employed as a data augmentation method, and the AdamW algorithm <cit.> was used for optimization. Decoding was performed with a beam search size of 1 and a greedy search size of 16. The results of this system are presented in Table <ref>.



The proposed WavLM-based system achieved a WER of 9.28% using the greedy search algorithm and 9.22% based on the beam search algorithm. These results demonstrate that the system effectively learned the acoustic parameters of whispered speech, significantly outperforming the OpenAI Whisper model baseline system. This improvement can be attributed to including whispered speech data in the training phase. Additionally, the Irish dialect, which posed a challenge for the baseline system, was effectively addressed during training using normal speech data from the CHAINS dataset. The choice of the WavLM network proved advantageous due to its suitability for training on small datasets and its superior acoustic modeling capabilities.



§ CONCLUSION


In this article, we proposed a system for speech recognition under the challenging conditions of whispered speech and the Irish dialect. To address these challenges, we utilized training data with an acoustic space similar to the test conditions, enabling the system to effectively model the complex acoustic characteristics in the test data. The results demonstrate that the proposed system, with a WER of 9.22%, outperforms the powerful OpenAI Whisper system, which served as a baseline. Our findings indicate that the proposed system achieves superior performance on the CHAINS dataset for whispered speech recognition.




IEEEtran









 Paper Title: Emotion-driven Piano Music Generation via Two-stage Disentanglement and Functional Representation.



[
    [
    
=====


§ ABSTRACT 

Managing the emotional aspect remains a challenge in automatic music generation.
Prior works aim to learn various emotions at once, leading to inadequate modeling. 

This paper explores the disentanglement of emotions in piano performance generation through a two-stage framework. 

The first stage focuses on valence modeling of lead sheet, and the second stage addresses arousal modeling by introducing performance-level attributes. 

To further capture features that shape  valence, an aspect less explored by previous approaches, we introduce a novel functional representation of symbolic music.

This representation aims to capture the emotional impact of major-minor tonality, as well as the interactions among notes, chords, and key signatures.

Objective and subjective experiments validate the effectiveness of our framework in both emotional valence and arousal modeling. 

We further leverage our framework in a novel application of emotional controls, showing a broad potential in emotion-driven music generation.




§ INTRODUCTION


With the recent advancements in symbolic music generation <cit.>, there has been a growing interest in controlling high-level musical features throughout the generation process.

Among these features, emotion-driven music generation <cit.> aims to generate music that conveys specific emotions, representing a crucial aspect for music appreciation and analysis. 

The downstream applications of such models have also been explored, such as music therapy for healthcare and educational purposes <cit.> and soundtrack generation for videos and movies <cit.>.


Emotion could be represented in two dimensions from the literature <cit.>: valence and arousal.

Valence refers to the positiveness of an emotion and arousal refers to energy or activation <cit.>.

These two dimensions can be further divided into four quadrants (4Q), namely high valence high arousal (Q1),  low valence high arousal (Q2), low valence low arousal (Q3), and high valence low arousal (Q4). 

In this paper, we focus on the emotion-driven piano performance generation of these four quadrants. 

Throughout prior works, we observe crucial challenges from the perspectives of both model design and musical inductive bias.


First, previous emotion-driven piano performance generation models <cit.> attempt to learn emotion quadrants and expressions in an end-to-end paradigm. 

In terms of model design, this approach poses training difficulty on the generation model, leading to the instability in achieving results of desired emotions. 

For example, many existing works <cit.> could effectively control the arousal levels of music, while their performance of valence modeling, especially in generating low valence (i.e., negative) music, is still poor. 

In terms of music, the creation process of music typically involves multiple stages, such as the lead sheet composition for melodies and chord progressions, and performance generation for textures and expressiveness. 

Consistently, emotion can be evoked through a combination of musical elements (e.g., melody, chord, texture). 

For example, major/minor chords have been found to seize different valence trends in psychological studies <cit.> and performance-level attributes like articulation, tempo, and velocity are more related to arousal <cit.>. 

It is worth to explore the potential relation between the disentanglement of the generation process and the emotion expression. 










Second, previous emotion-driven generation models have received limited attention regarding the influence of tonality on emotion modeling.

It has been widely shown that major-minor tonality in composition is highly related to valence perception <cit.>. 

For example, as depicted in Figure <ref>, the histogram of musical keys derived from the emotion-labeled music dataset EMOPIA <cit.> supports the distribution skews to major keys for high valence clips and opposite trend for low valence ones. 

Furthermore, different tonalities may reveal similar patterns in the relative relationships between melodies and chords, while the distribution of melodies, chords, and tonalities can exhibit distinct shapes across different emotions.

Current representations of symbolic music, such as REMI <cit.> and CP-Word <cit.>, do not explicitly incorporate such interactions nor address its connection to emotion adequately.


Therefore, it is necessary to consider a functional format of symbolic music representation considering the relationships between notes, chords and key signatures to better model the tonality in the emotion-driven music generation process.

In this paper, we contribute to combat above challenges:


    
  * We employ a two-stage Transformer-based model on emotion-driven piano performance generation. The first stage focuses on valence modeling via lead sheet composition, while the second stage addresses arousal modeling by introducing performance-level attributes.
    
    
  * We propose a novel functional representation for symbolic music, encoding both melody and chords with Roman numerals relative to musical keys, to consider the interactions among notes, chords and tonalities <cit.>.
    
    
  * Experiments demonstrate the effectiveness of our framework and representation on emotion modeling. Additionally, our method enables new capabilities to control the arousal levels of generation under the same lead sheet, leading to more flexible emotion controls.


As a minor contribution, we also refine key signature labels and extract lead sheet annotations for the EMOPIA dataset <cit.> to ensure the correct training of the two-stage framework. 

We share the data, open source our code[<https://github.com/Yuer867/EMO-Disentanger>] and present generation samples in the demo page.[<https://emo-disentanger.github.io/>]

















































§ RELATED WORK




 §.§ Emotion-driven Piano Performance Generation
 

Prior works apply emotion conditions on deep-learning models to guide the generation of piano performance <cit.>, or develop searching methods to generate music of desired emotions <cit.>.

Musical elements via feature disentanglement <cit.> or supervised clustering <cit.> can further be regarded as a bridge between emotion labels and performances for generation.

In contrast, our framework employs a two-stage generation approach to reduce the complexities of one-stage generation, fostering a more nature process of music creation as well as a better incorporation between emotion labels and generation results.





 §.§ Tonality, Functional Harmony, and Emotion


Musical keys and functional harmony have been explored in the field of roman numeral analysis <cit.>. 

The analysis of how modes and tonalities relate to mid-level perceptual features (e.g., dissonance, tonal stability, minorness) and affect the emotional perception of music pieces has also been discovered <cit.>.

While some music generation works attempted to combine key information into data representation <cit.>, loss function <cit.> and text conditions <cit.>, none of them explore the relation between musical keys and emotional perception. 

In this paper, we leverage both functional harmony knowledge and class-octave based pitch representation <cit.> to design a new data representation, incorporating the relationships between notes, chords and keys for emotion-driven music generation.



§ METHOD

In this section, we will first introduce the functional representation of symbolic music as the main generation unit.

Then we introduce the two-stage model as the main component of the emotion disentanglement and generation. 



 §.§ Functional Representation


Figure <ref> illustrates our proposed functional representation.

Its design is initially based on REMI <cit.>, a widely used event-based representation for symbolic music. 

We incorporate different note and chord events assisting to better learn the joint information of emotion and key signature.





  §.§.§ Emotion and Key Events


We follow CTRL <cit.> to set up the condition within the autoregressive generation process in Transformer architecture.

To denote distinct emotions and affect overall properties, we begin the event sequence with  event to indicate the emotion label of music clips. 

The  event is appended after  to provide the musical key property, with the total of 24 keys (12 tonic notes with two modes in EMOPIA <cit.>). 



  §.§.§ Bar, Sub-Beat, Tempo and EOS Events


Similar to REMI, 

a  event denotes the new start of a bar; 

a  event denotes one of 16 possible discrete beat locations within a bar;

a  event denotes local tempo changes every four beats;

and an  event denotes the end of sequence. 



  §.§.§ Chord Events


A musical chord name typically consists of root note and chord quality. 

For example,  represents the chord  with root  and major quality.


Such symbols describe correct note information in chord within the tonality, but they overlook the variations in chord functions of the same chord across different tonalities.

For example, while  serves the tonic function in F major scale, it serves the subdominant function in C major scale. 

Moreover, the chord progression follows these functional harmony rules to establish tonality and convey musical emotion <cit.>.

To introduce chord functions in the emotion modeling, we adopt Roman numerals from Roman Numeral Analysis <cit.> to notate chord roots in Figure <ref>.

Given the  event, root notes in the absolute pitch are directly converted into Roman numerals based on their scale degrees relative to the key (i.e., relative pitch). 

For roots outside the scale, we employ a direct conversion for , , ,  and  appearing in major keys, but randomly assign  and , which only appear in minor keys, as one of their neighboring degrees during the encoding and decoding process. 

This design ensures the notation to be key-independent and make every conversion of notes reasonable to the music theory. 

The notations of chord qualities remain unchanged, and the chord event  appears every four beats.





  §.§.§ Note-related Events
 

A note is denoted by ,  and  events, where  event indicates the onset of pitches from A0 to C8. 

Inspired by <cit.>, we decompose  into  and  events according to the note octave and degree in the certain key scale. 

The conversion rule from  to   is the same as that of chord roots in Figure <ref>.

For example, pitch D#4 is decomposed into  and  in c minor scale, but  in D# major scale.

Such degree-octave pitch representation narrows the difference between melodies, thus improves the learning of connections between emotions, chords, and melodies, as demonstrated in Figure <ref>.




 §.§ Two-stage Emotion Disentanglement


We use the idea of Compose & Embellish <cit.> to generate music in two stages: lead sheet first, and then piano performance. While Compose & Embellish is emotion-agnostic, we extend it so that the lead sheet model involves valence modeling and the performance model  arousal modeling. 











  §.§.§ Valence Modeling



The top left section of Figure <ref> denotes the first stage, where only emotion events  and  are considered as conditions.

The former includes music pieces of Q1 and Q4 (high valence) and the latter includes those of Q2 and Q3 (low valence).

The lead sheet model first predicts a key event k conditioned on the given emotion event e, and then generates the lead sheet sequence M={m_1, ⋯ m_T} of length T, as melody and chord progression, conditioned on previous tokens step-by-step:

    p(k, M|e) 
        = p(k|e)∏_t=1^T p(m_t | e, k, M_<t)  ,

where p(k|e) and p(m_t | e, k, M_<t) are jointly learned through the Transformer-based generation model <cit.>. 

Performance-related events  and  are removed in the first stage (i.e., lead sheet generation), as we mainly focus on the contributions of key, pitch and chord for valence perception.




  §.§.§ Arousal Modeling

The top right section of Figure <ref> denotes the second stage.

Given the lead sheet M, the performance model generates performance X conditioned on the true emotion label (Q1 to Q4). 

As the valence aspect has already been modeling in the first stage, this stage focuses on the generation of musical textures for the lead sheet, and more importantly, on how to perform it through variations of tempo, velocity, articulation, and other performance-level attributes that largely influence perceived arousal <cit.>.

During the training and inference phases, with the positions of  events, M and X are further segmented into {M_1, ⋯, M_b} and {X_1, ⋯, X_b}, where b is the number of bars. 

The segmented sequences are “interleaved” in the form of {⋯, M_i, , X_i ⋯} with additional  events to distinguish M and X tracks.

In that, the target performance bar X_i is appended to its corresponding conditions M_i, as mapping each lead sheet segment to its corresponding performance segment <cit.>.

With the emotion condition and key event from lead sheet as prefix tokens, the performance model is summarized as

    p(X|e, k, M) = ∏_i=1^b p(X_i|e, k, M_≤ i, X_<i)  .






  §.§.§ Training Objectives

Lead sheet and performance models are trained separately by both optimizing the negative log-likelihood loss of the sequence.

Since existing emotion-labeled music datasets are not large, we leverage large-scale music datasets without emotion annotations to pretrain both models for better music understanding.

During pretraining, the emotion event is marked as .

We then finetune two models on the emotion-labeled dataset (detail in Section <ref>)
to learn composition and performance styles specific to different emotion contexts.





  §.§.§ Two-stage Inference
 
The left bottom section of Figure <ref> denotes the inference process of both models.

In the first stage, the lead sheet model predicts the key event and generates the lead sheet sequence step-by-step given  or  event, creating a musical motif for the specific valence preference.

Even though our framework has the capability to generate any-key music of specific emotions, we observe that some generation results, such as a high-valence and high-arousal song with a minor key scale, may go beyond the current definition of emotion in <cit.>, where the valence naturally has a strong correlation to the major-minor tonality (Figure <ref>).

Therefore, we limit major keys to  and minor keys to  during the inference stage. 

We acknowledge that this can be overly simplifying.

Since this paper focuses mainly on the valence-arousal disentanglement during the generation process, we leave this exploration of generating any emotion within any key as an advanced topic for future research. 

In the second stage, the performance model generates piano performance with desired valence and arousal combination given the lead sheet from the first stage. 

For example, to generate a music piece of Q3, a “Negative” lead sheet and a “Q3” emotion event are selected as conditions.

Additionally, this two-stage framework enables the flexibility to generate different arousal levels of piano performance under the same lead sheet, delivering some scenarios when the music need to shift quickly to complement the scenes in movies or daily videos (detail in Section <ref> and the demo page). 





§ EXPERIMENTS




 §.§ Datasets and Preprocessing

As presented in Table <ref>, we collect different datasets for pretraining and finetuning phases as mentioned in Section  <ref>. 

For pretraining the lead sheet model, we use the HookTheory dataset <cit.>, where we choose 18,206 lead sheets with high-quality and human-transcribed melody, chord and key annotations in 4/4 time signature. 



We simplify 249 chord quality classes into 11 types[Major, minor, augment, diminish, suspend2, suspend4, major7, minor7, dominant7, diminish7, half-diminish7] as the same set in the other datasets below.

For pretraining the performance model, we use the Pop1k7 dataset <cit.>, consisting of 1747 transcribed pop piano performances.

Since Pop1k7 does not contain lead sheet annotations, we refer <cit.> to extract melodies using the skyline algorithm <cit.>, recognize chords using the chorder library <cit.>, and detect key signatures using <cit.> in MIDI Toolbox<cit.>.

For finetuning the models with emotion conditions, we use the EMOPIA dataset <cit.>, consisting of 1,071 music clips with human-annotated emotion labels. 

Similar to Pop1k7, we obtain the lead sheets of EMOPIA by extracting melodies using the algorithm in <cit.> and recognizing chords using the algorithm in<cit.>.


Empirically, we obverse that specifically in the EMOPIA dataset, melodies and chords extracted by these alternative algorithms are more correct compared to the skyline algorithm and the chorder library. 


Additionally, we found the key signature labels in EMOPIA are not fully correct since they are also obtained by the detection algorithm with error rates.


Since the valence modeling is strongly related to the musical keys and modes, we manually correct the key annotations of 367 clips in EMOPIA to ensure a high quality of lead sheets.

All datasets are randomly divided into respective training and validation sets at the ratio of 9:1. 

As a result in our functional representation, the vocabulary size of events is 215 for lead sheet and 324 for piano performance.








 §.§ Model Settings

The lead sheet model is a 12-layer Transformer Decoder <cit.> with 8 heads, 512 hidden dimensions and relative positional encoding <cit.>. 

The performance model is similar to the lead sheet model except with Performer attention <cit.>.

The total parameter sizes are 41 million and 38 million respectively.



Both models are trained with the batch size of 4, the maximum sequence length of 512 (lead sheet model) or 3072 (performance model), and the Adam optimizer with β=(0.99,0.9).

We adopt a 200-step warm-up to achieve the maximum learning rate of 1e-4 for pretraining and 1e-5 for finetune.

All models are implemented by PyTorch and trained on one NVIDIA Tesla V100 GPU. 

The lead sheet model took around 180,000 steps to converge and the performance model took around 200,000 steps.


Nucleus sampling <cit.> is employed in the inference phase.

We referred <cit.> to choose the sampling hyperparameters τ=1.2, p=0.97 for the lead sheet model and τ=1.1, p=0.99 for the performance model.



 §.§ Baseline and Ablations

We consider the emotion-driven piano performance generation model in EMOPIA <cit.> as our baseline, which generates music in an end-to-end paradigm instead of two stages. 

To ensure the fair comparison of generation performance, we trained the baseline model under ths same datasets in both pretraining and finetune phases, and replaced the original CP-Word representation with REMI as the former usually yields better generation performance and more comparable to our proposed functional representation.  


Two other related works <cit.> are not included in comparison due to the main reason that we focus more on the evaluation of the two-stage framework in valence and arousal modeling; and the partial reason that they are not open-source or releasing the reproducible model weights.

We conduct a comprehensive ablation study to evaluate if each proposed design benefits the emotion modeling of music generation. 

Specially, these designs include: 1) the two-stage generation, 2) the functional representation, and 3) the dataset pretraining.


In the following sections, models are denoted as <representation(stage)>. For example, REMI(one) denotes the one-stage generation model with REMI representation as the baseline, and REMI(two) denotes the two-stage generation as one variant.






 §.§ Objective Evaluation and Results

Even though previous studies <cit.> employ metrics, such as Pitch Range (PR) and Number of Pitch Classes (NPC), to evaluate the generation performance, they do not provide any evidences on the superiority of  melody development, chord progression, and texture arrangement of music. 

Therefore, a model with more similar PR and NPC values to those of the target dataset does not necessarily promise a better generation quality than others.

Instead of using such metrics, we wish to evaluate the consistency between the input musical conditions and the generation results. 

We introduce key consistency to assess if a model can generate music pieces that adhere to the desired input key signatures, which is highly correlated to the lead sheet development and valence modeling. 

Specifically, key consistency measures the match between the key condition  and the actual key detected in the generation using the algorithm <cit.> with an 81% accuracy rate. 

We compare REMI and our functional representation to determine if the functional representation can improve the key consistency via more close and interactive designs on key, melody, and chord. 

Since this metric requires the key as conditions, we add the  event in REMI after  (as REMI+key in Table <ref>) when training the model. 

The non-pretrained versions are also included for comparison.

Each model generates 200 lead sheets (100 high and 100 low valence) and 400 performance samples (100 per emotion quadrant) for evaluation.

From Table <ref>, the functional representation outperforms REMI (i.e., REMI+key) across all components and even achieves compatible accuracy to real data over the lead sheet component.

This demonstrates the effectiveness of the functional representation, by representing notes and chord roots relative to key events for key modeling.

In contrast, REMI struggles with associating chord events with keys due to the ignorance of chord labels serving different functions in different key scales. 

Moreover, pretraining process introduces musical priors to enhance the learning of key relationships with other musical elements, improving key consistency for both representations.







 §.§ Subjective Evaluation and Results

We leverage an online listening test to assess the emotion modeling ability of models. 

The test was conducted to collect user responses on three parts: 1) valence modeling, 2) arousal modeling, and 3) 4Q emotion modeling. 

During this test, the quality of the generated music has also been assessed implicitly as it is a prerequisite to the emotion expression in the music.

22 participants were engaging in this test, 5 with less than 2 years of musical training, 8 with 2-5 years, 3 with 5-10 years, and 6 with more than 10 years.




  §.§.§ Valence Modeling


In this part, each participant listened to 16 generated tracks of piano performance from four models [four tracks (two high valence and two low valence) per model]: 1) Real data; 2) REMI (one); 3) REMI (two); 4) Functional (two).


For each track, participants rated its positiveness from –2 (low valence) to 2 (high valence) with the step size 1.

The left of Figure <ref> (`(a)') presents the mean opinion scores for the valence-oriented test, where the Functional (two) model significantly outperforms both REMI (two) and REMI (one) models.

The REMI (two) model shows a slight improvement over REMI (one) due to its two-stage design.

Our proposed Functional (two) model even marginally exceeds real data in low valence scores, which could be due to the potential subjective biases in the negative emotion as discussed in <cit.>. 

And our model achieves both great performance in high valence and low valence results, demonstrating a good balance in valence modeling.





  §.§.§ Arousal Modeling


In the second part, the functional (two) and REMI (two) models are chosen to compare their arousal modeling performance.

Specifically, we wish to explore whether they can generate piano performance with either high or low arousal under the same lead sheet based on the given conditions (Q1 and Q4 for positive lead sheets, Q2 and Q3 for negative ones). 

Two pairs of generated tracks are randomly drawn for each model and each valence level, where every pair includes two tracks of different arousal conditions.

For each track, participants rated its arousal level from –2 (low arousal) to 2 (high arousal) with the step size 1. 

The right of Figure <ref>  (`(b)') presents the results. 

The Functional (two) model surpasses REMI (two) by an average of 0.5 point, highlighting its superior ability to differentiate between the musical features of the two arousal levels through performance. 

Additionally, it is rare for Functional (two) to be incorrectly identified as high arousal under low arousal conditions (Figure <ref> (b-2)).



  §.§.§ 4Q Emotion Judgement


In the last part, participates needed to choose the best option from four options (4Q) for each track, with 8 tracks in total for the two models the last section (4 tracks per model and 1 track per emotion).

Figure <ref> presents the confusion matrices of two models.

The Functional (two) model achieves the higher overall accuracy than that of REMI (two) (71.5% vs. 31.0%).

When examining each emotion category, Functional(two) demonstrates superior performance in Q3 and Q4 than Q1 and Q2. 

Furthermore, music pieces generated from it with high valence conditions are misidentified almost based on their arousal levels; for instance, pieces intended for Q1 are almost mistaken for Q4 and vice versa. 

In contrast, for REMI(two), the misclassifications are across all categories, demonstrating its limitations in modeling the four emotion classes although through two-stage generation.



All above evaluations support that the combination of two-stage framework and functional representation is effective in controlling the emotion of the music it generates to a certain extent.















§ CONCLUSION AND FUTURE WORK

In this paper, we first explore emotion disentanglement through a two-stage Transformer-based framework for emotion-driven piano performance generation. Then we propose a novel functional representation for symbolic music to capture the interactions among musical keys, modes, chords, and melodies in relation to the emotion contexts. 

An objective metric is designed to qualify the key modeling of the proposed method, and subjective evaluations further confirm its ability to convey desired emotional perception. 

In the future, we wish to focus on enhancing the flexibility of emotional music generation across all musical keys and investigating new applications fostered by our framework, such as the controls of valence and arousal attributes under similar music motifs.











§ ACKNOWLEDGMENT


The work is supported by a grant from the National Science and Technology Council of Taiwan (NSTC 112-2222-E-002-005-MY2). This work also benefits from the review comments of an unpublished work <cit.>, which is an early version of the present study. While the concept of functional representation has been mentioned in that previous work <cit.>, its focus was on emotion-driven melody harmonization and the effectiveness of the functional representation in that context was unclear, as the melody was pre-given and fixed. In the current work, melody and chords are generated together, allowing more flexibility for emotion control with the proposed functional representation. The source code of that previous work can be found at the GitHub repo: <https://github.com/Yuer867/EMO_Harmonizer>.





























 Paper Title: On the Problem of Text-To-Speech Model Selection for Synthetic Data Generation in Automatic Speech Recognition.



Spin-wave Goos-Hänchen effect induced by 360 degree domain walls in magnetic heterostructures
    Jie Lu
    August 5, 2024
=============================================================================================




§ ABSTRACT 
The rapid development of neural text-to-speech (TTS) systems enabled its usage in other areas of natural language processing such as automatic speech recognition (ASR) or spoken language translation (SLT).
Due to the large number of different TTS architectures and their extensions, selecting which TTS systems to use for synthetic data creation is not an easy task.
We use the comparison of five different TTS decoder architectures in the scope of synthetic data generation to show the impact on CTC-based speech recognition training.
We compare the recognition results to computable metrics like NISQA MOS and intelligibility, finding that there are no clear relations to the ASR performance.
We also observe that for data generation auto-regressive decoding performs better than non-autoregressive decoding, and propose an approach to quantify TTS generalization capabilities.




§ INTRODUCTION


The usage of synthetic data from text-to-speech (TTS) systems in the context of automatic speech recognition (ASR) has been explored in many different ways.
*Work done while being an internship student at JAIST
This ranges from simply generating new data <cit.> to jointly training ASR and TTS systems<cit.>.
Prior work improved the integration of the synthetic data into the ASR systems, for example by different weighting of synthetic inputs <cit.>, or the usage of internal ASR model statistics to reject synthetic samples not matching the real data<cit.>.
As most publications aim to introduce new methodologies, in each case they usually use one specific TTS architecture and one specific ASR architecture.
The number of fundamentally different ASR architectures is rather limited and roughly categorized into: Hybrid deep neural network hidden Markov model (DNN-HMM) <cit.>, connectionist-temporal-classification (CTC) <cit.>, Transducer <cit.> and Attention-Encoder-Decoder (AED) <cit.>. For TTS there is a vast amount of different architectures and models, especially when including different approaches of multi-speaker modeling<cit.>.
From the perspective of working on ASR this makes it very difficult to choose which TTS system to use for synthetic data generation.
In addition, human evaluations are not suited to evaluate data for training purposes, as conducting the evaluation usually takes more time than running the training itself. 
Then, evaluation for naturalness and human subjective evaluations alone can not be expected to be a sufficient criterion for utility estimation of synthetic data.
Instead, aspects like speaker diversity, temporal diversity or expressiveness in general are assumed to be relevant to yield good results by integrating synthetic data into the ASR process<cit.>. Unfortunately, those aspects are difficult to quantify.

Furthermore, LJSpeech <cit.> is often the only common dataset to compare new TTS architectures, which can not be expected to give representative results for the multi-speaker case<cit.>.
This is especially relevant as one might not have training data suited for TTS training at hand for a particular domain or language, but instead has to use the ASR data.

Motivated by those problems, we present an experiment pipeline to systematically analyze the relation of TTS generated data to ASR training.
For this we compare different decoder architectures, which are based on: Non-autoregressive (NAR) Transformer similar to FastSpeech-2 <cit.>, autoregressive (AR) long-short term memory (LSTM) similar to Non-attentive Tacotron <cit.>, Glow-TTS <cit.> as flow-based decoder and Grad-TTS <cit.> as diffusion based decoder.
We keep the rest of the TTS systems as simple as possible, using a Transformer encoder and convolution based duration prediction network, as used in <cit.>.

Given the large amount of possible TTS architectures, we choose these systems for multiple reasons:
First, they have a widespread recognition in the scientific community. Then, they already share similar encoder and up-sampling concepts, sometimes even using the same code, which makes comparison easier. Finally, they do not require training with an integrated vocoder model (as e.g. in VITS) which would make a direct comparison more difficult.  
While comparisons of different TTS architectures with respect to naturalness and expressiveness exist <cit.>, this is to our knowledge the first work covering a broader range of different TTS output characteristics and quality levels targeted for ASR training.
Also, we perform a direct comparison of NAR vs AR decoding for data generation with a consistent sequence mapping method.

Lastly, we present a way of measuring generalization of the TTS systems with respect to ASR training data generation by using different conditions for synthesizing the training data. The Sisyphus <cit.> based setup and additional materials are available on Github[<https://github.com/rwth-i6/i6_experiments/tree/main/users/rossenbach/experiments/jaist_project>].


§ TEXT-TO-SPEECH SYSTEMS




 §.§ General architecture


In the design of the TTS systems we follow what we found to be most prevalent in recent literature. After the introduction of Transformer blocks with convolution for TTS <cit.> and the establishing of duration based state up-sampling <cit.>, many systems continued using these components. Thus, for each system we use the same layout of a Transformer based-encoder, a convolution based duration predictor and a discrete encoder state up-sampling method. Only the decoder, which converts the up-sampled encoder states h_1^T into the final log-Mel spectrograms x_1^T, is changed. Figure <ref> shows the TTS architecture used in this work. We do not use explicit f_0 or energy prediction. The overall structure follows exactly Glow-TTS, in which many parts were taken from TransformerTTS <cit.> or FastSpeech-2. We also used the code of the official Glow-TTS repository as reference for our implementation.


 §.§ Decoder variants




A large amount of proposed TTS decoder architectures can be categorized into three groups: Direct prediction, flow and diffusion. Direct prediction means that the output of the TTS neural network is a log-Mel spectrogram which is trained to directly match the target x_1^T. Common losses are mean absolute error or mean squared error. The generation is a single forward pass through the decoder network g_θ, which can be written as:

    x̂_1^T = g_θ(h_1^T)

The generation can also be done in an auto-regressive manner:

    x̂_t = g_θ(h_1^t, x̂_1^t-1)

For flow-based TTS systems, the decoder is an invertible neural network that can be applied in both directions from the spectrograms x_1^T into latent space variables z_1^T and vice-versa. During training, the target features are transformed into the latent space and matched against a normal distribution with a mean μ_1^T = μ_θ(h_1^T) and unit variance. The target objective is to maximize the likelihood of the latent space samples given the distributions. For inference, the latent space values are sampled from the distribution with a mean function μ_θ, temperature τ, and passed backwards through the flow f_θ:

    ẑ_1^T ∼𝒩(μ_θ(h_1^T),τ1)    and   x̂_1^T = f^-1_θ(ẑ_1^T)

For the diffusion-based TTS system we follow the Grad-TTS architecture. A normal distribution as in the flow-based approach is used to sample a noisy output spectrogram X̂_I with X = x_1^T. The noise spectrogram is cleaned in I steps using a differential equation containing the solver network s_θ and pre-defined noise schedule β_i to reach the final output X̂_0:

    X̂_I    ∼𝒩(μ_θ(h_1^T),τ1)
    X̂_i-1   = X̂_i - 1/2I(μ_θ(h_1^T) - X̂_i - s_θ(h_1^T, X̂_i))β_i

For each of the decoder principles we implemented one model for which the details will be presented later in Section <ref>.

























§ EVALUATION PIPELINE AND METRICS



















To evaluate the TTS systems in the context of ASR, we train a CTC-based ASR system purely on data generated by TTS. We generate the same amount of data  as used for the baseline training, and measure the performance of the ASR systems in word error rate (WER) for recognizing real human speech. In the optimal case, the performance of the ASR system trained on the synthetic data would match the performance of the system trained on real data. In order to evaluate the generalization capabilities of TTS towards new conditions, we generate the synthetic data in 3 ways: a) using exactly the same text and speaker assignment as in training, b) using the same text but shuffling the speakers and finally c) using the same amount of text from a different corpus in the same domain. That way we can measure how good the TTS systems are at generalizing to new conditions different from the training. The main evaluation criterion to measure the utility of the TTS-generated data is the WER of the trained ASR systems calculated on the official LibriSpeech <cit.>  test sets. In addition, we use metrics directly on the synthetic data to evaluate their relation to the final WER. We use NISQA <cit.> as a tool to generate automatic mean-opinion-score (MOS) values. While computer generated MOS values are not a replacement for subjective evaluations, they can be used as tool to determine naturalness as possible indicator for the ASR performance. Subjective evaluation itself can not be useful to evaluate synthetic data in a development process, as doing the evaluation often takes more effort than simply training the ASR system. We also use a larger ASR model trained on the full LibriSpeech corpus to recognize the synthesized cross-validation data to check for completeness and pronunciation correctness. We refer to this metric as synthetic WER (sWER). In the literature this is often called intelligibility metric, and sometimes character error rate is used instead of WER , e.g. in <cit.>.












































§ EXPERIMENTS




 §.§ Data


We use the LibriSpeech train-clean-100 as baseline data to train the TTS and ASR models, and the transcriptions of train-clean-360 as additional text data. This is a common setup that has been used extensively in previous literature <cit.>. We sub-sample the exact same amount of utterances, roughly 28k, from train-clean-360 to have the same amount of unseen text as text in the training data. We take 4 utterances of each of the 251 speakers in train-clean-100 out of the TTS training data to have a cross-validation set to report MOS and sWER. We perform no further pre-processing of the data, unlike other work <cit.>. I.e. no excessive silence is removed, and no speakers are discarded. We use the official LibriSpeech lexicon for word-to-phoneme conversion and use Sequitur <cit.> to generate phonemes for words that are not in the lexicon. For ASR, we augment the phoneme set with end-of-word markers as presented in <cit.>.



 §.§ Automatic speech recogition

As ASR system we use a CTC model with a Conformer-based<cit.> encoder with 12 blocks, a base dimension of 384 and a convolutional frontend with a sub-sampling factor of 4. For recognition we use the lexicon-constrained Flashlight <cit.> CTC decoder via the available Torchaudio interface. The search process includes the official LibriSpeech 4-gram LM model. Since it has been shown that synthetic data generation is successful mostly for AED-ASR <cit.>, we chose the context-free phoneme-based CTC-ASR specifically for the reason that it should be influenced less by changing or adding new text.



 §.§ Training and model settings



All models are trained using 80-dimensional log-Mel spectrogram features, for TTS with 12.5ms shift and for ASR with 10ms shift and SpecAugment <cit.> applied.
The TTS models are trained with Adam <cit.> for 400 epochs. For all models except Grad-TTS we use linear learning rate (LR) scheduling, which we found to perform better than inverse square reduction. We increase the LR from 5e-5 to 5e-4 for the first 100 epochs, and reduce linearly from 5e-4 to 5e-7. For Grad-TTS we use a constant LR of 1e-4. The ASR models are trained using AdamW  with weight-decay 1e-3 for roughly 80 epochs and a maximum learning rate of 7e-4. We generate an alignment by training the Glow-TTS system once using implicit alignment search. In each TTS training the fixed alignment is used for up-sampling and as target for the duration predictor. For vocoding, we convert the log-Mel to linear features using a separate BLSTM-based network and apply the Librosa Griffin & Lim (G&L) <cit.> function with 32 iterations and momentum of 0.99.



  §.§.§ TTS parameter settings


For the different decoder architectures the number of parameters, training time and inference time differ, as shown in Table <ref>. Making a fair comparison is difficult, as for each system we would need to do extensive tuning experiments until we reach the optimal performance. Thus, we stick mostly to the sizes presented in previous publications.

Due to the high cost of subjectively evaluating TTS systems, ablation studies on the model sizes are usually not presented in scientific publications.



Base model: For the TTS Transformer encoder we use 6 layers with an internal dimension of 256, preceded by 3 convolution layers with 256 channels and kernel size 5. The duration prediction network uses 2 convolutions with kernel size 3 and 384 channels.


Transformer decoder: For the Transformer-based decoder we simply re-use the same structure as for the encoder Transformer, having 6 layers with an internal base dimension of 256.

The system is trained using L1 loss on the output spectrograms.


Autoregressive LSTM decoder: For the AR-LSTM decoder we are using 2 layers of 1024-dimensional Zoneout-regularized LSTM layers. We use a reduction factor of 2 to predict two feature frames per decoder step. The pre-net for the AR feedback loop and the convolutional NAR post-net are derived from <cit.>. We apply L1 loss for both the decoder output and the combined prediction with the post-net.


Non-autoregressive LSTM decoder: This decoder is similar to the AR decoder, but we remove the AR feedback loop network, and replace the uni-directional Zoneout LSTM cell by a normal bi-directional LSTM with dropout. Each LSTM direction has 512 dimensions. For Transformer, AR- and NAR-LSTM, a 256-dimensional speaker embedding vector is concatenated to each upsampled encoder state.


Glow-TTS decoder: For a flow-based decoder, including speaker conditioning, we follow exactly the architecture of Glow-TTS <cit.>. The decoder consists of 12 invertible coupling blocks with an internal dimension of 256. We merge each 2 consecutive frames for a latent-space dimension of 160.


Grad-TTS decoder: For a diffusion-based model we follow the architecture and settings of Grad-TTS <cit.>. As the original architecture is not designed for multi-speaker TTS, modifications were necessary for convergence. We added two additional Transformer layers on top of the 6 encoder layers which are conditioned on the speaker embedding. That way, we have a speaker-conditioned mean prediction network. We also used the same speaker embeddings to condition the u-net, which is used for the differential equation solving.
We found τ = 0.7 to be the optimal noise scale for both Glow-TTS and Grad-TTS.



§ RESULTS




 §.§ Synthetic data, automatic MOS and recognizability

































In a first comparison we evaluated the systems via the synthetic data only ASR training, the NISQA MOS and the sWER.
Table <ref> shows the results of each of the five decoders compared to the reference values of the real data. For all metrics we used text data unseen during training.
With respect to the final ASR training evaluation, all TTS systems have a reasonable performance and there is no strong outlier.
In general, using the synthetic data for training results in a system with double the error rate when training TTS only on the available ASR data, which is a consistent result to previous literature <cit.>.
<cit.> showed that even when using vast amounts of training data to create strong foundation-model sized TTS models, the generated synthetic data seems to be not yet on par with real data.
The AR-LSTM decoder produces the best synthetic training data, which is unexpected as the Glow-TTS and Grad-TTS should produce more expressive and variable audio due to their internal sampling process.
Also, Glow-TTS is the only system that exactly follows the given literature, while the other systems have small adaptations or are simplifications of the original model architecture.

The third and fourth column in Table <ref> show the NISQA MOS and the sWER on the TTS cross validation set. It is visible that there is no direct relation of the two metrics compared to the final ASR performance.

For example, when comparing Grad-TTS to the Glow-TTS and Transformer decoder, there is a lower MOS and a much higher sWER, but still the final system performance is better. This means both metrics are not reliable to determine which system to pick for data generation.



 §.§ Decoder characteristics

In Figure <ref> we show a selected sequence synthesized by four different decoders. The Transformer decoder spectrograms are smooth, lacking a lot of detail in the high-frequency area. The AR-LSTM spectrogram in comparison has much more details. The spectrogram from Glow-TTS shows visible noise resulting from the latent space sampling. It also has less distinct boundaries between the phonemes, but rather fuzzy transitions, which might be an explanation for the worse recognizability. The Grad-TTS has less distorted spectrograms than Glow-TTS, but we found that for very short sequences the output is sometimes just a noisy grid pattern. Also the voice lines become very blurred together for lower frequencies, causing a very unnatural sound, which matches with the low automatic MOS score. Thus we assume, that the NISQA MOS and the sWER are very sensitive to certain artifacts that have less impact in on the actual ASR training. One can see that all four TTS decoders produce spectrograms with different characteristics, even with other components being identical. Thus, it is important to not only consider a single TTS architecture when trying to show a general relation between the ASR performance and metrics calculated on the level of the TTS system or the synthetic data.

































 §.§ Generalization




Table <ref> shows how good the TTS systems generalize with respect to the synthetic data generation task. Recall that condition a) means to exactly reproduce the training data, so with perfect memorization during training it would be possible to produce identically performing data. It is visible that across all models the values for a) and c) are mostly not too far from each other, so the generalization abilities are high. This gives a hint that current models are under-fitting and much larger models might yield better results, even on very constrained datasets such as LibriSpeech 100h. For synthesizing new text, so comparing b) to c), most models show no or only a small change. Only for the AR LSTM and Glow-TTS decoder there is meaningful degradation in the WER, mostly for the "other" test. For the Glow-TTS a possible reason could be that it suffers from pronunciation issues, which get even stronger when synthesizing new text. Testing for the generalization capabilities of TTS this way might not be used only for the comparison of different models like in this work, but allow for easier hyperparameter tuning which is usually not done extensively in TTS literature.



 §.§ Discussion


When training with synthetic data only, we achieved a WER performance which is less than factor 2 worse compared to training on the real data. In relation, <cit.> reported reaching ratios of around 2 on the LibriSpeech-960 task, and <cit.> reported ratios of more than 3. In this work we specifically investigated decoder architectures. In future work this gap can be reduced by extending the results of this work with extensive hyperparameter tuning as well as prosody <cit.> and more sophisticated duration modeling <cit.>. We have seen that NISQA MOS and the sWER are not reliable to determine the usefulness of the TTS system for ASR data generation. This means there are more and/or different metrics needed to give accurate estimates about how good the synthetic data can be utilized for ASR training. While in our work synthesizing new data and training the ASR systems was done in less than a day, this becomes much more of a problem when training large models on magnitudes more of data. Thus, we think it is important to find ways to estimate the capabilities of TTS models before starting large-scale data generation.



§ CONCLUSION

In this work we evaluated different TTS decoder architectures w.r.t. their ability to generate synthetic data for ASR training. We compared five different TTS decoder architectures and found that for our setup the AR-LSTM-based system with L1 loss performed best in creating training data for ASR. We were able to create synthetic data that performs less than a factor of 2 worse than real data while training the TTS only on the same data as available for the ASR training. We observed that the systems' outputs differ from each other in many aspects such as naturalness or pronunciation accuracy. Still, the metrics we used to capture them showed a surprisingly low correlation to the performance of the ASR systems trained on such outputs. This indicates that better metrics have to be found in order to do a meaningful evaluation of TTS systems before using it to create synthetic training data. We also introduced a way to measure the generalization capabilities of TTS systems for data generation, which can be used to optimize TTS towards being able to generate data that reaches parity with real data.




§ ACKNOWLEDGEMENTS




This work was partially supported by NeuroSys, which as part of the initiative “Clusters4Future” is funded by the Federal Ministry of Education and Research BMBF (03ZU1106DA), and by the project RESCALE within the program AI Lighthouse Projects for the Environment, Climate, Nature and Resources funded by the Federal Ministry for the Environment, Nature Conservation, Nuclear Safety and Consumer Protection (BMUV), funding ID: 67KI32006A. Part of this work is supported by JSPS KAKENHI Grant Numbers JP21H05054 and JP23K21681. The authors thank Lukas Rilling, Peter Vieting and Benedikt Hilmes for their additional input and feedback.




IEEEtran









 Paper Title: Enhancing Partially Spoofed Audio Localization with Boundary-aware Attention Mechanism.



Spatial Transformer Network YOLO Model for Agricultural Object Detection


 



    Yash Vivek Zambre^1, Ekdev Rajkitkul^2, Akshatha Mohan^1, Joshua Peeples^1
 ^1Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX, USA

	^2Department of Computer Science and Engineering, Texas A&M University, College Station, TX, USA

		yashzambre@tamu.edu, ekdev@tamu.edu, akshatha.mohan@tamu.edu, jpeeples@tamu.edu
    
=========================================================================================================================================================================================================================================================================================================================================================================




§ ABSTRACT 
    
   The task of partially spoofed audio localization aims to accurately determine audio authenticity at a frame level. Although some works have achieved encouraging results, utilizing boundary information within a single model remains an unexplored research topic. In this work, we propose a novel method called Boundary-aware Attention Mechanism (BAM). Specifically, it consists of two core modules: Boundary Enhancement and Boundary Frame-wise Attention. The former assembles the intra-frame and inter-frame information to extract discriminative boundary features that are subsequently used for boundary position detection and authenticity decision, while the latter leverages boundary prediction results to explicitly control the feature interaction between frames, which achieves effective discrimination between real and fake frames. Experimental results on PartialSpoof database demonstrate our proposed method achieves the best performance. The code is available at <https://github.com/media-sec-lab/BAM>.



[Corresponding author.]





§ INTRODUCTION


With the rapid advancement of Artificial Intelligence Generated Content (AIGC), technologies like text-to-speech (TTS) <cit.> and voice conversion (VC) <cit.> can generate realistic human voices. Partially spoofed audio, where synthesized speech segments are inserted or spliced into genuine utterances, pose significant threats. An attacker can alter the meaning of sentences by manipulating small and specific units (e.g., words, characters, or even phonemes), deceiving both machines and humans.

In recent years, numerous methods and datasets have been developed for Partially Spoofed Audio Detection (PSAD). Yi et al. <cit.> constructed the first partially spoofed audio dataset named HAD, which replaces some nature segments with synthesized speech segments having different semantic content. The Audio Deep Synthesis Detection (ADD) challenge 2022 <cit.> involves a detection track containing partially spoofed audio. The organizers provided only real and entirely fake data for training, and the task was target to detect fake at utterance level. In this challenge, notable detection performance was achieved by fine-tuning pretrained self-supervised learning (SSL) models with different back-end classifiers <cit.>. Boundary detection task was only introduced as a proxy task to achieve utterance-level detection <cit.>.

Beyond merely detecting the presence of spoofing in audio, the Partially Spoofed Audio Localization (PSAL) task has become an emerging topic for better analyzing deepfake audio. Zhang et al. <cit.> developed a dataset named PartialSpoof, which includes segment-level labels. They then investigated an approach to detect spoofing at both segment and utterance levels within a multi-task learning framework <cit.>. Subsequently, they extended the PartialSpoof dataset and used Wav2vec2 (W2V2) <cit.> as a front-end to detect segment-level and utterance-level fakes simultaneously <cit.>. Moreover, the ADD 2023 <cit.> introduced a PSAL track, which has resulted in numerous promising works <cit.>. Among them, an effective approach was proposed to achieve the best localization score by integrating the decisions from three countermeasure (CM) systems including boundary detection, frame-level fake detection, and utterance-level fake detection <cit.>. Recently, an approach incorporating a contrastive learning  module and temporal convolution was proposed to effectively capture better features for localization <cit.>.



Some aforementioned methods <cit.> utilize boundary information solely for PSAD at utterance-level detection. However, leveraging boundary features to aid in pinpointing the exact locations of spoofing within a single CM system is still an unexplored topic. As we know, all frames within a segment delineated by boundaries are assigned the same label. Once the correct boundary position is identified, the authenticity of a frame can be determined by referencing other frames within the same segment. Thus, boundary information serves as vital auxiliary knowledge that can steer authenticity decisions at the frame level. To exploit this idea, in this paper, we have designed a new attention mechanism that leverages boundary information, as illustrated in Figure <ref>. Message passing between those frames that belong to different authenticity classes is explicitly weakened by the attention mechanism. Our proposed method is named Boundary-aware Attention Mechanism (BAM). Specifically, pre-trained W2V2 <cit.> or WavLM <cit.> is used as front-end feature extractor. Subsequently, a Boundary Enhancement (BE) module extracts inter-frame and intra-frame features to effectively detect the boundary frame position. Finally, a Boundary Frame-wise Attention (BFA) module leverages the boundary prediction results to make reliable  authenticity decisions at frame level. In this way, discriminative features are learned to better distinguish between real and fake frames, leading to improved localization performance. In summary, the contributions of this paper are as follows:


  * We propose a novel approach called BAM that takes boundary information as auxiliary attentional hint to guide localization. To the best of our knowledge, this work is the first attempt to leverage boundary information for PSAL within a single CM.

  * We develop a boundary enhancement module containing two branches for better exploiting intra-frame and inter-frame information to obtain discriminative boundary features.

  * We conducted experiments to demonstrate the effectiveness of the proposed method. Compared with existing methods, our method achieves the best localization performance on PartialSpoof dataset.




§ PROPOSED METHOD




The flowchart of our BAM is depicted in Figure <ref>. Firstly, we use a pre-trained SSL model to extract the features of speech and apply an attentive pooling layer <cit.> to make each frame represents a specific temporal resolution (e.g., 160). The output of the pooling layer is fed into the Boundary Enhance (BE) module to enhance boundary feature representation and identify the boundary frames with a simple fully connected layer. Then, the Boundary-aware Frame-wise Attention (BFA) module takes the boundary prediction results and the outputs of the pooling layer as inputs to capture the correlation information among frames. Finally, the outputs of the BFA module and the BE module, are concatenated and fed into a fully-connected layer to make frame-level authenticity decisions.



 §.§ Pretrained self-supervised front-end

We start by processing raw speech using a pretrained SSL speech model, W2V2 <cit.> or WavLM <cit.>, to extract effective front-end features. Compared to traditional handcrafted features (e.g., LFCC <cit.>, MFCC <cit.>), the SSL-based front-end features exploit extensive volumes of unlabeled data for pre-training, significantly improving data representation capabilities and facilitating the identification of intricate patterns within audio data. In our implementation, we utilize WavLM-large <cit.> or XLS-R-300M <cit.> and fine-tune their weights in conjunction with other modules, and use their last hidden states of the transformer as front-end features.



 §.§ Boundary enhancement module


In the context of PSAL, frames at the boundaries that contain both spoofed and genuine samples are labeled as spoofed. During training with binary authenticity labels, boundary frames (especially those with a smaller proportion of spoofed data) are close to fully genuine frames in the feature space, even though they have opposite labels. This situation can lead to training instability and performance degradation. To mitigate this issue, we have developed a boundary enhancement module specifically to model and distinguish boundary frames from non-boundary frames.

As shown in Figure <ref>, the boundary enhancement module transforms the front-end feature F_g ∈ℝ^T × D into the boundary feature F_b ∈ℝ^T × 2D, where F_g is the output feature of a pooling layer, T is the number of frames, and D is the feature dimension. The boundary feature serves dual purposes.. Firstly, it is fed into a fully-connected layer with a sigmoid function to obtain the boundary prediction probability b̂∈ℝ^T × 1 as following:

    b̂=sigmoid(ϕ(F_b)),

where the linear mapping function ϕ with a learnable weight matrix W and a bias term b is given by: 

    ϕ(x)=Wx+b,

Then we conduct binarization with a fixed threshold to make binary boundary prediction B̂. Secondly, the boundary feature is processed by a fully-connected layer with an activation function and concatenated for the final frame-level authenticity decision. 

The detail of BE module is illustrated in the middle of Figure <ref>. Specifically, two different branches are designed to respectively extract the intra-frame and inter-frame features. For the inter-frame branch, a Frame-wise Attention Block (FAB) is employed to capture inter-frame correlations using an attention mechanism. The right part of Figure <ref> illustrates the details of the FAB. We start by computing the attention score s ∈ℝ^T × T × D through the element-wise multiplication of each frame feature with others. Then, a learnable attention weight W_a ∈ℝ^D× H is defined to weigh the attention score. This process can be written as:

    A_t=tanh(ϕ(s))W_a,

where A_t∈ℝ^T× T× H represents the attention map. Subsequently, each frame within F_g aggregates information from other frames, utilizing the attention map as following:

    F_a=softmax(A_t)F_g,

where F_a ∈ℛ^H× T× D represents updated frame features. The residual structure is employed to stabilize training. The output of FAB is obtained from F_a and F_g according to:

    F_inter=𝒮(ℬ𝒩((ϕ(F_a) ⊕ (ϕ(F_g)))),

where ⊕ represents element-wise addition and ℬ𝒩 represents 1-D batch normalization, and 𝒮 is SELU <cit.> activation function. For the intra-frame branch, each frame is individually fed into a 1D-ResNet <cit.>, followed by a fully-connected layer to learn intra-frame features. Then, the intra-frame feature F_intra and inter-frame feature F_inter are concatenated to obtain the boundary feature F_b. 



 §.§ Boundary frame-wise attention module



As shown in Figure <ref>, the BFAM module consists of two stacked Boundary Frame-wise Attention Blocks (BFAB). Specifically, the structure of BFAB is similar to FAB but is equipped with a boundary masking component, which takes binary boundary prediction B̂ and F_g to weaken the message passing between those frames that belong to different genres. A boundary adjacency matrix A_b∈{0,1}^T × T is constructed based on the boundary prediction B̂. Its element is defined as follows:

    A_b_i,j=
        
        1   if  i = j, 
    ∏_n=i^j(1-B̂[n])   if  i<j, 
    ∏_n=j^i(1-B̂[n])   if  i>j,

where B̂[n] represents the n^th element of sequence B̂, and i,j∈[0,1,…,T-1] represent the indices of frames. In other words, equation <ref> determines whether there is a boundary between the i^th and the j^th frame; if there is, the value in the i^th row and the j^th column of A_b is set to 0, otherwise, it is set to 1. Once the adjacency matrix is obtained, the attention map (refer to equation <ref>) can be updated to a boundary attention map Â_̂t̂, as illustrated in Figure <ref>, and given as:

    Â_̂t̂=A_t⊗ A_b,

where ⊗ denotes element-wise multiplication.



 §.§ Loss function

We employ two loss functions to constrain the model for supervised training: boundary loss ℒ_b and frame-level authenticity loss ℒ_s as shown in Figure <ref>. For boundary loss, binary cross-entropy loss is used and the ground-truth boundary labels B can be derived from the segment-level authenticity labels Y. It is worth noting that we only set the boundary frames as label 1 and all other frames as label 0, which differs from previous methods <cit.> where they also set the label of frames near the boundary frames to 1. This labeling strategy <cit.> may introduce additional noise in our case and lead to diminished localization performance. For authenticity loss, we follow prior work and utilize the standard cross-entropy loss. In summary, the overall loss can expressed as:

    L = ℒ_s(ŷ,Y)+λℒ_b(b̂,B),

where ŷ∈ℝ^T× 1 represents the authenticity prediction result of model, and λ is set to 0.5 in practice.



§ EXPERIMENTS AND RESULTS




 §.§ Dataset and implementation details

To validate the effectiveness of our method, we conducted experiments on the PartialSpoof <cit.>. The dataset is constructed by segmenting the fully spoofed and genuine utterances from the ASVspoof2019 <cit.> LA database using Voice Activity Detection (VAD) and then concatenating those segments with basic digital signal processing. Four evaluation metrics are used to compare model performance: Equal error rate (EER), precision, recall and F_1 score.   

In the training phase, for experiments conducted at a 160 temporal resolution, we fix the length of training samples at 4s. Samples exceeding this length are randomly trimmed, while those shorter than are padded with zero. Then raw data is padded to correspond with the 20 shift of W2V2 and WavLM. Therefore, the attentive pooling stride is set to 8, the number of frames T is 25, the feature dimension D is 1024, block number N is 2, and the learnable weight head number H is 1. For experiments conducted at a 20 temporal resolution, the number of frames T is 200, and we use a linear transformation to reduce the feature dimension D to 256. Other hyperparameters remain consistent with those used in 160 experiments. In the test phase, variable-length test samples are directly fed into the model to obtain the prediction result. we use Adam optimizer with a learning rate of 10^-5 and halved every 10 epochs, and all of model are trained for 50 epochs.



 §.§ Comparison with existing methods

To ensure a fair comparison with existing methods <cit.>, we conducted experiments with 160 ms temporal resolution. The results are shown in Table <ref>. Based on our observation, WavLM achieves better results compared with W2V2-XLSR. Moreover, Our method gets EER as 3.58% and F1 score as 96.09%, reaching the best single CM performance on PartialSpoof dataset.






 §.§ Ablation study


We conducted ablation experiments to assess the effectiveness of each modules within our framework under two settings: 1) localization task and 2) boundary detection task. Table <ref> presents the results for localization task. The baseline model, denoted as 1 , is equipped with a WavLM front-end, succeeded by a max pooling layer and a fully-connected layer. The models 2 and 3 represent simplified versions of the BAM. Model 2 incorporates only the right branch as illustrated in Figure <ref>, while model 3 includes both the left and right branches but omits the middle pathway from the BE module to the BFA module. Model 4 is actually the proposed BAM. It can be seen that each module contributes to the enhancement of localization performance. Specifically, compared to the baseline model, our method reduced the EER by 2.21% and increased the F1 score by 1.73%.

Table <ref> presents the results for boundary detection task. It is clear that the inter-frame feature is more effective than intra-frame. The BE module combined with both inter-frame and intra-frame achieves the best boundary detection performance with EER as 3.33% and F1 score as 92.25%.




 §.§ Finer-grained resolution experiment




Extra experiment was conducted at a 20 temporal resolution. Specifically, we trained the BAM without pooling directly on the front-end feature with a 20 shift for 50 epochs as the base model. Subsequently, we removed the last fully-connected layer from pretrained base model and replaced it with an attentive pooling layer succeeded by a new fully-connected layer. We finetuned this model for 10 epochs to predict the authenticity at specific temporal resolutions. The results are shown in Table <ref>. The base model achieves remarkable localization performance at 20 resolution with EER as 5.20% and F1 score as 95.82%. As the temporal resolution decreases, the F1 score remains almost constant while the EER continually increase. We explain the reason behind this trend is that with the decreases in resolution, the total number of frames and the proportion of genuine frames both rise, and EER is particularly sensitive to changes in class proportions while the F1 score is relatively robust. Moreover, there is a marginal decrease in localization performance compared to BAM executed on frames of 160 (3.66% v.s 3.58%). We attribute this decline in performance to the increased challenge of boundary detection at finer resolutions. The base model gets EER as 5.12% and F1 score as 82.97% at 20 resolution for boundary detection task, which is worse than the boundary detection performance at 160 (refer to Table <ref>).
    


§ CONCLUSION

In this paper, we propose a novel partially spoofed audio localization method. Our method simultaneously conduct boundary detection and frame-level authenticity determination tasks within a single CM model. The boundary information is utilized to enhance the accuracy of localization. Experimental result show that the proposed method achieves the best performance on PartialSpoof dataset. The finer-grained resolution experiment demonstrates that accurately detecting boundary position at a finer resolution is a more challenging task and will be a direction of our future work.



§ ACKNOWLEDGEMENTS

This work was supported in part by the National Natural Science Foundation of China (Grant U23B2022, U22B2047, 62322120), and in part by the Guangdong Provincial Key Laboratory (Grant 2023B1212060076).

IEEEtran




 Paper Title: MMTrail: A Multimodal Trailer Video Dataset with Language and Music Descriptions.


Instantons and the Large =4 Algebra
    Edward Witten
    
===================================



[1]These authors contributed equally to this work.

[2] Corresponding authors.
[3]Github repository:https://github.com/litwellchi/MMTrailhttps://github.com/litwellchi/MMTrail
[4]Project Page:https://mattie-e.github.io/MMTrail/https://mattie-e.github.io/MMTrail/



§ ABSTRACT 


Massive multi-modality datasets play a significant role in facilitating the success of large video-language models. 
However, current video-language datasets primarily provide text descriptions for visual frames, considering audio to be weakly related information. They usually overlook exploring the potential of inherent audio-visual correlation, leading to monotonous annotation within each modality instead of comprehensive and precise descriptions. Such ignorance results in the difficulty of multiple cross-modality studies.
To fulfill this gap, we present MMTrail, a large-scale multi-modality video-language dataset incorporating more than 20M trailer clips with visual captions, and 2M high-quality clips with multimodal captions. 
Trailers preview full-length video works and integrate context, visual frames, and background music. In particular, the trailer has two main advantages: 
(1) the topics are diverse, and the content characters are of various types, , film,  news, and gaming.
(2) the corresponding background music is custom-designed, making it more coherent with the visual context. 
Upon these insights, we propose a systemic captioning framework, achieving various modality annotations with more than 27.1k hours of trailer videos. Here, to ensure the caption retains music perspective while preserving the authority of visual context, we leverage the advanced LLM to merge all annotations adaptively.
In this fashion, our MMtrail dataset potentially paves the path for fine-grained large multimodal-language model training. In experiments, we provide evaluation metrics and benchmark results on our dataset, demonstrating the high quality of our annotation and its effectiveness for model training. 


   

    




§ INTRODUCTION


AI-driven movies and shot video production have a wide range of applications in people's daily lives.
Clearly, creating vivid videos requires more than just visual frame generation or individual modality-based ones. 
Thanks to various large-scale video-language datasets, numerous generative multimodal large language models have been developed to achieve this goal <cit.>.
However, the existing video-language datasets <cit.> typically focus on visual-based text descriptions, and they overlook the significance of the inherent visual-audio dependencies.
It presents a complex challenge that demands cohesive integration of multiple modalities, yet remains largely unexplored.






Collecting high-quality multi-modality source data that preserves consistency between different modalities is challenging. 
Unlike previous datasets that only provide visual frame-based caption <cit.>, multimodal datasets contain complex data formats (, music), resulting in more labor-intensive and time-consuming costs in data processing and annotation.
Moreover, achieving a high correlation between audio and visual content presents challenges.



r0.5
    


    < g r a p h i c s >

Distribution of video categories of MMTrail dataset.


    

Targeting to fill the dataset gap by creating a comprehensive and accurate multi-modality visual-audio dataset, we first notice trailers. As a precursor to a full-length work, the video trailer has emerged as a vital tool for artists to showcase and disseminate their creations. These short videos typically combine the most compelling visual shots with carefully selected music, have high cross-modality consistency, and hold significant potential in broader multimodal research. The topics are diverse, and the content characters are of various types, e.g., film, comedy, and gaming, as shown in  <ref>. Significantly, the trailer format represents a unique, high-quality, video-centric multimodal data source that benefits further multi-modality research exploration and analysis.





In this work, we propose MMTrail, which aims to unlock the potential of multimodal content understanding and generation for innovative applications in video content generation. We first recognize the immense value of trailers as a video-centric dataset, especially considering the music alongside the videos. MMTrail contains 20M+ video clips from 290k trailer videos encompassing various source categories as shown in <ref>. To ensure the quality of our dataset, we have carefully designed a robust data filtering and cleaning methodology. We also provide extensive statistics works to demonstrate the diversity and complexity of our dataset.


To address the multimodal to language annotation challenge, we have designed a multimodal captioning pipeline incorporating diverse state-of-the-art (SOTA) captioning models <cit.>. Furthermore, we propose a language model fusion strategy to generate fine-grained multimodal captions. We have performed small-scale annotations on the entire dataset, created a multimodal annotation subset of 3 million samples MMTrail-2M, and provided a testing set MMTrail-Test with manually-adjusted multimodal caption.


We present evaluation metrics and benchmark results on our dataset, demonstrating the high quality of our annotations and their effectiveness for model training. Through extensive experiments and benchmarking, we showcase the difficulty and diversity of our dataset using various evaluation metrics. We also conduct human evaluations to validate the quality of our multimodal captioning pipeline. Furthermore, we fine-tune understanding models <cit.> and generative models <cit.> on a subset of our dataset, providing evidence of its high quality and efficacy. Additionally, we evaluate video understanding models on the MMTrail-Test, highlighting the challenges posed by our dataset, and evaluate video-music-based models to demonstrate the effectiveness of cross-modality tasks.







§ RELATED WORK







 §.§ Video generation and understanding

Video understanding and text-to-video generation are inherently connected tasks. In recent years, there has been remarkable progress in understanding models <cit.>, which have greatly contributed to the advancement of text-based video generation techniques. The availability of large-scale datasets and diffusion models has revolutionized video generation, moving from pixel-level approaches like <cit.> to latent-level video diffusion models <cit.>. Concurrently, understanding models have also witnessed significant improvements. A series of MLLM-based understanding models <cit.> has reach a satisfied understanding abilities. Besides, some surveys give detailed summaries of this area from different application areas or aspects <cit.>. The iterative interaction between video generation and understanding has led to the development of excellent large-scale datasets and models encompassing diverse approaches. Panda <cit.> introduced an auto-caption model distilled from video understanding models like VideoLlaMA <cit.>, MiniGPT4 <cit.>. Furthermore, leading to a large amount of multimodal generation models <cit.>.







 §.§ Video-Language datasets


Captioned video datasets are essential for text-to-video generation and understanding tasks. MSR-VTT <cit.>, UCF-101 <cit.> are commonly used as evaluation sets. Anna et al. <cit.> presented 118,081 movie clips with descriptions. ActivityNet Caption <cit.> by Ranjay et al. is a benchmark involving event detection, natural language description, and event localization. WebVid <cit.>, VideoFactory <cit.>, and other works <cit.>, contain multilingual video descriptions, video clips, metadata such as titles, descriptions, tags, and channel names, and are used for tasks like video understanding, text-to-video retrieval, and audio-video captioning with weak annotations. Several datasets and approaches have utilized audio to enhance video captioning <cit.>. These datasets consist of movies, web videos, YouTube videos, and high-resolution videos from popular YouTube categories, providing transcribed audio descriptions, narrations <cit.>, ASR transcriptions <cit.>, and multiple captions generated through a auto caption model <cit.>.

However, existing works only use audio to enhance the video caption, using metadata or ASR to provide extra information for the video caption. A large-scale video-centric dataset that cooperates with high-quality music and multimodal caption is still lacking. 




§ MMTRAIL-20M DATASET


To boost the performance of multimodal generation tasks, we construct a high-quality multimodal dataset based on trailers, which offers a wealth of multimodal information and diverse categories that distinguish them from existing large-scale video-language datasets.









 §.§ Data Collection Pipeline


Trailers, as unique, well-made video types, have particular data distribution and exceptional quality. The trailer videos usually contain multiple themes, including movies, TV shows, games, etc. They are well organized with the most attractive clips of the corresponding videos. At the same time, most of them are accompanied by corresponding background voices and music. Such rich and high-density information is a complete challenge for the current Artificial Intelligence Generation Content(AIGC) task and brings considerable difficulties to data cleaning. To solve this challenging problem, we designed a comprehensive data collection and cleaning process as shown in Fig. <ref> to deal with such complex videos, as described in this section.





Collection Strategies
To enrich the diversity of our MMTrail, we first employ the keyword “trailer” to retrieve various in-the-wild trailer videos. Such general-purpose trailer videos encapsulate a wide range of artistic works and genres. Then, we tailor the keywords more specifically to explicitly collect trailer videos with divergent sources. Those keywords include “Movie Trailers”, “Video Game Trailers”, “TV Show Trailers”, “Documentary Trailers”, etc. Through these two methods, we collect 285,518 comprehensive trailer videos with a total duration of 94,911,802.8 seconds. 


Trimming
To facilitate the extraction of various video information in subsequent analyses, we cut the original videos into clips based on the scenes. As the most mature and practical tool currently available, PySceneDetect[https://github.com/Breakthrough/PySceneDetect] offers robust functionalities for this purpose. Thus, we use ContentDetector of PySceneDetect to compare the difference in content between adjacent frames and then cut the videos according to the predetermined threshold of 30. We finally generated 21,588,792 clips with an average duration of 4.6 seconds. 

Motion Filtering 
Evaluating motion quality is a crucial step in selecting high-quality video content. Motion vectors and optical flow are both mainstream motion quality evaluation methods. In our case, trailer videos often have rapid cuts and transitions between scenes, posing challenges for optical flow-based analysis. Other than optical flow, motion vectors are more robust to these quick changes, as they rely on larger block units' displacement rather than individual pixels' continuous flow. On the other hand, motion vectors are more lightweight and can be calculated more efficiently. Thus, we leverage motion vectors to filter out clips with problems like static frames, title sequences, and slideshow-like playback.


Diversity
We evaluate the diversity and richness of our dataset from three aspects: theme, objects, and backgrounds. While collecting, we first assess the categories from the Yotoube metadata provided by the video provider, as shown in  <ref>. Furthermore, we generate an object-level caption list and background by Llava-NexT <cit.> for a more accurate category-based generation. The word cloud of objects and backgrounds is shown in  <ref>.








OCR

Trailer videos often have text-heavy sections with high-quality text animations, like opening and ending credits. To identify these text-rich segments, we utilize OCR to detect the text content in the video frames and calculate the bounding box area of the text. This measurement reflects the amount of text in the clips. 


Quality Statistics
In addition to text detection, we considered image quality <cit.> and aesthetic scores <cit.> to enhance our analysis of trailer videos. These measures allowed us to evaluate frames' visual fidelity, clarity, and aesthetic appeal, providing more comprehensive insights for trailer analysis and editing. 


Audio Collection
We extract the audio from video source segments with a sampling rate of 44.1 kHz. We use the PANNs <cit.> algorithm to perform music event detection, and over 70% of the audio segments contain music.










 §.§ Video Captioning Pipeline


The MMTrail contains many complex themes, like subtitles and character animations, as shown in  <ref>, which brings extraordinary complex work for video captioning. At the same time, smooth transition shots also make it impossible for traditional single-frame annotation methods to convey semantics coherently. Therefore, this section introduces a multi-temporal and multimodal caption pipeline containing a detailed video description from frame, motion, and music levels. 

Frame Caption


The auto-captioning pipeline has proven efficient in cutting-edge video generation foundation models. SVD <cit.> and Pandas <cit.> have given promising results and demonstrated the importance of high-quality frame captions for the generation model. We initially performed image-level captioning on the individual frames of the data. We employed coca <cit.> for each video clip to generate separate captions for three frames(first, middle, and last), resulting in relevant captions.

Clip Caption
Having obtained concise captions for three frames that capture the essential information, we aimed to obtain fine-grained captions and variations between frames in the video. We concatenated multiple frames into a comic strip format and employed the LLaVA <cit.> image model to guide the description of the dynamic differences between frames. Additionally, leveraging a powerful multimodal language model, we incorporated OCR and more detailed summary descriptions to expand the information within the frame captions.


Categories and Background
Noticing the LLM-based caption has hallucinations when describing the frame, we further generate word-level labels to enhance the annotation of the main objects and background. Initially, we utilized LLaVA's QA capabilities to have the model answer questions about the background. Subsequently, through QA, we prompted the model to provide relevant category information. We conducted the word cloud in  <ref>. and certified caption quality by subjective experience in  <ref>.



Music Caption
Moreover, given that trailer music usually has a well-designed audio effect and background music, we applied the music caption on our dataset rather than a standard audio caption. In our work, we used MusicCaps <cit.>, an LLM-based music captioning model. The caption format is well designed with its description pipeline, which first describes its sound quality, a generated speech style, and a detailed description of its instrument and music style. More examples are shown in the Appendix. We further evaluate the generation tasks and the text-to-music generation based on the music caption, which shows the efficiency of our captioning and dataset.

OCR The trailer contains many text animations. Captioning the context inside the movie can also be a challenging task. In this task, we utilize the OCR ability by LlaVA <cit.> and caption the context for 5 frames of each video. We merge reliable text animation captions based on the previous method.

General caption Combining all the captions mentioned above, we use the language model llama2-13B <cit.> to merge all the captions and generate complete and high-quality multimodal captions. We evaluate the caption accuracy and quality by human preference in  <ref>.





 §.§ SubSet Separation




We applied the frame caption <ref> for the full MMTrailer-20M video clips with 20M+ clips. We included the scale comparison with other large-scale datasets in  <ref>, showing that our dataset is a large-scale video-language dataset. MMTrail has a resolution no smaller than 720p, and MMTrail-20M clips are 4.6s long on average.


High-quality Subset
, named MMTrail-2M, contains a detailed multimodal caption. Compared to the original distribution, we samples MMTrial-2M to create a high-quality subset using the following criteria:
1. We filter out clips with motion scores below 0.45 or above 50.
2. We only retain the clips within the top 85% of image quality scores.
3. We only keep the clips within the top 85% of aesthetic scores.
Furthermore, all clips in MMTrail-2M are longer than 4s and provided with all the captions, as shown in  <ref>, including categories, background, frame captions, music captions, merged captions, etc.
We also compare MMTrail-2M with Video-Audio datasets as shown in  <ref>, MMTrail has a larger scale than existing datasets (Audioset <cit.> and Vggsound <cit.>). 
 
High-quality test set
is extracted from the MMTrail-2M, we extract a fine-branded testing set that contains 1k video clips and multiple multimodal captions. Then, we manually adjust the merged caption to a manual caption to build a testing subset with trust-wise multimodal prompts. We test several tasks and models on the test set in  <ref> to show the complexity and difficulties of MMTrail. The test set has 98.2 words of caption on average and includes 3.2hr video clips.











§ EXPERIMENTS



This section presents comprehensive experiments on multiple tasks to demonstrate our dataset's effectiveness, diversity, complexity, and difficulty. 



 §.§ Multimodal Captioning




We present the results of our human evaluation of video caption quality in  <ref>. Ten videos were randomly selected from the MMTrail-Test dataset. They were rated on a scale of 0 to 10 based on general impressions, including aspects such as correctness, level of detail, richness, and fluency. The results of more than 100 sets of samples indicate that the manually adjusted prompts rating of 8.12 outperforms the auto-caption pipeline, while our merged captions achieve the second-best performance of 6.62. Despite being short and straightforward, object/background labels achieve a 6.02 evaluation score, demonstrating more correctness than other captions. Frame caption, music caption, and llaVA caption obtain 4.3, 4.6, and 5.4, respectively, and these findings demonstrate the effectiveness of our captions and highlight the quality of our labeled captions by human annotators.



 §.§ Video Generation









The model was fine-tuned on the VideoCrafter-2.0 <cit.> dataset using 8 Tesla-H800 GPUs with a batch size of 3 for 10,000 steps at a learning rate of 6e-6. The training data was randomly sampled from the MMTrail-2M dataset, using the video captions as input.
The evaluation results, shown in  <ref>, include 9 matrices on the VBench <cit.>, indicating that fine-tuning the model on the MMTrail-2M dataset led to improvements of 0.6 in motion smoothness and 1.77 in subject consistency, with a slight overall performance boost(0.12 higher) compared to the official VideoCrafter-2.0 checkpoint.
Visual examples of the generated content are provided in  <ref>, and additional demonstrations and experiment details will be included in the supplementary material.
This thorough evaluation and comparison of the tuned model's performance on critical metrics provides valuable insights into the effectiveness of the fine-tuning process and the potential benefits of leveraging the MMTrail-2M dataset for video generation tasks.




 §.§ Video Understanding



Experiment Setting
To evaluate the capability of our dataset in multimodal video understanding, we choose Video-LLaMA <cit.> as the baseline for the video captioning task. We use same model and training config as Video-LlaMA, which useVicuna-v0-7B as llama model <cit.>, ViT <cit.> and Q-Former  <cit.> as the video encoder and the linear projection layer from MiniGPT-4 <cit.>. We train 4 epochs by MMTrail-2M, each containing 2500 iters with batch size 32. We compare it with two official model weights: the pre-train Video-LlaMA weight on WebVid (2.5M video-caption pairs) and the fine-tuned Video-LlaMA.

Evaluation Metric
We evaluate video understanding models on the MMTrail-Test. As for the evaluation metric, we choose the commonly used metrics in text generation tasks-BLEU-4 <cit.>, ROGUE-L <cit.>, METEOR <cit.>, and CIDEr <cit.> to evaluate our result.
All the metrics are computed using the pycocoevalcap <cit.> package. We also use BERTScore <cit.> to evaluate the contextual similarity for each token in the ground truth and the predicted captions. The results are reported in  <ref>. The official weights show relatively low performance, highlighting the challenge of MMTrail, and the data distribution differs from their training data.

In addition, we also evaluated three checkpoints from Video-LlaMA <cit.> by human evaluation in  <ref> and found that the Video-LlaMA-MMTrail evaluation result slightly lags behind Video-LlaMA-Finetune but performs significantly better than Video-LlaMA-Pretrain. We provide further details in  <ref> for a more comprehensive understanding of our model.



 §.§ Music Generation


We used text-to-music generation to evaluate the effectiveness of the video-music pair data and the labeled video caption and music caption. 
We use MusicGen <cit.> to generate music based on our video caption (VideoCap2Music) and music caption (MusicCap2Music). We use Kullback-Leibler Divergence (KL), Inception score (ISc), Frechet distance (FD), and Frechet Audio Distance (FAD) <cit.> to evaluate the generated music. Besides, we use the ImageBind-AV score (IB) to evaluate the audio-visual alignment between the video and the generated music. For the model with text input in Tab. <ref>, compared with video caption, the evaluation results on music caption are 0.13 better in KL, 0.65 in ISc, 3.64 in FD, and 1.21 in FAD, showing the domain gap of multimodal descriptions.
This comparison shows that there is still a significant research gap between caption-music-video.







§ CONCLUSION


We introduce MMTrail, a comprehensive and accurate multi-modality visual-audio dataset to address the dataset gap. By utilizing the inherent value of trailers, which integrate visual, audio, and contextual elements, MMTrail offers detailed and precise multi-modality annotations. Our systematic captioning framework adaptively merges visual and musical perspectives, ensuring that the annotations capture the richness of multimodal content. Experimental results demonstrate the high quality of the MMTrail dataset, its effectiveness for fine-grained multimodal-language model training, and a variety of downstream applications. We believe this innovative dataset will unlock new possibilities in video content generation and significantly advance research in visual-audio understanding. The comprehensive and diverse nature of MMTrail makes it a valuable asset for the research community, paving the way for novel applications that leverage the power of multimodal learning.




§ CAPTIONING PIPELINE




 §.§ Music Caption


As trailer videos often contain a cacophony of audio tracks, which typically include background music and vocals, captioning audio poses a significant challenge. Therefore, to achieve optimal music captioning results, we initially utilize Demucs <cit.> for vocal separation on each audio clip. Subsequently, LP-MusicCap <cit.> is leveraged to caption the resultant audio, devoid of vocals, from the separation process.



 §.§ Speech Recognition



Besides, the speech is also an important track of the trailer audio. Therefore, we further turn the speech into the text.
Similar to our approach in music captioning, we use Demucs initially to perform vocal separation on each audio clip. Following this, Whisper is utilized to caption the separated vocal audio.



 §.§ Merge Caption




§ CAPTIONING PIPELINE


We use LlaMA-13B to merge the multiple captions, by constricting a pre-designed caption prompt as follows:
    
    There are some descriptions of a video, 
    including video caption, music caption, background caption and the main objects in the video. 
    Please combine all the descriptions into an overall description of the video in only one paragraph.
    Finally, please rewrite and polish it into an overall video description in one paragraph and give me only the content of the video. 
        Background: background
        Main objects: objects
        Video caption 1: image_cap
        Video caption 2: frame_cap
        Music caption: music_cap
    

As illustrated in Figure <ref>, we use the LLaMA-13B to merge frame captions, lava captions, music captions, objects, and background into a final merge caption. By merging multiple captions, the merge caption offers a comprehensive and detailed description of the audiovisual content of videos. Its components mutually complement one another, ensuring that every aspect of the narrative receives attention and providing unique insights into various facets of the video content.



 §.§ Frame Caption

We utilize LLava-v1.6-vicuna-7B as the image understanding model to generate captions for video clips. Initially, we filter out clips with durations of less than 1.0 seconds. Subsequently, we sample three frames at the fractions of 0.2, 0.5, and 0.8 of each clip's total duration. The prompt is as follows:
 
 Please describe the image in detail.
 






 §.§ LLaVA Caption

We use the image understanding model LLaVA-13b <cit.> to caption each of our video clips. Specifically, for each clip, we sample frames at positions 0.1, 0.3, 0.5, 0.7, and 0.9, and then horizontally concatenate them into a single image, which serves as input for LLaVA-13b to generate captions. We construct the caption prompt as follows:
    
    These are some keyframes of a video. 
    Please use one sentence to summarize the content of the video in detail. 
    Summarize the content of the entire video but not describe keyframes frame by frame.
    
Moreover, we observe that the results of LLaVA captioning often contained some redundant information, as illustrated by the green sections in Figure <ref>. Therefore, we use LLaMA-13b to refine the LLaVA captions, eliminating much of the extraneous content and rendering the final captions more in line with human expression. We construct the prompt as follows:
   
    [caption] This is a description of a video. 
    Please polish it to an overall video description in one sentence and give me only the content of the video. 
    Do not use the words 'frame' and 'video'. 
    Describe the content of the video directly, which means do not start with 'The video...' or something like that.
    Do not add extra information that is not included in the original description.
    Here is an example: A dancer in a vibrant orange skirt and gray jacket moves gracefully across the stage, her movements fluid and expressive. 
    

By combining frames and secondary polishing, we finally obtain high-quality captions that contain the main content information of the clips.






§ ADDITIONAL EXPERIMENTS




 §.§ Video Understanding Model

We use VideoLLaMA as our base model, which contains a llama model, a q-former, and a llama projection model. For LLM, we use llama vicuna-v0-7B.  
We also use model weight pretrained on minigpt4 to initialize our llama projection. In the training stage, we train from scratch and froze the Visual Transformer and LLM models. Only the visual q-former and the llama projection are trained in the pretraining stage. Following the video llama setting, we use a batch size of 32, and each video uniformly samples 8 frames with a 224 resolution. We train 5 epochs on our 2M training data. Each epoch contains 2500 iters with a linear warmup cosine lr. The weight decays are 0.05, and the warmup lr is 1e-6 with 2500 warmup steps, a 1e-4 init lr, and an 8e-5 minimum lr. The results can be seen in Table <ref>








 §.§ Video to Music Generation


To further assess the effectiveness of the video-music pair data, we conduct an extended video-to-music generation task by training VidMuse <cit.> on an individual music subset of MMTrail-2M. The results, shown in Table <ref>, demonstrate the high audio quality and strong audio-visual alignment between the video and the generated music achieved by our dataset.







§ DATASET DETAILS


Here is one metadata example of our dataset. We introduce the basic information of video and clips in the "basic" tag, including their duration, quality evaluation score, etc.  The useful caption and description are saved in the "scene" tag.


    "basic": 
        "video_id": "-3r7ptfObEs",
        "video_path": "group_33/-3r7ptfObEs.mp4",
        "video_duration": 71.73333333333333,
        "video_resolution": [
            720,
            1280
        ],
        "video_fps": 30.0,
        "clip_id": "-3r7ptfObEs_0000000",
        "clip_path": "video_dataset_33/-3r7ptfObEs_0000000.mp4",
        "clip_duration": 7.033333333333333,
        "clip_start_end_idx": [
            0,
            211
        ],
        "image_quality": 36.83453941345215,
        "of_score": 12.92151,
        "aesthetic_score": [
            3.010026454925537,
            3.664743423461914,
            3.994750499725342
        ]
    ,
    "camera": 
        "view_scale": "",
        "movement": "",
        "speed": ""
    ,
    "misc": 
        "frame_caption": [
            "a person standing in a room with a laptop on their lap. ",
            "a black and blue background with the name of the series. ",
            "a black and white image of the words manga and comics. "
        ],
        "music_caption": [
            
                "text": "This is an indie rock music piece. There is a male vocalist singing melodically in the lead. The main tune is being played by the electric guitar while the bass guitar is playing in the background. The rhythm is provided by a simple acoustic drum beat. The atmosphere is easygoing. This piece could be used in the soundtrack of a teenage drama TV series as the opening theme.",
                "time": "0:00-10:00"
            
        ]
    ,
    "scene": 
        "objects": [
            " Matheus Trindade",
            " de",
            " original",
            "Series",
            " Manga     Comics"
        ],
        "background": "Dark",
        "ocr_score": 0.07147466104497355,
        "caption": "The video is a series of original de Matheus Trindade comics, showcasing the artist's unique style and storytelling.",
        "polish_caption": "Original de Matheus Trindade comics showcase the artist's unique style and storytelling, featuring a series of vibrant and expressive illustrations that bring characters and scenes to life.",
        "merge_caption": "In this captivating video, Matheus Trindade's original comics come to life with vibrant and expressive illustrations, showcasing the artist's unique style and storytelling. Set against a dark background, the video features a series of dynamic and colorful images, including a person sitting in a room with a laptop, a black and blue background with the name of the series, and a black and white image of the words m̈angaänd c̈omics.T̈he indie rock music piece, featuring a male vocalist and simple acoustic drum beat, adds an easygoing atmosphere to the video, making it perfect for the opening theme of a teenage drama TV series."
    



plain










 Paper Title: Towards Achieving Human Parity on End-to-end Simultaneous Speech Translation via LLM Agent.


UTF8gbsn
[colback=gray!20, colframe=gray!20,]

    0.1in
    Towards Achieving Human Parity on  End-to-end Simultaneous  Speech Translation via LLM Agent
     
    0.05in
    Cross Language Agent Team  
    0.05in
    ByteDance Research 
    0.05in
    July 20, 2024  
    



§ ABSTRACT 
  In this paper, we present Cross Language Agent - Simultaneous Interpretation, , a high-quality and human-like Simultaneous Speech Translation (SiST)[In this paper, we use Simultaneous Interpretation and Simultaneous Speech Translation interchangeably.] System.
  Inspired by professional human interpreters, we utilize a novel data-driven read-write strategy to balance the translation quality and latency.
  To address the challenge of translating in-domain terminologies, employs a multi-modal retrieving module to obtain relevant information to augment the translation.
  Supported by LLMs, our approach can generate error-tolerated translation by considering the input audio, historical context, and retrieved information.
  Experimental results show that our system outperforms other systems by significant margins. Aligned with professional human interpreters, we evaluate  with a better human evaluation metric, valid information proportion (), which measures the amount of information that can be successfully conveyed to the listeners. 
  In the real-world scenarios, where the speeches are often disfluent, informal, and unclear, achieves of 81.3% and 78.0% for Chinese-to-English and English-to-Chinese translation directions, respectively. In contrast, state-of-the-art commercial or open-source systems only achieve 35.4% and 41.6%. On the extremely hard dataset, where other systems achieve under 13% , can still achieve 70% . Demonstrations and human-annotated test sets are available at <>.






§ INTRODUCTION


Simultaneous speech translation (SiST) is recognized as one of the most challenging tasks in the translation domain <cit.>. 
Machine-assisted automatic interpretation has been receiving much attention in the natural language processing (NLP) community <cit.>. 
Traditional simultaneous translation approaches <cit.> 
usually employs a cascaded system, involving a streaming Automatic Speech Recognition (ASR) model, a punctuation model and a Machine Translation (MT) model.
However, such cascaded systems often suffer error propagation and latency from the ASR module. 
Despite these advancements in both academic SiST models <cit.> and commercial SiST engines, the translation quality is still far from satisfactory. 
As shown in <Ref>, we conduct a human assessment of the current accessible SiST systems. From the user-centered perspective, these systems only deliver less than 42% of the valid information to listeners, which heavily affects communication effectiveness. 
In contrast, professional human interpreters usually deliver more than 70% of the necessary information <cit.> and 95% ideally. Thus in this paper, we use 80% to indicate high-level human interpreters. 

Motivated by the huge success of LLMs in machine translation <cit.> and speech translation <cit.>, we propose to employ LLMs to accomplish the SiST task.
Specifically, we identify three primary challenges.
First, a key challenge for incorporating LLM into the SiST is the read-write policy, where LLM needs to provide partial translation for input speech. 
Second, achieving human equivalent performance requires understanding and translation of terminologies and uncommon phrases that LLMs cannot learn from training data. 
Lastly, the scarcity of training data continues to hinder the performance on the SiST task.


To address these challenges, we introduce our end-to-end approach, , a Cross-Lingual Agent that accomplishes Simultaneous Interpretation by iteratively performing multiple actions, as illustrated in Figure <ref>. 
Regarding the first challenge, we imitate professional human interpreters to learn their policy of segmenting a complete sentence into several semantic “chunks” through syntactic boundaries (pauses, commas, conjunctions, etc.) and contextual meaning. 
To enable to learn such a policy, we follow a data-driven policy learning process and invite human interpreters to annotate real-world speech, which includes the read-write timing for segmentation. From the data,  learns the robust read-write policy for SiST from humans.


For the second challenge, we include two external modules to augment our  agent: an external knowledge database that stores terminologies and paired translations, and a memory that stores the context of speech. 
However, the external knowledge database may contain tremendous terms that not only increase the inference time but may also lower the performance of our approach because of noisy intervention. Therefore, we propose a novel Multi-Modal Retrieval Augmented Generation (MM-RAG) process. 
A multi-modal retriever extracts knowledge from the external database based on the speech input. 
The retrieved information and the context from memory are then appended to the prompt of our LLM agent to augment the translation through in-context learning.

Addressing the data scarcity of the SiST task, we adopt a three-stage training methodology: pretraining, continual training, and fine-tuning. First, our LLM and audio encoder are independently pretrained on our large-size in-house datasets. 
Then, our model is continually trained with billions of tokens of mediocre-quality synthesized speech translation data, aiming to align the speech and text modalities.
We also include multiple tasks to enhance the in-context learning ability of LLM to better utilize the contextual information from the retriever and prior translation.
In the last stage, we fine-tune the model with a small amount of human-annotated data, further imitating professional human interpreters to improve the robustness and translation quality.



In addition, we would like to highlight that the conventional automatic evaluation metrics <cit.> of simultaneous interpretation might not be good indicators for reflecting the performance of SiST, which often contains compaction, abstraction, and paraphrasing. 
Aligned with human interpreters <cit.>, we propose a new evaluation metric named Valid Information Proportion ()[Detailed guidelines of our proposed metric can be found in Appendix <ref>.]. VIP represents the percentage of information that can be precisely delivered, reflecting the central objective of SiST: communication in real-time.
Through thorough human evaluation on diverse and challenging real-world long speech datasets, our approach outperforms other currently accessible systems by a large margin.
As shown in Figure <ref>, taking the Chinese-to-English direction as an example,  achieves a score of 81.3%, significantly narrowing the gap between machine-assisted systems and human interpreters.


Our contributions can be summarized as follows:

    
  * We introduce our end-to-end approach, , an LLM agent that is designed to perform high-quality and human-like simultaneous translation. Through human evaluation, our approach demonstrates significantly better performance compared to existing accessible SiST systems.
    
  * We propose a new data-driven read-write strategy by imitating professional human interpreters. Without the requirement of complicated human pre-design, the strategy could balance translation quality and latency effortlessly. Unlike most commercial systems where the outputs are frequently rewritten during the translation process for better quality, our strategy guarantees all the outputs are deterministic while maintaining high quality.
    
  * Motivated by the preparatory trajectory of human interpreters, we introduce a novel Multi-Modal Retrieval Augmented Generation (MM-RAG) process that empowers the LLM with domain-specific knowledge in real time. The proposed module
    further improves the translation quality with minimal computational overhead during inference. 
    
  * We work closely with professional human interpreters to develop our evaluation strategy, Valid Information Proportion (), and detailed guidelines are open-sourced. Meanwhile, we release a human-annotated test set focusing on diverse real-world scenarios and long speech translations.








§ METHODS






 §.§ Framework

<Ref> presents a flow of operation of . To perform the SiST task, we design 5 operations: , , ,  , and . The following sections describe the details of each operation.
As further illustrated in <Ref>,  is an LLM agent that can take input speech, instruction, relevant information retrieved from external knowledge, and last round memory as context. The memory stores previous transcriptions (optional) and translations. At round r, it first reads speech _t^r-1:T^r, where t^r-1 is the predicted cut-off time of round r-1 and T^r is the end time for audio stream at round r. 
Then the agent retrieves relevant information _r from the external knowledge and loads context _1:r-1 from the last round memory. 
Once “think” sufficient context is loaded, it generates the transcription (optional), translation, and cut-off timestamp t^r: 

    _r; t^r = TextDecoder(_t^r-1:T^r, _r, _1:r-1)

where t^r is the predicted cut-off timestamp indicating the end time for the current translation round r. 
_r is then forwarded to update the memory.
When instructed to output the transcription, the LLM optionally engages CoT to generate transcription first and then the speech translation.
For the following round r+1, the audio stream begins with the predicted cut-off timestamp t^r.


 §.§ Architecture

 employs an Encoder-Conditioned LLM architecture. As shown in <Ref>, the audio encoder transforms input speech stream  to a series of continuous representations . 
Then, the LLM takes the speech representation , retrieved knowledge , historical translation  and instruction  as a sequence of prompt (, , , ) to generate the translation result . 

Audio Encoder. The audio encoder module contains a large-scale speech conformer <cit.> pretrained on millions of hours of speech data to achieve human parity performance on ASR, and an audio adapter to connect the audio encoder and LLM. 
The adapter downsamples the speech representations and the resulting representations are linearly projected to match the dimension of the LLM embedding layer. 
The projected speech representations lower the computational latency for SiST.

Large Language Model. The language model[We use Doubao LLM as our foundation model.] is a medium size decoder-only transformer <cit.> to balance performance and computation efficiency. It is pretrained on a large amount of text data and fine-tuned with instructions. The LLM directly takes the continuous embedding from both the audio encoder and text embedder as input. It autoregressively generates the transcription and translation response of the provided speech stream. 

Multi-Modal Retriever. 
The multi-modal retriever framework employs audio and text encoders to independently encode the audio stream and text key of the terminologies in the external knowledge database.
To enhance the alignment between audio embeddings and text embeddings, we incorporate an embedding fusion layer, which includes a multi-head attention module followed by a pooling layer.
The resulting pooled representation is subsequently fed into a linear projection layer to produce the final scores, indicating the probability of the text key's presence in the audio stream.
Terminologies with top scores are forwarded to the  agent to enhance the translation quality.









 §.§ Data Driven Read-Write Policy:  and 


Unlike predetermined read-write probabilities and heuristic waiting policies detailed in prior research <cit.>, interpreters engage in a dynamic process of listening (read) and translating (write).
They attentively listen to the speaker's speech and segment lengthy sentences into semantic chunks, representing the smallest linguistic units capable of conveying a complete thought independently <cit.>.
Upon identifying a chunk that encapsulates sufficient information, they proceed to translate this segment into the target language, thereby providing an accurate and contextually appropriate translation.

Emulating the strategies of human interpreters,  does not require to explicitly define the read-write policy.  imitates their policies by waiting for complete semantic chunks. Specifically, given partial speech,  only generates the translation for the complete chunks of the input speech. The model is trained with segmented speech data to learn such ability.
Mathematically, given source audio _1:M, we segment its translation into a series of n “chunks” _1:n and obtain the corresponding pair {(_t^j:t^j+1, _j}_j=1^n, where _t^j:t^j+1 and _j denote the j-th segment of the audio and the corresponding translation. For training, our objective is to output all complete segmented translations and the cut-off time given random partial input audio _1:t

    min𝔼_t ∼𝒰[1, M] -log p_θ(_1:j; t^j|_1:t)    j = max_j{j|t^j < t}

where 𝒰 indicates uniform distribution over time of speech. 
Trained with <Ref>, learns to generate the cut-off time for the input speech. Additionally, the objective function makes the wait for appropriate time before starting translation as the LLM will output nothing when it “think” current speech stream does not contain a complete speech chunk.



 §.§ Context Information:  and 


The memory stores translations and transcriptions in previous rounds _1:r-1. It has two functions. Firstly, it works with the input speech to determine which part of the speech has been translated and which part has not, helping make the read-write decisions and outputs the translation of the unfinished parts. 
Secondly, understanding human speech often requires context. For example, when a speaker talks about “barrel bridge”, it often refers to the bridges built upon rivers that are supported by barrels. However, in the context of “watch”, it refers to a mechanical structure in the watch. The phenomenon of polysemy in different contexts can lead to vastly different translation outcomes. Therefore,  should be able to retrieve the context of the long speech for translating some keywords, and make appropriate translations under different contexts. 

As shown in <Ref>, at round r,  forwards relevant translations _1:r-1 to the LLM as a prompt. After agent generates the translation _r,  stores it to the memory and obtains _1:r.



 §.§ Multi-Modal Retrieval Augmented Generation: 


In real-world scenarios, the accurate speech transcription or translation of professional and domain-specific terminologies is challenging. Even human interpreters require prior domain knowledge to understand those terminologies, including names of people, locations, jargon, or special in-domain terms.
For example, an interpreter unfamiliar with the machine learning theory may not recognize the word “Rademacher complexity” when hearing it. 
Therefore, in various scenarios, human interpreters often prepare in advance to get familiar with the corresponding domain knowledge.

Motivated by the preparatory trajectory of human interpreters, we propose to integrate an external database to empower LLM with necessary domain-specific knowledge. Each item in the database contains a key and the corresponding value in text modality. The key, which may appear in the speech, is used as the input for the retriever. The value of the item may be itself, a paired translation of the target language, or even an explanation of the key. 

Theoretically, all items in the external database might be added into the prompt to provide information for the translation. However, the external knowledge database often contains tremendous items.
Simply prompting LLM with all the terms not only increases the
inference time but may also hurt the performance of because of noisy intervention. 
Therefore, we design a novel Multi-Modal Retrieval Augmented Generation (MM-RAG) process.
Our multi-modal retriever first retrieves the relevant terminologies from the database based on the input speech.
A small number of filtered items are incorporated into the prompt of  agent for in-context learning as shown in <Ref>. 

With the retrieved knowledge and previous context from the memory, our LLM has the in-context learning ability to better utilize the provided contextual information.
To achieve this, we collect a series of in-context learning data to train the model. Compared with the previous approaches for intervention, such as shallow-fusion <cit.> and traditional substitution-based methods <cit.>, which generates fixed translation for given translation pair. Our method achieves better results and generates more coherent text. For example, in some internet companies, “大盘 == overall performance”, while in most cases, it should be “stock market”. Our method can choose the correct translation given different context. Besides, our method can use monolingual text from both source and target language to help the translation.




§ MULTI-STAGE TRAINING

Our  follows a multi-stage training process: pretraining, multi-task continual training, and multi-task supervised fine-tuning.  
In the first stage, the LLM and audio encoder are separately pretrained with massive amounts of in-house speech and text data. 
Next, a large amount of speech-text paired data is used to align audio and text modalities, building the fundamental capability for cross-modal multitasking. 
In the final stage,  agent is fine-tuned with a small amount of human-annotated data to imitate the translation behavior of professional human interpreters.
Our multi-stage training process enables high efficiency of learning with a small amount of human-labeled data.



 §.§ Pretraining

The in-house LLM and audio encoder are independently pretrained on different modalities of data.
The LLM follows a decoder-only transformer architecture and is first pretrained on a massive amount of monolingual and bilingual text data with cross-entropy loss and then fine-tuned on instruction-following data. The LLM performs excellently on various downstream tasks, especially translation tasks.

The audio encoder also follows a classic pretrain-finetune paradigm <cit.> with a massive amount of speech-related data. 
The pretraining stage provides a proficiently trained LLM and audio encoder, setting a solid foundation for the following stages. 




 §.§ Multi-task Continual Training

Training Method.
We follow the work of <cit.> for multi-task training. Specifically, for streaming and higher-quality translation, we mainly focus on three tasks for training : Automatic Speech Recognition (ASR), Speech Translation (ST), and Text Translation (MT). 
To align the modalities of the pretrained LLM and audio encoder, 
 is continually trained on various tasks with a substantial volume of paired data. 
We further strengthen the in-context learning ability of our approach by incorporating translation in the memory and knowledge from external databases.
As a result, we expand the ST tasks to different configurations as shown in <Ref>. An ST translation can either be streaming or offline, direct or COT, with or without context, which leads to 8 different tasks.




Training Data Construction.
Both ASR and MT tasks have been last for a while, and there is a relatively large amount of ASR and MT data.
The major challenge of developing an end-to-end SiST model is the data scarcity of simultaneous ST. 
To this end, we propose a synthetic data construction pipeline. With a strong LLM, we synthesize two types of speech translation data for continual training: offline ST data and context-aware segmented streaming ST data.

1. Offline ST data: We mainly rely on ASR data to construct the offline ST data. Given the ground-truth transcription of speech, we use in-house LLM to translate the source language to target languages.
To ensure the readability and conciseness of the target language, the LLM is prompted to conduct Inverse Text Normalization (ITN), filler word smoothing, etc.

2. Context-aware segmented streaming ST data: The streaming ST data consists of fine-grained audio-text alignments and translation pairs for segmented semantic chunks.
Compared to offline ST data, streaming ST data
is even more challenging to collect. 
We find that human interpreter often segments long speech into a few semantic chunks, each of which can be translated independently to ensure an effective and smooth translation.
Motivated by such findings, we leverage LLM to construct streaming ST data by imitating the chunking process.
Long speech data are used to construct the streaming ST data, as the additional history can provide better contextual information.
First, we prompt the LLM to break down the ASR transcription into multiple independent semantic chunks, which are then translated into the target language. 
Subsequently, we align the semantic chunks with the corresponding audio chunks, obtaining the streaming ST data. 
Such data enable our model to handle incomplete speech inputs and generate partial translation in coherent semantics.

To measure the quality of the synthetic data, we conduct human evaluations based on our proposed metric. 
The synthetic data achieves a score of 81%, satisfying the minimal requirement for further training. 



 §.§ Multi-task Supervised Fine-tuning

Training Method.
Even though  possesses a good translation quality on the SiST tasks after the previous multi-task continual training stage, we further boost the performance by fine-tuning on human-annotated streaming ST data with diverse tasks listed in <Ref>. 
Such high-quality data enables our model to better align with the segmentation methodologies of professional human interpreters.
Furthermore, this process enhances our model's robustness to speech disfluencies such as stuttering, ensuring smoother communication in real-world scenarios.

Training Data Construction.
The source of human-annotated streaming ST data originates from real-world scenarios that contain various speech characteristics, such as disfluencies, stuttering, code-mixing, and specialized terminologies.
Such features ensure the robustness of our model in diverse conditions.
We engage professional human interpreters to provide high-quality annotations for simultaneous segmentation and interpretation of the speech data. 
Additionally, terminologies are identified and translated within the context, further strengthening the context-aware capabilities of . 



 §.§ Multi-Modal Retriever Training

The multi-modal retriever is independently trained with a substantial dataset of speech recognition data. 
During training, words are randomly selected from speech transcription to serve as the positive sample, indicating their appearance in the speech.
Negative words are selected from different sentences, indicating the speech does not mention these words.
We assign a label of 1 to positive samples and 0 to negative samples, aiming to minimize the Binary Cross Entropy (BCE) loss. This approach helps refine the model's ability to distinguish relevant from irrelevant information, enhancing its overall performance and accuracy.
We label the positive sample as 1 and the negative sample as 0, minimizing the Binary Cross Entropy (BCE) loss.
To evaluate the effectiveness of our retriever, we build an in-house retrieve development set. Each sample in the development set includes a short audio chunk and the mentioned terms in the audio. 
Note that the term here is defined as special keywords, such as name, location, abbreviation, and domain-specific word.




§ EXPERIMENTS




 §.§ Evaluation Benchmark

Quite a few number of evaluation benchmarks have been proposed for SiST over the past years, including MuST-C <cit.>, FLUERS <cit.>, CoVoST <cit.>, BSTC <cit.>, and GigaST <cit.>, etc. 
However, although much effort has been spent to build these benchmarks, they still suffer some shortcomings when facing real-world SiST applications. 

First, these benchmarks often contain speeches that are either recorded by volunteers (e.g. CoVoST and FLUERS) or collected from formally, clearly, and fluently talk and podcasts by well-prepared speakers (e.g.  MuST-C and GigaST). In real-world scenarios such as online meetings or social media videos, the characteristics of the speech might inevitably be informal, unclear, or disfluent. 
Second, these benchmarks provide a shortcut for evaluating the translation quality by giving the manually segmented sentences. Such a shortcut offers a gap between the current benchmark and real-world applications, where the models might need to take long speech and conduct segmentation <cit.> by themselves. Consequently, evaluations on manually segmented datasets are likely to overestimate the performances of a real-world SiST system.
These discrepancies result in the evaluation on these benchmarks are not reliable for practical SiST systems. 

As a preliminary attempt to address the shortcomings as mentioned earlier, we propose a new benchmark for Chinese-to-English (zh-en) and English-to-Chinese (en-zh). is collected from diverse sources, and most speakers talk naturally and casually without careful preparation.
We choose 10 popular domains: technology, healthcare, education, finance, law, environment, entertainment, science, sports, and art.
One video clip is selected for each domain from a well-known online video platform for both zh-en and en-zh settings.[is available at <https://github.com/byteresearchcla/RealSI>. We do not own the copyright of the videos and only release our annotations together with the publicly available website links of the corresponding videos. If anyone believes that the content constitutes infringement, please contact us. We will remove the relevant content as soon as it is confirmed.]
Each sample in is a nearly 5-minute speech to mock SiST without manual segmentation.
For systems that cannot take long-form audio as input, we also provide sentence-level timestamps for segmentation. 
<Ref> presents the detailed statistics of our .






 §.§ Baselines

We compare with the open-sourced SiST model, SeamlessStreaming <cit.>. In addition, because of the limited number of available SiST models, we choose to compare  with several commercial systems. We denote the commercial systems as Commercial 1-4. It is worth noting that unlike , most of the commercial SiST systems will first generate a temporary translation as soon as possible, then rewrite the temporary translation with a potentially better translation after getting more context.
Notwithstanding, we evaluate the finalized translation of these systems in all our experiments. 
Although this re-writing strategy could improve translation quality, continually revising existing translations might affect the user experience, potentially leading to additional confusion. 
We would also like to highlight that human interpreters usually do 
 not employ such a rewriting strategy during translation. 
 During the entire evaluation, we employ a general external knowledge database that maintains the same for all the evaluation in this paper. It does not contain domain-specific external knowledge to form unfair comparisons. The improvement of external knowledge is independently reported in <Ref>.



 §.§ Translation Quality

Evaluation Metrics.
Automatic evaluation metrics such as BLEU <cit.>, BLEURT <cit.>, and COMET <cit.> are widely used for evaluating the translation quality <cit.>. 
However, they may not be able to fully reflect the quality of the translation, especially for paragraph-level translation of long speech. It is argued that the current evaluation metrics are not sufficient for ST and SiST tasks <cit.>. The work of <cit.> also highlighted that there might be a discrepancy between automatic evaluation metrics with human evaluation. 

Therefore, besides the automatic evaluation, we collaborated with senior professional human simultaneous interpreters to standardize the guidelines for a more realistic human evaluation. 
Our proposed human evaluation metric focuses on whether the output of the translation model can accurately convey the speaker's original intention for each semantic fragment.
This is also the key objective of human interpreters in real-time translation. Note that a single semantic fragment indicates a complete piece of source speech. Typically, a single semantic fragment is a complete sentence. 
Detailed definition can be found in <Ref>.
The percentage of valid information fragments within a complete speech session is defined as , which is consistent with real-world criteria for human simultaneous interpretation <cit.>.




Quantitative Analysis. 
As shown in <Ref>, we compare  with the baseline methods on dataset. 
In terms of the reliable human evaluation metrics, ,  achieves scores of 81.3% and 78.0% for zh-en and en-zh, respectively. While all other models' scores are lower than 42%. For more references, we use 3 widely-used automatic evaluation metrics: BLEU[We use SacreBLEU <cit.> for all the BLEU calculations in this paper.], BLEURT, COMET. Under the automatic evaluation metrics,  also surpasses baselines by a large margin. The detailed human evaluation results of can be found in <Ref>. 

High marks  a practical system that can help listeners understand real-time speech without professional human interpreters. Note that we only consider a system is better than others when the is higher. For example, even though Commercial 1 achieves higher scores than SeamlessStreaming on BLEU and COMET, we still consider SeamlessStreaming is a better system for zh-en translation based on .



 §.§ Latency

Evaluation Metrics. Due to the differences of grammatical structures between languages, a delay in simultaneous interpretation is inevitable. In this paper, we adopt the widely-used Average Lagging (AL) <cit.>, Length Adaptive Average Lagging (LAAL) <cit.> for comparing the latency of different methods. 
To achieve a fair comparison with systems that rewrite the translation, we calculate the time of the definite translation of these systems. 
We also propose an additional metric, First Letter Appearance Lagging (FLAL), to reflect user experience on each system.
FLAL represents the time that each system outputs the first determined translation. 

Quantitative Results. <Ref> compares the latency of our model with various systems in terms of AL, LAAL, and our proposed FLAL on the and CoVoST.
We find that the existing metrics AL and LAAL are not suitable latency measurements of paragraph-level SiST on . 
When the results are significantly shorter or longer than the reference translation, AL and LAAL may be largely exaggerated, leading to unreliable high latency. 
In these scenarios, FLAL is a more reliable and stable metric for all the systems.

Besides the paragraph-level latency evaluation, we compare our approach with other systems on the sentence-level dataset CoVoST2 zh-en, where both AL and LAAL produce reasonable values and the results are shown on the right side of  <Ref>.
Since the commercial systems usually rewrite the translation, their latency is higher than the . 
Compared with the fastest approach SeamlessStreaming, achieves comparable latency but much better translation quality.




Discussion. While existing works put a lot of emphasis on the latency-quality trade-off <cit.>, human interpretation usually uses Ear-Voice-Span (EVS) to evaluate the lagging. EVS measures the average time from when the speaker finishes conveying a piece of information to when the audience hears the corresponding translation, which is similar to AL. The typical EVS of professional human interpreters usually ranges from 3 to 6 seconds <cit.> to achieve high-quality translation. 

Consequently, we perform user studies and argue that the latency is less important than the translation quality for a practical SiST system. In the recent IWSLT 2023 simultaneous track <cit.>, the ranking of models is also evaluated by the translation quality within certain latency constraints. We verify whether the latency of is acceptable to users through real-world user surveys. To the publication date of this paper, we collected 14 user surveys on zh-en direction, each user using for at least 30 minutes. Under the current latency performance shown in <Ref>, only 1/14 == 7% of them suggest that the latency significantly affects their user experiences while the rest think the improvement of translation quality outweighs the latency and overall output of largely helps them to understand the speech. Considering that the latency of is even lower than most of the commercial systems, We believe the latency of can be acceptable on most cases. 

Current latency metrics are proposed on sentence-level SiST. As shown in <Ref>, such metrics may not be suitable latency measurements for paragraph-level. 
As the importance of end-to-end evaluation for long speech keeps increasing, more refined metrics are required to measure the latency and provide a deeper insight into the systems.



 §.§ Supplementary Experiments


To ensure a comprehensive evaluation of , our model is further evaluated on four additional datasets, including BSTC (zh-en) <cit.>, CoVoST2 (zh-en) <cit.>, MuST-C (en-zh) <cit.>, and GigaST (en-zh) <cit.>[We use the subset from in https://github.com/SpeechTranslation/GigaS2SGigaS2S for evaluation].  
<Ref> presents the results of the automatic evaluation metrics for both zh-en and en-zh. 
Due to the high cost of human evaluation, we are not able to provide VIP for these four datasets.
We observe that our model achieves consistently better performance than the baseline models.
Even though our system achieves the best automatic evaluation results among all the compared systems, 
we still would like to emphasize that such a sentence-level evaluation scheme might overestimate the performance of SiST systems.






 §.§ MM-RAG Performance




  §.§.§ Retriever

Table <ref> presents the performance of various retrieve models on the development set of our proprietary dataset. Each sample in the test set includes a short audio chunk and the mentioned terms in the audio. 
Our MM-RAG retriever outperforms other open-source models by a large margin, achieving 91.3 % vs. 26.0% for Top-10 retrieve accuracy.
We compare two types of methodologies: audio-to-audio and audio-to-text. 
In the audio-to-audio approach, a Text-to-Speech (TTS) model is utilized to convert the text keys from the external knowledge database into audio format,
 forming a database with audio-based keys. 
The audio keys and the user-input audio are then encoded with the ASR model to produce the corresponding representations. 
The Top-k retrieved items are subsequently determined using the Maximum Inner Product Search (MIPS) algorithm.
For audio-to-text approach, we compare MM-RAG with CLAP <cit.>. As indicated in Table <ref>, the effectiveness of these models remains significantly below acceptable standards, and MM-RAG significantly outperforms them.

It is worth noting that the same audio encoder employed in our is utilized for generating audio embedding in the MM-RAG retriever.
Such a design ensures that the integration 
brings minimal computational latency to the overall framework.





  §.§.§ ICL Performance


By incorporating the retrieved terms from the external knowledge database as contextual information, our model's in-context learning ability significantly improves the performance of speech translation for in-domain terminologies. <Ref> compares our method with the widely-used shallow fusion for intervention in the generated conclusion. When calculating the Recall, we input 1 ground-truth keyword with 9 similar negative words as context. When calculating the false positive rate for precision, we input 10 similar negative words as context. ICL is able to achieve a high recall rate with good precision, obtaining the highest F1 while shallow fusion only gets a recall rate that is only half of ICL.

Additionally, we conduct an ablation study on our MM-RAG module within terminology-intensive scenarios incorporating the whole  pipeline.
The incorporation of the external knowledge database results in a significant increment in the score by about 10%, highlighting the effectiveness of our proposed MM-RAG.





 §.§ Case Study
 
We present case studies to show the ability of in translating complicated speech for zh-en and en-zh in <Ref> and <Ref>. We choose one of the most-performed cascaded systems for comparison. The adopted a cascaded approach for SiST and it is shown in <Ref> to be one of the best previous SiST systems. Detailed explanations are described in the tables. For zh-en direction, we present cases regarding robustness to recognition errors, reasoning ability, and trending words translation. For the en-zh direction, we present cases regarding native, expressive, and accurate terminology translations. More cases are shown in <Ref>.











§ RELATED WORK



  
Large language model.

The encoder-decoder architecture <cit.> has been widely explored in early speech translation research, but with the advent of large language models <cit.>, there has been a growing interest in employing decoder-only architectures <cit.> for sequence-to-sequence problems. While recent efforts have emerged in utilizing large language models for machine translation <cit.> and speech translation <cit.>, the application of such models in simultaneous translation tasks remains limited. Although there has been early attempts to utilize LLM for simultaneous machine translation <cit.>, to the best of our knowledge, no existing work has been found that explores the utilization of large language models for end-to-end simultaneous speech translation with such remarkable improvement.

Furthermore, LLMs have demonstrated impressive capabilities in tasks such as instruction following <cit.>, reasoning <cit.>, and planning <cit.>. Recent research studies have leveraged prompt engineering to develop remarkable LLM agents that autonomously tackle complex tasks in diverse environments <cit.>.  
In our work, we empower the LLM to perform sequential instructions to accomplish the simulation speech translation task.



  
Simultaneous Speech Translation.
One of the important components of simultaneous speech translation is the segmentation strategy, which determines how the speech frames are fed to the models.
Different strategies could affect the latency and performance of the translation. 
According to <cit.>, segmentation strategies can be classified into fixed-length, word-based, and adaptive segmentation. Fixed-length strategies <cit.> divide the speech into equally-length segments, while word-based strategies <cit.> identify word boundaries within the speech. Adaptive segmentation <cit.> detects boundaries for speech units. Among these categories, our method utilizes a fixed-length strategy.

Regarding the read/wait policy, the Wait-k method <cit.> and its variants <cit.> have been extensively studied in the context of text translation and speech translation. In comparison to these approaches, which explicitly learn the generation of read/write signals, another line of research focus on how to leverage offline translation models <cit.>.
When utilizing an offline translation model for simultaneous translation, it is important to address the stabilization of generated hypotheses to prevent excessive content refreshing experienced by the user.
<cit.> first proposed a local agreement policy to stabilize the partial hypothesis, while <cit.> introduced an incremental blockwise beam-search algorithm. In contrast to these methods, we enforce our model to generate consistent hypotheses by constraining the prompt to the language model.

For the model architecture, there are two primary methods for implementing speech translation systems: cascaded solutions <cit.> and end-to-end solutions <cit.>. Cascaded solutions involve separated ASR and MT components, while end-to-end solutions directly map speech to translations. Cascaded systems benefit from established techniques but suffer from latency and error propagation. End-to-end models offer real-time translation and improved quality through deep learning but require large-size training data. 
In our work, we implement an end-to-end model which combines the capabilities of ASR, MT, and ST.



  
Human Evaluation. 
In the realm of speech translation, the choice of evaluation metrics plays a crucial role in assessing the quality and effectiveness of translation systems. While automatic metrics, such as BLEU <cit.>, BLEURT <cit.>, and COMET <cit.>, have traditionally been relied upon for evaluation, there is a growing recognition that they may not be the most suitable or comprehensive measure of performance <cit.>.
We observe that in more recent works <cit.>, there is an increasing trend of evaluating systems using human assessments, particularly when LLM is employed in the work.
While human evaluation requires more resources and time compared to automatic metrics, its benefits outweigh the drawbacks. By incorporating human judgment, speech translation systems can be refined and optimized to align with user expectations, ensuring translations that are not only technically accurate but also linguistically and contextually appropriate.
In contrast to the existing human evaluation metrics, e.g.“continuous rating” <cit.> and MQM (Multidimensional Quality Metrics) <cit.>, we have taken inspiration from professional human interpreters <cit.> and propose to use VIP (Valid Information Proportion) as a human evaluation metric which precisely reflects the goal of the simultaneous translation task.







§ CONCLUSION

In this work, we introduce Cross Language Agent - Simultaneous Interpretation, , an LLM agent to produce end-to-end simultaneous speech translation. Benefits from massive pretraining and imitation learning, achieves significantly better performances than state-of-the-art systems. Take the Chinese-to-English direction as an example, under strict and challenging human-evaluated metrics proposed by professional human interpreters, Valid Information Proportion (),  significantly outperforms baselines by a large margin. While all other systems obtain by less than 40%, achieves a VIP of 81.3%, demonstrating human parity performance.
More specifically, we propose the following crucial components for the supreme performance of : (1) An encoder-conditioned LLM agent architecture that performs high-quality or even human-parity SiST process through simple actions. (2) Imitation learning from human interpreters for a natural read-write policy balances translation quality and latency in a data-driven manner, without complex human pre-designing. Under such policy, achieves a stable output scheme, where each output is deterministic, thus potentially better user experience than most commercial systems. (3) Motivated by the preparatory trajectory of human interpreters, could perform in-context learning from historical translations and external knowledge to provide sufficient information for translation. 
With the powerful translation ability of ,  we believe it can further make cross-lingual communication seamless across different places all over the world.



§ LIMITATION AND FUTURE WORK

Although we achieved significant improvements over commercial systems in Chinese-to-English and English-to-Chinese tasks, more languages should be considered in the future. 
In our current implementation, performs a full action sequence for each translation round. Some of the actions, e.g.,  is optional for easy translation scenarios since the model is capable of translating correctly without the help of external knowledge. Training the model to better determine whether to skip unnecessary actions is a future direction. 
For a product-level system, even though the latency of is acceptable in most cases, how to reduce the translation latency without lowering the translation quality is still interesting and potentially helpful for user experience. 
Furthermore, we argue that the current automatic metrics are not comprehensive for SiST evaluation. Most of the quality measurements do not consider key information, which is crucial in SiST scenarios. As such, we proposed for better human evaluation. Consequently, more reliable automatic quality and latency metrics should be proposed in the future as well. 
Reinforcement learning from human feedback (RLHF) has been proven to be effective in enhancing LLM performance. Although achieves significantly superior results than previous state-of-the-art systems, further studies on how to build better multi-modal reward models and better RL methods for SiST is also an important direction. Incorporating more modalities, for example, end-to-end speech-to-speech generation, or even end-to-end video-to-video generation are also promising research topics.



§ SOCIAL IMPACT

The powerful SiST system can be applied to various scenarios to facilitate cross-lingual communications. For example, it can be deployed to various conferences or daily meetings to help listeners understand speech in different languages. It can also be deployed as a system-level translation module to help users watch videos that are conveyed in different languages. For online gaming, it can also help to bridge the gap of cross-lingual communication and connect people speaking different languages. A powerful SiST system with human parity performance may significantly improve the efficiency of professional human interpreters.

Despite the huge positive social impact that may bring, every coin has two sides. Neglecting some low-resource languages may also bring unfairness to some minorities. Resolving these problems needs further cooperation from the society. We leave more languages supporting as our future work.





§ AUTHORSHIP AND ACKNOWLEDGEMENTS

All contributors are listed in alphabetical order by last name. Corresponding to this work can be sent to any core authors' email.

Core Authors. 
All core authors contributed equally to this work. 
2

    
  * Shanbo Cheng 
    
  * Zhichao Huang
    
  * Tom Ko
    
  * Hang Li
    
  * Ningxin Peng
    
  * Lu Xu
    
  * Qini Zhang
    
        
        
  * chengshanbo@bytedance.com
        
  * zhichao.huang@bytedance.com
        
  * tom.ko@bytedance.com
        
  * lihang.lh@bytedance.com
        
  * pengningxin@bytedance.com
        
  * xu.lu1@bytedance.com
        
  * qini.z@bytedance.com
    



Labeling, Evaluation and Interpretation Team. 
Our data labeling and human evaluation team led by Yifu Li, made diligent efforts in all kinds of help needed, which is irreplaceable in the success of this project. Special thanks to the human interpreter team led by Anna Liu for providing insightful and comprehensive analysis on data labeling, human evaluation, and other recommendations. 

    
  * Jingwen Chen
    
  * Xiaoya Chen
    
  * Yifu Li
    
  * Huiying Lin
    
  * Anna Liu


Engineering Team. 
We collaborated with our engineering team led by Tingshuai Yan, their infrastructure support is crucial for this project.

    
  * Weicheng Fu
    
  * Tingshuai Yan
    
  * Liehao Zou


Acknowledgements. We appreciate the Speech Understanding team for all kinds of help, especially data sharing, thanks to their tremendous work for all the in-house data. We would like to express our deepest thanks to all the contributors to this project, their brilliant work guarantees the success of this project.



plain








§ HUMAN EVALUATION GUIDELINES


In this Appendix, we provide detailed human evaluation guidelines which are formulated by professional human interpreters.


 §.§ Key Indicator
 
We define the key indicator as the “Valid Information Proportion,” denoted as . This metric measures the proportion of valid semantic fragments within a complete speech session. A semantic fragment is deemed valid if it effectively conveys the core information, accurately representing the speaker’s original intent. Typically, one complete sentence is considered a single semantic fragment. assesses the model’s ability to capture and communicate the essence of the spoken content. 



 §.§ Evaluation Process


First, the human evaluators segment the long translation result into semantic fragments according to the formal rules as follows:

    
  * Semantic Completeness: Each fragment should contain one complete concept or information point. For example, in a conference translation, an ideal semantic fragment often corresponds to a full sentence.
    
  * Natural Language Pauses: Pauses that naturally occur in speech often indicate the boundaries of semantic fragments. During segmentation, natural pauses should be extensively considered and avoid irrational interruptions. Also, punctuation and conjunction in the text should be considered to maintain integrity and clarity of information.
    
  * Logical Coherence: Each segment should contain information that are logically coherent and continuous. Conditional sentences, causative sentences, or both parts of antithesis sentences should be kept within the same segment.
    
  * Grammatical Completeness: Each segment should include all necessary grammatical components (e.g., subject, verb, object) and have a complete grammatical structure.
    
  * Proper Information Density: Each segment should have a moderate amount of information, avoiding information overload. It is recommended that each segment not exceed 50 words.


After segmentation, the human evaluators follow the instructions below to evaluate the validity of the semantic fragments:

    
  * Key Information Recognition. Key information refers to the content that can constitute core information, including but not limited to proper nouns, keywords, terminologies, sentence structures, etc.
    
  * Correctness Assessment. Evaluators assess whether the translation of key information is accurate and successful in conveying the correct spoken intentions. Misinterpretations of the speaker’s words, inaccuracies in analyzing the context, or erroneous translations of specific terms can all contribute to the failure of the assessment. 
    
  * Expressiveness Assessment. Evaluators assess whether the whole segment is translated accurately, comprehensibly, and expressively to humans. Assessing for any vague, ambiguous, or misleading statements. This indicator primarily evaluates the clarity, fluency, and intuitiveness of the translation, rather than its accuracy. Typically, verbosity, complex sentence structures, or challenging grammatical constructions that are unnecessary would reduce the expressiveness of the translation, thus leading to failure of the assessment. 

If the translation fails any of the above assessments, the translation will be marked as invalid. After the evaluators assessed all semantic fragments, the could be simply calculated as dividing the number of valid semantic fragments by the total number of fragments.


We illustrate the evaluation criteria with two examples in Table <ref>. Although these translations achieve “high accuracy” in automatic evaluations, we still categorize them as invalid. It’s important to note that our standard aims to emulate human interpreters, presenting a significant challenge to both human evaluators and translation systems.

UTF8gbsn






 §.§ Correlation with Automatic Metrics

<Ref> shows the distribution and regression curve for with regard to BLEU, BLEURT, and COMET, respectively. From the scatter points in <Ref> we may observe that as score increases, the growth of the automatic metric curves slows down and becomes less significant, making it hard to reflect the real changes in translation quality. The correlation curves in <Ref> also demonstrate the finding. Here, we calculate Kendall's Tau correlation coefficient <cit.>, which measures the monotonic correlation between two ordered variables. In low ranges, the correlation between and automatic metrics is observable; as the score increases, the correlation harshly drops, which indicates a significant distortion of the automatic metrics. A possible reason is that the translations may differ from the groundtruths by only a few words in the mediocre ranges. However, in a real simultaneous interpretation scenario, these words are likely to be keywords that play important roles in conveying precise information, which may significantly impact scores.





§ SUPPLEMENTARY MATERIALS




 §.§ Supplementary Case Study

We provide more case studies in <Ref>. In terms of informal, disfluent, code-mixing, and named-entity translation, could achieve much better results than the commercial products. Benefits from the end-to-end approach, could also understand the original speech tone and generate better translations.





 §.§ Example of Detailed Evaluation Result on 


We provide a detailed human evaluation result for in Figure <ref>, where we provide golden source transcription, output, human evaluation results, and reference translation. We randomly choose one of the test samples in . We share the full detailed evaluation results at https://docs.google.com/spreadsheets/d/1OBDbQMlixqFkBfzLW5pmP8WsI-RHqAtYbLnkd321N6Aonline sheets for academic reference. Note that to ensure fair comparison, when evaluating multiple systems, we randomly shuffle the ordering between systems for each semantic fragment so that human evaluators cannot identify the specific system.






 Paper Title: Bailing-TTS: Chinese Dialectal Speech Synthesis Towards Human-like Spontaneous Representation.




RDP: Ranked Differential Privacy for Facial Feature Protection in Multiscale Sparsified Subspace

    Lu Ou
Hunan University

Changsha, China 

oulu9676@gmail.com
Shaolin Liao
IIT

Chicago, USA 

sliao5@iit.edu
Shihui Gao
Hunan University

Changsha, China 

gaoqaq@hnu.edu.cn
Guandong Huang
State Grid Hunan Materials Company

Changsha, China 

huanggd@hnu.edu.cn
Zheng Qin
Hunan University

Changsha, Country 

zqin@hnu.edu.cn







    
================================================================================================================================================================================================================================================================================================================================================




§ ABSTRACT 
Large-scale text-to-speech (TTS) models have made significant progress recently. However, they still fall short in the generation of Chinese dialectal speech. To address this, we propose Bailing-TTS, a family of large-scale TTS models capable of generating high-quality Chinese dialectal speech. Bailing-TTS serves as a foundation model for Chinese dialectal speech generation. First, continual semi-supervised learning is proposed to facilitate the alignment of text tokens and speech tokens. Second, the Chinese dialectal representation learning is developed using a specific transformer architecture and multi-stage training processes. With the proposed design of novel network architecture and corresponding strategy, Bailing-TTS is able to generate Chinese dialectal speech from text effectively and efficiently. Experiments demonstrate that Bailing-TTS generates Chinese dialectal speech towards human-like spontaneous representation. Readers are encouraged to listen to demos at <https://giantailab.github.io/bailingtts_tech_report/index.html>.




§ INTRODUCTION

The goal of text-to-speech (TTS) systems <cit.> is to generate human-like speech from text. With the development of neural network and deep learning, large-scale training corpus of a human-like level <cit.> is established and corresponding TTS models is developed. However, such system <cit.> is only able to generate voice for non-dialectal speech. And the quality of the generated speech is not satisfactory <cit.>. Therefore, in order to generate human-like quality of speech from dialectal text, we propose Bailing-TTS, a family of speech generation models for synthesizing Chinese dialectal speech from text with human-level naturalness. 

Besides, in zero-shot speech synthesis, both the naturalness and the robustness of speaker modeling are challenging <cit.>. In previous studies, large-scale models are developed for speech synthesis<cit.>. Despite the advances from these methods, the generalization capability of these methods is still insufficient for the generation of high-quality dialectal speech. Therefore, we primarily propose Bailing-TTS to generate high quality Chinese dialectal speech from text.

We propose several specific extension techniques that significantly enhance the quality of Chinese dialectal speech generation. First, continual semi-supervised learning framework towards text and speech tokens is developed to facilitate the alignment of these multiple modalities. Second, a multiple-stage Chinese dialectal representation learning framework introduces a specific network architecture and the corresponding learning strategy to improve the quality of the generated Chinese dialectal speech. 

In the evaluation of the proposed Bailing-TTS, both the advantages and disadvantages of the proposed model for the dialectal speech generation is compared with human speakers. The evaluation results represent that Bailing-TTS generates good quality of Chinese dialectal speech.

Finally, both the potential applications and the limitation of Bailing-TTS are discussed. Then, the future work of Bailing-family foundation model is represented including taking multiple modalities as input and producing audio output. For example, it's supposed to produce the generation of music <cit.> from the text and video. Besides the generation of audio such as music or speech, in our next step, the video and audio are generated simultaneously. 

Our key contributions are represented as the following:



  * We introduce Bailing-TTS, a family of foundation models for synthesizing Chinese dialectal speech from text with human-level naturalness, such as spontaneous speech.

  * Both a continual semi-supervised representation strategy and Chinese dialectal representation with a specific mixture-of-expert network architecture are proposed to improve the quality of the generated Chinese dialectal speech. 

  * A variety of hierarchical reinforcement post-learning extension techniques are represented to significantly enhance the quality of Chinese dialectal speech generation.




§ BAILING-TTS

We propose Bailing-TTS based on an auto-regressive transformer model of multiple layers. The proposed Bailing-TTS is trained on a large-scale of dataset including a large part of dialectal data. Both non-precise annotation data and high-quality annotation data are in the dataset. Besides, we propose a multi-stage training strategy, based on a specific transformer architecture (Figure <ref>), to facilitate the generation of spontaneous and expressive speech from text.





 §.§ Overall Architecture

First, at the stage of token representation, a continual semi-supervised learning strategy of spontaneous, expressive text and speech token pairs is proposed to facilitate weak alignment between the two modalities. Second, a specific transformer-based architecture is proposed corresponding with a multi-stage training strategy. Third, during inference, it generates good quality of spontaneous Chinese dialectal speech from text. 



 §.§ Continual Semi-supervised Learning Towards Text and Speech Tokens

Spontaneous speaking style exhibits notable differences from other speaking styles due to various spontaneous phenomena (e.g., filled pauses, prolongation) and substantial prosody variation (e.g., diverse pitch and duration variation), posing challenges to modeling and prediction of spontaneous style. 

In order to generate spontaneous, expressive speech from text, we present a multi-stage and multi-modal (text and speech) pre-training learning framework for text-speech token alignment (Figure <ref>). In the first stage, an unsupervised sampling strategy is proposed on a large-scale non-precise annotation datasets, forming a multi-stage pre-training process. In the second stage, a fine-tune sampling strategy is proposed on a high-quality dataset containing spontaneous, expressive text and speech pairs. Therefore, the token-wise text-speech association is represented to facilitate alignment between the two modalities. 





 §.§ Multi-Chinese Dialectal Representation Learning

Dialect-specific TTS models are not satisfactory as dialect-specific data is scarce. Therefore, a single unified TTS model that generalizes well for many dialects is in demand. 

In order to train a unified TTS model for a variety of Chinese dialects, we present a specific token representation together with a multi-stage learning strategy. In the aspect of the token representation, a design of mixture-of-expert architecture (Figure <ref>) is proposed to learn the unified representation of multiple Chinese dialects and specific representation of each dialect. Similarly, we propose a cross-attention based mechanism to inject dialect tokens throughout the TTS model layers. 



 §.§ Hierarchical RL-based Posting-training Extension

Similar to text-based language models, four training stages are built for the representation learning of Bailing-TTS. The first stage is the pre-training stage aiming to maximize scenario and speaker coverage with a robust backbone for general speech representation of the Chinese dialect. This stage is to learn the efficient representation of a large-scale Chinese dialect dataset. The second stage is the fine-tuning stage in order to enhance the basic performance of the dialectal speech. 

Besides, we propose a hierarchical RL learning strategy as the third and the fourth step in order to generate the high-quality speech of multiple Chinese dialects. In the RL hierarchical training strategy, in order to let the high-level policy support exploration for good quality of spontaneous, expressive speech and the primary-level policy support good quality generation of dialectal speech simultaneously. We propose dynamic sub-optimization for the learning of the high-level policy at the base of the primary-level policy in the hierarchical learning strategy.



§ EXPERIMENTS AND RESULTS




 §.§ Experimental Settings

In this section, the details of the training, inference and evaluation for the proposed Bailing-TTS on Chinese dialectal speech synthesis are introduced. 



  §.§.§ Implementation Details

A custom dataset containing high-quality Mandarin and multiple Chinese dialects data is used for the training, inference and evaluation. This dataset contains 200k hours of labeled speech data. The annotation contains fine-grained labels between text and speech modalities. In addition, this large-scale dataset contains high-quality speech-text pairs. Furthermore, this custom Chinese dialect dataset is used for the pre-training stage of the continual semi-supervised learning. Clusters of GPUs (A100) are applied for the training. 



  §.§.§ Evaluation Dataset

A custom test set is used for the Chinese dialect TTS task. There are 50 distinct Chinese speakers for each Chinese dialect with 2000 clips. Following <cit.>, sentences are randomly chosen from each speaker, and 5-second to 15-second clips are randomly chosen as prompts from the same speaker's speech <cit.>. Besides, there are 25 male and 25 female Chinese speakers for each Chinese dialect. In the dataset, each Chinese dialect contains 50 common spontaneous words <cit.>. Besides, 2000 clips from DiDiSpeech dataset <cit.> are used for the evaluation of Mandarin speech. 




  §.§.§ Evaluation Metrics

Both objective metrics and subjective metrics are used for the evaluation. In the aspect of objective metrics: word error rate (WER) and speaker similarity (SIM) is evaluated. In the aspect of subjective metrics, comparative mean option scores (CMOS) and mean option score (MOS) are calculated to evaluate the naturalness and quality, respectively.



 §.§ Experimental Results on TTS of Multiple Chinese Dialects

In this section, the proposed Bailing-TTS is evaluated in terms of 1)generation quality 2)generation similarity 3)human-level naturalness. 



  §.§.§ Generation Quality

As shown in Table <ref>, we find that the proposed Bailing-TTS is close to the ground-truth recording. It demonstrates Bailing-TTS can generate high-quality and natural speech for the testing Chinese dialect data. Besides, the performance verifies the effectiveness of the proposed Bailing-TTS with specific network architecture and the corresponding training strategy. In details, for the evaluation of Mandarin speech, Bailing-TTS obtains 1.86 for word error rate (WER), 4.32 for mean option score (MOS). In addition, human speakers obtain 1.35 for word error rate (WER), 4.32 for mean option score (MOS). For the evaluation of dialectal speech, Bailing-TTS obtains 6.37 for word error rate (WER), 4.11 for mean option score (MOS). Besides, human speakers obtain 4.60 for word error rate (WER), 4.24 for mean option score (MOS).














  §.§.§ Human-Level Naturalness

The speech synthesized through Bailing-TTS is also in the comparison with the Ground Truth in Table <ref> for the evaluation of both Mandarin speech and Chinese dialectal speech. Bailing-TTS achieves -0.06 CMOS and -0.08 CMOS compared to the Ground Truth, which demonstrates that our method is comparable on human voice quality. 



  §.§.§ Zero-shot learning

Both objective and subjective evaluation for the zero-shot learning experiments are conducted. In one aspect, clips from custom corpora are used to measure the proposed model's performance on a variety of objective metrics. 2000 clips from custom dataset are used for the evaluation of Mandarin speech. To be noted, the custom clips contains highly expressive Mandarin speech with spontaneous word.

Besides, both reference utterance and corresponding target utterance are collected from the same speaker for each clip. The proposed Bailing-TTS family is designed to generate target text based on a prompt (the reference speech). Therefore, comparison between synthesized speech and ground truth speech from real human is conducted for both the objective and subjective evaluation.

For the comparison with real human speech, the word error rate (WER) and speaker similarity metrics (SIM) are used for objective evaluation. Particularly, Paraformer-zh <cit.> is used as the automatic speech recognition (ASR) for Mandarin, and we train a custom automatic speech recognition (ASR) for multiple Chinese dialects. Besides, comparative mean option scores (CMOS) are used for subjective evaluation. Particularly, first, native Chinese dialectal speakers are represented with a reference speech clip of the target speaker. Second, both the synthesized output of the proposed model and the corresponding targeting human speech are used. Third, evaluation is conducted to rate each given clip with higher speaker similarity and expressiveness to the reference clip on a scale between -3 to +3, where -3 and 3 indicate the least and strongest preference. The results are then collected and averaged over all human evaluators and test sentences. 

The results for both objective and subjective evaluation are reported in Table <ref>. The results demonstrate that the proposed Bailing-TTS model represents an expected performance in zero-shot learning. In details, for the evaluation of Mandarin speech (Zero-shot), Bailing-TTS obtains 3.48 for word error rate (WER), 0.72 for speaker similarity metrics (SIM). 




  §.§.§ Fine-tuning learning

Speaker fine-tuning (SFT) is conducted on the base of pre-trained Bailing-TTS model. For the evaluation of the fine-tuning learning, speech data from 10 speakers are combined for Mandarin fine-tuning learning dataset. Then, the comparison is made between the fine-tuned model (Bailing-TTS fine-tuned) and the base pre-trained model (Bailing-TTS Zero-shot). For the objective evaluation, the word error rate (WER) and speaker similarity metrics (SIM) are used for the objective evaluation. For the subjective evaluation, comparative mean option scores (CMOS) are used. To be noted, 15 seconds of speech clip is randomly sampled as the prompt for each speaker. Similarly, speech data from 10 speakers are combined for Chinese dialectal fine-tuning learning dataset. 

For the evaluation of Mandarin speech, the results of the speaker fine-tuning experiment and the base model are demonstrated in the Table <ref>. The results demonstrate that the fine-tuned models shows better performance in both objective and subjective metrics. In details, for the evaluation of Mandarin speech (Speaker fine-tuned),  Bailing-TTS obtains 2.98 for word error rate (WER), 0.77 for speaker similarity  metrics (SIM) and +0.18 for comparative mean option scores (CMOS). For the evaluation of Chinese dialectal speech (Speaker fine-tuned),  Bailing-TTS obtains 7.43 for word error rate (WER), 0.76 for speaker similarity metrics (SIM) and +0.11 for comparative mean option scores (CMOS). 





  §.§.§ Streaming processing

There are a range of challenges in the application of TTS systems in the real world. User experiences are not satisfactory due to the latency and the first packet delay. The computation cost of both the time and the memory is still large for the deployment on mobile hardware systems. Therefore, initial work on the deployment is conducted. A variety of techniques <cit.> are employed to reduce the inference cost and latency. 

In order to reduce the inference cost and the memory cost of attention layers, first, we apply efficient memory cost methods including grouped-query attention <cit.>, paged attention <cit.>, and flash attention <cit.>. Second, model quantization method is also applied to further reduce the computation cost <cit.>. Similarly, the consistency distillation <cit.> and a modified flow matching algorithm <cit.> are used to reduce the computation cost of the diffusion model. Third, to further reduce the computation cost of diffusion architecture, in one aspect, the consistency distillation <cit.> is applied to reduce the computation cost of the diffusion model, in the other aspect, the flow matching algorithm <cit.> is used to reduce the memory cost.

The initial results demonstrate that comparable performance is achieved after a variety of methods are applied on the deployed model. In details, as shown in Table <ref>, the latency is reduced to 0.13 ×, the RTF is reduced to 0.46 × and WER and SIM remains the same in the comparison with the offline Bailing-TTS model. 





§ APPLICATIONS AND DISCUSSION

We propose a family of Bailing-TTS towards Chinese dialectal speech synthesis with spontaneous words. It has the potential for real-world application. First, the dialectal speech is promising to provide rich experience of chat service in the real-world particularly companion chat service. Second, it's very likely to be beneficial to the facilitation of dialectal culture and cultural applications. 

Initial work on the family of Bailing-TTS is proposed while services such as speech synthesis with emotions, speech with support of other modalities are not well explored. Therefore, the exploration of expressive and emotional generation of Chinese dialectal speech is under-going. We are planing to develop the next version of Bailing-TTS family for the generation of high-quality audio (speech/music) from the input of video and text. Then, the generation of both the high-quality audio together with video will be explored.         
















plain



 Paper Title: TinyChirp: Bird Song Recognition Using TinyML Models on Low-power Wireless Acoustic Sensors.


IEEEexample:BSTcontrol




TinyChirp: Bird Song Recognition Using TinyML Models on Low-power Wireless Acoustic Sensors 
Corresponding author: zhaolan.huang@fu-berlin.de. 



    
    Z. Huang1,
    A. Tousnakhoff1,
    P. Kozyr2,
    R. Rehausen3,
    F. Bießmann25
    R. Lachlan4,
    C. Adjih6,
    E. Baccelli61
    
    1Freie Universität Berlin
    2Berlin University of Applied Sciences
    
    
    
    
    
    3Woks Audio GmbH
    4Royal Holloway University of London
    
    
    
    
    
    5Einstein Center for Digital Future
    6Inria, France
    
    
    
    
    

    

    August 5, 2024
======================================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Monitoring biodiversity at scale is challenging. Detecting and identifying species in fine grained taxonomies requires highly accurate machine learning (ML) methods. Training such models requires large high quality data sets. And deploying these models to low power devices requires novel compression techniques and model architectures. While species classification methods have profited from novel data sets and advances in ML methods, in particular neural networks, deploying these state of the art models to low power devices remains difficult. Here we present a comprehensive empirical comparison of various tinyML neural network architectures and compression techniques for species classification. We focus on the example of bird song detection, more concretely a data set curated for studying the corn bunting bird species. The data set is released along with all code and experiments of this study. In our experiments we compare predictive performance, memory and time complexity of classical spectrogram based methods and recent approaches operating on raw audio signal. Our results demonstrate that our TinyChirp approach can robustly detect individual bird species with precisions over 0.98 and reducing the energy consumption such that a deployment time on a single battery charge can be extended from 2 weeks to almost an entire season of 18 weeks.  



TinyML, Microcontroller, Acoustic Sensors, Machine Learning




§ INTRODUCTION


The typical data pipeline with bio-acoustic research requires the deployment of sensors, in which each node is battery-powered and left in the field to record environmental sound, continuously, for a long period (e.g. for a whole season). Thereafter, the recorded data is manually collected by researchers on-site, from each node, and further analyzed in laboratory. In the case of avian species for instance, only the targeted bird species is relevant within the recorded data, and the rest of the data is to be discarded ultimately. This is both a cumbersome process and a waste of resources, not only at this downstream stage in the lab, but also upstream in the field, on each node, while recording large amounts of irrelevant data. In this pipeline, continuous recording of audio thus creates a bottleneck in terms of memory and energy budgets available on individual sensors – typically limited to a small battery, and an SD card, respectively, driven by rudimentary software running on a low-power microcontroller. Such hardware is very energy-efficient, but very limited in memory resources, with RAM memory budgets in the order of 500 kiloBytes <cit.>, which yields specific constraints on software embedded on such devices <cit.>.

Meanwhile, as AI achieves excellent performance in pattern recognition of audio and biosignals, more and more bioacoustic researchers leverage ML  related methods to improve accuracy and efficiency. 
However, until recently, such ML models, e.g. BirdNet <cit.> were confined to lab use only, as their resource requirements (GigaBytes of RAM) are way beyond the capacity of microcontroller-based hardware available on sensors in the field. However, recent advances in TinyML, a lively field of research targeting machine learning for microcontrollers, offer a glimpse of hope that pattern recognition of bioacoustic signals might become possible, over longer periods, as required by the aforementioned use cases. 






However, ..., which required to:
[inline]to be finish

    
  * train ML models.
    
  * deploy models on sensor node.
    
  * evaluate the resource consumption of models on boards.


This paper thus introduces ....


Paper Contributions. In this paper, we explore the possibility of using TinyML in practice, on common low-power microcontroller hardware, for a concrete use case: monitoring corn bunting birds' songs, in a rural area in the UK, over several months in a row, using a fleet of energy-efficient acoustic sensors. We aim to answer the following questions: (Q1) Can we pre-screen the audio on the low-power sensor node, so as to only store the targeted bird songs on each sensor? (Q2) How does that extend the device's lifetime, in terms of memory and energy budgets? More in detail, our contributions are as follows:


    
  * We propose a pipeline to record, recognize and store specific bird songs on low-power microcontroller-based devices, which can be further scale-up to universal species;
    
  * We developed a neural network with high accuracy on bird song recognition. Furthermore, we optimize the network with partial convolution technology to minimize the memory consumption on deployment;
    
  * We propose a two-stage classification approach to reduce computational and storage cost and enhance the overall accuracy;
    
  * We provide experimental results on both the classification performance of models and resource consumption on low-power devices;
    
  * We publish a new data set of curated audio recordings snippets of corn bunting bird songs;
    
  * We provide open source code[see <https://github.com/TinyPART/TinyBirdSounds/>] to reproduce the results on common MCU.



§ BACKGROUND & RELATED WORK




Bird Song Recognition –
Bird song detection has significantly advanced through various machine learning techniques, particularly deep learning methods. Recent research has explored different strategies for recognizing bird songs, with substantial progress in preprocessing audio data for recognition.
One prevalent preprocessing technique involves transforming the audio signal into a spectrogram using STFT <cit.>. This spectrogram can subsequently be converted into a mel spectrogram <cit.>,  by applying a mel scale, followed by a log scale transformation <cit.>. This process effectively redefines the audio recognition task as an image recognition task. CNN <cit.> and Residual Neural Networks (ResNets) <cit.> are widely utilized for such image recognition tasks in bird song detection. For instance, one study employed CNNs to recognize the songs of 17 bird species <cit.>, while another study utilized ResNets to classify the songs of 19 bird species <cit.>. BirdNET <cit.>, a well-known neural network for bird song recognition, is derived from the family of residual networks and can recognize 6,000 of the world’s most common species at the time of writing.

Another approach involves transforming the audio signal into Mel-Frequency Cepstral Coefficients (MFCCs), which are then used as input for bird song classification <cit.>. Alternatively, feature extraction using wavelet decomposition can be employed for this purpose <cit.>. Additionally, certain studies combine different architectures; for example, CNN and Transformer <cit.>, and CNN and VGG <cit.> to enhance performance. While these methods are effective for bird song recognition, their computational requirements are typically too demanding for execution on microcontrollers.

Bioacoustic Audio Datasets –
High-quality and diverse datasets are crucial for training models to accurately recognize birds and differentiate between species by their unique songs. The Xeno-canto database [see <https://www.xeno-canto.org>] is one of the largest and most comprehensive collections of bird sounds, containing over 500,000 recordings from more than 10,000 species. In <cit.>, 2,699 bird song audio recordings from the Xeno-canto database, ranging in length from 20 to 0.74 seconds, were used to classify 17 bird species.
Another extensive collection of bird audio recordings is available in the Macaulay Library [see <https://www.macaulaylibrary.org/>], renowned for its high-quality bird audio recordings and detailed metadata, which includes information about species, location, and context. The combination of the Macaulay Library and Xeno-canto audio recordings was used to train BirdNet <cit.>.
In <cit.>, the public dataset CLO-43DS, containing recordings of 43 different North American wood-warblers, was utilized. Another notable collection is the Birdsdata dataset with audio files of 20 bird species from the Beijing Academy of Artificial Intelligence (BAAI) repository, which is frequently used in research and model training <cit.>.
To complement datasets focused on bird species, the Google AudioSet <cit.> and Urbansound8K <cit.> datasets are used to include non-bird species sounds.



TinyML –
For models to operate on low-power devices, they must be compact and computationally efficient. Studies have demonstrated the use of lightweight CNNs for speech recognition and age classification <cit.>, water leakage detection <cit.>, acoustic defect localization <cit.>, fall detection for the elderly <cit.> and other tasks <cit.>. Tiny vision transformers have also been employed for classification tasks in various studies <cit.>. Moreover, quantization techniques can be applied to models to reduce their size to fit within the constraints of low-power devices <cit.>.

Creating spectrograms from audio signals can be power-intensive; hence, some studies use raw time-series data as input for neural networks <cit.>. For instance, single-channel EEG was used as input for a model based on CNN and Transformer for sleep stage classification <cit.>. Raw audio signals were used for urban sound analysis <cit.>. Time-series data offer advantages for TinyML applications, including reduced computational load and suitability for low-power devices.


Embedded Software Platforms for TinyML – The widely used model transpiler TVM (Tensor Virtual Machine <cit.>) has recently been extended with uTVM, providing automated transpilation and compilation for models output by major ML frameworks (TFML, Pytorch, etc.). As such uTVM exposes low-level routines and  optimizes these for execution on different processing units, including for targets such as a large variety of microcontrollers. Prior works such as <cit.>, <cit.> or MLPerfTiny <cit.> focused on the production, performance and analysis of standard benchmark suites of representative TinyML tasks on different microcontrollers. Conversely, prior work such as U-TOE <cit.> or RIOT-ML <cit.> provide embedded operating system integration of TinyML, facilitating TinyML benchmarking and continuous deployment over low-power wireless network links such as IEEE 802.15.4 or BLE.



§ BIOACOUSTIC MONITORING SCENARIO



As depicted in Figure <ref>, a network of battery-powered, autonomous recording units (ARUs, i.e. microcontroller-based acoustic sensors) is deployed across a monitored area. An example of ARU is given in <cit.>.

These sensors remain in the field for an entire season, typically around six months. Locations are often remote and hard to access and devices may be distributed over a relatively wide area, making visiting them time-consuming. Birds typically do not sing consistently during the whole day, while each bird has several different song types that need to be regularly sampled. Moreover, each individual bird moves around its individual territory, singing from a range of song posts, not all of which will be adequately recorded from any one location. 

On the network aspect, the sensors are distributed approximately evenly, ensuring that each sensor can be wireless connected by more than two neighboring sensors. This arrangement facilitates approximate triangulation through distance estimation based on signal strength. The placement process can be streamlined using LEDs and a basic ultra-low-power wireless protocol, such as IEEE 802.15.4 or LoRa. For space reason, we do not detail further networking aspects in the paper, but rather focus on the on-device machine learning aspects. We nevertheless keep an eye on the total memory footprint on the device, to ensure that the operating system including application TinyML code, the OS and a typical wireless low-power network stack fit in typical resource budgets on common microcontroller-based boards.

Although our work is designed around one very specific real-world research scenario (monitoring common Corn Bunting bird songs) the architectures should be readily applicable to other audio data classification tasks. Examples of related scenarios to which our method could be extended include, surveying remote areas for targeted endangered species of conservation value, or broader surveys of songbird biodiversity.  Similar devices would even be useful in the search for very elusive species such as Ivory-billed Woodpecker <cit.>. 




 §.§ Corn Bunting Monitoring Use-Case




SongBeam <cit.> microcontroller-based recorders have been used to monitor Corn Bunting (Emberiza calandra) birds in Oxfordshire, UK. 
A set of 30-40 devices have been deployed since 2022, currently comprising 3 complete breeding seasons (February - July).

SongBeam devices are based on an ARM Cortex-M microcontroller, run on 4 D-cell batteries and record 4-channel WAV files onto 128  microSD cards. To maintain such a fleet of sensors, devices are currently checked approximately every 2 weeks, especially because memory space is soon exhausted on the microSD cards. 

The above deployment has been used to produce a significant part of the dataset we present in Section <ref>.
Our experience using SongBeam and a preliminary analysis of the raw dataset we produced shows that less than 10% of recording time contains useful recordings. 
Given the extended deployment period (6 months), remaining low-power while improving storage efficiency is thus paramount. 
Moreover, current prototypes use solar power, further emphasizing that the bottleneck is downloading data from the microSD cards to free space on-board. 

This description of the limitations of using SongBeam recorders to monitor Corn Buntings is likely to apply to many biomonitoring scenarios: solar power may allow for extending deployment lifetime in remote locations, but storage of recordings is a limiting factor.






Describe Robert's scenario.
A bunch of battery-powered, microcontroller-based acoustic sensors are disseminated across an area to be monitored. The sensors are left in the field for a whole season at a time ( 6months). Power and storage efficiency are paramount.
To emphasis the network aspect, we can assume that the sensors are placed approx. evenly so that each sensor hears N>2 other sensors beacons, so that approx. triangulation is possible via (signal strength) estimated distance. Placement can be simplified via LED and a basic ultra low-power wireless protocol such as IEEE 802.15.4 or LoRa.


§ EVALUATION METRICS



On the one hand, as the basic part of bird song recording, the model should identify the target bird song as much as possible, potentially requiring a complex structure with a large size. On the other hand, the limited resource budget allows only simple models to be deployed on low-power IoT devices.

Thus we considered two orthogonal types of metrics to evaluate the prediction performance on bird song classification and resource usage on low-power IoT devices of TinyML models. Preliminary results are presented in Section <ref>.




 §.§ On-device Resource Usage


We performed extensive profiling of all models on low-power devices to capture their effective memory and time complexity in a real-world application scenario. The metrics are helpful to investigate the energy efficiency of the models, complementing the commonly used metrics for predictive performance. 

Memory (RAM) Consumption –
This metric measures the amount of dynamic memory space (primary RAM) consumed by the model during inference. It reflects the memory footprint of the model activation and is important for low-power devices that have limited memory resources. Efficient memory utilization allows for the deployment of larger and more complex models on such devices.

Storage (Flash memory) Consumption –
This metric quantifies the amount of storage space, typically in terms of Flash memory region, required to store the compute instruction and associated parameters. 

Minimizing storage consumption allows for accommodating multiple models on the device or orchestrating with other essential applications.

Computational Latency –
This metric measures the time consumption of performing inference for each input sample at the model level. It reflects the inference speed of the model on the low-power device and plays a crucial role in real-time or latency-sensitive applications. Core clock frequency, cache strategies and communication latency between memory and working core have a great impact on this indicator.

Energy Consumption – 
This metric is crucial for battery-operated and resource-constrained devices, where efficient energy usage can significantly impact device longevity and performance. Energy consumption encompasses both active power (when the device is triggered to perform tasks) and idle power (when the device is in sleep mode).



 §.§ On-device Prediction Performance



Accuracy –
It is defined as the ratio of correctly predicted instances to the total instances in the dataset. While accuracy is intuitive, it can be misleading, especially in highly imbalanced datasets. 

To address this limitation, accuracy should be combined with other metrics that offer a more detailed view of model performance.

Precision and Recall –
They are two fundamental metrics of classification models used to evaluate the ability to distinguish true positives (TP) from false positives (FP) and false negatives (FN), particularly in the context of imbalanced datasets. 


F-Score –
There is often a trade-off between precision and recall; increasing one can lead to a decrease in the other. 

As two common instances, F_1-score weighs them evenly, while F_2-score treats recall as two times more important than precision, applying in scenarios where false positives are more tolerant than false negatives.

ROC Curve –
While the above metrics are sensitive to class imbalance – which occurs in many species classification and detection tasks – there are other metrics that are more robust towards class imbalance. The receiver-operator characteristic (ROC) curve depicts the  TPR and FPR across different decision thresholds.



The AUC is a key metric derived from the ROC curve that quantifies the overall performance of the model and is invariant with respect to class imbalance. An AUC of 1.0 indicates perfect classification, while an AUC of 0.5 suggests no discriminative ability.  


move to discussion part
    - Confusion matrix
    - ROC Curve -> different cut-off value





§ METHODOLOGY


We used a combination of project-specific and publicly accessible data to train and validate our classification models. In the data pre-processing phase, the raw audio signals were further segmented, labeled, down-sampled and divided into different groups to generate classification datasets. The corresponding Mel-spectrograms of the pre-processed audio segments were also created for spectral-based methods (models). During the pre-processing stage, we conducted a pilot analysis to establish guidelines and determine specific hyper-parameters for downsampling and spectrogram generation.




 §.§ Data Acquisition


The publicly accessible data originated from Xeno-Canto, Macaulay, Google AudioSet, while the project-specific data were previously collected in a long-term research of bird song patterns. We published the resulting dataset[see <https://www.kaggle.com/datasets/zhaolanhuang/tinychirp>].



    
  * Oxfordshire Corn Buntings. This library contains recordings of corn buntings along a transect of approximately 20km in Southern England. Corn buntings sing in a mosaic-like pattern of geographical variation called dialects; our sample contains approximately 6 different dialects. The recordings were performed with directional parabolic microphones as described in <cit.>.  
    
    
    
  * Macaulay Library. This library contains the necessary audio recordings for our target species as well as the other identified species. There are a total of 796 entries of audio recordings for the target species. We filtered the audio files by the song tag and removed 16 entries due to unclear metadata and missing catalog numbers. For other species as non-target, we limit the selection to a maximum of 30 recordings per species and choose those with the highest average community rating, acquiring a total of 1468 recordings.
    
    
  * Xeno-Canto. This library also contains recordings of target species and other species. We retrieved the recordings only marked with song and rated with the best quality level, resulting in 303 recordings of target species and 9622 recordings of other species as non-target.
    
    
  * Google AudioSet. This dataset provides environmental sounds which contain non-bird sounds. 


After gathering all audio recordings from the above libraries, we used BirdNET to identify and chop corn-bunting segments from all datasets, and labeled them as target bird songs; all segments were truncated or zero-padded to the length of 3 seconds. Meanwhile, we chose other 3-s segments randomly and ensured no overlap with corn-bunting segments, with labeled as non-target bird songs. Table <ref> shows the distribution of target and non-target segments over all datasets.



 §.§ Data Pre-processing



This phase contains the following steps:


    
  * We divided the segments into training, validation and test sets with the ratio of 80:10:10.

    
  * All segments were downsampled to 16  using zero-order holder, which constituted the audio dataset.
    
    
  * We transformed the audio segments into Mel-Spectrograms, and grouped them with the same splitting ratio to form the spectral dataset.



Pilot Analysis –

We averaged the STFT spectrograms of target and non-target segments in the training dataset to investigate their frequency characteristics, as depicted in Figure <ref>. Obviously, a bright band lays on the spectrogram of target segments roughly between 4000 and 8000 , hints that a sample rate with 16  should be sufficient to preserve all frequency components of corn-bunting song according to Nyquist–Shannon theorem. A highpass filter can also be applied to eliminate low-frequency noise without damaging target songs. Thus, we downsampled the segments from 48  to 16  to reduce the resource consumption and improve the efficiency of classification phase. In the baseline model we also designed a highpass filter to enhance SNR.




Mel-Spectrogram –

Mel-spectrogram is commonly used in audio classification tasks due to its ability to closely approximate human auditory perception. To create log Mel-spectrograms, we first generated magnitude spectrograms of STFT for all downsampled audio segments. The STFT was performed with a Hann window with a width of 1024 samples and a step size of 256 samples. This process generated 184 window frames and 513 frequency bins. Thereafter, the magnitude spectrograms were mapped onto the Mel scales with 80 Mel bins. In creating the Mel-spectrograms, we employed a sampling rate of 16  and focused on the frequency range of 80-8,000 Hz. This frequency range was chosen because it encompasses the typical frequency range of most bird songs, including that of the target Corn Bunting. Lastly, we converted the Mel-spectrograms into a logarithmic scale to obtain log Mel-spectrograms, with the shape of 184 × 80. 









| Data         | # Bird | # Non-Bird |
| ————— | —— | ———- |
| Roberts         |        |            |
| Xeon-Canto      |        |            |
| Macaulay        |        |            |
| Google AudioSet |        |            |


§ BASELINE & DECISION STRATEGY


We propose an approach to optimizing the screening process by combining a high-speed signal processing step (hereafter named "baseline") with the precision of an TinyML model, ensuring reliable results while maintaining resource-consumption efficiency.



 §.§ Baseline


We developed a lightweight pre-selector signal processing step, not relying on machine learning. First, the audio segments are normalized by min-max schema. Thereafter, a 9th-order highpass Butterworth filter with the cut-off frequency of 7000  is imposed on the normalized segments, in order to suppress noise and non-relative sound in low-frequency and to preserve only target components. Finally, we calculate the signal power P of the filtered segments,


    P = 1/N∑_n=1^N x(n)^2,

where x(n) and N denote the data points and the length of a segment, respectively. 
We define two thresholds: t_low and t_high. If P is smaller than t_low, the microcontroller remains in the idle/low-power state, and the audio sample is discarded directly. Else, P is compared to t_high. Depending on this result, and on the decision strategy (see below), the audio sample is either stored or further analyzed via inference using the machine learning model.





 §.§ TinyChirp Decision Strategy





We employ a double-phase approach for enhanced accuracy and efficiency of bird song recognition, combining the baseline with a TinyML model for further verification. The process is as follows:

    
  * Step 1: Baseline Filtering – Pre-screening is conducted using the baseline pre-processing step described in Section <ref>, designed to provide a quick and efficient preliminary analysis. If P is smaller than t_low, the microcontroller remains in the idle/low-power state, the audio sample is discarded and the process aborts. Else proceed to Step 2.
    


 

    
  * Step 2: Inference-based Classification – If the power-saving flag is set and if P is smaller than t_high a higher-accuracy classification is conducted via inference  using TinyML model. If the audio sample is not classified as the target, the audio sample is discarded and the process aborts. Else proceed to Step 3.


  * Step 3: Storage – Without further processing, the audio sample is logged, i.e. stored on the SD card. 

    


Note that the decision strategy can therefore be configured with two "knobs". On one hand the power-saving flag determines skipping (or not) some of the computation and can thus decrease energy consumption. On the other hand, different TinyML models can be inserted in Step 2.



Next, we thus study two families of TinyML models for Step 2: the first category of model takes a Mel-Spectrogram as input, while the second category of model takes time-series as input. Table <ref> presents a summary of the explored TinyML model structures. 



This study mainly investigated the performance of two types of ML models – CNN and Transformer – on time-series and Mel-spectral input. Due to the compute- and memory-intensive characteristic of ML methods, two major optimization strategies were leveraged to compress the models and improve their efficiency, as discussed in Section <ref>. 

This study also presents a baseline model built with classical digital signal processing methods, as performance reference of ML-based models. Furthermore, we integrated baseline and ML-based model into ensemble one to gain a optimal compromise between accuracy and efficiency resource consumption? on low-power devices. 






§ CLASSIFICATION BASED ON SPECTROGRAMS



In contrast to the computer vision community, where the transition from frequency decomposition based feature extraction to neural feature extraction on raw image data introduced in 2012 led to a significant increase in predictive performance <cit.>, neural feature extraction on raw audio time series did not contribute to a comparable breakthrough yet. Instead most ML techniques for audio pattern recognition (somewhat surprisingly) were relying on image processing pipelines – sometimes with neural networks trained on ImageNet data. Basically, image classification is performed, using a neural network, analyzing the spectrogram rendering of the audio trace.

For this reason, we initially consider the below two models, based on state-of-the-art neural network architectures.

CNN-Mel –
This model contains two 2D convolutional layers for Mel-spectral inputs as feature extraction and two fully connected layers as classifier. Both convolutional layers utilize 4 filters with a 3 × 3 kernel size and ReLU activation function, followed by 2 × 2 max pooling to reduce spatial dimensions. The output of the classifier are normalized by Softmax function as well.

SqueezeNet-Mel – 
This model is based on SqueezeNet <cit.>, an advanced backbone focused on optimizing the efficiency of computer vision applications by strategically reducing parameters, with comparable performance to AlexNet <cit.>. It leverages a so-called Fire module to achieve higher efficiency compared to standard convolutions with only a slight decrease in accuracy. The Fire module significantly reduces the number of parameters by the squeeze layer (1×1 convolutions), which exceedingly reduces the number of channels before the more parameter-heavy 3×3 convolutions in the expand layer. This reduction in intermediate channels means fewer parameters overall.
We aligned its input shape with the Mel-spectrogram and tailored the output for binary classification. Also offering potential for further compression using quantization techniques, SqueezeNet is thus appealing given memory constraints.





Limitations of Spectrograms on Microcontrollers
Practical experience on low-power microcontroller-based devices has shown however that producing and manipulating mel-spectrograms from audio signal streams on such devices is problematic. It inserts an additional step in the processing pipeline, which increases latency and memory requirements. Prior work (such as <cit.>, section 6.2) concludes that the CPU bottleneck is not just the model inference time, but also the spectrogram calculation, which, compared to inference alone, almost doubles latency. Other previous work such as <cit.> measures on a quite powerful STM32F7 microcontroller that computing and writing in memory the mel-spectrogram of 30 columns and 40 frequency bands takes approximately 1 second.  Note that these spectrogram dimensions are much smaller than our requirements (184 columns × 80 frequency bands) hence even more latency can be expected in our case.

For these reasons, we next explore pipelines which skip the spectral pre-processing stage as described below.



SqueezeNet<cit.> is a CNN characterized by comparable performance to AlexNet while using approximately 50 times fewer parameters. The compact architecture of SqueezeNet is achieved by employing eight unique building blocks called the Fire Modules. Each Fire Module starts with a "squeeze" convolution using 1x1 filters to reduce input channel dimensions, thereby cutting down on computational load. This initial step compresses data into fewer channels. Following this, an "expand" convolution layer mixes 1x1 and 3x3 filters: the former expands channel numbers, while the latter captures spatial details. Moreover, SqueezeNet offers the potential for further compression using quantization techniques, making it an efficient choice given memory constraints.

The implemented SqueezeNet-Time-Series architecture uses a reduced number of feature maps in compare to initial SqueezeNet. This compressed model has in the first convolutional layer 0.3 times the number of filters that initial model has in this layer. In the fire modules in convolutional layers, we use the integer value of 0.3 times the number of squeeze filters given to this layer in initial model. Since the number of expand filters depends on the number of squeeze filters and is four times greater, it also undergoes compression. The structure of the fire module in SqueezeNet-Time-Series is presented in Table 2. Therefore, if the initial number of squeeze filters in the fire module is x, the compressed number of squeeze filters in this fire module will be int(0.3× x). Then both the expand layers will have 4× int(0.3× x) filters.

Additionally, since the initial model uses 2D convolutions and time-series are 1D arrays, we converted all 2D convolutional layers to 1D convolutional layers. The model was trained using time-series data for 10 epochs with the Adam optimizer and cross-entropy loss function. After training, this model was further quantized using different post-training quantization techniques.





For spectrogram approach we aimed to create an ultra-compact CNN-Mel-Spec model, comprising just two convolutional layers and two fully connected layers. The input for the model is a log Mel spectrogram with a size of 184x80. Both convolutional layers utilized 4 filters with a 3x3 kernel size and ReLU activation function, followed by max pooling with a 2x2 pooling window to reduce spatial dimensions. After the last convolutional layer and max pooling, the data were reshaped into a one-dimensional vector for input into the subsequent fully connected layer. The first fully connected layer had 8 neurons and utilized ReLU activation. The final fully connected layer contained two neurons with a linear activation function. For optimization, we employed the Adam optimizer with a learning rate of 0.001, and the model was trained over 15 epochs, using cross-entropy as the loss function. In an effort to reduce model size, we converted it to TFLite format and applied several post-training quantization techniques: float16 quantization, dynamic range quantization, and full integer quantization.













§ CLASSIFICATION BASED ON TIME-SERIES


Contrary to the models described in the previous section and inspired by the success of neural feature extraction in the computer vision domain <cit.>, the pipelines we aim at next take directly the raw audio signal time-series as input. More precisely, we designed the three models described below.

CNN-Time –
This simple model performs feature extraction with two sequential (1D) temporal convolutional layers followed by average pooling. A max pooling is inserted in-between to reduce the dimension of feature maps and enhance the non-linearity of the network. Two fully connected layers act as classifier at the end of the network, with the probability outputs normalized by Softmax function.  

Transformer-Time –
Inspired by <cit.>, we designed an efficient, attention-based model with only one temporal convolutional layer and one single-head transformer. The convolutional and pooling layers serve as feature extraction to transfer raw audio signal into embeddings for the transformer. The activations of the fully connected layer are normalized by Softmax function as well.



SqueezeNet-Time –
This model is based on SqueezeNet <cit.>. We tailored this basis to align the input shape with the audio segment, and narrowed the output channels for binary classification. To adapt the backbone to time-series input, all 2D convolutional layers are replaced by temporal convolutional layers. Furthermore, to have a more compact structure, we decreased the filter number of all convolutional layers by roughly 70%, as presented in Table <ref>.





§ ADDITIONAL MODEL OPTIMIZATIONS FOR TINYML


In this study, we used two optimization techniques to compress the models and further reduce their memory consumption: model quantization on the one hand and on the other hand partial convolution, as described below.







 §.§ Model Quantization

Quantization reduces the model size and inference time by converting the weights and activations from higher precision (e.g., 32-bit floating-point) to lower precision (e.g., 8-bit integer) <cit.>. In this study we adopted PTQ with weights and activations quantized in 8-bit integer. To avoid substantial loss of prediction performance, the training dataset was used to find the optimal scale factor and zero-point of the activations.



 §.§ Partial Convolution

We observed that the outputs of the Conv1D layer require significant memory, which impedes deployment on resource-constrained tiny devices. As presented in Table <ref>, the peak memory consumption occurs at the first Conv1D layer both in CNN-Time and Transformer-Time with 768  and 3 , respectively. This indicates that deployment on resource-constrained tiny devices is impractical.

To address this issue and inspired by <cit.>, 

we exploited the fact that average pooling can be computed iteratively point-by-point on the output channels of the final Conv1D layer over a small sliding window of inputs. Each point in the output channels depends only on a small subset (kernel window) of the outputs from the previous layer. That is, for a block of L Conv1D layers following by an average pooling layer, its output y can be iteratively computed on the input sequence x(n), n= 1 … N as following:

    y_j(k)    = y_j(k-1) + 1/N_L A^L_j(k), 
    
        k    = 1 … N_L, j = 1 … C_L . 
     
        A^l_c(n)    = ∑_i=1^C_l-1 W^l_c(i) · [ A^l-1_i(n-K^l/2) ⋯
       =4A^l-1_i(n) ⋯ A^l-1_i(n+K^l/2) ]^T 
    
        A^0(n)    = x(n) , 
    
        l    = 1 … L, c = 1 … C_l, n = 1 … N_l, N_0 = N

where y_j(N_L) is the result of the average pooling of the j-th channel; A^l_c(n) denotes the n-th point in channel c of the l-th Conv1D layer calculated with kernel weights W. Each Conv1D layer contains C_l channels with filter size of K^l and output size of N_l. Without loss of generality, the formula of partial convolution contains only the linear components and considers only one-dimension case for the sake of simplicity; it can be generalized in high-dimension and integrated with non-linear building blocks (e.g., stride, non-linear activations, pooing layers, etc.).

Figure <ref> provides a comparison between classic convolution and partial convolution. Unlike classic convolution where entire channels (C × N) are computed and stored before being processed by the next layer, partial convolution requires only a small part of the channels (C × K, K << N) for each layer. In our case with K=3 and N=48000, it can be expected roughly 16000 × smaller memory consumption, making the implementation more suitable for tiny devices. We applied this strategy to the Conv1D layers combined with average pooling in the CNN-Time and Transformer-Time architectures.










§ TINYCHIRP PERFORMANCE EVALUATION


In the following, we conducted a comprehensive evaluation of the discriminative performance of TinyChirp with different TinyML models, as well as of resource consumption and computation time on typical low-power boards based on microcontrollers.



 §.§ Classification Performance Comparison







Our evaluation process began with the generation of ROC curves and the corresponding AUC on the training dataset, providing a clear picture of the models' abilities to distinguish between classes. As shown in Figure <ref>, the spectrogram-based models – CNN-Mel and SqueezeNet-Mel – achieved the best classification performance with AUCs of 1.0 and 0.99, respectively, followed by time-series models – CNN-Time and Transformer-Time – both with 0.98 AUC. Unexpectedly, with a more complex structure, SqueezeNet-Time performs worst (lower AUC than the baseline).


We next measured accuracy, precision, recall, F_1- and F_2-score, shown in Figure <ref>. Spectrogram-based models achieve the highest metrics for the whole range of threshold values. For time-series models, Transformer-Time worked overall better than CNN-Time; Again, SqueezeNet-Time performs worst (consistent with the above ROC analysis). 


Next, we focused on finding the threshold settings for optimal  F_2-score prioritizing recall, i.e. reducing the likelihood of mistakenly discarding records of the targeted birds). The results are shown in Table <ref> for training data, thereafter verified on test (previously unseen) data as shown in Table <ref>. Note that this threshold influences energy consumption: more target bird classification leads to more energy consumption (as data needs to we stored on to SD card in this case).

These results are promising and allowing to focus only on the relevant audio segments (10% of the total recording time), as the classification performance is compelling. Thus, TinyChirp can save 90% of SD card space compared to the initial monitoring scenario (Section <ref>), potentially extending the deployment time to 18 weeks. However, complementary experiments must now be carried out on targeted hardware, so as to evaluate other key metrics: computation time, energy consumption and memory footprint on typical microcontroller-based devices. The next sections focus on that part.




presents the optimized setting with corresponding metrics on the training dataset. This approach ensured that our models were fine-tuned to prioritize recall, thereby reducing the likelihood of missing critical positive cases. Also we emphasize that the decision threshold is directly related to energy consumption in the use case considered here: more positive predictions will lead to higher energy consumption. Hence high precision is desirable for low energy consumption, but recall should be traded with caution against energy consumption.

After determining the optimal threshold setting on the training dataset, we validated this threshold using the test dataset, as described in Table <ref>. This step was crucial to ensure that the threshold not only performed well on the training data but also generalized effectively to unseen data. By confirming the optimal threshold on the test dataset, we demonstrated the robustness and reliability of our TinyML models' predictive performance.













 §.§ Performance Evaluation on Microcontrollers


Experimental Setup – For our measurements, we used a common low-power off-the-shelf board: the nRF52840 Development kit (nrf52840dk). This board is based on an ARM Cortex-M4 processor, with a typical memory budget: 1  of Flash memory and 256  of RAM. As embedded software base we used RIOT-ML <cit.>. 

Throughout the experiments, we monitored the (RAM/Flash) resource footprint using built-in diagnostic tools provided by RIOT-ML and RIOT <cit.>, as well as external measurement equipment as described below.

Energy consumption was measured using an ampermeter to gauge the board's energy efficiency. This involved recording the current draw of the MCU and the external storage system (SD card) in different stages. We used a voltage regulator to supply 3.3  DC output and a logger to capture detailed current profiles over time.

Scope of the Measurements – Besides the model inference stage, we also considered the resource footprint of pre-processing during bird song recognition. Pre-processing refers to down-sampling and calculation of Mel-spectrogram as described in Section <ref>.

Also, the energy consumption in MCU's lowest power mode (idle stage) was measured as the reference of absence of sound with sufficient intensity.



Measurement Results on Microcontrollers – The first striking observation is that  SqueezeNet-Time and SqueezeNet-Mel were not deployable due to OOM (exceded memory budget) and were thus excluded in this experiment. All other models fit in the constraints of memory and storage budget, even when discounting the memory footprint of the OS and the network stack for wireless communication. 

The resource footprint of different stages and models measured on an nrf52840dk board are shown in Table <ref>.  We observe that the model inference stage takes the bulk of the overall energy consumption, as it requires the most process time and MCU in full power mode. 
As expected, the idle stage requires the least power, as all peripherals and the MCU are in the lowest power (standby) mode (until woken up by sound with sufficient amplitude). 











Spectrogram vs Time-series – Considering only the prediction performance in Section <ref>, it would seem natural to choose CNN-Mel as the best candidate, since it also provides the best inference latency. However, taking pre-processing time into account (which for CNN-Mel includes producing a spectrogram) shows a different picture. It takes roughly 2.4 seconds in total for a 3 second audio snippet, which is dangerously near the real-time constraint and costs high power consumption. With the best prediction performance and the lowest total compute latency among time-series models, the Transformer-Time now appears like the best choice to deploy on tiny devices.



Next, we consider jointly the overall resource consumption and the classification performance to discuss variants for the decision strategy described in Section <ref>:

Baseline Only –
With the Baseline t_low (Step 1 and Step 3 only) setting, the required computational resources are minimal, while achieving a recall of 0.99; however, 
due to the low precision, many false positives are stored, leading to the lowest storage efficiency.

TinyChirp Skipping Baseline –
With this variant, the TinyML model is applied immediately, without pre-screening by the baseline (Step 2 and Step 3 Only), leading to high predictive performance and high storage efficiency, but requiring higher energy consumption. Even when using the Transformer-Time with the lowest energy footprint, this requires more than 9 times the energy required for the baseline. This variant is ideal for devices currently running out of storage space while still with substantial battery left.

TinyChirp –
This variant refers to the full series of Step 1-3 described in Section <ref>, whereby the samples not discarded by the baseline are double-checked by the TinyML model. This approach offers a good compromise of compute, storage and energy footprint, by discarding early, without impacting too much the overall prediction performance achievable by the TinyML model on its own. 

TinyChirp with Power-saving –
This variant (Step 1-3 with the power-saving flag enabled) provides a more energy-efficient twist to TinyChirp.

However, the overall precision is determined by the threshold setting t_high, which causes an extra 40% of false positives to be stored onto the SD card, leading to lower storage efficiency. This variant seems more suitable for currently running out of battery, but with sill sufficient storage resources.






§ CONCLUSION

In this paper we present an empirical study of a bioacoustic monitoring use case showing that machine learning is usable directly on autonomous recording units based on low-power microcontrollers. We focused on a concrete use-case: monitoring Corn Bunting birds in rural UK. We publish a data set of the recordings, based on which we demonstrated that TinyML can perform on-device high accuracy classification for bird song classification. Compared to the traditional approach used so far, our approach TinyChirp can extend from 2 weeks to 18 weeks – almost a full season – the time intervals until which researchers must harvest overfull data storage embedded on the recording units deployed in the field. Our experiments also observe hands-on how performing inference directly on the audio time-series is the better approach on microcontroller-based hardware, compared to inference on mel-sepctrograms typically used in audio pattern recognition. Beyond the specific use case of monitoring Corn Bunting birds' songs, based on our general-purpose TinyML approach and our open-source code implementation integrated into the RIOT operating system, TinyChirp is applicable to a wide range of similar bioacoustic use cases, using a large variety of low-power microcontroller-based hardware.




§ ACKNOWLEDGMENT

The research leading to these results partly received funding from the MESRI-BMBF German/French cybersecurity program under grant agreements No. ANR-20-CYAL-0005 and 16KIS1395K. We acknowledge the receipt of media from the Macaulay Library at the Cornell Lab of Ornithology and the Xeno-Canto database.  Felix Bießmann received support from the Einstein Center Digital Future, Berlin,
the German Research Foundation (DFG) (528483508 - FIP 12), the Federal Ministry of Economic Affairs and Climate Action (RIWWER 01MD22007H), the German Federal Ministry of the Environment (GCA 67KI2022A) and the German Federal Ministry
of Education and Research (KIP-SDM 16SV8856).


IEEEtran 









 Paper Title: Towards Explainable and Interpretable Musical Difficulty Estimation: A Parameter-efficient Approach.






Substrate stabilization of Jahn–Teller distortion in a single molecule
    Andrea Donarini
    August 5, 2024
======================================================================



§ ABSTRACT 
Estimating music piece difficulty is important for organizing educational music collections. This process could be partially automatized to facilitate the educator's role. Nevertheless, the decisions performed by prevalent deep-learning models are hardly understandable, which may impair the acceptance of such a technology in music education curricula. Our work employs explainable descriptors for difficulty estimation in  symbolic music representations. Furthermore, through a novel parameter-efficient white-box model, we outperform previous efforts while delivering interpretable results. These comprehensible outcomes emulate the functionality of a rubric, a tool widely used in music education.
Our approach, evaluated in piano repertoire categorized in 9 classes, achieved  
41.4% accuracy independently, with a mean squared error (MSE) of 1.7, showing precise difficulty estimation. 
Through our baseline, we
illustrate how building on top of past research can  offer alternatives for music difficulty assessment which are explainable and interpretable. With this, we aim to promote a more effective communication between the Music Information Retrieval (MIR) community and the music education one.








§ INTRODUCTION



Estimating the difficulty of music pieces aids in 
organizing large collections for music education purposes. 
However, manually assigning difficulty levels is laborious and might lead to
subjective errors <cit.>. To address this, 
Music Information Retrieval (MIR) research has focused on automating this process for piano works represented in 
various modalities <cit.> as well as repertoires from 
other instruments <cit.>. Furthermore, the interest of companies like Muse Group <cit.> and Yousician <cit.> highlights the industry's recognition of the importance of the task.

Previous work in this field has mainly focused on processing machine-readable symbolic  scores <cit.>.  
These, unlike acoustic features extracted from audio whose understanding depends on signal processing knowledge, are both analyzable by computers and interpretable by humans. Musicians find also easier to understand symbolic features since 
based on music theory knowledge. Initial 
works towards interpretable difficulty assessment focused on visualization 
<cit.>, with Chiu and Chen
 <cit.> making the first attempt to classify difficulty in the piano repertoire with 
explainable descriptors. 




Still, the continually increasing trend towards deep-learning based solutions <cit.>, whose lack of transparency  limits  users'  understanding and therefore leads to an  eventual non-acceptance  in real life applications <cit.>, can impair a fruitful implementation of such technologies in  music educational  practices.





With this background, we propose a white-box <cit.> model (cf. Figure <ref>), 
which 
through the concept of a 

rubric, 

an evaluation instrument from 
music education  used  to support objective assessment  <cit.>,  allows a transparent interpretation of music difficulty. From this point forward, the white-box model will be denoted by . 
Furthermore, to gain a profound understanding of what 
music difficulty means from an explainable perspective, we build upon the 
descriptors  of  
Chiu and Chen
 <cit.> by proposing a new one focusing on music repetitive patterns. 


We also provide
an interactive companion page[At: <https://pramoneda.github.io/rubricnet>] to visualize 
the evaluated 
data and 
scrutinize the results 
in light of its 
interpretability from a musical point of view. 







Through 
eXplainable Artificial Intelligence (XAI), we aim to contribute to music education by 
facilitating the understanding of measurable 
factors that determine a piece's difficulty. 
Our work builds on methods from 
music education, 
where 
objectively assessing  abstract competences through measuring concrete criteria 
is consolidated by employing rubrics, tools which, unlike a black-box, break down complex concepts into simpler ones <cit.>.







Our interpretable methodology aims to bridge the gap between computational models and practical music education needs, enabling educators to make facilitated, but also informed decisions about curriculum development based on the difficulty levels of pieces. 

We release all the code and models of this research[At: <https://github.com/pramoneda/rubricnet>], in order to offer a baseline
for further research in music difficulty assessment.









§ RELATED WORKS


Previous research aiming to  automatically assess  the difficulty of piano repertoire
examined the link between 
fingering patterns and the pieces'
difficulty level <cit.>. 
Recent studies <cit.> have also made significant contributions. 
In <cit.>, representations are used to feed three deep learning models—covering music notation, physical gestures, and expressiveness—to emulate Cook's dimensions <cit.>. These models' predictions are merged using an ensemble method to estimate the scores' difficulty. 
While we appreciate their musicology-inspired approach, its lack of interpretability harms its usability.






Difficulty estimation of 
piano pieces has also been investigated through 
hybrid methods that merge features with deep learning models <cit.>. However, the absence of publicly shared data and code complicates performing comparative analyses these works. 
In <cit.>, the authors  combine the methods from Chiu and Chen
 <cit.> with deep learning models trained using piano roll as input. 
In a similar vein, <cit.> 
uses JSymbolic features <cit.> and deep learning models on a proprietary dataset. 



In the study by Chiu and Chen
<cit.>, 159 pieces from the 8notes website were used, whereas <cit.> utilized 1800 MIDI files from the same source. The categorization of these pieces, provided by users of 8notes, raises concerns about their reliability. Unfortunately, neither study provides access to their data or details on how they were segmented. Other work <cit.> has attempted to understand the effectiveness of various features, including those proposed in <cit.> for categorizing the grade levels of a specific piano curriculum. Recent efforts by Zhang et al. <cit.> and Ramoneda et al. <cit.> have focused on compiling datasets with difficulty annotations from the established piano publisher Henle Verlag, with the latter's dataset not only being the most extensive but also the only one made publicly available. Therefore, for our comparative analysis, we will use the  open-source datasets presented in <cit.>, namely Can I Play It? (CIPI), which has 9 levels of difficulty, and Mikrokosmos-difficulty (MKD), which includes 3 levels.
 


Finally, in order to validate our approach, we consider a different and established feature set, i.e., the standard music symbolic features available through Music21 library <cit.>, which includes (amongst others) established JSymbolic features <cit.>,  
thus facilitating a meaningful comparison with our proposed descriptors. In addition, 
we also contrast the results achieved with our novel descriptors with those obtained with the features by Chiu and Chen
 <cit.>, which are also reimplemented and open-sourced in this study.
Note that none
of the approaches previously mentioned 
has focused on the interpretability of the descriptors, which is a key contribution of our work.




§ INTERPRETABLE 


The  model (cf. Figure <ref>) is designed to provide interpretability akin to a rubric, enabling its analysis and results to be intuitively aligned with established practices in music education. This approach ensures that the model's logic and outcomes are easily comprehensible, facilitating their usage in music education 
along to traditional 
tools.



 §.§ Model Architecture



The network, 
comprising a series of linear layers dedicated to process individual input descriptors and followed by a nonlinear activation function,  
is formulated as follows:

Given a set of N input descriptors, each descriptor x_i is first processed through its dedicated linear layer with weight 
w_i and bias b_i, followed by a hyperbolic tangent activation function to yield:

    s_i = tanh(w_i · x_i + b_i)

where s_i represents the processed score for the i-th descriptor. 
Scores are then aggregated  
in a single 
score S_agg:

    S_agg = ∑_i=1^n s_i


The aggregated score S_agg is then 
passed through a final linear layer to obtain the logits for the class predictions, which are  
mapped to probabilities with a 
sigmoid function:

    P⃗ = σ(S_agg·w_f  + b_f)

where σ denotes the sigmoid function, 
w_f and b_f are the weight and bias of the final linear layer, respectively. 








 §.§ Ordinal Optimization


This model applies an ordinal optimization approach <cit.>, predicting ordered categorical outcomes, difficulty levels such as beginner (1), intermediate (2), and  advanced (3), 
through logits. These logits, computed using a mean squared error (MSE) loss, indicate the model's predictions on the ordinal scale.

Difficulty level is then obtained as:


    max{ i  where  P_i ≥ 0.5  and  P_j ≥ 0.5, ∀ j < i}







 §.§ Interpretability




In , the descriptors (automatically computed from the data) are, to some extent, comparable to the formalized evaluation criteria defined in traditional rubrics; similarly, the aggregated score, might be comparable to a final grade/mark assigned in an educational scenario. Given the correspondences between both, we could consider the model a “white-box” approach, able to promote 
transparency and interpretability, similarly to a  
rubric. 


It uses independent linear transformations on input descriptors to generate scores between -1 and 1, which directly influence the regression output, S_agg. 
Since negative scores, might be not fully understood in terms of difficulty level, 
we normalize scores between 0 and 1, rescaling  between 0 and 12. 
This approach mirrors rubric's ability to provide objective and 
structured feedback, with the simplicity of these transformations aiding in understanding the impact of features in predictions.


The interpretability of the model lies in its ability to dissect each descriptors' influence on a piece's difficulty level. 
Consequently, analyzing each descriptor's scores might reveal its overall importance on the prediction.
Lastly, S_agg is a continuous-ordered scalar with rank correlation to difficulty. 
Therefore, from S_agg, we retrieve ordered and discrete categories  with clear decision boundaries.









§ EXPLAINABLE DESCRIPTORS








From codified musical scores, we extracted numeric features which are feed to a classification
algorithm. 
We re-implemented a set of features from the literature <cit.> while proposing
a novel one,
Pitch Set LZ.
In addition to explaining the features (cf. Table <ref>), we will provide their technical descriptions and analyze their relevance to difficulty and interdependencies using the data.





 §.§ Descriptors


In our work, we analyze music sheets encoded in symbolic format, focusing on extracting pitch and timing. Following the approach suggested by Chiu and Chen <cit.>, we process left and right hand parts separately to clarify pedagogical aspects of musical difficulty. Our primary analysis involves sequences of pitch set events, each characterized by a pitch set S and onset time T. Pitch sets, represented by sets of MIDI numbers, are defined over the alphabet of all pitch sets 𝐒 that occurred in a score part, while onset times are calculated in seconds from the performance start by the music21 library <cit.> marked tempo information. This method emphasizes the timing of note attacks, duration and rests. Additionally, we consider a collection of pitch events, each defined by pitch P over the alphabet of all pitches 𝐏. Our analysis started with the five features identified by Chiu and Chen <cit.> as most relevant to understanding musical difficulty.



Pitch Entropy.
The entropy of pitches in the pitch events:

    -∑_i∈𝐏 p(P = i)log_2p(P = i)



Pitch Range.
The distance between the minimum and maximum MIDI pitches in a score part.



Average Pitch.
The average MIDI pitch in a music sheet.





Displacement Rate. 
Initially proposed by <cit.>, it quantifies the extent of hand movement across the
keyboard during the performance of a score. It analyzes maximum pitch distances between consecutive pitch set events and is calculated as a weighted average of three categories: distances less than 7 semitones (assigned a weight of zero); distances over 7 semitones but under an octave (assigned a weight of one); and distances of an octave or larger (assigned a weight of two to emphasize larger movements).



Average IOI:
Average Inter Onset Interval.
A concept similar to the “Playing speed”  
introduced by
 <cit.>, a term we consider deceptive since it actually decreases as the hand's “speed” increases. This is an average time in seconds between onsets of two consecutive pitch set events. Let's denote i^th onset time with T_i, then the value is:

    ∑_1≤ i ≤ N^events - 1(T_i+1 - T_i)/N^events-1

In 23% of the scores, information about the recommended performance tempo is missing.

We then assume the tempo is 100 beats per minute (bpm). 
Thus, in cases of missing bpm, the Average IOI feature might not be relevant.



Pitch Set LZ. Lempel-Ziv complexity of pitch set sequence. Before introducing our proposed descriptor,
it is crucial to provide context and motivation.
Pitch Entropy, as emphasized by Chiu and Chen <cit.>,
is particularly relevant—a conclusion supported by the analysis of correlations between difficulty
and features in the following section, as well as by informal experiments.
As Sayood discusses <cit.>, there's a link between entropy of a task
and the cognitive load it imposes on the performer, a concept that may also apply to music performance <cit.>.
However, music is often perceived in terms of larger structures like phrases
and sections, not just isolated pitches, prompting us to seek a descriptor that captures the “repetitiveness”
of music on a broader scale. To this end, we employ LZ-complexity, a measure of redundancy introduced by Lempel and Ziv <cit.>.
In context of music research, it was used for binary encoded rhythm analysis by Shmulevich and Povel
<cit.>. We apply LZ-complexity to sequence of pitch sets:
scan a score part, identify all subsequences of pitch sets that cannot be reproduced
from preceding material through a recursive copying procedure.
The number of such unique subsequences is defined as the LZ-complexity of the part.
This approach allows us to assess the structural complexity and redundancy of a musical piece,
highlighting the cognitive demands placed on performers.




 §.§ Feature Analysis





We assume that, for easier interpretability,
features must on average change monotonically with the difficulty level.
To measure this quality, 

we use the τ_c version of Kendall rank correlation coefficient due to 
its ability to deal with
“heavily tied” rankings <cit.> (many musical pieces have the same difficulty, hence, we have multiple ties in the ranking by difficulty). 
τ_c is equal to 1 when feature and difficulty rankings are perfectly aligned in the same direction, -1 if they are aligned in opposite directions. As the number of nonconcordant cases increases, the coefficient approaches zero. 
In <ref>,
the results show that
the features related to pitch organization are the most correlated to difficulty.
Hand displacement and Inter-onset intervals are less correlated,
while average pitch seems almost irrelevant.


In addition, we aim to uncover dependencies among the features themselves while mitigating the influence of difficulty, with whom  most features are correlated.
To achieve this,
we calculate conditional τ_c correlations for all feature pairs given a fixed difficulty level, 
and average the coefficients across all difficulty levels.
We then convert these coefficients into a distance matrix
and apply hierarchical agglomerative clustering based on average distance to identify clusters of correlated features.
From the resulting dendrogram (cf. <ref>), we observe that features correlated with
difficulty—namely Pitch Entropy, Pitch Set LZ, and Pitch Range—are also interrelated.
This is remarkable because the three most correlated features are not inherently dependent:
 one could envision a music piece with any of them maximized while maintaining low values for the others.
However, pieces in CIPI
typically exhibit coordinated values in these descriptors.
Thus, we mostly observe the combined effect of these features,
making it challenging to reliably decompose “difficulty” into an aggregate of independent components.


























§ EXPERIMENTS



 §.§ Experimental Setup


To evaluate the effectiveness of our proposed method, we utilized the Mikrokosmos-difficulty (MKD), and Can I play it? (CIPI) datasets <cit.>.
For fair comparison,
we use the 5-fold cross-validation approach defined in <cit.>. 
In each split, 60% of the data is used as a train set, while the remaining  is equally divided into  validation and test sets.
 

As in <cit.>,  
we employ 
mean squared error (MSE) and accuracy within n classes (-n) for evaluation. These metrics are chosen for their applicability to ordinal classification challenges, with -n assessing the model's accuracy for n classes from the true labels, and MSE measuring the average squared prediction error across classes.
The effects of dataset imbalances and a fair evaluation across classes are mitigated by macro-averaged metrics.





We optimize the models during training through Adam optimizer with a learning rate of 10^-2.
The training process incorporates early stopping, based on the -n and MSE metrics from the validation set, to prevent overfitting. Through 
Ordinal Loss, 
we frame difficulty prediction as an ordinal classification task, as mentioned in Section 3. 
We apply a standard scaler and dropout to the features to prevent individual ones 
from dominating. 
For each experiment, we look for the best hyperparameters using Bayesian optimization <cit.>: batch size within the range from 16 to 128, dropout rate between 0.1 and 0.5, learning rate decay from 0.1 to 0.9, and the learning rate itself, tested over a logarithmic scale from 1e-5 to 1e-1. This 
approach allows us to systematically explore the hyperparameter space and identify the optimal settings for our models; thus, enabling a
fair comparison between experiments.








 §.§ Experimental Results





In Table <ref>, the results from the comparison between the performance of our novel approach  with the presented descriptors  (cf. Sections 3 and 4) and the results achieved by three previous models from the literature (argnn <cit.>, virtuoso <cit.>, pitch) as well as their collective ensemble, are shown. 
Our model 
achieves the highest
Acc-9 score of 41.4 (±3.1) in CIPI, surpassing the ensemble's 39.5 (±3.4), 
while displaying the second lower 
MSE of 1.7 (±0.5), only overtaken by 
the ensemble's 1.1 (±0.2). 
With an Acc-3 score of 79.6 (±8.8) in the MKD dataset, our approach is superior to previous ones but with a higher standard deviation.



























































In the following, we 

examine the impact of various feature and model configurations on  performance (cf.  Table <ref>).
As baseline for comparison, we consider the configuration previously discussed (cf. Ours in Table <ref>).


Employing only the five Chiu and Chen <cit.> descriptors, excluding Pitch Set LZ, leads to a decrease in Acc-9 by -5.2, reflecting a performance drop from the baseline. The use of Music21 <cit.> descriptors, which include JSymbolic <cit.> and other descriptors widely used in the community, results in a decrease in Acc-9 by -4.7 and a decrease in MSE by -0.4, showing slight improvements in MSE but not in accuracy. However, note that a larger number of descriptors could decrease the explainability.
Combining all the descriptors  
slightly decreases Acc-9 and MSE:   -2.5 and -0.4, respectively; with the accuracy results  still under the baseline. 
These results indicate that 
the  descriptors discussed 
in Section 4 constitute the best option for difficulty estimation on CIPI. 
Since average pitch showed no relation to difficulty in the feature analysis, we repeated the experiments without this feature. This lead, however, to non-significant worsening of the results.








Concerning the impact of different model configurations, we replace the tanh by sigmoid non-linearities to guarantee positive scores. The obtained MSE rate is similar but the accuracy drops by -3.1. This means that negative scores could aid in training, which is why we keep them, but normalize the scores after the training phase. Besides, substituting the ordinal encoding used in the baseline 
with a traditional one-hot encoding with cross-entropy loss, results in a decrease in Acc-9 by -5.2 and an increase in MSE by +0.4, highlighting the importance of ordinal regression in achieving lower MSE rates. Lastly, logistic regression with ordinal loss 
decreases the Acc-9 by -1.4 while showing a decrease of MSE by -0.2. This offers a compromise for both metrics
but without beating our setup and to our understanding, being less interpretable.  












Overall, the gains offered by Rubricnet with the features proposed are relatively modest compared to the baselines. However, having a 
smaller feature set is necessary for explainability. The novelty of our approach lies in aligning the interpretability of music education with rubric-like interpretability feedback. This alignment is essential for a successful 
application 
of our model in 
practice, as we will discuss in further sections. 




 §.§ Decision Boundaries


In , the input features are combined into a single scalar before performing
the final ordinal classification. Analysis of the results shows that the final layer
defines optimized decision boundaries, setting thresholds for S_agg that progressively increase along with difficulty levels.
Because of the final sigmoid activation, once  exceeds a boundary, the 
corresponding difficulty level will always be active, which guarantees the ordinality of the predictions.




By examining the decision boundaries (cf. Figure <ref>), we observe that the trends are similar across splits, displaying shorter valid ranges around intermediate levels. Note that in split 2, there are only 8 classes because the model ignored the last class. This can happen as we use numeric optimization, which sometimes falls into local minima. These minima might seem optimal based on the validation metrics but do not meet our overall performance expectations.









§ DISCUSSION AND LIMITATIONS



























Now, we analyze whether  is interpretable from a musical point of view. 

To understand  how features impact the final level suggested by the model, we evaluate the contribution of each descriptor to the aggregated score. 

Since learning to play an instrument is a progressive process,  relative contributions of features to different levels the grade 1 are displayed instead of  absolute values.
These contributions are averaged across splits on the test set and shown in  <ref>.









We observe a trend of higher contributions when the level
increases for every descriptor. 

This observation is consistent 

with the fact that  value increases for higher levels (cf. <ref>).
The most discriminative features are 
pitch entropy and 
pitch range, 
as well as the LZ descriptor for 
higher levels. 

Conversely, some features, average IOI 
or the average pitch,  
have low contribution 
to the model's decisions, as shown by their relatively constant and small values across grades. 



The latter is expected, since 
very different pieces could have the same average pitch, not disclosing anything about difficulty.


The former might be explained by the averaging, which
can remove information, especially when a piece can alternate between fast and slow parts. Besides, as mentioned before, tempo is often poorly annotated in the dataset.









To better understand the explainable capabilities of the proposed descriptors, in the following, we provide a musical examination of two concrete samples, by this demonstrating the interpretability of our approach.
Nocturne op. 9, no. 3 by F. Chopin, is labelled as level 7, but classified as level 2. All the descriptors are below the grade average, as shown in the rubric (cf. Grade Divergence in Figure <ref>).
Our hypothesis is that this nocturne contains many challenges that go beyond the descriptors used. There are constant changes in dynamics, a variety of articulations, and as a key difficulty aspect, many types of polyrhythms between the right and left hands. Further research should address all the types of difficulty challenges, probably underrepresented in the existing datasets.




The piece Berceuse in D-flat major, Op.57 by F. Chopin, shown in <ref> is appropriately classified as grade 7. This is because it maintains a left-hand accompaniment with few changes, in contrast to the higher virtuosity of the right hand. 
The left hand has scores below average in most descriptors because of its few changes: Pitch Range (-0.41), Average IOI (-0.04), Pitch Set LZ (-0.34), Average Pitch (-0.01), and Pitch Entropy (-0.52). In contrast, the right hand shows more virtuosity, with higher than average scores for all the right hand features.
These scores collectively contribute to a final cumulative score that accurately reflects the overall difficulty.


































Finally, it should  be noted that our approach primarily focuses on
descriptors
related to pitch sequences and
onsets, while disregarding others. 
Still, the ablation study showed that other features sets (those from music21), even  covering aspects like rhythm variety,  
do not  enhance our classifier's performance either. In addition, expressive elements <cit.> such as dynamics, tempo changes, and articulation, since often left to performers' interpretation, 
are  not always captured in musical notation <cit.>, 
and therefore is 
a dimension our score-based model does not consider.







§ CONCLUSION AND FUTURE WORK



In our study, we proposed a novel white-box 
parameter-efficient model aligned with the music education community tools, rubrics, which outperforms previous approaches on difficulty estimation. 
In addition, we  
created an interactive companion page for visualizing CIPI and MKD datasets.
In summary,   
we showed that analyzing explainable 
descriptors, unlike deep learning models, 
offers clarity, 
which  gives both teachers and students specific insights into pieces.   
This approach not only underscores the importance of explainable artificial intelligence (XAI) in understanding music difficulty, but also emphasizes the potential for such technologies to contribute to the broader field of music education.
For future research, we consider interesting to creating a dataset based on technical challenges like finger fluency and polyphonic complexity, as well as user studies for understanding the perception of interpretable feedback by music education community.















§ ETHICS STATEMENT


The system presented in this paper aims at obtaining the difficulty of a musical
piece through several descriptors. In previous work, descriptors were not available, limiting access to the area. This situation underscores the need for open science practices. Therefore, we open our implementation, to facilitate access for new researchers. Besides, the dataset used for this study is available upon request for non-profit and academic research purposes.
While this limits its use in commercial applications, it ensures the reproducibility of the results. The data consists of open-source scores of music that
is no longer copyrighted, its use for open research can thus be considered fair.


The proposed work belongs to the area of assisted music learning. One might
argue that such a tool can have a detrimental impact on music teaching jobs. 
While this is a valid concern, we think that an eventual solution of the addressed task,  would not endanger music educators profession, whose role naturally goes much beyond than  categorizing music in difficulty levels.  Instead, this technology should be seen as a way to support them in the own   teaching practices, for instance, by alleviating their burden on some duties, such as exploring large collections, and by this enabling them to easily discover forgotten  musical works from our cultural heritage which fit students' needs. 
Moreover, through this research, we also aim to convey the message that the path to advancement does not solely lie in acquiring more data or creating larger models. 
By highlighting what drives its decisions, our proposed model aligns with the goals of eXplainable AI, something crucial for its acceptance in music education.  
Although our efforts in making the system interpretable and explainable will partly answer  the common criticisms made to black-box approaches, the real impact of our system remains to be verified by  its future use in real scenarios. 





§ ACKNOWLEDGEMENTS


We want to thank Alia Morsi's previous work on difficulty estimation from a feature engineering perspective, which encouraged research in tabular data <cit.>. The team would also acknowledge Marius Miron for continuously insisting on aligning difficulty estimation with explainability. He also highlighted the direction of using a rubric-like explainability feedback, pointing out Ustun et al.'s work <cit.>. We also thank Roser Batlle-Roca for helping to discuss some concepts between interpretability and explainability based on her research <cit.>.

This work was supported by “IA y Música: Cátedra en Inteligencia Artificial y Música’' (TSI-100929-2023-1), funded by the Secretaría de Estado de Digitalización e Inteligencia Artificial, and the European Union-Next Generation EU, under the program “Cátedras ENIA 2022 para la creación de cátedras universidad-empresa en IA’'. This work was also supported by the ANR project TABASCO (ANR-22-CE38-0001) and the travel grant MERMOZ2-012047. 
Finally, this work was also possible through the support of the Hightech Agenda Bayern, funded by the  Free State of Bavaria (Germany).






































 Paper Title: Computational music analysis from first principles.





^1Department of Music, Princeton University, Princeton, NJ 19147, USA
^2Center for the Study of Complex Systems, University of Michigan, Ann Arbor, MI 48109, USA
^3Santa Fe Institute, 1399 Hyde Park Road, Santa Fe, NM 87501, USA

§ ABSTRACT 
We use coupled hidden Markov models to automatically annotate the 371 Bach chorales in the Riemenschneider edition, a corpus containing approximately 100 000 notes and 20 000 chords.  We give three separate analyses that achieve progressively greater accuracy at the cost of making increasingly strong assumptions about musical syntax.  Although our method makes almost no use of human input, we are able to identify both chords and keys with an accuracy of 85% or greater when compared to an expert human analysis, resulting in annotations accurate enough to be used for a range of music-theoretical purposes, while also being free of subjective human judgments.  Our work bears on longstanding debates about the objective reality of the structures postulated by standard Western harmonic theory, as well as on specific questions about the nature of Western harmonic syntax. 

Computational music analysis from first principles
    Dmitri Tymoczko^1,3 and Mark Newman^2,3
    August 5, 2024
==================================================






§ INTRODUCTION

More than two hundred years ago, Gottfried Weber argued that traditional Western harmony was pervasively ambiguous <cit.>.   One problem involves chords and keys: a single progression such as C major  F major can be experienced as a tension-reducing dominant-tonic in F major or a tension-increasing tonic-subdominant in C major (Fig. <ref>).  A second problem involves chords themselves: a voice moving from D6 to C6 over a stable {F4, A4} can signify either a change of harmony, from an inverted D minor to a root-position F major, or a “merely linear” passing motion that creates an F major chord as a nonsyntactical byproduct.  A third problem involves incomplete sonorities, for instance bare thirds that could plausibly belong to multiple triads, or triads belonging to multiple seventh chords.  Together these ambiguities ensure that even relatively simple passages can be analyzed in multiple ways, with no agreed-upon rules for choosing among them.  Music analysis thus requires complex holistic judgments, balancing vague and often-conflicting heuristics <cit.>.  It is not surprising, therefore, that we find widespread disagreement about analyses even among human experts—not only about general principles, but also specific passages (Fig. <ref>).  





This holism helps explain the somewhat disappointing performance of unsupervised computational musical analysis.  Micchi et al. <cit.>, for example, review a variety of attempts at automated analysis, none of which were able to assign Roman numerals with greater than 50% overall accuracy.  In a different vein, Quinn and White <cit.> used hidden Markov models to classify progressions in the Bach chorales but their results depart substantially from standard music-theoretical practice, for instance by including seemingly “nonharmonic” tones within chords, and by assigning tonic and subdominant to the same harmonic category.  Currently, no existing studies have managed to use unsupervised methods to  annotate substantial musical corpora in a way that closely captures the judgments of human annotators.  Some approaches have achieved a good match to human intuition in more limited contexts, such as the studies of Noland and Sandler <cit.> and Nápoles López et al. <cit.>, which both focused on the narrower question of identifying key centers rather than chords and were able to do so with about 90% accuracy.

Substantial human disagreement, coupled with the lack­luster performance of computational methods, together raise the skeptical possibility that human judgments might not be reducible to objective algorithms.  Weber himself argued that musical analysis required intuitive human judgment <cit.>.  Quinn <cit.> has argued that the notion of a chord is theoretically superfluous, suggesting that one can accurately capture the syntax of traditional Western tonal music without distinguishing harmonic from nonharmonic tones.  In a similar vein, Gjerdingen <cit.> proposes that the syntax of Western music is grounded in “schemas” that do not not require the apparatus of standard theory.  Perhaps the familiar theoretical picture, in which nonharmonic tones decorate tertian harmonies, is not so much an empirical claim as an interpretive grid we impose onto music.

Against this background, we present in this paper a simple and largely assumption-free computational method that does a good job at reproducing expert human analysis, achieving about 88% overall correctness at the task of identifying keys and chords in the 371 Bach chorales of the Riemenschneider edition, compared to a human analysis of the same corpus.  Our approach encompasses a family of related algorithms all sharing the same basic structure: for a given input, they find a fixed number of transposable probability distributions that best represent the occurrence of elements (notes or chords) in that input.  When fed raw notes and told to find a small number of probability distributions, the output can be interpreted as a series of chords; when fed these same chords as input, the output can be interpreted as a series of keys.  The significance of these results lies not so much in their accuracy alone but in the combination of accuracy, simplicity, and lack of presupposition: our algorithm performs well with very little information about musical structure, using well-known and straightforward techniques, and without the need for extensive training sets of expert human analysis.  In using familiar methods to reproduce the standard chord-and-key picture, it answers skeptics who have questioned the objectivity of Western musical concepts <cit.>.  Conversely, the close correspondence to human annotations suggests that those annotations do not  rest on large numbers of arbitrary decisions.  Finally, by retracing the steps required to approximate human analyses, our algorithm clarifies the assumptions underlying traditional analytical practices.

The structure of the paper is as follows.  Section <ref> describes the statistical model we use in detail and places it in the context of previous work on related models.  Section <ref> applies this model to the corpus of Bach chorales and compares the results to those of a human analysis, finding reasonably good agreement.  Section <ref> improves these already-good results by applying a few additional rules of thumb concerning modulatory tendencies, local harmonic “laws,” and the principles of nonharmonic tone usage.  Section <ref> considers some broader methodological issues arising from our investigation, as well as potential uses for the machine-annotated corpus generated by our calculations.  A number of technical issues are discussed in appendices.  Data and code are freely available online at .




§ HIDDEN MARKOV MODELS FOR ANALYZING

KEYS AND CHORDS


Our basic goal is to take a sequence of notes, represented as notation and divided among some number of voices, and infer from them the underlying chords and keys at each point in a work.  We ignore lengths of notes, rests, and other rhythmic elements of the music: all notes are given equal weight, whether short or long, and whether they fall on the beat or off it.  Whenever any voice begins a note or rest, we consider it a new data point for our analysis.  The key or chord need not change at every such data point however.

To perform our analysis we make use of hidden Markov models, a standard statistical tool for the analysis of discrete time-series data.  In a hidden Markov model it is assumed that a system of interest is governed by “hidden” factors, changing over time, which are not directly observed but which nonetheless influence the observed behavior of the system.  For example, the observed behavior might be the sung notes of a Bach chorale and the hidden factors the underlying chords, which one can model (for example) with Roman numerals.  The chords are hidden in the sense that the Roman numerals are not stated explicitly by the composer or written on the score, but they determine the observed data—the notes—in a probabilistic fashion.  If the underlying chord is C major, for example, then there is a high probability that we will hear certain notes {C, E, G}, a lower probability that we will hear some others {D, A, B}, and a still-lower probability for others {C, E, A}.

The hidden factors can change over time.  Our C major chord might transition to G major, then to A minor, and so forth, but some transitions are more common than others: C major is unlikely to be followed by C major for example.  To capture these differences, a hidden Markov model incorporates a set of transition probabilities that represent how likely every chord is to be followed by every other, and we expect that some of these probabilities will be  large and others small.

Collectively, these elements define the hidden Markov model: chords change over time with some transitions being more likely than others, and notes reflect the current chord in a probabilistic fashion, the probabilities of individual notes being dependent on the underlying chord.

The hidden Markov model can be thought of as a machine for generating sequences of notes from underlying chords.  In the present instance, however, we want to use the model in reverse, we want to infer the chords from the notes.  We ask, “If these are the notes we hear, what is our best guess about the chords that produced them?”  We can answer this question by combining the hidden Markov model with standard numerical techniques of Bayesian inference: the Baum-Welch and Viterbi algorithms, described below.

A central goal of our work is to perform these calculations while making as few assumptions as possible ahead of time.  We could for instance make use of guesses about the model parameters based on input from human experts, and analyses of this kind have been performed in the past <cit.>, but we will not do that here.  The parameters in our calculations are given random initial values, which are then automatically refined by the Baum-Welch algorithm to fit the data.  We give the calculation as little help as possible in performing its analysis; we are interested in what it comes up with when given free rein to take any approach it likes.

Once we have inferred a collection of chords we use them as input to a second layer of analysis in which we infer the prevailing key at each moment from the chords.  This second analysis is also performed using a hidden Markov model.  The hidden features of this model are now the keys and it is assumed that chords are produced with probabilities that depend on the current key.  If we are in C major for instance, then some chords will have a high probability of occurrence (C major, G7), while others will have lower probability (C minor, A major).  Again we make only the most basic prior assumptions about the hidden Markov model and we initialize the parameter values at random; we are interested to see what the algorithm comes up with on its own.  In effect, the computer “listens” to a large corpus and “learns” how music works from first principles, based on what it hears.

Our only assumptions about our models are the following:


  * Music is built from an ordered set of 12 octave-free pitch classes, laid out in a circular arrangement like the numbers on a clock. Musical transposition is represented by addition modulo 12.

  * The probability of a particular note at a particular time is dictated by an underlying “chord,” a transposable probability distribution anchored on one of the 12 tones, and belonging to a small number of types, which is fixed at three in our calculations.  In practice these types turn out to correspond to accepted classes such as major and minor chords, but we do not specify this: we specify only the number of chord types and not their structure.

  * The probability of transition from one chord to the next depends on chord type and the interval by which the chords are separated.  We do not specify what the probabilities are, only that different probabilities can exist for different transitions.

  * The probabilities of appearance of chords are dictated by an underlying “key,” which is another transposable probability distribution anchored on one of the 12 tones, and belonging to a small number of types, fixed at two in our calculations.  In practice these types turn out to correspond to major and minor keys, but again we do not specify this, only that different types exist.

  * The probabilities of transition between keys may vary depending on the type of key and the interval between successive keys, but we do not specify what those probabilities are.


Arguably the biggest assumption here is about the number of chord and key categories (three and two respectively), and these choices are certainly rooted in our pre-existing knowledge about the style.  The choice of two key categories, in particular, is based on the expectation that we will find major and minor keys only.  Other styles could employ a larger number of modes and demand a model with more key types.

The number of chords is more flexible.  Three categories gives reasonable results without artificially shoehorning highly dissimilar chords into a single category.  But we have performed analyses with larger numbers of chord types and the results are reasonable.  One can gain additional nuance by allowing more types, at the expense of more complex analytic output and also longer computation time.  Using three chord types, however, gives excellent results, so a larger number, while allowable, is not necessary.




 §.§ Related previous work

A number of previous authors have employed hidden Markov models for harmonic analysis, particularly for the task of inferring the key of a piece of music, a calculation similar to the second half of our analysis described above.  The goal of this work has generally been to create machinery that can identify keys with high accuracy.  To do this, most authors have tuned their algorithms using knowledge from human experts concerning occurrence of notes within keys and transitions between keys.  While this can indeed produce high accuracy, it conflicts with our goal of determining how much the model can achieve on its own.  The difference of approach is analogous in some ways to different philosophies in the design of programs for playing chess.  Early chess programs were trained with large amounts of expert human knowledge about the game, such as opening books and endgames, and these programs can perform well.  But more recent work focused on a new class of algorithms, exemplified by DeepMind's Alpha­Zero, which eschews human expertise and learns from scratch by playing against itself.  Starting with only the basic rules of the game, such as how the pieces move and what constitutes a win, the program is able in less than 24 hours to graduate from beginner chess to superhuman skill without outside help.

Most previous work on hidden Markov models for musical analysis has been analogous to the first approach, making use of human knowledge to build a working machine.  Noland and Sandler <cit.>, for example, made use of a hidden Markov model to learn keys from chord sequences in music by the Beatles, attaining 91% accuracy in key identification compared to a human analysis.  The parameters of the model were initialized by hand using values assigned by experts.  Nápoles López et al. <cit.> took a similar approach, but inferred keys directly from notes without using chords as input.  Their approach works best with symbolic input but can also be applied directly to audio data.  A number of other authors have also developed methods for analyzing audio data, most often in the form of pitch-class profiles <cit.>.  Chai and Vercoe <cit.> used this approach, for instance, to deduce keys directly from notes, while Sheh and Ellis <cit.> and Burgoyne and Saul <cit.> both trained hidden Markov models to infer chords from pitch-class profiles.  Burgoyne and Saul went further to infer keys as well, although again using hand-tuned parameters.

Our approach makes use of no hand-tuned parameters—the only assumptions are those laid out in Section <ref>.  In effect we tell the program only the “rules of chess” and ask whether it can teach itself, from scratch, to interpret harmony as a human being does, simply by listening to enough of it.  We also use a larger corpus, annotated in much more detail, than the corpora featuring in previous work: our annotations record not only global keys but also local key changes, and include approximately 20 000 chords.[Albrecht and Shanahan <cit.> use a corpus of 982 pieces, but each is annotated only with a single global key; our corpus instead includes within-piece key changes.]  The corpus has been checked for accuracy both by hand and automatically, and it has been used in the past for a variety of purposes <cit.>.

More recently, several authors have trained machine learning algorithms on corpora of hand-annotated scores <cit.>.  This strategy once again treats expert judgments as a capacity to be emulated.  Our work instead seeks to use machine learning to understand that capacity, and in particular to determine the extent to which it is guided by arbitrary conventions.



 §.§ Inferring chords from notes


We use two specific hidden Markov models in our calculations.  The first represents how notes are related to the underlying chords.  As applied to a complete corpus of musical works, such as a set of chorales, this model assumes a sequence of chords for each work, transition probabilities between chords, and note probabilities that depend on chords.  Chords are defined by an anchor pitch r and a type t.  The anchor can be thought of roughly as the root of the chord, although in some cases, as we will see, it does not correspond to the conventional root.  Moreover, we are at pains to avoid any presuppositions about chord structure, so we use the generic term “anchor” here.

As discussed in Section <ref>, we fix the number of chord types at three.  We do not specify what the types represent—that is for the algorithm to learn—but as we will see they end up being major and minor triads plus a combined third type that captures both diminished and dominant chords.  For convenience we will often denote a chord by a single symbol c, which is synonymous with the anchor/type pair (r,t).

As also mentioned previously, we identify a new data point every time any voice starts a note or rest.  We will refer to these data points as steps of the work.[Quinn and White <cit.> call them “salami slices.”] Different steps do not necessarily correspond to the same note value—some could be quarter notes while others are eighth notes for instance.  The underlying chord may change at any step, or it may stay the same.  All chords are considered a priori equally likely at the start of a work and we define a set of probabilities α_c_1c_2 for subsequent transitions between chords c_1=(r_1,t_1) and c_2=(r_2,t_2) thus:

    α_c_1c_2 = P( c_2 | c_1 ) = f(r_2-r_1,t_1,t_2).

That is, the probability that the next chord will be c_2 given that the previous one was c_1 is a function of the types of the chords and the interval between their anchors.  Here r_2-r_1 indicates the upward interval from the first anchor to the second, in semitones modulo 12.  Thus the interval from B3 to C4 is 1 semitone while the interval from C4 to B3 is 11 semitones.  The probability of not changing chord at a particular step is

    α_c_1c_1 = f(0,t_1,t_1).

Note that this does not depend on the anchor r_1 of the chord but does depend on the type t_1 and hence is potentially different for chords of different type.

We also define a set of probabilities for pitch classes to occur given the current chord.  Since pitch classes are octave free there are only 12 possibilities, which we represent by integers n = 0…11.  Then

    μ_cn = P( n | c )

is the probability that a sounding note has pitch n given that the current chord is c.

This fully defines our first model.  Our goal is now to take the notes of all works in a corpus and fit the model to them, meaning we want to infer the chords and probabilities given the observed notes.  We fit a single instance of the model to all works in the corpus, under the assumption that similar rules of harmony apply to all works.  There are standard techniques for performing this type of fit, which comprise two stages of analysis <cit.>.  In the first stage we use the Baum-Welch expectation-maximization algorithm to calculate the values of the probabilities.  Then in the second stage, given those values, we use the Viterbi dynamic-programming algorithm to calculate the best estimate of the chords.  A technical description of these two well-known algorithms is given in the appendices.

The Baum-Welch algorithm is an iterative algorithm that starts from an initial guess about the values of the parameters and repeatedly refines that guess until a locally optimal value is attained.  In previous work, authors have used input from human experts to make the initial guess, but in order to avoid relying on expert opinion we use random values produced by a high-quality random number generator.  Like all expectation-maximization algorithms, it is known that the Baum-Welch algorithm can converge to sub-optimal solutions from certain initial conditions.  To guard against this we perform 50 separate runs of the algorithm with different random starting conditions and take as our final solution the one that gives the highest value of the data likelihood—see Section <ref> and Appendix <ref>.

Note that, since our calculations are performed using pitch classes as input, the chords produced by our calculation do not include information about inversions.  In Section <ref> we discuss simple rules for deducing inversions automatically by referring back to the score.




 §.§ Inferring keys from chords


Once we have determined the chords, we use our second hidden Markov model to also determine the prevailing key at each step.  This model assumes a sequence of keys, one at each step, transition probabilities between keys, and probabilities of chords given the prevailing key.  Keys are represented by an anchor s and a type u, and again for brevity we will use a single symbol k to represent a key, which is equivalent to the anchor/type pair (s,u).  (Again, the anchor can be intuitively associated with a tonic, but we do not assume this to be the case.)  We allow two types of keys but do not stipulate what they should be, although as we will see they strongly resemble major and minor.

Transitions between keys are parameterized by probabilities

    β_k_1k_2 = P( k_2 | k_1 ) = g(s_2-s_1,u_1,u_2),

which is a function of the types of the keys and of the interval between their anchors (measured again as the upward interval in semitones, modulo 12).  All keys are considered a priori equally likely at the start of a work and the probability of not changing key is

    β_k_1k_1 = g(0,u_1,u_1),

which thus depends on the type of key (as it should) but not on the anchor.  The probabilities of chords are

    ν_kc = P( c | k ),

which is the probability that a sounding chord is c given the prevailing key k.

This second model is fitted to the chord sequences generated by the first model (Section <ref>), one sequence for each work in the corpus, using the Baum-Welch algorithm to determine the probabilities and the Viterbi algorithm to determine the specific keys.  Again we fit a single instance of the model to all works in the corpus, and again the initial parameter values are generated randomly and we repeat the entire calculation 50 times to guard against the possibility of converging to a sub-optimal solution.

The end result of applying both models to the corpus is a set of keys and chords at every step in every work.  As a by-product we also derive the values of the probability parameters, which are of interest in their own right, as discussed in Section <ref>.




§ ANALYSIS OF BACH CHORALES


Our analysis focuses on the 371 Bach chorales in the Riemenschneider edition.  As described in Section <ref>, our calculations are divided into two parts, the first focusing on the chords and the second on the keys.   Each of these is then further subdivided into the calculation of the probability parameters and the calculation of the chords and keys themselves.

Turning first to the chords, our algorithm looks for chords of three types, which are distinguished by the probability distributions of their notes.  Over the course of the 50 runs of the algorithm, the best fit we find to the data is one in which the three chord classes correspond to major, minor, and dominant seventh chords, as described below.  The next best fit has classes corresponding to major, minor, and dominant seventh with suspended fourth, but this fit is enormously worse than the first-place winner: the data likelihood—the probability of the notes given the inferred chord classes—is more than a million times worse.  This allows us to say with the some confidence that the major/minor/dominant seventh analysis is the best fit to the data in the present case.  (See Appendix <ref> for further discussion of this point.)

The distributions of notes within each chord class found by our method when applied to the chorales are shown in Fig. <ref>, with the convention that the highest-probability note is placed in the anchor position in each case.  The first two plots correspond clearly to major and minor chords—note the high probabilities assigned to the root, major third, and fifth in the first case and root, minor third, and fifth in the second case.  Other notes get much smaller probabilities, though nonzero, in part because these notes can occur as passing notes, neighbor notes, or other embellishments, and also because they occur as sevenths.  (See, for instance the minor seventh, 10 semitones above the anchor in the middle panel.  By the same token, the major sixth in the same panel, 9 semitones above the anchor, indicates the presence of a half-diminished seventh chord {D, F, A, B}.)





The third category of chord, represented by the bottom panel in Fig. <ref>, has high probabilities for the anchor pitch, the minor third, and the diminished fifth, suggesting a diminished triad, but also for the minor sixth (eight semitones above the anchor), suggesting a dominant seventh.[Note that the anchor in this case—the most probable note—is, from a conventional perspective, the third of the dominant seventh.]  Thus the third category captures both diminished and dominant seventh chords, a sensible marriage since diminished chords often play a role functionally similar to dominant sevenths, particularly in a major mode.  Note that this category can also represent a pure major triad, with notes 0, 3, and 8 semitones above the anchor—a dominant triad without a seventh.

Once the probabilities are determined, we calculate the chords themselves.  Figure <ref> shows a typical example, for Riemenschneider No. 88, along with a human analysis produced by one of us (DT).  Overall, our algorithm does a good but not perfect job of identifying chords.  Out of 45 chords, the calculation gets 40 correct and 5 wrong (highlighted in the figure).  Often the errors are understandable.  For instance, identifying a B dim at the start of measure 5 is certainly reasonable, though most human annotators would consider the G to represent a chord change.  (In part this is because the progression A minor-B diminished-A minor is rare.)  Less obvious is the A minor instead of D minor on the final beat of the same bar: here the algorithm takes the presence of the A-E fifth to suggest an A minor chord, whereas the human considers the E a nonharmonic suspension.

Turning to the keys, our calculation looks for keys of two types, characterized by the probability of occurrence of chords of each of the three types in each possible transposition relative to the anchor pitch.  Again the calculation finds the probabilities first.  The values of the probabilities are shown in Fig. <ref> and correspond closely to our expectations for major and minor keys.  In the first key type, shown in the upper panel of the figure and corresponding to major keys, major triads occur with high probability at zero, five, and seven semitones above the anchor and minor triads—with somewhat lower probabilities—two, four, and nine semitones above the anchor.  Diminished/dominant seventh chords are almost always 11 semitones above the anchor, corresponding to a dominant seventh on the fifth or a diminished on the leading tone (which play similar roles).

In the second key type, shown in the lower panel of Fig. <ref> and corresponding to minor keys, major triads occur mostly on the fifth scale degree, minor triads on the first and fourth, and diminished on the major seventh again.  All of these probabilities agree well with our expectations.

Over the 50 repetitions of the calculation with different random initial values of the parameters, these are the only probability distributions of chords found by our algorithm.  No other distribution was found even once, indicating with high certainty that this is the best fit to the data.

In passing, we note that although our algorithm operates exclusively in chromatic space, these chord probabilities allow us to infer the presence of familiar scales.  Taking the most probable notes in the most probable chords gives us the diatonic scale above the major-key tonic, (C, D, E, F, G, A, B) in C major, and the harmonic minor scale above the minor-key tonic, (A, B, C, D, E, F, G) in A minor.  This suggests the presence of an additional hierarchical scalar level intermediate between chord and chromatic aggregate.



Once the probabilities are determined, the algorithm makes an estimate of the prevailing key at each step in each work and, like the chords, the results are accurate but not perfect.  Again Fig. <ref> gives an example, with disagreements highlighted.  The main error—and one that we see repeatedly in our results—is that our calculation tends to locate key changes slightly earlier than the human analysis does.  For example, the first key change from A minor to G major is marked on the fourth beat of measure 1 by our algorithm but on the first beat of measure 2 by the human analysis.  The algorithm changes keys at the moment the chords become more probable in the new key than in the old, whereas humans typically default toward remaining in a single key, requiring more definitive confirmation that a new key has arrived (e.g., a V-I progression in the new key).  It is debatable whether this is really a disagreement: there are arguably two separate questions here, one about when the key-change process begins, and the other about when it ends; our algorithm answers the first question, whereas human analysts often answer the second.  Furthermore, even expert listeners may differ by a beat or two on exactly when a modulation occurs.

A more significant disagreement is seen at the end of measure 9, where our calculation identifies a modulation to A minor that the human analysis omits.  This also is a fairly common issue: the computer appears willing to identify brief excursions from the current tonal center as modulations, where the human analysis considers the key to be unchanged.  In this case the algorithm sees A minor and B dim chords as implying a temporary key change to A minor.  The issue here is that the algorithm has no notion of “being in one key while briefly using progressions characteristic of another”; it simply calculates the key most likely to produce each sequence of chords.

Once our calculations are complete we can make a quantitative estimate of the accuracy of our method by comparing all chords and keys between the human and computer analyses of all 371 chorales.  For the chords there are a number of ways to perform this comparison, as we discuss in Section <ref>, but the simplest is to take each human-annotated chord and translate it into one of the three categories used by the computer—major, minor, and diminished/dominant 7th—then compare this to the category found at the same point by our algorithm.  When we do this, we find agreement in 87.9% of cases, averaged over all 371 chorales.  Comparing keys is more straightforward, since the computer's two categories clearly correspond to standard major and minor, so no translation between human and machine language is necessary.  We do allow some flexibility about the timing of key changes: we do not penalize cases in which the computer analysis finds the keys correctly but there is disagreement about the timing of the change by up to two steps (as in measures 1 and 2 of Fig. <ref>).  With this provision, we find agreement of keys 84.3% of the time.  These numbers are comparable with the accuracy of previous automated approaches for finding chords and keys, and in the vicinity of what we estimate, intuitively, to be the level of agreement among human annotators.  Exact comparisons with previous work are difficult because other approaches have typically focused on determining either the chords or the keys, but not both, and have used a variety of data sets varying in size, content, and style.  Burgoyne and Saul <cit.> report 83% accuracy in determining chords from pitch-class profiles, a little worse than our result; Sheh and Ellis <cit.> report 75% on a similar task.  Nápoles López  <cit.> summarize a number of previous attempts at key identification, some of which achieve somewhat higher accuracy than ours, though without inferring chords, and in some cases using smaller or simpler data sets or hand-tuned parameter values.






 §.§ Probabilities for transitions between chords and keys

Each of our two hidden Markov models also contains probability parameters that represent the likelihood of transitions, between chords on the one hand and keys on the other.  Figure <ref> depicts the chord-transition probabilities.  The chords, represented by the dots in the figure, are arranged around a circle of fifths and divided into three concentric rings, corresponding to the three types, major triads in the outermost ring, then minor triads and diminished/dominant seventh chords.  Transitions, represented by the arrows, are transposed so as to begin at a chord with a C anchor.  Thus a transition up a fifth is represented by an arrow from C to G; down a fifth is represented by an arrow from C to F.  The varying probabilities themselves are represented by the thickness of the arrows.

Once again, the results found by the algorithm correspond closely to accepted understanding of chord movements, with strong transitions of major chords up and down a fifth (C major  F major and C major  G major) and minor chords down a fifth (C minor  F minor) and a third (C minor  A dim, the latter potentially standing in for F7).  The most frequent diminished-chord transitions are C dim  D major and C dim  D minor, with C dim here potentially standing for A7, so that these correspond to V7  I, V7  i, vii  I, or vii  i.

The corresponding results for transitions between keys are shown in Fig. <ref> and again coincide with intuition, the most frequent changes being between the relative major and minor (C minor  E major and C major  A minor), one step around the circle of fifths (C  G and C  F), and two steps (C major  D minor and C minor  B major).

While these probabilities are broadly consistent with standard music theory, the correspondence is somewhat indirect.  This is because the probabilities are defined in terms of the movement of the anchor—a C major chord goes to F major for instance, or the key of C major goes to D minor—but not relative to any key center.  By contrast, music theoretical concepts are typically defined in a key-relative fashion: the tendency of the dominant chord to progress to a tonic, or for the dominant key to modulate to the relative minor.  (In the case of chord progressions the relevant key is the local key while in the case of modulations it is the global tonic of the entire piece.)  Our model's probabilities thus average over syntactically distinct transitions: the probability of the chord progression C major  F major is an average of I-IV in C, V-I in F, and so on.






§ MACHINE AND HUMAN ANALYSES


In Section <ref> we evaluated our model's performance by translating human annotations into the language of the hidden Markov model.  In this section we do the reverse, translating the model output into standard harmonic labels, with the goal of both examining the algorithm's performance in greater detail and also of producing an annotated corpus potentially useful to other researchers.

There is some disagreement about the optimal terminology for harmonic analysis.  Generally speaking, music theorists have taken one of three approaches: functional theories use flexible chord categories that do not completely determine a chord's note content (e.g., “dominant”); scale-degree theories use Roman numerals that presuppose tertian (“stack of thirds”) structure (e.g., “V7”); and figured-bass theories simply list intervals above a bass scale degree (e.g., “a sixth, fourth, and third above the supertonic”).  These competing approaches complicate the attempt to find a ground truth against which we can compare our model, as the choice of terminology is itself somewhat controversial.

The target language we use here combines Roman numerals indicating a chord's pitch-class content relative to the key with figured-bass symbols that indicate both the bass note and the presence or absence of a seventh.  Table <ref> summarizes the labeling system, which is described more fully in <cit.>.  These categories are sharp, imposing strict limits on which notes can be considered harmonic at any time: in C major the symbol V65 corresponds to a harmony containing the notes {B, D, F, G} with B in the bass.  Notes may be omitted but no other notes can be included within the harmony.  (To be sure, the style permits “nonharmonic” tones—suspensions, neighbors, passing tones, pedals, and anticipations—but these are typically understood as extraneous to the harmony, and subject to additional contrapuntal laws.)  The sharpness of these labels contrasts with the more flexible categories used in jazz and other styles, which permit the addition of non-notated harmonic tones such as ninths, elevenths, and thirteenths.



Our two-level hidden Markov model approach labels both chords and keys with ordered pairs (x, y), where x represents the anchor pitch and y is a probability distribution denoting a class of chords or keys.  For keys, the interpretation is straightforward: the anchor is the tonic note and the two probability distributions of Fig. <ref> give the probabilities of occurrence of various chords in major and minor keys.  As mentioned, this approach matches our ground-truth keys 84.3% of the time.

The interpretation of chords is less obvious.  Each probability distribution in Fig. <ref> encompasses a number of different chords, so there is no straightforward translation from distributions to chord symbols.  An effective strategy is to use the score to disambiguate the analysis, inferring the harmonic symbol that best reflects the sounding notes at each step.  Essentially the hidden Markov model constrains the range of possibilities to the point where simple rules can fill in the remaining details.

In this section we describe three different translations of the hidden Markov model output into chord symbols, ordered by their degree of reliance on the score.  For the interested reader, the results are available for download in the RomanText format, which can be read by software packages such as music21 <cit.>.



  
*Method 1.  Our first translation assigns the first probability distribution in Fig. <ref> to the major triad and the second to the minor triad (ignoring the small probability of a sixth or seventh above the bass).  The third category, however, can correspond to five different chords: diminished triads, diminished sevenths, and half-diminished sevenths with root given by the anchor pitch and dominant sevenths or major triads with root a major third below the anchor.[In a major key these chords are fairly similar and can often be aggregated without much loss of information.  In a minor key, however, they can be quite different: in A minor, the diminished triad B-D-F is typically a predominant progressing to E7, whereas the dominant seventh B-D-F-G is an applied dominant typically progressing to C.]  We resolve this ambiguity by taking as harmonic all those notes present in the score that are either 0, 3, 6, or 8 semitones above the anchor.  This allows for diminished triads, major triads, and dominant sevenths.  Diminished and half-diminished sevenths are identified by first identifying a diminished triad and then the appropriate seventh from the score.  We identify bass notes as the lowest chord tone in the sounding harmony.  The resulting analyses are available in the file “TN2022raw.zip.”





This simple translation matches our ground-truth chords 79% of the time, conditional on getting the key correct.  The lower success rate reflects the fact that this comparison is more demanding than the one we performed in Section <ref>, where we asked only whether the hidden Markov model has assigned each chord to the correct one of the three categories.  Here a correct result requires identifying key, Roman numeral, bass note, and the presence of a seventh.  Table <ref> shows the most common mistakes made by the algorithm, relative to the human ground truth.  In this table “suspension” refers to the misanalysis of a chord produced by a suspension, “sixth/seventh” refers to the analysis of a seventh chord as a triad with added sixth, and “missing seventh” refers to the neglect of a harmonic seventh present in the music.



  
*Method 2.  We now consider several ways to improve the translation, by developing better rules that more accurately reflect human judgment.  As we will see, many of these rules reflect notational conventions rather than substantive syntactical questions.



  
*Keys.  One issue with the key identification is that the hidden Markov model tends to modulate more readily than a human analyst would.  We can improve the accuracy somewhat, relative to the human ground truth, by eliminating very small key regions that are sandwiched between two identical keys, such as a stretch of C major followed by a two-chord stretch of D minor, followed by a stretch of C major again.  This modestly improves the fit between the hidden Markov model and our human corpus, from 84.3 to 85.6%.  It is unclear how the results might be further improved: unlike chord identification, key identification does not seem to be amendable to simple algorithmic fixes. 



  
*Bass notes. In Bach's chorales the bass part normally articulates the true bass note even if it briefly crosses above the tenor.  This is because it was often doubled in a lower octave by the organ.  Thus rather than taking the lowest sounding note to be the bass, we should consider the bass voice to be the carrier of the bass note at all times.  Like the other musical voices, however, the bass sometimes sounds nonharmonic tones not belonging to the chord.  To deal with this issue, we distinguish the step or instant of musical time from the chord span, a maximal contiguous span of steps assigned identical analytical symbols, with the understanding that the true bass note may not be present in every step in a span.  We can define the initial bass note in a given chord span as the first harmonic note in the bass part.  This remains in force until it is displaced by another harmonic note in the bass part within the same span (Fig. <ref>).



  
*Chord labels. We have seen that the third chord category in Fig. <ref> is ambiguous, encompassing a diminished triad, diminished or half-diminished seventh, dominant triad, or dominant seventh a third below.  Close inspection shows that the other two categories are also somewhat ambiguous.  The first, shown at the top of Fig. <ref>, has 0, 4, and 7 semitones as its most probable intervals above the anchor, with smaller probabilities for 9 and 11 semitones.  This means that it could potentially represent {C, E, G}, {C, E, G, B}, {C, E, G, A}, or {C, E, A}.  Similarly, the second category could represent the minor triad {C, E, G}, the minor seventh {C, E, G, B}, or the half-diminished seventh {C, E, G, A}.  Thus, the same set of notes can be analyzed in multiple ways: {A, C, E, G} can be either category 1 or 2 (corresponding to either C major with an added sixth or A minor with an added seventh) and {A, C, E, G} can be analyzed as either category 2 or 3 (corresponding to C minor with an added sixth or A diminished with an added seventh).

As we discuss in the next section, this sort of multivalence is characteristic of some harmonic theories.  Here, however, our goal is to match a particular style of analysis in which chord symbols set a strict limit on the allowed harmonic scale degrees.  To do this, we start by distinguishing the sounding notes occurring within a particular step from the persistent notes, those sounding in every step of the chord span.  We then develop rules that determine whether a given note is harmonic or not, depending on whether it is found in the persistent notes. The best match to the human analysis requires a slightly different strategy for each of our three chord categories.  

For the first (the “major chord”) we consider the major sixth or major seventh to be harmonic only if they belong to the persistent notes, but the minor seventh to be harmonic if it belongs to the sounding notes.  This is because the major sixth and major seventh often appear as unlabeled passing tones, while the minor seventh signals a dominant seventh, which is very likely to be labeled in a human analysis.  The final possibility, the major triad with a minor sixth, is very rare in Bach's chorales.

For the second category (“minor chord”) we consider the sixth to be harmonic only if it belongs to the persistent notes, but the minor seventh to be harmonic if it belongs to the sounding notes.  (The major seventh over a minor triad is again very rare.)  For the third category we consider the minor sixth, diminished seventh, and minor seventh to be harmonic if they occur in the sounding notes.  

These rules reflect both notational conventions, such as when to label a seventh, and substantive beliefs about musical structure, e.g., that a major chord with a minor seventh almost always represents a dominant seventh chord.  Requiring that a note belong to the persistent notes sets a high standard for harmonicity: such notes only count as chord tones if they last for an entire chord span, in which case they are very unlikely to be nonharmonic.



  
*Cadential formulas. A final issue involves idiomatic cadential suspensions that human beings would normally consider nonharmonic, such as (G3, C4, D4)  (G3, B3, D4).  Our algorithm often analyzes this progression as two different chords, a C major (missing the E and with a D in the persistent notes) followed by G major.  We can eliminate most of these errors by hand-coding a small set of standard suspension patterns: for instance, if the persistent notes in a category-1 chord include the major second above the anchor (e.g.,{C, D, G}), and if the music progresses to another category-1 chord up by fifth (e.g., {B, D, G}), then we eliminate the initial chord, extending the second chord backward to cover its predecessor.  These rules are summarized in Table <ref>.  Here it is worth noting that four-chord versions of our model typically settle on suspension formations as a fourth probability distribution, underscoring their importance to the style.



Thus improved, the translation—our second of three—now agrees with the human annotations on about 87% of the chordal assignments, conditional on agreeing about the key.  (The annotations are available in the file “TN2022.zip.”)  We have obtained this agreement at the cost of using a modest amount of knowledge about the underlying musical style: without this it would be difficult to translate the three-category output of the hidden Markov model into the much richer symbol system of standard harmonic analysis.  Importantly, this knowledge is embodied by general rules that are independent of local musical context: relative to key, the same set of notes is always treated the same way. 



  
*Method 3.  Our third translation improves accuracy still further by allowing for “nonharmonic consonances”—triads and seventh chords produced by nonharmonic tones and not part of the harmonic narrative.  (Gottfried Weber called these “apparent chords” <cit.>.)  These nonharmonic consonances are an important source of the holism mentioned in the introduction: choosing between the analyses in Figs. <ref> and <ref> requires a sense of which harmonic progressions are likely to occur, expectations that must be in place prior to analyzing a given passage.  Our third method thus introduces a degree of context-dependency into the analyses.

We can simulate background knowledge using the results of our hidden Markov model analysis.  First we count every two-chord progression in the dataset annotated using our previous method and use these counts to estimate the frequency of each progression.  Second, we encode a basic set of rules about the nonharmonic tones permissible in Bach's style—the “contrapuntal grammar” that was largely inherited from the Renaissance.  (Thus we stipulate, for example, that passing and neighboring tones are out-of-chord notes that are preceded and followed by chord-tones to which they are connected by step.) These rules give us a set of contrapuntally permissible analyses for every step.  Third, whenever we find two eighth-note harmonies occurring within a single quarter-note beat, we ask whether either of those harmonies could be analyzed as nonharmonic.  This gives us a range of possible chord progressions, from which we choose the one whose chord progressions have the highest average frequency in our count of two-chord progressions.  The result can be thought of as the “most likely analysis,” given harmonic expectations based on the raw hidden Markov model output and the additional rules described here.

Figure <ref> shows how the algorithm treats four superficially similar passages.  Notes marked “p?” are flagged as potentially nonharmonic and we consider one reading in which the note is a passing tone and another in which it is harmonic.  The numbers underneath each two-chord progression show the number of occurrences of this progression in our previous analysis.  The algorithm chooses the reading with the highest average frequency, shown in bold and surrounded by a box.  This sometimes requires treating superficially similar passages in different ways.  Our third approach achieves this context-dependency in a rule-governed way, relying on first-pass statistics to generate a most-probable interpretation from among a set of contrapuntally permissible alternatives.  By contrast, human beings make context-dependent judgments whose origin and justification can be hard to understand.



This pruning of the analysis improves accuracy to the point where we match the human chord symbols 90% of the time, assuming agreement about the key.  The resulting data set, which is accurate enough for a range of music-theoretical purposes, is available as “TNpruned.zip.”  The high degree of agreement with the human analysis demonstrates that analytical competence does not require much training or experience.  Instead it can be learned from a suitable corpus of music, along with a modest amount of style-specific knowledge.  The subtle analytical judgments that go into Fig. <ref> can be simulated with relatively presupposition-free methods, making almost no assumptions about the specific harmonic syntax of the music.




§ TESTS OF WESTERN HARMONIC THEORY


The most obvious use for these datasets is as a neutral testing-ground for theories of Western harmonic structure.  The biggest question here is whether traditional music has a chord-to-chord harmonic structure akin to that of a first-order Markov model, articulated by rules of the form “V chords generally go to I chords but not IV chords.”  Many theorists believed such structure exists, including Rameau, Weber, Riemann, Piston, Kostka, Payne, and other contemporary pedagogues.  A more recent alternative is that harmony is characterized by a hierarchical structure rooted in melodic laws—a view originating with Heinrich Schenker <cit.> and endorsed by post-Schenkerians such as Lerdahl and Jackendoff <cit.>.  In recent years, this alternative perspective has been incorporated into introductory textbooks <cit.>.

Our data provide strong evidence for the existence of chord-to-chord syntax.  The most popular chords and transitions found in the major-mode passages of the chorales can be visualized as shown in Fig. <ref>.  In this figure, symbol size corresponds to frequency while arrow size indicates the relative frequency of the transitions.[The figure shows Roman numerals accounting for at least 5% of all chords in the chorales' major-key passages (disregarding inversion), along with all transitions accounting for at least 0.75% of all transitions.] This representation is reminiscent of the prescriptive maps found throughout the theoretical and pedagogical literature <cit.>.  While the presence of this chord-to-chord structure does rule out additional hierarchical organization, it does suggest that any such organization must either produce or complement the music's nonhierarchical organization.  This is significant insofar as Schenkerian theorists have sometimes rejected <cit.> or neglected <cit.> this sort of chord-to-chord structure.

Figure <ref> provides the same information for modulations, transposing all major-key pieces to C major and all minor-key pieces to A minor.[In making this graph we selected only those chorales that start and end in the same key, transposing them to so that the starting key was either C major or A minor.] Here the most common modulations are oscillatory: keys tend to move from the tonic to a subsidiary key and back, rather than on longer journeys through multiple subsidiary keys.  This is likely a feature of the chorales in particular, and not characteristic of genres such as the classical sonata.  In this respect the modulatory grammar is unlike the chord-to-chord grammar, which describes a much wider range of harmonic practices—indeed, a good portion of eighteenth- and nineteenth-century music.





Another longstanding area of theoretical controversy is the principles governing “doubling,” or the duplication of elements in four-part triadic harmony.  Aarden and von Hippel have surveyed the astonishing range of disagreement surrounding this subject <cit.>.  Table <ref> shows which notes Bach tends to double in major mode, as extracted from our automated analysis.  In all cases we find that the bass is the most frequently doubled, unless it is a leading tone in which case it is not doubled.  Note for instance that the bass of the first-inversion tonic chord is frequently doubled despite longstanding injunctions to the contrary.  For all the controversy surrounding the subject, Bach's practice is fairly clear.



There are other potential analytical uses for our data.  Combined with the musical score, our analysis allows one to algorithmically catalog nonharmonic tones (Table <ref>), identify the specific voice leadings used in a given harmonic progression, explore the resolution or nonresolution of tendency tones (e.g., leading tones and sevenths), detect parallel perfect intervals either on the musical surface or masked by nonharmonic tones, and search for examples of various phenomena of interest (e.g., unusual chord progressions).  In this way our data provides a quantitative tool for grounding theoretical assertions that have traditionally been supported intuitively.

Our work also suggests some more general theoretical lessons.  Theories of Western harmony are subject to a tradeoff between the determinacy of chord categories and the complexity of chord grammar.  This issue dates back to Jean-Philippe Rameau (1683–1784), celebrated as “the Newton of music” and in many ways the progenitor of contemporary harmonic theory.  Rameau is sometimes credited with being the inventor of “inversional equivalence,” the thesis that pitch-class content determines chordal identity.  From this perspective the notes {C, E, G} always constitute a C-major chord no matter how they are arranged in register.  When pitch classes are labeled relative to a tonal center, this implies that a chord's scale-degree content determines its harmonic category.  In fact, however, Rameau was not a pure scale-degree theorist, and his chord-categories had a flexible relation to pitch-class content.  For example, his 1722 Treatise on Harmony explained the deceptive cadence V7-vi as a descending-fifth progression in which the second chord was represented not by the root, third, and fifth but by root, third, and sixth <cit.>.[Rameau later abandoned this analysis of deceptive cadences, although followers such as d’Alembert continued to uphold it <cit.>.  However, Rameau never abandoned the analogous analysis of the predominant chord with added sixth, arguing that {F, A, C, D} in C major could have a fundamental bass of either F or D, or indeed both.]  Thus on this view the notes {A, C, E} could represent either a chord whose root is A or a chord whose root is C.



Here Rameau opted for a simpler harmonic grammar at the expense of a more complex categorization scheme: he described both the authentic and deceptive progressions with a single rule, “G7 chords progress down by a fifth to C chords.”  But this simplification came at the cost of a categorization scheme in which chordal identity could not be identified from note content alone.  Instead, context determined whether the notes {A, C, E} represented C major or A minor.  This can be considered a functional approach to chord categorization, as a chord's label depends on its local behavior.  Functional approaches can be controversial insofar as they identify chords on the basis of pre-existing theory: {A, C, E} might be identified as a kind of C chord when preceded by G7 but a kind of A chord when preceded by E7.  Thus the claim that chords descend by a fifth is not simply a hypothesis supported by data, but a rule that regulates analysis.  

Something similar can be said of Hugo Riemann, one of the most influential nineteenth-century theorists.  Riemann considered the basic chords to be tonic, subdominant, and dominant, or I, IV and V in Roman numeral notation, arguing that tonal phrases typically articulate a series of I-IV-V-I progressions <cit.>.  However these basic chords were subject to category-preserving transformations: the “parallel” transform replaces a chordal fifth with the sixth above the root, and the “leittonwechsel” replaces a chordal root with the note a semitone below.  This allowed Riemann to describe both G7  C and G7  Am as instances of a single underlying dominant-to-tonic pattern.  In G7  C, the tonic is represented by its primary triad, while in G7  Am it is represented by the primary triad's parallel variant.  Once again, the result is an ambiguity at the level of chord categories: the chord {A, C, E} can represent either a tonic (with A replacing the fifth of {C, E, G}) or subdominant (with E replacing the root of {F, A, C}).  

Our unsupervised hidden Markov model learns a series flexible chord categories that resemble those of Rameau and Riemann.  Like those theorists, it groups the diminished triad {B, D, F} with the dominant seventh {G, B, D, F} into a single category.  Like Rameau, it analyzes the notes {D, F, A, C} sometimes as an F major chord with added sixth and sometimes as a D minor chord with added seventh.  This is not because it considers these chords to “sound similar,” nor because it is committed to metaphysical claims about the priority of descending-fifth progressions.  Rather it does so solely for reasons of information-theoretic efficiency, mutually adjusting probability distributions and transition matrices so as to maximize its ability to predict the future.  What is surprising is that this process reproduces results familiar from the history of music theory.  Those claims have sometimes been denigrated as methodologically disreputable <cit.>, but our work suggests they can be given an algorithmic foundation: musical “function” can be understood information theoretically, as the maximally parsimonious description of the input data.[This result is consistent with work of Jacoby, Tishby, and Tymoczko <cit.>, who use the “information bottleneck” to cluster Roman numerals into categories, recovering something resembling traditional functional theory.]

Contemporary North American theory tends to prefer more determinate chord labels at the cost of a more complex harmonic grammar: in this approach, {C, E, G} is a C-major chord and {A, C, E} is an A minor chord regardless of context.  This in turn requires a more complex set of harmonic rules according to which G7 progresses most often to {C, E, G} (the “authentic” resolution) and less often to {A, C, E} (the “deceptive” resolution).  This approach generally allows for a fine-grained description of musical practice, at the cost of being less enlightening about why G7 progresses as it does.[There are some circumstances where contemporary North American theory uses functional rather than scale-degree labels.  The most common involves cadential _4^6 chords such as {G3, C4, E4} in C major.  Where a pure scale-degree theory would label this chord I64, many contemporary theorists use V64 to record the fact that it typically embellishes and precedes a dominant.  Other circumstances include the assimilation of vii to V (as a “dominant seventh lacking its root” <cit.>), the distinction between “plagal” and “predominant” functions of IV <cit.>, and the correct labeling of chords like {G3, B3, E4} in C major (which can be considered a dominant or a mediant <cit.>).]  In standard Roman-numeral theory this is just a fact about the grammar, whereas for Riemann and Rameau {A, C, E} is portrayed as being intrinsically similar to {C, E, G}.  In this way, the different approaches serve somewhat different purposes: the general categories of Rameau and Riemann provide a partial explanation for tonal practice, one that was useful for composers working in the style.  The more precise categories of contemporary theory, by contrast, may help scholars articulate a more detailed grammar of historical practice.

This suggests a pluralistic picture in which there is more than one way to describe harmony.  The most specific simply represents harmonies as collections of scale-degrees above a bass.  This is likely how the earliest functional composers thought—musicians like Corelli, Vivaldi, Handel, and J. S. Bach, who wrote before the widespread acceptance of inversional equivalence.  More general are the theory-laden harmonic categories of North American theory, Roman numerals presupposing stack-of-thirds harmonic structure.  These facilitate the description of general patterns that can appear over a variety of different bass notes, such as “V chords generally progress to I chords.”  The even more general theories of Rameau and Riemann use still more flexible categories that help explain the principles underlying the grammar.  Our hidden Markov model natively produces something very much like this last approach, though as we have seen it can be translated into Roman numerals.

These different approaches each have their partisans among contemporary theorists: Gjerdingen <cit.> and Holtmeier <cit.> have advocated for figured-bass representations; Tymoczko <cit.> for Roman numerals; and Agmon <cit.> for harmonic functions.  Yet the different analytical languages are arguably optimized for different musical purposes.  Instead of asking which approach is best, it might be better to ask which is better for a particular application.  In particular, it seems likely that more flexible categories are better suited for introductory or creative contexts, where the goal is to apply broad principles in novel contexts, while the more fine-grained language may be better for detailed syntactical investigation.  In other words, introductory textbooks might benefit from the functional approach, while more specialized investigations might need the language of Roman numerals.



§ CONCLUSIONS



There is at best a tenuous consensus about the basic concepts of standard harmonic theory.  Schenker and many of his successors deny that harmonies obey any local chord-to-chord grammar, proposing that music is instead governed by purely linear laws.  Functional theorists follow Rameau and Riemann in promoting flexible categories that can be represented by many different triads.  Schoenbergians, along with Ian Quinn and Christopher White, challenge the distinction between harmonic and nonharmonic—–the very picture of tertian harmonies decorated by “merely linear” tones.  And figured-bass theorists from J. S. Bach to Holtmeier and Gjerdingen have advocated for figured-bass symbols.

Against this background of widespread disagreement, it is remarkable that our largely assumption-free computation manages to reproduce so many familiar features of traditional theory.  Its two key categories are directly interpretable as major and minor.  Its three chordal categories are broadly synonymous with “major,” “minor” and “diminished or dominant.”  Yet at a more fine-grained level these categories admit a degree of flexibility that recall the function-theoretical ideas of Riemann and Rameau: the notes {F, A, C, D} can be interpreted either as a D minor chord with added seventh or an F major chord with added sixth.  We have seen that it is possible to translate these flexible categories into the more precise terminology of North American theory, where added sixths are unknown and there is a sharp distinction between D minor and F major.  Indeed, our automated analyses largely accord with the human ground-truth annotations.  

Our work results in several datasets that can be used for further music-theoretical and musicological inquiry.  Though perhaps less accurate than purely human analytical corpora, they are also less subjective, being largely independent of human biases: thus we can be fairly sure that regularities appearing in these data are not a mere byproduct of analytical preconceptions.  Furthermore, all the assumptions that go into producing the corpora are clearly spelled out, embodied in the computer programs we have used. Though this paper has not focused on detailed corpus study, we have already observed some non-obvious features of musical structure, such as oscillatory rather than progressive modulatory norms (Fig. <ref>).  One project for future work is to explore these datasets more thoroughly.

Another is to develop methods that deal with a broader range of music.  Bach's chorales are unusual in having a fixed number of voices, avoiding rests and incomplete chords, using an extremely regular harmonic rhythm, and explicitly delineating voices and phrases in their notation.   Classical-style harmonies, by contrast, change irregularly, are often incomplete, and feature a variable number of voices not explicitly embodied in the notation.  It is not obvious that we should be able to analyze this music with largely assumption-free methods.  Instead, it may be necessary to make further assumptions about classical syntax.  Solving this problem, however, would open the door to a new era for music theory and music history, one that exploits big data and large-scale statistical analysis.





6cm0.5pt









§ RESULTS OF INDIVIDUAL RUNS


Our calculations consist of two separate parts: the analysis of notes to identify chords and the analysis of the resulting chords to identify keys.  Over multiple repetitions of the calculation the second step—the identification of keys—reliably converges to only a single solution with key classes corresponding to major and minor keys.  No other solution is ever found.  The first step, in which the chords are identified, converges most often to the solution described in Section <ref>, with three chord classes corresponding to major, minor, and diminished/dominant seventh chords.  In 37 of the 50 runs we performed—roughly 75% of the time—we found these chord classes.  In the remaining 13 runs we found other solutions, typically with two of the conventional chords plus one alternate.  A common alternate, for instance, is the dominant seventh with suspended fourth {C, F, G, B}, which occurs in place of the dominant seventh.

These findings, however, somewhat understate our results, in that the alternate solutions, while occurring a moderate fraction of the time, do so with hugely diminished probability.  The probability of these solutions, as quantified by the data likelihood P(𝒞|θ) (see Appendix <ref>), is much lower than for the major/minor/dominant solution.  For example, the highest likelihood solution found overall—which has the major/minor/dominant form—has a data log-likelihood of -118926.5 (base e).  The best of the alternate solutions has a log-likelihood of -121413.4.  This means that even this best alternate is lower in probability by the stupendously large factor e^-2486.9≃ 10^1080.  Thus the major/minor/dominant classification is very heavily favored in our analysis.

Figure <ref> shows the note probabilities for the chord classes in the top 20 of our calculation runs, those with the highest data likelihoods—each panel is the equivalent for the given run of Fig. <ref> in the main text.  Of these twenty plots, every single one corresponds to the major/minor/dominant solution.  In fact, all of the top 31 runs give this solution; one has to go down to the 32nd-best run out of 50 to find one of the alternates.  Again the results speak clearly: the standard major/minor/dominant combination is the best way to describe the structure of these data.




§ THE BAUM-WELCH ALGORITHM


The Baum-Welch algorithm is an expectation-maximization (EM) algorithm for estimating the optimal values of the parameters of a hidden Markov model.  We describe the process here in the language of keys and chords, but the same algorithm applies to any hidden Markov model.

Suppose we have a corpus of N works labeled by i=1… N, with the ith work having M_i chords C_i = ( c^i_1, …, c^i_M_i ).  The parameters of the hidden Markov model, which are the same for every work, are the transition probabilities β = {β_kk'} and chord probabilities ν = {ν_kc} as defined in Section <ref>, plus the prior probabilities π = {π_k } that the first key in a work is k.  (In our calculations we assume π_k to be the same for all keys, so it plays no role, but we include it in the derivations for the sake of completeness.)  Denoting the parameters collectively by θ = {π, β, ν}, the probability within the model of a sequence of keys K_i = ( k^i_1,…,k^i_M_i ) for the M_i steps of work i is

    P(K_i|θ) = P(k^i_1|θ) ∏_t=2^M_i P(k^i_t|k^i_t-1,θ)
      = π_k^i_1∏_t=2^M_iβ_k^i_t-1k^i_t,

and the probability of the chords given the sequence of keys is

    P(C_i|K_i,θ) = ∏_t=1^M_i P(c^i_t|k^i_t)
      = ∏_t=1^M_iν_k^i_tc^i_t.

Hence the probability of both the keys and the chords is

    P(K_i,C_i|θ) = P(C_i|K_i,θ) P(K_i|θ)
      = π_k^i_1ν_k^i_1c^i_1∏_t=2^M_iβ_k^i_t-1k^i_tν_k^i_tc^i_t

and the probability for the entire corpus, with keys = { K_1,…, K_N } and chords = { C_1,…, C_N }, is the product of this expression over all works:

    P(,|θ) = ∏_i=1^N P(K_i,C_i|θ)
      = ∏_i=1^N π_k^i_1ν_k^i_1c^i_1∏_t=2^M_iβ_k^i_t-1k^i_tν_k^i_tc^i_t.


Using Bayes rule we write the probability of the unknowns  and θ as

    P(,θ|) = P(,|θ) P(θ) P(),

which for given chords  is proportional to P(,|θ) assuming a uniform prior P(θ)= on the parameters θ.

We find our best estimate of the parameter values θ by maximizing the log-probability

    log P(θ|) = log∑_ P(,θ|) ∝log∑_ P(,|θ).

We now apply Jensen's inequality, which says that for any set of positive quantities x_i,q_i>0 with i=1… n and ∑_i=1^n q_i = 1 we have

    log∑_i x_i ≥∑_i q_i logx_i q_i.

Applied to Eq. (<ref>) this tells us that

    log∑_ P(,|θ) ≥∑_ q() logP(,|θ) q(),

where q() is any probability distribution over  satisfying ∑_ q() = 1.  The exact equality is achieved when

    q() = P(,|θ)∑_ P(,|θ),

which necessarily maximizes the right-hand side of (<ref>), and a subsequent maximization with respect to θ will then find the optimal parameter values.  Put another way, a double maximization of the right-hand side gets us the answer we want.  In the EM approach we perform this double maximization by simply maximizing with respect to q() with θ held constant, then with respect to θ with q() constant, and repeating until we reach convergence.  Maximization with respect to q() is achieved by applying Eq. (<ref>) as we have said.  Maximization with respect to θ is achieved by differentiating the right-hand side of (<ref>) and setting the result to zero.

For instance, the derivative with respect to β_kk' takes the form

    ∂∂β_kk'[ ∑_ q() log P(,|θ) - λ∑_l β_kl] = 0,

where we have enforced the necessary constraint ∑_l β_kl = 1 with a Lagrange multiplier λ.  Substituting from Eq. (<ref>) for P(,|θ), taking the log, and performing the derivative, we then find that

    β_kk' = ∑_i=1^N ∑_t=1^M_i-1 q^i_t,t+1(k,k')∑_k'∑_i=1^N ∑_t=1^M_i-1 q^i_t,t+1(k,k'),

with

    q^i_t,t+1(k,k') = ∑_ q() δ_k^i_t,kδ_k^i_t+1,k',

where δ_ij is the Kronecker delta, which is 1 when i=j and 0 otherwise.  A similar derivative with respect to ν_kc, enforcing the constraint ∑_c ν_kc = 1 with another Lagrange multiplier, gives

    ν_kc = ∑_i=1^N ∑_t=1^M_i q^i_t(k) δ_c,c^i_t∑_i=1^N ∑_t=1^M_i q^i_t(k),

where

    q^i_t,t+1(k,k') = ∑_ q() δ_k^i_t,kδ_k^i_t+1,k'.


The EM algorithm involves choosing an initial set of parameter values θ = {π, β, ν} and applying Eq. (<ref>) to calculate q(), then Eqs. (<ref>), (<ref>), (<ref>), and (<ref>) to calculate new values of the parameters and iterating until the parameter values converge.  It can be proved that this procedure always converges to a local optimum of the parameter values, but not to the global optimum.  In practice, therefore, one typically runs the algorithm several times and, if different runs converge to different optima, accepts the solution with the highest log-probability, Eq. (<ref>).

Equations (<ref>) and (<ref>) are easily implemented.  The tricky part of the calculation is evaluating the probability distribution q() of Eq. (<ref>) and from it the quantities q^i_t(k) and q^i_t,t+1(k,k').  A naive version of this calculation would involve performing the sum in the denominator of (<ref>), which would be computationally exorbitant, but luckily there exists an efficient algorithm for computing q^i_t(k) and q^i_t,t+1(k,k') directly, the forward-backward algorithm.




 §.§ The forward-backward algorithm

The quantity q^i_t(k) defined in Eq. (<ref>) is equal to the probability that the prevailing key is k at step t of work i.  Similarly q^i_t,t+1(k,k') is the probability that the keys at consecutive steps t and t+1 are k and k'.  The forward-backward algorithm computes these probabilities using a dynamic programming approach.  Focusing on a single work i, we write the joint probability of the key and chord at the first step as

    P(k_1,c_1|θ) = P(k_1|θ) P(c_1|k_1,θ) = π_k_1ν_k_1c_1,

where we temporarily suppress the superscript i for the sake of clarity, since it will be the same throughout.

Given this starting point, we can now calculate the set of subsequent probabilities

    P(k_t,c_1… c_t) = ∑_k_t-1 P(k_t-1,c_1… c_t-1)
      P(k_t|k_t-1) P(c_t|k_t),

where we have also suppressed θ.  Defining the shorthand a_t(k_t) = P(k_t,c_1… c_t), this can then be written as

    a_t(k) = ν_kc_t∑_k'β_k'k a_t-1(k').

Equation (<ref>) tells us that a_1(k) = π_k ν_kc_1, then applying (<ref>) repeatedly allows us to quickly calculate a_t(k) for all keys k and steps t.  This is the “forward” half of the forward-backward algorithm.

The backward half defines a complementary set of quantities b_t(k_t) = P(c_t+1… c_M|k_t), where M is the total number of steps in the work.  We write

    P(c_t… c_M|k_t-1) = ∑_k_t P(c_t+1… c_M|k_t) P(k_t|k_t-1) P(c_t|k_t),

which can be written in the form

    b_t-1(k) = ∑_k'ν_k'c_tβ_kk' b_t(k'),

from which, starting with b_M(k)=1 for all k, we can quickly calculate all b_t(k).

Given the complete set of a_t(k) and b_t(k), we now have

    P(k_t,C) = P(k_t,c_1… c_t) P(c_t+1… c_M|k_t) = a_t(k_t) b_t(k_t),

where C = ( c_1,…,c_M ) as previously.  Thus

    P(k_t|C) = P(k_t,C) P(C) = P(k_t,C)∑_k_t P(k_t,C),

or equivalently

    q^i_t(k) = a_t(k) b_t(k)∑_k a_t(k) b_t(k).

Repeating this calculation for each work i we then have the full set of probabilities q^i_t(k).

Similarly we can write

    P(k_t,k_t+1,C)    = P(k_t,c_1… c_t) P(c_t+2… c_M|k_t+1) 
       × P(k_t+1|k_t) P(c_t+1|k_t+1) 
       = a_t(k_t) b_t+1(k_t+1) β_k_tk_t+1ν_k_t+1c_t+1,

so that

    P(k_t,k_t+1|C) = P(k_t,k_t+1,C) P(C)
      = P(k_t,k_t+1,C)∑_k_t,k_t+1 P(k_t,k_t+1,C),

or

    q_t,t+1(k,k') = a_t(k) b_t+1(k') β_kk'ν_k'c_t+1∑_kk' a_t(k) b_t+1(k') β_kk'ν_k'c_t+1.

Between them, Eqs. (<ref>) and (<ref>) now allow us to complete our EM algorithm and calculate the optimal values of the parameters.




§ THE VITERBI ALGORITHM


Using the Baum-Welch algorithm we can calculate the optimal values of the probability parameters β_kk' and ν_kc.  Given these values we can then calculate the best estimate of the hidden variables using the Viterbi algorithm.  Again we use the language of keys and chords, in which case the hidden variables are the keys.

We define x_t(k_t) to be the probability that the key is k_t on step t and the chords up to that point are c_1… c_t, given that the preceding keys k_1… k_t-1 take their most likely (i.e. highest probability) values and given the values of the parameters θ.  That is,

    x_t(k_t) = max_k_1… k_t-1 P(k_1… k_t,c_1… c_t|θ).

We will henceforth suppress θ, since the parameters are fixed throughout this part of the calculation.

Now we note that

    P(k_1… k_t,c_1… c_t) 
            = P(k_1… k_t-1,c_1… c_t-1) P(k_t|k_t-1) P(c_t|k_t),

and hence

    max_k_1… k_t-1 P(k_1… k_t,c_1… c_t|θ) 
        = max_k_t-1max_k_1… k_t-2
               P(k_1… k_t-1,c_1… c_t-1) P(k_t|k_t-1) P(c_t|k_t),

or

    x_t(k) = ν_kc_tmax_k'β_k'k x_t-1(k').

Starting from x_1(k) = π_k ν_kc_1, we can use Eq. (<ref>) to calculate the full set of quantities x_t(k) for all t and k.

The last of these quantities gives us the probability of k_M when all other keys take their most likely values, so the most likely value of k_M is simply

    k_M = _k x_M(k),

where _k x(k) indicates the value of k that gives the largest value of x.  Then the most likely values of the remaining keys can be calculated in reverse order from

    k_t = _k β_kk_t+1 x_t(k).

The end result is the complete sequence of keys k_1… k_M that has the highest probability for the given values of the probability parameters.




§ IMPLEMENTATION AND RUNNING TIME


Implementation of the Baum-Welch algorithm is a straightforward numerical realization of the equations.  We choose random initial values for each of the probability parameters β_kk' and ν_kc, and then for each work we repeatedly apply Eq. (<ref>) to calculate the quantities a_t(k), Eq. (<ref>) to calculate the b_t(k), and then Eqs. (<ref>) and (<ref>) to calculate q^i_t(k) and q^i_t,t+1(k,k').  Armed with these we then use Eqs. (<ref>) and (<ref>) to calculate updated values of β_kk' and ν_kc, repeating the whole process until the values converge within a predetermined tolerance.

A technical issue arises with applications to longer works because the quantities a_t(k) and b_k(t) can become very small and underflow the numerical resolution of the computer.  To avoid this, we normalize these quantities at each step within a work.  For instance, after calculating a_t(k) in Eq. (<ref>) we compute a normalized version

    ã_t(k) = a_t(k) A_t,

with

    A_t = ∑_k a_t(k),

then use this version in the next step of Eq. (<ref>).  We also perform a similar normalization of b_k(t).  These changes have no effect on Eqs. (<ref>) and (<ref>) since any overall rescaling of a_t(k) or b_t(k) cancels in numerator and denominator of both equations, but they prevent underflows by ensuring that a_t(k) and b_t(k) both sum to 1, so that their average values cannot become arbitrarily small even for long works.

Implementation of the Viterbi algorithm is a similarly straightforward realization of Eq. (<ref>).  No iteration is needed here: one applies the algorithm just once to compute the optimal key sequence for each work.  Potential underflows can be prevented by normalizing the values of x_t(k) while having no effect on the keys computed in Eq. (<ref>).

Most of the running time is taken up by the calculation of the optimal parameter values using the Baum-Welch algorithm, and most of that time is taken by the forward-backward algorithm.  If there are R distinct keys (where R=24 in our case) and if the total number of steps in all works in the corpus is T = ∑_i=1^N M_i, then the calculation of a_t(k) from Eq. (<ref>) for all t and k takes time (TR^2), and similarly for the calculation of b_k(t).  This time gets multiplied by the number of iterations of the Baum-Welch algorithm needed to achieve convergence.  In our calculations we iterate until the values of the parameters change by less than one part in 10 million on each iteration, and this point is typically reached after about 100 iterations.  Total running time in practice for the full corpus of Bach chorales is about five minutes on a standard laptop (circa 2024).

For practical applications to finding keys and chords in written music, however, one would not need to run the Baum-Welch algorithm at all.  In such applications one would normally pre-compute the optimal values of the probability parameters then apply those to infer keys and chords for any given work.  This would require running only the Viterbi algorithm for each work of interest, which is very fast, taking only a fraction of a second on a typical computer.




§ CALCULATION OF THE DATA LIKELIHOOD


The Baum-Welch algorithm maximizes the probability of the model parameters given the data, Eq. (<ref>), but is not guaranteed to find a global maximum.  Like all expectation-maximization algorithms the Baum-Welch algorithm may erroneously converge to a local maximum of the probability.  To guard against this possibility we perform multiple runs of the algorithm with different random initial conditions and then use the results from the run that gives the highest probability.  This does not guarantee that we find the global maximum, but in practice we see the same best result occurring repeatedly, which strongly suggests that there is no better result to be found.

Calculating the value of the posterior probability P(θ|𝒞) is equivalent to calculating the data likelihood P(𝒞|θ), since we assume a uniform prior on the parameters θ, and the likelihood can be calculated conveniently from results we already have.  As described in Appendix <ref>, we calculate as part of the Baum-Welch algorithm the quantities ã_t(k) defined in Eq. (<ref>).  Applying this equation repeatedly we have

    a_M(k_M) = P(k_M,c_1… c_M|θ) = ã_M(k_M) ∏_t=1^M A_t

with A_t as in Eq. (<ref>) and M being the total number of steps in the work.  Summing over k_M, we then have

    P(c_1… c_M|θ)    = ∑_k_M P(k_M,c_1… c_t|θ) 
       = ∑_k_Mã_M(k_M) ∏_t=1^M A_t
      = ∏_t=1^M A_t ,

where we have made use of the fact that ∑_k_Mã_M(k_M) = 1 because of the normalization in Eq. (<ref>).  The total likelihood P(𝒞|θ) of all works in the entire data set is then the product of this quantity over all works.


10
url<#>1urlprefixURL 

WEBER1846
G. Weber, Theory of Musical Composition. Wilkins, Carter, and Company,
  and O. C. B. Carter, Boston ([1817–1821] 1846).

TAOM
D. Tymoczko, Tonality: An Owner's Manual. Oxford University Press, New
  York (2023).

Micchi
G. Micchi, M. Gotham, and M. Giraud, Not all roads lead to Rome: Pitch
  representation and model architecture for automatic harmonic analysis.
  Transactions of the International Society for Music Information
  Retrieval 3, 42–54 (2020).

QUINNWHITE
I. Quinn and C. White, Chord context and harmonic function in tonal music.
  Music Theory Spectrum 40, 314–337 (2018).

NOLAND2006
K. C. Noland and M. B. Sandler, Key estimation using a hidden Markov model.
  In Proceedings of the 7th International Conference on Music
  Information Retrieval (ISMIR 2006), pp. 121–126 (2006).

NAPOLESLOPEZ2019
N. Nápoles López, C. Arthur, and I. Fujinaga, Key-finding based on a hidden
  Markov model and key profiles. In Proceedings of the 6th
  International Conference on Digital Libraries for Musicology, pp. 33–37
  (2019).

QUINN
I. Quinn, Tonal harmony. In A. Rehding and S. Rings (eds.), The Oxford
  Handbook of Critical Concepts in Music Theory, pp. 467–497, Oxford
  University Press, New York (2018).

GJERDINGEN2007
R. Gjerdingen, Music in the Galant Style. Oxford University Press, New
  York (2007).

BURGOYNE2005
J. Burgoyne and L. Saul, Learning harmonic relationships in digital audio with
  Dirichlet-based hidden Markov models. In Proceedings of the 6th
  International Conference on Music Information Retrieval (ISMIR 2005), pp.
  438–443 (2005).

FUJISHIMA1999
T. Fujishima, Realtime chord recognition of musical sound: A system using
  common Lisp music. In Proceedings of the International Computer
  Music Conference, pp. 464–467 (1999).

CHAI2005
W. Chai and B. Vercoe, Detection of key change in classical piano music. In
  Proceedings of the 6th International Conference on Music Information
  Retrieval (ISMIR 2005), pp. 468–473 (2005).

SHEH2003
A. Sheh and D. P. Ellis, Chord segmentation and recognition using EM-trained
  hidden Markov models. In Proceedings of the 4th Internation
  Conference on Music Information Retrieval (ISMIR 2003), pp. 185–191 (2003).

ALBRECHTSHANAHAN
J. Albrecht and D. Shanahan, The use of large corpora to train a new type of
  key-finding algorithm: An improved treatment of the minor mode. Music
  Perception 31, 59–67 (2013).

napoleslopez21augmentednet
N. Nápoles López, M. Gotham, and I. Fujinaga, AugmentedNet: A Roman
  numeral analysis network with synthetic training examples and additional
  tonal tasks. In Proceedings of the 22nd International Conference on
  Music Information Retrieval, pp. 404–411 (2021).

ZUCCHINI2021
W. Zucchini, I. L. MacDonald, and R. Langrock, Hidden Markov Models for
  Time Series. Chapman & Hall, London (2021).

TYMOCZKO2019
D. Tymoczko, M. Gotham, M. S. Cuthbert, and C. Ariza, The RomanText format: A
  flexible and standard method for representing Roman numeral analyses. In
  A. Flexer, G. Peeters, J. Urbano, and A. Volk (eds.), Proceedings of
  the 20th International Conference on Music Information Retrieval, pp.
  123–129 (2019).

CUTHBERT2010
M. S. Cuthbert and C. Ariza, Music21: A toolkit for computer-aided musicology
  and symbolic music data. In J. S. Downie and R. C. Veltkamp (eds.),
  Proceedings of the 11th International Conference on Music Information
  Retrieval, pp. 637–642 (2010).

SCHENKER1997
H. Schenker, The Masterwork in Music: Volume 3, 1930: A Yearbook.
  Cambridge University Press, Cambridge ((1930) 1997).

JACKENDOFF1983
F. Lerdahl and R. Jackendoff, A Generative Theory of Tonal Music. MIT
  Press, Cambridge, MA (1983).

LAITZ2008
S. Laitz, The Complete Musician. Oxford University Press, New York
  (2008).

CLENDINNING2016
J. P. Clendinning and E. W. Marvin, The Musician’s Guide to Theory
  and Analysis. Norton, New York, 3rd edition (2016).

PISTON1941
W. Piston, Harmony. Norton, New York (1941).

KOSTKA2003
S. Kostka and D. Payne, Tonal Harmony. Alfred A. Knopf, New York, 4th
  edition (2016).

TYMOCZKO2011
D. Tymoczko, A Geometry of Music. Oxford University Press, New York
  (2011).

LERDAHL2001
F. Lerdahl, Tonal Pitch Space. Oxford University Press, New York
  (2001).

AARDEN2004
B. Aarden and P. T. von Hippel, Rules for chord doubling (and spacing): Which
  ones do we need? Music Theory Online 10(2) (2004).

RAMEAU1971
J. P. Rameau, Treatise on Harmony. Dover, New York ([1722] 1971).

CHRISTENSEN1993
T. Christensen, Rameau and Musical Thought in the Enlightenment.
  Cambridge University Press, Cambridge (1993).

BURNHAM1992
S. Burnham, Method and motivation in Hugo Riemann's History of Harmonic
  Theory. Music Theory Spectrum 14(1), 1–14 (1992).

JACOBY2015
N. Jacoby, N. Tishby, and D. Tymoczko, An information theoretical approach to
  chord classification. Journal of New Music Research 44,
  219–244 (2015).

DAHLHAUS1990
C. Dahlhaus, Studies in the Origins of Harmonic Tonality. Princeton
  University Press, Princeton (1990).

SWINDEN2005
K. Swinden, When functions collide: Aspects of plural function in chromatic
  music. Music Theory Spectrum 27, 249–282 (2005).

ROTHGEB1996
J. Rothgeb, Re: Eytan Agmon on functional theory. Music Theory
  Online 2(1) (1996).

HOLTMEIER2007
L. Holtmeier, Heinichen, Rameau, and the Italian thoroughbass tradition:
  Concepts of tonality and chord in the rule of the octave. Journal of
  Music Theory 51, 5–49 (2007).

AGMON1995
E. Agmon, Functional harmony revisited: A prototype-theoretic approach.
  Music Theory Spectrum 17(2), 196–214 (1995).





