MusicMamba: A Dual-Feature Modeling Approach for Generating Chinese Traditional Music with Modal Precision
Jiatao Chen Tianming Xie Xing Tang Jing Wang Wenjing Dong Bing Shi


[
    [
    September 9, 2024
=====================



§ ABSTRACT 
In recent years, deep learning has significantly advanced the MIDI domain, solidifying music generation as a key application of artificial intelligence. However, existing research primarily focuses on Western music and encounters challenges in generating melodies for Chinese traditional music, especially in capturing modal characteristics and emotional expression. To address these issues, we propose a new architecture, the Dual-Feature Modeling Module, which integrates the long-range dependency modeling of the Mamba Block with the global structure capturing capabilities of the Transformer Block. Additionally, we introduce the Bidirectional Mamba Fusion Layer, which integrates local details and global structures through bidirectional scanning, enhancing the modeling of complex sequences. Building on this architecture, we propose the REMI-M representation, which more accurately captures and generates modal information in melodies. To support this research, we developed FolkDB, a high-quality Chinese traditional music dataset encompassing various styles and totaling over 11 hours of music. Experimental results demonstrate that the proposed architecture excels in generating melodies with Chinese traditional music characteristics, offering a new and effective solution for music generation.





Music generation, music information retrieval, music generation, neural networks, deep learning, machine learning










§ INTRODUCTION



Recent advances in deep learning have significantly impacted the MIDI domain, making music generation a key application of artificial intelligence. Melody generation, a central task in music composition, involves creating musical fragments through computational models and presents more challenges than harmony generation and arrangement. A successful model must capture essential features like pitch and rhythm while producing melodies that align with specific styles and emotions. However, most existing methods, whether based on Recurrent Neural Networks<cit.> or Transformer architectures <cit.>, struggle with the complexity and structure of melodies. For example, while <cit.> generated long-term structured melodies, Transformers excel in capturing global dependencies, demonstrating strong performance across various melody tasks <cit.>.

Several studies have integrated music theory into the generation process. For instance, <cit.> introduced chord progressions for melody generation, <cit.> controlled polyphonic music features through chords and textures, and <cit.> improved beat structure representation. Additionally, <cit.> generated harmonious jazz melodies by adjusting harmonic and rhythmic properties, while other works have explored structured music generation using note-to-bar relationships <cit.> and melody skeletons <cit.>.

Meanwhile, State Space Models (SSMs) have also advanced in modeling long-sequence dependencies, particularly in capturing global musical structures. Models like S4 <cit.> and S5 <cit.> have significantly improved parallel scanning efficiency through new state space layers. Mamba <cit.>, a successful SSM variant, enhances parallel computation and has been applied across various fields, including visual domains with VMamba <cit.> and large-scale language modeling with Jamba <cit.>. Recognizing Mamba's potential in sequence modeling, we applied it to symbolic music generation.

However, these methods primarily focus on Western music and struggle with generating Chinese traditional melodies. While they can produce smooth melodies, they often align with modern styles, failing to capture the unique contours and rhythms of Chinese traditional music. As shown in <ref>, existing methods underperform in preserving the stylistic elements of Chinese music. Modes play a central role in Chinese melodies, determining note selection and arrangement, while conveying specific emotions and styles <cit.>. Due to significant differences in scales, pitch relationships, and modal structures between Western and Chinese music, these methods fail to capture these modal characteristics, leading to discrepancies in style and emotional expression <cit.>. The lack of high-quality Chinese traditional music datasets further limits their effectiveness.



To address these issues, we propose a new architecture, the Dual-Feature Modeling Module, which combining the long-range dependency modeling of the Mamba Block with the global structure capturing of the Transformer Block. We also designed the Bidirectional Mamba Fusion Layer, which integrates local details and global structures through bidirectional scanning, enhancing complex sequence modeling. This comprehensive architecture enables the generation of Chinese traditional music with complex structures and coherent melodies. Specifically, our contributions are:


    
  * Mamba architecture to the MIDI domain. We applied the Mamba architecture to MIDI music generation, proposing the Dual-Feature Modeling Module, which combines the strengths of Mamba and Transformer Blocks. Through the Bidirectional Mamba Fusion Layer, we integrated local details with global structures, achieving excellent performance in long-sequence generation tasks.
    
    
  * REMI-M Representation. We extended the REMI representation with REMI-M, introducing mode-related events and note type indicators, allowing the model to more accurately capture and generate modal information in melodies.
    
    
  * FolkDB. We created a high-quality Chinese traditional music dataset, FolkDB, designed for studying Chinese traditional music. With over 11 hours of music covering various styles, FolkDB fills a gap in existing datasets and provides a foundation for further research.





§ PROPOSED METHOD







 §.§ Problem Formulation




In melody generation, the condition sequence is typically defined as x_1:t = [x_1,...,x_t] and the target sequence as y_1:k=[y_1,...,y_k], where k>t. The prediction of the j-th element in the target sequence can be expressed as y_j|[x_1,...,x_t]∼p(y_j|x_1,...,x_t) where p(y_j|x_1,...,x_t) represents the conditional probability distribution of y_j given the condition sequence.
Chinese traditional music often includes various modes. For example, in pentatonic modes, if a note N_i∈{C,D,E,G,A} serves as the tonic note, then the following notes, if they follow a specific interval relationship, form a mode M. Therefore, to generate Chinese music with mode characteristics, the target sequence can consist of multiple modes and transition notes, represented as y_1=(M_1,f(M_1),M_2,f(M_2),...,M_l,f(M_l)), where M_i corresponds to a subsequence of notes within a specific mode. where M_i corresponds to a subsequence of notes within a specific mode, and M_i represents the transition note sequence following M_i. The task of generating melodies with Chinese modes can ultimately be formulated as the following autoregressive problem:


    p(𝐲|𝐱, M) = ∏_i=1^l p(M_i|C_i) · p(f(M_i)|C'_i) ,


where M is the collection of multiple modes, C_i = (𝐱, 𝐲_<i) and C'_i = (𝐱, 𝐲_≤ i). During the step-by-step generation of notes, the corresponding mode sequence M_i is generated first, followed by the generation of the transition note sequence f(M_i) based on the mode sequence.





 §.§ REMI-M Representation




In generating traditional Chinese music, mode generation is a crucial and complex component. Chinese music often features intricate modal structures, such as pentatonic and heptatonic scales, where the selection and transition of modes are vital to the style and expression of the music. However, existing music representation methods face significant limitations in capturing and generating these modal structures. Although the REMI representation <cit.> effectively captures rhythm, pitch, and velocity information through events such as bar, position, tempo, and note, it struggles with complex modal structures, particularly when handling the dynamic modes in Chinese music.

To address this issue, we extended REMI by introducing two new events in the REMI-M representation to explicitly describe modes:


    
  * Note type event. Distinguishes between mode notes and transition notes, helping the model to more accurately capture modal information.

    
  * Mode-related events. Include the start, end, and type of mode, enabling REMI-M to explicitly annotate and generate modal changes in the music.



As shown in <ref>, the original REMI and MIDI-Like encodings result in low mode generation rates, whereas REMI-M demonstrates significant improvements, achieving mode generation rates exceeding 0.8 across all tested music lengths. These enhancements allow REMI-M to better handle complex modal structures, significantly improving the stylistic consistency and theoretical accuracy in generated music.




    



 §.§ Model



In music generation tasks, it is crucial to capture both local melodic details and global musical structure dependencies within long contexts. To achieve this, as shown in Figure <ref>, we designed a hierarchical feature extraction and integration architecture named the Dual-Feature Modeling Module, which combines the long-range dependency modeling capability of the Mamba Block with the global structure capturing ability of the Transformer Block.

Dual-Feature Modeling Module. In music sequence generation, both melodic details (like note variations and modal transitions) and overall structure (such as phrases and repetition patterns) are crucial. Traditional architectures often struggle to capture these levels of features simultaneously. Let 𝐇 represent the feature matrix. The Mamba Block captures melodic details and modal dependencies by computing a dot product between the mode mask and melody tokens, generating the feature representation 𝐇_1. This provides essential long-range and local information for integration. The Transformer Block primarily models global structural information, processing input melody embeddings with positional encoding to obtain the structural representation 𝐇_2.

Bidirectional Mamba Fusion Layer. To integrate the outputs of the Mamba Block and Transformer Block, we introduce the Bidirectional Mamba Fusion Layer. This layer simultaneously receives the long-range features 𝐇_1 generated by the Mamba Block and the global features 𝐇_2 generated by the Transformer Block. Through a bidirectional scanning mechanism, the forward and backward features are processed separately to obtain F_forward and F_backward. Then, self-attention is applied to the forward and backward features to extract key information:


    F_1 = Attention(F_forward),    F_2 = Attention(F_backward) ,

Next, these two directional features are concatenated to obtain the fused feature 𝐇_fusion, and further processed by a linear layer:

    Output= Linear(𝐇_fusion) ,

The fused feature 𝐇_fusion combines the long-range dependencies and global structure of the melody, providing complete information support for generating complex and coherent music sequences. Finally, the linear layer maps the fused features to the output space, generating the final music sequence.







§ EXPERIMENTS







 §.§ Implementation Details





  §.§.§ Dataset


We used two datasets: the POP909 dataset<cit.> and a self-collected Chinese Traditional Music dataset (referred to as the FolkDB). The POP909 dataset contains rich musical information, particularly in chords and melodies. Pre-training on this dataset allows the model to learn fundamental musical structures and elements, helping it to adapt more quickly to our FolkDB. Additionally, to address the lack of cultural diversity in the POP909 dataset, we have compiled a dataset of approximately 300 Chinese traditional music pieces. This dataset contains about 11 hours of piano MIDI works, featuring traditional modes such as the pentatonic and heptatonic scales, showcasing the diverse styles and modal characteristics of Chinese music.

In terms of data preprocessing, since the original data consists of single-track Chinese traditional music melodies, we performed additional processing on the self-collected Chinese traditional music dataset to ensure that the model can effectively capture and generate music with Chinese cultural characteristics. The specific steps are as follows:


    
  * Tonic Track Extraction. We employed the tonic extraction framework mentioned in the Wuyun model<cit.>. This framework uses a layered skeleton-guided approach, first constructing the skeleton of the melody and then extending it. 
    
    
  * Mode Detection and Annotation. After extracting the tonic track, we conducted mode detection on the melody using interval relationships. By analyzing the intervals between each pair of tonic notes and leveraging the knowledge-enhanced logic within the Wuyun model, we obtained the mode track for each piece of music. 





  §.§.§ Model Settings


We adopted an architecture based on the MambaBlock2 module<cit.>, with the model's hidden dimension set to 256 and the feedforward network's intermediate layer dimension set to 1024. Additionally, we employed GatedMLP to enhance the model's nonlinear representation capabilities. During training, we used the Adam optimizer with an initial learning rate of 2 × 10^-4, dynamically adjusted via the LambdaLR scheduler. The training data was processed in batches, with each batch containing 8 samples and a fixed sequence length of 512 tokens. We used the cross-entropy loss function to measure the difference between the model's predictions and the target labels. This loss function is defined as follows:

    Loss = L_CE - λ_1L_NT - λ_2L_MR ,

Among these, L_CE is used to focus on the model's ability to accurately predict musical events, while L_NT and L_MR correspond to note type events and mode-related events, respectively. λ_1 and λ_2 are used to balance the contributions of the two losses, with both values typically ranging between 0 and 1.







 §.§ Objective Evaluation





  §.§.§ Metric


To evaluate our music generation model, we selected the following four objective metrics: Pitch Class Entropy, Groove Consistency<cit.>, Style Consistency, and Mode Consistency. 


    
  * Pitch Class Entropy. This metric reflects the diversity of pitch distribution, with higher entropy indicating a more dispersed distribution of generated notes, while lower entropy indicates a more concentrated distribution.

    
  * Groove Consistency. Higher groove consistency indicates less variation in rhythm, resulting in a smoother, more stable musical flow.

    
  * Style Consistency. A higher style consistency score indicates that the generated music aligns more closely with the expected style.

    
  * Mode Consistency. This metric evaluates whether the notes in the generated music conform to the predefined mode structure. We improved the traditional scale consistency metric <cit.> to better align with the modal characteristics of Chinese folk music. The specific formula is as follows:







    Consistency Score = |𝒫_melody∩𝒫_scale|/|𝒫_melody|× 100% ,
 

Here, 𝒫_melody represents the set of melody notes, 𝒫_scale represents the set of scale notes, |𝒫_melody∩𝒫_scale| denotes the size of the intersection between the melody note set and the scale note set, and 𝒫_melody represents the size of the melody note set. The consistency score is determined by calculating the overlap ratio between the sets of notes in the melody and scale tracks.


   





  §.§.§ Results


Before introducing the objective indicator test results, we tested the key restoration of each model, and it is clear from the <ref> that MusicMamba not only effectively restores the key in the original sequence, but also introduces additional key changes, while MusicTransformer, although it captures some keys, is not as comprehensive and diverse as MusicMamba. Experimental results show that MusicMamba is better at generating melodies with traditional Chinese music styles, and can generate richer and more consistent sequences.

We conducted two sets of comparative experiments using the MusicTransformer <cit.> and MelodyT5 <cit.> models as baselines. In each experiment, we randomly generated approximately 50 songs for each model and calculated objective metrics, which are displayed in the table. When evaluating the quality of generated music, we consider values that are closer to real data as better. As shown in the <ref>, our model's generated music is closer to the real values in terms of pitch entropy, outperforming the other models. Our model also excels in style consistency and rhythm consistency. Notably, in terms of mode consistency, over 70% of the music generated by our model exhibits a detectable modal structure, and more than 60% of the music performs well in mode consistency. The above metrics are shown in the <ref>.







 §.§ Subjective Listening Test



To evaluate the quality of the music samples generated by the model, we designed a subjective listening test. We recruited 10 music enthusiasts from social networks, each of whom plays at least one musical instrument. Each participant was asked to listen to 10 generated audio samples. They rated the samples based on three criteria: coherence, richness, and style, with scores ranging from 0 to 10. In the subjective evaluation results, MusicMamba outperformed all baseline models in coherence, richness, and style, showing the best overall performance.




§ CONCLUSION



This paper proposes a new architecture that combines the long-range dependency modeling capability of the Mamba Block with the global structure capturing ability of the Transformer Block. We also designed the Bidirectional Mamba Fusion Layer to effectively integrate local and global information. By introducing the REMI-M representation, we were able to more accurately capture and generate modal features in Chinese traditional music. Experimental results show that the combination of REMI-M and MusicMamba more accurately reproduces and generates specific modes in Chinese traditional music, with the generated music outperforming traditional baseline models in terms of stylistic consistency and quality. Our research provides a new direction and technical foundation for exploring more complex modes in various types of ethnic music, as well as for generating melodies with distinctive styles through the incorporation of traditional instruments.






§ ACKNOWLEDGEMENTS



Hao-Wen thanks J. Yang and Family Foundation and Taiwan Ministry of Education for supporting his PhD study. This project has received funding from the European Research Council (ERC REACH) under the European Union’s Horizon 2020 research and innovation programme (Grant agreement #883313).











IEEEbib


