SymPAC: Scalable Symbolic Music Generation With Prompts And Constraints
Haonan Chen Jordan B. L. Smith Bochen Li Ju-Chiang Wang Janne Spijkervet Pei Zou Xingjian Du Qiuqiang Kong



MobileUNETR: A Lightweight End-To-End Hybrid Vision Transformer For Efficient Medical Image Segmentation
    Shehan Perera10009-0005-3831-0404 Yunus Erzurumlu10009-0006-5798-5842 Deepak Gulati20000-0003-3374-5992 Alper Yilmaz10000-0003-0755-2628
    September 9, 2024
============================================================================================================================================



§ ABSTRACT 
Progress in the task of symbolic music generation may be lagging behind other tasks like audio and text generation, in part because of the scarcity of symbolic training data.
In this paper, we

leverage the greater scale of audio music data
by applying pre-trained MIR models (for transcription, beat tracking, structure analysis, etc.) to extract symbolic events and encode them into token sequences.
To the best of our knowledge, this work is the first to demonstrate the feasibility of training symbolic generation models solely from auto-transcribed audio data.
Furthermore, to enhance the controllability of the trained model, we introduce SymPAC (Symbolic Music Language Model with Prompting And Constrained Generation), which is

distinguished by using
(a) prompt bars in encoding and (b) a technique called Constrained Generation via Finite State Machines (FSMs) during inference time.
We show the flexibility and controllability

of this approach,
which may be critical in making music AI useful to creators and users.






§ INTRODUCTION


The success of language models — especially large ones — has demonstrated that with
more data and larger models,
using a simple language model objective can endow a model with powerful natural language generation capabilities.
On the other hand, although symbolic music and natural language share many similarities,
no music model has yet seemed to match the capabilities of generative text models.
One reason for this gap is the insufficient amount of symbolic music data.

To address this, previous efforts in symbolic music generation have involved combining limited manually annotated data with data obtained by automatic transcription <cit.>, or collecting private symbolic training datasets <cit.>.
By contrast, in this work, we demonstrate that a high-quality, multi-track symbolic music generation model can be trained just using results from running Music Information Retrieval (MIR) models on audio music data.
In this way,
our framework eliminates the need for manually annotated symbolic music data, allowing for expansion purely through audio datasets.

On the other hand,
there has been a recent surge of efforts
that directly generate the auditory modality of music <cit.>.
This is useful for some applications, but typically precludes fine-grained control and
editing the outcome,
which is crucial for composers who wish to shape their musical ideas precisely.
In contrast, outputting symbolic data gives composers the ability to interactively shape and modify their musical ideas.

Considering such advantages, the problem of how to integrate user input to control the generation of symbolic music has been a popular research topic. In previous works, two methods for incorporating control signals are usually used. The first approach is based on a Variational Autoencoder (VAE) <cit.>, wherein the control is exerted within the VAE's latent space.
The second approach is to embed control information directly into the encoding of symbolic music and implant control inputs during inference <cit.>.

In this work, we introduce the SymPAC framework (Symbolic Music Language Model with Prompting And Constrained Generation),
designed to work with decoder-only language models to enable user input controls.
The SymPAC framework consists of the following two parts.
First, inspired by the prompting mechanism used in the natural language domain <cit.>, we introduce prompt bars in our symbolic music encoding, which consolidates all control signals into a separate prompt section before encoding the actual musical notes.
This design is essential for a decoder-only language model to have the full context of control signals during the generation of music.
Second,
in the controlled symbolic music generation setting, the generated tokens should not only comply with the encoding grammar but also adhere to user inputs.
Thus we propose to use Constrained Generation via Finite State Machines (FSMs), which constrains the sampling of tokens at each time step to a subspace.

We will discuss the advantages of SymPAC over previous methods in Section <ref>, and provide more details of how SymPAC can be used for various types of user inputs in Sections <ref> and <ref>.

We collected roughly
one million in-house audio samples and extracted MIR information for each, using
pre-trained models for beat tracking <cit.>, chord detection <cit.>, section detection <cit.>, multi-track transcription <cit.>, and music tagging <cit.>.
The MIR results were transformed into various tokens, and then integrated into an extended REMI <cit.> encoding to train a language model based on Llama <cit.> architecture.
To summarize, our main contributions are:

Scalability: We demonstrate that a high-quality symbolic music generation model can be trained solely with transcribed data, without the need of manually annotated symbolic music, and can be scaled by amassing more audios.

Controllability: We propose the SymPAC framework, which enables flexible user input controls on a decoder-only language model while retaining good quality.






§ RELATED WORK



 §.§ Training Data For Symbolic Music

In Table <ref>, we summarize some popular music datasets in the symbolic and audio domains, together with our in-house audio dataset, and compare their sizes.
The Lakh MIDI Dataset <cit.> is one of the biggest public datasets, containing 170K multitrack pieces in MIDI format.
Many researchers use publicly available symbolic music datasets for training, but some collect and use large-scale ones that are not disclosed; e.g., MusicBERT <cit.> was trained on the Million-MIDI Dataset (MMD).

Although the combined size of the public datasets in Table <ref> is large, combining them is not straightforward since they vary in format.
For example, the Maestro dataset consists of transcriptions of piano performances where note timings reflect actual performance timings, whereas datasets like Lakh are quantized to metrical time with alignment to beats. The inclusion of instrument tracks and additional information (e.g., chords, sections) also differs between datasets.
To expand the scale of training data by combining these datasets, it is necessary to unify their formats first, which may be tedious and introduce errors. 

On the other hand, publicly available audio datasets are much larger in scale. The Million Song Dataset (MSD) <cit.>, for example, contains 1M songs, or 709M notes in total after being run through a 5-track transcription model <cit.>.
The recently published DISCO-10M <cit.> is of an even larger scale.
Furthermore, by using a single set of MIR models to annotate all the audio data,
we do not need to be concerned about the issue of inconsistent data formats.
This makes it easier to scale up the training dataset.



 §.§ Encoding For Symbolic Music

Since the introduction of the Music Transformer <cit.>, language models based on the transformer architecture have become a popular choice for symbolic music generation.
One of the most critical research questions has been how to encode symbolic music that is amenable to processing by such a model, which, in the context of language models, involves converting the piece into a sequence of tokens.

Early transformer-based models for symbolic music predominantly employed a MIDI-like encoding scheme, by treating MIDI event sequences almost identically as input token sequences <cit.>.
Later, the Revamped MIDI (REMI) encoding <cit.> was proposed, which modified the MIDI encoding by replacing time shift events with duration events for each note and introducing bar and beat concepts to adopt metrical time instead of absolute time. These modifications facilitated the model's learning of rhythmic patterns within the music, 
improving the quality of the output.
Building upon REMI, several extensions have been proposed to support encoding multitrack <cit.> and various control tokens <cit.>.
Our work is based on the multitrack REMI encoding, and given the MIR models we have, 
it incorporates 
control tokens such as genre, chord, and section tokens to the encoding.



 §.§ Controllable Symbolic Music Generation

Previous methods for controlling symbolic music generation have typically fallen into two categories.
The first is based on Variational Autoencoders (VAEs) <cit.>.
VAEs aim to find a latent space for representing music that encodes distinct musical attributes in independent dimensions. This disentanglement allows for specific attributes of generated music (e.g., rhythm, genre, or timbre) to be individually manipulated by altering corresponding dimensions in the latent space without affecting other attributes, thereby enhancing the controllability of music generation.

The second approach is to include control tokens in the encoding of symbolic music. For example, MMM <cit.> includes instruments and note density tokens in the encoding, which can be specified at inference. Similarly, FIGARO <cit.> uses “expert descriptions” 
indicating time signature, note density, mean pitch, mean velocity and mean duration as well as instruments and chords.
It then uses an encoder-decoder model to learn a mapping from descriptions to sequences of a piece of music.
Driven by the development of Large Language Models (LLMs), recent work has also explored using natural language to control symbolic music generation <cit.>. Natural language text can also be treated as control tokens, with the key distinction that it usually requires pre-training the LLM on text.

In our work, the proposed SymPAC framework is designed to work with a decoder-only language model.
In a controlled generation setting, prompt bars that conform with user input control signals are generated first. The generation of musical part comes after that, in which the model will have full context of control signals from prompt bars. These two generation stages are both controlled by an FSM, which takes into consideration the grammar of the encoding and user inputs.
There are two main
differences between SymPAC and previous works


  * We encode control signals as tokens and use FSM to enforce input control signals during inference.
In contrast, for VAE-based control methods, control signals are converted into latent embeddings, and the model is not guaranteed to follow these control signals.

  * Since we use a decoder-only language model, the tokens in prompt bars are also learned simultaneously.
Consequently, the user is only required to input a portion of the control information, with the model being able to automatically generate missing controls.
In contrast, an encoder-decoder framework like the one described in <cit.> would require a complete encoder input during inference, which lacks flexibility.




§ METHOD






 §.§ Symbolic Music Encoding And Prompt Bars

Our data representation is based on the REMI+ <cit.> representation, an extension of REMI <cit.> that supports multitrack data.


An illustration of our encoding is shown in Fig. <ref>.
The fundamental unit of our encoding is a bar, of which there are two types: prompt bar and song bar.
The token sequence of a song bar can be divided into four parts:



  * The meta part includes four tokens for the , ,  (for section type name), and  (which indicates the tempo range).


  * The chord part consists of alternating  and  tokens.

  * Each instrument track part consists of a  token, followed by one or more groups of ,  and  tokens.

  * The drum track part consists of a  token, 
followed by one or more groups of  and  (drum MIDI) tokens.


Here are further explanations of ,  and  tokens[Details of all token types are provided in supplementary materials]:



  * : Represents the starting position of subsequent ,  or  token within a bar. Each bar is divided into 16 steps, so that position ranges from 0/16 to 15/16.

  * : Ranges from the minimum time division of 1/16 bar to a maximum of 2 bars, or 32/16.

  * : A track token will only exist if there is at least one note in the bar for the corresponding instrument.
This allows the user to control which instruments are used within a bar.


Prompt bars contain a subset of tokens in song bars, retaining only tokens that represent control signals.
In our case, these include genre, section, tempo, chords and tracks.
As future work, this encoding could be extended to include more control signals (e.g. note density for a track).
The encoding of a full piece of music will consist of: all prompt bars in the piece; then, a special  token; then, all song bars in the piece; and finally a special  token.

During training stage, the model is trained to predict tokens in prompt bars as well,
not distinguishing them from
tokens in song bars.
As mentioned previously, this design enables the user to input partial control signals (or no input at all), and the model is able to infer the missing ones.





 §.§ Constrained Generation via FSM

In the controlled symbolic music generation setting, there are two types of constraints:

Grammar constraint: The encoding of symbolic music follows a specific format. For example, for our proposed encoding shown in Fig. <ref>, a  token will always be followed by a  token.

User input constraint: 
Generated token sequence should conform with user inputs.
For example, if the user wants to generate “rock” style music, the  token can only be .

Since we are already aware of these constraints in advance,
there is no need to sample from the entire vocabulary space during inference.
Instead, we can sample from a subspace that is in accordance with the constraints.

To achieve this, we employ a Finite State Machine (FSM) to interact with the language model ℳ during inference. Let x_t denote the token generated by ℳ at time step t. The FSM takes x_t, the current state q_t and the predetermined rule set ℛ, and outputs a subset of the vocabulary 𝒱_t+1, from which the language model ℳ can sample at time t+1. We call this procedure Constrained Generation via FSM, which is formally defined in Algorithm <ref>.
This algorithm is analogous to regular expression matching, where it checks if a given input string conforms to a specified pattern.
Here the pattern and input string are equivalent to rule set ℛ and token sequence s_t respectively.



§ EXPERIMENTS AND RESULTS



To validate our contributions, we conduct experiments to assess whether the system is scalable (i.e., improves when scaling up training data) and controllable (i.e., there is consistency between generation output and user inputs).

In Sec. <ref>, we conduct a quantitative analysis to compare models trained on different amounts of training data, in order to assess scalability.
In Sec. <ref>, we examine two common types of control inputs: chord progression and section structure.
The impact of these control inputs is tested through both quantitative metrics and qualitative examples.
Lastly, in Sec. <ref>, we compare our models trained on different datasets with other baseline symbolic music generation systems through subjective evaluation.



 §.§ Datasets

We use three datasets in our experiments. We always use each dataset individually; i.e., we never merge the datasets to train a single model. The datasets are:




Lakh MIDI Dataset (LMD) <cit.>. A dataset in MIDI format, containing around 170K songs.
We use this to compare with models trained on transcribed audio data.

Million Song Dataset (MSD) <cit.>. A public dataset used extensively by MIR researchers. We use the 
30–60s preview audio clips, representing the highlight of the song.

In-House Dataset (IHD). We use a licensed internal collection with about 1M 
full songs in audio format, covering a wide range of Western modern genres.



 §.§ Training Settings

We train a decoder language model with the Llama <cit.> architecture. We set the number of layers, number of attention heads and embedding dimensions to be 12, 12 and 768 respectively, resulting in a model with about 86M trainable parameters.
We concatenate token sequences of all pieces into a 1-D array, and randomly pick a window of size 10,240 as one training sample. As the average sequence lengths of LMD, MSD and IHD are 900, 1500 and 8000 respectively, this window size would contain 11.4, 6.8 and 1.3 pieces on average for each dataset.

When training data are limited, data augmentation and data filtering (to ensure that unusual data do not pollute the training) are commonly used.
However, we adopt neither approach, for two reasons.
First, since we have a large dataset of audio samples,  the training data are likely to cover a broad spectrum of examples already, reducing the need to filter out unusual data points.
Second,  augmentation may alter the training data in unwanted ways.
For example, a common augmentation approach is to transpose all the pitches in a piece <cit.>.
However, this may distort the pitch ranges of each instrument: e.g., if the input bass parts are transposed up and down, the model will not learn the correct range of realistic bass notes.

 



 §.§ Unconditioned Generation


Intuitively, increasing the amount of data should enhance the performance of the model.
In this experiment, we use objective metrics to validate this.
Designing objective metrics to evaluate symbolic music remains an open question.
A common approach is to prepare a reference dataset,
calculate embeddings or metrics
of the generated samples and reference set,
and then compare these using distance metrics such as the Fréchet Distance or Kullback-Leibler Divergence (KLD). 
For a detailed review on evaluation methods for symbolic music, see <cit.>.

In our experiment, we prepare a held-out validation set with 3000 samples.
We use a range of metrics that can be categorized into the following classes: chord, structure, instrument note (including vocals, guitar, piano and bass) and drum note.
Detailed definitions are provided in supplements. In general, the metrics in each class are as follows:


  * Chord: chord label, chord root, chord transition;

  * Structure: section label, section label bigram, instrument labels per bar;

  * Instrument Note: note pitch, note duration, pitch class, min/max pitch per bar, max number of notes per bar, uniformity of number of notes per bar;

  * Drum Note: drum key, max number of notes per bar, uniformity of number of notes per bar, and unique drums per bar.


We compare models trained on  100%, 10% and 1% of the IHD data,
and do generation in an unconditioned setting.
For each model, we generate 800 samples to compute metric distributions.
KLD values are then computed between distributions of generated samples and distribution of the validation set for each metric. 
Lower KLD indicates that two distributions are closer, suggesting the generated samples sound more similar to the validation set.
We report the average KLD values for the same class, and provide a full list of KLDs for each metric in supplements.

The results are shown in Table <ref>. We can see that the model trained with 100% IHD data has the lowest KLD against the validation set on 6 out of 7 classes, and the model trained on only 1% data has the highest KLD on 6 out of 7 classes.
The results confirm that a model trained on more data can generate samples closer to the training data.
Furthermore, we observed that
the benefit of using more data is greater for the 'Note' metrics than for the 'Chord' or 'Structure' ones.
This is likely because note tokens are more numerous and have complex distributions, which needs larger scale of data to learn.
Counterintuitively, the KLD for 'Structure' was better when using 10% of the data instead of 100%. We speculate that since the structure tokens are scarcest, this could be the result of a lucky alignment between the validation set at the 10% of the data used, but this deserves more study.






 §.§ Controlled Generation


The SymPAC framework aims to give users flexible control over the music generation process. However, we need to verify that this control is effective: do the notes generated agree with the control inputs?
To this end, we conduct controlled generation experiments on two input scenarios: chord progression inputs and section structure inputs.

Chord Progression Inputs.
In this experiment, we randomly pick 20 top trending chord progressions from HookTheory[https://www.hooktheory.com] as the chord progression inputs.
We only include major and minor triad chords.
We then let the model generate 64 bars of music by looping the chord progressions.
To evaluate the match between the input chord progression and the output, we apply a symbolic chord detection method on the generated samples. Details about the method can be referred in the supplementary materials.

The accuracy of detected chord from the input chord progression is shown in Table <ref>.
As shown in the result, the models trained on MSD, IHD 100% and IHD 10% all have similar overall accuracy, with MSD slightly out-performing the others. But the model trained on IHD 1% (just 10K songs) is much worse than the other three. This suggests that a dataset at the scale of 100K songs is sufficient to model low-level control signals like chord, given the model and encoding we are using here.
We also provide examples in supplementary audios of outputs when given unusual chord progressions.



Section Structure Inputs.
In this experiment, we take 10 typical section sequences as inputs (listed in supplements), ranging in length from 4 to 13 sections (16 to 68 bars), and use each model to generate 100 outputs per prompt.
We compare the same 4 models from the previous section.
For each generated output,
we leverage a Music Structure Analysis (MSA) algorithm <cit.> to predict its structure, and compare this to the input structure.
The MSA algorithm's predictions may be inaccurate, but we still expect that a greater match between the intended and estimated structure indicates more success at controlling the structure.
We use Foote's algorithm <cit.> for segmentation and the 2D-Fourier magnitude algorithm <cit.>
for section labeling, with a beat-wise feature embedding that averages the pitch-wise MIDI piano rolls within a beat interval. We evaluate the results using  <cit.>, and report three metrics: boundary prediction f-measure with a 3-second tolerance (HR3F); pairwise clustering f-measure (PWF); and the normalized entropy score f-measure (Sf). To test directly how similar the repeated sections are, we also report PWF and Sf when the ground-truth segmentation is used.

We find that all metrics are worse (lower) when the system is trained on MSD or IHD 1%, and improve substantially when at least 10% of the data are used (Table <ref>). This is expected, since the audio clips in MSD are only excerpts and thus not instructive for modelling full-song structure.



Fig. <ref> shows the piano roll 

of a typical output, where the match between the intended and predicted structure was average (Sf = 0.508).
Even so,
the match between the intended and realized structure is evident in the piano roll: the chorus sections are similar but not identical to each other, and so are the verse sections.

In both controlled generation experiments, the gap between 100% and 10% IHD is very small, indicating that 10% IHD data combined with SymPAC is sufficient for achieving good adherence to control inputs. However, it is important to remember that the metrics of these two experiments only reflect whether the control signals are well-followed, not the overall quality of the generated pieces.




 §.§ Subjective Evaluation


The models tested so far were all trained on transcribed audio data, so it is worth comparing
with models trained directly on MIDI data.
In this experiment, we compare our model trained on LMD, MSD and IHD, and also two baselines, FIGARO <cit.> and MMT <cit.>, in a subjective listening test.
We recruited 12 participants with the background of MIR researchers or music producers.
Similar to <cit.>, we asked each participant to rate 10 audio samples generated by each model on a 5-point Likert scale on five criteria: coherence, richness, arrangement, structure and overall [These criteria are described as: (1) Coherence: The rhythm is stable; The chord progression develops logically; Dissonant notes are not excessive. (2) Richness: The melody and acccompaniment are interesting and diverse.
(3) Arrangement: Collaboration among multiple instruments is harmonious and natural; Arrangements for different instruments are diverse and reasonable.
(4) Structure: The piece includes a clear and engaging structure with appropriate repetitions and variations; The piece has obvious connections and reasonable developments between sections.
(5) Overall: I like this piece in general.
].



The result is summarized in Table <ref>. All of our 
proposed models outperform the baselines in all dimensions.
Our model trained on IHD has higher performance than the other two training data setups,
which attests to the viability of leveraging audio data by running MIR models at scale.
The result using LMD was better than MSD, despite having fewer songs;
this could be due to LSD having more notes than MSD (see Tab <ref>), or due to it containing full songs instead of only excerpts.
We only compare FIGARO and Ours (LMD) with a statistical test, since these were trained on the same dataset. Mann-Whitney U tests found significant differences in Richness (p = .005), Structure (p = .0005), and Overall (p = .027) ratings, but not in Coherence (p = .85) or Arrangement (p = .122).



§ CONCLUSIONS AND FUTURE WORK

We trained a language model for symbolic music generation leveraging audio data and pre-trained MIR models.
We proposed the SymPAC framework, which includes prompt bars in encoding and Constrained Generation via FSM during inference time.
We showed how  combining these two components enables a user to control the generation process, and we evaluated the results through quantitative and qualitative analysis.

Future work could improve at least two aspects of this system:
(1) We quantified position and duration to 1/16 per bar, which does not support 3/4 or 6/8 time signatures well. Also, the chord detection model we used only supports 12 major and minor chords, limiting the user input options. We can expand the encoding to support finer-grained quantization and more advanced chords.
(2) Our token sequence length is long: 8000 on average for samples in IHD.
We could use tokenization methods such as Byte Pair Encoding <cit.> or use compound word tokens <cit.> to compress sequences and improve training efficiency.




























supplements
