Multi-Track MusicLDM: Towards Versatile Music Generation with Latent Diffusion Model
Tornike Karchkhadze Mohammad Rasool Izadi Ke Chen Gerard Assayag Shlomo Dubnov


  

MT-MusicLDM




















T. Karchkhadze et al.



UC San Diego, CA 92093, USA Bose Corp., Framingham, MA 01701, USA Institute for Research and Coordination in Acoustics and Music (IRCAM), Paris, France 

{tkarchkhadze, kchen, sdubnov}@ucsd.edu, Russell_Izadi@bose.com.com, Gerard.Assayag@ircam.fr




Multi-Track MusicLDM: Towards Versatile Music Generation with Latent Diffusion Model
    Tornike Karchkhadze1 Mohammad Rasool Izadi2 Ke Chen1 Gerard Assayag3Shlomo Dubnov1
    Accepted Sep 3 2024 to ApJ Letters

======================================================================================



§ ABSTRACT 
Diffusion models have shown promising results in cross-modal generation tasks involving audio and music, such as text-to-sound and text-to-music generation. These text-controlled music generation models typically focus on generating music by capturing global musical attributes like genre and mood. However, music composition is a complex, multilayered task that often involves musical arrangement as an integral part of the process. This process involves composing each instrument to align with existing ones in terms of beat, dynamics, harmony, and melody, requiring greater precision and control over tracks than text prompts usually provide. In this work, we address these challenges by extending the MusicLDM—a latent diffusion model for music—into a multi-track generative model. By learning the joint probability of tracks sharing a context, our model is capable of generating music across several tracks that correspond well to each other, either conditionally or unconditionally. Additionally, our model is capable of arrangement generation, where the model can generate any subset of tracks given the others (e.g., generating a piano track complementing given bass and drum tracks). We compared our model with existing multi-track generative model and demonstrated that our model achieves considerable improvements across objective metrics, for both total and arrangement generation tasks. Sound examples can be found at https://mt-musicldm.github.iohttps://mt-musicldm.github.io









§ INTRODUCTION


In recent years, diffusion models <cit.> have demonstrated their ability to learn complex distributions, rendering them well-suited for data types such as raw audio. These advancements have significantly impacted the domains of speech and general audio generation <cit.>, as well as the generation of music directly within the audio domain <cit.>. User-controlled neural audio synthesis, particularly in music generation, has the potential to revolutionize music industry by providing musicians with tools for quick compositional prototyping, speeding up the creative process. Additionally, these tools contribute to the democratization of the field by allowing amateur musicians to leverage generated pieces to compose without extensive knowledge of instruments, music theory and years of musical education.

Most generative music models operate by conditioning music on high-level ideas expressed as text, such as genre and mood, leading to text-to-music (TTM) task. However, TTM paradigms have conceptual problems. One challenge is that music, as an abstract entity, is generally difficult to describe with words. Another issue is that text is not an effective medium to convey time-dependent musical attributes, which are crucial for musical expression. Furthermore, music is a multilayered art in which many tracks of instruments simultaneously play their unique roles while being in correspondence with each other on lower-level attributes like notes, timbre, dynamics, harmony, and rhythm.  The essence of musical composition often boils down to arrangement—structuring the piece, orchestrating interactions, and determining the overall sonic character by distributing texture among different instruments or voices, a complexity that is challenging to convey through text.

To bridge the conceptual gaps in current music generation models, we introduce the Multi-Track MusicLDM, a diffusion-based model that generates coherent music in multiple tracks or stems (terms we use interchangeably), ensuring they correspond and collectively create a unified musical piece. To achieve this, we utilized the MusicLDM <cit.> model, which is an adaptation of AudioLDM <cit.> for music, and transformed it into a multi-track audio diffusion model. Taking a multi-track inspiration from the recent work MSDM <cit.> and operating on the latent space,  our model learns a joint probability distribution for tracks that share a contextual structure and generates music mixtures in separate tracks, a process referred to as total generation. Leveraging the text and music conditioning capabilities of the CLAP <cit.> component, our model provides options for additional conditioning. Audio conditioning can be applied by using an existing reference track processed through CLAP's audio branch to influence the generation. This track can guide the overall character and content of the generated music, effectively enabling a transfer that preserves the essence while allowing creative deviations. Similarly, text conditioning can influence the genre, mood and overall character of the generated tracks. Additionally, by employing a well-known method from diffusion model, inpainting, our model is capable of imputation of tracks or generating any subset of tracks given others. We refer to this process as arrangement generation. Arrangement generation enables, for example, the creation of specific musical parts, like piano or guitar, to accompany existing bass and drum tracks. The conditioning mentioned above can be used in combination with arrangement generation, giving our model an additional edge in creative endeavors. By designing desired instrument combinations and using audio and text conditioning, users have the flexibility to generate specific arrangements or full musical pieces, tailoring the model to their compositional needs.


In our experiments, we demonstrate that our model can generate realistic music across various scenarios: total track-by-track music generation, conditional generations, and arrangement generation with any combination of stems. Furthermore, we compared our model with the existing open source multi-track generative model, MSDM, and demonstrated that our model, trained on the same dataset, achieves considerable improvements in the Fréchet Audio Distance (FAD) <cit.> score compared to the baseline. 

As part of our commitment to reproducibility and open science, the code and checkpoints of this study will be made available on GitHub upon acceptance of this paper.



 





§ BACKGROUND
 




Music Generation. Our system, Multi-Track MusicLDM (MT-MusicLDM), depicted in Figure <ref>, is an extended version of the MusicLDM model capable of learning and generating multiple simultaneous music stems. MusicLDM shares its architecture with AudioLDM, which in turn is based on a Latent Diffusion Model (LDM) <cit.> framework with a cascaded model structure. The system comprises a text-audio encoder, an LDM generator, a Variational Autoencoder (VAE) <cit.>, and a vocoder. The role of text and audio encoding is played by CLAP <cit.>, serving as the system's encoder and mapping audio and text into a shared embedding space for conditioning. This is followed by an LDM that acts as the main generator, operating in the latent space (instead of directly on audio or spectrograms), allowing for training and inference on limited computational resources while retaining generation quality. This latent space is achieved by a  VAE that manages the data dimensionality, pre-trained to compress and reconstruct Mel-spectrograms into latent representations. Finally, a HiFi-GAN <cit.> vocoder synthesizes the audio output from the generated Mel-spectrograms. 

Arrangement generation. The most similar work to ours is the commercial product JEN-1 Composer <cit.>, a framework that uses a latent diffusion-based music generation system for versatile multi-track music generation. Different from our approach, they use a latent space obtained directly from audio with their own pre-trained autoencoder model for audio reconstruction, following the Jen-1 architecture proposed in <cit.>. Additionally, the dataset used is also different from ours, as they used their own private dataset, which is larger and of higher quality than the publicly available Slakh2100 <cit.> dataset that we used. Unfortunately, as it is a commercial product, they do not share code, datasets, or provide a free API for audio generation, making direct comparison to our work impossible. Other recently published works have explored music-to-music and arrangement-like music generation. STEMGEN <cit.> presented an alternative paradigm for music generation by introducing a model that can listen and respond to musical context. Different from ours, STEMGEN uses a transformer language model-like architecture on hierarchical discrete representations from VQ-VAEs <cit.> to model musical tracks. Some other works approach the arrangement task with a one-to-many or many-to-one paradigm. For example, <cit.> generates bass accompaniment using audio autoencoders. On the other hand, SingSong <cit.>, utilizing an architecture similar to AudioLM <cit.>, proposed a method for generating instrumental music to accompany input vocals and demonstrated promising results. However, these works primarily concentrate on one instrument, offering limited versatility. Another work that serves as our baseline, MSDM <cit.>, introduced a novel diffusion-based, multi-source generative framework trained via denoising score-matching <cit.>. MSDM is capable of synthesizing music, creating arrangements, and separating musical sources while operating in the raw waveform domain. While MSDM is notable for its flexibility, it exhibits limitations in audio quality and musical coherence. Interestingly, although our model surpasses MSDM in most tasks, it struggles with source separation—a limitation we attribute to its operation within a latent space where mixtures and stems do not maintain a linear relationship. This insight is prompting our future research in this direction.

Arrangement generation was extensively studied in the symbolic music domain. Using the audio-to-MIDI generation paradigm, these works typically focus on generating harmonization for a given melody, such as in <cit.>, or extracting pitch information from input vocals and generating chords suitable for the melody, as seen in the commercial product Microsoft Songsmith, inspired by <cit.>. Other significant contributions include the works by <cit.> and <cit.>, which explored generating kick drum and bass accompaniments, respectively, in the MIDI domain. The multi-track MIDI music generation paradigm was also employed in systems like MuseGAN <cit.>, MIDI-Sandwich <cit.>, Multitrack Music Machine <cit.>, and MTMG <cit.>. Additionally, MIDI accompaniment generation based on audio conditioning was suggested in <cit.>.





§ METHOD


We introduce MT-MusicLDM, depicted in Figure <ref>,  as an integration of MusicLDM and MSDM. In section 1, we expand the LDM latent space to adopt MusicLDM for multi-stem music generation. This expansion enhances our control and versatility over the music generation process. Next, in section 2, we create musical arrangements for a given subset of tracks. Section 3 explores the use of CLAP encoders to control the style of music with text and musical prompts. 




 §.§ Multi-Track LDM




Following AudioLDM and MusicLDM work, we employ denoising diffusion probabilistic models (DDPMs) <cit.> for audio/music generation. DDPMs belong to a category of latent generative variable models. Given two mappings between a time-domain sample x and its corresponding latent space representation x_latent, i.e. x ↔ x_latent, the generation problem is to model q(x_latent) instead of q(x). We first describe the mappings to and from the latent-space for multi-track music and then explain the training and inference of the LDM.


Let x_mix be a mixture composed of S stems x_s with a duration of T_mix for s ∈{1, …, S} such that x_mix = ∑_s=1^S x_s. Also, denote the stack of stems as x with dimensions S × T_mix.
As shown in Figure <ref>, the individual stack of stems x is transformed into a Mel-spectrogram x_Mel, using short-time Fourier transform (STFT) and Mel operations, with dimensions S × T × F where T and F show the time and frequency dimensions of Mel-spectrogram, respectively. Subsequently, the VAE encoder translates x_Mel into a latent, compressed representation x_latent with dimensions S × C ×T/r×F/r, where r indicates the VAE's compression ratio, and C denotes the number of channels in the latent space. Following this, the VAE decoder converts the latent, compressed representation, x_latent, to the Mel-spectrogram domain as x̂_Mel, which is then transformed to the time-domain x̂ by the HiFi-GAN vocoder. Therefore, the time-domain to latent space mapping, x → x_latent, is composed of STFT, Mel, and VAE encoder, while the reverse mapping x_latent→ x, employs VAE decoder and HiFi-GAN vocoder.


The generative model for x_latent via the diffusion process is defined as p_θ(z_0) = ∫ p_θ(z_0:N) dx_1:N where z_0 = x_latent is the latent representation and θ corresponds to the parameters of the LDM model. Within the DDPM framework, the LDM generator operates in the latent space to generate a latent representation z_0 ∼ q(z_0), either conditionally or unconditionally, from Gaussian noise z_N ∼𝒩(0, σ_N^2 I). The variable n in z_n, where n ∈ [1, …, N], represents the step number in the diffusion model's forward or reverse process, and N denotes the number of total steps. The forward pass gradually introduces Gaussian noise, ϵ∼𝒩(0, I), to z_0 = x_latent, i.e. z_n = z_0 + σ_n ϵ ultimately resulting in isotropic Gaussian noise z_N with a distribution 𝒩(0, I) over N steps. Conversely, the reverse process aims to eliminate noise by estimating the injected noise, ϵ, at every step and obtaining z_n-1 from z_n, thereby incrementally reconstructing z_0 = x̂_latent. The DDPM parameterizes the reverse Gaussian distribution p(z_n-1|z_n) with a neural network ϵ_θ(z_n, n):

    p(z_n-1|z_n) =𝒩( z_n-1 | μ_θ(z_n, n), σ_n^2) 
    μ_θ(z_n, n) = ((σ_n^2 - σ_n-1^2) ϵ_θ(z_n, n) + σ_n-1^2 z_n ) / σ_n^2.

Optimizing the evidence lower bound on the log-likelihood q(z_0) simplifies to minimizing the mean squared error between the predicted noise ϵ_θ and the Gaussian noise ϵ, at each step, as follows:

    L(θ) = 𝔼_z_0, ϵ, nϵ - ϵ_θ(z_n,n, [c_cond]) ^2

where [c_cond] denotes the optional use of conditioning, which we will touch upon in more detail later in the section. Therefore, having the time-domain mapping, x → x_latent, one can estimate the LDM θ by minimizing the loss function L(θ), presented in Eq. <ref>. For inference, the generation begins with the Gaussian noise prior z_N, followed by an iterative backward sampling process from p(z_n-1|z_n) for each n ∈{N, …, 1}, as outlined in Eq. <ref>. The final step involves mapping the generated latent representation, x̂_latent = z_0, to the time-domain using x_latent→ x. To form the mixture of tracks, one can simply add all the rows of x̂, i.e. x̂_mix = ∑_s=1^S x̂_s, with each x̂_s being the s-th row of x̂.




For high-quality reconstructions using DDPM, a large number of steps, typically N = 1000, are traditionally necessary. However, to streamline the process and reduce computational demands, denoising diffusion implicit models (DDIM) <cit.> offer a compelling alternative. In this study, we utilize the DDIM protocol, which allows for a substantial reduction in the number of required steps, down to approximately N = 200 during inference, while still preserving the generative quality.



As Audio/MusicLDM frameworks for audio and music generation are heavily inspired by methods originally used in image generation, their network architecture is borrowed from this domain. Like in image generation, a large UNet <cit.> architecture is a common choice for diffusion models in audio and music domain. The UNet architecture consists of two symmetrical halves: an encoder and a decoder, both enhanced with skip connections that bridge corresponding layers. To accommodate z_n with an additional dimension compared to typical image or single-channel audio representations, S × C ×T/r×F/r vs × C ×T/r×F/r in Audio/MusicLDM, we extend the UNet architecture by using 3D convolutional operations. We effectively enhance the operational dimensionality of UNet by interpreting the channel dimension of z_n as an additional spatial dimension. This adjustment leads to the stems’ dimension now serving as the channel dimension.





 §.§ Conditional Generation

To have control over the generation p_θ(z_0), one can introduce some condition c to the diffusion process, resulting in p_θ(z_0|c). The conditional diffusion process is defined similarly to the unconditional process p_θ(z_0:N ) as follows:

    p_θ(𝐳_0:N | c) = p(𝐳_N) ∏_n=1^N p_θ(𝐳_n-1 | 𝐳_n, c).

Additionally, to enable more controllable generation, LDM  models often employ classifier-free guidance (CFG) <cit.>.
CFG is a technique in diffusion models that enhances control over the adherence to conditioning information during inference. 
This is achieved by randomly dropping the conditioning information during training, thereby simultaneously training both conditional and unconditional versions of the LDM model. In the inference time, the strength of the conditioning can be modulated by the CFG weight ϵ̂ = wϵ_u + (1 - w)ϵ_c, where w is the guidance scale weight that balances the model's unconditional ϵ_u and conditional ϵ_c predictions.

In this study, we employ CLAP to convert text prompts and musical tracks into embeddings, which serve as the basis for conditioning the LDM.
For example, users can specify the type of guitar by conditioning on the CLAP embedding of a reference track.
Additionally, the strength of the conditioning can be adjusted using the CFG weight, increasing their creative options.




 §.§ Arrangement generation

From a musical perspective, arrangement composition refers to the task of creating plausible musical accompaniments for a particular subset of given tracks. In a broader machine learning literature setting, the task of filling a partially observed the data is commonly referred to as imputation (inpainting in the image domain) and aims to fill out the missing segments of variable. Learning the joint distribution of musical tracks offers us a clear path to explore this task in the latent space.

For a given subset of tracks, x_I = { x_s | s ∈ I }, arrangement generation task is to find x_I̅ = { x_s | s ∈I̅}, as follows

    _x_I̅ p_θ( x_I̅ | x_I),

where I̅ = {1, …, S} - I.
Given the LDM, the search for x_I̅ happens in the latent space, i.e. z_I̅ given z_I.
Note that given I, one can find the latent representation of x_I and x_I̅ as follows:

    x_I → m ⊙ z_0, 
    
    x_I̅→ (1 - m) ⊙ z_0,

where m is a binary mask in the latent space with the same dimension as z_0 and m_s = 1 if s ∈ I and m_s = 0 otherwise.

The generation of x_I̅ starts with sampling a Gaussian noise p(z_N) ∼ N(0, I) as the total generation, but each denoising step in the reverse process if followed by

    z_n-1← (1-m) · z_n-1 + m · z'_n-1.

where z'_n-1∼𝒩(z_n-1 | z_0 , σ^2_n-1) is obtained by adding n-1 noise steps to z_0 through the forward process.

In essence, arrangement generation becomes a generation problem where, at every step, parts of the latent space corresponding to the given tracks are masked and replaced with their noise-added version. This approach compels the model to perform generation under constraints, ensuring that the generated arrangement tracks align well with the given ones.







  


§ EXPERIMENTAL SETUP
 



 §.§ Dataset


Following a similar research path as MSDM and to facilitate direct comparison, we used Slakh2100 <cit.>, a dataset widely recognized as a benchmark in the domain of music source separation. Although source separation was not our primary focus, the choice of Slakh2100 was motivated by our need for clean and high-quality multi-track audio data for learning multi-track generation. Synthesized from MIDI files using premium virtual instruments, the dataset consists of 2100 individual tracks into subsets of 1500 for training, 375 for validation, and 225 for testing. While the original Slakh compilation offers up to 31 distinct instrumental classes, our experiment and subsequent analyses were limited to S=4 most prevalent classes: Bass, Drums, Guitar, and Piano. These classes were selected due to their dominant presence in the dataset, ensuring a robust and consistent basis for our evaluations.


In our experiments, we performed preprocessing steps on the dataset. We downsampled the audio from its original 22.05 kHz to 16 kHz to align with the specifications of our model. We read audio from the original tracks, creating audio segments of 10.24 seconds with an additional random shift for training samples. To convert the audio clips into a suitable feature representation, we utilized a window length of 1024 and a hop size of 160 samples to generate Mel-spectrograms with dimensions F × T = 64 × 1024. For the creation of mixed audio samples, individual tracks were combined to form mixtures. Differing from the original MusicLDM, we abstained from normalizing the separate tracks or their mixtures to prevent any potential peaking in the audio signals.








 §.§ Model, Training and Evaluation Specifics


Our parameter configuration closely mirrors that of MusicLDM <cit.>, with only minimal adjustments. We extended the LDM model to accommodate the stem dimension of S=4, transforming it into a 3D LDM model. For our new 3D LDM model, we employed a UNet architecture comprising 2 encoder blocks, a middle block transof, and 2 decoder blocks. We maintained the settings consistent with previous configurations of MusicLDM, with the sole modification being the adaptation to 3D convolutional layers. Additionally, we switched from the "spatial transformer" used by MusicLDM and AudioLDM to a generic attention block transformer, as the former was tailored for 2D, picture-like data, which did not align with our model's 3D data processing requirements.

To obtain a latent representation of Mel-spectrograms, we employed a mono-track VAE from the MusicLDM model, which boasts a compression ratio r = 4, effectively encoding stacks of Mel-spectrograms of size S × T × F = 4 × 1024 × 64 into a 3D LDM latent vector with dimensions S × C ×T/r×F/r = 4 × 8 × 256 × 16, which respectively represent stems, channels, time, and frequency. The components of MusicLDM—including the CLAP encoder, VAE, and the HiFi-GAN vocoder—were taken from the pre-trained, publicly released checkpoint of MusicLDM[https://github.com/RetroCirce/MusicLDMhttps://github.com/RetroCirce/MusicLDM]. This checkpoint was trained on an extensive collection of music audio data. However, it is worth mentioning that these components were not trained or fine-tuned for processing separate tracks, which imposes certain limitations on the final audio quality of our model. These components remained unchanged during the training phase of the 3D LDM generator, as illustrated in Figure <ref>. We utilized pre-trained weights for MusicLDM components from publicly available checkpoints.

We trained our model, MT-MusicLDM, with a dropout rate of 0.1 applied during conditional generation, effectively resulting in training both unconditional and conditional models. Training was conducted using the Adam optimizer with a learning rate of 3 × 10^-5 for a duration of up to 1000 epochs. The number of denoising steps for the LDM is set at N = 1000 during training and reduced to N = 200 for DDIM sampling during inference. 

We evaluated our models using the Frechet Audio Distance (FAD) <cit.> metric, a widely recognized benchmark in music quality assessment. This metric was employed across all our experiments, including total generation, arrangement generation, and both audio- and text-conditioned tasks.









§ EXPERIMENTS AND RESULTS
 



 §.§ Total Generation


We evaluated our MT-MusicLDM model in unconditional mode on the total music generation task using the Slakh test dataset. For this evaluation, we generated audio stems, mixed them to create mixtures, and then calculated the FAD between these generated mixtures and the mixtures from the test set. Given that MSDM was the first and only open source model capable of generating music in individual parts, we selected it as our baseline for comparison. Table  <ref> shows the performance of our model, reported as "MT-MusicLDM (Uncond)", compared to MSDM on the same dataset. We observed that our model significantly outperforms MSDM, with a dramatic reduction in FAD scores from 6.55 to 1.36. This substantial improvement highlights our model's capability to generate high-quality and coherent music audio track-by-track.

For broader context, we also incorporated the benchmark MusicLDM scores from <cit.> in the table. We took the highest-performing variant, "MusicLDM w/ BLM Text-Finetune." This model underwent specialized training with beat-synchronous latent mixup and was further enhanced through fine-tuning on text prompts. We denote MusicLDM with an asterisk in the table to indicate that evaluations were performed on distinct datasets: Audiostock <cit.> for MusicLDM and Slakh for our model. Given the significant differences in dataset characteristics these comparisons should be viewed as contextual rather than direct.





 §.§ Audio Conditional Generation


In the audio-conditioned experiment, we explored the MT-MusicLDM conditioned through CLAP audio branch, focusing on total audio generation with existing audios used for conditioning. We utilized audio stems mixed from the Slakh test dataset as inputs for our model's CLAP encoder and generated audio by conditioning our model with a CFG weight of w =2.0. Subsequently, these generated stems were summed to form new audio mixtures. We then calculated the FAD score between these mixtures and the Slakh test set to evaluate the model's performance in generating coherent audio outputs. In the Table  <ref>, we reported our result as "MT-MusicLDM (Audio Cond)." It is evident that audio conditioning with CLAP noticeably steers generation towards the test set, resulting in further improvement in FAD score. This validates our hypothesis and underscores the potential of our model 
for audio content adaptation with CLAP conditioning, leveraging a user-selected reference track as a dynamic catalyst for creativity, opening new avenues for personalized and expressive music generation. 





 §.§ Text Conditional Generation



To elucidate the impact of text prompts on the generation capabilities of our model, we employed the MT-MusicLDM conditioned through CLAP's text encoder branch and used the Audiostock <cit.> dataset as a validation set. We searched for tag words "energetic" and "soft" within the dataset, creating subsets of corresponding audio files. Then, we generated audio files conditioning our model with text prompts "soft music" and "energetic music" through CLAP text encoder, with CFG weight w =2.0. We calculated FAD score across generated and target audios. Table <ref> presents the results of this experiment. Notably, the cross-prompt and target folders yielded higher FAD scores, suggesting that the model successfully follows the text prompts. It is worth mentioning that the model was neither trained nor fine-tuned on text, nor on the Audiostock dataset. These results underscore the potential of the model being effectively conditioned on text prompts to influence the perceptual quality of generated music, aligning with our statement that the model represents a step forward towards a versatile audio model.









 §.§ Arrangement Generation


In our arrangement generation experiment, we provided the model with a subset of stems and tasked it with generating the remaining stems unconditionally. We conducted 14 distinct experiments, each focused on generating a specific stem, starting from a singles and expanding to include all possible combinations of them. To assess the model's performance, we calculated the FAD scores for each combination by comparing the generated stems to their corresponding targets from the test dataset.

We chose MSDM as our baseline for the arrangement generation task, as it is the only other open source model known to us capable of generating arrangements in a similar manner. For direct comparisons, we adopted the performance evaluation approach used by MSDM, which in turn utilized a generalized version of the FAD protocol from <cit.>, designed for arrangement generation involving multiple tracks. According to this protocol, generated tracks are mixed with existing originals, and the FAD score is calculated for the resultant mixtures, providing a robust measure of the model’s performance in producing coherent total audio outputs. Additionally, to gain a comprehensive picture, we utilized a publicly available implementation of MSDM

to generate arrangements for all combinations for direct comparison of FAD scores between solely generated stems rather than just mixtures. We pursued this approach because we believe that mixing with given stems can mask a model's performance details, thus not allowing for a detailed analysis of the model's capacity to generate each stem subset.

In the Table <ref>, we report the FAD score for all instrument stems  (B: Bass, D: Drums, G: Guitar, P: Piano) and their combinations. The upper two rows of table present a comparison of FAD scores for mixes following the protocol described above, while the lower two rows detail the FAD scores between generated tracks and their targets. Our model significantly outperforms MSDM in every combination except for guitar stem generation. By analyzing the values in the bottom row and reflecting on our listening experiences during experiments, we noticed that our model demonstrates notably stronger performance on drums and bass compared to guitar and piano outputs, which are occasionally slightly inferior, as evidenced by the scores (0.76 and 1.07 versus 2.76 and 1.80), and at times exhibit similarities with each other. This observation is further supported, notably by the GP combination, which refers to guitar and piano pair generation and registers as the highest FAD score among all combination categories. Additionally, we observed that when drums are not provided and the model lacks clear rhythmic cues, it often struggles to maintain rhythmic coherence of generated tracks with the given ones. This limitation is reflected by slightly higher FAD scores for the combinations where drums are not included in provided stems. Addressing these limitations and achieving balanced performance across all stems poses an interesting challenge for future research.







§ CONCLUSION AND FUTURE WORK


We proposed the MT-MusicLDM model, a versatile framework designed to empower creators to generate and compose music in a variety of modes. This includes total track-by-track generation, conditioning generation with reference music tracks or textual inputs, and creating arrangements using any combination of given and generated instrument tracks. Our experiments and evaluations demonstrate that our model produces high-quality sounds across these generative tasks, achieving musical coherence and significantly outperforming the baseline. This work opens several promising avenues for future research.

Although our generative modeling MT-MusicLDM model has shown significant results, limitations remain. These limitations stem from the fact that our model relies on pre-trained components from MusicLDM, such as the VAE and HiFi-GAN vocoder, which are not fine-tuned and specialized in processing individual stems but were trained on music mixes. Additionally, the use of the Mel-spectrogram domain further limits our model's capacity to yield high-fidelity state-of-the-art audio compared to commercial counterparts. Another source of limitation is the choice of dataset, as Slakh2100 is very small considering the data-extensive nature of diffusion models and the task of audio generation. Furthermore, the Slakh dataset doesn't contain any tags or textual descriptions for genre or any other contextual information about its musical pieces.

Looking ahead, we aim to enhance the musical and rhythmic coherence of our model and increase its versatility by expanding the list of available instruments, potentially allowing for user-specified combinations. We intend to extend our investigations by moving away from the Mel-spectrogram domain and possibly shifting to higher sample rates. We plan to incorporate more state-of-the-art VAEs and move to larger datasets beyond Slakh, including large text-to-music datasets, as demonstrated in <cit.>. Additionally, we will explore incorporating different methods of conditioning for source separation to achieve truly versatile music generation—a comprehensive music generation framework that supports a wide range of creative expressions, including generation, arrangement, and separation.



§ ACKNOWLEDGMENTS

We thank the Institute for Research and Coordination in Acoustics and Music (IRCAM) and Project REACH: Raising Co-creativity in Cyber-Human Musicianship for their support. This project received support and resources in the form of computational power from the European Research Council (ERC REACH) under the European Union’s Horizon 2020 research and innovation programme (Grant Agreement 883313).

splncs04


























































































































