# M2UGen: Multi-modal Music Understanding and Generation
with the Power of Large Language Models

Atin Sakkeer Hussain1
National University of Singapore
<EMAIL_ADDRESS>Shansong Liu2†
ARC Lab, Tencent PCG
<EMAIL_ADDRESS>Chenshuo Sun1
National University of Singapore
<EMAIL_ADDRESS>Ying Shan2
ARC Lab, Tencent PCG
<EMAIL_ADDRESS>
###### Abstract

The current landscape of research leveraging large language models (LLMs) is
experiencing a surge. Many works harness the powerful reasoning capabilities
of these models to comprehend various modalities, such as text, speech,
images, videos, etc. They also utilize LLMs to understand human intention and
generate desired outputs like images, videos and music. However, research that
combines both understanding and generation using LLMs is still limited and in
its nascent stage. To address this gap, we introduce a Multi-modal Music
Understanding and Generation (M2UGen) framework that integrates LLM’s
abilities to comprehend and generate music for different modalities. The
M2UGen framework is purpose-built to unlock creative potential from diverse
sources of inspiration, encompassing music, image and video through the use of
pretrained MERT, ViT, and ViViT models, respectively. To enable music
generation, we explore the use of AudioLDM 2 and MusicGen. Bridging multi-
modal understanding and music generation is accomplished through the
integration of the LLaMA 2 model. Furthermore, we make use of the MU-LLaMA
model to generate extensive datasets that support text/image/video-to-music
generation, facilitating the training of our M2UGen framework. We conduct a
thorough evaluation of our proposed framework. The experimental results
demonstrate that our model achieves or surpasses the performance of the
current state-of-the-art models.

Figure 1: Multi-modal music understanding and generation by our proposed
M2UGen framework.

${\dagger}$${\dagger}$footnotetext: Corresponding author.

##  1 Introduction

Multi-modal large language models (MLLMs) have emerged as a thriving area of
research, captivating the current scholarly landscape [78]. They primarily
serve as a bridge connecting diverse modalities, such as visual [2, 40, 73],
audio [60, 33, 47], 3D [74, 59, 69] and so on, transcending mere textual
interactions. This significant advancement greatly expands the application
scenarios of large language models (LLMs).

Large language models are typically composed of a large number of parameters
and trained on extensive datasets, endowing them with powerful comprehension
and reasoning capabilities. Leveraging these qualities, researchers have
utilized LLMs to achieve semantic understanding across various modalities.
Examples include engaging in free-form conversations with humans [54, 65],
comprehending audio/video events and performing event-based question answering
[60, 33, 53, 86], as well as captioning images/3D point cloud data [8, 40, 74,
69]. In addition to harnessing the capabilities of LLMs for multi-modal
understanding, researchers have also strived to utilize these models to grasp
the creative intentions of humans. For instance, they have explored generating
images [7], videos [30], audio [48], or music [9] based on textual
descriptions, thereby providing valuable assistance in artistic pursuits.

By integrating understanding and generation tasks within the framework of
LLMs, we have the potential to significantly enhance the user experience. For
example, users can leverage LLMs to summarize videos and generate accompanying
audio commentary or suitable background music, thus assisting them in their
video creation process. However, research that combines both understanding and
generation using LLMs is still limited and in its nascent stage [52, 20, 33,
71, 26, 76]. Among these few existing studies, NExT-GPT [71] stands out as a
notable advancement: it is a multi-modal large language model (MLLM) that
excels in both understanding and generation tasks. NExT-GPT showcases several
promising abilities, such as image/video question answering, text to
image/video generation, audio understanding and generation, image/video
generation for audio, and more. Despite these advancements, the exploration of
music understanding and generation leveraging LLMs remains relatively
unexplored. While NExT-GPT exhibits some capabilities in music understanding
and generation, its proficiency in music-related tasks is modest due to the
absence of specialized training on music datasets. To bridge this gap, we
explore the use of LLMs for music understanding and multi-modal music
generation in this work.

In the domain of music AI, significant progress has been made in developing
tailored models for music understanding [50, 13, 47, 19] and multi-modal music
generation [11, 81, 87, 1, 9, 46]. For music understanding, the MU-LLaMA model
[47] stands as a representative, which is trained on a dedicated music
question-answering dataset. It employs the MERT model [41] as the music
encoder and combines it with the LLaMA 2 model [65], demonstrating superior
performance on music question answering and captioning tasks. Regarding multi-
modal music generation, it can be divided into text-to-music generation,
represented by models like MusicLM [1], MusicGen [9], and AudioLDM 2 [46], as
well as image/video-to-music generation, exemplified by models like Vis2Mus
[81], CMT [11] and V-MusProd [87]. To the best of our knowledge, prior to the
completion of this paper, there have been no other works capable of
simultaneously encompassing music understanding and multi-modal music
generation tasks using LLMs, except for the limited musical capabilities
demonstrated by NExT-GPT. Therefore, in this work, we aim to contribute to
this field by presenting our research findings.

In this work, we propose the Multi-modal Music Understanding and Generation
(M2UGen) framework capable of music understanding and drawing inspiration from
multi-modal inputs to generate music. Our framework employs multiple modal
encoders to represent image, video and music inputs, respectively. In line
with the conventions of the computer vision field, we select ViT [15] and
ViViT [4] as the encoders for the image and video modalities. The MERT model
[41], which has exhibited outstanding performance in downstream music tagging
tasks in the MU-LLaMA work [47], is chosen as our music encoder. The feature
representations obtained from the input encoders of different modalities are
then fed into their respective understanding adaptors. The LLaMA 2 model [65]
comprehends these modality signals and input intentions to carry out
downstream tasks. For the music generation task, we explore and compare two
music decoders, which are AudioLDM 2 [46] and MusicGen [9], while music
understanding is directly addressed by the LLaMA 2 model.

In the realm of LLM-assisted music understanding and generation, there is a
notable scarcity of readily available training data. The MusicCaps dataset
[1], which stands as the largest publicly available dataset for text-to-music
generation, only comprises approximately 28.52 hours of music accompanied by
captions. Moreover, there is a dearth of datasets for the training of
image/video-to-music generation. Hence, in order to tackle this data scarcity
issue, we employ MU-LLaMA [47] and MPT-7B [63] models to generate diverse
modality-music pairs for training our models. Furthermore, we will release our
constructed datasets later to contribute to the development of the community.

Our contributions are summarized as follows:

  1. 1)

We introduce the M2UGen framework, an advancement capable of simultaneously
encompassing music understanding and multi-modal music generation tasks,
aiming to assist users in music related artistic creation.

  2. 2)

We propose a systematic approach for generating large multi-modal music
oriented instruction datasets for the training of our M2UGen model.

  3. 3)

We conduct comprehensive evaluations on various subtasks such as music
question answering, text/image/video-to-music generation and music editing,
showcasing performance levels that surpass or are on par with the state-of-
the-art (SOTA) models.

##  2 Related Works

Figure 2: Multi-modal Music Understanding and Generation Model (M2UGen). The
model is divided into four parts (from left to right): (1) Pre-trained feature
encoders to generate representations from music/images/videos; (2) Multi-modal
understanding adapters to fuse the modality representations into the LLaMA 2
model; (3) The LLaMA 2 model that takes input from the adapters to learn
context information; (4) The output projection layer that projects the LLaMA 2
model’s output embeddings for the music decoder.

#### Multi-modal understanding.

Our world is filled with diverse multi-modal information, while a myriad of AI
models incessantly emerges, striving to comprehend various forms of
information. The tasks themselves encompass a wide spectrum, ranging from
audio/visual classification [28, 15, 4], audio/visual question answering [17,
3, 38, 75], audio/visual captioning [51, 77, 34] or tagging [24, 84, 35],
audio/video event detection [12, 42], audio/video summarization [66, 36], and
so forth. These techniques have witnessed a rapid evolution over the past few
decades, flourishing with remarkable vitality. In these research works, ViT
[15] was a milestone in the field of computer vision, and many existing
visual-related researches [2, 39, 79] utilized ViT as the image encoder. ViViT
[4], which was inspired by ViT, incorporated temporal and spatial information
to perform video representation. In our work, we adopt ViT and ViViT as
encoders for image and video modalities, respectively. Regarding the music
modality, the MU-LLaMA paper [47] compared several SOTA models in their music
feature representation section, including ImageBind [23], Jukebox [10], MERT
[41], and others. The authors revealed that the MERT encoder achieved the best
performance in downstream music tagging tasks. Hence, following [47], we
choose MERT as the music encoder in this work.

#### Multi-modal Music Generation.

Research in music generation has made significant strides in recent years,
particularly with the rise of Transformer [67] and diffusion [29] models. Some
notable works representing the utilization of text as input for guiding music
generation include MusicLM [1], MusicGen [9], Moûsai [57], Noise2music [32],
and AudioLDM 2 [46]. Among these, AudioLDM 2 and MusicGen stand out for their
exceptional music generation abilities. AudioLDM 2 is a general-purpose audio
generation model based on the diffusion process, while MusicGen consists of a
single autoregressive Transformer decoder. In the proposed M2UGen framework,
we explore and compare these two music decoders. There have been a few works
in investigating music generation guided by alternative modalities, such as
Vis2Mus [81] for generating music from images, and CMT [11] and V-MusPod [87]
for generating music from videos. However, these approaches are limited to a
single modality as the instruction condition. In contrast, this paper presents
a novel approach that integrates multiple modalities, including text, image,
and video, leveraging the LLM framework to achieve multi-modal music
generation.

#### LLM-assisted Multi-modal Understanding and Generation.

MLLMs have emerged as a prominent research topic, with researchers tirelessly
equipping these models with various functionalities. For example, Macaw-LLM
[49] integrates text, images, videos, and audio into a unified framework,
enabling multi-modal comprehension. DreamLLM [14] and InternLM-XComposer [80]
generate text and images in an interleaved manner, enhancing the multi-modal
conversational abilities between users and LLMs. For instance, InternLM-
XComposer can compose an article with accompanying illustrations. SEED-LLaMA
[21] leverages LLaMA [65] and diffusion models for image understanding and
question answering, as well as image generation. The NExT-GPT model [71],
which is most relevant to our proposed framework, facilitates interleaved
conversations involving text, images, videos, and audio. However, its limited
music training data restricts its musical capabilities. In this paper, we
introduce the M2UGen framework, which enables music understanding and multi-
modal music generation guided by multiple modalities. It also allows for
modifying input music based on music prompts and text. We believe that our
work will make a valuable contribution to the community of AI-assisted
artistic creation.

##  3 M2UGen Model Architecture & Training

The architecture of the M2UGen model is illustrated in Figure 2. In this
section, we provide an in-depth description of the M2UGen architecture and
elaborate on the training methods employed during the development of this
model.

###  3.1 Multi-modal Feature Encoders

In order to accomplish multi-modal music understanding and generation, the
M2UGen model needs to be able to process multi-modal inputs. To achieve this,
it incorporates frozen pre-trained encoders capable of comprehending various
modalities, including music, images, and videos. These encoders extract
relevant information to address tasks associated with music question answering
and music generation within the M2UGen framework. Specifically, the following
encoders are employed for processing involved modalities:

#### MERT Encoder

Notably, the MERT model [41] has demonstrated exceptional performance in
downstream music tagging tasks, as highlighted by Liu et al. (2023) [47],
surpassing alternative models such as Wav2CLIP [70], ImageBind [23], and
Jukebox [10]. Hence, we incorporate the MERT model as the music encoder to
generate feature embeddings for music inputs. The shape of the output
embedding is $(25,1024)$, which is obtained by stacking the 24 hidden layers
and the final output layer of the MERT model.

#### ViT Encoder

Vision Transformer (ViT) [15] is a prominent breakthrough due to its
performance and a prevailing image encoder in the field of computer vision. It
splits an image into a series of fixed-sized patches and transforms them into
patch embeddings, which are then fed into the Transformer encoder along with
positional encodings. We adopt ViT as the encoder for image inputs, and it
produces feature embeddings with a dimension of $(197,768)$, where $197$ is
the number of $16\times 16$ patches in a $224\times 224$ input image plus the
final output layer, while $768$ corresponds to the hidden size of the
Transformer.

#### ViViT Encoder

The Video Vision Transformer (ViViT) model, as introduced by Arnab et al.
(2021) [4], represents one of the initial successful implementations of purely
Transformer-based models for video comprehension. The ViViT model extracts
spatio-temporal tokens from the input video and subsequently processes them
through a sequence of Transformer layers to generate feature embeddings. The
ViViT model produces embeddings with a shape of $(3137,768)$, where $3137$ is
derived from the total count of $16\times 16$ patches sampled uniformly from
32 frames of size $224\times 224$, including the final output layer, and $768$
is the hidden size of the Transformer.

Figure 3: A Detailed Structure of Multi-modal Understanding Adapter’s Dense
Network. Each sub-block consists of normalization, a linear layer, and SiLU
activation function components. The input from the previous layer is also
propagated to the next layer through a skip connection.

###  3.2 Multi-modal Understanding Adapters

To integrate the feature embeddings into the LLaMA 2 model [65], we have
developed multi-modal understanding adapters (see in Figure 2) to align the
output of multi-modal encoders with the input of the LLaMA 2 model. Different
modal adapters share a unified architecture which includes a 1D convolutional
layer, a linear projection layer, and a dense network composed of three sub-
blocks, as depicted in Figure 3. The process can be formulated as follows:

$\displaystyle\begin{aligned}
X_{i}=X_{i-1}+L_{2,i}(SiLU(L_{1,i}(N_{i}(X_{i-1})))\\\ \times
L_{3,i}(N_{i}(X_{i-1})))\end{aligned}$

where $X_{i}$ denotes the output embedding after the $i$-th sub-block,
$L_{j,i}$ is the $j$-th linear layer in the $i$-th sub-block, and $N_{i}$
represents the normalization layer in the $i$-th sub-block. SiLU [16] is the
activation function. The architectural design of the adapter is inspired by
the work of Liu et al. (2023) [47] which demonstrated promising results in the
field of music understanding. Subsequently, the output embeddings from the
adapters, with a dimension of $4096$, serve as inputs for the LLaMA 2 model,
providing multi-modal context information to enhance LLM’s capabilities in
music understanding, question answering, and guiding downstream music
generation.

###  3.3 LLM as a Bridge

To introduce multi-modal context information to the LLM, we merge outputs from
previous multi-modal understanding adapters into some specific hidden layers
of the LLaMA 2 model. As shown in the light blue box of Figure 2, the total
number of hidden layers is $N=32$, and we introduce one modality-specific
information every $L$-th layer ($L=6$) starting from the top (last) layer. For
the lower $(N-3L-1)$ hidden layers, vanilla attention is employed, while the
remaining layers above utilize zero-initialized attention. The input
instruction prompt is fed into the first layer which is at the bottom of the
Transformer, while the embedding vectors from music, image, and video are
respectively imported into the $L$-th, $2L$-th, and $3L$-th layers starting
from the top (last) layer. Through this approach, the LLaMA 2 model can
effectively reason and comprehends multi-modal inputs.

###  3.4 Music Understanding and Generation

Inspired by models such as NExT-GPT [71], the M2UGen model incorporates
specialized audio tokens of the form $[AUD_{i}]$, where
$i\in\\{0,1,\cdots,7\\}$, to distinguish between music question answering and
generation tasks. The number of audio tokens is a hyper-parameter that
determines the dimension of the input to the music output Transformer, also
known as the output projector, during music generation. In the training phase,
instruction sets containing music as the output are adjusted by appending such
audio tokens to the end of the output. In the inference phase, the M2UGen
model will generate audio tokens only if the instruction prompt requires a
music output. Through this method, the M2UGen model exhibits the capability to
produce text-only outputs for music question answering and music outputs for
music generation within the same framework.

When performing the music generation task, we leverage the output from the
output projector to modulate the music generation process. As each output
token is mapped to a hidden embedding in the final layer of the LLaMA 2 model,
we combine these hidden embeddings corresponding to the audio tokens with the
audio token embeddings themselves as the input to the output projector. The
subsequent output generated by the output projector serves as a crucial
conditioning signal for the AudioLDM 2 [46] / MusicGen [9] models, guiding the
generation of the final output music.

###  3.5 Training Method

Training a MLLM model from scratch is computationally expensive, which has led
several models [71, 58, 40] to adopt the LoRA fine-tuning approach. In our
training method, we alleviate the computational burden by freezing the
encoders and generation models, focusing the training efforts on the multi-
modal understanding adapters and output projector. This strategy not only
reduces computational costs but also enhances training efficiency. To further
simplify the training process and minimize the number of trainable parameters,
we apply the LoRA method [31] to train the LLaMA 2 model. In the training
pipeline, we utilize the following loss function to guide the optimization
process:

$Loss=\begin{cases}L_{CE}(y_{tokens},f(y)_{logits})&\\\
+\|y_{embeddings}-g(f(x)_{hidden})\|,&\text{if music}\\\ \\\
L_{CE}(y_{tokens},f(y)_{logits}),&\text{else}\end{cases}$

where $y_{tokens}$ denotes target output tokens, $y_{embeddings}$ denotes the
target embeddings for AudioLDM 2/MusicGen, $f(\cdot)$ represents the output
from M2UGen’s LLaMA 2 model, $g(\cdot)$ represents the output from M2UGen’s
output projection layer, and $L_{CE}$ is the cross entropy (CE) loss. The loss
function used by the M2UGen model is either CE alone or a combination of CE
and mean squared error (MSE). During training, if the task involves only text
token generation, the model is guided by the CE loss alone. If the task
involves music generation, both CE and MSE are used together, with MSE
calculated between the conditioning embedding generated by the output
projector and the target music caption’s text encoding from the music
generation model. This dual-loss strategy ensures that the M2UGen model is
adept at both text token generation and generating embeddings for conditioning
the downstream music generation models (AudioLDM 2 or MusicGen).

##  4 Music Oriented Instruction Dataset

Training MLLMs requires a significant amount of data. However, there is a
noticeable deficiency of multi-modal datasets that specifically focus on
music-related tasks. Currently, MusicCaps [1] and MusicQA [47] stand as the
two largest publicly available dedicated datasets for music-related tasks,
specifically in the areas of music captioning and music question answering,
which are directly relevant to music understanding. Nevertheless, for the task
of multi-modal music understanding and generation addressed in this paper,
they are still far from sufficient.

To train our M2UGen model, the availability of multi-modal instruction
datasets that support any-to-music generation is imperative. Moreover,
extensive datasets that include pairs of different modalities, such as text-
image pairs, are essential for modality alignment training. We make use of
established datasets like Alpaca [62] with general knowledge for instruction
following and COCO [45] for aligning the image encoder. In addition to
utilizing existing resources, we also collect our own dataset. We adopt an
automated approach to overcome the labor-intensive and time-consuming nature
of manual data collection. Specifically, inspired by previous works [47, 25],
the MU-LLaMA [47] and MPT-7B [63] models are utilized to generate required
datasets. In the following subsections, we provide a comprehensive overview of
the methodologies employed in crafting the datasets used for training the
M2UGen model.

###  4.1 MUCaps Dataset

We develop the MUCaps dataset which is composed of text-music pairs,
encompassing approximately 1,200 hours of music sourced from AudioSet [22] and
some publicly accessible music websites. We leverage the MU-LLaMA model to
caption the music files. For the captioning process, the MU-LLaMA model is
presented with each music file along with the following question: “Describe
the music in detail, including aspects such as instruments used, tempo, and
the mood of the song”. The generated MUCaps dataset is employed for the
purpose of encoder and decoder alignment training.

###  4.2 MUEdit Dataset

To empower our model with the capability of performing music editing in
response to prompts, we have curated the MUEdit dataset, which includes 55.69
hours of 10-second music pairs. The generation of this dataset is carried out
through the following steps:

  1. 1)

Generate captions for all the music files acquired using the MU-LLaMA model.

  2. 2)

Select pairs from a music pool, employing metrics such as tempo, beats, pitch,
and magnitude to ensure that the chosen pairs exhibit similar rhythmic
characteristics.

  3. 3)

For each selected pair, the MPT-7B model is employed to generate instructions.
To create the human side of the conversation, the model is provided with the
captions of the music files as input, accompanied by the following
instructions:

     * –

You are given description of input and target music

     * –

You will give a single line instruction of the form to change input music to
target music

     * –

Give the instruction based on the music descriptions

For generating the model side of the conversation, the model is supplied with
the caption of the output music file, along with the following instructions:

     * –

You are given description of a music

     * –

You will give a single line answer of the form ‘Here is a music that is ….’

     * –

Complete the answer based on the music description

###  4.3 MUImage Dataset

We present the MUImage dataset for generating appropriate music for input
images. The MUImage dataset is assembled by obtaining music samples from the
AudioSet with paired videos. A random frame is selected from each video to
serve as the input image. The process of creating this dataset involves the
following steps:

  1. 1)

Generate captions for all the music files acquired using the MU-LLaMA model.

  2. 2)

Generate captions for the corresponding images using the BLIP image captioning
model [39].

  3. 3)

For each pair of music and image, the MPT-7B model is employed to generate
instructions. The music and image captions are used as inputs. To create the
human side of the conversation, the model is provided with the following
instructions:

     * –

You are given description of a music and an image

     * –

You will give a single line instruction of the form ‘Generate a music for the
image that is ….’ based on the image

     * –

Complete the instruction based on the music and image description

For generating the model side of the conversation, the model is presented with
the following instructions:

     * –

You are given description of a music and an image

     * –

You will give a single line answer of the form ‘Here is a music that is ….’
based on the image

     * –

Complete the answer based on the music and image description

###  4.4 MUVideo Dataset

Similar to MUImage, we also create the MUVideo dataset to enable our proposed
M2UGen framework to accomplish the task of video-to-music generation. The
MUVideo dataset is curated by gathering music samples from the AudioSet as
well with their corresponding videos. To construct this dataset, the following
steps are designed:

  1. 1)

Generate captions for all the music files acquired using the MU-LLaMA model.

  2. 2)

Generate captions for the corresponding videos using the VideoMAE captioning
model [64].

  3. 3)

For each pair of music and video, the MPT-7B model is employed to generate
instructions. The music and video captions are used as inputs. To create the
human side of the conversation, the model is presented with the following
instructions:

     * –

You are given description of a music and a video

     * –

You will give a single line instruction of the form ‘Generate a music for the
video that is ….’

     * –

Complete the instruction based on the music and video descriptions

For generating the model side of the conversation, the model is given the
following instructions:

     * –

You are given description of a music and a video

     * –

You will give a single line answer of the form ‘Here is a music that is ….’

     * –

Complete the answer based on the music and video descriptions

In these four datasets, we make efforts to minimize overlaps among the music
files. Additionally, we establish evaluation splits for each dataset to
facilitate the comparison of our model’s performance with that of the current
state-of-the-art (SOTA) models in their respective domains.

##  5 Model Evaluation

Given various capabilities demonstrated by our M2UGen model, such as music
understanding and music generation from multi-modal inputs, we conduct a
comprehensive evaluation of the model in this section, assessing its
performance across different subtasks. We also present a comparative analysis
with other pertinent models. One such model demonstrating the capability of
any-to-any generation is NExT-GPT[71]; however, since the checkpoint released
by the authors can not function as expected and has issues generating desired
outputs, a direct comparison with NExT-GPT for large-scale evaluation is
currently unfeasible. During the evaluation, we set the hyper-parameters of
the M2UGen model as follows: temperature$\ =0.6$, top_p$\ =0.8$ and max target
length$\ =512$. We have also made sure that all models leveraging LLMs, such
as LLaMA-Adapter [18] and SALMONN [60], use the same hyper-parameters for
evaluation to ensure a fair comparison.

###  5.1 Music Understanding

To evaluate the music understanding capabilities of the M2UGen model, we
employ the MTG-eval-QA subset of the MusicQA dataset proposed by Liu et al.
(2023) [47] as our evaluation set, consisting of 4,500 music question-answer
pairs. The SOTA models for comparison include LTU [25], LLaMA-Adapter [18],
SALMONN [60] and MU-LLaMA [47]. Among these models, MU-LLaMA stands out as the
only one that was specifically trained on music-related datasets. The
evaluation metrics used for music understanding follow the work of [47],
containing BLEU (B-U) [56], METEOR (M-R) [6], ROUGEL (R-L) [44], and BERT-
Score (BERT-S) [83].

Table 1: Comparison of models for music understanding. The best values of different metrics are made bold. Model | B-U$\uparrow$ | M-R$\uparrow$ | R-L$\uparrow$ | BERT-S$\uparrow$
---|---|---|---|---
LTU | 0.242 | 0.274 | 0.326 | 0.887
LLaMA Adapter | 0.273 | 0.334 | 0.413 | 0.895
SALMONN | 0.286 | 0.332 | 0.371 | 0.898
MU-LLaMA | 0.306 | 0.385 | 0.466 | 0.901
M2UGen | 0.308 | 0.393 | 0.476 | 0.902

It is evident from the Table 1 that the LTU, LLaMA Adapter, and SALMONN models
are lagging behind MU-LLaMA and our M2UGen model in the four metrics compared,
since the former three models have not been specifically trained on music-
related datasets. Both the MU-LLaMA and our M2UGen models have been trained on
the MusicQA dataset [47], demonstrating impressive music understanding
capabilities. It is worth noting that our model still outperforms the MU-LLaMA
model. One possible reason is that, in addition to the MusicQA dataset, we
also have the MUCaps dataset for text-music alignment training, which
contributes to enhancing our model’s music understanding abilities. However,
since the MUCaps dataset is generated by the MU-LLaMA model, there may be
limitations on further improving our model’s music understanding capabilities.
This could be one of the factors preventing our model from significantly
surpassing MU-LLaMA, which implies that in our future work, we may need to
explore other music datasets to make further improvements.

###  5.2 Text to Music Generation

For text-to-music generation, we use the evaluation set from the MUCaps
dataset. This set comprises 5,000 text-music pairs. SOTA models selected for
comparison include CoDi [61], AudioLDM 2 [46], and MusicGen [9]. Among these
models, MusicGen is the sole one explicitly trained for music generation,
leading us to anticipate its superior performance compared to other models. In
this and the following experiments, we evaluate the M2UGen model with both
AudioLDM 2 (M2UGen v1) [46] and MusicGen (M2UGen v2) [9] decoders. Following
MusicGen, we use the Fréchet Audio Distance (FAD) [37], Kullback-Leibler
divergence (KL), and CLAP score [72] as the evaluation metrics.

Table 2: Comparison of models for text-to-music generation. The best values of different metrics are made bold. Model | FADvgg$\downarrow$ | KL$\downarrow$ | CLAPscore$\uparrow$
---|---|---|---
CoDi | 16.201 | 6.021 | 0.143
AudioLDM 2 | 11.619 | 4.074 | 0.238
MusicGen | 10.697 | 3.909 | 0.289
M2UGen v1 | 11.143 | 3.982 | 0.282
M2UGen v2 | 10.498 | 3.769 | 0.313

From Table 2, we can see that CoDi performs the worst of all the models for
the task of text-to-music generation. This is likely due to the fact that it
has not been trained on music data. As mentioned above, our model incorporates
AudioLDM 2 and MusicGen as music decoders, so in Table 2, M2UGen v1 is
compared to AudioLDM 2, and M2UGen v2 is compared to MusicGen. It can be
observed that our M2UGen model performs better when given AudioLDM 2 or
MusicGen as the music decoder compared to using them alone. Particularly, the
generated music is more relevant to the input instructions, as indicated by
the improvement in CLAP score. This can be attributed to the use of LLMs,
which makes it easier for the model to understand the input instructions and
use them to guide music generation.

###  5.3 Prompt Based Music Editing

M2UGen is one of the few existing models that support music editing using
natural language. In contrast, models like AUDIT [68] and InstructME [27]
require specific prompt words such as “Add”, “Drop”, “Remove” and others to
edit music. Loop Copilot [85] is another model that supports music editing
using natural language. However, since the authors have not made their model
open-source, we are unable to compare it. Both AUDIT and InstructME also lack
open-sourced models, but InstructME provides a few samples that we can utilize
for comparison. Following AUDIT, in addition to using FAD and KL for
evaluation, we introduce another metric called log spectral distance (LSD) to
assess the music editing subtask.

Table 3: Comparison of models for prompt based music editing. The best values of different metrics are made bold. Model | FADvgg$\downarrow$ | KL$\downarrow$ | LSD$\downarrow$
---|---|---|---
AUDIT | 2.855 | 6.267 | 0.987
InstructME | 2.442 | 6.018 | 0.846
M2UGen v1 | 2.223 | 5.654 | 0.790
M2UGen v2 | 2.191 | 5.118 | 0.735

Table 3 illustrates the superior performance of our M2UGen model compared to
the AUDIT and InstructME models. This achievement can be attributed to the
utilization of the LLaMA 2 model for comprehending prompts in the editing
task. This allows the model to effectively edit music based on natural
language prompts. Furthermore, the use of the MERT Encoder enhances the
model’s capability to better understand the source music, consequently
improving its effectiveness during the editing process.

Table 4: Comparison of models for image-to-music generation. The best values of different metrics are made bold. Model | FADvgg$\downarrow$ | KL$\downarrow$ | IB Rank$\uparrow$
---|---|---|---
CoDi | 10.788 | 9.925 | 0.493
M2UGen v1 | 7.326 | 6.014 | 0.688
M2UGen v2 | 6.968 | 5.878 | 0.819
Table 5: Comparison of models for video-to-music generation. The best values of different metrics are made bold. Model | FADvgg$\downarrow$ | KL$\downarrow$ | IB Rank$\uparrow$
---|---|---|---
CoDi | 11.273 | 6.267 | 0.212
CMT | 9.021 | 5.991 | 0.629
M2UGen v1 | 8.171 | 5.284 | 0.721
M2UGen v2 | 8.002 | 4.939 | 0.850

###  5.4 Multi-modal Music Generation

Multi-modal music generation from images/videos is a crucial ability of our
M2UGen model. CoDi [61] is an any-to-any generation model, thus it is involved
for both image-to-music (I2M) and video-to-music (V2M) generation experiments
(Tables 4 and 5), while CMT [11] is for video-to-music generation. The
evaluation sets for I2M and V2M consist of 2,500 pairs of image-music and
video-music, respectively. In these two sets of experiments, apart from FAD
and KL, we introduce a new evaluation metric called ImageBind Ranking (IB
Rank) [23] to assess the alignment between the image/video modality and the
generated music. Specifically, we use the ImageBind model to obtain embeddings
for the images/videos and the generated music, and then calculate their
similarity scores for ranking purposes.

From Tables 4 and 5, it can be seen that our M2UGen model demonstrates
exceptional capabilities in multi-modal music generation, both in terms of the
quality of generated music and the relevance to the input modality.
Furthermore, it consistently outperforms other SOTA models.

###  5.5 Subjective Evaluation for Music Generation

In order to provide a subjective assessment of our model’s music generation
capabilities, we conduct a subjective evaluation involving 40 participants. A
total of 20 questions are created for three subtasks: text-to-music (T2M),
image-to-music (I2M), and video-to-music (V2M) generation. Each question has
options generated by the models to be compared, which are randomly shuffled to
avoid any preference bias from the testers. Since the sample size for
subjective evaluation is relatively small, we use the interactive demo
released by the authors of NExT-GPT to generate evaluation results for the T2M
and I2M subtasks. However, for the V2M experiment, the NExT-GPT demo
occasionally fail to function properly, so we decide to omit its comparison
for V2M. The results are presented in Table 6. It shows that our proposed
M2UGen model consistently receive the highest preference among the testers for
all three subtasks.

Table 6: Subjective comparison of models for music generation tasks. The best values of different metrics are made bold. Model | T2M | I2M | V2M
---|---|---|---
CoDi | 14.75% | 18.5% | 17.5%
AudioLDM 2 | 19.25% | N/A | N/A
MusicGen | 21.5% | N/A | N/A
NExT-GPT | 15% | 23.5% | N/A
CMT | N/A | N/A | 37.5%
M2UGen v2 | 29.5% | 58% | 45%

##  6 Conclusion and Future Work

This paper introduces the M2UGen model, which utilizes a large language model
(LLM) to achieve music understanding and multi-modal music generation within a
unified framework. Furthermore, we present a comprehensive methodology for
generating the datasets used to train our model. The experiments show that our
proposed M2UGen model outperforms or achieves SOTA performance in various
tasks, including music understanding, music editing, and text/image/video-to-
music generation. Our future work will focus on further enhancing the model’s
fine-grained music understanding capabilities, as well as improving the
correlation between generated music and input instructions.

## References

  * Agostinelli et al. [2023] Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al.  MusicLM: Generating Music from Text.  _arXiv preprint arXiv:2301.11325_ , 2023.
  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: A Visual Language Model for Few-Shot Learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736, 2022.
  * Antol et al. [2015] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh.  VQA: Visual Question Answering.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2425–2433, 2015.
  * Arnab et al. [2021] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lucic, and Cordelia Schmid.  ViViT: A Video Vision Transformer.  _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 6816–6826, 2021.
  * Banerjee and Lavie [2005a] Satanjeev Banerjee and Alon Lavie.  METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments.  In _ACL Workshop_ , pages 65–72, 2005a.
  * Banerjee and Lavie [2005b] Satanjeev Banerjee and Alon Lavie.  METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments.  In _ACL Workshop_ , pages 65–72, 2005b.
  * Brade et al. [2023] Stephen Brade, Bryan Wang, Mauricio Sousa, Sageev Oore, and Tovi Grossman.  Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models.  In _Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology_ , pages 1–14, 2023.
  * Chen et al. [2022] Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny.  VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18030–18040, 2022.
  * Copet et al. [2023] Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez.  Simple and Controllable Music Generation.  _arXiv preprint arXiv:2306.05284_ , 2023.
  * Dhariwal et al. [2020] Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever.  Jukebox: A Generative Model for Music.  _arXiv preprint arXiv:2005.00341_ , 2020.
  * Di et al. [2021] Shangzhe Di, Zeren Jiang, Si Liu, Zhaokai Wang, Leyan Zhu, Zexin He, Hongming Liu, and Shuicheng Yan.  Video Background Music Generation with Controllable Music Transformer.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , pages 2037–2045, 2021.
  * Dinkel et al. [2021] Heinrich Dinkel, Mengyue Wu, and Kai Yu.  Towards Duration Robust Weakly Supervised Sound Event Detection.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:887–900, 2021.
  * Doh et al. [2023] SeungHeon Doh, Keunwoo Choi, Jongpil Lee, and Juhan Nam.  LP-MusicCaps: LLM-Based Pseudo Music Captioning.  _arXiv preprint arXiv:2307.16372_ , 2023.
  * Dong et al. [2023] Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al.  DreamLLM: Synergistic Multimodal Comprehension and Creation.  _arXiv preprint arXiv:2309.11499_ , 2023.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  In _International Conference on Learning Representations_ , 2021.
  * Elfwing et al. [2018] Stefan Elfwing, Eiji Uchibe, and Kenji Doya.  Sigmoid-weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning.  _Neural networks_ , 107:3–11, 2018.
  * Fayek and Johnson [2020] Haytham M Fayek and Justin Johnson.  Temporal Reasoning via Audio Question Answering.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 28:2283–2294, 2020.
  * Gao et al. [2023] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao.  LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model.  _arXiv preprint arXiv:2304.15010_ , 2023.
  * Gardner et al. [2023] Josh Gardner, Simon Durand, Daniel Stoller, and Rachel Bittner.  LLark: A Multimodal Foundation Model for Music.  _arXiv preprint arXiv:2310.07160_ , 2023.
  * Ge et al. [2023a] Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan.  Planting a Seed of Vision in Large Language Model.  _arXiv preprint arXiv:2307.08041_ , 2023a.
  * Ge et al. [2023b] Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan.  Making LLaMA SEE and Draw with SEED Tokenizer.  _arXiv preprint arXiv:2310.01218_ , 2023b.
  * Gemmeke et al. [2017] Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter.  Audio Set: An Ontology and Human-labeled Dataset for Audio Events.  In _Proc. IEEE ICASSP 2017_ , 2017.
  * Girdhar et al. [2023] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra.  ImageBind: One Embedding Space To Bind Them All.  In _CVPR_ , 2023.
  * Gong et al. [2021] Yuan Gong, Yu-An Chung, and James Glass.  PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregation.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:3292–3306, 2021.
  * Gong et al. [2023] Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James Glass.  Listen, Think, and Understand.  _arXiv preprint arXiv:2305.10790_ , 2023.
  * Guo et al. [2023] Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al.  Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following.  _arXiv preprint arXiv:2309.00615_ , 2023.
  * Han et al. [2023] Bing Han, Junyu Dai, Xuchen Song, Weituo Hao, Xinyan He, Dong Guo, Jitong Chen, Yuxuan Wang, and Yanmin Qian.  InstructME: An Instruction Guided Music Edit And Remix Framework with Latent Diffusion Models.  _arXiv preprint arXiv:2308.14360_ , 2023.
  * Hershey et al. [2017] Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al.  CNN Architectures for Large-scale Audio Classification.  In _2017 ieee international conference on acoustics, speech and signal processing (icassp)_ , pages 131–135. IEEE, 2017.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising Diffusion Probabilistic Models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Hong et al. [2023] Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang.  CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Hu et al. [2022] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  LoRA: Low-Rank Adaptation of Large Language Models.  In _International Conference on Learning Representations_ , 2022.
  * Huang et al. [2023a] Qingqing Huang, Daniel S Park, Tao Wang, Timo I Denk, Andy Ly, Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Frank, et al.  Noise2Music: Text-conditioned Music Generation with Diffusion Models.  _arXiv preprint arXiv:2302.03917_ , 2023a.
  * Huang et al. [2023b] Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, et al.  AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head.  _arXiv preprint arXiv:2304.12995_ , 2023b.
  * Iashin and Rahtu [2020] Vladimir Iashin and Esa Rahtu.  Multi-modal Dense Video Captioning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , pages 958–959, 2020.
  * Ilyas and Rehman [2019] Sadia Ilyas and Hafeez Ur Rehman.  A Deep Learning based Approach for Precise Video Tagging.  In _2019 15th International Conference on Emerging Technologies (ICET)_ , pages 1–6. IEEE, 2019.
  * Ji et al. [2019] Zhong Ji, Kailin Xiong, Yanwei Pang, and Xuelong Li.  Video Summarization with Attention-based Encoder-Decoder Networks.  _IEEE Transactions on Circuits and Systems for Video Technology_ , 30(6):1709–1717, 2019.
  * Kilgour et al. [2019] Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi.  Fréchet Audio Distance: A Reference-Free Metric for Evaluating Music Enhancement Algorithms.  In _Interspeech_ , 2019.
  * Lei et al. [2018] Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg.  TVQA: Localized, Compositional Video Question Answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1369–1379. Association for Computational Linguistics, 2018.
  * Li et al. [2022] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation.  In _International Conference on Machine Learning_ , pages 12888–12900. PMLR, 2022.
  * Li et al. [2023a] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.  _arXiv preprint arXiv:2301.12597_ , 2023a.
  * Li et al. [2023b] Yizhi Li, Ruibin Yuan, Ge Zhang, Yinghao Ma, Xingran Chen, Hanzhi Yin, Chenghua Lin, Anton Ragni, Emmanouil Benetos, Norbert Gyenge, et al.  MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training.  _arXiv preprint arXiv:2306.00107_ , 2023b.
  * Li et al. [2019] Zhihui Li, Lina Yao, Xiaojun Chang, Kun Zhan, Jiande Sun, and Huaxiang Zhang.  Zero-shot Event Detection via Event-adaptive Concept Relevance Mining.  _Pattern Recognition_ , 88:595–603, 2019.
  * Lin [2004a] Chin-Yew Lin.  ROUGE: A Package for Automatic Evaluation of Summaries.  In _Text summarization branches out_ , pages 74–81, 2004a.
  * Lin [2004b] Chin-Yew Lin.  ROUGE: A Package for Automatic Evaluation of Summaries.  In _Text summarization branches out_ , pages 74–81, 2004b.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft COCO: Common Objects in Context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * Liu et al. [2023a] Haohe Liu, Qiao Tian, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D. Plumbley.  AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining.  _arXiv preprint arXiv:2308.05734_ , 2023a.
  * Liu et al. [2023b] Shansong Liu, Atin Sakkeer Hussain, Chenshuo Sun, and Ying Shan.  Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioning.  _arXiv preprint arXiv:2308.11276_ , 2023b.
  * Liu et al. [2023c] Xubo Liu, Zhongkai Zhu, Haohe Liu, Yi Yuan, Qiushi Huang, Jinhua Liang, Yin Cao, Qiuqiang Kong, Mark D Plumbley, and Wenwu Wang.  WavJourney: Compositional Audio Creation with Large Language Models.  _arXiv preprint arXiv:2307.14335_ , 2023c.
  * Lyu et al. [2023] Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu.  Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration.  _arXiv preprint arXiv:2306.09093_ , 2023.
  * Manco et al. [2021] Ilaria Manco, Emmanouil Benetos, Elio Quinton, and György Fazekas.  MusCaps: Generating Captions for Music Audio.  In _2021 International Joint Conference on Neural Networks (IJCNN)_ , pages 1–8. IEEE, 2021.
  * Mei et al. [2021] Xinhao Mei, Xubo Liu, Qiushi Huang, Mark D. Plumbley, and Wenwu Wang.  Audio Captioning Transformer.  In _Proceedings of the 6th Detection and Classification of Acoustic Scenes and Events 2021 Workshop (DCASE2021)_ , pages 211–215, 2021.
  * Moon et al. [2022] Jong Hak Moon, Hyungyung Lee, Woncheol Shin, Young-Hak Kim, and Edward Choi.  Multi-Modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training.  _IEEE Journal of Biomedical and Health Informatics_ , 26(12):6070–6080, 2022.
  * Muhammad Maaz and Khan [2023] Salman Khan Muhammad Maaz, Hanoona Rasheed and Fahad Khan.  Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models.  _arXiv preprint arXiv:2306.05424_ , 2023.
  * OpenAI [2023] OpenAI.  ChatGPT (Mar 14 version) [Large language model], 2023.
  * Papineni et al. [2002a] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  BLEU: A Method for Automatic Evaluation of Machine Translation.  In _ACL_ , pages 311–318, 2002a.
  * Papineni et al. [2002b] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  BLEU: A Method for Automatic Evaluation of Machine Translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318, 2002b.
  * Schneider et al. [2023] Flavio Schneider, Zhijing Jin, and Bernhard Schölkopf.  Mo$\backslash$^ usai: Text-to-Music Generation with Long-Context Latent Diffusion.  _arXiv preprint arXiv:2301.11757_ , 2023.
  * Su et al. [2023] Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai.  PandaGPT: One Model To Instruction-Follow Them All.  _arXiv preprint arXiv:2305.16355_ , 2023.
  * Sun et al. [2023] Chunyi Sun, Junlin Han, Weijian Deng, Xinlong Wang, Zishan Qin, and Stephen Gould.  3D-GPT: Procedural 3D Modeling with Large Language Models.  _arXiv preprint arXiv:2310.12945_ , 2023.
  * Tang et al. [2023a] Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang.  SALMONN: Towards Generic Hearing Abilities for Large Language Models.  _arXiv preprint arXiv:2310.13289_ , 2023a.
  * Tang et al. [2023b] Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal.  Any-to-Any Generation via Composable Diffusion.  _arXiv preprint arXiv:2305.11846_ , 2023b.
  * Taori et al. [2023] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford Alpaca: An Instruction-following LLaMA model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * Team [2023] MosaicML NLP Team.  Introducing MPT-7B: A New Standard for Open-Source, Commercially Usable LLMs, 2023.  Accessed: 2023-05-05.
  * Tong et al. [2022] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang.  VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open Foundation and Fine-tuned Chat Models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Vartakavi et al. [2021] Aneesh Vartakavi, Amanmeet Garg, and Zafar Rafii.  Audio Summarization for Podcasts.  In _2021 29th European signal processing conference (EUSIPCO)_ , pages 431–435. IEEE, 2021.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is All You Need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. [2023a] Yuancheng Wang, Zeqian Ju, Xu Tan, Lei He, Zhizheng Wu, Jiang Bian, and Sheng Zhao.  AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models.  _arXiv preprint arXiv:2304.00830_ , 2023a.
  * Wang et al. [2023b] Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, and Zhou Zhao.  Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenes.  _arXiv preprint arXiv:2308.08769_ , 2023b.
  * Wu et al. [2022] Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello.  Wav2CLIP: Learning Robust Audio Representations From CLIP.  In _ICASSP_ , pages 4563–4567. IEEE, 2022.
  * Wu et al. [2023a] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua.  NExT-GPT: Any-to-Any Multimodal LLM.  _arXiv preprint arXiv:2309.05519_ , 2023a.
  * Wu et al. [2023b] Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov.  Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation.  In _IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP_ , 2023b.
  * Xu et al. [2023a] Li Xu, Bo Liu, Ameer Hamza Khan, Lu Fan, and Xiao-Ming Wu.  Multi-modal Pre-training for Medical Vision-language Understanding and Generation: An Empirical Study with A New Benchmark.  In _Proceedings of the Conference on Health, Inference, and Learning_ , pages 117–132, 2023a.
  * Xu et al. [2023b] Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin.  PointLLM: Empowering Large Language Models to Understand Point Clouds.  _arXiv preprint arXiv:2308.16911_ , 2023b.
  * Yang et al. [2021] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid.  Just Ask: Learning to Answer Questions from Millions of Narrated Videos.  In _ICCV_ , 2021.
  * Yang et al. [2023] Zhen Yang, Yingxue Zhang, Fandong Meng, and Jie Zhou.  TEAL: Tokenize and Embed ALL for Multi-modal Large Language Models.  _arXiv preprint arXiv:2311.04589_ , 2023.
  * Yao et al. [2017] Ting Yao, Yingwei Pan, Yehao Li, Zhaofan Qiu, and Tao Mei.  Boosting Image Captioning with Attributes.  In _Proceedings of the IEEE international conference on computer vision_ , pages 4894–4902, 2017.
  * Yin et al. [2023] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen.  A Survey on Multimodal Large Language Models.  _arXiv preprint arXiv:2306.13549_ , 2023.
  * Zhai et al. [2022] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer.  Scaling Vision Transformers.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12104–12113, 2022.
  * Zhang et al. [2023a] Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuangrui Ding, Songyang Zhang, Haodong Duan, Hang Yan, et al.  InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition.  _arXiv preprint arXiv:2309.15112_ , 2023a.
  * Zhang et al. [2022] Runbang Zhang, Yixiao Zhang, Kai Shao, Ying Shan, and Gus Xia.  Vis2Mus: Exploring Multimodal Representation Mapping for Controllable Music Generation.  _arXiv preprint arXiv:2211.05543_ , 2022.
  * Zhang* et al. [2020a] Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi.  BERTScore: Evaluating Text Generation with BERT.  In _ICLR_ , 2020a.
  * Zhang* et al. [2020b] Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi.  BERTScore: Evaluating Text Generation with BERT.  In _ICLR_ , 2020b.
  * Zhang et al. [2016] Yang Zhang, Boqing Gong, and Mubarak Shah.  Fast Zero-shot Image Tagging.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5985–5994. IEEE, 2016.
  * Zhang et al. [2023b] Yixiao Zhang, Akira Maezawa, Gus Xia, Kazuhiko Yamamoto, and Simon Dixon.  Loop Copilot: Conducting AI Ensembles for Music Generation and Iterative Editing.  _arXiv preprint arXiv:2310.12404_ , 2023b.
  * Zhao et al. [2023] Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar.  Learning Video Representations from Large Language Models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6586–6597, 2023.
  * Zhuo et al. [2023] Le Zhuo, Zhaokai Wang, Baisen Wang, Yue Liao, Chenxi Bao, Stanley Peng, Songhao Han, Aixi Zhang, Fei Fang, and Si Liu.  Video Background Music Generation: Dataset, Method and Evaluation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 15637–15647, 2023.

The appendix presents supplementary details that extend beyond the content of
the manuscript, aiming to enhance comprehension of the M2UGen model.
Comprehensive information is provided concerning the model’s training dataset
and training methodology, encompassing explicit insights into the utilized
training approach and the corresponding model hyperparameters. Additionally, a
thorough exposition is given regarding the composition of the evaluation sets
employed in our study, accompanied by a delineation of the evaluation
methodology and metrics applied to assess the performance of our model. To
elucidate the diverse capabilities of our model, illustrative demo examples
are also included.

##  Appendix A Music Oriented Dataset Information

We generate 4 different datasets to train the M2UGen model: MUCaps, MUImage,
MUVideo and MUEdit datasets. The statistics of the datasets are given in Table
7. An example of each from the 4 datasets are shown in Figure 4.

Table 7: Dataset Statistics. The number of instructions in the dataset and total hours of music files in the dataset Dataset | Instruction Count | Hours of Music
---|---|---
MUCaps | 21966 | 1273.78
MUImage | 9966 | 27.72
MUVideo | 13203 | 36.72
MUEdit | 10815 | 60.22
Figure 4: Music Oriented Dataset. Examples from the MUCaps, MUEdit, MUImage
and MUVideo datasets used to train the M2UGen model.

##  Appendix B Model Training

In this section, we detail the training strategy for the M2UGen model along
with parameters used for training.

###  B.1 Model Training Strategy

The M2UGen model adopts the adapter training strategy, implementing a three-
step training regimen. In the first phase, all parameters, with the exception
of those associated with the Multi-modal Understanding Adapters, undergo
freezing. The training dataset is configured to incorporate the MUCaps dataset
for music understanding, the COCO dataset for image comprehension, and the
captions sourced from the MUVideo dataset for video understanding. During this
training stage, the Cross Entropy Loss function is applied to compute the
disparity between the caption generated by the LLaMA 2 model and the target
caption corresponding to the input modality. This process is illustrated in
Figure 5.

Figure 5: Training Stage 1: The Multi-modal Understanding Adapters are trained
to integrate multi-modal features into the different layers of the LLaMA 2
model.

In the second training stage, the output projector is trained to generate
conditional embeddings using input captions processed by the LLaMA 2 model.
The LLaMA 2 model produces specialized audio tokens, denoted as [AUD$i$] where
$i\in\\{1,2,\ldots,K\\}$ (with $K$ as a hyperparameter representing the number
of special audio tokens added to the LLaMA 2 model’s vocabulary) when
processing input captions. The special audio tokens serve as signaling
indicators, aiding the model in determining whether to generate text+music or
solely text. In training, these audio tokens are added to the end of the text
output in datasets requiring music output. During inference, if the M2UGen
model generates audio tokens, downstream music decoders (MusicGen/AudioLDM 2)
will perform music generation, otherwise, solely text will be produced.

The hidden embeddings corresponding to these audio tokens from the last layer
of the LLaMA 2 model is then input into the output projection layer,
generating the conditional embedding for the Music Generation model. The
MUCaps dataset is utilized to train this stage, with captions serving as
inputs to the model and the target output tokens set as the special audio
tokens.

Figure 6: Training Stage 2: The Output Projection Layer is trained to generate
the conditioning embedding for the MusicGen/AudioLDM 2 model.

Assuming a total of $N$ tokens generated by the LLaMA 2 model, where [AUD$i$]
with $i\in\\{1,2,\ldots,8\\}$ constitutes the last $8$ tokens. The hidden
embeddings size is $(1,N,4096)$, and the last 8 tokens are extracted along
dimension $-1$, resulting in an input embedding size of the Output Projection
layer as $(1,8,4096)$. The output size from the projection layer varies based
on the Music Generation model: for AudioLDM2, it is $(1,512)$, and for
MusicGen, it is $(512,768)$.

In the final training stage, the LoRA training strategy is employed to train
the LLaMA 2 model, concurrently fine-tuning the Multi-modal Understanding
Adapter and Output Projection layer. This stage utilizes datasets including
Alpaca, MusicQA, MUImage, MUVideo, and MUEdit. To signal the M2UGen model to
generate both music and text, the output text in MUImage, MUVideo, and MUEdit
datasets is extended with the special audio tokens.

Figure 7: Training Stage 3: The Multi-modal Understanding Adapter and Output
Projection Layer are fine-tuned while the LoRA-enabled LLaMA 2 model is
trained in this stage.

###  B.2 Reasoning for Training Strategy

Using the initial training stage, the model undergoes training with the
objective of comprehending diverse modalities by leveraging extensive
captioning datasets for music, image, and video. The subsequent training stage
focuses on refining the LLaMA 2 model’s capability to condition music
generation based on input captions.

These dual training stages equip the model with the ability to grasp various
modalities and generate distinct music conditions based on input captions.
This proficiency significantly contributes to the final training stage. In
this ultimate phase, the model leverages the trained Multi-modal Understanding
Adapters and Output Projection layers to bridge the gap between them, honing
the LLaMA 2 model’s skills through the utilization of multi-modal music
generation and music understanding datasets.

###  B.3 Model Training Parameters

We conduct training for the three stages of our model, employing 5, 5, and 2
epochs, respectively. The training process incorporates the following hyper-
parameters: $N=32$, $L=6$, $\text{number of Audio Tokens}=8$, and
$lr=10^{-4}$. This choice of hyper-parameters, coupled with our training
strategy, allows for the effective use of a reduced number of epochs and a
smaller dataset during the final stage of model training.

##  Appendix C Model Evaluation

In this section, we elaborate on the datasets employed to assess the various
capabilities of the M2UGen model, followed by a discussion of the evaluation
metrics utilized.

###  C.1 Evaluation Datasets

For each of the training datasets generated—MUCaps, MUImage, MUVideo, and
MUEdit—we create a corresponding evaluation set. The methodology employed for
generating the evaluation dataset mirrors that of the training dataset
generation. Detailed statistics for the evaluation datasets are provided in
Table 8. However, for the MUEdit dataset, the evaluation set could not be
utilized for model evaluation due to the unavailability of code bases and
trained checkpoints for InstructME[27] and AUDIT[68]. Consequently, we resort
to utilizing samples from InstructME’s demo website, which includes samples
from both AUDIT and InstructME, to assess our model’s performance. For
evaluating the M2UGen’s music understanding capabilities we utilize the
evaluation split of the MusicQA dataset.

Table 8: Evaluation Dataset Statistics. The number of instructions in the evaluation dataset and total hours of music files in the dataset Dataset | Instruction Count | Hours of Music
---|---|---
MUCaps | 4000 | 265.35
MUImage | 2500 | 6.94
MUVideo | 2500 | 6.94
MUEdite | 2000 | 5.55

###  C.2 Evaluation Metrics

To assess music question answering, we adopt the metrics employed in [47],
namely BLEU (B-U) [55], METEOR (M-R) [5], ROUGEL (R-L) [43], and BERT-Score
(BERT-S) [82]. These metrics are widely used for evaluating text generation.
For all music generation tasks, we employ Fréchet Audio Distance (FAD) [37]
and Kullback-Leibler divergence (KL), as these metrics are commonly utilized
to assess the quality of generated audio. In addition to these general
metrics, task-specific metrics are applied for each of the music generation
tasks, namely Text-to-Music, Image-to-Music, Video-to-Music, and Music
Editing.

In the context of Text-to-Music, we employ the CLAP[72] score, calculated by
determining the cosine similarity between the CLAP embedding for the generated
music and the text input:

$CLAPScore(M,T)=\max(100\times\cos(E_{M},E_{T}),0)$

Here, $M$ represents the generated music, $T$ denotes the text input, and
$E_{M}$, $E_{T}$ represent the CLAP embeddings for the music and text,
respectively.

For the Music Editing task, we leverage the Log Spectral Distance (LSD) to
assess the disparity between the generated music and the target music. This
metric facilitates the evaluation of whether the frequencies in the generated
music, post-editing, align with those in the target music.

For Image-to-Music and Video-to-Music tasks, we introduce the ImageBind[23]
Ranking (IB Rank), akin to the CLAP score, to quantify the alignment between
the input modality and the generated music. Considering $N$ distinct models
producing $N$ music files, we generate ImageBind embeddings for the music
files, denoted as $E_{M1},E_{M2},\ldots,E_{MN}$, as well as the ImageBind
embedding for the input modality, denoted as $E_{I/V}$. The embeddings for the
music files are ranked based on their cosine similarity to $E_{I/V}$.
Subsequently, after ranking all the samples in the evaluation set, the
ImageBind Ranking is computed by calculating the ranking score using the
individual rankings.

Using these evaluation metrics, we are able to evaluate the M2UGen model
against other state-of-the-art models for the different tasks.

##  Appendix D Model Demonstration

In this section, we present screenshots of the M2UGen model demo, illustrating
various capabilities of the model.

Figures 8, 9, and 10 showcase the M2UGen model’s ability to generate music
directly from textual prompts and draw inspiration from images and videos,
both with and without textual guidance. Figure 11 exemplifies M2UGen’s
proficiency in music editing guided by natural language prompts. Additionally,
Figures 12, 13, and 14 illustrate the utilization of M2UGen’s editing
capabilities to further refine music generated from different modalities.
Collectively, the M2UGen model proves to be a robust framework for Music
Understanding, Generation, and Editing.

Figure 8: Text-To-Music Generation and Understanding: The M2UGen model is
capable of generating music from text prompts and also answering questions
regarding the generated music. Figure 9: Image-To-Music Generation and
Understanding: The M2UGen model is capable of generating music for images and
also answering questions regarding the generated music. Figure 10: Video-To-
Music Generation and Understanding: The M2UGen model is capable of generating
music for videos and also answering questions regarding the generated music.
Figure 11: Music Editing and Understanding: The M2UGen model is capable of
editing input music based on natural language prompts and also answering
questions regarding the generated music. Figure 12: Text-To-Music Generation
and Understanding + Music Editing: The M2UGen model is capable of generating
music from text prompts, answering questions regarding the generated music and
also editing the generated music using Natural Language prompts. Figure 13:
Image-To-Music Generation and Understanding + Music Editing: The M2UGen model
is capable of generating music for images, answering questions regarding the
generated music and also editing the generated music using Natural Language
prompts. Figure 14: Video-To-Music Generation and Understanding + Music
Editing: The M2UGen model is capable of generating music for videos, answering
questions regarding the generated music and also editing the generated music
using Natural Language prompts.


# Evaluating NLP Systems On a Novel Cloze Task: Judging the Plausibility of
Possible Fillers in Instructional Texts

Zizhao Hu
University of Southern California
Ravikiran Chanumolu
University of Southern California
Xingyu Lin
University of Southern California
Nayela Ayaz
University of Southern California
Vincent Chi
University of Southern California

###### Abstract

Cloze task is a widely used task to evaluate an NLP system’s language
understanding ability. However, most of the existing cloze tasks only require
NLP systems to give the relative best prediction for each input data sample,
rather than the absolute quality of all possible predictions, in a consistent
way across the input domain. Thus a new task is proposed: predicting if a
filler word in a cloze task is a good, neutral, or bad candidate. Complicated
versions can be extended to predicting more discrete classes or continuous
scores. We focus on subtask A in Semeval 2022 task 7, explored some possible
architectures to solve this new task, provided a detailed comparison of them,
and proposed an ensemble method to improve traditional models in this new
task.

##  1 Introduction & Related Work

WikiHow is the largest and the most trusted how-to website. It has more than
300,000 articles and over 2.5 million registered users. The primary goal of
the articles is to help users achieve a specific goal like how to cook a
Thanksgiving turkey or how to deal with an online stalker. The instructional
articles frequently undergo revisions to correct grammatical errors, elaborate
on the underspecified text and remove ambiguities. These clarifications are
necessary to ensure the best user experience. The goal of the SemEval-2022
Task 7 is to evaluate the ability of NLP systems to automatically distinguish
between plausible and implausible clarifications of instructional text. The
task is formulated as a cloze task, in which clarifications are presented as
possible fillers in a masked sentence. For each (sentence, filler) pair the
system must find if the clarification is plausible, implausible, or neutral.

TF-idf, Glove, and Word2Vec are widely used non-contextual embedding
techniques, and such embeddings containing useful information on word-level
without contextual information can be used to solve various NLP tasks.

However, recent discoveries in contextualized embeddings and language models
have shown their dominance. The current best architecture for solving language
understanding tasks is based on fine-tuning a large pre-trained language model
on the downstream task [Radford et al., 2018]. Pre-trained language models
such as BERT [Devlin et al., 2018], ELMo [Peters et al., 2018], OpenAI GPT
[Radford et al., 2018] have shown state-of-the-art performance on multiple
language understanding tasks. And evidence of such models’ ability to have
commonsense knowledge [Cui et al., 2020] gives the empirical basis to use such
models for the cloze task, which is a sub-task that requires commonsense in
the language understanding domain.

Masked language modeling(MLM) task, a self-supervised, cloze task format
pretraining objective, is widely used in the pre-training for transformer
based language models. And due to its training nature, the pre-trained models
with such objectives without fine-tuning can be directly applied to solve a
general cloze task. Further fine-tuning on the training dataset can be
conducted in the same manner as pre-training, to possibly improve the
performance.

The ability of the NLP system to give consistent judging(scoring) across the
whole input domain has been exploited in the domain of Out-of-
distribution(OOD) detection. There is proof that a well-designed scoring
system can perform well in OOD detection. Such scoring system can be directly
built upon the softmax output of the classification task) [Hendrycks and
Gimpel, 2016] or distance based [Chang et al., 2017] (Training confidence
calibrated classifiers for detecting out-of-distribution samples [Lee et al.,
2017]). Inspired by these discoveries, we designed our scoring systems

##  2 Methods

###  2.1 Data

We took the data and labels from Semeval 2022 task 7. The data consists of the
article title, the subheading, a masked sentence with its previous and next
instructions, and the possible fillers with corresponding labels and ratings
that predict the plausibility score. The label has 3 classes, positive
(plausible), neutral, and negative (implausible), each indicating the quality
of each candidate word. Numerical plausibility scores from 1-5 are also given
for each training data example, and we can see the distribution in figure 3.

Following methods are used to preprocess the input texts:

  1. 1.

Preprocessing Method 1: Concatenating titles, subheading, previous
instruction, masked sentence, next instruction into one long sentence.

  2. 2.

Preprocessing Method 2: Concatenating previous instruction, masked sentence,
next instruction into one long sentence.

  3. 3.

Preprocessing Method 3: Only use the masked sentence.

  4. 4.

Additional Preprocessing for MLM: Remove the first word if the filling word is
a two-word pair: ’My book’, ’The table’. Since the model can predict one
masked word at a time.

Figure 1: Sample Training Data with scores

Figure 2: Sample Training Data with labels

Figure 3: Training Data Distributions

###  2.2 High Level Approaches

We define a set of variables and functions
$\\{X,X^{\prime},y,s,l,f_{emb},f_{score},f_{cls},f_{reg}\\}$. Here
$\\{x_{1},x_{2},...x_{n}\\}\in X$ is a set of the masked input text. For each
masked input text $x_{n}$, it has 5 candidate filling words
$\\{y_{n_{i}},i=1,2,...5\\}$. Each filling word has a corresponding label
$l_{n_{i}}\in\\{positive(5),neutral(3),negative(1)\\}$ and a corresponding
score $s_{n_{i}}\in[1,5]$. $\\{x^{\prime}_{n_{i}},i=1,2,...5\\}\in X^{\prime}$
is the set of filled input text. $f_{emb}$ is a non-contextual embedding
method or contextual embedding taken from a language model. $f_{score}$ is a
function that translates a filling word probability into a score, we’ll
discuss this in detail in section 3.1.2. $f_{cls}$ is a multi-class
classifier. $f_{reg}$ is a regression model.

####  2.2.1 Approach 1

Use embeddings to train a classifier
$f_{cls}(f_{emb}(x^{\prime}_{n_{i}}),l_{n_{i}})$. This approach corresponds to
Experiment sections 3.1.1.

####  2.2.2 Approach 2

Use a score to train a
classifier$f_{cls}(f_{prob}(P(y_{n_{i}}|context)),l_{n_{i}})$, here the
context is $x_{n}$ for MLM and neighboring words for N-gram models. This
approach corresponds to Experiment sections 3.1.2 and 3.1.2.i.

####  2.2.3 Approach 3

Use a score to train a regression model
$f_{reg}(f_{prob}(P(y_{n_{i}}|context)),l_{n_{i}})$. This approach corresponds
to Experiment sections 3.1.3 and 3.1.4.ii. The rule to decide labels is
discussed in detail in the corresponding sections.

####  2.2.4 Evaluation Methods

We use overall accuracy and class-wise fscore to measure the classification
performance. In addition, we also use Spearman’s rank correlation
coefficient(Spearman rank) to assess monotonic relationships between the true
scores and predicted labels. Since the labels have ordinal properties, we
translate them to scores to make this evaluation method possible. the possible
values ranging from -1 to 1. With 1 indicating the strongest relationship, -1
the opposite.

###  2.3 Language Models

In this section we describe the different transformer models used in this
work.

  1. i.

Bert: Bert [Devlin et al., 2018] is a multi-layer transformer encoder model
which achieved state-of-the-art performance on a wide variety of NLP tasks[].
The key technical innovation in Bert is using bidirectional self-attention to
learn text representations. The paper’s results show that a language model
which is bi-directionally trained can have a deeper sense of language context
than single-direction language models. Bert is therefore an ideal LM candidate
for our task as it can score the correctness of each (sentence, filler) pair
by jointly conditioning on both the right and left context.

  2. ii.

XLNet: XLNet [Yang et al., 2019] is a generalized autoregressive language
model that is trained by maximizing log-likelihood over all the possible
permutations of words in a sentence, thereby getting bidirectional context
information. XLNet is a good candidate for our task because such a model
learns to get info from all positions in a sentence to determine the
likelihood of a word at a certain position, which makes it a good candidate
for our task, as plausibility depends on the entire sequence.

  3. iii.

Electra: Electra replaces the MLM training objective of BERT with Replaced
Token Detection (RTD) [Clark et al., 2020]. The Electra model has two
components a generator model and a discriminator model. The discriminator
model takes as input a corrupted sequence. For each token in the sequence, it
outputs a probability score based on whether the token is corrupted or not.
The generator model is used to create the corrupted sequence. This is achieved
by masking an input sentence and using a small Bert model, trained through MLM
to replace the mask with the most likely token. This formulation allows the
decoder to learn from every token in the input sequence. In contrast, with the
MLM objective, Bert only learns from 15% of the tokens. After pretraining, the
generator model is discarded and the discriminator model is used for fine-
tuning the downstream tasks.

##  3 Experiments

###  3.1 Main Experiments

Here we tested several model architectures:

####  3.1.1 Classification with sentence embeddings

In this approach, we convert each (sentence, filler) pair to a fixed size
representation and train a simple classification model over them. To obtain
the sentence representations from word embeddings we use the following
approaches: When using Glove we use the simple average of word embeddings to
obtain sentence representation. In the case of transformer models, we take the
average of the word embeddings obtained from the last layer. Further, we use
the embedding of the first word-piece token as the word embedding. We tried
other variations which included averaging and concatenating the embeddings of
different layers in the transformer. But this did not result in performance
improvement. We omit these variations in the results section due to limited
space. Below we enumerate different models using this approach. For each
model, we show the embedding method followed by the classifier.

  1. (1)

glove + Logistic Regression

  2. (2)

BERT + Logistic Regression

  3. (3)

XLNet + Logistic Regression

  4. (4)

Electra + Logistic Regression

####  3.1.2 Classification with MLM scores

In this approach, we take advantage of the similarity between our task and the
MLM task and use the output logits for the whole vocabulary to train a simple
classification model to map the logits to classification labels. We first
translate the logits to scores(will discuss below). Then a classification
model is used to map the score to predicted labels. Here are some ways to
translate the logits to scores:

  1. i.

Logits: Use the logits directly as scores.

    1. (9)

BERT + Naive Bayes Classifier

  2. ii.

Soft-max: Add a soft-max layer to the MLM and use the soft-max probability as
scores.

    1. (10)

BERT + Naive Bayes Classifier

  3. iii.

Soft-max + Top-1 Embedding Similarity: Add a soft-max layer to the MLM, take
the word with the highest probability. Filling the word into the masked
sentence and acquiring the average embedding according to 3.2.1.i. Fill the
candidate word into the masked sentence and acquire the average embedding. Use
the cosine similarity between these two embeddings as the score.

    1. (11)

BERT + Naive Bayes Classifier

  4. iv.

Soft-max + Weighted Top-5 Embedding Similarity: The same way to acquire the
cosine similarity score. But we choose the top 5 highest soft-max probability
words and use the sum of the 5 cosine similarity score weighted by the
corresponding soft-max probabilities as the score.

    1. (12)

BERT + Naive Bayes Classifier

  5. v.

Soft-max + Maximum Top-5 Embedding Similarity: The same way to acquire the
cosine similarity score. But we choose the top 5 highest soft-max probability
words and use the largest cosine similarity as the score.

    1. (13)

BERT + Naive Bayes Classifier

####  3.1.3 Regression with MLM scores

For a regression model, we use the values 1, 3, 5 for negative, neutral, and
positive labels as the dependent variable. Then we set 2 thresholds on the
dependent values, above which neutral and positive are predicted respectively
otherwise negative. The thresholds are set to values that will predict the
same amount of labels for each class as there are in the training set. The
scores are acquired in the same way as mentioned in the previous section:

  1. i.

Logits:

    1. (14)

BERT + Linear Regression

  2. ii.

Soft-max:

    1. (15)

BERT + Linear Regression

Method | accuracy | fscore0 | fscore1 | fscore2 | spearman rank
---|---|---|---|---|---
Task Baseline
tf-idf + NB | 0.38 | 0.01 | 0.15 | 0.56 | 0.024
Classification with sentence embeddings
glove + LR | 0.34 | 0.09 | 0.22 | 0.52 | 0.098
BERTbase \+ LR | 0.38 | 0.33 | 0.23 | 0.50 | 0.074
XLNet + LR | 0.36 | 0.29 | 0.25 | 0.46 | 0.019
Electra + LR | 0.40 | 0.33 | 0.25 | 0.53 | 0.150
Classification with MLM scores
BERTbase \+ NB | 0.45 | 0.59 | 0.01 | 0.26 | 0.244
BERTbase+adapt \+ NB | 0.45 | 0.59 | 0.02 | 0.27 | 0.251
BERTlarge \+ NB | 0.46 | 0.60 | 0.02 | 0.28 | 0.288
XLNet + NB | 0.44 | 0.04 | 0.24 | 0.60 | 0.095
Classification with RTD scores
Electra + NB | 0.39 | 0.01 | 0.25 | 0.58 | 0.088
Regression with MLM scores
BERTbase \+ LinearReg | 0.47 | 0.49 | 0.27 | 0.58 | 0.350
BERTlarge \+ LinearReg | 0.48 | 0.50 | 0.28 | 0.60 | 0.398
Using N-gram probabilities
N-gram + NB | 0.38 | 0.00 | 0.25 | 0.58 | 0.107
N-gram + LinearReg | 0.34 | 0.21 | 0.28 | 0.47 | 0.109
Pretraining with version distinction task
Electra + NB | 0.36 | 0.18 | 0.20 | 0.52 | 0.164

Table 1: Results

In this table we show performance of models listed under section 3.1 - Main
Experiments. Note: All transformer models are base models unless explicitly
mentioned.

Method | accuracy | fscore0 | fscore1 | fscore2 | spearman rank
---|---|---|---|---|---
Bertlarge \+ Naive Bayes(10)
Pre-processing 1 | 0.46 | 0.60 | 0.02 | 0.28 | 0.288
Pre-processing 2 | 0.46 | 0.60 | 0.02 | 0.28 | 0.267
Pre-processing 3 | 0.44 | 0.59 | 0.05 | 0.22 | 0.216
Bertlarge \+ Linear Regression(15)
Pre-processing 1 | 0.48 | 0.50 | 0.28 | 0.60 | 0.398
Pre-processing 2 | 0.48 | 0.48 | 0.29 | 0.60 | 0.363
Pre-processing 3 | 0.45 | 0.43 | 0.28 | 0.58 | 0.300
Table 2: Compare Preprocessing Methods Method | accuracy | fscore0 | fscore1 | fscore2 | spearman rank
---|---|---|---|---|---
Naive Bayes
Bertlarge(10) | 0.46 | 0.60 | 0.02 | 0.28 | 0.289
N-gram(16) | 0.38 | 0.00 | 0.25 | 0.58 | 0.107
Electra | 0.39 | 0.01 | 0.25 | 0.58 | 0.088
Ensemble(10)+(16) | 0.46 | 0.60 | 0.02 | 0.29 | 0.290
Ensemble(10)+(16)+Electra | 0.46 | 0.60 | 0.02 | 0.29 | 0.295
Linear Regression
Bertlarge(10) | 0.48 | 0.50 | 0.28 | 0.60 | 0.398
N-gram(16) | 0.34 | 0.21 | 0.28 | 0.47 | 0.109
Electra | 0.36 | 0.38 | 0.22 | 0.44 | 0.151
Ensemble(10)+(16) | 0.44 | 0.44 | 0.29 | 0.53 | 0.334
Ensemble(10)+(16)+Electra | 0.42 | 0.41 | 0.26 | 0.52 | 0.339
Table 3: Compare Ensemble Methods

####  3.1.4 Using N-gram Probabilities

We use the previous word, the filling word, and the next word to form a tri-
gram(or quadri-gram if the filling words contain 2 words). We acquire the
n-gram frequency from Google Books N-gram model 1900 - 2019 and use it in
replacement of the score mentioned in the previous section. We also applied
similar classification and regression models. An additional rule is added to
the regression model: if the n-gram frequency is 0, it’s automatically mapped
to negative:

  1. i.

Classification:

    1. (16)

N-gram + Naive Bayes Classifier

  2. ii.

Regression:

    1. (17)

N-gram + Linear Regression

####  3.1.5 Classification with replaced token detection (RTD) scores

As described in section 2.3 (ii) the Electra decoder takes a text sequence as
input and for each token in the sequence outputs the probability that the
token in corrupted/replaced. We use the RTD score corresponding to the filler
word as input to the Naive Bayes classifier to get the predicted class for the
(sentence, filler) pair.

####  3.1.6 Domain adaption with MLM

Although BERT aims to learn contextualized representations across a wide range
of tasks, leveraging BERT alone still leaves the domain challenge and low
resource challenges unsolved. BERT is trained on large text corpus(Wikipedia)
and also requires a large number of samples to do fine-tuning. It suffers
considerably at low-resource when applied to a different domain. However, in
this task, we only have a limited number of samples(thousands of sentences),
which is insufficient to fine-tune BERT to ensure full task awareness. So, we
adopt BERT as the base model and adapt it to our specific domain using a
masked language modeling(MLM) loss.

We conduct domain adaption using the WikiHow dataset, a large-scale dataset
using the online WikiHow knowledge base. It contains more than 200K articles,
each of which is formed by merging multiple paragraphs.

####  3.1.7 Pretraining with version distinction task

Apart from pretraining the transformer models on the wikiHow dataset using
MLM, we also try to pretrain using the version distinction task [Anthonio et
al., 2020] [Bhat et al., 2020]. The version distinction task involves
differentiating between the older and newer version of a sentence in wikiHow.
The dataset for this task was created using the revision history of the
wikiHow corpus [Anthonio et al., 2020]. Similar to the Bert Model discussed by
the authors, we add a regression head to a transformer model to predict a
score $s$ given an instruction. More formally, given a sentence pair
$(s_{i},s_{j})$ corresponding to an old and new sentence, we first convert the
sentences to numerical embeddings $(e_{i},e_{j})$ using a transformer encoder.
Then the embeddings are passed to a stack of linear layers to obtain scores
$(\phi(e_{i}),\phi(e_{j}))$. The transformer encoder and the linear layers are
trained using the following pairwise ranking loss:

$l_{p}=max(0,\phi(e_{i})-\phi(e_{j})+1)$

We then train a simple Naive Bayes classifier using the 1-D scores from the
above-described model to get the (sentence, filler) classification label. We
hypothesize that this model learns to score a sentence based on its
grammatical and semantic correctness. As a result, for our cloze task, the
model should give a higher score to plausible(sentence, filler) pairs compared
to implausible pairs. Lastly, we use an Electra model as the transformer
encoder as it is faster to train.

####  3.1.8 Baselines

Organisers of the task provided the following model as the baseline to improve
on.

  * (0)

tf-idf + Naive Bayes Classifier

###  3.2 Compare Preprocessing Methods

Model (10) and Model (15) are used to evaluate different preprocessing
methods(1, 2, and 3).

###  3.3 Ensemble Experiments

We use an ensemble score from (10) and (16) to train a classifier, and (15)
and (17) to train a regression model, to see if there is an improvement
compared to single models:

##  4 Results

From Table 1 under ’classification with sentence embeddings’ we see that the
Electra model has the best overall performance. In both Bert and XLNet the
[MASK] token is only present during pre-training, which precedes the fine-
tuning step. This results in different token distributions for the two stages.
In contrast, the Electra decoder is trained on complete sequences with no
masking. In the sentence embedding approach also, we replace the blank in the
instruction with the filler and pass the complete sequence as input to the
transformer.

Further from Table 1, we observe that using MLM scores Bert outperforms both
XLnet and Electra. The reason for this could be the bidirectional training of
Bert. Further, as described in section 2.3 (ii) the noise in the input
sequence to Electra discriminator is generated by a Masked Language model that
is much smaller than the Bert base model. Therefore the corrupted sequences
that the discriminator sees are not too complicated. As a result, the MLM
score of Bert is more powerful than the RTD score to find the probability of a
word given its context.

By comparing regression models with classification models under the same pre-
trained BERT model, we observed that regression models outperform
classification models in total accuracy and predicting neutral cases while
having high F1 scores for both positive and negative cases. The reason for
this might be that regression takes the ordinal nature of the labels into
account, and forced a balanced prediction at the same time. This total effect
increased the model’s performance in detecting neutral cases.

For the 5 ways to get a score using MLM models in section 3.1.2, we discovered
that the distance-based scores(model(11), (12), and (13)) perform
poorly(30.6%, 29.7%, 32.9% accuracy). And soft-max probability as a score
performs slightly better than logit-based score. The reason behind this is
unclear, but based on this empirical result, for other experiments, we decided
to use the soft-max score-based models for MLM models.

We also find that BERTbase+adapt is slightly outperforming BERTbase and close
to BERTlarge, especially in the accuracy and F1 score of different classes.
This implies that domain adaption works, but due to the similarity of WikiHow
and Wikipedia, the improvement is not huge. Similarly, pretraining using
version distinction task also doesn’t give high performance. The reason for
this could be that a majority of the Wiki-Revisions dataset is characterized
by the addition and deletion of tokens. Therefore the model does not see many
cases similar to our downstream task where a particular token in instruction
is replaced with tokens that can be lexically similar.

From Table 2, we find that concatenating titles, subheadings, and context
instructions will improve the performance of pretrained language models(BERT).
With context instructions having a larger impact compared to titles and
subheadings. This indicates BERT can understand the long sequence
dependencies, even when the constituent sub-sentences are from different parts
of a document.

Lastly, from Table 3, Ensemble methods that combine raw prediction scores can
further improve the classification performance on the best single model. To do
so it disregards some underrepresented labels(Neural). For regression-based
prediction models, the ensemble methods make balanced predictions on all
possible classes. A possible explanation based on our experiment is: Bert
captures the feature of the negative case well, while N-grams and Electra
capture the features of the positive case well. A classification model is
optimized towards higher prediction accuracy, thus considering the advantages
of all models, while the shared disadvantage(underrepresented neutral class)
remains. The regression model is designed to give balanced prediction results
that are similar to the training set. Thus the underrepresented class is given
more attention, at the cost of bringing down the prediction performance for
other classes in the best models. This might result in worse performance
compared to the best single model.

##  5 Conclusion

Our experiments are done in a manner that covers a broad range of models
rather than exploiting a single model’s performance with in-depth fine-tuning.
Given the limited amount of training data, we deem it more meaningful to look
at this task in the big picture and explore general directions to improve the
performance of models for this task. In comparison to the baseline provided by
the semeval task organizers, we improve accuracy by 10%. For future work on a
similar task, the biggest takeaways from our experiments are 1. Pre-trained
transformers as off-the-shelf models do not give high performance in
classification settings where the classes have ordinal nature and the
difference in characteristics of the adjacent classes is small. 2. Ensemble
predictions from multiple models can improve the performance for the well-
represented classes(binary: good or bad), but not the underrepresented
ones(neutral). 3. Given a small dataset(the size that’s reasonable for a
binary classification problem in language understanding), fine-tuning a pre-
trained model cannot capture the standard of the labeler used to classify a
neutral class.

## References

  * [Anthonio et al., 2020]  Talita Anthonio, Irshad Bhat, and Michael Roth.  2020\.  wikihowtoimprove: A resource and analyses on edits in instructional texts.  In Proceedings of The 12th Language Resources and Evaluation Conference, pages 5721–5729.
  * [Bhat et al., 2020]  Irshad Bhat, Talita Anthonio, and Michael Roth.  2020\.  Towards modeling revision requirements in wikihow instructions.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8407–8414.
  * [Chang et al., 2017]  Chia-Yang Chang, Shie-Jue Lee, and Chih-Chin Lai.  2017\.  Weighted word2vec based on the distance of words.  In 2017 International Conference on Machine Learning and Cybernetics (ICMLC), volume 2, pages 563–568. IEEE.
  * [Clark et al., 2020]  Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning.  2020\.  Electra: Pre-training text encoders as discriminators rather than generators.  arXiv preprint arXiv:2003.10555.
  * [Cui et al., 2020]  Leyang Cui, Sijie Cheng, Yu Wu, and Yue Zhang.  2020\.  Does bert solve commonsense task via commonsense knowledge?  arXiv preprint arXiv:2008.03945.
  * [Devlin et al., 2018]  Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  2018\.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805.
  * [Hendrycks and Gimpel, 2016]  Dan Hendrycks and Kevin Gimpel.  2016\.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  arXiv preprint arXiv:1610.02136.
  * [Lee et al., 2017]  Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin.  2017\.  Training confidence-calibrated classifiers for detecting out-of-distribution samples.  arXiv preprint arXiv:1711.09325.
  * [Peters et al., 2018]  Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer.  2018\.  Deep contextualized word representations.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana, June. Association for Computational Linguistics.
  * [Radford et al., 2018]  Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.  2018\.  Improving language understanding by generative pre-training.
  * [Yang et al., 2019]  Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le.  2019\.  Xlnet: Generalized autoregressive pretraining for language understanding.  Advances in neural information processing systems, 32.

Github111https://github.com/rchanumo/CS544_project (Be careful of the
underscore when copying) and Video
Presentation222https://www.youtube.com/watch?v=iWvDm0xILfw