11institutetext: Centre for Digital Music, Queen Mary University of London,
United Kingdom 11email<EMAIL_ADDRESS>{p.p.sarmento,
<EMAIL_ADDRESS>

# LooperGP: A Loopable Sequence Model for Live Coding Performance using
GuitarPro Tablature

Sara Adkins 11 Pedro Sarmento 11 0000-0002-4518-0194 Mathieu Barthet 11
0000-0002-9869-1668

###### Abstract

Despite their impressive offline results, deep learning models for symbolic
music generation are not widely used in live performances due to a deficit of
musically meaningful control parameters and a lack of structured musical form
in their outputs. To address these issues we introduce LooperGP, a method for
steering a Transformer-XL model towards generating loopable musical phrases of
a specified number of bars and time signature, enabling a tool for live coding
performances. We show that by training LooperGP on a dataset of 93,681 musical
loops extracted from the DadaGP dataset [22], we are able to steer its
generative output towards generating 3x as many loopable phrases as our
baseline. In a subjective listening test conducted by 31 participants,
LooperGP loops achieved positive median ratings in originality, musical
coherence and loop smoothness, demonstrating its potential as a performance
tool.

###### Keywords:

Controllable Music Generation Sequence Models Live Coding Transformers AI
Music Loops Guitar Tabs

##  1 Introduction

Sequence models such as the Pop Music Transformer [11] and Transformer-XL [8]
are able to generate symbolic polyphonic music that maintains a coherent
musical structure over several minutes of audio. However, these types of
models present several issues that prevent them from being viable in the
context of a live performance. They offer no control over important musical
parameters such as key signature, time signature, instrumentation, genre and
duration. While the outputs generated may be musically sound, they are
difficult to incorporate into a performance if these structural music
parameters are uncontrollable. Furthermore, the slow speed of inference makes
it very difficult to develop a usable interface for running sequence models in
traditional performance settings.

Live coding, an artform in which a performer writes code that synthesizes
music in real time, has the potential to be a performance environment in which
sequence models are viable. The live coding language can be thought of as a
score notation tool, where edits are evaluated in real time [14]. In a typical
performance, the performer will alter the score by generating a symbolic
pattern through algorithmic composition techniques [6] or hard-coding. The
pattern is then typically added to the audio mix and repeated as a loop [16].
Rather than traditional pattern generation techniques, a sequence model could
be used to generate pattern options for the performer to choose from.

In this paper, we address the issue of controllable generative music by
presenting a training and inference algorithm for steering a Transformer model
towards generating loopable phrases of a specified length, key and time
signature. We then evaluate the success of our loop model, coined as LooperGP,
by comparing the number of viable loops generated to a baseline model.
Finally, we present results from a listening test that compares participants’
ratings of creativity, likability, coherence and smoothness between human- and
machine-made loops in order to subjectively evaluate LooperGP and identify
areas for improvement. Our loop model and inference controls are intended to
enable the use of deep generative models in a live coding context by
addressing the constraints of ensuring model outputs fit within the context of
an ongoing musical performance. LooperGP evaluations in live performance will
be explored in future work.

##  2 Related Work

###  2.1 Sequence Models for Music Generation

A variety of deep learning architectures such as Recurrent Neural Networks,
Variational Autoencoders and Generative Adversarial Networks have been used to
generate symbolic music [5], [12], and Transformers have been particularly
successful at modeling temporal dependencies [8]. As demonstrated by the Music
Transformer in [11], the self-attention mechanism of the Transformer
architecture allows the model to refer back to previously generated motifs and
phrases, making it a promising architecture for symbolic music generation. The
Pop Music Transformer paper further shows that the way in which musical data
is stored can have a significant effect on output quality. For instance,
[11]’s results demonstrated that encoding musical knowledge such as metrical
position into the training data resulted in better beat and meter salience.

###  2.2 Controllable Music Generation

A large barrier stifling the adoption of symbolic music generation systems in
mainstream music creation is a lack of control. Often the only available
control is the temperature parameter [2], used to select a token from the
model’s probability distribution. One exception to this is the DeepBach system
[9], which allows the user to impose constraints on generation by fixing
specific notes in a Bach-style chorale. Given these note constraints, the bi-
directional long short-term memory (LSTM) model fills in the rest of the
chorale using Gibbs sampling. This level of customization would be useful in a
live coding context for generating phrases with a good loopable cadence, as
start and endpoint notes could be hard-coded. However, the DeepBach method is
constrained to a fixed receptive field and has only been tested on Bach
chorales, potentially limiting its use in performance of modern genres.

Vanilla Transformers often expand upon a motif they are primed with, but this
behavior is not guaranteed. The Theme Transformer in [23] introduces a new
model architecture that conditions the Transformer encoder on a specific
motif/theme. The training data is manipulated such that “theme-composition
pairs” are extracted from each training example by clustering similar measures
together. By training the model on these pairs, it learns how motifs are
expanded upon throughout a piece and is rewarded for repeating and varying
themes. This ability to expand upon existing material is useful in live
performance, as it ensures generated output is related to existing music
material.

###  2.3 Live Coding

[20] defines live coding as “the act of programming a computer under concert
conditions,” where we are specifically interested in programs that produce
musical output. While traditional music performance involves specifying a
stream of music on a note by note or chord by chord basis, live coding
involves performing at a higher level of abstraction through score-level
control over the creation of music [20]. A large focus of live coding practice
is on control over generative processes; a performer controls the parameters
of a generative algorithm that creates the musical output such that writing
software becomes an act of performing[6].

Prior work which focused on incorporating deep learning (DL) into live coding
performances includes Cibo [24], a text-based sequence-to-sequence model that
generates Tidal Cycles [16] code, and RaveForce [13], a pipeline for
translating synthesizer parameters derived from a DL model to a SuperCollider
server using Open Sound Control (OSC) [15].

###  2.4 DadaGP Dataset

A generative model used in a live coding context will ideally be capable of
outputting music from a variety of modern genres to match the styles typically
found at these types of performances. The DadaGP dataset [22] contains over
26,000 songs covering 700 genres and a dedicated encoder represent songs as
tokens compatible with sequence models. The dataset emphasizes rock and metal,
but also contains jazz, pop, classical and electronic dance music (EDM).
Furthermore, expressive performance information for string instruments such as
tremolo, palm muting and hammer-ons/pull-offs are included in the token set,
giving it an advantage over MIDI. The song diversity and token expressivity of
the DadaGP dataset make it an ideal training corpus candidate to base music
generation on for live coding performances.

[22] also presents a Transformer-XL model with 8 attention heads and 12 self-
attention heads trained on the DadaGP dataset. We use this model as a baseline
and starting point for our LooperGP model presented in the next section.

##  3 Methodology

The Theme Transformer in [23] demonstrates promising results in reorganizing
training data to steer the model towards certain types of outputs. The Pop
Music Transformer in [11] showed that including music information retrieval
(MIR) features such as beat position, bar position and chord labels in the
tokens improved higher level music features for the generated content like
beat salience. Combining these ideas, we extract “loopable phrases” from the
training data to create a new training corpus consisting only of segments that
loop naturally. Each loop will be stored as an ordered list of DadaGP tokens
[22], where the tokens describe the score and performance details for each
activated instrument in the loop. The goal is to further steer the model
towards generating loopable phrases during inference, so they can be
incorporated into a live coding performance.

###  3.1 Defining a Loop

Previous work on loop extraction has classified a loop as a short segment of
audio that transitions seamlessly when repeated [21], [7]. Seamless
“loopability” is ultimately a subjective consideration from listeners, but we
can focus on the structural attributes of the composition to identify
repeatable sections. For our purposes, we consider a segment of a song to be
loopable if it is bookended by a repeated phrase of a minimum length111We
focus here on loops where the exact same content is repeated, but it is worth
noting that a more general definition could encompass loops where certain
types of musical variations can occur across repetitions (e.g. modulation)..
An example loop from AC/DC’s “You Shook Me All Night Long” is shown in Figure
1.

Figure 1: An example loop from AC/DC’s “You Shook Me All Night Long.” Only the
guitar part is shown, though the loop is multi-track in the DadaGP dataset.

###  3.2 Extracting Loops

To identify all the loops in a given song, we must first identify all the
repeated phrases, or “bookends” as shown in Figure 1. To extract the
repetitions from a given song, we use the correlative matrix approach
presented in [10]. The DadaGP dataset [22] is provided in a text format
designed for the multi-track tablature editing software Guitar Pro. A Guitar
Pro formatted song is first converted into a list $M$ of active note sets as
shown in Equation 1. $N(t,d)$ represents the set of all active notes at tick
time $t$ with duration $d$, such that $t_{0}=0$ and $t_{n}=t_{n-1}+d_{n-1}$.

$M=[N(t_{0},d_{0}),N(t_{1},d_{0}),...,N(t_{n},d_{n})]$ (1)

Let $C_{s}$ and $D_{s}$ be $L\times L$ matrices for song $s$ with melody line
$M_{s}$, where $L$ is the length of $M_{s}$. $C_{s}$ is populated as a
correlation matrix for notes as shown in Equation 2, where $C_{s}(i,j)$
represents the number of notes in the repeated segment that ends at both
$M_{s}[i]$ and $M_{s}[j]$. Similarly, $D_{s}(i,j)$ is a correlation matrix
that stores the duration (in ticks) of a repeated segment that ends at both
$M_{s}[i]$ and $M_{s}[j]$.

$C_{s}[i,j]=C_{s}[i-1,j-1]+1\textrm{ if }M_{s}[i]=M_{s}[j]\textrm{ else }0$
(2)

The matrices $C_{s}$ and $D_{s}$ allow us to identify loop candidates and
filter repeated phrases by both duration and note length. The recursive nature
of the correlation matrix ensures that for any given cell $C_{s}[i,j]>0$; it
follows that $C_{s}[i-1,j-1]=C_{s}[i,j]-1$. Therefore, we can identify the
starting points of all repetitions of a minimum length $L_{min}$ by searching
for cells in the correlation matrix where $C_{s}[i,j]=L_{min}$. The two
starting indices of the repeated phrase $[s_{i}:s_{j}]$ are calculated in
Equation 3, and form a potential loop $p_{i,j}$ as defined in Equation 4.

$[s_{i},s_{j}]=(i-L_{min},j-L_{min})\textrm{ iff }C_{s}[i,j]=L_{min}$ (3)
$p_{i,j}=M[s_{i}:s_{j}]$ (4)

Given a list of all potential loops $p_{i,j}$ and their corresponding
endpoints $[s_{i},s_{j}]$, we can filter out loops that are trivial, too
lengthy, or have repeated bookends that are too short. The following length
parameters are configurable:

  * •

Minimum Repetition Notes ($L_{min}$): minimum number of notes in bookend

  * •

Minimum Repetition Beats ($RD_{min}$): minimum number of beats in bookend

  * •

Minimum Loop Bars ($LB_{min}$): minimum number of bars between bookends

  * •

Maximum Loop Bars ($LB_{max}$): maximum number of bars between bookends

We filter for $LB_{min}$ and $LB_{max}$ by comparing the timestamp difference
between $M[s_{i}]$ and $M[s_{j}]$, and converting this tick duration to number
of bars using the time signature metadata for the song. Next, we filter for
$RD_{min}$ by locating the end of each loop bookend. Given a loop with
endpoints $[s_{i},s_{j}]$, we can find the end of the repeated sub-phrase by
traversing along the diagonal of $C_{s}$ until we hit a decrease in cell value
as shown in Equation 5. This decrease indicates that the matching between
$s_{i}$ and $s_{j}$ has ended, and we have found the end of the bookend. We
can then use Equation 6 to lookup the duration of the bookend in ticks and
filter out loops with duration shorter than desired.

$n_{end}=n|(C_{s}[i+n,j+n]<C_{s}[i+n-1,j+n-1])$ (5)
$D_{ticks}=D_{s}[i+n_{end}-1,j+n_{end}-1]$ (6)

A final filtering strategy to improve extracted loops’ musical interest is to
filter by note density, in order to filter out loops made up primarily of
rests or long held notes. We defined note density ($\rho$) for a loop
$p_{i,j}$ as the average number of note onsets in a measure, scaled by the
number of instrumental tracks. This definition is formalized in Equation 7,
where $B$ is the number of bars in the loop and $T$ is the number of
instrument tracks. A $\rho$ value of 4.0 for instance would mean each track in
$p_{i,j}$ has on average four note onsets per measure.

$\rho_{p_{i,j}}=len(p_{i,j})\times\frac{T}{B}$ (7)

In addition to loops extracted using the correlative matrix algorithm, we can
also identify built-in loops by simply searching for repeat signs in the
training corpus and filtering by the duration in beats. The full loop dataset
is then made up of both extracted and built-in loops. Table 1 shows how
different combinations of loop extraction parameters affect the number of
loops identified. We chose the bolded parameter settings (total of 93,681
loops), as it was the median of all the parameter combinations explored.

Table 1: Loop Extraction Parameter Statistics. Bars | Density | L min | RD min | Loops | Avg. Loops Per Song
---|---|---|---|---|---
4-8 | 4 | 4 | 4 | 79,066 | 3.02
4-8 | 3 | 4 | 4 | 80,557 | 3.008
4-8 | 4 | 2 | 4 | 82,244 | 3.14
4-8 | 3 | 2 | 4 | 83,787 | 3.20
4-16 | 4 | 4 | 4 | 88,842 | 3.34
4-16 | 3 | 4 | 4 | 90,512 | 3.46
4-8 | 4 | 4 | 2 | 92,325 | 3.52
4-16 | 4 | 2 | 4 | 92,422 | 3.53
4-8 | 3 | 4 | 2 | 93,681 | 3.59
4-16 | 3 | 2 | 4 | 94,138 | 3.60
4-8 | 4 | 2 | 2 | 99,331 | 3.79
4-8 | 3 | 2 | 2 | 100,930 | 3.89
4-16 | 4 | 4 | 2 | 104,910 | 4.01
4-16 | 3 | 4 | 2 | 106,607 | 4.08
4-16 | 4 | 2 | 2 | 113,547 | 4.34
4-16 | 3 | 2 | 2 | 115,347 | 4.41
Table 2: Loop Dataset Statistics. Data Format | Files | Max Tokens Per Song | Avg. Tokens Per Song | Size
---|---|---|---|---
Hard | 66,637 | 12,253 | 1,376 | 9.13GB
Barred | 23,024 | 43,954 | 2,270 | 5.19GB
DadaGP | 26,158 | 51,997 | 4,456 | 9.56GB

###  3.3 Training

We experimented with two different loop training dataset formats. The first
format (“Barred Repeats”) surrounds each loop in a given song with
`repeat_open` and `repeat_close` tokens, then concatenates these barred loops
into a single file. The second format (“Hard Repeats”) stores each loop in a
separate file, where the loop is manually repeated 4 times. We hypothesized
that by training on the Barred Repeats dataset format, the model would learn
where to place repeat open/close tokens during generation to best form
musically coherent loops. The Hard Repeats model may better learn smooth
transitions due to the multiple repetitions of each loop. However, as shown in
Table 2, the Hard Repeats storage format is much less space efficient than
Barred Repeats, and therefore takes longer to run through an epoch during
training. We used the Pop Music Transformer-XL architecture presented in [22]
to train LooperGP. For each data format, we trained one model from scratch and
a second starting from epoch 200 of the DadaGP model. The pretrained DadaGP
model has already learned how to create structurally coherent music, but early
testing revealed that it tended to switch to different thematic sections at
random. By training this model further on a loop dataset, we hoped to teach
the model to bias its generation towards coherent loops that we can then
extract using the filtering procedure presented in Section 3.2. The results of
the different training configurations are covered in the results section. For
all four models, we used a learning rate of 0.0001, with 15% dropout and an
Adam optimizer as used by [22].

###  3.4 Controllable Inference

In a live coding performance, it is important that the performer has control
over the musical parameters of generation. In order to fit a generated loop
into an ongoing performance, the loop must match the key and time signature of
what is currently playing. It is also essential to control the length of the
loop in bars, to ensure the phrase lines up (or intentionally doesn’t line up)
with other patterns that are currently active in the set.

[22] observed that priming inference with one or more notes from the desired
instruments forming the root chord of the key is generally successful at
generating music in that key and instrumentation, at least over a short
duration. Figure 2 shows how priming with an eighth note of drums (unpitched),
guitar (A3) and bass (A2) results in the model continuing with the established
instrumentation in the key of A minor for several measures.

Figure 2: Example output generated from the DadaGP model, with primer shown in
the black box. Chord symbols were analyzed by hand.

We further explored how to control time signature and duration during
inference. The DadaGP Transformer model operates by generating a set number of
tokens [22], but these tokens do not have any relationship to duration. To
address this unpredictability, we modified the inference method to keep track
of cumulative duration by summing up the tick values of the `wait:xxx` tokens
and artificially inserting `new_measure` tokens after a specified amount of
ticks to enforce a measure duration. If the tick duration of a wait token
crosses a bar boundary, we shorten it to fit within the measure. We also keep
a cumulative count of the `new_measure` tokens, and artificially insert an
“end” token to replace the last `new_measure` token when the desired number of
bars is reached. A shortcoming of this approach is that it does not allow us
to differentiate between time signatures with the same measure duration, e.g.
between 3/4 and 6/8.

After LooperGP has generated a desired number of bars, the next step is to
extract loops using our filtering procedure. By setting the $LB_{min}$ and
$LB_{max}$ parameters to the same value, we can specify the exact length of
loops to output. As LooperGP was trained specifically on loop data, the
generated outputs should contain significantly more loops compared to the
baseline model. We compare the loop generation rate between different versions
of LooperGP and the baseline DadaGP model in the results section.

##  4 Results

###  4.1 Training Performance

The four model configurations described in Section 3.3 were each trained for
20 epochs (due to resource constraints) with an 85/15 training/validation
split on the loop dataset. Cross-entropy training and validation loss curves
are shown in Figure 3. As expected, the pretrained model configurations have a
much flatter loss curve than the scratch models, as they have already
converged on the original DadaGP dataset. Validation loss decreases over time
in all but the Pretrained Barred Repeats model.

Figure 3: Training loss (blue) and Validation loss (orange) across 20 epochs
for the 4 LooperGP model configurations

###  4.2 Loop Performance

To evaluate the quality of output generated by each of the model
configurations, we generated 25 excerpts from each model and calculated the
average number of loops extracted per excerpt and the average loop note
density as defined in Equation 7. Each model was configured to generate 16
bars of 4/4 music, primed with a random tempo and instrument configuration
extracted from the training data. We then ran the loop extraction algorithm
from Section 3.2 with the following parameters: $L_{min}=4$, $RD_{min}=2$,
$LB_{min}=4$ and $LB_{max}=4$, chosen as the median of Table 1

The results from the inference analysis are shown in Table 3. It is
immediately clear that the Hard Repeats model trained from scratch was not
able to generate meaningful loops; very few were extracted and the average
note density was extremely low. Manually viewing the generated results from
this model revealed most of the outputs were comprised of long held notes and
excessive rests, indicating it was not able to sufficiently learn musical
phrases. The Pretrained Hard Loops model did not have this issue with
excessive rests, however it failed to perform better than the baseline DadaGP
model, generating only 1 more valid loop across the total 25 excerpts.

Table 3: Loop Extraction Statistics Between Models. Model | Loops Found | Avg. Loops | Avg. Note Density
---|---|---|---
DadaGP Baseline | 20 | 0.80 | 6.03
Pretrained Hard | 21 | 0.84 | 6.84
Scratch Hard | 3 | 0.12 | 0.25
Pretrained Barred | 88 | 3.52 | 6.96
Scratch Barred | 77 | 3.08 | 9.80

The Barred Repeats models both performed well above the baseline in terms of
number of loops generated. In both cases three or more 4-bar loops were
extracted from each 16 bars of generation, with high average note density.
While these results are promising, they do not give us any indication as to
the musical quality of the extracted loops. To prepare stimuli for a
subjective listening test, we continued to train the Pretrained Barred Repeats
model, stopping at epoch 40 with a Cross-Entropy loss of 0.148.

###  4.3 Subjective Evaluation

To evaluate the subjective quality of loops generated by LooperGP, we
conducted an online listening study which had participants evaluate machine-
generated loops and human-composed loops extracted from the training data.
Participants listened to 50 excerpts, each containing a 4-bar loop that
repeated four times. 25 of the excerpts were human-composed, and the other 25
were machine-generated222Link to listening test excerpts:
https://drive.google.com/drive/folders/1I0MCPYjj8nXqKkmDN-
d-C2ETOHJpCZyn?usp=share_link. Excerpts were presented to participants in a
random order, without revealing the generation method. Participants were asked
the following list of questions about each excerpt, and also filled out the
Goldsmiths Music Sophistication Index (MSI) survey [19]:

  1. 1.

Have you heard the music in this excerpt before? (Y/N)

  2. 2.

Do you think the music in this loop has been produced by a human or by a
machine? (Human/Machine)

  3. 3.

Please rate to what extent you agree or disagree with the following
statements: (7-point Likert scale)

     * Q1: The music in this loop is original and musically creative.

     * Q2: I like the music in this loop.

     * Q3: Within the loop, the music is coherent rhythmically, melodically and harmonically.

     * Q4: The transition between loop repetitions is smooth rhythmically, melodically and harmonically.

  4. 4.

Briefly explain the reasoning behind your ratings. You may list keywords
rather than using full sentences.

####  4.3.1 Listening Test Excerpts

The 25 human-made loops were randomly sampled from the set of all 4-bar loops
extracted from the DadaGP training data. The other 25 excerpts were machine-
generated by the LooperGP model. To generate an excerpt, a primer tempo,
instrumentation and 1st note were randomly sampled from the 4-bar loop DadaGP
dataset, then inference was run for 16 bars of 4/4. Next, all 4-bar loops were
extracted from the generated output and a random one was chosen as the final
excerpt. This procedure was repeated 25 times. All excerpts were rendered as
MP3s (variable 260 kbps) using the default virtual instruments in GuitarPro 7.
Participants were instructed to focus on the quality of the music composition
and not on the quality of the virtual instruments or the music production mix
when answering questions.

####  4.3.2 Participants

We recruited 31 participants in total, six of whom responded to a department-
wide participant invitation email, and 25 from the data collection website
Prolific. The inclusion criteria were access to headphones, normal hearing, no
amusia (tone deafness), and having music as a hobby. Participants were roughly
evenly split between genders, with 17 male and 14 female participants.

####  4.3.3 Ordinal Data Transformation

The Likert scale data collected from the four statements in Question 3 is
ordinal data. Opinions are mixed as to whether or not Likert data can be
treated as continuous data in statistical tests ([4] and [25]). The mean value
is not necessarily a useful metric, as participants may not assume a constant
distance between Likert items [26]. To work around this issue, we implemented
the Snell scaling procedure proposed in [26], which uses Maximum Likelihood
Estimation to place the ordinal Likert categories as points on a continuous
scale so that mean-based parametric methods may be applied. Applying their
algorithm to our listening test data, we mapped the seven Likert values
(Strongly Disagree, Disagree, Slightly Disagree, Neutral, Slightly Agree,
Agree, Strongly Agree) to the values shown in Figure 4.

####  4.3.4 Rating Results and Mixed Effects Models

We used a linear mixed-effects model (LMM) [17] to test the effect of the
generation type on the results from the survey questions. A LMM was chosen as
it is suitable for repeated measures and individual differences can be modeled
by using different random intercepts for each participant. Participants’ MSI
scores [19] were included as an additional independent variable to evaluate
the relationship between music background and ratings. Finally, we used
participant ID (PID) as a random effect to account for variation in rating
tendencies between individual participants. The mixed effects model was
implemented using the random intercept model from the StatsModel 0.13.5 Python
library [1]. In all four questions, generation type had a significant effect
on the rating results, with $p<.001$. The MSI score did not have a significant
effect on any of the dependent variables, with all $p\geq 0.22$. Table 4 shows
the breakdown of regression results by question.

Figure 4: 7-point Likert scale values converted to a continuous scale (Snell method [26]). Table 4: LMM Results for Fixed Effect (Coefficients, Standard Errors, and p-value) and Random Effect (PID Variance and Standard error) | Coef. | Std. Err. | $P>|z|$
---|---|---|---
| Q1 | Q2 | Q3 | Q4 | Q1 | Q2 | Q3 | Q4 | Q1 | Q2 | Q3 | Q4
Intercept | 3.388 | 3.463 | 4.019 | 3.966 | 0.584 | 0.581 | 0.495 | 0.643 | 0.001 | 0.001 | 0.001 | 0.001
Gen. Type | -0.072 | -0.897 | -0.621 | -0.565 | 0.072 | 0.082 | 0.070 | 0.069 | 0.001 | 0.001 | 0.001 | 0.001
MSI | 0.009 | 0.005 | 0.006 | 0.008 | 0.007 | 0.007 | 0.006 | 0.008 | 0.220 | 0.446 | 0.315 | 0.310
PID Var. | 0.420 | 0.403 | 0.292 | 0.522 | 0.086 | 0.075 | 0.063 | 0.110 | - | - | - | -

To visualize the rating difference between generation type, we plotted the
Likert distributions shown in Figure 5 using the original Likert data. With
the exception of Q2:Machine (“I like the music in this loop”), the median of
all response categories are positive. For Likert questions 2-4, the human-
composed excerpts are rated approximately one category higher than the
machine-generated excerpts. The median ratings for Q1 (“The music in this loop
is original and musically creative”) were equivalent across generation type.
Mean, variance and standard deviation for the Snell-scaled [26] ratings are
shown in 5.

####  4.3.5 Musical Turing Test Results

We evaluated participants ability to discern between human- and machine-made
loops using a logistic regression model. As with the Likert rating questions,
both generation type and MSI score were used as independent variables in the
model. To ensure participants’ answers would not be biased towards human-made
loops due to familiarity, we discarded 83 (out of 1550) datapoints where the
participants indicated they recognized the music in the loop. We used the
logistic regression implementation from the StatsModel Python library [1] to
calculate the model results. Generation type was found to have a significant
effect on perceived generation source (Human or Machine), with $p<0.01$. MSI
score did not have a significant effect on perceived generation source, with
$p\geq 0.28$. To further discern how well participants were able to
differentiate between human- and machine-made loops, we plotted the confusion
matrix in Table 6. Though generation type was found to have a significant
effect on perceived generation source, the hit rate is only 10% above chance.
Participants correctly predicted generation type about 60% of the time, and
the distribution is similar for both human and machine sources.

Figure 5: Comparing 7-point Likert rating distributions between human and machine generated loops for each evaluation question. Table 5: Likert Rating Statistics. H: Human; M: Machine. | Median | Mean | Var | Std Dev
---|---|---|---|---
| H | M | H | M | H | M | H | M
Q1 | Slightly Agree | Slightly Agree | 4.09 | 3.38 | 2.24 | 2.62 | 1.50 | 1.62
Q2 | Slightly Agree | Neutral | 3.89 | 3.00 | 2.93 | 3.04 | 1.71 | 1.74
Q3 | Agree | Slightly Agree | 4.50 | 3.88 | 1.83 | 2.55 | 1.35 | 1.60
Q4 | Agree | Slightly Agree | 4.60 | 4.03 | 1.94 | 2.73 | 1.39 | 1.65
Table 6: Confusion Matrix of Ground Truth and Predicted Generation Type. | Human | Machine
---|---|---
Human | 0.6 | 0.4
Machine | 0.42 | 0.58

####  4.3.6 Free Response Results

Participants were given a text box to explain their ratings for each excerpt
in the listening test. We visualized the top 40 most frequently occurring
words in answers for each generation type using a word cloud generated by the
WordCloud Python library [18]. The following words were used as stop words and
removed from the cloud as they occurred frequently but were not informative:
loop, melody, sounds and feels. As shown in Figure 6, “simple”, “boring” and
“repetitive” are used more often to describe machine excerpts than human ones.
On the other hand, “interesting”, “smooth” and “coherent” are roughly equal
size in both types of excepts. “Good” is a top word in both categories, but is
more often used to describe human excerpts.

Figure 6: Frequency-based word clouds for machine-generated (left) and human-
generated (right) excerpts from listening test free response.

##  5 Discussion

Our approach to restructuring the DadaGP dataset was overall successful in
encouraging loopable outputs. However, there was a surprising difference in
output quality between the Hard Repeats and Barred Repeats dataset formats.
The Pretrained Hard Repeats model produced the same amount of loops on average
as the baseline model, indicating the model was unable to learn what features
constitute a loopable phrase. The Hard Repeats model performed even worse when
trained from scratch, falling into a local minimum in which the best it could
do was generate rests and the occasional long note. Seemingly, the model
learned to focus on the easier low density tracks while ignoring the pertinent
features in the high density tracks of a song.

The Barred Repeats model was much more successful at generating loops during
inference, with both the pretrained and scratch models generating, on average,
3x as many loops as the baseline. Despite high performance on this loop
metric, the validation loss of the pretrained Barred Repeats model strictly
increased over time, indicating that it was over fitting the training data.
Evidently the model learned to place repeat bars, but it did not learn to
place them in meaningful places. Unlike the pretrained version, the scratch
Barred Repeats model did converge and generated loops with a higher average
note density, indicating that it was successful at learning musically
meaningful loops.

Our subjective listening test allowed us to evaluate the quality of generated
loops beyond the objective note density metric. Our statistical evaluation
showed a significant effect of generation type on all four Likert rating
questions, indicating that the generated outputs are distinguishable from
human-composed ones. However, it is promising that median ratings were
positive across all questions and suggests our method has potential as a
performance tool.

Breaking down the Likert ratings by question, it is peculiar that Q1, which
asked for ratings of originality and musicality, had the most similar ratings
across generation types. Musicality and originality are particularly human
characteristics that AI models traditionally have difficulty emulating, so
this result is surprising. It is possible that unexpected musical outputs were
interpreted as “original” by participants, inflating ratings for machine-made
excerpts. Q2, which asks participants if they like an excerpt, had the lowest
median rating for both machine and human excerpts. Though we asked
participants to focus on the musical score rather than the quality of the
digital instruments, for preference questions such as this one it may have
been difficult for participants not to be influenced by the synthetic timbres
and lack of human expression in the renditions of both the human and machine
excerpts, biasing the scores negatively. Q3 and Q4, which asked about musical
coherence and loop-point smoothness respectively, had very similar rating
distributions as shown in Figure 5, hinting that perhaps participants had
trouble differentiating between the meaning of musical coherence and loop-
point smoothness. This may be in part caused by the presence of sub-loops
within some of the excerpts. For instance, with our extraction algorithm, it
is possible a 4-bar loop could actually be made up of a repeated 2-bar loop.
This may have made it difficult for participants to focus on the exact loop
point when answering Q4 since the amount of repetitions varied.

The musical Turing test data showed a significant effect of generation type on
perceived generation srouce, but results were still close to chance for both
types. While the usefulness of Turing tests in evaluating the musicality of
algorithmic composition systems is disputed [3], in our case it is a useful
metric of how well LooperGP is able to mimic the loopability of the training
data. In other words, it offers insight into whether LooperGP has learned to
place repeat bars in convincing places. Given our results, LooperGP is able to
somewhat convincingly emulate human composers, but there is room for
improvement.

The sample size for the listening test remains fairly small (31 participants
who each assessed 25 items per condition, so a total of about 775 observations
per condition). However, for fixed effects, power in LMMs does not necessarily
increase as the total sample of observations increases [17].

##  6 Conclusion and Future Work

As the motivation behind LooperGP is for it be used in live coding, we shall
discuss its potential in this context. Reference [6] argues that for a
generative process to be useful in a live coding environment, it must be
succinct to call, applicable to a variety of musical circumstances, modifiable
and require limited temporal scope. Given that LooperGP focuses on loopability
of output and can be primed with a specific instrumentation, key/time
signature and bar length, it can be configured to fit a wide variety of
musical circumstances. Our algorithm requires no temporal scope, as the model
does not have any temporal dependencies. A downside of LooperGP, and many
other DL methods for music generation, is a lack of modification control. Once
a loop has been generated, there is no way to algorithmically adjust it.
However, it could be feasible to generate variations on a generated loop by
passing it back through the system as a primer, as observed in [23]. Finally,
an inference of LooperGP can be easily generated via a succinct Python
function call, as the only required parameters are key, duration and bar
length.

Overall, LooperGP was able to increase the frequency of high-density loopable
outputs compared to the baseline. Subjective ratings showed a positive
response to the loops overall, but also identified some key areas for
improvement in reducing simplicity and repetitiveness. LooperGP exhibits many
of the characteristics required for a generative process to be useful in live
coding, with a specific focus on generating controllable output able to fit
most musical circumstances. The duration control and loopable generation of
LooperGP make it a viable option for use in a live performance context,
although this should be the object of further assessment with live coders.

To address the listening test comments that machine-made loops were
“repetitive” and “boring”, future work could focus on increasing the average
complexity of extracted loops by adjusting the loop extraction algorithm to
identify and filter out sub-loops. On the training side, more of the
hyperparameter space could be explored, such as increasing dropout or
decreasing layer size, to prevent the early overfitting demonstrated by some
of the training setups.

Finally, an exciting area for future work is integrating the LooperGP system
into a live coding framework such as Tidal Cycles [16] or SuperCollider [15].
The OSC pipeline for communicating between Python and SuperCollider presented
in [13] could be used to prototype this concept without requiring the model to
be exported to a new environment. It is also important that LooperGP be
wrapped into a usable interface with a succinct API. Ideally, the performer
should be able to cherry-pick a loop from a set of potential options generated
by the model and automatically transpose it to their desired key signature.
Once LooperGP has been wrapped into a performer-friendly interface, its
potential as a live coding feature can be truly evaluated by incorporating it
into a concert and collecting feedback from audience members and performers.

##  7 Acknowledgements

This work has been partly supported by the EPSRC UKRI Centre for Doctoral
Training in Artificial Intelligence and Music (Grant no. EP/S022694/1).

## References

  * [1] Statsmodels. https://github.com/statsmodels/statsmodels (2022), accessed: 2022-08-14
  * [2] Ackley, D.H., Hinton, G.E., Sejnowski, T.J.: A learning algorithm for boltzmann machines. Cognitive Science pp. 147–169 (1985)
  * [3] Ariza, C.: The interrogator as critic: The turing test and the evaluation of generative music. Computer Music Journal 33, 48–70 (2009)
  * [4] Bishop, P.A., Herron, R.L.: Use and misuse of the likert item responses and other ordinal measures. Int. Journal of Exercise Science 8, 297–302 (2015)
  * [5] Briot, J.P., Hadjeres, G., Pachet, F.D.: Deep learning techniques for music generation, vol. 1. Springer (2020)
  * [6] Brown, A.R., Sorensen, A.: Interacting with generative music through live coding. Contemporary Music Review 28, 17–29 (2009)
  * [7] Chandna, P., Ramires, A., Serra, X., Gómez, E.: Loopnet: Musical loop synthesis conditioned on intuitive musical parameters. In: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 3395–3399. IEEE (2021)
  * [8] Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q.V., Salakhutdinov, R.: Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860 (2019)
  * [9] Hadjeres, G., Pachet, F., Nielsen, F.: Deepbach: a steerable model for bach chorales generation. International Conference on Machine Learning pp. 1362–1371 (2017)
  * [10] Hsu, J.L., Liu, C.C., Chen, A.L.: Discovering nontrivial repeating patterns in music data. IEEE Trans. on Multimedia 3, 311–325 (9 2001)
  * [11] Huang, Y.S., Yang, Y.H.: Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions. In: Proceedings of the 28th ACM International Conference on Multimedia. pp. 1180–1188 (2020)
  * [12] Ji, S., Luo, J., Yang, X.: A comprehensive survey on deep music generation: Multi-level representations, algorithms, evaluations, and future directions. arXiv preprint arXiv:2011.06801 (2020)
  * [13] Lan, Q., Tørresen, J., Jensenius, A.R.: Raveforce: A deep reinforcement learning environment for music. In: Proceedings of the smc conferences. pp. 217–222. Society for Sound and Music Computing (2019)
  * [14] Magnusson, T.: Sonic writing: technologies of material, symbolic, and signal inscriptions. Bloomsbury Publishing USA (2019)
  * [15] McCartney, J.: Supercollider: A new real-time sound synthesis language. Proceedings of the Int. Computer Music Conference pp. 257–258 (1996)
  * [16] McLean, A., Wiggins, G.: Tidal–pattern language for the live coding of music. In: Proceedings of the 7th sound and music computing conference. pp. 331–334 (2010)
  * [17] Meteyard, L., Davies, R.A.: Best practice guidance for linear mixed-effects models in psychological science. Journal of Memory and Language 112, 104092 (2020)
  * [18] Mueller, A.: Word cloud. https://github.com/amueller/word_cloud (2022), accessed: 2022-08-14
  * [19] Müllensiefen, D., Gingras, B., Musil, J., Stewart, L.: The musicality of non-musicians: An index for assessing musical sophistication in the general population. PloS one 9(2), e89642 (2014)
  * [20] Nilson, C.: Live coding practice. In: Proceedings of the 7th international conference on New interfaces for musical expression. pp. 112–117 (2007)
  * [21] Ramires, A., Font, F., Bogdanov, D., Smith, J.B., Yang, Y.H., Ching, J., Chen, B.Y., Wu, Y.K., Wei-Han, H., Serra, X.: The freesound loop dataset and annotation tool. arXiv preprint arXiv:2008.11507 (2020)
  * [22] Sarmento, P., Kumar, A., Carr, C., Zukowski, Z., Barthet, M., Yang, Y.H.: DadaGP: a Dataset of Tokenized GuitarPro Songs for Sequence Models. In: Proc. of the 22nd Int. Soc. for Music Information Retrieval Conf. (2021)
  * [23] Shih, Y.J., Wu, S.L., Zalkow, F., Muller, M., Yang, Y.H.: Theme transformer: Symbolic music generation with theme-conditioned transformer. IEEE Trans. on Multimedia pp. 1–1 (3 2022)
  * [24] Stewart, J., Lawson, S.: Cibo: An autonomous tidalcyles performer. In: Proceedings of the Fourth International Conference on Live Coding. p. 353 (2019)
  * [25] Sullivan, G.M., Artino, A.R.: Analyzing and interpreting data from likert-type scales. Journal of Graduate Medical Education 5, 541–542 (12 2013)
  * [26] Wu, C.H.: An empirical study on the transformation of likert-scale data to numerical scores. Applied Mathematical Sciences 1, 2851–2862 (2007)