# Toward Expressive Singing Voice Correction: On Perceptual Validity of
Evaluation Metrics for Vocal Melody Extraction

###### Abstract

Singing voice correction (SVC) is an appealing application for amateur
singers. Commercial products automate SVC by snapping pitch contours to equal-
tempered scales, which could lead to deadpan modifications. Together with the
neglect of rhythmic errors, extensive manual corrections are still necessary.
In this paper, we present a streamlined system to automate expressive SVC for
both pitch and rhythmic errors. Particularly, we extend a previous work by
integrating advanced techniques for singing voice separation (SVS) and vocal
melody extraction. SVC is achieved by temporally aligning the source-target
pair, followed by replacing pitch and rhythm of the source with those of the
target. We evaluate the framework by a comparative study for melody extraction
which involves both subjective and objective evaluations, whereby we
investigate perceptual validity of the standard metrics through the lens of
SVC. The results suggest that the high pitch accuracy obtained by the metrics
does not signify good perceptual scores.

Index Terms—  Singing voice correction, audio synchronization, vocal melody
extraction, evaluation

##  1 Introduction

Automatic singing voice correction (SVC) has found many applications in the
era of digital music, ranging from mobile apps, karaoke devices, to music
production tools [1, 2, 3]. Commercial products mostly rely on an equal-
tempered musical scale that is defined by a reference song, whereby out-of-
tune pitches are modified to match the nearest notes on the scale [4, 3]. The
naive pitch snapping, however, could lead to modifications lacking lively
expressions. Additionally, pitch errors that deviate much could not be
rectified. Meanwhile, most of the existing solutions limit their scope to
pitch correction and ignore rhythmic errors. All the mentioned problems
necessitate extensive manual processes after the modification.

In this paper, we discuss a streamlined framework which automates expressive
SVC for both pitch and rhythmic errors under real-world scenarios.
Particularly, the framework is built upon a previous work based on audio
synchronization [5]. This approach regards SVC as a task of dynamic time
warping (DTW) followed by vocoding: acoustic features of the input audio
(source) and the reference (target) are temporally aligned, whereby the
corrected singing voice is synthesized with the pitch contour modified based
on the synchronization. To deal with general sources and targets which might
contain musical accompaniments, in this work advanced algorithms for singing
voice separation (SVS) and vocal melody extraction are incorporated into the
system, which takes a step forward to achieve the real-world application.

In comparison to the previous works [4, 3, 6], the presented method leverages
human performances as the target, thereby directly rendering the expressive
pitch contours for the correction and avoiding deadpan modifications. A data-
driven model for SVC is proposed recently [7], which predicts pitch deviations
caused by data augmentation, thereby waiving the need of a reference. However,
the model is incapable of fixing large pitch errors as the pitch augmentation
used for training is bounded to $\pm{100}$ cents. Our method, albeit requiring
the target, alleviates the demand by the SVS that allows for the leverage of
professional renditions which are easily accessible from online streaming
services, and is able to rectify pitch errors up to an octave. Moreover, the
derived temporal synchronization allows for correcting rhythmic errors in
addition to pitch. A similar method focuses on transferring singing style [8]
instead of SVC. On the other hand, we can also achieve style transfer in terms
of pitch as a by-product.

Fig. 1: The framework for singing voice correction.

We expect that in the proposed framework, the performance of SVC would benefit
from advanced melody extraction techniques. Therefore, we also conduct a
comparative study for state-of-the-art (SoTA) melody extractors. We then
investigate perceptual validity of the standard metrics for melody extraction
through the lens of the application of SVC. The contributions are summarized
as follows:

  * •

We present a streamlined framework which automates expressive SVC. It enables
both pitch and rhythmic correction, and allows for polyphonic sources and
targets, with the aids of SVS and vocal melody extractors. Audio samples can
be accessed from https://yjlolo.github.io/expressive-svc/.

  * •

In conducting the perceptual evaluation, we try to answer, in what aspect the
advance of the vocal melody extraction benefits the task of SVC. Specifically,
we perform correlation analysis for the standard metrics and the subjective
judgements, whereby we highlight the strengths and limitations of the existed
annotations, metrics, and melody extractors.

The remainder of the paper is structured as follows. We first present the
framework in Section 2, and then describe the experimental setups along with
the results in Section 3. We have an in-depth discussion in Section 4, and
conclude the study in Section 5.

##  2 Methods

As shown in Figure 1, the proposed method consists of SVS, acoustic feature
extraction, temporal alignment, feature stretching, and audio re-synthesis.
The main architectural differences between the proposed and the previous work
[5] are the integration of SVS and the comparative study of SoTA vocal melody
extractors.

The inputs to the system are a source $s$ and a target $t$. In our
experiments, $s$ contains only singing voice of the source, while $t$ has
singing voice of the target overlapped with background music. The goal is to
correct $s$ in terms of pitch and rhythmic by aligning and rendering
parameters from $t$, while keeping vocal timbre of $s$.

Note that although monophonic $s$ is not a requirement for our method, we
believe that using monophonic sources and polyphonic targets matches well with
the real-world use cases for SVC.

###  2.1 Singing Voice Separation

We integrate SVS to extract the target singing from the professional
renditions $t$ that are easily accessible through online streaming services.
This facilitates the later stages of feature extraction and temporal alignment
for singing voice. We consider SoTA models, namely Open-Unmix [9] and Demucs
[10] for the task. In the rest of the article, we denote $t$ as the extracted
target singing voice.

###  2.2 Acoustic Feature Extraction

Given an input audio resampled to 16,000Hz, we employ WORLD vocoder [11] to
extract parameters using short-time Fourier transform with a window size of
64ms and a hop size of 5ms, including pitch contour F0, spectral envelope SP,
and aperiodicity AP. Frame-wise 24-ordered Mel-cepstral coefficients (MCEPs)
are then derived from SP which serve as the feature for aligning $s$ and $t$.

Because the framework directly replaces $\text{F0}_{s}$ with $\text{F0}_{t}$
after the alignment to achieve SVC, we conduct a comparative study for SoTA
vocal melody extractors in order to study the relation between accuracy of the
extracted F0 and the consequent performance of SVC. Particularly, we include
CREPE [12], PATCH [13], and SEG [14] in our study.

###  2.3 Singing Voice Alignment

Following the previous work [5], we use canonical time warping (CTW) [15] to
temporally align $s$ and $t$. CTW iteratively projects the input sequences to
a common subspace by canonical correlation analysis, and performs DTW therein.
This can be seen as endowing DTW with a feature selection mechanism.

###  2.4 Feature Stretching and Audio Re-synthesis

As illustrated in Figure 1, the modified source performance $s^{\prime}$ is
re-synthesized by WORLD, which consumes the speech parameters $\text{F0}_{t}$,
$\text{SP}^{\prime}_{s}$, and $\text{AP}^{\prime}_{s}$; where
$\text{SP}^{\prime}_{s}$ and $\text{AP}^{\prime}_{s}$ are obtained by temporal
stretching of $\text{SP}_{s}$ and $\text{AP}_{s}$ given the optimal path of
alignment $W$ derived from CTW.

In particular, let $f$ denote collectively SP and AP. Given temporal
associations that align $f_{s}$ and $f_{t}$, we either locally average the
values of $f_{s}$ or interpolate them with constants. This makes the sequence
length of the modified feature $f^{\prime}_{s}$ equal to that of $f_{t}$,
thereby correcting the rhythmic errors. Note that we do not directly replace
$f_{s}$ with $f_{t}$ as we do for $\text{F0}_{s}$, which is to preserve the
vocal timbre of $s$ represented by $\text{SP}_{s}$. More advanced
manipulations for the feature stretching are left for future investigations.

Fig. 2: The quality scores measured in JODs for the six criteria. Different
numbers of asterisks denote increasing levels of statistical difference, i.e.,
$p<0.05$, $p<0.01$, and $p<0.001$, respectively. Left: methods based on Open-
Unmix for SVS. Right: Demucs for SVS. Fig. 3: Pearson correlation between the
standard metrics and the criteria. Correlation coefficients are shown in the
legend. Different numbers of asterisks that precede the criteria denote
increasing levels of statistical difference, i.e., $p<0.05$, $p<0.01$, and
$p<0.001$, respectively.

##  3 Experiments

###  3.1 Subjective Evaluation

We use six amateur recordings of 30-second singing voice from MIR-1K [16] as
the sources $s$. The dataset features 1000 clips of mandarin pop songs, with
singing voice separated from background music. For the targets $t$, we
leverage the six songs performed by the original singers, respectively,
accessed from YouTube.111https://www.youtube.com

We conduct an experiment of pairwise comparison with online questionnaires.
From the SVS and melody extraction algorithms mentioned in Section 2.1 and
2.2, we form eight methods from the two SVS and the four vocal melody
extractors. By dividing the eight methods into two groups based on SVS, we
have 12 pairs for each of the six songs ($2\text{ groups of SVS}\times 6\text{
pairs formed by the four melody extractors}$), which results in 72 pairs in
total ($12\text{ pairs}\times 6\text{ source songs}$). We do not exhaust all
possible pairs by considering inter-group (SVS) combinations, which is in
order to reduce the number of pairs involved in the experiment, thereby
shortening the survey and avoiding fatigue. The 72 pairs of comparison are
distributed uniformly in 12 questionnaires, each has six pairs to evaluate
corresponding to the six sources.

A subject is asked to first listen to a source, and is then presented a pair
of corrected sources by different methods. The task is to select one that is
better than the other, in terms of six criteria usually seen in singing
contests [17]: pitch accuracy, rhythm accuracy, fluency, naturalness, vocal
timbre similarity between the modified and the original source, and overall
audio quality. The subject is tasked to repeat the process for the six songs
in a session. We employ a scheme of two-alternative-forced-choice, which is
because our pilot study suggests that not providing “no preference” encourages
the subjects to scrutinize the samples before making decisions.

We gather responses from 115 subjects and scale the pairwise comparison data
to obtain quantitative scores. The scaling approach [18] defines the quality
score of a tested method as a one-dimensional Gaussian distribution, with the
mean representing the overall quality and the standard deviation addressing
variance of subjective judgement. It employs a maximum-likelihood estimator
such that the scaled quality scores best explain the collected data whose
likelihood function is modeled as a binomial distribution:

$L(\hat{q}_{i}-\hat{q}_{j}|c_{ij},n_{ij})=\binom{n_{ij}}{c_{ij}}P(r_{i}>r_{j})^{c_{ij}}(1-P(r_{i}>r_{j}))^{n_{ij}-c_{ij}},$
(1)

where $\hat{q}_{i}$ and $\hat{q}_{j}$ denote the estimated quality scores of
the tested methods $i$ and $j$, respectively; $r_{i}$ and $r_{j}$ refer to the
random variables sampled from the underlying Gaussian distributions. $c_{ij}$
denotes the number of responses that $i$ is selected as being better than $j$,
and $n_{ij}$ is the total number of trials that $i$ and $j$ are compared. We
refer readers to the literature [18] for more details.

The quality scores estimated this way are reported to have a strong linear
relation with mean opinion score (MOS) obtained by direct rating [19].
Conducting the pairwise comparison is nevertheless easier for non-expert
raters, more sensitive to subtle but noticeable quality differences, and
invariant to inter-rater scale difference [20]. Confidence intervals for the
scaled quality scores can be computed using bootstrapping techniques, which
gives the $95\%$ confidence intervals for the estimated quality scores.

Method | OA | RPA | RCA | VR | VFA
---|---|---|---|---|---
Open- Unmix | WORLD | 61.29 | 71.93 | 73.43 | 96.89 | 74.46
SEG | 76.61 | 82.09 | 83.72 | 77.59 | 0.99
PATCH | 71.22 | 79.99 | 82.82 | 96.09 | 56.79
CREPE | 69.38 | 78.45 | 80.35 | 84.53 | 41.92
Table 1: Evaluation of the standard metrics for melody extraction.

###  3.2 Objective Evaluation

In order to investigate the interplay between the extracted pitch contours and
the perceptual study, we also report standard metrics for melody extraction
implemented by mir_eval [21], including raw pitch accuracy (RPA), raw chroma
accuracy (RCA), and overall accuracy (OA) for pitch; and voice recall (VR) and
voice false alarm (VFA) for voicing [22].

In particular, the eight methods that we use for the subjective experiment are
evaluated with the 1000 clips of music mixtures from MIR-1K. For the methods
based on deep learning, we use pre-trained models provided from the
literature. The averaged scores over individual clips are reported for all the
metrics.

Fig. 4: Illustration of the pitch contours and the alignment. Left: WORLD as
the vocal melody extractor. Center: SEG as the vocal melody extractor. Right:
the red rectangle highlights the alignment from a plain singing (source) to an
expressive one (target). Fig. 5: Pitch contours extracted by WORLD and SEG.

###  3.3 Results

Figure 2 shows results of the subjective evaluation, where the left and the
right figures correspond to SVS being Open-Unmix and Demucs, respectively. The
scaled values of the pairwise comparison data are measured in Just-
Objectionable-Distances (JODs), which quantify the relative quality difference
among the evaluated methods [18]. Note that the scaling is performed
separately for the two groups of SVS, and individually for each of the
criteria. As a result, we can only meaningfully compare within-criteria JODs,
and the common x- and y-axes across the criteria are solely for visualization.
Also, the minimum JOD from each of the criteria is set to one, which does not
affect interpretability because we focus on the relative difference.

We summarize major findings as that WORLD outperforms SEG significantly across
all criteria, and that WORLD is the best-performing method in terms of fluency
and quality. We will extend the discussion in Section 4. Due to the space
constraint, we carry on our analysis with Open-Unmix for SVS, as our pilot
subjective study with a professional singer indicates superiority of the
methods based on Open-Unmix over that based on Demucs.

As shown in Table 1, methods based on SEG outperform other melody extractors
in terms of OA, RPA, RCA, and VFA; while WORLD prevails only in terms of VR.
Interestingly, the results are at odds with our findings in the subjective
evaluation where WOLRD clearly outperforms SEG. We measure Pearson correlation
to gain more insights, and report in Figure 3 the correlation coefficients
calculated from the JODs and the objective scores. In general, the correlation
coefficients are positive for VR and VFA, and are negative for OA, RPA and
RCA. Surprisingly, both of the criteria pitch and rhythm negatively correlate
with OA and RPA.

Before continuing our discussion on the relation between the subjective and
the objective evaluation, we demonstrate that the proposed framework is able
to render target singing style as illustrated in the right panel of Figure 4.
This can be seen as a by-product of SVC. As addressed by a professional singer
during our pilot study, the corrected source is reminiscent of the original
artist in terms of articulation and particular style of vibrato.

##  4 Discussion

We first note that the melody extractor of WORLD, Harvest [23], is able to
estimate frequency values in an arbitrary fine resolution (continuous
outputs), and is tuned to avoid wrongly estimating voiced frames as unvoiced
(thus the high VR and correspondingly the high VFA). These characteristics are
claimed to be critical in high-quality speech synthesis [23] which are
relevant to the subjective evaluation. The striking difference can be observed
from the left and the center panel in Figure 4. It is evidenced that frames of
SEG-based contours are much more likely to be predicted as unvoiced,
especially seen in the target, which could dramatically hinder the synthesis
quality when the frames are actually voiced [24]. Meanwhile, the high VFA of
WORLD does not seem to bring degradation to perceptual quality, which is also
shown in the previous works [23, 24]. Moreover, as the framework re-
synthesizes the corrected source with the target pitch contour, the predicted
unvoiced frames are harmful to the quality. These observations could explain
the positive correlation between VR/VFA and JODs, and that WORLD prevails in
terms of quality.

On the other hand, the SoTA melody extractors such as SEG cast the task of
pitch extraction as a problem of classification (discrete outputs), and are
trained to optimize the standard metrics which are calculated based on human
annotations of pitch contours. The contrast of WORLD and SEG is further
demonstrated in Figure 5. Once again, one can spot more gaps between segments
along the SEG-based contour. This actually aligns with the ground-truth and
implies the limitations of annotation that will be addressed later. The
discontinuity, together with the discrete outputs, might explain how SoTA
models are outperformed by WORLD in terms of fluency. We can also observe
“pitch spikes” along the WORLD-based contour, which might signify consonants
and could be the main culprit that causes the inferior OA, RPA, and RCA.
Modeling the consonants instead of suppressing them as unvoiced, however,
improves quality of synthesis as the latter hinders the estimation of SP and
AP.

Additionally, WORLD is shown to outperform SEG in terms of naturalness in
Figure 2, and the criteria is shown to have negative correlation coefficients
with OA, RPA, and RCA; and positive ones with VR and VFA from Figure 3. This
implies that an accurate pitch contour, evaluated by the existed metrics, is
not necessarily natural in perception. Similarity is positively correlated
with VR and VFA, which implies that the artifacts caused by the discontinuity
of pitch contours could hinder the perception of vocal similarity.

Based on the discussion, we summarize the lessons that we could learn as
follows. (1) Limitations of human annotations: Annotations of pitch contours
are hardly agreed by different human annotators. Time instants associated with
ambiguous pitch values further aggravate the problem, such as boundaries of
pitched segments and voiced consonants. On singing voice, a particular
methodology is introduced to note-level transcription [25, 26], but is not yet
a practice for the pitch contour estimation [22]. An established annotation
protocol which precisely accommodates the ambiguity is therefore necessary in
order to perform a fairer benchmark with both continuously and discretely
valued pitch estimators. (2) Limitations of the standard metrics: The
tolerance of $\pm{50}$ cents for estimation errors is usually used as a
workaround for the subjective nature of the annotation. An improved metric
thus can be the one considering different tolerance levels by e.g. weighted
average with weights inversely proportional to the values of thresholds. A
similar idea is proposed for the voicing estimation [27]. Moreover, according
to our experimental results, continuity of the pitch contour may play an
important role in perceptual evaluation, which is also taken into account by
previous works [28, 29]. (3) Limitations of classification-based melody
extractors: Discrete predictions of pitch values may lead to contours lacking
in naturalness and fluency, which are detrimental to synthesis. They are
however favored by the standard metrics. On the other hand, the F0 estimator
that produces continuous outputs, albeit inferior in terms of the metrics due
to the pitch variations nearby pitched segments, is favored perceptually. This
again hints towards the lack of the generalizability of the existed metrics
[30].

##  5 Conclusion

We devise a streamlined framework for automatic SVC which accommodates real-
world use cases, correct both pitch and rhythmic errors, and render expressive
parameters from professional singers. We investigate the perceptual validity
of the standard metrics for vocal melody extraction through the lens of the
application of SVC. The results show that a good performance in terms of most
of the metrics does not signify better perceptual scores, whereby we highlight
the challenges posed to the existed annotations, metrics, and melody
extractors. Paths forward involve devising a robust quantitative metric which
accounts for the issues presented in the discussion.

## References

  * [1] S. Salazar, R. Fiebrink, G. Wang, M. Ljungstrom, J. C. Smith, and P. R. Cook,  “Continuous score-coded pitch correction,” 2015,  Sept. 29 2015, US Patent 9,147,385.
  * [2] P. Cano, A. Loscos, J. Bonada, M. de Boer, and X. Serra,  “Voice morphing system for impersonating in karaoke applications,”  in International Computer Music Conference, 2000.
  * [3] “Melodyne by celemony,” http://www.celemony.com/en/melodyne/what-is-melodyne,  [Online; accessed 24-May-2020].
  * [4] “Auto-tune and vocal processing tools by antares audio technologies,” http://www.antarestech.com,  [Online; accessed 24-May-2020].
  * [5] Y.-J. Luo, M.-T. Chen, T.-S. Chi, and L. Su,  “Singing voice correction using canonical time warping,”  in International Conference on Acoustics, Speech and Signal Processing, 2018, pp. 156–160.
  * [6] E. Azarov, M. Vashkevich, and A. Petrovsky,  “Guslar: a framework for automated singing voice correction,”  in International Conference on Acoustics, Speech and Signal Processing, 2014, pp. 7919–7923.
  * [7] S. Wager, G. Tzanetakis, C.-I. Wang, and M.J. Kim,  “Deep autotuner: a pitch correcting network for singing performances,”  in International Conference on Acoustics, Speech and Signal Processing, 2020, pp. 246–250.
  * [8] Yong S. and Nam J.,  “Singing expression transfer from one voice to another for a given song,”  International Conference on Acoustics, Speech and Signal Processing, pp. 151–155, 2018.
  * [9] F.-R. Stöter, S. Uhlich, A. Liutkus, and Y. Mitsufuji,  “Open-unmix - a reference implementation for music source separation,”  Journal of Open Source Software, vol. 4, no. 41, pp. 1667, 2019\.
  * [10] A. Défossez, N. Usunier, L. Bottou, and F. Bach,  “Music source separation in the waveform domain,”  arXiv preprint arXiv:1911.13254, 2019.
  * [11] M. Morise, F. Yokomori, and K. Ozawa,  “World: A vocoder-based high-quality speech synthesis system for real-time applications,”  IEICE Transactions on Information and Systems, vol. 99, no. 7, pp. 1877–1884, 2016.
  * [12] J. W. Kim, J. Salamon, P. Li, and J. P. Bello,  “Crepe: A convolutional representation for pitch estimation,”  in International Conference on Acoustics, Speech and Signal Processing, 2018, pp. 161–165.
  * [13] L. Su,  “Vocal melody extraction using patch-based cnn,”  in International Conference on Acoustics, Speech and Signal Processing, 2018, pp. 371–375.
  * [14] W.-T. Lu and L. Su,  “Vocal melody extraction with semantic segmentation and audio-symbolic domain transfer learning,”  in International Society for Music Information Retrieval Conference, 2018, pp. 521–528.
  * [15] F. Zhou and F. Torre,  “Canonical time warping for alignment of human behavior,”  in Advances in neural information processing systems, 2009, pp. 2286–2294.
  * [16] C.-L. Hsu and J.-S. R. Jang,  “On the improvement of singing voice separation for monaural recordings using the mir-1k dataset,”  Transactions on Audio Speech and Language Processing, vol. 18, no. 2, pp. 310–319, 2010.
  * [17] “i-sing,” https://i-singworld.com,  [Online; accessed 24-May-2020].
  * [18] Maria Perez-Ortiz and Rafal K. Mantiuk,  “A practical guide and software for analysing pairwise comparison experiments,” 2017.
  * [19] E. Zerman, Hulusic V, G. Valenzise, R. Mantiuk, and F. Dufaux,  “The relation between mos and pairwise comparisons and the importance of cross-content comparisons,”  in Human Vision and Electronic Imaging, 2018.
  * [20] N. Shah, S. Balakrishnan, J. Bradley, A. Parekh, K. Ramchandran, and M. Wainwright,  “Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependence,”  San Diego, California, USA, 09–12 May 2015, vol. 38 of Proceedings of Machine Learning Research, pp. 856–865, PMLR.
  * [21] C. Raffel, B. McFee, E. J. Humphrey, J. Salamon, O. Nieto, D. Liang, and D. P. W. Ellis,  “Mir_eval: A transparent implementation of common MIR metrics,”  in International Society for Music Information Retrieval Conference, 2014, pp. 367–372.
  * [22] J. Salamon, E. Gomez, D. P. W. Ellis, and G. Richard,  “Melody extraction from polyphonic music signals: Approaches, applications, and challenges,”  IEEE Signal Processing Magazine, vol. 31, no. 2, pp. 118–134, 2014\.
  * [23] M. Morise,  “Harvest: A high-performance fundamental frequency estimator from speech signals,”  Interspeech, p. 2321–2325, 2017.
  * [24] D. Thomas, H Goeric, K. Viacheslav, and M. Alexis,  “Traditional Machine Learning for Pitch Detection,”  IEEE Signal Processing Letters, vol. 25, no. 11, pp. 1745–1749, 2018.
  * [25] J. Mora, F. Gómez, E. Gómez, and F. J. Borrego J. Díaz-Báñez,  “Characterization and melodic similarity of a cappella flamenco cantes,”  International Society for Music Information Retrieval Conference, pp. 351–356, 01 2010.
  * [26] E. Molina, A. Barbancho, L. Tardón, and I. Barbancho,  “Evaluation framework for automatic singing transcription,”  in International Society of Music Information Retrieval, 2014.
  * [27] Rachel M. Bittner and Juan J. Bosch,  “Generalized metrics for single-f0 estimation evaluation,”  in International Society of Music Information Retrieval, 2019.
  * [28] K. Frieler, D. Basaran, F. Höger, H.-C. Crayencour, G. Peeters, and S. Dixon,  “Don’t hide in the frames: Note- and pattern-based evaluation of automated melody extraction algorithms,”  2019, DLfM ’19, p. 25–32.
  * [29] J. Boschn, R. Marxer, and E. Gómez,  “Evaluation and combination of pitch estimation methods for melody extraction in symphonic classical music,”  Journal of New Music Research, vol. 45, pp. 1–17, 05 2016.
  * [30] J. Salamon,  “What’s broken in music informatics research? three uncomfortable statements,”  in International Conference of Machine Learning, 2019.