# Clinical BERTScore: An Improved Measure of Automatic Speech Recognition
Performance in Clinical Settings

Joel Shor
Verily Life Sciences, USA
<EMAIL_ADDRESS>Ruyue Agnes Bi
MIT, USA
<EMAIL_ADDRESS>Subhashini Venugopalan
Google Research, USA Steven Ibara
Verily Life Sciences, USA Roman Goldenberg
Verily Life Sciences, Israel Ehud Rivlin
Verily Life Sciences, Israel

###### Abstract

Automatic Speech Recognition (ASR) in medical contexts has the potential to
save time, cut costs, increase report accuracy, and reduce physician burnout.
However, the healthcare industry has been slower to adopt this technology, in
part due to the importance of avoiding medically-relevant transcription
mistakes. In this work, we present the Clinical BERTScore (CBERTScore), an ASR
metric that penalizes clinically-relevant mistakes more than others. We
demonstrate that this metric more closely aligns with clinician preferences on
medical sentences as compared to other metrics (WER, BLUE, METEOR, etc),
sometimes by wide margins. We collect a benchmark of 18 clinician preferences
on 149 realistic medical sentences called the Clinician Transcript Preference
benchmark (CTP) and make it publicly available111https://osf.io/tg492/ for the
community to further develop clinically-aware ASR metrics. To our knowledge,
this is the first public dataset of its kind. We demonstrate that CBERTScore
more closely matches what clinicians prefer.

##  1 Introduction

Clinicians in a number of disciplines work in an overburdened healthcare
system that leads to difficult working environments and an epidemic of
physician burnout Dzau et al. (2018). AI-related technologies have the
potential for improving efficiency on repetitive tasks, therefore increasing
both patient throughput and decreasing physician burnout. For example,
physicians in a number of disciplines spend as much time doing paperwork as
with patients Tai-Seale et al. (2017). However, the adoption of speech
technology in the medical community has been slow Latif et al. (2021), and
there are a number of speech technologies that could improve efficiency.

Speech technology can be applied to a number of medical problems including
transcribing patient-physician conversations Shafran et al. (2020), helping
dysarthric patients communicate Shor et al. (2020), and diagnosing medical
conditions from speech Shor et al. (2022); Shor and Venugopalan (2022);
Peplinski et al. (2021); Venugopalan et al. (2021). In this work, we focus on
the task of generating a report after a colonoscopy procedure.

One of many reasons for the lower adoption of time-saving speech transcription
technologies is that the ASR systems often don’t perform as well in real-world
clinical settings as they do on evaluation benchmarks. The most common metric
for measuring ASR performance, Word Error Rate (WER), has significant
practical drawbacks Wang et al. (2003); Morris et al. (2004); He et al.
(2011). First, all mistakes are treated equally. In clinical settings,
however, medical words are more important (e.g. "had complete resection"
$\rightarrow$ "had complete c-section" is a worse mistake than $\rightarrow$
"has complete resection", but both have equal WER). Second, some mistakes
affect the overall intelligibility more than others (e.g. "was no perforation"
$\rightarrow$ "was no puffer age" vs "was not any perforation"). Although
researchers have proposed alternatives to the WER, no metric combines medical
domain knowledge with recent AI advances in language understanding.

In this work, we make the following contributions:

  1. 1.

Generate a collection of realistic medical sentences and transcripts with
plausible ASR errors and collect preferences from 18 clinicians on 149
sentences. We publicly released this dataset for reference and future studies.
This is the first public dataset of its kind.

  2. 2.

Present the Clinical BERTScore (CBERTScore) and demonstrate that it more
closely matches clinician preferences on medical transcripts than other ASR
metrics (WER, BLEU, METEOR, BERTScore).

  3. 3.

Demonstrate that CBERTScore does not perform worse than other metrics on non-
medical transcripts.

##  2 Related work

Figure 1: Left: Background of the clinicians who were surveyed to create the
Clinician Transcript Preference (CTP) dataset. Right: Some examples of triplet
medical sentences, which transcript clinicians prefer, and which transcript
scores better based on different metrics.

There are a number of ways to evaluate transcript quality. The Word Error Rate
(WER), is the simplest to compute and most common. It counts the number of
insertions, deletions, and substitutions between two text strings, and
normalizes by the length of the reference string. The Bilingual Evaluation
Understudy (BLEU) Papineni et al. (2002) measures the amount of n-gram overlap
between two text strings (where n is often 4). It captures the intuition that
groups of words are important in addition to individual words. METEOR Banerjee
and Lavie (2005) focuses on unigrams, but computes an explicit alignment
between two strings and takes both precision and recall into consideration.
While these techniques are cheap to compute, they primarily focus on character
or string similarity, not semantic similarity.

Our work most closely follows the BERTScore Zhang et al. (2019). This metric
computes a neural word embedding for each word in the reference and candidate.
Embeddings are matched using cosine distance instead of string similarity, and
the final score takes precision and recall into account (see Fig.1). This
method takes semantic similarity into account, but not that some words are
more important to preserve in clinical contexts.

Structured graphs are one way to encode real-world knowledge in a machine-
readable format. The Knowledge Graph (KG) Singhal (2012) is a publicly
available structure that encodes medical knowledge. Previous work has used the
medical subset of the KG to learn medical entity extraction Shafran et al.
(2020). We primarily follow this approach to determine which words are
clinically significant.

##  3 Methods

###  3.1 Clinical BERTScore

Our proposed metric, the Clinical BERTScore (CBERTScore), combines the
BERTScore Zhang et al. (2019) and the medical subset of the Knowledge Graph
Shafran et al. (2020).

BERTScore is a relatively novel language generation evaluation metric proposed
in Zhang et al. (2019) based on pre-trained BERT contextual embeddings. It is
designed to capture semantic similarity between two sentences, instead of
simple string matching. Given a reference sentence $x=\langle
x_{1},...,x_{k}\rangle$ and a candidate sentence
$\hat{x}=\langle\hat{x}_{1},...,\hat{x}_{l}\rangle$, we first represent each
token by a contextual embedding, and then calculate the cosine similarities
between the tokens. Each token in the reference sentence is matched to the
most similar token in the candidate sentence, and vice versa. The former is
used to compute the recall $R_{\textsc{bert}}$, and the latter to compute the
precision $P_{\textsc{bert}}$. Precision and recall are then combined into a
single score BERTScore as follows:

R_bert & = 1|x| ∑_x_i ∈x max_^x_j ∈^x x_i^T ^x_j ,
P_bert = 1|^x|∑_^x_j ∈^x max_x_i ∈x x_i^T^x_j
BERTScore = 2Pbert⋅RbertPbert\+ Rbert

Building on this insight, we define CBERTScore as:

CBERTScore(x, ^x) = & k ×BERTScore_medical(x, ^x) +
(1-k) ×BERTScore_all(x, ^x)
, where $0\leq k\leq 1$

BERTScoreall is computed over all words in the sentences, and BERTScoremedical
is computed over a subset of them that are medically relevant. If there are no
medical terms in either the reference or candidate sentence, we define the
CBERTScore to be the standard BERTScore (on all words), i.e., $k$ is set at
$0$.

We inject medical information into this metric in two ways. First, we compute
a weighted score on a subset of words involving medical terms, as determined
by the Knowledge Graph Shafran et al. (2020). Second, we tune the weight of
the clinical term penalty to best match a clinician transcript dataset (CTP)
that we collected. We describe our method for determining $k$ in Sec. 3.1.2.

####  3.1.1 Medical Entities

Similar to Shafran et al. (2020), we derive roughly 20K medically relevant
words from Google’s Knowledge graph Singhal (2012). These words come from
entities with properties such as “/medicine/disease", “/medicine/drug",
“/medicine/medical_treatment", and “/medicine/medical_finding". We also
include numbers for the CBERTScore algorithm, since numerical accuracy is
important in medical contexts.

####  3.1.2 Tuning the medical entities weight factor

CBERTScore has a parameter controlling the weight of the clinical component.
To determine this factor, we picked the best performing $k$ on the training
subset of the Clinician Transcript Preference (CTP) dataset (Sec. 3.2). We
evaluated $k$ using 11 points evenly spaced between 0 and 1, and performed the
evaluation methodology in Sec. 3.2 for each. We then used this value for all
subsequent results and analyses.

###  3.2 Clinician Transcript Preference (CTP) Dataset

In order to compare CBERTScore’s agreement with human preference, we sent out
a Qualtrics survey to elicit judgment specifically from clinicians222Broadly
defined as a person with extensive clinical experience or from a clinical
research background, for our purpose.. We call this dataset the Clinician
Transcript Preference dataset (CTP), and we make it publicly available on the
Open Science Framework (OSF). To our best knowledge, this is the first
publicly available dataset with clinician preferences of transcript errors.

We collected data on 150 sentences. They were divided into three groups, each
containing 50 trials. 18 subjects with clinical backgrounds responded to more
than half the questions. Fig. 1 (left) describes clinician backgrounds. Each
participant was randomly assigned to a group to ensure approximately uniform
response coverage. For each trial, participants are given a ground truth
sentence and two “transcripts" and asked to select the less useful one or to
indicate the two are about the same. An example of such a triplet is as
follows: “Patient elects to go under Propofol sedation." #1: Patient elects to
go under Prilosec sedation. #2: Patient selects to go under Propofol sedation.
The survey was designed to take no more than 20 min to minimize the cognitive
strain on participants. One sentence was malformed, resulting in 149 sentences
for the final dataset.

Figure 2: Comparison of different metrics’ agreement with human rater
transcript preferences. Process of deriving a prediction from metric values is
described in Sec. 3.2. In all plots, "CBERTScore1.0“ is the performance from
only the medical term component ($k=1.0$ in Sec. 3.1). "CBERTScore0.4“ uses
the optimal value of $k$ according to the train set. Left: Agreements with
clinicians on the CTP benchmark when labels are derived using majority voting.
Center: Agreements with clinicians on the CTP benchmark when restricted to
questions with unanimous answers. Right: Agreement with speech pathologist
raters on the non-medical dataset, when restricting the data to cases where
there is a fidelity difference between two candidate transcripts.

####  3.2.1 Constructing the CTP triplets

To generate the triplets of (target, transcript #1, transcript #2) used in the
survey, we started by downloading publicly available YouTube videos on
colonoscopies created by GI physicians and educational institutes. The target
sentences were transcribed by Google’s publicly available Speech-to-Text
medical dictation model Soltau et al. (2021) and manually checked for
accuracy. Filler words such as “uh" and repeated words were edited out.
Sentences longer than 30 words or less than 5 were discarded.

For each target sentence, transcript #1 was generated by one of Google’s
other, non-medical, publicly available ASR models. Transcripts with an edit
distanceedi  outside [1, 3] were discarded. This procedure generated 1220
candidate sentences.

To ensure that the two transcripts were roughly comparable in terms of
fidelity, transcript #2 was generated synthetically. We used a publicly
available English word frequency dictionaryGoldhahn et al. (2012) to select
words in the target sentence that were candidates for synthetic errors.
Candidate words were at least 5 characters, appeared in the 1M word dictionary
fewer than 10 times, and were not proper nouns. 486 candidate sentences
matched these criteria. Finally, transcript #2 was generated by deleting the
candidate word or manually substituting it with a phonetically similar word or
phrase333A Python fuzz search algorithm based on CMU Pronouncing Dictionary
was used for consistency.. We discarded similar sentences and selected 150
triplets for the final survey. The ordering of the two transcripts was
randomized, and so were the sentences.

####  3.2.2 Evaluating metrics on the CTP

To compare the ability of different metrics to agree with rater preference
from the CTP, we define a 3-class classification problem as follows:
Predicted& better transcript(M)(gt, t_1, t_2) =
{t_1 M(gt, s_1) - M(gt, s_2) > l
t_2 M(gt, s_1) - M(gt, s_2) < -l
same else where $M$ is an evaluation metric, $gt$ is the ground truth
sentence, and $t_{i}$ are the transcripts. Note the predictions are reversed
for the WER, since lower values indicate higher fidelity. $l$ is a free
variable, which we optimize separately for each metric. We split the data into
two halves, choose the best performing $l$ on one half, and report the
accuracy using that $l$ on the second half.

####  3.2.3 Non-medical sentences

To demonstrate that CBERTScore doesn’t degrade on non-medical speech, we
compare the metrics’ agreement with rater preferences on a dataset with
annotations similar to Tobin et al. (2022). Part of this dataset consists of
5-tuples of (ground truth sentence, transcript 1, transcript 2, assessment 1,
assessment 2), where the sentence assessments describe how much of the ground
truth sentence’s meaning is captured in the transcript. We used a subset of
103 utterances from our annotated data where the ratings were not the same,
and at least one transcript was rated as having “Major errors". We report
performance using a similar formulation as on the CTP evaluation in Sec.
3.2.2: we frame this as a 2-way classification problem (no cutoff is needed
since we exclude tuples that have the same rating).

##  4 Results

###  4.1 Clinician responses

18 clinicians responded to a total number of 149 triplet questions. Each
question had 5 or 6 responses. 78% of questions had more than half agreement
on which transcript was less useful and 42% had more than 80% agreement.
Clinicians thought transcripts were the same usefulness in 21% of cases.

###  4.2 Metric agreement on medical text

We report 3-way accuracy classification on the CTP dataset using two labeling
schemes (Fig. 2). In the first, we only look at the questions where more than
half the respondents agreed. In the second, we report accuracy on the
questions where more than 4/5 of the respondents agreed. For both numbers, we
determine the cutoff from one half the data and report accuracy on the second
half.

First, the metric ordering by performance is the same using both labeling
schemes, and the best CBERTScore medical weighting factor was the same using
both label schemes. Second, BERTScore and CBERTScore are significantly more
closely aligned with clinician preferences than other metrics. Third,
CBERTScore weighted entirely toward medical terms outperforms or ties with
BERTScore agreement. Fourth, the weighted combination of medical and non-
medical terms outperforms other metrics in terms of clinician agreement.
Fifth, the medical component meaningfully improves the performance of
CBERTScore over BERTScore (75.9% vs 67.2% and 87.5% vs 84.4%).

###  4.3 Metric agreement on non-medical text

CBERTScore was the second best-performing metric on non-medical text.
Importantly, the addition of the medical component did not degrade the
performance compared to BERTScore.

##  5 Discussion

###  5.1 Knowledge Graph medical terms wins and losses on the CTP

The CTP (Sec. 3.2) had 127 distinct words that were the source of transcript
errors, and 684 distinct other words. The medically-relevant terms used in the
CBERTScore algorithm, identified primarily from the Knowledge Graph as
described in Sec. 3.1.1, intersected with 99 of the 127 transcript error
words. By manual inspection, 25 of the 28 transcript error words in the CTP
not included in the CBERTScore word list were used in a medical context but
were not only medical in meaning (ex. “surveillance", “tethered", and
“longitudinal"). 3 of the 28 missed words did have a primarily medical
meaning, but were not included in the CBERTScore list either due to errors in
the KG or errors in the queries generating the list (“cologuard",
“colonoscope", “protuberance"). Some of the words have a clear meaning in a
medical context, and could be manually added to the list for future
applications (“snare", “suctioning", etc.).

The CBERTScore word list included 100 words that weren’t selected for
transcript errors. Many of these are medical in nature, but were not selected
for synthetic transcript errors via the method described in Sec. 3.2 (ex.
“endoscope", “hypoplastic", “lymphoma").

###  5.2 CBERTScore performance on the CTP

Figure 3: Fraction of cases where metric Y is correctly conditioned on metric
X and Y disagreeing. An indicator of how similar the pattern of mistakes is
between metrics.

####  5.2.1 CBERTScore wins

Fig. 3 left shows the degree to which better-performing metrics subsume other
metrics, or make a different pattern of mistakes. The plot shows the
$(\text{MetricYcorrect})/(\text{MetricXandYdisagree)}$. Metrics that have
higher clinician agreement and a high fraction on this plot are strictly
better, whereas metrics with higher agreement but a low value in this plot
indicate that another metric might have an additional signal. We see that
CBERTScore is nearly strictly better than the other metrics, with the possible
exception of METEOR (when they differ, METEOR gives the correct rating in
roughly a third of cases).

There were some triplets that CBERTScore got correct that no other metric did.
The improvements over BERTScore always involved a medical term, and sometimes
involved encouraging the metric to prioritize medical mistakes (ex. "Marked
the site with 5 cc's of indigo carmine.“ $\rightarrow$ "Marked the site with 5
cici's of indigo carmine.“ vs "Marked the sight with 5 cc's of indigo
carmine.“)

There were thirteen triplets that the neural word embeddings predicted
correctly that other metrics did not. Many of these wins came from the
strength of neural word embeddings penalizing less for semantically similar
mistakes (ex. "Small burst of coagulation to create a darkish white ablation.“
$\rightarrow$ "Small burst of coagulation to create a darkish white oblation.“
vs "Small burst of coagulation to create a dark white ablation.“).
Furthermore, BERTScore agreed with clinicians on some medical word mistakes,
likely due to the BERT embedding somewhat understanding when a transcript
error leads to a large semantic change in a medical term (ex. "No ongoing
infection or coagulopathy.“ $\rightarrow$ "No on going infection or
coagulopathy.“ vs "No ongoing infection or glomerulopathy.“).

####  5.2.2 CBERTScore mistakes

Fig. 3 shows that METEOR made the most correct predictions when CBERTScore was
incorrect. Some mistakes are due to the KG medical list being incomplete. For
example, "longitudinal“ was not included, but has medical meaning in clinical
contexts (ex. "The longitudinal extent of the hot snare.“ $\rightarrow$ "The
long eternal extent of the hot snare.“ vs "The longitudinal extend to the hot
snare.“).

Another pattern of mistake is when a non-medical adjective contains an error,
but the adjective modifies a medical term in an important way. For example,
"vessel“ is a medical term, but "feeding“ is not (ex. "This polyp is at high
risk of bleeding, with multiple feeding vessels.“ $\rightarrow$ "This polyp is
at high risk of bleeding, with multiple seeding vessels.“ vs "This polyp is at
high risking bleeding, with multiple feeding vessels.“). This suggests that
future work might include modifications and dependencies when calculating
clinical importance.

Finally, a third pattern of mistake involves the fact that METEOR penalizes
complex correspondences between candidate and reference sentences, while
CBERTScore only considers the best pairwise word matches. One example in the
CTP preserves most of the words, but reorders them (ex. "Inject into the head
of the polyp, another 1 to 2 cc.“ $\rightarrow$ "Injectant the head of the
polyp, another 1 to 2 cc.“ vs "Inject into the head of the polyp, another 1 2
to cc.“).

##  6 Conclusions

We present CBERTScore, a novel metric that combines medical domain knowledge
and recent advances in neural word embeddings. We collect and release a
benchmark of clinician rater preferences on transcript errors, demonstrate
that CBERTScore is more closely aligned with clinician preferences, and
release the benchmark for the research community to continue to improve ASR in
medical contexts.

## References

  * (1) Edit distance.  https://en.wikipedia.org/wiki/Edit_distance.  Accessed: 2023-03-03.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.  In _Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization_ , pages 65–72.
  * Dzau et al. (2018) Victor J Dzau, Darrell G Kirch, Thomas J Nasca, et al. 2018.  To care is human—collectively confronting the clinician-burnout crisis.  _N Engl J Med_ , 378(4):312–314.
  * Goldhahn et al. (2012) Dirk Goldhahn, Thomas Eckart, and Uwe Quasthoff. 2012.  Building large monolingual dictionaries at the Leipzig corpora collection: From 100 to 200 languages.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)_ , pages 759–765, Istanbul, Turkey. European Language Resources Association (ELRA).
  * He et al. (2011) Xiaodong He, Li Deng, and Alex Acero. 2011.  Why word error rate is not a good metric for speech recognizer training for the speech translation task? In _2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 5632–5635.
  * Latif et al. (2021) Siddique Latif, Junaid Qadir, Adnan Qayyum, Muhammad Usama, and Shahzad Younis. 2021.  Speech technology for healthcare: Opportunities, challenges, and state of the art.  _IEEE Reviews in Biomedical Engineering_ , 14:342–356.
  * Morris et al. (2004) Andrew Cameron Morris, Viktoria Maier, and Phil Green. 2004.  From wer and ril to mer and wil: improved evaluation measures for connected speech recognition.  In _Eighth International Conference on Spoken Language Processing_.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Peplinski et al. (2021) Jacob Peplinski, Joel Shor, Sachin Joglekar, Jake Garrison, and Shwetak Patel. 2021.  FRILL: A non-semantic speech embedding for mobile devices.  In _Interspeech 2021_. ISCA.
  * Shafran et al. (2020) Izhak Shafran, Nan Du, Linh Tran, Amanda Perry, Lauren Keyes, Mark Knichel, Ashley Domin, Lei Huang, Yu-hui Chen, Gang Li, Mingqiu Wang, Laurent El Shafey, Hagen Soltau, and Justin Stuart Paul. 2020.  The medical scribe: Corpus development and model performance analyses.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 2036–2044, Marseille, France. European Language Resources Association.
  * Shor et al. (2022) Joel Shor, Aren Jansen, Wei Han, Daniel Park, and Yu Zhang. 2022.  Universal paralinguistic speech representations using self-supervised conformers.  In _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE.
  * Shor et al. (2020) Joel Shor, Aren Jansen, Ronnie Maor, Oran Lang, Omry Tuval, Félix de Chaumont Quitry, Marco Tagliasacchi, Ira Shavitt, Dotan Emanuel, and Yinnon Haviv. 2020.  Towards Learning a Universal Non-Semantic Representation of Speech.  In _Proc. Interspeech 2020_ , pages 140–144.
  * Shor and Venugopalan (2022) Joel Shor and Subhashini Venugopalan. 2022.  TRILLsson: Distilled universal paralinguistic speech representations.  In _Interspeech 2022_. ISCA.
  * Singhal (2012) Amit Singhal. 2012.  Introducing the knowledge graph: things, not strings.  https://blog.google/products/search/introducing-knowledge-graph-things-not/.  Accessed: 2023-03-03.
  * Soltau et al. (2021) Hagen Soltau, Mingqiu Wang, Izhak Shafran, and Laurent El Shafey. 2021.  Understanding medical conversations: Rich transcription, confidence scores & information extraction.  In _Interspeech_.
  * Tai-Seale et al. (2017) Ming Tai-Seale, Cliff W Olson, Jinnan Li, Albert S Chan, Criss Morikawa, Meg Durbin, Wei Wang, and Harold S Luft. 2017.  Electronic health record logs indicate that physicians split time evenly between seeing patients and desktop medicine.  _Health affairs_ , 36(4):655–662.
  * Tobin et al. (2022) Jimmy Tobin, Qisheng Li, Subhashini Venugopalan, Katie Seaver, Richard Cave, and Katrin Tomanek. 2022.  Assessing ASR Model Quality on Disordered Speech using BERTScore.  In _Proc. 1st Workshop on Speech for Social Good (S4SG)_ , pages 26–30.
  * Venugopalan et al. (2021) Subhashini Venugopalan, Joel Shor, Manoj Plakal, Jimmy Tobin, Katrin Tomanek, Jordan R. Green, and Michael P. Brenner. 2021.  Comparing Supervised Models and Learned Speech Representations for Classifying Intelligibility of Disordered Speech on Selected Phrases.  In _Proc. Interspeech 2021_ , pages 4843–4847.
  * Wang et al. (2003) Ye-Yi Wang, A. Acero, and C. Chelba. 2003.  Is word error rate a good indicator for spoken language understanding accuracy.  In _2003 IEEE Workshop on Automatic Speech Recognition and Understanding (IEEE Cat. No.03EX721)_ , pages 577–582.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019.  BERTScore: Evaluating Text Generation with BERT.  _arXiv e-prints_ , page arXiv:1904.09675.


# Score-informed Networks for Music Performance Assessment

###### Abstract

The assessment of music performances in most cases takes into account the
underlying musical score being performed. While there have been several
automatic approaches for objective music performance assessment (MPA) based on
extracted features from both the performance audio and the score, deep neural
network-based methods incorporating score information into MPA models have not
yet been investigated. In this paper, we introduce three different models
capable of score-informed performance assessment. These are  (i) a
convolutional neural network that utilizes a simple time-series input
comprising of aligned pitch contours and score,  (ii) a joint embedding model
which learns a joint latent space for pitch contours and scores, and  (iii) a
distance matrix-based convolutional neural network which utilizes patterns in
the distance matrix between pitch contours and musical score to predict
assessment ratings.  Our results provide insights into the suitability of
different architectures and input representations and demonstrate the benefits
of score-informed models as compared to score-independent models.

##  1 Introduction

A performance is a sonic rendition of a written musical score (in the case of
Western classical music). The characteristics of a music performance play a
major role in how listeners perceive music, even if performances are based on
the same underlying score [14]. To perform a musical piece, the performer must
first parse the score, interpret or modify the musical information, and
utilize complex motor skills to render the piece on their instrument [21].

From the perspective of the performer, mastery over the art of music
performance is often a journey spanning several years of instruction and
practice. A major factor in learning and improving one’s skill as a performer
is to analyze and obtain feedback regarding the performance. Due to the
complex nature of music performance, students require regular feedback from
trained professionals. Teachers are expected to grade or rate students based
on various performance criteria such as note accuracy or musicality. These
criteria are often ill-defined and subject to interpretation, thus making
objective and consistent music performance assessment (MPA) rather difficult
[29, 26]. Regardless, this subjective manner of MPA is still used, e.g., in
school systems where ensemble members are selected based on instructors’
assessments of student auditions.

Wu et al. discussed the notion of objective descriptors (features) which are
potentially useful for automatic MPA [30]. Such features are computed by
applying signal processing methods to recorded performances and are used to
model teachers’ assessments of the performances using machine learning. With
the rise of deep learning, neural networks were found to outperform the
classical pipeline of feature extraction followed by regression [22]. However,
one issue with these approaches is that they ignore the score that the
students are meant to play. We will refer to such approaches as score-
independent. The idea of incorporating score-based features utilizing audio to
score alignment was explored, e.g., by Vidwans et al. [27]. Further analysis
of hand-crafted features for MPA showed the relative importance of score-based
features over score-independent ones [8]. Therefore, the design of deep
architectures that incorporate score information is an obvious and overdue
extension of previous approaches.

The goal of this paper is to explore different methods to incorporate this
score information. Our hypothesis is that including score information will
lead to improved performance of deep networks in the objective MPA task. To
this end, we present three architectures which combine score and audio
features to make a score-informed assessment of a music performance. First, we
concatenate aligned pitch contours and scores into a 2-dimensional time-series
feature representation that is fed to a convolutional neural network (CNN).
Second, we propose a joint embedding model for aligned score and pitch
contours. The assessment ratings are predicted using the cosine similarity
between the score and performance embeddings. Third, we utilize the distance
matrix, a mid-level representation combining both the score and pitch contour,
as the input to a deep CNN trained to predict the teachers’ assessments.
Finally, using a fairly large scale dataset of middle school and high school
student auditions, we perform an in-depth evaluation comparing these proposed
architectures against each other and with a score-independent baseline
approach for MPA .

##  2 Related Work

MPA deals with the task of assessing music performances based on audio
recordings. Progress in MPA is roughly categorized into feature design-based
approaches [13, 20, 30, 24, 8] and feature learning-based approaches [31, 22,
9]. Feature design-based methods rely on signal processing techniques to
either extract standard spectral and temporal features [13], or use expert
knowledge to extract perceptually motivated features capable of characterizing
music performances [20, 24]. The extracted features are then fed into simple
machine learning classifiers to train models which predict different
performance assessment ratings. Feature learning-based approaches, on the
other hand, stem from the argument that important features for modeling
performance assessments are not trivial and cannot be easily described. Hence,
they rely on using mid-level representations (such as pitch contours or mel-
spectrograms) as input to sophisticated machine learning models such as sparse
coding [9, 31] and neural networks [22].

Most performances of Western music, however, are based on written musical
scores. Hence, performances are also assessed based on their perceived
deviations from the underlying score. There has been some prior research on
incorporating the score information into the assessment modeling process. Most
of the these approaches rely on computing descriptive features using some
notion of distance between the score representation and the performance
representation [17, 3, 19, 6, 11]. The most common approach has been to first
use an alignment algorithm, e.g., Dynamic Time Warping (DTW) [25], to
temporally align the performance recording with the score and then compute
descriptive features which characterize the deviations of the performance from
the score [27, 1]. However, to the best of our knowledge, incorporating score
information directly into neural network-based models for MPA has not been
investigated before.

Score-informed approaches have helped improve results for both related
performance analysis tasks and other music information retrieval tasks. Most
of these methods have also relied on an alignment between the audio recording
and the score as the primary tool for incorporating score information.
Aligning audio recordings with scores has been useful for several tasks such
as detecting expressive features in music performances [15], identifying
missing notes and errors in piano performances [5], and segmenting syllables
in vocal performances[23]. Scores have also been used to generate soft labels
and/or artificial training data for tasks such as source separation [18, 4].

##  3 Methods

We propose and compare three different approaches to incorporate the score
information with audio features for MPA.111The code is available at:
https://github.com/biboamy/FBA-Fall19 The score information is represented as
the MIDI pitch sequence (in ticks) obtained from the sheet music of the score
to be performed. Henceforth, the MIDI pitch sequence will be referred to as
the score. The student’s performance is represented by the pitch contour of
the audio. We use pitch contour since it captures both pitch and rhythmic
information. Musical dynamics and timbre are ignored in this study; while
dynamics are an important expressive tool for the performer [14], the score
usually lacks specificity in dynamics instructions and cannot serve as the
same absolute reference as for pitch and rhythm.

###  3.1 Score-Informed Network (SIConvNet)

Figure 1: Schematic for the SIConvNet. The aligned score and pitch contour are
stacked together and fed into a 4-layer CNN to directly predict the assessment
ratings.

The first approach that we use is probably the most straightforward way of
incorporating score information into the assessment model. A simple CNN is
used that relies on both the score and performance as the input and directly
predicts the assessment ratings.

####  3.1.1 Input Representation

The input representation for this approach is constructed by simply stacking
an aligned pitch contour and score pair to create a $N\times 2$ matrix, where
$N$ is the sequence length of the pitch contour. The two channels correspond
to the pitch contour and score, respectively.

In order to obtain this representation, we first consider a pitch contour
snippet of length $N$ (sequence of logarithmic frequencies). Then, we find the
corresponding part of the score using a DTW-based alignment process. The
obtained score snippet (sequence of MIDI note numbers) is then resampled to
have the same length $N$ as the pitch contour.

####  3.1.2 Model Architecture

A schematic of the model architecture is shown in Figure 1. We use a simple
4-layer CNN based on the architecture proposed by Pati et al. [22] and append
a single linear layer which predicts the assessment. Each convolutional stack
consists of a 1-D convolution followed by a 1-D batch normalization layer [12]
and ReLU non-linearity. The linear layer at the end comprises of a dense layer
followed by Leaky ReLU non-linearity.

###  3.2 Joint Embedding Network (JointEmbedNet)

Figure 2: Schematic of the JointEmbedNet architecture.

The second approach is based on the assumption that performances are rated
based on some sort of perceived distance between the performance and the
underlying score being performed. Consequently, we use two separate encoder
networks to project the score and the pitch contour to a joint latent space
and then use the similarity between the embeddings to predict the assessment
ratings.

####  3.2.1 Input Representation

This approach uses the same input representation as SIConvNet (see Section
3.1.1). However, instead of stacking the aligned pitch contour and the score,
the individual $N\times 1$ sequences are fed separately to the two encoders.

####  3.2.2 Model Architecture

This network (see Figure 2) uses two 1-D convolutional encoders having the
same architecture as SIConvNet. Each encoder has 4 convolutional blocks to
extract high level feature embeddings. The performance encoder is expected to
extract relevant features pertaining to the performance from the pitch
contour. On the other hand, the score encoder is expected to extract the
important features from the score. Assuming that the assessment rating for the
performance is high if these two embeddings are similar, we use the cosine
similarity $\mathrm{cos}(E_{\mathrm{score}},E_{\mathrm{performance}})$ between
the two embeddings to obtain the predicted assessment rating.
$E_{\mathrm{score}}$ and $E_{\mathrm{performance}}$ are the embeddings
obtained from the score and performance encoders, respectively. If the two
embeddings are similar, the cosine similarity is close to one, and the model
will predict a higher rating.

###  3.3 Distance Matrix Network (DistMatNet)

The final approach uses a distance matrix between the pitch contour and the
score as the input to the network. Given the information from both the pitch
contour and the score, the task of performance assessment might be interpreted
as finding a performance distance between them. Thus, the choice of the
distance matrix as the input representation allows the model to learn from the
pitch differences. A Residual CNN [10] architecture is chosen for the network.

Figure 3: Schematic of the DistMatNet architecture.

####  3.3.1 Input Representation

The distance matrix elements are the pair-wise wrapped distances between the
pitch contour and the MIDI pitch sequence. The octave-independent wrapped-
distance is used to compensate for possible octave errors made by the pitch
tracker. To ensure a uniform input size to the network, the matrix is
resampled to a square shape of a fixed size. Thus, a performance with constant
tempo would result in an aligned path located on the diagonal. Unlike the
previous two methods where the input pitch contour and the score are aligned
using DTW, the distance matrix input avoids any error propagation caused by
alignment errors. The choice of this input representation stems from the
success of distance matrices (or self-similarity matrices) in other areas of
MIR such as structural segmentation [7, 2] and music generation [28].

####  3.3.2 Model Architecture

The model architecture is shown in Figure 3. It is composed of 3 residual
blocks. Each residual block has 2 convolutional layers. Dropout and max-
pooling are added between each residual block. A classifier with two linear
layers (128 features) with one ReLU and dropout layer in between is used after
the residual network to perform regression prediction. We use (3,3) kernal
size and 4 feature maps for all convolutional layers, 0.2 dropout rate, and a
(3,3) kernal size for all max-pooling layers.

##  4 Experiments

###  4.1 Dataset

The dataset we use to evaluate our methods is a subset of a large dataset of
middle school and high school student performances. These are recorded for the
Florida All State Auditions, which are separated into three bands:  (i) middle
school band,  (ii) concert band, and  (iii) symphonic band.  The recordings
contain auditions spanning 6 years (from 2013 to 2018), and feature several
monophonic pitched and percussion instruments. Each student performs rehearsed
scores, scales and a sight reading exercise. For the purpose of this study we
limit our experiments to the technical etude for middle school and symphonic
band auditions. We choose Alto Saxophone, Bb Clarinet and Flute performances
due to these being the most popular across all pitched instruments. Table 1
shows the distribution of data across different instruments. The average
duration of each performance is 30 s for middle school and 50 s for symphonic
band students. The dataset also includes the musical scores that the students
are supposed to perform for each exercise. The average length (in notes) of
the musical scores are 136 for middle school and 292 for symphonic band. Note
that these scores are the same across all students performing the same
instrument in the same year but vary across years and instruments.

The dataset also contains expert assessments for each exercise of a student
performance. Each performance is rated by one expert along $4$ criteria
defined by the Florida Bandmasters’ Association (FBA):  (i) musicality,  (ii)
note accuracy,  (iii) rhythmic accuracy, and  (iv) tone quality.  All ratings
are on a point-based scale and are normalized to range between $0$ to $1$ by
dividing by the maximum. Since we focus on pitch contours as the primary audio
feature, tone quality is excluded from this study.

| Middle School | Symphonic Band
---|---|---
Alto Saxophone | 696 | 641
Bb Clarinet | 925 | 1156
Flute | 989 | 1196
Table 1: Number of performances for the different instruments per band.

####  4.1.1 Data pre-processing

The pitch contours are extracted using the pYIN algorithm [16] with a block
size and hop size of $1024$ and $256$ samples, respectively. The audio
sampling rate is 44100 Hz. The extracted frequencies are converted from Hz to
MIDI pitch (unlike the MIDI pitches from the musical score, these can be
floating point numbers). Both the resulting pitch contour and musical score
are normalized by dividing by $127$. Finally, for the purpose of model
training and evaluation, we divide our dataset into three randomly sampled
subsets: training, validation, and testing. We use a ratio of $8\colon 1\colon
1$ for splitting the dataset.

We use random-chunking as a data augmentation tool when training SIConvNet and
JointEmbedNet since it has shown to be useful in improving model performance
[22]. First, the pitch contour is chunked into snippets of length $N$ by
randomly selecting the starting position. The corresponding aligned and
length-adjusted score snippet is obtained using the method described in
Section 3.1.1. We assume the chunked segment has the same assessment score as
the whole recording. We do not perform chunking on our distance matrix since
the matrix has already been resampled into a smaller resolution. Instead, we
discuss how varying the resampling size could effect the performance in one of
the experiments.

###  4.2 Experimental Setup

We present three experiments to evaluate our proposed methods. First, we
compare the overall performance of the proposed architectures against a score-
independent baseline system PCConvNet [22] which uses only the randomly-
chunked pitch contour as input. This experiment also gives us an indication of
the effectiveness of each of the proposed methods. Second, we look at the
sensitivity of the SIConvNet and JointEmbedNet to the chunk size $N$. Finally,
we investigate the effect of varying the resolution of the input distance
matrix for the DistMatNet model. The latter two experiments were aimed at
understanding the effects of the different hyper-parameters used while
constructing the input data for each model. These helped us arrive at the best
parameters for each approach.

aaSIConvNetaa | JointEmbedNet |  aaDistMatNetaa
---|---|---
3,089 | 6,144 | 63,417
Table 2: Number of parameters for each model.

The number of trainable parameters for each method is shown in Table 2.
DistMatNet has a higher number of parameters because it uses a higher-
dimensional input with a deeper architecture to capture high level information
[10].

For each method, we trained separate models to predict each assessment
criterion. Moreover, to measure the variation of each model, we trained each
model on 10 different random seeds. We used $M_{i}$ to represent the model
training on different random seed where $i=0\ldots 9$. A boxplot with median
and variation of each $M_{i}$ is shown to demonstrate the results. All the
models are trained based on the mean squared error between estimated and
ground truth ratings. All the models are trained with a stochastic gradient
descent optimizer with a 0.05 learning rate. We apply early stopping if the
validation loss does not improve for 100 epochs. The performance of all models
is measured using the coefficient of determination ($R^{2}$ score):

$R^{2}=1-\frac{\sum_{i}{}(y_{i}-\hat{y_{i}})^{2}}{\sum_{i}{}(y_{i}-\bar{y_{i}})^{2}}\,,$
(1)

where $y_{i}$ is the ground truth rating, $\hat{y_{i}}$ is the estimated
rating, and $\bar{y_{i}}$ is the average of the ground truth rating. $R^{2}$
is a common metric to evaluate the fit of a regression prediction to the
ground truth value.

##  5 Results & Discussion

---
(a) Middle School

---
(b) Symphonic Band

Figure 4: Box plots showing comparative performance (higher is better) across
different models and assessment criteria.

###  5.1 Overall Performance

Figure 4 shows the comparative performance for all models for middle school
and symphonic band. We can make the following observations (with independent
t-test results reported):

  1. (i)

We compare the performance of various models trained on different band
performances. All systems perform better (higher $R^{2}$ value) on the middle
school recordings than on the symphonic band recordings ($p<0.01$ except
JointEmbedNet for musicality). One possible explanation for this is that
symphonic band scores are usually more complicated and longer. For example,
symphonic band scores tend to be performed at high tempo with high note
density. The chunking into smaller lengths (and the downsampling of the
distance matrix) compared to the score length might lead to a less accurate
mapping to the assessment rating. An additional factor is that most performers
in the symphonic band auditions exhibit greater skill level than middle school
performers thus making it potentially more difficult to model the differences
in proficiency levels.

  2. (ii)

All score-informed models generally outperform the baseline, implying that
score information is indeed helpful for MPA ($p<0.01$ except SIConvNet for
musicality, DistMatNet for musicality and note accuracy on middle school). We
notice, however, that the difference between the score-independent baseline
and the score-informed models is smaller for the middle school than for
symphonic band. Given the significant improvement over the baseline for
symphonic band performances (which have complicated scores), we speculate that
the score-informed models benefit more from access to score information. In
other words, a score reference becomes more impactful with increasing
proficiency level while the pitch contour alone contains most relevant
information for medium proficiency levels.

  3. (iii)

While the two models SIConvNet and JointEmbedNet both use the same input
features, JointEmbedNet either outperforms or matches SIConvNet in all
experiments. The main difference between these two architectures is that
SIConvNet simply performs a regression to estimate the assessments while
JointEmbedNet learns a similarity in the embedding space to model the
assessments. Therefore, we can assume that JointEmbedNet is able to explicitly
model the differences between the input pitch contour and score especially in
the case of symphonic band where the scores are more complicated.

  4. (iv)

We observe that while DistMatNet and JointEmbedNet both utilize the similarity
between the score and pitch contour, albeit at different stages of the
network, JointEmbedNet typically performs better across categories and bands,
and the gap is larger for musicality than for the other two categories. It is
possible that the absolute pitch at the input may be important for the final
assessment (octave jumps, for example, would not be properly modeled in the
distance matrix). More likely, however, is that the significantly larger input
dimensionality of the matrix (compared to the aligned sequences for
JointEmbedNet) negatively impacts performance. Most of the relevant
information for MPA centers around the diagonal of the distance matrix with
relatively small deviations depending on the students’ tempo variation. Most
of the distance matrix elements far from the diagonal contain redundant or
irrelevant information, thus complicating the task. Another advantage that
JointEmbedNet might have over DistMatNet in terms of overall assessment is
that the distance is computed on the whole performance while DistMatNet
computes a frame-level pitch distance, potentially complicating the task for
overall quality measures like musicality.

---
(a) Middle School

---
(b) Symphonic Band

Figure 5: Box plots showing comparative performance (higher is better) across
different chunk sizes for SIConvNet and JointEmbedNet.

###  5.2 Chunk Size

In this experiment, we look at the impact of two different chunk sizes for the
first two methods. Figure 5 shows the results on middle school (a) and
symphonic band (b). For both SIConvNet and JointEmbedNet, a chunk size of 10 s
outperforms that of 5 s across all the bands.

Chunking with random sampling is a form of data augmentation. By using the
ground truth rating of the whole performance, the chunks are assumed to
reflect the quality of the whole performance. The results show that 5 s chunks
might be too short to evaluate the whole performance while 10 s chunks are
much better suited regardless of category and score complexity. Chunk lengths
greater than 10 s were not tested because we restricted ourselves to the
length of the shortest performance in the dataset. Consequently, we used a 10
s chunk size for the experiment in Figure 4.

---
(a) Middle School

---
(b) Symphonic Band

Figure 6: Box plots showing comparative performance (higher is better) across
different matrix sizes for the distance matrix network (DistMatNet).

###  5.3 Distance Matrix Resolution

In this experiment, we study the impact of the different input matrix
resolutions $400\times 400$, $600\times 600$, and $900\times 900$, for the
DistMatNet model. The results for both middle school and symphonic band are
shown in Figure 6. First, the performance of rhythmic accuracy criterion tends
to decrease with increasing distance matrix resolution. It might be more
difficult for the same model structure to capture the complexity inside a
larger matrix. This can also explain the result for middle school: although
increasing the input resolution from $400\times 400$ to $600\times 600$ will
capture more details, the performance decreases when the matrix resolution is
further increased. Second, an input matrix size of $600\times 600$ leads to a
slightly higher average score (0.46) on both symphonic and middle school than
the other two resolutions (0.45 for $400\times 400$ and 0.43 for $900\times
900$). We ended up using the $600\times 600$ resolution for the experiment in
Figure 4.

##  6 Conclusion

This paper presents three novel neural network-based methods that combine
score information with a pitch representation of an audio recording to assess
a music performance. The methods include:  (i) a CNN with aligned pitch
contour and score as the input,  (ii) a joint embedding model that learns the
assessment as the cosine similarity of the embeddings of both the aligned
pitch contour and the score, and  (iii) a distance-matrix based CNN, using a
differential representation of pitch contour and score at the input.  The
results show that all the methods outperform the score-independent baseline
model. The joint embedding model achieves the highest average performance.

Beyond the obvious applications in software-based music tutoring systems,
score-informed performance assessment models (and objective MPA in general)
can benefit the broader area of music performance analysis. Models capable of
rating performances along different criteria could serve as useful tools for
objective evaluation of generative systems of music performance. In addition,
such models could also be explored for objective analysis of inter-annotator
differences in rating music performances.

In the future, we plan to incorporate timbre and dynamics information into the
models as it has been shown to improve accuracy [22]. This will also enable
the model to assess performances in terms of tone quality, the criterion
ignored in this study. We also plan to investigate other instruments and to
examine cross-instrument relationships by training instrument-specific models.
Furthermore, the musical score reference could be replaced with other
representations such as the pitch contour of a highly-rated performance.

##  7 Acknowledgment

We would like to thank the Florida Bandmasters Association for providing the
dataset used in this study. We also gratefully acknowledge Microsoft Azure who
supported this research by providing computing resources via the Microsoft
Azure Sponsorship.

## References

  * [1] B. Bozkurt, O. Baysal, and D. Yuret.  A dataset and baseline system for singing voice assessment.  In Proc. of International Symposium on Computer Music Multidisciplinary Research (CMMR), pages 430–438, Matosinhos, Porto, 2017.
  * [2] Alice Cohen-Hadria and Geoffroy Peeters.  Music structure boundaries estimation using multiple self-similarity matrices as input depth of convolutional neural networks.  In Proc. of Audio Engineeing Society (AES) International Conference on Semantic Audio, Erlangen, Germany, 2017.
  * [3] Johanna Devaney, Michael I Mandel, and Ichiro Fujinaga.  A Study of Intonation in Three-Part Singing using the Automatic Music Performance Analysis and Comparison Toolkit (AMPACT).  In Proc. of 13th International Society of Music Information Retrieval Conference (ISMIR), Porto, Portugal, 2012.
  * [4] Sebastian Ewert and Mark B Sandler.  Structured dropout for weak label and multi-instance learning and its application to score-informed source separation.  In Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2277–2281, New Orleans, USA, 2017.
  * [5] Sebastian Ewert, Siying Wang, Meinard Müller, and Mark Sandler.  Score-informed identification of missing and extra notes in piano recordings.  In Proc. of 17th International Society for Music Information Retrieval Conference (ISMIR), New York City, USA, 2016.
  * [6] Felipe Falcao, Baris Bozkurt, Xavier Serra, Nazareno Andrade, and Ozan Baysal.  A Dataset of Rhythmic Pattern Reproductions and Baseline Automatic Assessment System.  In Proc. of 20th International Society for Music Information Retrieval Conference (ISMIR), Delft, The Netherlands, 2019\.
  * [7] Thomas Grill and Jan Schluter.  Music boundary detection using neural networks on spectrograms and self-similarity lag matrices.  In Proc. of 23rd European Signal Processing Conference (EUSIPCO), pages 1296–1300, Nice, France, 2015.
  * [8] Siddharth Gururani, Ashis Pati, and Alexander Lerch.  Analysis of objective descriptors for music performance assessment.  In Proc. of International Conference on Music Perception and Cognition (ICMPC), Montréal, Canada, 2018.
  * [9] Yoonchang Han and Kyogu Lee.  Hierarchical approach to detect common mistakes of beginner flute players.  In Proc. of 15th International Society of Music Information Retrieval Conference (ISMIR), pages 77–82, Taipei, Taiwan, 2014.
  * [10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, Las Vegas, USA, 2016.
  * [11] Jiawen Huang and Alexander Lerch.  Automatic assessment of sight-reading exercises.  In Proc. of 20th International Society for Music Information Retrieval Conference (ISMIR), Delft, The Netherlands, 2019.
  * [12] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In Proc. of 32nd International Conference on Machine pyin (ICML), pages 448–456, Lille, France, 2015.
  * [13] Trevor Knight, Finn Upham, and Ichiro Fujinaga.  The potential for automatic assessment of trumpet tone quality.  In Proc. of 12th International Society of Music Information Retrieval Conference (ISMIR), pages 573–578, Miami, USA, 2011.
  * [14] Alexander Lerch, Claire Arthur, Ashis Pati, and Siddharth Gururani.  Music performance analysis: A survey.  In Proc. of 20th International Society for Music Information Retrieval Conference (ISMIR), Delft, The Netherlands, 2019.
  * [15] Pei-Ching Li, Li Su, Yi-Hsuan Yang, Alvin WY Su, et al.  Analysis of expressive musical terms in violin using score-informed and expression-based audio features.  In Proc. of 16th International Society of Music Information Retrieval Conference (ISMIR), pages 809–815, Málaga, Spain, 2015.
  * [16] Matthias Mauch and Simon Dixon.  pYIN: A fundamental frequency estimator using probabilistic threshold distributions.  In Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 659–663, Florence, Italy, 2014.
  * [17] Oscar Mayor, Jordi Bonada, and Alex Loscos.  Performance analysis and scoring of the singing voice.  In Proc. of Audio Engineering Society (AES) Convention, pages 1–7, London, UK, 2009.
  * [18] Marius Miron, Jordi Janer Mestres, and Emilia Gómez Gutiérrez.  Monaural score-informed source separation for classical music using convolutional neural networks.  In Proc. of 18th International Society for Music Information Retrieval Conference (ISMIR), Suzhou, China, 2017.
  * [19] Emilio Molina, Isabel Barbancho, Emilia Gómez, Ana Maria Barbancho, and Lorenzo J Tardón.  Fundamental frequency alignment vs. note-based melodic similarity for singing voice assessment.  In Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 744–748, Vancouver, Canada, 2013.
  * [20] Tomoyasu Nakano, Masataka Goto, and Yuzuru Hiraga.  An automatic singing skill evaluation method for unknown melodies using pitch interval accuracy and vibrato features.  In Proc. of International Conference on Spoken Language Processing (INTERSPEECH), pages 1706–1709, Pittsburg, USA, 2006.
  * [21] Caroline Palmer.  Music performance.  Annual review of psychology, 48(1):115–138, 1997.
  * [22] Ashis Pati, Siddharth Gururani, and Alexander Lerch.  Assessment of student music performances using deep neural networks.  Applied Sciences, 8(4):507, 2018.
  * [23] Jordi Pons Puig, Rong Gong, and Xavier Serra.  Score-informed syllable segmentation for a cappella singing voice with convolutional neural networks.  In Proc. of 18th International Society for Music Information Retrieval Conference (ISMIR), Suzhou, China, 2017.
  * [24] Oriol Romani Picas, Hector Parra Rodriguez, Dara Dabiri, Hiroshi Tokuda, Wataru Hariya, Koji Oishi, and Xavier Serra.  A real-time system for measuring sound goodness in instrumental sounds.  In Proc. of Audio Engineering Society Convention, Philadelphia, USA, 2015.
  * [25] Hiroaki Sakoe and Seibi Chiba.  Dynamic Programming Algorithm Optimization for Spoken Word Recognition.  Transactions on Acoustics, Speech, and Signal Processing, 26(1):43–49, 1978.
  * [26] Sam Thompson and Aaron Williamon.  Evaluating evaluation: Musical performance assessment as a research tool.  Music Perception: An Interdisciplinary Journal, 21(1):21–41, 2003\.
  * [27] Amruta Vidwans, Siddharth Gururani, Chih-Wei Wu, Vinod Subramanian, Rupak Vignesh Swaminathan, and Alexander Lerch.  Objective descriptors for the assessment of student music performances.  In Proc. of Audio Engineeing Society (AES) International Conference on Semantic Audio, Erlangen, Germany, 2017.
  * [28] I-Chieh Wei, Chih-Wei Wu, and Li Su.  Generating strcutured drum pattern using variational autoencoder and self-similarity matrix.  In Proc. of 20th International Society for Music Information Retrieval Conference (ISMIR), Delft, The Netherlands, 2019.
  * [29] Brian C Wesolowski, Stefanie A Wind, and George Engelhard.  Examining rater precision in music performance assessment: An analysis of rating scale structure using the multifaceted rasch partial credit model.  Music Perception: An Interdisciplinary Journal, 33(5):662–678, 2016\.
  * [30] Chih-Wei Wu, Siddharth Gururani, Christopher Laguna, Ashis Pati, Amruta Vidwans, and Alexander Lerch.  Towards the objective assessment of music performances.  In Proc. of International Conference on Music Perception and Cognition (ICMPC), pages 99–103, San Francisco, USA, 2016.
  * [31] Chih-Wei Wu and Alexander Lerch.  Learned features for the assessment of percussive music performances.  In Proc. of International Conference on Semantic Computing (ICSC), Laguna Hills, California, USA, 2018.