Pianist Identification Using Convolutional Neural Networks
This paper presents a comprehensive study of automatic performer
identification in expressive piano performances using convolutional neural
networks (CNNs) and expressive features. Our work addresses the challenging
multi-class classification task of identifying virtuoso pianists, which has
substantial implications for building dynamic musical instruments with
intelligence and smart musical systems. Incorporating recent advancements, we
leveraged large-scale expressive piano performance datasets and deep learning
techniques. We refined the scores by expanding repetitions and ornaments for
more accurate feature extraction. We demonstrated the capability of
one-dimensional CNNs for identifying pianists based on expressive features and
analyzed the impact of the input sequence lengths and different features. The
proposed model outperforms the baseline, achieving 85.3% accuracy in a 6-way
identification task. Our refined dataset proved more apt for training a robust
pianist identifier, making a substantial contribution to the field of automatic
performer identification. Our codes have been released at
https://github.com/BetsyTang/PID-CNN.
Jingjing Tang, Geraint Wiggins, Gyorgy Fazekas
# Pianist Identification Using Convolutional Neural Networks

###### Abstract

This paper presents a comprehensive study of automatic performer identification in expressive piano performances using convolutional neural networks (CNNs) and expressive features. Our work addresses the challenging multi-class classification task of identifying virtuoso pianists, which has substantial implications for building dynamic musical instruments with intelligence and smart musical systems. Incorporating recent advancements, we leveraged large-scale expressive piano performance datasets and deep learning techniques. We refined the scores by expanding repetitions and ornaments for more accurate feature extraction. We demonstrated the capability of one-dimensional CNNs for identifying pianists based on expressive features and analyzed the impact of the input sequence lengths and different features. The proposed model outperforms the baseline, achieving 85.3% accuracy in a 6-way identification task. Our refined dataset proved more apt for training a robust pianist identifier, making a substantial contribution to the field of automatic performer identification. Our codes have been released at [https://github.com/BetsyTang/PID-CNN](https://github.com/BetsyTang/PID-CNN).

 performer identification, expressive piano performance, deep neural networks

## I Introduction

Performers, with their individual phrasing, dynamics, and interpretive choices, bring their personal aristry to each piece they play, resulting in distinguishable styles. Researchers who focus on studying expressive musical performances have been investigating computational models for performer identification [1, 2, 3, 4, 5]. A reliable pianist identifier holds great potential for not only studying the styles of different performers, but also various applications in music education, music information retrieval and smart musical instruments [6]. As an illustration, a pianist identification model could aid piano students wishing to emulate the performances of virtuoso pianists. With an upsurge in embedded devices, the vision of smart musical systems--ones that can discern different performers or styles and provide real-time feedback or adjustments--becomes closer to reality. Imagine a smart piano capable of tailoring its settings to mirror the nuances of iconic pianists, or a wearable accessory that offers pianists instant feedback, juxtaposing their performance against the masterpieces of legendary artists. Networked musical instruments could use style information or the features extracted by the proposed system in educational, retrieval or networked performance contexts, similar to those proposed by Turchet et.al. in [7]. These groundbreaking applications will not only resonate with the principles of Internet of Musical Things (IoMusT) [8] and the Internet of Audio Things (IoAuT) [9] but also elevate their potential, transforming basic devices into dynamic musical instruments with intelligence in the context of the Internet of Sounds (IoS) [10].

Automatic performer identification is usually regarded as a multi-class classification task where the system is designed to infer the performer of the given music performance. Early studies [1, 2] mainly applied traditional machine learning algorithms such as K-means clustering, decision trees, and discriminant analysis to this task. More recent research [3, 4] calculated the KL-divergence between performers' feature distributions and identified performer by performing similarity estimation based on the KL-divergence. Zhao et al. [11] utilised transfer learning for classifying violinists, adopting pre-trained models for music tagging and singer identification. With the emergence of large-scale expressive piano performance datasets [12, 13], two projects [5, 11, 12] recently applied deep learning techniques to pianist identification task. Rafee et al. [5] proposed a RNN-based hierarchical neural network for pianist identification. Zhang et al. [12] has applied convolutional neural networks (CNNs) to a 16-way pianist identification task, achieving less than 50% accuracy. However, this work paid insufficient attention to extracting expressive features which have been proven effective for deep neural networks that model expressiveness and performance styles of pianists [5, 14].

This paper details our exploration of the potential of CNNs in identifying virtuoso pianists using various expressive features. We obtained a subset consisting of both performance and score midis from the ATEPP dataset, refining the scores by extending the repetitions and ornaments in the corresponding midis, thus generating the most comprehensive and accurate dataset currently available for pianist identification. We conducted experiments to investigate the effectiveness of different expressive features and the impact of input sequence
lengths. The proposed one-dimensional CNN surpassed the baseline model [5], attaining an 85.3% accuracy for a 6-way identification task. In addition, our dataset was shown to be more suitable for training a robust pianist identifier compared to the one proposed previously [5].

The rest of this paper is organised as follows: Section II elaborates on the methodology, providing details of the dataset, the feature extraction process, and the model architecture. Section III outlines the experiment set-ups employed for model training. Section IV discusses the experiment results and the ensuing discussions. Lastly, Section V concludes the paper.

## II Methodology

### _Dataset_

As discussed by Rafee et al. [5], the lack of large datasets containing multiple performances of the same compositions by different pianists results in the lack of investigation in deep neural networks for pianist identification. However, the recent proposed expressive piano performance midi dataset, ATEPP [12], enabled us to create subsets which are balanced in the number of performances for six virtuoso pianists including Alfred Brendel, Claudio Arrau, Daniel Barenboim, Friedrich Gulda, Sviatoslav Richter, and Wilhelm Kempf. In our research, we consider two subsets as shown in the Table I:

1. _ID-400_: we created an updated version of the proposed subset by Rafee et al. [5] by removing corrupted transcription results as well as repeated performances following the latest version of the ATEPP dataset1. Footnote 1: [https://github.com/BetsyTang/ATEPP](https://github.com/BetsyTang/ATEPP)
2. _ID-1000_: we chosen a larger subset containing more compositions and performances by the same pianists to increase robustness and verify the capability of our model.

All movements in both subsets are by Beethoven or Mozart. Each movement corresponds to at least one performance by each pianist, making it possible to compare the differences in performance style of each individual performer. In order to maintain similar data distributions in training, validation, and testing sets, we divided the datasets alongside the number of performances of a composition by each pianist. To achieve a 8:1:1 train-valid-test split, we followed the Algorithm 1 to assign performances to _Train_, _Valid_ and _Test_ subsets. The Algorithm 1 is designed to guarantee that each split contains at least one performance of the composition by a performer, especially when there are fewer than 10 performances by that performer.

```
# Let \(C\) be the set of compositions, \(P\) be the set of pianists. # Info returns composition and pianist of a performance \(i\). # Count gives the number of performances in a set \(S\). # RandomSplit randomly splits a set \(S\) of size \(n\) into subset \(a\) of size \(rn\) and subset \(b\) of size \((1-r)n\), where \(r\in[0,1)\). # Random generates a number between \([0,1)\) following the uniform distribution. # \(\leftarrow\) means "assigned to". for(\(c\), \(p\)) in (\(C\), \(P\))do \(n=Count(I)\), where if \(i\in I\), Info\((i)=(c,p)\) if\(n\leq 1\)then\(Train\gets I\) elseif\(n=2\)then\(a,b=\)RandomSplit\((I,r=1/n)\) \(m=\)Random\(()\), \(Train\gets b\) if\(m\leq 0.5\)then\(Valid\gets a\) elseif\(m>0.5\)then\(Test\gets a\) endif elseif\(3\leq n\leq 9\)then\(a,b=\)RandomSplit\((I,r=\frac{1}{n})\) \(b,c=\)RandomSplit\((b,r=\frac{1}{n-1})\) \(Valid\gets a\), \(Test\gets b\), \(Train\gets c\) elseif\(10\leq n\)then\(a,b=\)RandomSplit\((I,r=\frac{4}{5})\) \(b,c=\)RandomSplit\((b,r=\frac{1}{2})\) \(Train\gets a\), \(Valid\gets b\), \(Test\gets c\) endif endfor
```

**Algorithm 1** Data Splitting

### _Score and Performance Alignment_

Inspired by previous research [1, 3, 5] focusing on pianist identification, we used an alignment algorithm proposed by Nakamura et al. [15] to establish correspondences between performance midi data and score midi data, which allowed us to extract performance-related features. While the algorithm exhibited promising results in most cases, it demonstrated limited capability in handling annotated repetitions and ornaments found in the scores. To address this limitation, we manually expanded the repetitions and added ornament notes to the score midi files, thereby enhancing the accuracy of the alignment results. The improved alignment results more accurately captured the nuances of performances, aiding in distinguishing among performers.

After performing the alignments, we proceeded to filter out two types of discrepancies: _missing notes_ (representing notes present in the scores but not successfully aligned to performances) and _extra notes_ (representing notes present in performances but not successfully aligned to scores). Then we quantified the extent of information loss caused by the
alignment algorithm for each performance, as captured by Equation 1:

\[\textit{Loss of Information}=\frac{N_{e}}{N_{p}}\times 100\%, \tag{1}\]

where \(N_{e}\) denotes the number of extra notes and \(N_{p}\) refers to the total number of notes in the performance.

The distributions of information loss in the datasets _ID-400_ and _ID-1000_ are presented in Fig. 1. Our analysis reveals that more than 95% of performances in both datasets exhibit less than 15% information loss.

### _Feature Extraction_

After aligning the performances and scores, we extracted input features following the process outlined in the study by Rafee et al. [5]. We derived deviations between the scores and performance for note-wise features, encompassing aspects such as timing and velocity. Beyond considering feature deviations, we also incorporated the original note-wise features as part of our input data. A full list of features used for our experiments are summarised in the Table II-C. Two note-level features are defined as follows: _Inter-onset Interval_ (IOI), representing the temporal duration between the onset times of two consecutive notes, and _Offset Time Duration_ (OTD), signifying the time interval between the offset time of a note and the onset time of its subsequent note. To process the features into suitable input for our model, we organized them into sequences, preserving the order of the notes. These sequences were then stacked together to create the final input. The resulting shape of the input would be (_batch size_, _sequence length_, _number of features_), as shown in the Fig. 2 at the left side.

To examine the performance of our model under circumstances of limited information, we divided the sequences into segments of varying lengths respectively. This allowed us to gauge the model's capacity to manage scenarios with limited data availability, detailed further in Section IV.

### _Model Architecture_

In light of the promising performance demonstrated by Convolutional Neural Networks (CNNs) in various classification tasks across different domains, we proposed a novel one-dimensional CNN model to address the pianist identification task. The architecture was determined through an empirical grid search, focusing on structural hyperparameters such as the number of layers and kernel size. The model architecture, depicted in Fig. 2, encompasses five convolutional layers followed by one dense layer, strategically designed to efficiently process the input data. All convolution layers are followed by a ReLU activation and a batch normalization layer. Dropout layers are added in order to avoid overfitting problem.

## III Experiments

We implemented our model using PyTorch [16], and monitored and recorded the experimental progress through the use of Wandb [17]. To achieve optimal model performance, we conducted an extensive hyperparameter tuning process using grid search. We specifically focused on parameters such as learning rate, weight decay, batch size, and the number of training epochs. This process was enhanced by leveraging the powerful capabilities of Wandb Sweeps. Consequently, our model underwent training with a batch size of 16 for a total of 1500 epochs, employing the Adam optimizer with an initial learning rate set to 8e-5 and a weight decay rate of 1e-7.

Our proposed model, which has only 6.1 million trainable parameters, showcasing remarkable efficiency. On average, a single experiment on a GeForce RTX 2080 Ti GPU takes

Fig. 1: Boxplots of information loss caused by the alignment process in _ID-400_ and _ID-1000_ datasets

Fig. 2: Model architecture of the proposed one-dimensional CNN
approximately 1.2 hours. This duration stands in stark contrast to the significantly lengthier training times encountered in the context of RNN-based hierarchical models, as proposed by Rafee et al. [5].

## IV Results

To thoroughly evaluate our proposed CNN model in addressing the pianist identification task, we conducted three studies. These studies examined the impacts of variable input sequence lengths, the diverse expressive features, and the datasets on the model's performance. To ensure a reliable assessment of the model, each experiment was repeated three to five times under consistent experimental settings. For a more straightforward comparison with the state-of-the-art [5], both Study I and II were conducted using the _ID-400_ dataset.

### _Study I: Effect of Varying Input Music Sequence Lengths_

The reliable identification of a pianist necessitates stable performance regardless of variations in the length of the musical input. We embarked on a series of experiments using all the features delineated in Section II-C to train our model. Experiments were conducted on complete musical pieces and segments of varying lengths, utilizing the _ID-400_ dataset. Mean values along with standard deviations pertaining to accuracy and F1-score for each experiment are tabulated in Table III. As inferred from the outcomes, our model demonstrated uniform high performance when dealing with sequences comprising 1000 notes or less. However, incorporating the full scope of performances substantially bolstered the model's performance as opposed to relying solely on performance segments. Furthermore, our model surpassed the benchmark set by the state-of-the-art RNN-based Hierarchical model [5] when we integrated more features into the training at both piece-wise and segment-wise levels. Our model attained a commensurate level of accuracy when trained with the same number of features as their study.

### _Study II: Effect of Different Input Features_

In order to investigate the impact of various input features, we elected five feature combinations and executed corresponding experiments on each group. These combinations are displayed in Table IV, where **D** symbolizes the usage of the deviation feature as a replacement for the original note-wise feature.

**C1** embodies 7 original note-wise features; **C2** omits the singular frequency-based feature, pitch, from **C1**; **C3** comprises only deviation features; **C4** replicates the same combination used in the study [5]; while **C5** incorporates all available features. Experiments were conducted on the _ID-400_ dataset utilizing music segments of 1000 notes. The mean accuracy from five iterations along with the standard deviation for each feature combination experiment is detailed in Table V.

The results highlight negligible differences when employing either note-wise features or deviation features in isolation for training the model. Incorporation of all the features collectively yields the optimal performance, suggesting that the model is more adept at identifying a performer's style when given the full set of related features. The comparison between **C1** and **C2** groups suggests that the frequency-based feature does not make a significant contribution to the identification process. Concurrently, the outcomes provide further evidence that the combination of velocity, duration, and IOI deviations proves to be a more reliable choice when solely utilizing deviation features, as discussed in [5].

### _Study III: Comparison between ID-400 with ID-1000_

Despite the implementation of a carefully designed data splitting algorithm, the relatively small size of the subset _ID-400_ impedes the creation of training, testing, and validation sets that maintain similar data distributions. Employing the
same algorithm, we generated five varied data splits for both the _ID-400_ and _ID-1000_ datasets, each of which underwent model testing. The Table VI presents the average test accuracy across all data splits, alongside the highest accuracy achieved by the best models on both datasets. Experiments were conducted using sequences of 1000 notes and 13 features. The outcomes, as outlined in the Table VI and Fig. 3, reveal in data splits, thereby improving the robustness in identifying the six pianists.

## V Conclusion

We presented our investigation of the application of convolutional neural networks to the pianist identification task. Our proposed convolutional neural network model shows promising results in identifying virtuoso pianists. Three studies were conducted, analysing the effects of varying input sequence lengths, the utilization of diverse expressive features, and the impacts of different datasets on the model's performance. Our findings suggest that our model performs best when handling complete musical performances rather than fragments, outperforming the state-of-the-art with 85.3% accuracy when integrating a larger set of features into the training phase. Our model uses less computational resource, leading to significant time savings during the training process compared with the state-of-art. In addition, training on our proposed larger _ID-1000_ dataset resulted in a model less sensitive to alterations in data splits, thereby improving the robustness in identifying the six pianists.

Our model serves as an exemplar for embedded systems that aspire to decode and respond to nuanced musical cues. Just as voice-operated devices discern users' vocal nuances, our proposed model distinguishes pianists based on their expressive nuances. There are numerous further applications of the technology in the IoS and IoMusT contxts, including the population of music related ontologies [18, 19, 20, 21] with performer identity or style related information.

Future work could extend these findings, utilising the proposed model to develop identifiers for more pianists. Such extensions will offer a more comprehensive understanding of pianist-specific performance characteristics, and enrich the applications of the current system. It would also be beneficial to evaluate the model's generalization abilities on unseen compositions.
