Onsets and Velocities: Affordable Real-Time Piano Transcription Using
  Convolutional Neural Networks
Polyphonic Piano Transcription has recently experienced substantial progress,
driven by the use of sophisticated Deep Learning approaches and the
introduction of new subtasks such as note onset, offset, velocity and pedal
detection. This progress was coupled with an increased complexity and size of
the proposed models, typically relying on non-realtime components and
high-resolution data. In this work we focus on onset and velocity detection,
showing that a substantially smaller and simpler convolutional approach, using
lower temporal resolution (24ms), is still competitive: our proposed
ONSETS&VELOCITIES model achieves state-of-the-art performance on the MAESTRO
dataset for onset detection (F1=96.78%) and sets a good novel baseline for
onset+velocity (F1=94.50%), while having ~3.1M parameters and maintaining
real-time capabilities on modest commodity hardware. We provide open-source
code to reproduce our results and a real-time demo with a pretrained model.
Andres Fernandez
# Onsets and Velocities: Affordable Real-Time Piano Transcription Using Convolutional Neural Networks

###### Abstract

Polyphonic Piano Transcription has recently experienced substantial progress, driven by the use of sophisticated Deep Learning approaches and the introduction of new subtasks such as note onset, offset, velocity and pedal detection. This progress was coupled with an increased complexity and size of the proposed models, typically relying on non-realtime components and high-resolution data. In this work we focus on _onset_ and _velocity_ detection, showing that a substantially smaller and simpler convolutional approach, using lower temporal resolution (24ms), is still competitive: our proposed Onsets&Velocities (O&V) model achieves state-of-the-art performance on the MAESTRO dataset for onset detection (F=96.78%) and sets a good novel baseline for onset+velocity (F\({}_{1}\)=94.50%), while having \(\sim\)3.1M parameters and maintaining real-time capabilities on modest commodity hardware. We provide open-source code to reproduce our results and a real-time demo with a pretrained model 1.

 deep learning, polyphonic piano transcription

Footnote 1: Setup [https://github.com/andres-fr/aimusica_training](https://github.com/andres-fr/aimusica_training)

## I Introduction

### _Polyphonic Piano Transcription_

The task of _Polyphonic Piano Transcription_ (PPT) is useful for downstream tasks like musical analysis and resynthesis. Consider an audio _waveform_\(x(t)\in\mathbb{R}^{T}\) with time \(T\) that corresponds to a piano performance of a _score_\(\mathcal{S}\); then the task of PPT is to recover \(\mathcal{S}\) from \(x\). Here, \(\mathcal{S}\) is a collection of \(N\)_note events_\(\{\mathcal{N}_{n}:=(k_{n},v_{n},\perp_{n},\uparrow_{n})\}_{n=1}^{N}\), where \(k\in\{1,\ldots,K\}\) specifies the _key_ (typically \(K=88\)). The value \(v\in[0,1]\) indicates the intensity of the event (also called key _velocity_). The key _onset_ (pressing) and _offset_ (releasing) timestamps are specified by \(\downarrow\) and \(\uparrow\), respectively, where \(0\leq\downarrow_{n}<\uparrow_{n}\leq T\)\(\forall n\).

There has been extensive effort in automating PPT, typically articulated through challenges like the popular _Music Information Retrieval Evaluation eXchange_ (MIREX) [1] and featuring different techniques like handcrafted features, spectrogram factorization, probabilistic models [2, 3] and, more recently, _Deep Learning_ (DL) [4]. PPT is typically evaluated by comparing the recovered score \(\mathcal{S}\) with the ground truth \(\mathcal{S}\) on a test set, in an event-wise manner. Prominent efforts in curating datasets like MAPS [5, 6], SMD [7] and MusicNet [8] were affected by imprecise annotations, insufficient training data, unrealistic interpretations and/or constrained recording conditions, which made evaluation more difficult and impeded the establishment of a unified benchmark for PPT [9]. The introduction of the MAESTRO dataset [9] addressed many of these issues, by providing \(\sim\)200 hours of precisely annotated, high-quality audio data, encompassing a large variety of virtuositic compositions, pianists and recording conditions, and incorporating evaluation splits. As a result, it quickly became a popular benchmark. Still, all pianos in MAESTRO are fairly similar: to capture more general settings and satisfy the ever-growing demand for training data, [10] curated the GiantMIDI dataset, by sourcing over 1000 hours of piano music from YouTube and annotating them using DL [11].

### _The State of the Art in PPT_

One influential effort applying DL to PPT was [12]. Their work cemented the following main trends:

SpectrogramsDespite promising efforts to use the \(x(t)\) waveforms as DL inputs [13], time-frequency representa

Fig. 1: Log-mel spctrogram (\(X\)) of a virtuosistic excerpt from the MAESTRO test set, followed by the corresponding velocity (\(\mathcal{\hat{R}}_{V}\)) and last-stage onset (\(\mathcal{\hat{R}}_{1}^{(3)}\)) predictions, as well as the ground truth _piano roll_\(\mathds{1}_{{}_{1}_{3}}\) (see Section II for details).
tions like the spectrogram [14, ch.19] remain competitive for PPT and discriminative audio tasks in general [11, 15].

Piano roll supervisionConsider an alternative representation of \(\mathcal{S}\), called _piano roll_\(\mathcal{R}\in[0,1]^{K\times T}\) (see Figure 1), where entries \(\mathcal{R}(k,t)\) encode the activity of channel \(k\) at time \(t\) (zero if inactive). This type of supervision consists in training the model to output a piano roll \(\mathcal{R}\) that predicts some ground truth \(\mathcal{R}\), by minimizing the binary cross-entropy loss: \(l_{BCE}(\mathcal{R},\mathcal{\hat{R}})\!=\!\langle\mathcal{R},-log(\mathcal{ \hat{R}})\rangle\!+\!\langle(1\!-\!\mathcal{R}),-log(1\!-\!\mathcal{\hat{R}})\rangle\). Often, the ground truth is binarized and we have \(\mathds{1}\in\{0,1\}^{K\times T}\) instead of \(\mathcal{R}\). This approach requires to _decode_ the predicted piano roll \(\mathcal{\hat{R}}\) to obtain the event-based representation \(\mathcal{\hat{S}}=dec_{H}(\mathcal{\hat{R}})\), typically by using a heuristic \(H\), e.g. grouping consecutive active frames into singe notes.

Computer VisionPPT can be tackled effectively by treating spectrograms and piano rolls as images, and models like Convolutional Neural Networks (CNNs) work well with minor adaptions.

A major turning point was Onsets&Frames (O&F) [16], which uses a sub-network to first predict a piano roll \(\mathcal{\hat{R}}_{\downarrow}\) encoding the probability of an _onset_ (trained with a mask \(\mathds{1}_{\downarrow}\) that is active only when a key is pressed), and then uses another subnetwork to predict the _frames_\(\mathcal{\hat{R}}_{\mathcal{N}}\) conditioned on \(\mathcal{\hat{R}}_{\downarrow}\) (trained with a mask \(\mathds{1}_{\mathcal{N}}\), active for the whole duration of each note). O&F is then trained jointly via a multi-task loss \(l_{BCE}(\mathds{1}_{\downarrow},\mathcal{\hat{R}}_{\downarrow})+l_{BCE}( \mathds{1}_{\mathcal{N}},\mathcal{\hat{R}}_{\mathcal{N}})\). O&F achieved a steep improvement in all PPT benchmarks, and also introduced a novel subtask, _note velocity_, modelled with a third sub-network that predicts a velocity piano roll \(\mathcal{\hat{R}}_{V}\) trained via masked \(\ell_{2}\)-norm loss \(l_{V}=\langle\mathds{1}_{\downarrow},(\mathcal{R}_{V}-\mathcal{\hat{R}}_{V})^{ 2}\rangle\). Due to its unprecedented effectiveness and versatility, O&F became a popular baseline [17, 18], but this was at the expense of increased complexity, including more elaborate decoder heuristics, a larger model, and the incorporation of bi-RNN layers [19, 20], which preclude real-time applications.

More recently, [11] pointed out issues with temporal precision on piano rolls, incorporating a trainable Regression model to enhance precision. They further expanded the model including sustain pedal detection capabilities. In [21], an _off-the-shelf_ Transformer[22] setup was used to produce \(\mathcal{\hat{S}}\) directly from spectrograms in an end-to-end fashion. Apart from their good performance, both systems have in common their substantial size, increased time resolution and replacing decoder heuristics with an end-to-end differentiable solution, suggesting that decoding is a performance bottleneck.

### _Proposed Contribution for PPT_

These state-of-the-art improvements in performance came entangled with increased complexity in the form of larger models, additional components and new sub-tasks [18]. Understanding and disentangling this complexity is an active field of research: Alternative PPT sub-task factorizations that do not rely on O&F were proposed, like nonlinear denoising vs. linear demixing [23], sound source vs. note arrangement [24] and ADSR envelopes [25]. General approaches like using invertible neural networks [26], reconstruction tasks [27] and additive attention [18] were also explored.

In this work, we pursue the orthogonal goal of achieving real-time capabilities. For that, we observe that the masked loss \(l_{V}\) imposes time-locality around the onsets, and follow up on several ideas: the importance of the onsets [16] as well as decoder heuristics [18], and the idea that note velocity is naturally associated with the onset [11]. We propose that _a convolutional end-to-end method for onsets and velocities leads to efficiency gains and affordable real-time capabilities without compromising performance_, and that _efficient decoding heuristics replace the need for high temporal resolution and complex inference schemes_.

We present Onsets&Velocities (O&V), featuring:
1. State-of-the-art performance for _onset_ detection and a good baseline for _onsets_+_velocities_ on MAESTRO.
2. A substantially reduced CNN (no recurrent layers) based on piano rolls at 24ms resolution, enabling affordable real-time inference on modest hardware.
3. A straightforward decoding mechanism, enabling a multi-task training scheme without any data augmentations or extensions.

In Section II we present our O&V method. Section III presents experiments substantiating our claims. We also provide a PyTorch [28] open-source implementation with a real-time demo. Section IV concludes and proposes future work.

## II Methodology

### _Model_

Given a waveform \(x(t)\in\mathbb{R}^{T}\) at 16kHz, we compute its Short-Term Fourier Transform (STFT) [14] with a Hann window of size 2048, and a hop size \(\delta\)=384 (i.e. a time resolution of \(\Delta_{t}\)=24ms). We then map it to 229 mel-frequency bins [29] in the 50Hz-8000Hz range, and take the logarithm, yielding our input representation: a _log-mel spectrogram_\(X(f,t^{\prime})\in\mathbb{R}^{229\times T^{\prime}}\), where \(T^{\prime}=\frac{T}{\delta}\) is the resulting "compact" time domain (see Figure 1). We also compute the first time-derivative \(\hat{X}(f,t^{\prime}):=X(f,t^{\prime})-X(f,t^{\prime}-1)\) and concatenate it to \(X\), forming the CNN input. Using the same \(\Delta_{t}\), we time-quantize the MIDI annotations into a piano roll \(\mathcal{R}_{V}\in[0,1]^{88\times T^{\prime}}\), where \(\mathcal{R}_{V}(k_{n},t^{\prime}_{n})\) contains the velocity if key \(k_{n}\) was pressed at time \(\Delta_{t}t^{\prime}_{n}\pm\frac{\Delta t}{2}\), and zero otherwise. We further binarize \(\mathcal{R}_{V}\), yielding \(\mathds{1}_{\downarrow}\).

The complete CNN is presented in Figure 2. We highlight the following design principles:

No recurrent layersMotivated by [30, 31], we follow the established CNN design of convolutional stem and body, followed by a fully connected head, making use of residual bottlenecks [32].

No poolingMotivated by [33], all residual bottlenecks maintain activation shape, and conversion from input to output shape is done in a single depthwise convolution layer [34], shown to be efficient and effective [35]. Note that the convolutions in the input domain (spectrogram) have vertical
dimensions but the convolutions in the output domain (piano roll) do not, since we assume that neighbouring frequencies are related but neighbouring piano keys aren't.

Multi-stageInspired by OpenPose [36], O&V features a series of residual stages that sequentially refine and produce the output. This is useful for real-time applications, since stages can be easily removed without need for retraining.

Temporal contextAt its core, O&V features the Context-Aware Module (CAM) [37], which is a residual bottleneck that combines time-dilated convolutions and channel attention [38]. Inspired by TCNs [39] and Inception [40], we aim to capture the temporal vicinity of an onset efficiently.

Model regularizersAt the input and before each output, O&V features _Sub-Spectral Batch Normalization_ (SBN), i.e. one individual BN per vertical dimension [41, 42]. We add dropout [43, 44] after the parameter-heavy layers. We use leaky ReLUs [45, 46] as nonlinearities.

Time localityThe time-derivative \(\hat{X}\) is a handcrafted input feature that directly represents intensity variations.

During inference, O&V produces one piano roll per onset stage \((\hat{\mathcal{R}}_{1}^{(1)},\hat{\mathcal{R}}_{1}^{(2)},\hat{\mathcal{R}}_{ 1}^{(3)})\) and one velocity piano roll \(\hat{\mathcal{R}}_{V}\) (see Figure 2(d)). Then, our proposed decoder \(\hat{\mathcal{S}}:=dec_{\sigma,\rho,\hat{\mathcal{R}}}(\hat{\mathcal{R}}_{1}^ {(3)},\hat{\mathcal{R}}_{V})\) follows a simple heuristic: temporal Gaussian smoothing (_smooth_) with variance \(\sigma^{2}\) followed by non-maximum suppression (_mms_), thresholding \(\rho\) and shifting \(\mu\), yielding the predicted score \(\hat{\mathcal{S}}\) with note onsets and velocities:

\[\begin{split}\hat{\downarrow}&:=\{(k,t^{\prime}):~{ }nms\big{(}smooth_{\sigma}(\hat{\mathcal{R}}_{1}^{(3)}))(k,t^{\prime})\geq\rho \}\\ \hat{\mathcal{S}}&:=\{(k_{n},~{}\hat{\mathcal{R}}_{V }(k_{n},t^{\prime}_{n}),~{}\Delta_{t}t^{\prime}_{n}+\mu):~{}(k_{n},t^{\prime}_ {n})\in\hat{\downarrow}_{\rho}\}\end{split} \tag{1}\]

The _mms_ operation consists in zeroing out any \((k,t)\) entry that is strictly smaller than \((k,t+1)\) or \((k,t-1)\). The note events are read at the resulting locations, and shifted by a global constant \(\mu\). In this work, we use the values \(\sigma\!=\!1,\mu\!=\!-0.01s,\rho\!=\!0.74\), obtained via cross-validation of the trained CNN on a subset of the MAESTRO validation split (note that this is different from the test split used for evaluation). While the optimal \(\rho\) fluctuates during training, we found \(\sigma\!=\!1,~{}\mu\!=\!-0.01s\) to be stable.

### _Training_

We train our CNN to predict onset probability and velocity jointly via minimization of the following multi-task loss:

\[\begin{split}& l_{1V}(\mathds{1}_{13},(\hat{\mathcal{R}}_{1}^{(1)}, \hat{\mathcal{R}}_{1}^{(2)},\hat{\mathcal{R}}_{1}^{(3)}),\mathcal{R}_{V_{3}},\hat{\mathcal{R}}_{V}):=\\ &\quad\sum_{i}^{3}l_{BCE}(\mathds{1}_{13},\hat{\mathcal{R}}_{1}^ {(i)})+\lambda\cdot l_{V^{\prime}}(\mathcal{R}_{V_{3}},\hat{\mathcal{R}}_{V}) \text{, where}\\ & l_{V^{\prime}}(\mathcal{R}_{V_{3}},\hat{\mathcal{R}}_{V}):=\\ &\quad\big{\langle}\mathds{1}_{13},~{}\big{(}\mathcal{R}_{V_{3}} \!\cdot\!-\!log(\hat{\mathcal{R}}_{V})\big{)}\big{(}(1\!-\!\mathcal{R}_{V_{3}} )\cdot\!-\!log(1\!-\!\hat{\mathcal{R}}_{V})\big{)}\big{\rangle}\end{split}\]

The \(\mathds{1}_{13}\) and \(\mathcal{R}_{V_{3}}\) rolls are a straightforward modification of \(\mathds{1}_{1}\) and \(\mathcal{R}_{V}\), where each active frame at \((k,t)\) is also extended into \(t+1\) and \(t+2\) (i.e. note onsets span 3 frames instead of one). This simple extension was crucial to achieve target performance, and combined with our decoder, allowed

Fig. 2: Our proposed CNN. Rank-4 tensor dimensions are Batch\(\times\)Channel\(\times\)Height\(\times\)Width. Design principles, interfaces and loss functions are described in Section II.
to bypass the need for elaborate decoding schemes as the ones discussed in [11]. The masked loss \(l_{V^{\prime}}\) is a cross-entropy variant of the previously mentioned \(l_{V}\), introduced in [16] and [11], that encourages to predict the right velocity only in the vicinity of onsets.

All model weights are initialized with the Gaussian-He distribution [47], and biases with 0, except the CAM channel attention biases (right before the sigmoid), initialized with 1 to promote signal flow. We use the Adam optimizer with a decoupled weight decay [48, 49] of 3\(\times\)10\({}^{-4}\), trained with random batches of 5-second segments (batch size 40, \(\sim\)14k batches per epoch) for \(\sim\)70K batches. For the learning rate, we start with a ramp-up from 0 to 0.008 across 500 batches, followed by cosine annealing with warm restarts [50], using cycles of 1000 batches and decaying by 97.5% after each cycle. BN/SBN momentum is 95%, dropout 15% and leaky ReLUs have a slope of 0.1. In \(l^{\prime}_{V}\), we use \(\lambda\!=\!10\). To compensate that \(\mathds{1}_{3\downarrow}\) is sparse, we give positive entries a weight 8 times bigger than negative entries inside of \(l_{BCE}(\mathds{1}_{3\downarrow},\cdot)\). Training speed was 1800 batches per hour on a 2080Ti NVIDIA GPU.

### _Evaluation_

Following the same evaluation procedure as O&F [16], Regression[11] and Transformer[21], and applying standard metrics from [51] implemented in the mir_eval library [52], we report precision (P), recall (R) and F\({}_{1}\)-score for the predicted _onsets_, considered correct if they are within 50ms of the ground truth. The _onset+velocity_ evaluation, following O&F [16, 3.1], has an added constraint: the predicted velocity must also be within 0.1 of the ground truth normalized between 0 and 1.

Note that the MAESTRO dataset is being actively extended and curated, presenting 3 versions so far. We report the respective versions in Table I, noting that versions 2 and 3 are almost identical, although comparisons across versions should be taken approximately.

## III Experiments and Discussion

We trained O&V on the MAESTRO v3 training split without any extensions or augmentations, achieving state-of-the-art performance in onset detection (see Table I). In the following we discuss some implications:

Temporal resolutionOur results seem to counter the need for increased temporal resolution expressed in [11] (which use 8ms), showing that 24ms piano rolls coupled with our decoder presented in Equation (1) are competitive.

Reduced memory footprintTable I reports model parameters for the components that are responsible exclusively for onset and frame detection (Transformer has \(\sim\)54M parameters in total, but it transcribes everything jointly so it cannot be fairly compared). O&V outperforms the best alternative, Regression, with \(\sim\)4 times less parameters.

Affordable real-time inferenceBi-recurrent layers like the ones used in O&F and Regression are unsuited for real-time processing. Transformer took \(\sim\)380s to transcribe a 120s file on an Intel Xeon CPU (1 core), and \(\sim\)20s on a Tesla-T4 GPU (including offsets) when run on the official Colab implementation2. O&V took less than 2s to process the same file on an 8-core Intel i7-11800H CPU. Even accounting for the number of cores, O&V is approximately one order of magnitude faster than Transformer.

Footnote 2: [https://github.com/magenta/mt3](https://github.com/magenta/mt3)

Conceptual simplicityIn essence, O&V revives the simplicity from [12] by applying a feedforward CNN to a discriminative task via piano rolls, followed by a simple decoding heuristic. Its architecture allows to remove onset stages without retraining, providing a flexible trade-off between runtime and performance with little added complexity.

LatencyThe receptive field for our O&V proposed components is: Stem: 60 frames (1.44s), Stage\({}_{1}\)\({}^{(i)}\): 99 frames (\(\sim\)2.38s), and Stage\({}_{\text{v}}\): 35 frames (0.84s). This would theoretically impose a latency of over 9s, which is far from a responsive system. We informally note that the latency can be truncated without drastically affecting results (we used a latency of 4s in a live workshop), and encourage practical applications. We also note that our focus was on finding a CNN with affordable inference and competitive performance, and we did not optimize for low receptive field, which may be obtainable with minor variations to the architecture (e.g. reducing the number of consecutive stages or CAMs).

## IV Conclusion and Future Work

We presented Onsets&Velocities, a convolutional setup that achieves real-time capabilities on modest commodity hardware without compromising performance, and with a substantially reduced size and temporal resolution. O&V achieves state-of-the-art performance in PPT note _onset_ detection, and establishes a good baseline on _onset+velocity_ detection. Future work could include reducing the receptive field, extensions to _offset_ and _pedal_ detection, training and evaluation on different instruments, and analysis of design choices via ablation studies (e.g. number of stages and temporal resolution).

\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Onset+Velocity**} & \multirow{2}{*}{**Architecture**} & \multirow{2}{*}{**Offset/pedal**} & \multicolumn{3}{c}{**Onset (\%)**} & \multicolumn{3}{c}{**Onset+Velocity (\%)**} & \multirow{2}{*}{**MAESTRO**} \\  & & & & P & R & F\({}_{1}\) & P & R & F\({}_{1}\) & **version** \\ \hline O\&F [16] & 10M & bi-RNN & ✓ / ✗ & 98.27 & 92.61 & 95.32 & - & - & - & v1 \\ Regression[11] & 12M & bi-RNN & ✓ / ✓ & 98.17 & 95.35 & 96.72 & - & - & - & v2 \\ Transformer[21] & – & Transformer & ✓ / ✗ & - & 96.13 & - & - & - & v3 \\ O\&V (ours) & **3.13M** & CNN & ✗ / ✗ & 98.58 & 95.07 & **96.78** & 96.25 & 92.86 & 94.50 & v3 \\ \hline \hline \end{tabular}
\end{table} TABLE I: Comparison of top-performing models in terms of specifications (number of parameters for onset+velocity only, architecture and functionality) and performance (precision, recall, F\({}_{1}\)-score and MAESTRO version).
## Acknowledgments

A.F. wants to thank Jesus Monge Alvarez and Christian J. Steinmetz for their valuable feedback, the _Institut d'Estudis Balearics_ for supporting this work with research grant 389062 INV-23/2021, and the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for further support.
