# Semi-supervised Neural Chord Estimation
Based on a Variational Autoencoder
with Latent Chord Labels and Features

Yiming Wu, Tristan Carsault, Eita Nakamura, Kazuyoshi Yoshii  Manuscript
received March 18, 2020; revised XXXX XX, 2020; accepted XXXX XX, 2020; Date
of publication XXXX XX, 2020; date of current version XXXX XX, 2020. This work
was supported by JST ACCEL No. JPMJAC1602 and JSPS KAKENHI Nos. 19H04137,
19K20340, and 16H01744. The associate editor coordinating the review of this
manuscript and approving it for publication is XXX XXX. (Corresponding author:
Yiming Wu.) Y. Wu, E. Nakamura, and K. Yoshii are with Graduate School of
Informatics, Kyoto University, Kyoto 606-8501, Japan (email: {wu, enakamura,
yoshii}@sap.ist.i.kyoto-u.ac.jp). E. Nakamura is also with Hakubi Center for
Advanced Research, Kyoto University, Kyoto 606-8501, Japan. T. Carsault is
with Représentations Musicales, IRCAM, 75004 Paris, France (email:
tristan.carsault@ircam.fr). Digital Object Identifier
10.1109/TASLP.2020.XXXXXXX

###### Abstract

This paper describes a statistically-principled semi-supervised method of
automatic chord estimation (ACE) that can make effective use of music signals
regardless of the availability of chord annotations. The typical approach to
ACE is to train a deep classification model (neural chord estimator) in a
supervised manner by using only annotated music signals. In this
discriminative approach, prior knowledge about chord label sequences (model
output) has scarcely been taken into account. In contrast, we propose a
unified generative and discriminative approach in the framework of amortized
variational inference. More specifically, we formulate a deep generative model
that represents the generative process of chroma vectors (observed variables)
from discrete labels and continuous features (latent variables), which are
assumed to follow a Markov model favoring self-transitions and a standard
Gaussian distribution, respectively. Given chroma vectors as observed data,
the posterior distributions of the latent labels and features are computed
approximately by using deep classification and recognition models,
respectively. These three models form a variational autoencoder and can be
trained jointly in a semi-supervised manner. The experimental results show
that the regularization of the classification model based on the Markov prior
of chord labels and the generative model of chroma vectors improved the
performance of ACE even under the supervised condition. The semi-supervised
learning using additional non-annotated data can further improve the
performance.

###### Index Terms:

Automatic chord estimation, semi-supervised learning, variational autoencoder.

##  I Introduction

Achord is a mid-level representation of polyphonic music that lies between the
highly-abstracted musical intentions of humans and actual musical sounds.
Unlike musical notes, chord sequences do not tell the actual pitches, but
abstractly represent the harmonic content evolving over time. In lead sheets
(a form of music notation consisting of melody, chords, and lyrics), chord
labels are shown to musicians to roughly suggest the intentions of how musical
notes should be arranged and played in each bar.

Automatic chord estimation (ACE), which aims to recognize the chord sequence
behind a music signal, has been one of the fundamental research topics in the
field of music information retrieval (MIR). For example, it forms the basis of
music content visualization (e.g., Songle [1]), and higher-level MIR tasks
such as genre classification [2] and cover song retrieval [3]. The diversity
and complexity of the acoustic characteristics of music signals make the ACE
task very challenging. ACE studies have thus been focusing on data-driven
methods based on statistical models[4], which are roughly categorized into
generative and discriminative approaches.

Figure 1: The proposed variational autoencoder consisting of a deep generative
model of chroma vectors, a deep classification model of chord labels, and a
deep recognition model of latent features. Dashed arrows indicate stochastic
relations. These three models are trained jointly in a semi-supervised manner
by using annotated and non-annotated music signals.

Early studies took the generative approach based on a classical probabilistic
latent variable model. For example, a hidden Markov model (HMM) is often used
to represent the stochastic generative process of chroma vectors (observed
variables) from chord labels (latent variables) [5]. Given chroma vectors, the
chord labels are inferred such that their likelihood is maximized. An
advantage of this approach is that the prior distribution of chord labels
(language model) can be naturally introduced. To encourage the estimated chord
labels to be consistent with musical grammar, the temporal continuity and
dynamics of chord labels should be considered. Another advantage is that in
theory generative models can be trained in a semi-supervised or even
unsupervised manner. However, the expressive power of simple generative models
like HMMs have been insufficient to represent the temporal structure of chord
labels and the relationships between acoustic features and chord labels.

The discriminative approach, in contrast, aims to directly convert acoustic
features into chord labels. Recently, deep neural networks (DNNs) have often
been used for estimating the posterior probabilities of chord labels. Because
the generative process is not considered explicitly, the oversimplified
assumptions posed by the generative approach can be avoided[6], and chord
label inference is more straightforward. A problem here is that the classifier
is usually trained for learning a frame-level audio-to-label mapping without
considering the characteristics of chord label sequences and the generative
process of music. In addition, the model needs to be trained in a fully-
supervised manner. Manual chord annotation is a labor-intensive task, and the
performance of ACE heavily depends on the amount, diversity and quality of
annotated music signals[7] because of the nature of supervised learning.

Given that the performance of ACE could be improved by integrating the
complementary generative and discriminative approaches, we propose a semi-
supervised neural chord estimator based on a variational autoencoder (VAE)[8]
(Fig. 1). Because chroma vectors can be represented more precisely by
considering their continuous latent features (deviations from basic chroma
patterns) in addition to the underlying discrete chord labels, we formulate a
DNN-based generative model $p_{\theta}$ that represents the generative process
of a sequence of chroma vectors from that of chord labels and that of latent
features. To complete the Bayesian formulation, we introduce prior
distributions on the latent variables. Specifically, the chord labels are
assumed to follow a first-order Markov model favoring self transitions, and
the latent features, which abstractly represent the fine structure of the
chroma vectors, are assumed to follow a standard Gaussian distribution. In the
framework of amortized variational inference (AVI) [9], DNN-based
discriminative models $q_{\alpha}$ and $q_{\beta}$ are then introduced as
variational posterior distributions to approximate the posterior distribution
of the latent variables from an observed chroma sequence. The generative and
discriminative models can be trained jointly in a semi-supervised manner by
using music signals with and without chord annotations.

The main contribution of this paper is to draw the potential of the powerful
deep discriminative model for ACE, by integrating it into the principled
statistical inference formalism of the generative approach. This is the first
attempt to use a VAE for semi-supervised ACE. A key feature of our VAE is that
a Markov model is used to define the prior probability of chord label
sequences, which works as a chord language model and prevents frequent frame-
level transitions of chord labels in the joint training of the generative and
discriminative models for given chroma vectors. We experimentally show the
effectiveness of this regularization on semi-supervised learning.

The remainder of this paper is organized as follows. Section II reviews
related work on the generative and discriminative approaches to ACE. Section
III describes the proposed ACE method based on the semi-supervised VAE with
the Markov prior on latent chord labels. Section IV reports the experimental
results and discusses the effectiveness and limitations of the proposed
techniques. Section V summarizes the paper.

##  II Related Work

This section reviews related work on ACE based on generative and
discriminative approach, and machine-learning strategies for integrating these
approaches.

###  II-A Generative Approach

In the generative approach, a probabilistic generative model
$p(\mathbf{X},\mathbf{S})=p(\mathbf{X}|\mathbf{S})p(\mathbf{S})$ has often
been formulated as an HMM[10], where $\mathbf{X}$ and $\mathbf{S}$ denote a
sequence of acoustic features (e.g., chroma vectors [11]) and a sequence of
chord labels [12], respectively. The emission probabilities of $\mathbf{X}$
for each chord, $p(\mathbf{X}|\mathbf{S})$, and the chord transition
probabilities, $p(\mathbf{S})$, are typically given by a Gaussian mixture
model (GMM) and a first- or higher-order Markov model (n-gram model),
respectively. Given $\mathbf{X}$ as observed data, the optimal chord label
sequence maximizing the posterior probability $p(\mathbf{S}|\mathbf{X})$ can
be estimated efficiently by using the Viterbi algorithm.

Many HMM-based methods have been proposed for ACE. Lee and Slaney [5], for
example, proposed joint training of multiple HMMs corresponding to different
keys. Another research direction is to simultaneously deal with multiple kinds
of musical elements in addition to chord labels by explicitly considering the
relationships of those elements. Mauch and Dixon [13] proposed a dynamic
Bayesian network (DBN) that represents the hierarchical structure over
metrical positions, musical keys, bass notes, and chords, and formulates the
generative process of treble and bass chroma vectors. Similarly, Ni et al.[14]
proposed a harmony progression analyzer (HPA) based on a DBN whose latent
states represent chords, inversions, and musical keys.

Some studies have focused on the temporal characteristics of frame-level chord
labels. To represent repetitions of chord patterns, for example, multi-order
HMMs[15] and duration-explicit HMMs[16] have been proposed, where the results
of ACE were found to be insensitive to the frame-level transition
probabilities of chord labels [16]. Although such a frame-level language model
is incapable of learning typical chord progressions at the symbol level, it is
still effective for encouraging the continuity of chord labels at the frame
level.

###  II-B Discriminative Approach

Recently, DNNs have intensively been used as a powerful discriminative model
for directly estimating the posterior probability $p(\mathbf{S}|\mathbf{X})$.
Humphrey and Bello [17], for example, used a convolutional neural network
(CNN) for learning an effective representation of raw audio spectrograms. A
number of ACE methods using DNN classifiers have then been proposed for using
low-level audio representations such as constant-Q transform (CQT)
spectrograms[18]. In general, these DNN-based methods outperform HMM-based
generative methods[19].

Because typical DNN-based methods estimate the posterior probabilities of
chord labels at the frame level, some smoothing technique is often used for
estimating temporally-coherent chord labels. An HMM [20] or a conditional
random field (CRF) [19], for example, can be used for estimating the optimal
path of chord labels from the estimated posterior probabilities. Recurrent
neural networks (RNNs) have recently been used as a language model that
represents the long-term dependency of chord labels [21, 22]. Note that RNN-
based models are still incapable of learning the symbol-level syntactic
structure from frame-level chord sequences[23]. To solve this problem,
Korzeniowski and Widmer [24] formulated a symbol-level chord transition model
with a duration distribution. Chen and Su [25] proposed an extension of the
transformer model [26] for joint chord segmentation and labeling in a context
of multi-task learning.

Even the state-of-the-art neural chord estimators have difficulty in
identifying infrequent chord types because the numbers of occurrences of chord
types are highly imbalanced. In fact, the majority of chords is occupied by
major and minor triads. Another difficulty lies in distinguishing chords that
commonly include several pitch classes as chord notes (e.g., C9 and Csus2).
When the chord vocabulary gets larger, i.e., the chord classes are defined at
a finer level, the subjectivity of annotators leads to the inconsistency of
chord annotations [27]. This would make infrequent chord types harder to
learn. This problem can be mitigated by reflecting the musical knowledge about
the constituent tones and taxonomy of chord labels [28, 29, 30] into objective
functions and/or by using an even-chance training scheme [31]. However,
dealing with a large chord vocabulary is still an open problem.

###  II-C Unified Generative and Discriminative Approach

Integration of deep generative and discriminative models has actively been
explored in the context of unsupervised or semi-supervised learning. One of
the most popular strategies is to use a variational autoencoder (VAE) [8] that
jointly optimizes a deep generative model $p(\mathbf{X}|\mathbf{S})$ and a
deep recognition model $q(\mathbf{S}|\mathbf{X})$ that approximates
$p(\mathbf{S}|\mathbf{X})$ such that the lower bound of the marginal
likelihood $p(\mathbf{X})$ is maximized. An extension, known as conditional
VAE[32], can be used for semi-supervised learning of latent representations
disentangled from given labels (conditions) [33]. In the field of Automatic
Speech Recognition (ASR), some studies tried to jointly train a speech-to-text
model with a text-to-speech model to improve the performance of ASR by using
both annotated and non-annotated speech signals [34].

In this paper, we use a VAE with two different latent variables corresponding
to chord labels (categorical variables) and latent features (continuous
variables). This model is similar to JointVAE [35] in a sense that both
discrete and continuous representations are learned jointly. In our model, a
Markov prior favoring self transitions is put on chord labels for regularizing
the recognition model in the VAE training. In conventional studies, in
contrast, the Markov prior is used only for smoothing chord labels estimated
by a recognition model. To our knowledge, our work is the first attempt to
integrate the generative and discriminative processes between chord labels and
acoustic features into a unified jointly-trainable DNN.

##  III Proposed Method

This section describes the proposed unified generative and discriminative
approach to ACE (Fig. 2). To tackle the frame-level ACE (Section III-A), we
formulate a probabilistic model $p_{\theta}$ representing the generative
process of chroma vectors from chord labels and latent features (Section
III-B), and then introduce neural statistical estimators $q_{\alpha}$ and
$q_{\beta}$ that respectively infers chord labels and latent features from
chroma vectors (Section III-C). These three models are jointly trained in the
framework of amortized variational inference (AVI)[9]. In theory, $q_{\alpha}$
can be trained in an unsupervised manner only from chroma vectors without
referring to their chord labels (Section III-D). In practice, $q_{\alpha}$ is
trained in a supervised or semi-supervised manner by using paired data
(Sections III-E).

Figure 2: The computation flow of the unsupervised learning. The gray areas
correspond to the four terms of the cost function $\mathcal{L}_{\mathbf{X}}$
given by (10).

###  III-A Problem Specification

For simplified explanation, suppose that we have one musical piece. Let
$\mathbf{X}=\\{\mathbf{x}_{1},\cdots,\mathbf{x}_{N}\\}$ be a sequence of
chroma vectors extracted from the music signal, where
$\mathbf{x}_{n}\in\\{0,1\\}^{D}$ is a $D$-dimensional binary vector
representing the activations of $D$ pitch classes at frame $n$, and $N$ is the
number of frames. In this paper, $\mathbf{x}_{n}$ is a 3-channel chroma
vector, each channel representing the lower, middle, and higher pitch ranges
respectively ($D=12\times 3=36$), as defined in [36]. Because the true pitch
activations are not given in reality, $\mathbf{x}_{n}$ is obtained by using a
neural multi-pitch estimator [36] that estimates frame-wise pitch-class
distributions with a DNN trained from 6,000 pieces (audio signals synthesized
from MIDI files)111 https://github.com/Xiao-
Ming/ChordRecognitionMIDITrainedExtractor. We thus relax the binary constraint
and assume $\mathbf{x}_{n}\in[0,1]^{D}$.

Let $\mathbf{S}=\\{\mathbf{s}_{1},\cdots,\mathbf{s}_{N}\\}$ be a sequence of
chord labels corresponding to $\mathbf{X}$, where
$\mathbf{s}_{n}\in\\{0,1\\}^{K}$ is a categorical variable ($K$-dimensional
one-hot vector) indicating the chord label of frame $n$. In this paper, the
chord vocabulary consists of all possible combinations of twelve root notes
with six types of triad chords (with shorthands maj, min, dim, aug, sus2, and
sus4) and two types of power chords (with shorthands 1 and 5), and a no-chord
label (with shorthand N), i.e., $K=12\times 8+1=97$. We follow Harte’s
notation [37], e.g., C:maj and C#:sus4.

Let $\mathbf{Z}=\\{\mathbf{z}_{1},\cdots,\mathbf{z}_{N}\\}$ be a sequence of
latent features, where $\mathbf{z}_{n}\in\mathbb{R}^{L}$ is a continuous
variable that abstractly represents how $\mathbf{x}_{n}$ is deviated from a
basic chroma pattern specified by $\mathbf{s}_{n}$ ($L=64$). i.e., not only
$\mathbf{s}_{n}$ but also $\mathbf{z}_{n}$ are needed to fully describe
$\mathbf{x}_{n}$. If a region of C:maj includes some passing notes with
different volumes, for example, the chroma vectors are deviated from a basic
pattern that takes one at the dimensions corresponding to C, E, and G.

Our goal is to train a classification model $p(\mathbf{S}|\mathbf{X})$ and use
it for estimating chord labels behind unseen music signals. In an unsupervised
condition, the classification model should be trained from $\mathbf{X}$
without using the ground-truth data of $\mathbf{S}$. In a supervised
condition, the classification model can be trained by using paired data of
$\mathbf{X}$ and $\mathbf{S}$. In a semi-supervised condition, some part of
$\mathbf{S}$ is given as ground-truth data.

###  III-B Generative Model

We formulate a probabilistic hierarchical generative model (joint probability
distribution) of chord labels $\mathbf{S}$, latent features $\mathbf{Z}$, and
chroma vectors $\mathbf{X}$ as follows:

$\displaystyle
p_{\theta,\phi}(\mathbf{X},\mathbf{S},\mathbf{Z})=p_{\theta}(\mathbf{X}|\mathbf{S},\mathbf{Z})p_{\phi}(\mathbf{S})p(\mathbf{Z}),$
(1)

where $p_{\theta}(\mathbf{X}|\mathbf{S},\mathbf{Z})$ is a likelihood function
of $\mathbf{S}$ and $\mathbf{Z}$ for $\mathbf{X}$, $p_{\phi}(\mathbf{S})$ is a
prior distribution of $\mathbf{S}$, $p(\mathbf{Z})$ is the prior of
$\mathbf{Z}$, and $\theta$ and $\phi$ are model parameters. While standard
HMMs used for ACE are represented as
$p_{\theta,\phi}(\mathbf{X},\mathbf{S})=p_{\theta}(\mathbf{X}|\mathbf{S})p_{\phi}(\mathbf{S})$
[6], we use both $\mathbf{S}$ and $\mathbf{Z}$ for precisely representing
$\mathbf{X}$, i.e., $p_{\theta}(\mathbf{X}|\mathbf{S},\mathbf{Z})$ is a deep
generative model represented as follows:

$\displaystyle
p_{\theta}(\mathbf{X}|\mathbf{S},\mathbf{Z})=\prod_{n=1}^{N}\prod_{d=1}^{D}\mathrm{Bernoulli}(x_{nd}|[\bm{\omega}_{\theta}(\mathbf{S},\mathbf{Z})]_{nd}),$
(2)

where $\bm{\omega}_{\theta}(\mathbf{S},\mathbf{Z})$ is the $ND$-dimensional
output of a DNN with parameters $\theta$ that takes $\mathbf{S}$ and
$\mathbf{Z}$ as input and the notation $[\mathbf{A}]_{ij}$ indicates the
$ij$-th element of $\mathbf{A}$. Although $x_{nd}$ should take a binary value
in theory, we allow it to take a real value between $0$ and $1$ from a
practical point of view.

If we have no prior knowledge on chord labels $\mathbf{S}$, a natural choice
of $p_{\phi}(\mathbf{S})$ is a uniform distribution as follows:

$\displaystyle
p_{\phi}(\mathbf{S})=\prod_{n=1}^{N}\mathrm{Categorical}\left(\mathbf{s}_{n}|\textstyle\left[\frac{1}{K},\cdots,\frac{1}{K}\right]\right),$
(3)

Since chord labels $\mathbf{S}$ have temporal continuity, i.e., change
infrequently at the frame level, we follow the common assumption on chord
labels [12] and use a first-order Markov model favoring self-transitions of
chord labels as the chord label prior $p_{\phi}(\mathbf{S})$ as follows:

$\displaystyle p_{\phi}(\mathbf{S})$
$\displaystyle=p(\mathbf{s}_{1})\prod_{n=2}^{N}p(\mathbf{s}_{n}|\mathbf{s}_{n-1})$
$\displaystyle=\prod_{k=1}^{K}\phi_{k}^{s_{1k}}\prod_{n=2}^{N}\prod_{k^{\prime}=1}^{K}\prod_{k=1}^{K}\phi_{k^{\prime}k}^{s_{n-1,k^{\prime}}s_{nk}},$
(4)

where $\phi_{k}$ is the initial probability of chord $k$ and
$\phi_{k^{\prime}k}$ is the transition probability from chord $k^{\prime}$ to
chord $k$. Since $\phi_{k}$ and $\phi_{k^{\prime}k}$ represent the
probabilities, $\sum_{k=1}^{K}{\phi_{k}=1}$,
$\sum_{k=1}^{K}{\phi_{k^{\prime}k}=1}$. In the ACE researches based on HMMs,
the transition probability matrices typically have high self-transition
probability, which reflects the continuity of chord label sequences[16]. In
this paper, we set $\phi_{kk}=0.9$. The effects of (3) an (4) are compared in
Section IV-B.

Since we have no strong belief about the abstract latent features
$\mathbf{Z}$, the prior $p(\mathbf{Z})$ is set to a standard Gaussian
distribution as follows:

$\displaystyle
p(\mathbf{Z})=\prod_{n=1}^{N}\mathcal{N}(\mathbf{z}_{n}|\mathbf{0}_{L},\mathbf{I}_{L}),$
(5)

where $\mathbf{0}_{L}$ is the all-zero vector of size $L$ and $\mathbf{I}_{L}$
is the identity matrix of size $L\times L$.

###  III-C Classification and Recognition Models

Given chroma vectors $\mathbf{X}$ as observed data, we aim to infer the chord
labels $\mathbf{S}$ and the latent features $\mathbf{Z}$ from $\mathbf{X}$ and
estimate the model parameters $\theta$ and $\phi$ in the maximum-likelihood
framework. Because the DNN-based formulation makes the posterior distribution
$p_{\theta,\phi}(\mathbf{S},\mathbf{Z}|\mathbf{X})\propto
p_{\theta,\phi}(\mathbf{X},\mathbf{S},\mathbf{Z})$ analytically intractable,
we compute it approximately with an AVI technique. More specifically, we
introduce a sufficiently-expressive variational distribution
$q_{\alpha,\beta}(\mathbf{S},\mathbf{Z}|\mathbf{X})$ parametrized by $\alpha$
and $\beta$ and optimize it such that the Kullback-Leibler (KL) divergence
from $q_{\alpha,\beta}(\mathbf{S},\mathbf{Z}|\mathbf{X})$ to
$p_{\theta,\phi}(\mathbf{S},\mathbf{Z}|\mathbf{X})$ is minimized. Considering
that both $\mathbf{S}$ and $\mathbf{Z}$ make an effect on the generative model
$p_{\theta}(\mathbf{X}|\mathbf{S},\mathbf{Z})$, they are assumed to be
conditionally independent in the inference model
$q_{\alpha,\beta}(\mathbf{S},\mathbf{Z}|\mathbf{X})$ as follows:

$\displaystyle
q_{\alpha,\beta}(\mathbf{S},\mathbf{Z}|\mathbf{X})=q_{\alpha}(\mathbf{S}|\mathbf{X})q_{\beta}(\mathbf{Z}|\mathbf{X}),$
(6)

where $q_{\alpha}(\mathbf{S}|\mathbf{X})$ and
$q_{\beta}(\mathbf{Z}|\mathbf{X})$ are classification and recognition models
that infer $\mathbf{S}$ and $\mathbf{Z}$, respectively. In our study, (6) was
found to work better than
$q_{\alpha,\beta}(\mathbf{S},\mathbf{Z}|\mathbf{X})=q_{\alpha}(\mathbf{S}|\mathbf{X})q_{\beta}(\mathbf{Z}|\mathbf{X},\mathbf{S})$
respecting the chain rule of probability as in [32]. In this paper, these
models are implemented with DNNs parameterized by $\alpha$ and $\beta$ as
follows:

$\displaystyle q_{\alpha}(\mathbf{S}|\mathbf{X})$
$\displaystyle=\prod_{n=1}^{N}\mathrm{Categorical}(\mathbf{s}_{n}|[\bm{\pi}_{\alpha}(\mathbf{X})]_{n}),$
(7) $\displaystyle q_{\beta}(\mathbf{Z}|\mathbf{X})$
$\displaystyle=\prod_{n=1}^{N}\mathcal{N}(\mathbf{z}_{n}|[\bm{\mu}_{\beta}(\mathbf{X})]_{n},[\bm{\sigma}^{2}_{\beta}(\mathbf{X})]_{n}),$
(8)

where $\bm{\pi}_{\alpha}(\mathbf{X})$ is the $NK$-dimensional output of the
DNN with parameters $\alpha$, and $\bm{\mu}_{\beta}(\mathbf{X})$ and
$\bm{\sigma}^{2}_{\beta}(\mathbf{X})$ are the $NL$-dimensional outputs of the
DNN with parameters $\beta$. Similar to the deep generative model, the outputs
of the DNNs represent the parameters of probabilistic distributions.

###  III-D Unsupervised Learning with Non-Annotated Data

Instead of directly maximizing the log-marginal likelihood $\log
p_{\theta,\phi}(\mathbf{X})$ with respect to the model parameters $\theta$ and
$\phi$, we maximize its variational lower bound
$\mathcal{L}_{\mathbf{X}}(\theta,\phi,\alpha,\beta)$ derived by introducing
$q_{\alpha,\beta}(\mathbf{S},\mathbf{Z}|\mathbf{X})$ as follows:

$\displaystyle\log p_{\theta,\phi}(\mathbf{X}) $ $\displaystyle=\log\iint
p_{\theta,\phi}(\mathbf{X},\mathbf{S},\mathbf{Z})d\mathbf{S}d\mathbf{Z}$
$\displaystyle=\log\int\\!\\!\\!\\!\int\\!\frac{q_{\alpha,\beta}(\mathbf{S},\mathbf{Z}|\mathbf{X})}{q_{\alpha,\beta}(\mathbf{S},\mathbf{Z}|\mathbf{X})}p_{\theta,\phi}(\mathbf{X},\mathbf{S},\mathbf{Z})d\mathbf{S}d\mathbf{Z}$
$\displaystyle\geq\int\\!\\!\\!\\!\int\\!q_{\alpha,\beta}(\mathbf{S},\mathbf{Z}|\mathbf{X})\log\frac{p_{\theta,\phi}(\mathbf{X},\mathbf{S},\mathbf{Z})}{q_{\alpha,\beta}(\mathbf{S},\mathbf{Z}|\mathbf{X})}d\mathbf{S}d\mathbf{Z}$
$\displaystyle\stackrel{{\scriptstyle\triangle}}{{=}}\mathcal{L}_{\mathbf{X}}(\theta,\phi,\alpha,\beta),$
(9)

where the equality holds, i.e.,
$\mathcal{L}_{\mathbf{X}}(\theta,\phi,\alpha,\beta)$ is maximized, if and only
if
$q_{\alpha,\beta}(\mathbf{S},\mathbf{Z}|\mathbf{X})=p_{\theta,\phi}(\mathbf{S},\mathbf{Z}|\mathbf{X})$.
Note that this condition cannot be satisfied because
$p_{\theta,\phi}(\mathbf{S},\mathbf{Z}|\mathbf{X})$ is hard to compute.
Because the gap between $\log p_{\theta,\phi}(\mathbf{X})$ and
$\mathcal{L}_{\mathbf{X}}(\theta,\phi,\alpha,\beta)$ in (9) is equal to the KL
divergence from $q_{\alpha,\beta}(\mathbf{S},\mathbf{Z}|\mathbf{X})$ to
$p_{\theta,\phi}(\mathbf{S},\mathbf{Z}|\mathbf{X})$, the minimization of the
KL divergence is equivalent to the maximization of
$\mathcal{L}_{\mathbf{X}}(\theta,\phi,\alpha,\beta)$.

To approximately compute $\mathcal{L}_{\mathbf{X}}(\theta,\phi,\alpha,\beta)$,
we use Monte Carlo integration as follows:

$\displaystyle\mathcal{L}_{\mathbf{X}}(\theta,\phi,\alpha,\beta)$
$\displaystyle=\mathbb{E}_{q_{\alpha}(\mathbf{S}|\mathbf{X})q_{\beta}(\mathbf{Z}|\mathbf{X})}[\log
p_{\theta}(\mathbf{X}|\mathbf{S},\mathbf{Z})]$
$\displaystyle\quad+\mathbb{E}_{q_{\alpha}(\mathbf{S}|\mathbf{X})q_{\beta}(\mathbf{Z}|\mathbf{X})}[\log
p(\mathbf{Z})-\log q_{\beta}(\mathbf{Z}|\mathbf{X})]$
$\displaystyle\quad+\mathbb{E}_{q_{\alpha}(\mathbf{S}|\mathbf{X})}[\log
p_{\phi}(\mathbf{S})-\log q_{\alpha}(\mathbf{S}|\mathbf{X})]$
$\displaystyle\approx\frac{1}{I}\sum_{i=1}^{I}\log
p_{\theta}(\mathbf{X}|\mathbf{S}_{i},\mathbf{Z}_{i})-\mathrm{KL}(q_{\beta}(\mathbf{Z}|\mathbf{X})||p(\mathbf{Z}))$
$\displaystyle\quad+\mathrm{Entropy}[q_{\alpha}(\mathbf{S}|\mathbf{X})]+\mathbb{E}_{q_{\alpha}(\mathbf{S}|\mathbf{X})}[\log
p_{\phi}(\mathbf{S})],$ (10)

where $\\{\mathbf{S}_{i},\mathbf{Z}_{i}\\}_{i=1}^{I}$ are $I$ samples drawn
from $q_{\alpha}(\mathbf{S}|\mathbf{X})q_{\beta}(\mathbf{Z}|\mathbf{X})$.
Following the typical VAE implementation, we set $I=1$ and hence the index $i$
can be omitted. To make (10) partially differentiable with respect to the
model parameters $\theta$, $\phi$, $\alpha$, and $\beta$, we use
reparametrization tricks [8, 38] for deterministically representing the
categorical variables $\mathbf{S}$ and the Gaussian variables $\mathbf{Z}$ as
follows:

$\displaystyle\bm{\epsilon}^{\mathbf{s}}_{n}$
$\displaystyle\sim\mathrm{Gumbel}(\mathbf{0}_{K},\mathbf{1}_{K}),$ (11)
$\displaystyle\mathbf{s}_{n}$
$\displaystyle=\mathrm{softmax}((\log[\bm{\pi}_{\alpha}(\mathbf{X})]_{n}+\bm{\epsilon}^{\mathbf{s}}_{n})/\tau),$
(12) $\displaystyle\bm{\epsilon}^{\mathbf{z}}_{n}$
$\displaystyle\sim\mathcal{N}(\mathbf{0}_{L},\mathbf{I}_{L}),$ (13)
$\displaystyle\mathbf{z}_{n}$
$\displaystyle=[\bm{\mu}_{\beta}(\mathbf{X})]_{n}+\bm{\epsilon}^{\mathbf{z}}_{n}\odot[\bm{\sigma}_{\beta}(\mathbf{X})]_{n},$
(14)

where (11) indicates the standard Gumbel distribution, $\mathbf{1}_{K}$ is the
all-one vector of size $K$, $\odot$ means the element-wise product, and
$\tau>0$ is a temperature parameter that controls the uniformity of
$\mathbf{s}_{n}$ ($\tau=0.1$ in this paper).

We can now compute the four terms of (10) evaluating the fitness of
$\mathbf{S}$ and $\mathbf{Z}$: a reconstruction term indicating the likelihood
of $\mathbf{S}$ and $\mathbf{Z}$ for $\mathbf{X}$ and three regularization
terms making the posterior $q_{\beta}(\mathbf{Z}|\mathbf{X})$ close to the
prior $p(\mathbf{Z})$, increasing the entropy of
$q_{\alpha}(\mathbf{S}|\mathbf{X})$, and making $\mathbf{S}$ temporally
coherent, respectively (Fig. 2). More specifically, the first term is given by
(2) and the second and third terms are given by

$\displaystyle\mathrm{KL}(q_{\beta}(\mathbf{Z}|\mathbf{X})||p(\mathbf{Z}))$
$\displaystyle=\\!\sum_{n=1}^{N}\sum_{k=1}^{K}\\!\left(\frac{[\bm{\sigma}_{\beta}(\mathbf{X})]_{nk}^{2}\\!+\\![\bm{\mu}_{\beta}(\mathbf{X})]_{nk}^{2}\\!-\\!1}{2}-\log([\bm{\sigma}_{\beta}(\mathbf{X})]_{nk})\\!\right)\\!,$
(15) $\displaystyle\mathrm{Entropy}[q_{\alpha}(\mathbf{S}|\mathbf{X})]$
$\displaystyle=-\sum_{n=1}^{N}\sum_{k=1}^{K}[\pi_{\alpha}(\mathbf{X})]_{nk}\log([\pi_{\alpha}(\mathbf{X})]_{nk}).$
(16)

The last term $\mathbb{E}_{q_{\alpha}(\mathbf{S}|\mathbf{X})}[\log
p_{\theta}(\mathbf{S})]$ can be calculated analytically. If
$p_{\theta}(\mathbf{S})$ is the uniform distribution given by (3), we have

$\displaystyle\mathbb{E}_{q_{\alpha}(\mathbf{S}|\mathbf{X})}[\log
p(\mathbf{S})]=-\sum_{n=1}^{N}\sum_{k=1}^{K}[\bm{\pi}_{\alpha}(\mathbf{X})]_{nk}\log
K.$ (17)

If $p_{\theta}(\mathbf{S})$ is the Markov model given by (4), we use a dynamic
programming technique similar to the forward algorithm of the HMM. Let
$\gamma(\mathbf{s}_{n})=\mathbb{E}_{q_{\alpha}(\mathbf{S}|\mathbf{X})}[\log
p(\mathbf{s}_{1:n})]$ be a forward message at frame $n$, which can be
calculated recursively as follows:

$\displaystyle\gamma(\mathbf{s}_{1})=\log p(\mathbf{s}_{1}),$ (18)
$\displaystyle\gamma(\mathbf{s}_{n})=\sum_{\mathbf{s}_{n-1}}q_{\alpha}(\mathbf{s}_{n-1}|\mathbf{X})\bigl{(}\gamma(\mathbf{s}_{n-1})+\log
p_{\phi}(\mathbf{s}_{n}|\mathbf{s}_{n-1})\bigr{)},$ (19)
$\displaystyle\mathbb{E}_{q_{\alpha}(\mathbf{S}|\mathbf{X})}[\log
p_{\phi}(\mathbf{S})]=\sum_{\mathbf{s}_{N}}q_{\alpha}(\mathbf{s}_{N}|\mathbf{X})\gamma(\mathbf{s}_{N}).$
(20)

We now have a VAE (Fig. 2) that consists of:

  * •

the deep classification model $q_{\alpha}(\mathbf{S}|\mathbf{X})$ given by (7)
with the reparametrization trick given by (11) and (12),

  * •

the deep recognition model $q_{\beta}(\mathbf{Z}|\mathbf{X})$ given by (8)
with the reparametrization trick given by (13) and (14), and

  * •

the deep generative model $p_{\theta}(\mathbf{X}|\mathbf{S},\mathbf{Z})$ given
by (2) with the prior distributions $p_{\phi}(\mathbf{S})$ and $p(\mathbf{Z})$
given by (4) and (5), respectively.

In the unsupervised condition, all models are jointly optimized in the
framework of the VAE by using a variant of stochastic gradient descent such
that (10) is maximized with respect to $\theta$, $\alpha$, and $\beta$.

###  III-E Supervised Learning with Annotated Data

Under the supervised condition that chroma vectors $\mathbf{X}$ and the
corresponding chord labels $\mathbf{S}$ are given as observed data, we aim to
maximize a variational lower bound
$\mathcal{L}_{\mathbf{X},\mathbf{S}}(\theta,\beta)$ of the log-likelihood
$\log p_{\theta}(\mathbf{S}|\mathbf{X})$, which can be derived in a way
similar to (9) as follows:

$\displaystyle\log p_{\theta}(\mathbf{X}|\mathbf{S})$
$\displaystyle=\log\int\\!\frac{q_{\beta}(\mathbf{Z}|\mathbf{X})}{q_{\beta}(\mathbf{Z}|\mathbf{X})}p_{\theta}(\mathbf{X}|\mathbf{Z},\mathbf{S})d\mathbf{Z}$
$\displaystyle\geq\int
q_{\beta}(\mathbf{Z}|\mathbf{X})\log\frac{p_{\theta}(\mathbf{X}|\mathbf{Z},\mathbf{S})}{q_{\beta}(\mathbf{Z}|\mathbf{X})}d\mathbf{Z}$
$\displaystyle\stackrel{{\scriptstyle\triangle}}{{=}}\mathcal{L}_{\mathbf{X},\mathbf{S}}(\theta,\beta).$
(21)

Using Monte Carlo integration,
$\mathcal{L}_{\mathbf{X},\mathbf{S}}(\theta,\beta)$ can be approximately
computed as follows:

$\displaystyle\mathcal{L}_{\mathbf{X},\mathbf{S}}(\theta,\beta)$
$\displaystyle\stackrel{{\scriptstyle\triangle}}{{=}}\mathbb{E}_{q_{\beta}(\mathbf{Z}|\mathbf{X})}(\log
p_{\theta}(\mathbf{X}|\mathbf{S},\mathbf{Z})+\log p(\mathbf{Z})-\log
q_{\beta}(\mathbf{Z}|\mathbf{X}))$
$\displaystyle\approx\frac{1}{I}\sum_{i=1}^{I}\log
p_{\theta}(\mathbf{X}|\mathbf{S}_{i},\mathbf{Z}_{i})+\mathrm{KL}(q_{\beta}(\mathbf{Z}|\mathbf{X})\|p(\mathbf{Z})),$
(22)

where $\\{\mathbf{Z}_{i}\\}_{i=1}^{I}$ are $I$ samples drawn from
$q_{\beta}(\mathbf{Z}|\mathbf{X},\mathbf{S})$ using the reparametrization
trick ($I=1$ in this paper). Because the chord estimator
$q_{\alpha}(\mathbf{S}|\mathbf{X})$, which plays a central role in ACE, does
not appear in (22), $q_{\alpha}(\mathbf{S}|\mathbf{X})$ cannot be trained only
by maximizing (22). As suggested in the semi-supervised learning of a VAE
[32], one could thus define an alternative objective function
$\mathcal{L}_{\mathbf{X},\mathbf{S}}(\theta,\alpha,\beta)$ including $\alpha$
by adding a classification performance term to (22) as follows:

$\displaystyle\mathcal{L}_{\mathbf{X},\mathbf{S}}(\theta,\alpha,\beta)\stackrel{{\scriptstyle\triangle}}{{=}}\mathcal{L}_{\mathbf{X},\mathbf{S}}(\theta,\beta)+\log
q_{\alpha}(\mathbf{S}|\mathbf{X}).$ (23)

A problem of this approach, however, is that the regularization terms (16) and
(20) enhancing the entropy of $q_{\alpha}(\mathbf{S}|\mathbf{X})$ (preventing
the overfitting) and smoothing the output of
$q_{\alpha}(\mathbf{S}|\mathbf{X})$, respectively, are not taken into account.

To solve this problem, we propose a new objective function
$\mathcal{L}_{\mathbf{X},\mathbf{S}}(\theta,\phi,\alpha,\beta)$ by summing
(10) and (23) as follows:

$\displaystyle\mathcal{L}_{\mathbf{X},\mathbf{S}}(\theta,\phi,\alpha,\beta)={\mathcal{L}_{\mathbf{X}}(\theta,\phi,\alpha,\beta)+\mathcal{L}_{\mathbf{X},\mathbf{S}}(\theta,\alpha,\beta)}.$
(24)

In this function, the chroma vectors $\mathbf{X}$ with the annotations
$\mathbf{S}$ are used twice as unsupervised and supervised training data as if
they were not annotated (first term) and as they are (second term),
respectively.

###  III-F Semi-supervised Learning

Under the semi-supervised condition that partially-annotated chroma vectors
are available, we define an objective function by summing the objective
functions (10) and (24) corresponding to the unsupervised and supervised
conditions, respectively, as follows:

$\displaystyle\mathcal{L}^{\prime}_{\mathbf{X},\mathbf{S}}(\theta,\phi,\alpha,\beta)$
$\displaystyle\stackrel{{\scriptstyle\triangle}}{{=}}\\!\sum_{\mbox{\scriptsize{$\mathbf{X}$
w/o
$\mathbf{S}$}}}\\!\\!\mathcal{L}_{\mathbf{X}}(\theta,\phi,\alpha,\beta)+\\!\sum_{\mbox{\scriptsize{$\mathbf{X}$
with
$\mathbf{S}$}}}\\!\mathcal{L}_{\mathbf{X},\mathbf{S}}(\theta,\phi,\alpha,\beta),$
$\displaystyle=\ \
\sum_{\mbox{\scriptsize{$\mathbf{X}$}}}\mathcal{L}_{\mathbf{X}}(\theta,\phi,\alpha,\beta)+\\!\sum_{\mbox{\scriptsize{$\mathbf{X}$
with
$\mathbf{S}$}}}\\!\mathcal{L}_{\mathbf{X},\mathbf{S}}(\theta,\alpha,\beta).$
(25)

Note that $q_{\alpha}(\mathbf{S}|\mathbf{X})$ can always be regularized by
$p_{\theta}(\mathbf{X}|\mathbf{S},\mathbf{Z})$ and $p_{\phi}(\mathbf{S})$
regardless of the availability of annotations.

###  III-G Training and Prediction

The model parameters $\theta$, $\phi$, $\alpha$, and $\beta$ can be optimized
with a stochastic gradient descent method (e.g., Adam [39]) such that the
objective function (10), (24), or (25) is maximized. Nonetheless, in this
paper, $\phi$ is fixed (not optimized) because of its wide acceptable range
(see Section IV-B3). Under the semi-supervised condition, each mini-batch
consists of non-annotated and annotated chroma vectors (50%–50% in this paper)
randomly selected from the training dataset.

In the test phase, using the neural chord estimator
$q_{\alpha}(\mathbf{S}|\mathbf{X})$, a sequence of the posterior probabilities
of chord labels $\mathbf{S}$ are calculated from a sequence of chroma vectors
$\mathbf{X}$ extracted from a target music signal. The optimal temporally-
coherent path of chord labels is then estimated via the Viterbi algorithm with
the transition probabilities $\bm{\phi}$.

##  IV Evaluation

This section reports comparative experiments conducted for evaluating the
effectiveness of the proposed method. Specifically, we investigate the
effectiveness of the VAE-based regularized training, that of using the Markov
prior on chord labels, and, that of using external non-annotated data.

###  IV-A Experimental Conditions

We here explain compared ACE methods, network configurations, datasets, and
evaluation procedures and measures.

####  IV-A1 Compared Methods

As listed in Table I, we trained a chord estimator
$q_{\alpha}(\mathbf{S}|\mathbf{X})$ in five different ways:

  * •

ACE-SL (baseline): As in most ACE methods, $q_{\alpha}(\mathbf{S}|\mathbf{X})$
was trained in a supervised manner by minimizing the cross-entropy loss for
the ground-truth labels $\mathbf{S}$, i.e., maximizing the following objective
function:

$\displaystyle\mathcal{L}_{\mathbf{X},\mathbf{S}}(\alpha)=\log
q_{\alpha}(\mathbf{S}|\mathbf{X}).$ (26)

  * •

VAE-UN-SL: $q_{\alpha}(\mathbf{S}|\mathbf{X})$ was trained in a supervised
manner by maximizing (24) with the uniform prior (3).

  * •

VAE-MR-SL: $q_{\alpha}(\mathbf{S}|\mathbf{X})$ was trained in a supervised
manner by maximizing (24) with the Markov prior (4).

  * •

VAE-UN-SSL: $q_{\alpha}(\mathbf{S}|\mathbf{X})$ was trained in a semi-
supervised manner by maximizing (25) with the uniform prior (3).

  * •

VAE-MR-SSL: $q_{\alpha}(\mathbf{S}|\mathbf{X})$ was trained in a semi-
supervised manner by maximizing (25) with the Markov prior (4).

For convenience, let ‘*’ denote the wild-card character, e.g., VAE-*-SL means
VAE-MR-SL or VAE-UN-SL.

Comparing ACE-SL with VAE-*-SL, we evaluated the effectiveness of the VAE
architecture in regularizing $q_{\alpha}(\mathbf{S}|\mathbf{X})$. Comparing
VAE-*-SSL with VAE-*-SL, we evaluated the effectiveness of the semi-supervised
learning. Comparing VAE-MR-* with VAE-UN-*, we evaluated the effectiveness of
the Markov prior on $\mathbf{S}$.

TABLE I: Experimental conditions
---
|  | Training
---
| Regularization
---
| Prior
---
ACE-SL (baseline) | Supervised | NA | NA
VAE-UN-SL | Supervised | VAE-based | Uniform
VAE-MR-SL | Supervised | VAE-based | Markov
VAE-UN-SSL | Semi-supervised | VAE-based | Uniform
VAE-MR-SSL | Semi-supervised | VAE-based | Markov
TABLE II: Durations of chord types in datasets used for evaluation
---
Chord type | Duration [h]
maj | 53.09
min | 16.63
aug | 0.15
dim | 0.36
sus4 | 1.63
sus2 | 0.25
1 | 0.76
5 | 0.84

####  IV-A2 Network Configurations

Each of the classifier $q_{\alpha}(\mathbf{S}|\mathbf{X})$, the recognizer
$q_{\beta}(\mathbf{Z}|\mathbf{X})$, and the generator
$p_{\theta}(\mathbf{X}|\mathbf{S},\mathbf{Z})$ was implemented with a three-
layered BLSTM network [40] followed by layer normalization [41]. The final
layer of $q_{\alpha}(\mathbf{S}|\mathbf{X})$ consisted of softmax functions
that output the frame-level posterior probabilities of $K$ chord labels. The
final layer of $q_{\beta}(\mathbf{Z}|\mathbf{X})$ consisted of linear units
that output $\bm{\mu}_{\beta}(\mathbf{X})$ and
$\log\bm{\sigma}^{2}_{\beta}(\mathbf{X})$. The final layer of
$p_{\theta}(\mathbf{X}|\mathbf{S},\mathbf{Z})$ consisted of sigmoid functions
that output $\bm{\omega}_{\theta}(\mathbf{S},\mathbf{Z})$. Each hidden layer
of the BLSTM had 256 units. Because the architecture of
$q_{\alpha}(\mathbf{S}|\mathbf{X})$ was similar to that of a state-of-the-art
chord estimator [36], ACE-SL was considered to be a reasonable baseline
method.

We used Adam optimizer[39], where the learning rate was first set to 0.001 and
then decreased exponentially by a scaling factor of 0.99 per epoch. Gradient
clipping with norm 5 was additionally applied to the optimization process.
Each mini-batch consisted of 16 sequences, each of which contained 645 frames
(1 min). The parameter of the Markov prior $\phi$ was fixed as stated in
III-B, and $\theta$, $\alpha$ and $\beta$ were iteratively updated. In all the
methods and configurations, the number of training epochs was set to 300,
which was sufficiently large to reach convergence (early stopping was not
used).

####  IV-A3 Datasets

We collected 1210 annotated popular songs consisting of 198 songs from
Isophonics [37], 100 songs from RWC-MDB-P-2001 [42], 186 songs from uspop2002
[43]222 The annotations for RWC-MDB-P-2001 and uspop2002 are provided by the
Music and Audio Research Lab at NYU., and 726 songs from McGill Billboard
dataset [44]. As shown in Table II, the six types of triad chords and the two
types of power chords are heavily imbalanced in the chord annotations, where
most of the annotated chord labels belong to the major and minor triads. We
also collected 700 non-annotated popular songs composed by Japanese and
American artists.

Each music signal sampled at 44.1kHz was analyzed with constant-Q transform
(CQT) with a shifting interval of 4096 points and a frequency resolution of
one semitone per bin. The CQT spectrogram and its 1-, 2-, 3-, and 4-octave-
shifted versions were stacked to yield a five-layered harmonic CQT (HCQT)
representation, which was fed to a neural multi-pitch estimator [36] for
computing chroma vectors $\mathbf{X}$. To compensate for the imbalance of the
ratios of the 12 chord roots in the training data, chroma vectors and chord
labels were jointly pitch-rotated by a random number of semitones on each
training iteration. Pitch shifting of the multi-channel chroma vectors can be
done by performing vector rotation on each channel, and pitch shifting of
chord annotations can be done by changing the root notes.

####  IV-A4 Evaluation Procedures

To conduct five-fold cross validation, we divided the 1210 annotated songs
into five subsets (242 songs each). In each fold, one of the subsets was kept
as test data and the remaining four subsets were used as training data, in
which $I$ and $4-I$ subsets were treated as annotated and non-annotated songs,
respectively ($I\in\\{1,2,3,4\\}$). Not only the classifier
$q_{\alpha}(\mathbf{S}|\mathbf{X})$ but also the recognizer
$q_{\beta}(\mathbf{Z}|\mathbf{X})$ and the generator
$p_{\theta}(\mathbf{X}|\mathbf{S},\mathbf{Z})$ were trained jointly by using
all the training data (VAE-*-SSL) or only the annotated data (VAE-*-SL). In
ACE-SL, in contrast, only the classifier $q_{\alpha}(\mathbf{S}|\mathbf{X})$
was trained by using the annotated data.

VAE-*-SSL was additionally tested under extended semi-supervised conditions
that annotated 976 songs (four subsets) and $M$ non-annotated songs
($M\in\\{250,500,700\\}$) were used as training data in each fold. Note that
the performance was measured on the remaining 242 annotated songs, which did
not overlap with the 700 non-annotated songs.

####  IV-A5 Evaluation Measures

The chord estimation performance (accuracy) of each method was measured in
terms of the frame-level match rate between the estimated and ground-truth
chord sequences. The weighed accuracy for each song was measured with mir_eval
library [45] in terms of the majmin criterion considering only the major and
minor triads plus the no-chord label ($K=25$) and the triads criterion with
the vocabulary defined in Section III-A ($K=97$). The overall accuracy was
given as the average of the song-wise accuracies weighed by the song lengths.

###  IV-B Experimental Results

The overall accuracies of ACE-SL and VAE-*-* with respect to the numbers of
annotated and non-annotated songs (denoted as $A{+}B$) used for training are
shown in Fig. 3. To investigate the temporal continuity of $\mathbf{S}$
induced by the Markov prior, we tested VAE-MR-SSL with a self-transition
probability $\phi_{kk}\in\\{1/K,0.3,0.5,0.7,0.9\\}$. The song-wise accuracies
and average chord durations at $976{+}0$ and $976{+}700$ are compared in Fig.
4. Examples of estimated chord sequences estimated by ACE-SL, VAE-UN-SSL, and
VAE-MR-SSL with $\phi_{kk}=0.9$ are shown in Fig. 5.

####  IV-B1 Evaluation of VAE-Based Regularized Training

We confirmed the effectiveness of the VAE-based regularized training of
$q_{\alpha}(\mathbf{S}|\mathbf{X})$ under all the conditions. As shown in Fig.
3 and Fig. 4(a), VAE-*-SL clearly outperformed ACE-SL by a large margin around
1 pts. $q_{\alpha}(\mathbf{S}|\mathbf{X})$ was regularized effectively by
considering its entropy, the Markov or uniform prior of $\mathbf{S}$, and the
reconstruction of $\mathbf{X}$ based on $\mathbf{S}$ and $\mathbf{Z}$.

####  IV-B2 Evaluation of Semi-supervised Learning

We confirmed the effectiveness of the semi-supervised learning of
$q_{\alpha}(\mathbf{S}|\mathbf{X})$. Under the semi-supervised conditions at
$244{+}732$, $488{+}488$, and $732{+}244$ in the left half of Fig. 3, VAE-MR-
SSL and VAE-UN-SSL outperformed VAE-MR-SL and VAE-UN-SL, respectively, where
the proposed VAE-MR-SSL worked best. The success of the semi-supervised
learning was attributed to the fact that the musical and acoustic
characteristics of the 976 songs used for training were consistent with those
of the 244 songs used for testing in the five-fold cross validation with the
1210 songs. Under the supervised condition at $976{+}0$, VAE-MR-SSL and VAE-
UN-SSL were equivalent to VAE-MR-SL and VAE-UN-SL, respectively.

Under the extended semi-supervised conditions at $976{+}250$, $976{+}500$, and
$976{+}700$ in the right half of Fig. 3, the performance once dropped and then
barely recovered according to the increase of non-annotated songs used for
training. The large difference in the musical and acoustic characteristics of
the annotated and non-annotated songs is considered to have hindered the semi-
supervised learning.

####  IV-B3 Evaluation of Markov Prior

We confirmed the effectiveness of the Markov prior on $\mathbf{S}$ under all
the conditions. Under the semi-supervised conditions at $244{+}732$,
$488{+}488$, and $732{+}244$ and the supervised condition at $976{+}0$ in the
left half of Fig. 3, VAE-MR-SL and VAE-MR-SSL outperformed VAE-UN-SL and VAE-
UN-SSL, respectively. The Markov prior played a vital role under the extended
semi-supervised conditions at $976{+}250$, $976{+}500$, and $976{+}700$ in the
right half of Fig. 3. While VAE-UN-SSL at $976{+}250$ significantly
underperformed VAE-UN-SSL at $976{+}0$, VAE-MR-SSL did not experience large
performance drop and achieved slightly better performance at $976{+}700$ by
encouraging $q_{\alpha}(\mathbf{S}|\mathbf{X})$ to yield a temporally-smooth
estimate of $\mathbf{S}$.

Fig. 4(b) shows that the Markov prior mitigated the negative effect of the
semi-supervised learning. VAE-UN-SSL (VAE-MR-SSL with $\phi_{kk}=1/K$)
performed better, but yielded finer chord segments than ACE-SL because
$q_{\alpha}(\mathbf{S}|\mathbf{X})$ was trained under a condition that chord
labels were allowed to change frequently for non-annotated songs such that the
unsupervised learning objective (10) was maximized. In contrast, VAE-MR-SSL
with a higher self-transition probability yielded longer chord segments and
the chord durations were distributed in a way similar to the ground-truth
data. As shown in Fig. 4(a), the choice of the self-transition probability
between $0.3$ and $0.9$ had a small impact on the overall chord estimation
accuracies.

Figure 3: The experimental results of the five-fold cross validation using the
1210 annotated songs and the 700 external non-annotated songs. Figure 4:  The
song-wise accuracies (in majmin criterion) and average durations of chord
labels estimated by ACE-SL and VAE-MR-SSL with different self-transition
probabilities ($\phi_{kk}\in\\{1/K,0.3,0.5,0.7,0.9\\}$), where VAE-MR-SSL with
$\phi_{kk}=1/K$ is equivalent to VAE-UN-SSL. The chord durations were measured
before the Viterbi post-filtering.  Figure 5:  An example of chord label
sequences estimated by the supervised and semi-supervised methods without the
Viterbi post-filtering. For readability, only the first 24 dimensions (bass
and middle channels) of the chroma vectors are displayed.

In Fig. 5, some errors made by ACE-SL (light-green segments) were corrected by
the VAE-based methods. The chord label sequence obtained by VAE-UN-SSL,
however, included a number of incorrect short fragments caused by the local
fluctuations of the chroma vectors. As discussed above,
$q_{\alpha}(\mathbf{S}|\mathbf{X})$ was encouraged to frequently vary over
time such that the fine structure of chroma vectors could be reconstructed
precisely in the VAE framework. In contrast, the chord label sequence obtained
by VAE-MR-SSL included fewer transitions and was much closer to the ground-
truth sequence.

Figure 6: Confusion matrices with respect to chord types. Estimated chords
with wrong root notes are not counted.

Figure 7: The probability distributions obtained by
$p_{\theta}(\mathbf{X}|\mathbf{S},\mathbf{Z})$ conditioned by different chord
labels $\mathbf{S}$. Only the first 24 dimensions (bass and middle channels)
of the chroma vectors are displayed.

###  IV-C Further Observations

Fig. 6 shows the confusion matrices obtained by ACE-SL and VAE-MR-SSL. While
the accuracies on the maj, min, aug and dim types were improved by VAE-MR-SSL,
the accuracies on the other uncommon chord types were significantly degraded.
Rare chords tended to be wrongly classified to the maj and min triads.
Interestingly, the accuracies on the eight chord types were not necessarily
correlated to their ratios in the training data (Table II), where the sus4
triads were much more frequently used than the aug and dim triads. The same
problem occurred in VAE-UN-SSL because the unsupervised learning objective
(10) has no mechanism to prevent $q_{\alpha}(\mathbf{S}|\mathbf{X})$ from
excessively yielding popular chord types.

As shown in Fig. 7, the generative model
$p_{\theta}(\mathbf{X}|\mathbf{S},\mathbf{Z})$ successfully reconstructed
chroma vectors, conditioned by the maj, min, aug, and dim triads, i.e., the
reconstructed chroma vectors had high probabilities on the pitch classes of
the chord notes. In contrast, $p_{\theta}(\mathbf{X}|\mathbf{S},\mathbf{Z})$
failed to reconstruct chroma vectors, when conditioned by the sus2 and sus4
triads and the power chords. Comparing Fig. 6 with Fig. 7, we investigate the
relationships between the estimation accuracy of chord labels and the
reconstruction quality of chroma vectors. The generator
$p_{\theta}(\mathbf{X}|\mathbf{S},\mathbf{Z})$ failed to learn the pitch-class
distributions of several chord classes (e.g., sus2 and sus4) whose chroma
vectors often had no clear peaks on the chord notes. For such chroma vectors,
$p_{\theta}(\mathbf{X}|\mathbf{S},\mathbf{Z})$ always gave a lower probability
even if $\mathbf{S}$ was the ground-truth chord classes corresponding to
$\mathbf{X}$. Note that $p_{\theta}(\mathbf{X}|\mathbf{S},\mathbf{Z})$ was
used for regularizing the classifier $q_{\alpha}(\mathbf{S}|\mathbf{X})$ in
the unsupervised learning objective (10), where
$q_{\alpha}(\mathbf{S}|\mathbf{X})$ was trained to avoid sus2 and sus4 triads
and favor maj triads to maximize
$p_{\theta}(\mathbf{X}|\mathbf{S},\mathbf{Z})$.

##  V Conclusion

This paper described a statistical method that trains a neural chord estimator
in a semi-supervised manner by constructing a VAE with latent chord labels and
features. This is a new approach to ACE that unifies the generative and
discriminative methods. Our method incorporates a Markov prior on chord labels
to encourage the temporal continuity of chord labels estimated with the chord
estimator. The comparative experiment clearly showed the effectiveness of
using the generative model of chroma vectors and the Markov prior on chord
labels in regularizing the chord estimator for performance improvement. The
limitation of the proposed semi-supervised learning is that the trained chord
classifier tends to mistakenly classify some specific chord types into popular
types. This may become more problematic when a larger chord vocabulary
including seventh chords and chord inversions is used.

The success of the semi-supervised VAE for ACE indicates the effectiveness of
unifying the deep generative and discriminative methods in automatic music
transcription (AMT). Using the AVI framework, we can explicitly introduce
prior knowledge on musical symbol sequences as a regularization term. Such
knowledge is hard to automatically extract from training data in supervised
discriminative methods. One way to improve the performance of ACE is to
replace the frame-level Markov prior of chord labels with a beat- or symbol-
level language model, which has been considered to be effective for solving
the ambiguity in acoustic features [23, 24]. We also plan to develop a
comprehensive AMT system based on a unified VAE that can treat mutually-
dependent musical elements such as keys, beats, and notes as latent variables.

## References

  * [1] M. Goto, K. Yoshii, H. , Fujihara, M. Mauch, and T. Nakano, “Songle: A web service for active music listening improved by user contributions,” in _proceedings of the International Society for Music Information Retrieval (ISMIR)_ , 2011, pp. 311–316.
  * [2] A. Anglade, R. Ramirez, and S. Dixon, “Genre classification using harmony rules induced from automatic chord transcriptions.” in _proceedings of the International Society for Music Information Retrieval (ISMIR)_ , 2009, pp. 669–674.
  * [3] J. P. Bello, “Audio-based cover song retrieval using approximate chord sequences: Testing shifts, gaps, swaps and beats.” in _proceedings of the International Society for Music Information Retrieval (ISMIR)_ , vol. 7, 2007, pp. 239–244.
  * [4] J. Pauwels, K. O’Hanlon, E. Gómez, and M. B. Sandler, “20 years of automatic chord recognition from audio,” in _proceedings of the International Society for Music Information Retrieval (ISMIR)_ , 2019, pp. 54–63.
  * [5] K. Lee and M. Slaney, “Acoustic chord transcription and key extraction from audio using key-dependent hmms trained on synthesized audio,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 16, no. 2, pp. 291–301, Feb 2008.
  * [6] M. McVicar, R. Santos-Rodriguez, Y. Ni, and T. D. Bie, “Automatic chord estimation from audio: A review of the state of the art,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 22, no. 2, pp. 556–575, 2014.
  * [7] E. J. Humphrey and J. P. Bello, “Four timely insights on automatic chord estimation,” in _proceedings of the International Society for Music Information Retrieval_ , 10 2015, pp. 673–679.
  * [8] D. P. Kingma and M. Welling, “Auto-encoding variational Bayes,” in _proceedings of the International Conference on Learning Representations (ICLR)_ , 2014, pp. 1–14.
  * [9] S. Gershman and N. Goodman, “Amortized inference in probabilistic reasoning,” in _Proceedings of the annual meeting of the cognitive science society_ , vol. 36, no. 36, 2014, pp. 517–522.
  * [10] L. R. Rabiner, “A tutorial on hidden markov models and selected applications in speech recognition,” _Proceedings of the IEEE_ , vol. 77, no. 2, pp. 257–286, 1989.
  * [11] T. Fujishima, “Realtime chord recognition of musical sound: A system using common lisp music,” in _Proceedings ofthe International Computer Music Con- ference (ICMC)_ , 1999, pp. 464–467.
  * [12] A. Sheh and D. Ellis, “Chord segmentation and recognition using EM-trained hidden Markov models,” in _proceedings of the International Society for Music Information Retrieval (ISMIR)_ , 2003, pp. 185–191.
  * [13] M. Mauch and S. Dixon, “Approximate note transcription for the improved identification of difficult chords,” in _proceedings of the International Society for Music Information Retrieval (ISMIR)_ , 2010, pp. 135–140.
  * [14] Y. Ni, M. McVicar, R. Santos-Rodriguez, and T. De Bie, “An end-to-end machine learning system for harmonic analysis of music,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 20, no. 6, pp. 1771–1783, 2012.
  * [15] M. Khadkevich and M. Omogolo, “Use of hidden markov models and factored language models for automatic chord recognition,” in _proceedings of the International Society for Music Information Retrieval (ISMIR)_ , 2009, pp. 561–566.
  * [16] R. Chen, W. Shen, A. Srinivasamurthy, and P. Chordia, “Chord recognition using duration-explicit hidden Markov models,” in _proceedings of the International Society for Music Information Retrieval (ISMIR)_ , 2012, pp. 445–450.
  * [17] E. J. Humphrey and J. P. Bello, “Rethinking automatic chord recognition with convolutional neural networks,” in _2012 11th International Conference on Machine Learning and Applications_ , vol. 2, 2012, pp. 357–362.
  * [18] C. Schörkhuber and A. Klapuri, “Constant-q transform toolbox for music processing,” in _7th Sound and Music Computing Conference_ , 2010.
  * [19] F. Korzeniowski and G. Widmer, “A fully convolutional deep auditory model for musical chord recognition,” in _IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP)_ , 2016, pp. 13–16.
  * [20] Y. Wu, T. Carsault, and K. Yoshii, “Automatic chord estimation based on a frame-wise convolutional recurrent neural network with non-aligned annotations,” in _27th European Signal Processing Conference (EUSIPCO)_ , 2019.
  * [21] N. Boulanger-Lewandowski, Y. Bengio, and P. Vincent, “Audio chord recognition with recurrent neural networks.” in _proceedings of the International Society for Music Information Retrieval (ISMIR)_ , 2013, pp. 335–340.
  * [22] S. Sigtia, N. Boulanger-Lewandowski, and S. Dixon, “Audio chord recognition with a hybrid recurrent neural network,” in _proceedings of the International Society for Music Information Retrieval (ISMIR)_ , 2015, pp. 127–133.
  * [23] F. Korzeniowski and G. Widmer, “On the futility of learning complex frame-level language models for chord recognition,” in _AES Conference on Semantic Audio_ , 2017, pp. 2–6.
  * [24] ——, “Improved chord recognition by combining duration and harmonic language models,” in _proceedings of the International Society for Music Information Retrieval (ISMIR)_ , 2018, pp. 10–17.
  * [25] T.-P. Chen and L. Su, “Harmony transformer: Incorporating chord segmentation into harmony recognition,” in _proceedings of the International Society for Music Information Retrieval (ISMIR)_ , 2019, pp. 259–267.
  * [26] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Łukasz Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2017, pp. 5998–6008.
  * [27] H. V. Koops, W. B. de Haas, J. A. Burgoyne, J. Bransen, A. Kent-Muller, and A. Volk, “Annotator subjectivity in harmony annotations of popular music,” _Journal of New Music Research_ , vol. 48, no. 3, pp. 232–252, 2019.
  * [28] B. Mcfee and J. P. Bello, “Structured training for large-vocabulary chord recognition,” in _proceedings of the International Society for Music Information Retrieval (ISMIR)_ , 2017, pp. 188–194.
  * [29] T. Carsault, J. Nika, and P. Esling, “Using musical relationships between chord labels in automatic chord extraction tasks,” in _proceedings of the International Society for Music Information Retrieval (ISMIR)_ , 2018, pp. 18–25.
  * [30] J. Jiang, K. Chen, W. Li, and G. Xia, “Large-vocabulary chord transcription via chord structure decomposition,” in _proceedings of the International Society for Music Information Retrieval (ISMIR)_ , 2019, pp. 644–651.
  * [31] J. Deng and Y. K. Kwok, “Large vocabulary automatic chord estimation with an even chance training scheme,” in _proceedings of the International Society for Music Information Retrieval (ISMIR)_ , 2017, pp. 531–536.
  * [32] D. P. Kingma, S. Mohamed, D. J. Rezende, and M. Welling, “Semi-supervised learning with deep generative models,” in _NIPS_ , 2014, pp. 3581–3589.
  * [33] L. Maaløe, C. K. Sønderby, S. K. Sønderby, and O. Winther, “Auxiliary deep generative models,” in _proceedings of the International Conference on Machine Learning (ICML)_ , 2016, pp. 1445–1453.
  * [34] T. Hori, R. F. Astudillo, T. Hayashi, Y. Zhang, S. Watanabe, and J. L. Roux, “Cycle-consistency training for end-to-end speech recognition,” in _proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2019, pp. 6271–6275.
  * [35] E. Dupont, “Learning disentangled joint continuous and discrete representations,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2018, pp. 710–720.
  * [36] Y. Wu and W. Li, “Automatic audio chord recognition with MIDI-trained deep feature and BLSTM-CRF sequence decoding model,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 27, no. 2, pp. 355–366, 2019.
  * [37] C. Harte, “Towards automatic extraction of harmony information from music signals,” Ph.D. dissertation, Queen Mary University of London, 2010.
  * [38] E. Jang, S. Gu, and B. Poole, “Categorical reparameterization with Gumbel-Softmax,” in _proceedings of the International Conference on Learning Representations (ICLR)_ , 2017.
  * [39] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _proceedings of the International Conference on Learning Representations (ICLR)_ , 2015, pp. 1–15.
  * [40] A. Graves, N. Jaitly, and A. Mohamed, “Hybrid speech recognition with deep bidirectional LSTM,” in _2013 IEEE Workshop on Automatic Speech Recognition and Understanding_ , Dec 2013, pp. 273–278.
  * [41] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” _arXiv preprint arXiv:1607.06450_ , 2016.
  * [42] M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka, “RWC music database: Popular, classical, and jazz music databases,” in _proceedings of the International Society for Music Information Retrieval (ISMIR)_ , 2002, pp. 287–288.
  * [43] A. Berenzweig, B. Logan, D. Ellis, and B. Whitman, “A large-scale evaluation of acoustic and subjective music-similarity measures,” _Computer Music Journal_ , vol. 28, no. 2, pp. 63–76, 2004.
  * [44] J. A. Burgoyne, J. Wild, and I. Fujinaga, “An expert ground truth set for audio chord recognition and music analysis,” in _proceedings of the International Society for Music Information Retrieval (ISMIR)_ , 2011, pp. 633–638.
  * [45] C. Raffel, B. McFee, E. J. Humphrey, J. Salamon, O. Nieto, D. Liang, and D. P. W. Ellis, “mir_eval: A transparent implementation of common MIR metrics,” in _proceedings of the International Society for Music Information Retrieval (ISMIR)_ , 2014, pp. 367–372.


disposition

# Running on Raygun

Alexander Hirsch  Peter Thoman

###### Abstract

With the introduction of Nvidia RTX hardware, ray tracing is now viable as a
general real time rendering technique for complex 3D scenes. Leveraging this
new technology, we present Raygun, an open
source111https://github.com/W4RH4WK/Raygun (MIT license) rendering,
simulation, and game engine focusing on simplicity, expandability, and the
topic of ray tracing realized through Nvidia’s Vulkan ray tracing extension.

##  1 Introduction

In the beginning, Nvidia RTX [7] could only be used via Nvidia OptiX [6],
their dedicated ray tracing application framework, or via Microsoft DXR [5].
As many developers were hoping for a more open approach to this technology,
Nvidia released an (experimental) extension for Vulkan [2] with their 411.63
graphics driver [9]. This advancement in technology is what makes Raygun
possible.

At the time consumers got their hands on Nvidia RTX capable graphics cards,
none of the modern, publicly accessible video game engines allowed using
Nvidia RTX for the _whole_ rendering process. Initially we looked at
Unity222https://unity.com, Unreal333https://www.unrealengine.com, and
Godot444https://godotengine.org just to find out that RTX is not yet available
in these engines. Unity’s engine is closed source, Unreal is far too large for
us to add RTX support in a tangible amount of time, and Godot has not made the
transition (from OpenGL) to Vulkan yet. This spawned the incipient idea of
Raygun. While Quake II RTX [1], an implementation of RTX ray tracing using the
Quake II engine, already existed at this time, we decided to create our own
engine using C++ and modern libraries, focusing on a slim core with
expandability in mind.

Nvidia RTX ray tracing works in two steps. First, a two-layered data structure
is instantiated with the vertex information (and transformation matrices) of a
scene’s 3D models. Next, a dispatch call initiates the ray tracing process
causing the execution of a ray generation shader. This shader controls the
initial casting of rays. Different shaders are invoked depending on whether a
ray intersects with any triangle (closest hit shader and any hit shader) or
not (miss shader). This is done for each ray. The invoked shader has the
option to spawn additional rays, continuing the ray tracing process555The
number of these recursive steps is limited by hardware parameters and ray
payload size.. Each ray can carry a payload (limited, user-defined data)
between shaders.

##  2 Architecture

Raygun’s main goal is to combine a Vulkan-based ray tracer with other systems
typically found in video game engines (e.g. scene graph, physics engine, audio
system). It is designed to be used as malleable framework for creating simple,
ray traced 3D video games. The division between game code and engine code can
be maintained, as suggested by the file structure666See subfolders raygun vs.
example..

Abstractions over systems are kept to a minimum so that access to a system’s
more exotic features is retained. For instance, we expose the underlying
objects of the physics engine so that the physics engine’s full potential can
be leveraged without adjusting multiple layers of abstractions. This benefits
not only rapid prototyping, but also enables one to swap out systems with
little effort at the start of a new project (e.g. replacing Nvidia
PhysX777https://developer.nvidia.com/physx-sdk with
Bullet888https://github.com/bulletphysics/bullet3).

As Vulkan is cross-platform, Raygun targets Windows as well as Linux. Third-
party libraries have been selected accordingly. CMake999https://cmake.org is
used as build system.

###  2.1 Render System

In addition to the commonly used C API, the Khronos Group also provides a C++
API [3]. Although using the C++ API can introduce breaking changes across
Vulkan SDK versions, it is still preferred to the C API due to Vulkan being a
low-level API. Rendering a simple cube in Vulkan requires a moderate amount of
code (compared to high-level APIs like DirectX or OpenGL) [8]. Code becomes
more manageable with the C++ API as language features like
RAII101010https://en.wikipedia.org/wiki/Resource_acquisition_is_initialization,
generic programming, and exceptions are utilised. While implementing the
render system, we introduced thin wrappers around commonly used Vulkan objects
(e.g. buffers, descriptor sets) to improve the developer experience even more.

The ray tracer is based on the official Nvidia Vulkan ray tracing tutorial
[4]. The code has been rewritten using Vulkan’s C++ API and now supports
rendering multiple objects. To highlight the benefits of ray tracing over
rasterization we extended the shader code to simulate reflection and
refraction in addition to shadow casting.

But Vulkan is not just a 3D graphics API, it also provides support for compute
workloads. Considering this, a compute system has been integrated into the
engine alongside the render system. Putting this compute system to use, we
also added screen-space roughness approximation and fast approximate anti-
aliasing (FXAA).

Shaders are implemented using OpenGL shading language (GLSL) and compiled to
SPIR-V during built-time. Additionally, shaders can be tweaked, recompiled,
and reloaded during runtime.

A profiler has been fitted into the engine, using Dear
ImGui111111https://github.com/ocornut/imgui for visualisation. Material
properties, used by shaders, can be manipulated in real time via an on-screen
material editor.

Due to the complexity of Vulkan’s API, introducing bugs is quite likely. To
contain this trouble, the Vulkan SDK features a validation layer that
intercepts and checks Vulkan API calls at runtime. This mechanism is enabled
in Raygun debug builds. Note that, Vulkan also allows one to tag Vulkan
objects with a custom string so they can be identified with ease in profilers
and logging output.

###  2.2 Other Systems

Nvidia PhysX is used as physics engine because of its industry roots, license
(BSD 3), and visual debugger support. We added a simple mechanism to register
trigger and collision call-backs for physics actors and also forward log
messages to the Raygun-wide logging infrastructure121212Using spdlog, see
https://github.com/gabime/spdlog.

An OpenAL131313https://www.openal.org based audio system allows playing of
positional sound effects as well as background music. Audio sources are
manipulated through a thin wrapper class as OpenAL uses a very primitive C
API. Sound files are Opus141414https://opus-codec.org encoded and stored using
the Ogg151515https://www.xiph.org/ogg container format.

The aforementioned systems are connected via a basic scene graph. This scene
graph is to be used as tree of entities (i.e. game objects) where each entity
holds instance specific data for all systems (e.g. model and materials used
for rendering). Strictly speaking, Raygun does not use an entity component
system (ECS)161616https://en.wikipedia.org/wiki/Entity_component_system
architecture, we just adopted the terms entity and system.

For in-game user interfaces we came up with a UI generator that loosely
follows ImGui’s API. Predefined widgets (e.g. labels, buttons, sliders) are
programmatically composed and automatically arranged. All UI widgets are ray
traced 3D objects — no textures needed.

Window management and input processing is handled through
GLFW171717https://www.glfw.org. Mouse and gamepads are supported, despite
using only keyboard input in the provided example application.

3D models are loaded from COLLADA181818https://en.wikipedia.org/wiki/COLLADA
files using custom JSON documents for material properties. To minimize
material data duplication, materials can inherit properties from other
materials.

A dedicated resource manager allows caching of loaded resources.

##  3 Future Work

Raygun is considered feature complete and enables one to experience ray
tracing in a developer friendly environment. Some parts of the code-base may
receive additional clean-up and code-quality improvements in the near future.
We intend to keep the engine up-to-date with respect to the Vulkan SDK,
leveraging new features given their applicability.

The Khronos Group is working on a vendor independent ray tracing extension for
Vulkan. [10] As this addition will be based on the currently used,
experimental Nvidia extension, migration is expected to be straightforward.
Using a vendor independent extension empowers Raygun to also run on non-Nvidia
platforms.

C++20 will bring lots of interesting features to the language, giving us
additional opportunities to further improve code-quality. Especially modules,
designated initializers, and ranges promise to greatly enhance code
composition. Coroutines, although stack-less, could facilitate the creation of
a scripting system using C++ as embedded domain-specific language (eDSL).

## References

  * Burnes [2019] Andrew Burnes.  Quake II RTX: Re-Engineering a Classic with Ray Tracing Effects on Vulkan, 2019.
https://www.nvidia.com/en-us/geforce/news/quake-ii-rtx-ray-tracing-vulkan-
vkray-geforce-rtx.

  * Khronos Group [2016] Khronos Group.  Vulkan, 2016.
https://www.khronos.org/vulkan.

  * Khronos Group [2019] Khronos Group.  Open-Source Vulkan C++ API, 2019.
https://github.com/KhronosGroup/Vulkan-Hpp.

  * Lefrançois and Gautron [2019] Martin-Karl Lefrançois and Pascal Gautron.  NVIDIA Vulkan Ray Tracing Tutorial, 2019.
https://developer.nvidia.com/rtx/raytracing/vkray.

  * Microsoft D3D Team [2018] Microsoft D3D Team.  Announcing Microsoft DirectX Raytracing!, 2018.
https://devblogs.microsoft.com/directx/announcing-microsoft-directx-
raytracing.

  * Nvidia [2018a] Nvidia.  NVIDIA OptiX Ray Tracing Engine, 2018a.
https://developer.nvidia.com/optix.

  * Nvidia [2018b] Nvidia.  NVIDIA RTX platform, 2018b.
https://developer.nvidia.com/rtx.

  * Overvoorde [2019] Alexander Overvoorde.  Vulkan Tutorial, 2019.
https://vulkan-tutorial.com.

  * Subtil [2018] Nuno Subtil.  Introduction to Real-Time Ray Tracing with Vulkan, 2018.
https://devblogs.nvidia.com/vulkan-raytracing.

  * Subtil [2019] Nuno Subtil.  Bringing Ray Tracing to Vulkan, 2019.
https://www.youtube.com/watch?v=xpxVAoXaVgg.