# Amplitude analysis of four-body decays using a massively-parallel fitting
framework

C Hasse1,2,3 J Albrecht2 A A Alves Jr.3 P d’Argent4 T D Evans5 J Rademacker6
and M D Sokoloff3 1 CERN, CH-1211 Geneva 23, Switzerland  2 Experimentelle
Physik V, Technische Universität Dortmund, Otto-Hahn-Straße 4, 44227 Dortmund,
Germany  3 Physics Department, University of Cincinnati, 2600 Clifton Ave.
Cincinnati, OH 45221, USA  4 Physikalisches Institut, Universität Heidelberg,
Im Neuenheimer Feld 226, 69120 Heidelberg, Germany  5 Department of Physics,
University of Oxford, Parks Road, Oxford OX1 3PU, UK  6 School of Physics,
University of Bristol, Tyndall Avenue, Bristol BS8 1TL, UK
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>and
<EMAIL_ADDRESS>
###### Abstract

The GooFit Framework is designed to perform maximum-likelihood fits for
arbitrary functions on various parallel back ends, for example a GPU. We
present an extension to GooFit which adds the functionality to perform time-
dependent amplitude analyses of pseudoscalar mesons decaying into four
pseudoscalar final states. Benchmarks of this functionality show a significant
performance increase when utilizing a GPU compared to a CPU. Furthermore, this
extension is employed to study the sensitivity on the $\PDzero-\APDzero$
mixing parameters $x$ and $y$ in a time-dependent amplitude analysis of the
decay $\PDzero\rightarrow\PKp\Ppiminus\Ppiplus\Ppiminus\ $. Studying a sample
of $50\,000\text{\,}\text{/}$ events and setting the central values to the
world average of $x=$0.49\pm 0.15\text{\,}\mathrm{\char 37\relax}\text{/}$$
and $y=$0.61\pm 0.08\text{\,}\mathrm{\char 37\relax}\text{/}$$, the
statistical sensitivities of $x$ and $y$ are determined to be
$\sigma(x)=$0.019\text{\,}\mathrm{\char 37\relax}\text{/}$$ and
$\sigma(y)=$0.019\text{\,}\mathrm{\char 37\relax}\text{/}$$.

##  1 Introduction

In physics analyses it is common to fit a theoretical model to observed data
to extract parameters of interest. This involves minimizing the differences
between a model and data, which is mostly done by performing a minimization of
a cost function, for example the likelihood function. However, problems arise
because the computations become very expensive as the complexity of the models
and number of events increases. The GooFit [1, 2, 3] framework has been
designed to address this issue by allowing such computations to be performed
in parallel. It is built upon the Thrust library [4] to be able to run on
different parallel architectures, while maintaining a control flow similar to
the RooFit package [5], which is commonly used in high energy physics to fit
theoretical models to data, and which only runs on CPUs. While GooFit has been
successfully employed in several analyses, even for complex models such as
time-dependent mixing in three-body decays, it did not allow for performing a
time-dependent amplitude analyses of four-body decays. This functionality was
recently added and will be described in this paper.

##  2 Mixing in the decay $\PDzero\rightarrow\PKp\Ppiminus\Ppiplus\Ppiminus\ $

Mixing or oscillation of neutral mesons is a process during which a particle
transitions into its antiparticle or vice versa. This process has been
observed in the , , and systems. The system is the only one comprised of up-
type quarks.

$\mathcal{A}_{f}$Mixing$\mathcal{\bar{A}}_{f}$$\PKp\pi^{+}\pi^{-}\pi^{-}$
Figure 1: Schematic view of the two possible decay paths for a decaying into a
final state. The top path corresponds to the direct decay, while the bottom
path shows the mixing transition of a into a followed by a decay into the
final state.

One possible decay to study the phenomenon of mixing in the neutral charm
meson system is the decay of to $\PKp\Ppiminus\Ppiplus\Ppiminus$. This decay
can proceed via two different decay amplitudes, which are depicted in figure
1. The top arrow depicts the direct decay subscribed $\mathcal{A}_{f}$, while
the bottom arrow represents the decay proceeding via mixing into a which
decays into the final state via an amplitude subscribed
$\mathcal{\bar{A}}_{f}$. Due to the mixing of a into a being time-dependent,
the overall decay rate becomes time-dependent. Analysing such time-dependent
decay rates allows extraction of mixing properties of the system.The
expression for the time-dependent decay rate of the , assuming no CP
violation, can be derived to be [6],

$\displaystyle\begin{split}\frac{d\Gamma(\mathcal{A}_{f})}{e^{-\Gamma
t}\mathcal{N}_{f}}=&\left(\left|\mathcal{A}_{f}\right|^{2}+\left|\mathcal{\bar{A}}_{f}\right|^{2}\right)\cosh(y\Gamma
t)+\left(\left|\mathcal{A}_{f}\right|^{2}-\left|\mathcal{\bar{A}}_{f}\right|^{2}\right)\cos(x\Gamma
t)\\\ -&2\Re\left(\mathcal{A}_{f}\mathcal{\bar{A}}_{f}^{*}\right)\sinh(y\Gamma
t)-2\Im\left(\mathcal{A}_{f}\mathcal{\bar{A}}_{f}^{*}\right)\sin(x\Gamma
t)\,.\end{split}$ (1)

Most of the complexity of this expression lies within the model used to
describe the two amplitudes $\mathcal{A}_{f}$ and $\mathcal{\bar{A}}_{f}$

##  3 Structure and implementation of four-body amplitudes

While equation 1 is completely general, the amplitudes that encode the
properties of the decay are functions of the position in phase space occupied
of the final state of the decay. The amplitude structure of a four-body decay
is significantly more complicated than that of three-body decays because their
phase space is five dimensional while three-body decays merely occupy a two-
dimensional phase space.

Similar to other amplitude models, the implemented functionality assumes that
multi-body decays mostly proceed via quasi two-body processes, which include
two-body resonances.

Figure 2: Possible quasi two-body decay topologies of a four-body decay. Left,
a meson decays into two resonances $R_{1}$ and $R_{2}$ , which decay into two
particles each. Right, a meson decays into a particle $a$ and a resonance
$R_{1}$ , which proceeds to decay into a resonance $R_{2}$ and a final state
particle. $R_{2}$ then decays into the remaining two final state particles.

This leads to two possible decay chain topologies depicted in figure 2, where
$R_{1}$ and $R_{2}$ are intermediate resonances and $a,b,c$ and $d$ are the
four final decay products, in various configurations. Here, $R_{1}$ and
$R_{2}$ can take the form of multiple kinematically allowed resonance states,
resulting in many possible decay chains. A complete amplitude will therefore
be modelled by a coherent sum over these decay chains $\mathcal{A}_{i}$ as,

$\mathcal{A}_{f}=\sum_{i}c_{i}\mathcal{A}_{i}\,,\qquad
c_{i},\mathcal{A}_{i}\in\mathbb{C}.$ (2)

Each decay chain $\mathcal{A}_{i}$ is constructed by the user from classes
representing form factors, spin factors, resonance lineshapes, and possibly,
in the case of two identical final state particles, Bose-symmetrization. After
successfully constructing all necessary decay chains the user constructs two
amplitude class instances representing $\mathcal{A}_{f}$ and
$\mathcal{\bar{A}}_{f}$, which each hold the necessary decay chains to fit the
theoretical model. The model creation is finalized by creating an instance of
the time-dependent amplitude model class and passing the two amplitudes just
created by the user. Upon creation the time-dependent model class
automatically checks for recurring form factors, spin factors, and lineshapes
in all decay chains. In case of multiple occurrences, these instances are
substituted by a link to a single instance, thus removing redundant
calculations. The proceeding steps of the internal model building process are
explained in detail in [1, 2].

###  3.1 Normalization and event generation

During the fitting procedure the complete expression in equation 1 must be
normalized accurately. As it is not feasible to find an analytic expression
for such a complex function, the normalization is computed numerically. In our
study, this requires evaluating the function at several million phase space
points. To achieve a sufficiently fast generation of phase space events, we
integrated the MCBooster library [7, 8], which allows very fast generation of
phase space events on the GPU. This also enables the generation of pseudo-
events, which are uniformly distributed phase space events weighted by the
previously created amplitude model.

###  3.2 Validation

As this work implemented various new building blocks to model four-body decay
amplitudes in GooFit it was important to validate the correctness of each of
these new components. A cross check of the implementation was performed by
comparing the newly implemented functionality of GooFit to the software
package MINT3 [9]. MINT3 is based upon the MINT (Minuit Interface) package
[10], which is used to perform time-integrated amplitude analyses of three-
and four-body decays. Additionally, it supports the generation of pseudo-
events. We generate $500,000$ pseudo-events for a specific amplitude model,
which includes all newly implemented building blocks, and compare the
resulting event samples. This comparison is performed by studying the phase
space projections of the samples given the five variables
$m_{12},m_{34},\cos_{12},\cos_{34}$ and $\phi$, where the subscript 12 refers
to the pair and 34 to the pair.

Figure 3: Comparison between the generated pseudo events from the MINT3 (dots)
and GooFit (solid) frameworks. Shown are the distributions of the five
variables used to parametrize the phase space. Additionally, the normalized
pull distributions and p-value are shown. The pulls should follow a normal
distribution with a mean of zero (blue line) and a standard deviation of one.
The red lines mark the 2$\sigma$ region.

As shown in figure 3, there are no significant differences observed and the
pull distribution as well as the p-value indicate that both samples are drawn
from the same distribution.

##  4 Statistical sensitivity to the charm mixing parameters $x$ and $y$

The novel functionality of GooFit has successfully been used to determine the
statistical sensitivity on the charm mixing parameters $x$ and $y$ in a time-
dependent amplitude analysis of the decay
$\PDzero\rightarrow\PKp\Ppiminus\Ppiplus\Ppiminus\ $. This study did not
account for resolution effects, background in the data, and did not allow the
model to float. Therefore, the real sensitivity will be worse than shown in
table 1. Nevertheless, this study proves the capabilities of the newly
implemented extension in GooFit to be fully functional.

Table 1: Summary of the obtained statistical sensitivities of $x$ and $y$ in the case of $x=$0.49\text{\,}\mathrm{\char 37\relax}\text{/}$$ and $y=$0.61\text{\,}\mathrm{\char 37\relax}\text{/}$$ [11]. Events | Sensitivity of $x$ $[$\text{\,}\mathrm{\char 37\relax}\text{/}$]$ | Sensitivity of $y$ $[$\text{\,}\mathrm{\char 37\relax}\text{/}$]$
---|---|---
20000 | $0.030\text{\,}\text{/}$ | $0.031\text{\,}\text{/}$
50000 | $0.019\text{\,}\text{/}$ | $0.019\text{\,}\text{/}$
70000 | $0.016\text{\,}\text{/}$ | $0.017\text{\,}\text{/}$

##  5 Performance comparison between CPU and GPU

Lastly, we present a performance comparison of the newly implemented
functionality, between the CPU and GPU. Two different test cases are used to
study the performance. The first one targets the generation speed of pseudo-
events according to a time-dependent amplitude-model. This generation is
repeated for three different sample sizes to study the scaling behavior.
Secondly, the performance of the fitting procedure is studied, where the
scaling behavior is studied by increasing the number of used events in the
normalization while leaving the sample size one fits to constant.

These tests are repeated on three different platforms: a server with two Intel
Xeon E5-2680 v3 CPUs, each with 12 physical cores that can run two concurrent
threads, a NVIDIA K40 GPU and a mid-range mobile gaming GPU NVIDIA GeForce GT
525M. The results are obtained by an average over 5 runs, and listed in tables
2 and 3. They show a significant speedup when utilizing the K40 and even the
outdated mid-range mobile graphics card was able to perform surprisingly well
compared to the other two platforms, but due to insufficient memory it was not
able to complete all tests.

Table 2: Pseudo-event generation according to a time-dependent model using a Monte-Carlo accept/reject method. Events | 2 $\times$ Intel Xeon | NVIDIA
---|---|---
E5-2680 v3 2.50GHz | GT 525M | K40
24 Cores | 48 Cores | 96 Cores | 2880 Cores
$20\,000\text{\,}\text{/}$ | $179.7\text{\,}\mathrm{s}\text{/}$ | $156.7\text{\,}\mathrm{s}\text{/}$ | $195.8\text{\,}\mathrm{s}\text{/}$ | $25.0\text{\,}\mathrm{s}\text{/}$
$50\,000\text{\,}\text{/}$ | $451.9\text{\,}\mathrm{s}\text{/}$ | $378.7\text{\,}\mathrm{s}\text{/}$ | $484.6\text{\,}\mathrm{s}\text{/}$ | $58.8\text{\,}\mathrm{s}\text{/}$
$70\,000\text{\,}\text{/}$ | $598.0\text{\,}\mathrm{s}\text{/}$ | $524.0\text{\,}\mathrm{s}\text{/}$ | $677.4\text{\,}\mathrm{s}\text{/}$ | $79.0\text{\,}\mathrm{s}\text{/}$

Table 3: Fit to $100\,000\text{\,}\text{/}$ generated pseudo-events, with
varying number of points used to calculate the normalization. Fixed model,
floating $x$ and $y$.

Points | 2 $\times$ Intel Xeon | NVIDIA
---|---|---
E5-2680 v3 2.50GHz | GT 525M | K40
24 Core | 48 Cores | 96 Cores | 2880 Cores
$750\,000\text{\,}\text{/}$ | $4.2\text{\,}\mathrm{s}\text{/}$ | $3.3\text{\,}\mathrm{s}\text{/}$ | $8.2\text{\,}\mathrm{s}\text{/}$ | $0.6\text{\,}\mathrm{s}\text{/}$
$1\,500\,000\text{\,}\text{/}$ | $7.7\text{\,}\mathrm{s}\text{/}$ | $6.5\text{\,}\mathrm{s}\text{/}$ | - | $1.0\text{\,}\mathrm{s}\text{/}$
$3\,000\,000\text{\,}\text{/}$ | $14.8\text{\,}\mathrm{s}\text{/}$ | $12.4\text{\,}\mathrm{s}\text{/}$ | - | $1.7\text{\,}\mathrm{s}\text{/}$
$6\,000\,000\text{\,}\text{/}$ | $30.0\text{\,}\mathrm{s}\text{/}$ | $22.5\text{\,}\mathrm{s}\text{/}$ | - | $3.2\text{\,}\mathrm{s}\text{/}$

While the non-linear scaling from 24 to 48 cores was expected as one only
increases the logical number of cores by running two threads per core, the
expected performance gain from the K40 compared to the GT 525M was less than a
priori expected. Using the available NVIDIA profiler, we are able to determine
that the source of the throttled performance on the K40 is due to memory
latency. We hope to reduce this in the future by reducing the used memory as
well adapting the current memory layout to make memory transfers more
efficient.

##  6 Summary

In conclusion, we have presented a novel extension to the GooFit framework
which allows for performing a time-dependent amplitude analysis of a
pseudoscalar meson decaying into four pseudo-scalar final states.
Additionally, this extension allows the user to generate pseudo-events
according to a previously defined time-dependent amplitude model. This
functionality was successfully validated by comparing the results to an
existing software package and furthermore used to study the sensitivity to the
charm mixing parameters in the decay
$\PDzero\rightarrow\PKp\Ppiminus\Ppiplus\Ppiminus\ $. Lastly, it is shown that
there is a significant speedup gained by utilizing the GPU, while an even
bigger performance gain is forseen once the memory layout in GooFit has been
adapted to minimize memory latency on high performance GPUs like the K40.

The GooFit package can be found on GitHub at https://github.com/GooFit I would
like to thank the authors and maintainers of the MINT and MINT3 framework, P.
d’Argent, T.D. Evans and J. Rademacker, as their work and support has been
most helpful in implementing the presented extension to GooFit.

Work sponsored by the Wolfgang Gentner Programme of the Federal Ministry of
Education and Research.

The development of this extension has been in part supported by the National
Science Foundation under grant number PHY-1414736.

NVidia provided K40 GPUs for our use through its University Partnership
program.

## References

  * [1] Andreassen R, Meadows B T, de Silva M, Sokoloff M D and Tomko K 2014 Journal of Physics: Conference Series 513 052003
  * [2] Andreassen R E, de Silva W M, Meadows B T, Sokoloff M D and Tomko K A 2014 IEEE Access 2 160–176
  * [3] The GooFit package https://github.com/GooFit/GooFit
  * [4] The Thrust library https://thrust.github.io/
  * [5] Verkerke W and Kirkby D P 2003 eConf C0303241 (Preprint physics/0306116)
  * [6] Sozzi M S 2008 Discrete symmetries and CP violation: From experiment to theory
  * [7] The MCBooster library https://github.com/MultithreadCorner/MCBooster
  * [8] Alves Jr A A et al. 2016 MCBooster: a library for fast Monte Carlo generation of phase-space decays in massively parallel platforms. http://indico.cern.ch/event/505613/contributions/2230884/
  * [9] LHCb Collaboration Analysis in progress
  * [10] Artuso M et al. (CLEO) 2012 Phys. Rev. D85 122002 (Preprint 1201.5716)
  * [11] Amhis Y et al. (Heavy Flavor Averaging Group (HFAG)) 2014 Updated results and plots available at http://www.slac.stanford.edu/xorg/hfag/. (Preprint 1412.7515)


# Conditional WaveGAN

Chae Young Lee 11footnotemark: 1
HAFS
<EMAIL_ADDRESS>Anoop Toffy
IIIT Bangalore
<EMAIL_ADDRESS>
Gue Jun Jung
SK Telecom
<EMAIL_ADDRESS>Woo-Jin Han
Netmarble IGS
<EMAIL_ADDRESS>These two authors contributed equally

###### Abstract

Generative models are successfully used for image synthesis [11] in the recent
years. But when it comes to other modalities like audio, text etc little
progress has been made. Recent works focus on generating audio from a
generative model in an unsupervised setting. We explore the possibility of
using generative models conditioned on class labels. Concatenation based
conditioning and conditional scaling were explored in this work with various
hyper-parameter tuning methods [22] [15]. In this paper we introduce
Conditional WaveGANs (cWaveGAN). Find our implementation at
https://github.com/acheketa/cwavegan

##  1 Introduction

Generative adversarial networks [8] are being widely used for synthesizing
realistic images [13, 20, 1]. But very little has been explored in the area of
audio generation. A few research works has been made in the area of
unsupervised generative models in audio. One of them is WaveGAN [5], which
trains a generative model in an unsupervised setting. In this work we use
WaveGAN model as our baseline model. Audio samples generated from WaveGAN are
human-recognizable and have a relatively good inception scores [2]. But the
samples generated are completely random.

The need for generating synthetic data has many applications in various
fields. For example, in digital image enhancement, synthesizing new images can
be used for smoothing (denoising), filling missing pieces (inpainting),
improving resolution (super-resolution imaging) etc. Other areas include
reinforcement learning, where agents can be trained with more training
examples, and automatic speech recognition, where the synthesized data from
generative models can be used as training data for training acoustic models.
More direct usage is in the field of semi-supervised learning techniques [23,
18, 22]. In the recent 2018 ACM webinar [12] Ian Goodfellow talks about the
semi-supervised learning techniques and their applications in various fields.

In this work we explore a way to generate audio samples conditioned on class
labels. That is, given a class label whether the generator of GAN can generate
a particular audio waveform. In the history of GAN such type of conditioning
has been explored in the synthesis of images. Conditional Generative
Adversarial Nets introduced in 2016 by Mehdi Mirza et. al [17] was the first
to try concatenation based conditioning. The conditional GANs were able to
synthesize realistic images on MNIST and MIR Flickr datasets.

##  2 Related Work

Ever since the introduction of Generative Adversarial Networks by Ian
Goodfellow et. al. [8] in 2014 a lot of varieties of GANs has been implemented
for accomplishing various tasks. It is one of the much explored generative
models when compared to Variational Autoencoders (VAEs) and autoregressive
models, assisted by a great tutorial given by Ian Goodfellow in NIPS 2016 [7].
Most of the notable advancements were made in the field of image synthesis. In
his 2018 paper, He Huang et al. [11] talks more about the usage of GANs for
image synthesis and various techniques in this fields.

Usage of generative models in natural language processing is also in progress
with notabld fields including text modelling [28, 21], dialogue generation
[16], and neural machine translation [27]. When it comes to audio generation,
several approaches have been explored. Nonetheless, models that can generate
audio waveforms directly (as opposed to some other representation that can be
converted into audio afterwards, such as spectrograms or piano rolls) are only
starting to be explored. Autoregressive models were initially used for the
generation of raw audio. WaveNet by Van Den Oord et al. [25] is a
convolutional model with dilated convolutions that learns to generate raw
audio by autoregressive modelling. While a single WaveNet can capture the
characteristics of many different speakers with equal fidelity and can switch
between them by conditioning on the speaker identity, WaveRNN [14] describes a
single-layer recurrent neural network with a dual softmax layer that matches
the quality of the state-of-the-art WaveNet model. WaveNets have been applied
to music generation as well. The paper by Sander Dieleman et al. (2018) [4]
talks about the challenges in realistic music generation.

The application of GAN in speech were used mainly for audio enhancements
techniques and little on raw audio generation. SEGAN by Santiago Pascual et al
[19] uses deep networks which operates on waveform level, training the model
end-to-end for denoising waveform chunks. The model works as a fully
convolutional encoder-decoder structure. Tacotron [26] is an end-to-end
generative text-to-speech model that synthesizes speech directly from
characters. Since Tacotron generates speech at the frame level, it’s
substantially faster than sample-level autoregressive methods. In Tacotron2
[24] a neural network architecture synthesizes speech directly from the text.
Tacotron2 achieves a mean opinion score (MOS) of 4.53, whereas a
professionally recorded speech obtained a MOS of 4.58.

In the recent work on speech synthesis, Chris Donahue et al. (2018) [5]
introduces two GAN models, WaveGAN and SpecGAN. The waveGAN works in the time-
domain and specGAN works in the frequency-domain. WaveGAN can produce
intelligible words from a small vocabulary of human speech, as well as
synthesize audio from other domains such as bird vocalizations, drums, and
piano. It uses GANs in an unsupervised setting. WaveGAN is based on the DCGAN
[20] architecture, which became famous by its usage in image synthesis. We
focus on this work to make the WaveGAN conditioned on class labels by
introducing various conditioning techniques discussed in the following
sections.

##  3 Adversarial Nets

###  3.1 Generative Adversarial Nets

GAN was introduced as a novel method to train generative models. They are
composed of two adversarial models: the generator, G and the discriminator, D.
The generator, G captures the data distribution while the discriminator, D
estimates the probability that the data generated by generator is coming from
the data distribution or not. Both G and D can be non-linear mapping
functions.

To learn the data distribution $P_{D}$ over data $x$, the generator builds a
mapping from a noise distribution $P_{z}$ to data space as $G(z,\theta_{g})$.
And the discriminator $D(x,\theta_{d})$ outputs as scalar representing that
the training data $p_{x}$ rather than $p_{g}$.

${G}$ and ${D}$ are both trained simultaneously: we adjust parameters for
${G}$ to minimize ${log(1-D(G(\boldmath{z}))}$ and adjust parameters for $D$
to minimize ${logD(X)}$, as if they are following the two-player min-max game
with value function ${V(G,D)}$:

$\min_{G}\max_{D}V(D,G)=\mathbb{E}_{\boldmath{x}\sim
p_{\textit{data}}(\boldmath{x})}[\log
D(\boldmath{x})]+\mathbb{E}_{\boldmath{z}\sim
p_{z}(\boldmath{z})}[\log(1-D(G(\boldmath{z})))].$ (1)

###  3.2 Conditional Generative Adversarial Nets

The conditional GANs (cGANs) [17] were introduced to make use of the
additional information available in the form of labels, $y$. So instead of
generating a random data from the generator, the auxiliary information like
labels, $y$ are passed to both the generator and the discriminator along with
the input features, ($z$ and $x$) so as to generate data conditioned on class
labels.

In both the generator and the discriminator, the label information, $y$ is
augmented with the input features ($z$ and $x$).

The objective function of a two-player minimax game would be as Eq. 2.

$\min_{G}\max_{D}V(D,G)=\mathbb{E}_{\boldmath{x}\sim
p_{\textit{data}}(\boldmath{x})}[\log
D(\boldmath{x}|\boldmath{y})]+\mathbb{E}_{\boldmath{z}\sim
p_{z}(\boldmath{z})}[\log(1-D(G(\boldmath{z}|\boldmath{y})))].$ (2)

##  4 Conditioning methodologies

In this work we explore a few conditioning mechanisms discussed in Dumoulin
et. al [6] article in this section. We are interested in generating raw audio
of a particular given class label from the generator output. That is, the
model takes as input a class and a source of random noise (e.g., a vector
sampled from a normal distribution) and outputs a raw audio sample for the
requested class as shown in Fig. 1.

Figure 1: Synthesizing raw audio by conditioning

###  4.1 Concatenation based conditioning

First approach is to embed the class information to the input feature vector.
That is, before training we would concatenate a representation of conditioning
information to the noise vector and use it as the model’s input as shown in
Fig. 2

Figure 2: Concatenation based conditioning

###  4.2 Conditional scaling

In the other type of conditioning we implemented, we scaled the hidden layers
based on the conditioning representation and multiplied it with the input
vector for both the discriminator and the generator input. Also the scaling is
applied to each layer of the convolution model.

Figure 3: Conditional scaling

##  5 Experimental Setting

We based our architecture on WaveGAN model used by Chris Donahue et. al. [5]
which inturn is based on the DCGAN model by Radford et. al [20]. We modified
the input vector dimension to 8192 by removing the beginning and ending
silence to better capture the word alone. As in WaveGAN paper, we used one-
dimensional filters of length 25 instead of two-dimensional filters of size
5x5 used in DCGAN model architecture but changed the strides from 4 to 2 just
to incorporate the change in input vector dimension. The architecture of
generator and discriminator model is provided in Table 5 and Table 5
respectively.

As a pilot study for this project we trained our model with DCGAN [20] and
WAGAN-GP [10] losses with and without batch normalization. We compare two
combinations in the following table 3. We tried with different learning rate
apart from $\alpha=2\mathrm{e}{-4}$, ie with $\alpha=2\mathrm{e}{-2}$ and
$\alpha=4\mathrm{e}{-4}$. We also tried different learning rates with
generator and disriminator since we found discriminator loss graph spiking.

We train our networks using batches of size 64 on four NVIDIA Telsa V100
accelerators and batches of 1024 on a single 8 core TPUv2. We trained it over
200 epochs for around 3 days.

Table 1: Conditional WaveGAN generator architecture

Operation | Kernel Size | Output Shape
---|---|---
Input $\boldmath{z}\sim\textit{Uniform}(-1,1)$ |  | ($n$, $100$)
Dense 1 | ($100$, $256d$) | ($n$, $256d$)
Reshape |  | ($n$, $16$, $16d$)
ReLU |  | ($n$, $16$, $16d$)
Trans Conv1D (Stride=$4$) | ($25$, $16d$, $8d$) | ($n$, $64$, $8d$)
ReLU |  | ($n$, $64$, $8d$)
Trans Conv1D (Stride=$4$) | ($25$, $8d$, $4d$) | ($n$, $256$, $4d$)
ReLU |  | ($n$, $256$, $4d$)
Trans Conv1D (Stride=$4$) | ($25$, $4d$, $2d$) | ($n$, $1024$, $2d$)
ReLU |  | ($n$, $1024$, $2d$)
Trans Conv1D (Stride=$4$) | ($25$, $2d$, $d$) | ($n$, $4096$, $d$)
ReLU |  | ($n$, $4096$, $d$)
Trans Conv1D (Stride=$2$) | ($25$, $d$, $c$) | ($n$, $8192$, $c$)
Tanh |  |
Table 2: Conditional WaveGAN discriminator architecture

Table 3: Conditional WaveGAN hyperparameters Name | Value (GPU) | Value (GPU) | Value (TPU) | Value (TPU)
---|---|---|---|---
Input data type |  $16$-bit PCM |  $16$-bit PCM |  $16$-bit PCM |  $16$-bit PCM
Model data type |  $32$-bit float |  $32$-bit float |  $32$-bit float |  $32$-bit float
Num channels ($c$) | 1 | 1 | 1 | 1
Batch size ($b$) | 64 | 64 | 1024 | 1024
Model size ($d$) | 64 | 64 | 64 | 64
Phase shuffle (WaveGAN) | 2 | 2 | 2 | 2
Loss | WGAN-GP [10] | DCGAN [20] | WGAP-GP | DCGAN
$D$ updates per $G$ update | 5 | 5 | 5 | 5
Optimizer | Adam ($\alpha=1\mathrm{e}{-4}$) | Adam ($\alpha=2\mathrm{e}{-4}$) | Adam ($\alpha=2\mathrm{e}{-4}$) | Adam ($\alpha=2\mathrm{e}{-4}$)

##  6 Dataset

We used the Speech Commands Dataset [3] released by Google AI Team for
conducting our priliminary experiments. This dataset consists of many speakers
recording individual words in uncontrolled recording conditions. We used a
subset of the dataset as done by the authors of WaveGAN paper, ie. Speech
Commands Zero Through Nine (SC09) subset, which reduces the vocabulary of the
dataset to ten words: the digits “zero” through “nine”.

Each recording utterance is of one second in length and the training set
consists of 1850 recording for each word adding upto 5.3hr of speech data. The
wide variety of alignments, speakers and recording conditions make this a
challenging dataset from a generative perspective.

##  7 Preliminary Results

We were able to generate human realistic raw audio from the conditional
WaveGAN. But most of the time we faced the difficulty of having generated
noisy output. The conditioning that we tried was not consistent more often and
we ended up generating a false positive sample. So we will research further to
stabilize the GAN training and to tune our training parameters to generate raw
audio out that are realistic to real world data.

This is a pilot study which explores the possibility of making use of
synthesized data from generative models as training data by other
discriminative models.

##  8 Conclusion

Synthesizing audio is a area that has been growing recently and is much
focused by researcher since it has its application in building robust speech
recognition systems and improved text-to-speech systems. Much research were
focused on generation on random audio from unlabeled training data. In this
work we focused on generation of audio given a class label. We were able to
get human recognizable synthesized audio waveform from conditional waveGANs,
but the presence of noise in the sample generated needs to be smoothed as a
continuation of the present work, which gives a lot of scope for future
improvements. Secondly, hyperparameter tuning of GAN needs to be explored
since presently the training is unstable. Thirdly, the conditioning method
explored so far is increasing training time of GANs, so a more novel
conditioning method need to be discovered.

We hope this work gives direction to more research on conditional generation
of audio wave-forms and gets extended to other frontiers.

#### Acknowledgments

We would like thank Tensorflow Korea and Google Korea, and all other sponsors
of Deep Learning Camp Jeju 2018, http://jeju.dlcamp.org/2018/. This work was
done as a part of the DL Jeju Camp. We would like to convey our sincere
gratitude to our mentors, Dr.Gue Jun Jung (SK Telecom), Dr. Woo-Jin Han
(Netmarble IGS) and all other men-tees of the camp.

We are also delighted to use TPUs [9] for our training process which made a
considerable improvement in making the training process faster.

## References

  * Agustsson et al. [2018] E. Agustsson, M. Tschannen, F. Mentzer, R. Timofte, and L. Van Gool.  Generative adversarial networks for extreme learned image compression.  _arXiv preprint arXiv:1804.02958_ , 2018.
  * Barratt and Sharma [2018] S. Barratt and R. Sharma.  A note on the inception score.  _arXiv preprint arXiv:1801.01973_ , 2018.
  * Buchner [2017] J. Buchner.  Synthetic speech commands dataset.  _https://www.kaggle.com/jbuchner/synthetic-speech-commands-dataset_ , 2017\.
  * Dieleman et al. [2018] S. Dieleman, A. v. d. Oord, and K. Simonyan.  The challenge of realistic music generation: modelling raw audio at scale.  _arXiv preprint arXiv:1806.10474_ , 2018.
  * Donahue et al. [2018] C. Donahue, J. McAuley, and M. Puckette.  Synthesizing audio with generative adversarial networks.  _arXiv preprint arXiv:1802.04208_ , 2018.
  * Dumoulin et al. [2018] V. Dumoulin, E. Perez, N. Schucher, F. Strub, H. d. Vries, A. Courville, and Y. Bengio.  Feature-wise transformations.  _Distill_ , 3(7):e11, 2018.
  * Goodfellow [2016] I. Goodfellow.  Nips 2016 tutorial: Generative adversarial networks.  _arXiv preprint arXiv:1701.00160_ , 2016.
  * Goodfellow et al. [2014] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial nets.  In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, _Advances in Neural Information Processing Systems 27_ , pages 2672–2680. Curran Associates, Inc., 2014.  URL http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf.
  * Google [2016] Google.  Cloud tpu.  _https://cloud.google.com/tpu/_ , 2016.
  * Gulrajani et al. [2017] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville.  Improved training of wasserstein gans.  In _Advances in Neural Information Processing Systems_ , pages 5767–5777, 2017.
  * Huang et al. [2018] H. Huang, P. S. Yu, and C. Wang.  An introduction to image synthesis with generative adversarial nets.  _arXiv preprint arXiv:1803.04469_ , 2018.
  * Ian Goodfellow [2016] G. B. Ian Goodfellow, Staff Research Scientist.  Adversarial machine learning.  _https://goo.gl/NVWF7j_ , 2016.
  * Isola et al. [2017] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros.  Image-to-image translation with conditional adversarial networks.  _arXiv preprint_ , 2017.
  * Kalchbrenner et al. [2018] N. Kalchbrenner, E. Elsen, K. Simonyan, S. Noury, N. Casagrande, E. Lockhart, F. Stimberg, A. v. d. Oord, S. Dieleman, and K. Kavukcuoglu.  Efficient neural audio synthesis.  _arXiv preprint arXiv:1802.08435_ , 2018.
  * Kurach et al. [2018] K. Kurach, M. Lucic, X. Zhai, M. Michalski, and S. Gelly.  The gan landscape: Losses, architectures, regularization, and normalization.  _arXiv preprint arXiv:1807.04720_ , 2018.
  * Li et al. [2017] J. Li, W. Monroe, T. Shi, S. Jean, A. Ritter, and D. Jurafsky.  Adversarial learning for neural dialogue generation.  _arXiv preprint arXiv:1701.06547_ , 2017.
  * Mirza and Osindero [2014] M. Mirza and S. Osindero.  Conditional generative adversarial nets.  _arXiv preprint arXiv:1411.1784_ , 2014.
  * Odena [2016] A. Odena.  Semi-supervised learning with generative adversarial networks.  _arXiv preprint arXiv:1606.01583_ , 2016.
  * Pascual et al. [2017] S. Pascual, A. Bonafonte, and J. Serra.  Segan: Speech enhancement generative adversarial network.  _arXiv preprint arXiv:1703.09452_ , 2017.
  * Radford et al. [2015] A. Radford, L. Metz, and S. Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks.  _arXiv preprint arXiv:1511.06434_ , 2015.
  * Rajeswar et al. [2017] S. Rajeswar, S. Subramanian, F. Dutil, C. Pal, and A. Courville.  Adversarial generation of natural language.  _arXiv preprint arXiv:1705.10929_ , 2017.
  * Salimans et al. [2016] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen.  Improved techniques for training gans.  In _Advances in Neural Information Processing Systems_ , pages 2234–2242, 2016.
  * Schoneveld [2017] L. Schoneveld.  Semi-supervised learning with generative adversarial networks.  2017\.
  * Shen et al. [2017] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerry-Ryan, et al.  Natural tts synthesis by conditioning wavenet on mel spectrogram predictions.  _arXiv preprint arXiv:1712.05884_ , 2017.
  * Van Den Oord et al. [2016] A. Van Den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. W. Senior, and K. Kavukcuoglu.  Wavenet: A generative model for raw audio.  In _SSW_ , page 125, 2016.
  * Wang et al. [2017] Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio, et al.  Tacotron: Towards end-to-end speech synthesis.  _arXiv preprint arXiv:1703.10135_ , 2017.
  * Yang et al. [2017] Z. Yang, W. Chen, F. Wang, and B. Xu.  Improving neural machine translation with conditional sequence generative adversarial nets.  _arXiv preprint arXiv:1703.04887_ , 2017.
  * Yu et al. [2017] L. Yu, W. Zhang, J. Wang, and Y. Yu.  Seqgan: Sequence generative adversarial nets with policy gradient.  In _AAAI_ , pages 2852–2858, 2017.