Introduction to Jet Images and Computer Vision

Tiago Vale - LIP

Adapted from a tutorial by Michela Paganini

Material:
 1. Ben Nachman's plenary talk at ACAT 2017 (slides)
 1. My talk on Generative Adversarial Networks for jet images at the 2017 IML Workshop (video)
 1. Jannicke Pearkes's talk on boosted top tagging with jet images at the 2017 IML Workshop (video)
 1. Michael Kagan's overview talk at LHCP 2017 (slides)
 1. ATLAS PUB Note on quark vs gluon tagging with jet images(note)
 1. Lynn Huynh's summer report on jet image pre-processing (write-up)
 1. Ben Nachman's talk at DataScience@LHC 2015 (video)

# Dataset

[](https://doi.org/10.17632/4r4v785rgx.1)

Although the dataset was released in conjunction with the arXiv publication of our [work on Generative Adversarial Networks for jet images](https://arxiv.org/pdf/1701.05927.pdf), it was previously used in the original "[Jet Images -- Deep Learning Edition](https://link.springer.com/article/10.1007%2FJHEP07%282016%29069)" work on jet image classification. Feel free to explore the dataset and use it for any project you have in mind (please cite the dataset and relevant publications explaining its generation!)

### Download dataset from Zenodo

```python
import os
from keras.utils.data_utils import get_file

# Info for downloading the dataset from Zenodo
MD5_HASH = 'f9b11c46b6a0ff928bec2eccf865ecf0'
DATAFILE = 'jet-images_Mass60-100_pT250-300_R1.25_Pix25.hdf5'
URL_TEMPLATE = 'https://zenodo.org/record/{record}/files/{filename}'

print('[INFO] MD5 verification: {}'.format(MD5_HASH))

datafile = get_file(
    fname=DATAFILE,
    origin=URL_TEMPLATE.format(record=269622, filename=DATAFILE),
    md5_hash=MD5_HASH
)
```

# Jets at the LHC

Jets are the observable result of quarks and gluons scattering at high energy. A collimated stream of
protons and other hadrons forms in the direction of the initiating quark or gluon. Clusters of such
particles are called jets.

# Jet Images

Mature field of research! (image courtesy of B.P.Nachman)

### What is a jet image?

> A jet image is a two-dimensional representation of the radiation pattern within a jet: the distribution of the locations and energies of the jet’s constituent particles. The jet image consists of a regular grid of pixels in η×φ.

Advantages of this data format include: easy visual inspection, fixed-length representation, suitable for application of computer vision techniques.

### Pre-processing

In the dataset we will be using today:

> The finite granularity of a calorimeter is simulated with a regular 0.1×0.1 grid in η and φ. The energy of each calorimeter cell is given by the sum of the energies of all particles incident on the cell. Cells with positive energy are assigned to jets using the anti-kt clustering algorithm with a radius parameter of R = 1.0 via the software package FastJet 3.2.1.

> To mitigate the contribution from the underlying event, jets are are trimmed by re-clustering the constituents into R = 0.3 kt subjets and dropping those which have less than 5% of the transverse momentum of the parent jet. Trimming also reduces the impact of pileup: multiple proton-proton collisions occurring in the same event as the hard-scatter process. Jet images are formed by translating the η and φ of all constituents of a given jet so that its highest pT subjet is centered at the origin.

> A rectangular grid of η × φ ∈ [−1.25, 1.25] × [−1.25, 1.25] with 0.1 × 0.1 pixels centered at the origin
forms the basis of the jet image. The intensity of each pixel is the pT corresponding to the energy
and pseudorapditiy of the constituent calorimeter cell, pT = E_cell/ cosh(η_cell). The radiation pattern
is symmetric about the origin of the jet image and so the images are rotated. The subjet with the
second highest pT (or, in its absence, the direction of the first principle component) is placed at an
angle of −π/2 with respect to the η − φ axes. Finally, a parity transform about the vertical axis is
applied if the left side of the image has more energy than the right side.

  Learning Particle Physics by Example: Location-Aware Generative Adversarial Networks for Physics Synthesis

  [arXiv:1701.05927](https://arxiv.org/pdf/1701.05927.pdf)

References:
* Section 3 of [arXiv:1511.05190](https://link.springer.com/content/pdf/10.1007%2FJHEP07%282016%29069.pdf)
* Appendix B of [arXiv:1701.05927](https://arxiv.org/pdf/1701.05927.pdf)

### Uniqueness with respect to natural images in ML literature

* Sparse (low occupancy)
* High dynamic range (pixel intensity represents pT of particles and spans several orders of magnitude)
* Pixel activations and positions are physically meaningful
* Small variations can drastically modify physical properties of a jet

# Hands-on tutorial

```python
import h5py
import numpy as np
import os
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
import matplotlib.cm as cm
from matplotlib.colors import LogNorm, Normalize
%matplotlib inline
```

```python
# number of images to load
nb_points = 800000
```

```python
# open hdf5 data file
d = h5py.File(datafile, 'r')
```

```python
# content of the dataset
d.items()
```

    ItemsViewHDF5(<HDF5 file "jet-images_Mass60-100_pT250-300_R1.25_Pix25.hdf5" (mode r)>)

```python
# extract a random subset of samples
ix = list(range(d['image'].shape[0])) # get indices
np.random.shuffle(ix) # shuffle them
ix = ix[:nb_points] # select out nb_points

# extract data from dataset
images, labels = d['image'][:][ix], d['signal'][:][ix]
mass = d['jet_mass'][:][ix]
delta_R = d['jet_delta_R'][:][ix]
tau_21 = d['tau_21'][:][ix]
```

You can compute the quantities above directly from the images:

\begin{align}
&p_\text{T}^2(I) =\left(\sum_{i=0}^{N} I_i\cos(\phi_i)\right)^2+\left(\sum_{i=0}^{N} I_i\sin(\phi_i)\right)^2
\label{eq:pt}
\\
&m^2(I) = \left(\sum_{i=0}^{N} I_i\right)^2-p_\text{T}^2(I)-\left(\sum_{i=0}^{N} I_i\sinh(\eta_i)\right)^2
\label{eq:m}
\\
&\tau_{21}(I)=\frac{\tau_2(I)}{\tau_1(I)},
\label{eq:tau21}
\end{align}
where:
\begin{equation*}
\tau_{n}(I)\propto\sum_{i=0}^{N} I_i \min_{1\leq a\leq n}\left\{\sqrt{\left(\eta_i-\eta_a\right)^2+\left(\phi_i-\phi_a\right)^2}\right\}
\end{equation*}

```python
def plot_jet_image(content, output_fname=None, vmin=1e-6, vmax=300, title=''):
    '''
    Function to help you visualize a jet image on a log scale
    Args:
    -----
        content : numpy array of dimensions 25x25, first arg to imshow,
            content of the image
            e.g.: images.mean(axis=0) --> the average image

        output_fname : string, name of the output file where the plot will be
            saved.

        vmin : (default = 1e-6) float, lower bound of the pixel intensity
            scale before saturation

        vmax : (default = 300) float, upper bound of the pixel intensity
            scale before saturation

        title : (default = '') string, title of the plot, to be displayed
            on top of the image
    '''
    fig, ax = plt.subplots(figsize=(7, 6))
    extent = [-1.25, 1.25, -1.25, 1.25]

    im = ax.imshow(content, interpolation='nearest',
                   norm=LogNorm(vmin=vmin, vmax=vmax), extent=extent)

    cbar = plt.colorbar(im, fraction=0.05, pad=0.05)
    cbar.set_label(r'Pixel $p_T$ (GeV)', y=0.85)
    plt.xlabel(r'[Transformed] Pseudorapidity $(\eta)$')
    plt.ylabel(r'[Transformed] Azimuthal Angle $(\phi)$')
    plt.title(title)

    if output_fname is None:
        plt.savefig('output_fname')
```

```python
def plot_diff_jet_image(content, output_fname=None, extr=None, title='',
                        cmap=matplotlib.cm.seismic):
    '''
    Function to help you visualize the difference between two sets of jet
        images on a linear scale
    Args:
    -----
        content : numpy array of dimensions 25x25, first arg to imshow,
            content of the image
            e.g.: sig_images.mean(axis=0) - bkg_images.mean(axis=0)
        output_fname : string, name of the output file where the plot will be
            saved.
        extr : (default = None) float, magnitude of the upper and lower
            bounds of the pixel intensity scale before saturation (symmetric
            around 0)
        title : (default = '') string, title of the plot, to be displayed on
            top of the image
        cmap : (default = matplotlib.cm.PRGn_r) matplotlib colormap, ideally
            white in the middle
    '''
    fig, ax = plt.subplots(figsize=(6, 6))
    extent = [-1.25, 1.25, -1.25, 1.25]
    if extr == None:
        extr = max(abs(content.min()), abs(content.max()))
    im = ax.imshow(
        content,
        interpolation='nearest',
        norm=Normalize(vmin=-extr, vmax=+extr), extent=extent,
        cmap=cmap
    )

    plt.colorbar(im, fraction=0.05, pad=0.05)
    plt.xlabel(r'[Transformed] Pseudorapidity $(\eta)$')
    plt.ylabel(r'[Transformed] Azimuthal Angle $(\phi)$')
    plt.title(title)

    if output_fname:
        plt.savefig(output_fname)

```

```python
# visualize a jet image
plot_jet_image(images[0])
```

```python
# visualize the average jet image
plot_jet_image(images.mean(axis=0))
```

```python
# visualize the difference between the average signal and the average background image
plot_diff_jet_image(
    images[labels == 1].mean(axis=0) - images[labels == 0].mean(axis=0)
)
```

## Looking at Physics features

```python
def plot_physics_feature(feature_name, feature, labels, bins=None, output_fname=None):

    # if bins are not defined when function is called, define them here
    if not bins:
        bins = np.linspace(feature.min(), feature.max(), 50)

    fig, ax = plt.subplots(figsize=(6, 6))

    _ = plt.hist(feature[labels == 1], bins=bins, histtype='step',
                 label=r"Signal ($W' \rightarrow WZ$)",
                 color='red')

    _ = plt.hist(feature[labels == 0], bins=bins, histtype='step',
                 label=r'Background (QCD dijets)', color='blue')

    plt.xlabel(r'Discretized {} of Jet Image'.format(feature_name))
    plt.ylabel(r'Units normalized to unit area')
    plt.legend()

    if output_fname:
        plt.savefig(output_fname)

```

```python
plot_physics_feature(r'$m$', mass, labels)
```

```python
plot_physics_feature(r'$\Delta R$', delta_R, labels)
```

```python
plot_physics_feature(r'$\tau_{2,1}$', tau_21, labels)
```

# Jet Image Classification

We can now try to use various techniques to classify the jet images into signal (i.e. originating from boosted W bosons) and background (QCD).

We will start with a classic feature-based classifier, which will use properties of the jet such as mass, tau_21, and delta_R (known to have good discriminative power) to separate the two classes.

Then, we will construct different networks that operate directly at the pixel level and compare them all.

## Simple feature-based classifier

#### Data processing

Follow the procedure from yesterday to create your matrix of features X. Shuffle its entries, split them into train, test, and validation set, and scale them to zero mean and unit standard deviation.

```python
features = np.stack((mass, tau_21, delta_R)).T # What we called X yesterday
```

```python
features
```

    array([[7.4899811e+01, 1.3971861e-01, 5.7381898e-01],
           [9.6999718e+01, 3.8425145e-01, 7.0187241e-01],
           [8.1882729e+01, 3.6674374e-01, 5.5160850e-01],
           ...,
           [8.1609734e+01, 3.1000379e-01, 6.3720000e-01],
           [7.3569969e+01, 8.4428534e-02, 4.9061748e-01],
           [7.7200653e+01, 1.0726629e-01, 5.4942137e-01]], dtype=float32)

```python
from sklearn.model_selection import train_test_split
```

```python
# 80% train+validate, 20% test
images_train, images_test, \
    labels_train, labels_test, \
    features_train, features_test \
    = train_test_split(images, labels, features,
                       test_size=0.2)

# 64% train, 16% validate
images_train, images_val, \
    labels_train, labels_val, \
    features_train, features_val \
    = train_test_split(images_train, labels_train, features_train,
                       test_size=0.2)
```

```python
print ('{} training samples\n{} validation samples\n{} testing samples'
       .format(images_train.shape[0], images_val.shape[0], images_test.shape[0]))
```

    512000 training samples
    128000 validation samples
    160000 testing samples

```python
from sklearn.preprocessing import StandardScaler
```

```python
scaler = StandardScaler()
features_train = scaler.fit_transform(features_train)
features_val = scaler.transform(features_val)
features_test = scaler.transform(features_test)
```

#### Model

Build a simple keras model made of fully-connected (Dense) layers. Remember the steps:
1. Define the symbolic graph by connecting layers
1. Define an optimizer and a loss function to minimize
1. Train ('fit') the model to the training dataset, monitoring whether the validation loss continues to decrease
1. Stop the training automatically when the validation loss stops going down
1. Evaluate performance on test set

Recall activation functions: Rectified Linear Unit (relu) vs. Leaky Rectified Linear Unit

```python
from keras.layers import Input, Dense, Dropout
from keras.models import Model
from keras.callbacks import EarlyStopping, ModelCheckpoint
```

```python
x = Input(shape=(features_train.shape[1], ))
h = Dense(64, activation='relu')(x)
h = Dense(64, activation='relu')(h)
h = Dense(64, activation='relu')(h)
y = Dense(1, activation='sigmoid')(h)
```

```python
baseline_model = Model(x, y)
baseline_model.compile('adam', 'binary_crossentropy', metrics=['acc'])
```

The command below trains the model. However, in the interest of time, I will not train the network on the spot. I will instead load in pre-trained weights from a training I performed last night.

```python
baseline_model.fit(
    features_train, labels_train, # X and y
    epochs=200,
    batch_size=128,
    validation_data=(features_val, labels_val), # validation X and y
    callbacks=[
        EarlyStopping(verbose=True, patience=15, monitor='val_loss'),
        ModelCheckpoint('./models/baseline-model.h5', monitor='val_loss',
                        verbose=True, save_best_only=True)
    ]
)
```

    Epoch 1/200
    4000/4000 [==============================] - 4s 936us/step - loss: 0.4614 - acc: 0.7879 - val_loss: 0.4503 - val_acc: 0.7918

    Epoch 00001: val_loss improved from inf to 0.45032, saving model to ./models/baseline-model.h5
    Epoch 2/200
    4000/4000 [==============================] - 4s 908us/step - loss: 0.4514 - acc: 0.7925 - val_loss: 0.4509 - val_acc: 0.7923

    Epoch 00002: val_loss did not improve from 0.45032
    Epoch 3/200
    4000/4000 [==============================] - 4s 883us/step - loss: 0.4508 - acc: 0.7924 - val_loss: 0.4493 - val_acc: 0.7931

    Epoch 00003: val_loss improved from 0.45032 to 0.44934, saving model to ./models/baseline-model.h5
    Epoch 4/200
    4000/4000 [==============================] - 4s 881us/step - loss: 0.4511 - acc: 0.7931 - val_loss: 0.4498 - val_acc: 0.7926

    Epoch 00004: val_loss did not improve from 0.44934
    Epoch 5/200
    4000/4000 [==============================] - 4s 915us/step - loss: 0.4508 - acc: 0.7926 - val_loss: 0.4494 - val_acc: 0.7931

    Epoch 00005: val_loss did not improve from 0.44934
    Epoch 6/200
    4000/4000 [==============================] - 4s 919us/step - loss: 0.4498 - acc: 0.7934 - val_loss: 0.4489 - val_acc: 0.7933

    Epoch 00006: val_loss improved from 0.44934 to 0.44887, saving model to ./models/baseline-model.h5
    Epoch 7/200
    4000/4000 [==============================] - 4s 1ms/step - loss: 0.4509 - acc: 0.7934 - val_loss: 0.4490 - val_acc: 0.7927

    Epoch 00007: val_loss did not improve from 0.44887
    Epoch 8/200
    4000/4000 [==============================] - 4s 955us/step - loss: 0.4494 - acc: 0.7937 - val_loss: 0.4493 - val_acc: 0.7925

    Epoch 00008: val_loss did not improve from 0.44887
    Epoch 9/200
    4000/4000 [==============================] - 4s 937us/step - loss: 0.4496 - acc: 0.7934 - val_loss: 0.4491 - val_acc: 0.7932

    Epoch 00009: val_loss did not improve from 0.44887
    Epoch 10/200
    4000/4000 [==============================] - 4s 993us/step - loss: 0.4508 - acc: 0.7926 - val_loss: 0.4495 - val_acc: 0.7929

    Epoch 00010: val_loss did not improve from 0.44887
    Epoch 11/200
    4000/4000 [==============================] - 4s 949us/step - loss: 0.4490 - acc: 0.7935 - val_loss: 0.4500 - val_acc: 0.7928

    Epoch 00011: val_loss did not improve from 0.44887
    Epoch 12/200
    4000/4000 [==============================] - 4s 952us/step - loss: 0.4501 - acc: 0.7929 - val_loss: 0.4487 - val_acc: 0.7932

    Epoch 00012: val_loss improved from 0.44887 to 0.44868, saving model to ./models/baseline-model.h5
    Epoch 13/200
    4000/4000 [==============================] - 4s 970us/step - loss: 0.4500 - acc: 0.7931 - val_loss: 0.4493 - val_acc: 0.7927

    Epoch 00013: val_loss did not improve from 0.44868
    Epoch 14/200
    4000/4000 [==============================] - 4s 940us/step - loss: 0.4496 - acc: 0.7936 - val_loss: 0.4493 - val_acc: 0.7926

    Epoch 00014: val_loss did not improve from 0.44868
    Epoch 15/200
    4000/4000 [==============================] - 4s 941us/step - loss: 0.4499 - acc: 0.7932 - val_loss: 0.4489 - val_acc: 0.7930

    Epoch 00015: val_loss did not improve from 0.44868
    Epoch 16/200
    4000/4000 [==============================] - 4s 941us/step - loss: 0.4504 - acc: 0.7930 - val_loss: 0.4496 - val_acc: 0.7925

    Epoch 00016: val_loss did not improve from 0.44868
    Epoch 17/200
    4000/4000 [==============================] - 4s 939us/step - loss: 0.4502 - acc: 0.7923 - val_loss: 0.4490 - val_acc: 0.7930

    Epoch 00017: val_loss did not improve from 0.44868
    Epoch 18/200
    4000/4000 [==============================] - 5s 1ms/step - loss: 0.4501 - acc: 0.7928 - val_loss: 0.4485 - val_acc: 0.7930

    Epoch 00018: val_loss improved from 0.44868 to 0.44849, saving model to ./models/baseline-model.h5
    Epoch 19/200
    4000/4000 [==============================] - 4s 1ms/step - loss: 0.4495 - acc: 0.7935 - val_loss: 0.4491 - val_acc: 0.7927

    Epoch 00019: val_loss did not improve from 0.44849
    Epoch 20/200
    4000/4000 [==============================] - 4s 1ms/step - loss: 0.4490 - acc: 0.7938 - val_loss: 0.4487 - val_acc: 0.7934

    Epoch 00020: val_loss did not improve from 0.44849
    Epoch 21/200
    4000/4000 [==============================] - 4s 1ms/step - loss: 0.4501 - acc: 0.7928 - val_loss: 0.4488 - val_acc: 0.7930

    Epoch 00021: val_loss did not improve from 0.44849
    Epoch 22/200
    4000/4000 [==============================] - 4s 1ms/step - loss: 0.4509 - acc: 0.7925 - val_loss: 0.4486 - val_acc: 0.7927

    Epoch 00022: val_loss did not improve from 0.44849
    Epoch 23/200
    4000/4000 [==============================] - 4s 949us/step - loss: 0.4503 - acc: 0.7930 - val_loss: 0.4486 - val_acc: 0.7930

    Epoch 00023: val_loss did not improve from 0.44849
    Epoch 24/200
    4000/4000 [==============================] - 4s 932us/step - loss: 0.4501 - acc: 0.7927 - val_loss: 0.4488 - val_acc: 0.7928

    Epoch 00024: val_loss did not improve from 0.44849
    Epoch 25/200
    4000/4000 [==============================] - 4s 909us/step - loss: 0.4495 - acc: 0.7931 - val_loss: 0.4486 - val_acc: 0.7929

    Epoch 00025: val_loss did not improve from 0.44849
    Epoch 26/200
    4000/4000 [==============================] - 4s 915us/step - loss: 0.4488 - acc: 0.7944 - val_loss: 0.4485 - val_acc: 0.7934

    Epoch 00026: val_loss did not improve from 0.44849
    Epoch 27/200
    4000/4000 [==============================] - 4s 921us/step - loss: 0.4501 - acc: 0.7929 - val_loss: 0.4485 - val_acc: 0.7933

    Epoch 00027: val_loss improved from 0.44849 to 0.44846, saving model to ./models/baseline-model.h5
    Epoch 28/200
    4000/4000 [==============================] - 4s 916us/step - loss: 0.4500 - acc: 0.7933 - val_loss: 0.4488 - val_acc: 0.7932

    Epoch 00028: val_loss did not improve from 0.44846
    Epoch 29/200
    4000/4000 [==============================] - 4s 921us/step - loss: 0.4491 - acc: 0.7935 - val_loss: 0.4486 - val_acc: 0.7932

    Epoch 00029: val_loss did not improve from 0.44846
    Epoch 30/200
    4000/4000 [==============================] - 4s 922us/step - loss: 0.4503 - acc: 0.7932 - val_loss: 0.4487 - val_acc: 0.7934

    Epoch 00030: val_loss did not improve from 0.44846
    Epoch 31/200
    4000/4000 [==============================] - 4s 1ms/step - loss: 0.4488 - acc: 0.7935 - val_loss: 0.4495 - val_acc: 0.7932

    Epoch 00031: val_loss did not improve from 0.44846
    Epoch 32/200
    4000/4000 [==============================] - 4s 1ms/step - loss: 0.4510 - acc: 0.7918 - val_loss: 0.4484 - val_acc: 0.7934

    Epoch 00032: val_loss improved from 0.44846 to 0.44842, saving model to ./models/baseline-model.h5
    Epoch 33/200
    4000/4000 [==============================] - 4s 934us/step - loss: 0.4496 - acc: 0.7938 - val_loss: 0.4486 - val_acc: 0.7932

    Epoch 00033: val_loss did not improve from 0.44842
    Epoch 34/200
    4000/4000 [==============================] - 4s 915us/step - loss: 0.4492 - acc: 0.7936 - val_loss: 0.4484 - val_acc: 0.7932

    Epoch 00034: val_loss did not improve from 0.44842
    Epoch 35/200
    4000/4000 [==============================] - 4s 917us/step - loss: 0.4507 - acc: 0.7925 - val_loss: 0.4493 - val_acc: 0.7926

    Epoch 00035: val_loss did not improve from 0.44842
    Epoch 36/200
    4000/4000 [==============================] - 5s 1ms/step - loss: 0.4488 - acc: 0.7942 - val_loss: 0.4492 - val_acc: 0.7931

    Epoch 00036: val_loss did not improve from 0.44842
    Epoch 37/200
    4000/4000 [==============================] - 5s 1ms/step - loss: 0.4484 - acc: 0.7941 - val_loss: 0.4487 - val_acc: 0.7932

    Epoch 00037: val_loss did not improve from 0.44842
    Epoch 38/200
    4000/4000 [==============================] - 5s 1ms/step - loss: 0.4487 - acc: 0.7936 - val_loss: 0.4484 - val_acc: 0.7933

    Epoch 00038: val_loss did not improve from 0.44842
    Epoch 39/200
    4000/4000 [==============================] - 5s 1ms/step - loss: 0.4489 - acc: 0.7933 - val_loss: 0.4487 - val_acc: 0.7928

    Epoch 00039: val_loss did not improve from 0.44842
    Epoch 40/200
    4000/4000 [==============================] - 4s 1ms/step - loss: 0.4499 - acc: 0.7932 - val_loss: 0.4484 - val_acc: 0.7932

    Epoch 00040: val_loss improved from 0.44842 to 0.44842, saving model to ./models/baseline-model.h5
    Epoch 41/200
    4000/4000 [==============================] - 4s 1ms/step - loss: 0.4502 - acc: 0.7933 - val_loss: 0.4489 - val_acc: 0.7931

    Epoch 00041: val_loss did not improve from 0.44842
    Epoch 42/200
    4000/4000 [==============================] - 4s 877us/step - loss: 0.4486 - acc: 0.7938 - val_loss: 0.4485 - val_acc: 0.7930

    Epoch 00042: val_loss did not improve from 0.44842
    Epoch 43/200
    4000/4000 [==============================] - 4s 979us/step - loss: 0.4481 - acc: 0.7946 - val_loss: 0.4484 - val_acc: 0.7933

    Epoch 00043: val_loss improved from 0.44842 to 0.44839, saving model to ./models/baseline-model.h5
    Epoch 44/200
    4000/4000 [==============================] - 4s 974us/step - loss: 0.4493 - acc: 0.7937 - val_loss: 0.4492 - val_acc: 0.7930

    Epoch 00044: val_loss did not improve from 0.44839
    Epoch 45/200
    4000/4000 [==============================] - 3s 872us/step - loss: 0.4489 - acc: 0.7939 - val_loss: 0.4485 - val_acc: 0.7933

    Epoch 00045: val_loss did not improve from 0.44839
    Epoch 46/200
    4000/4000 [==============================] - 4s 914us/step - loss: 0.4499 - acc: 0.7931 - val_loss: 0.4484 - val_acc: 0.7933

    Epoch 00046: val_loss did not improve from 0.44839
    Epoch 47/200
    4000/4000 [==============================] - 4s 908us/step - loss: 0.4495 - acc: 0.7941 - val_loss: 0.4489 - val_acc: 0.7929

    Epoch 00047: val_loss did not improve from 0.44839
    Epoch 48/200
    4000/4000 [==============================] - 3s 862us/step - loss: 0.4491 - acc: 0.7940 - val_loss: 0.4488 - val_acc: 0.7931

    Epoch 00048: val_loss did not improve from 0.44839
    Epoch 49/200
    4000/4000 [==============================] - 4s 946us/step - loss: 0.4501 - acc: 0.7930 - val_loss: 0.4489 - val_acc: 0.7927

    Epoch 00049: val_loss did not improve from 0.44839
    Epoch 50/200
    4000/4000 [==============================] - 4s 903us/step - loss: 0.4491 - acc: 0.7934 - val_loss: 0.4488 - val_acc: 0.7937

    Epoch 00050: val_loss did not improve from 0.44839
    Epoch 51/200
    4000/4000 [==============================] - 4s 896us/step - loss: 0.4494 - acc: 0.7936 - val_loss: 0.4486 - val_acc: 0.7930

    Epoch 00051: val_loss did not improve from 0.44839
    Epoch 52/200
    4000/4000 [==============================] - 4s 886us/step - loss: 0.4495 - acc: 0.7934 - val_loss: 0.4484 - val_acc: 0.7934

    Epoch 00052: val_loss did not improve from 0.44839
    Epoch 53/200
    4000/4000 [==============================] - 3s 856us/step - loss: 0.4501 - acc: 0.7930 - val_loss: 0.4485 - val_acc: 0.7929

    Epoch 00053: val_loss did not improve from 0.44839
    Epoch 54/200
    4000/4000 [==============================] - 3s 870us/step - loss: 0.4498 - acc: 0.7935 - val_loss: 0.4488 - val_acc: 0.7931

    Epoch 00054: val_loss did not improve from 0.44839
    Epoch 55/200
    4000/4000 [==============================] - 3s 873us/step - loss: 0.4478 - acc: 0.7945 - val_loss: 0.4485 - val_acc: 0.7929

    Epoch 00055: val_loss did not improve from 0.44839
    Epoch 56/200
    4000/4000 [==============================] - 4s 971us/step - loss: 0.4484 - acc: 0.7941 - val_loss: 0.4489 - val_acc: 0.7932

    Epoch 00056: val_loss did not improve from 0.44839
    Epoch 57/200
    4000/4000 [==============================] - 4s 998us/step - loss: 0.4497 - acc: 0.7930 - val_loss: 0.4485 - val_acc: 0.7931

    Epoch 00057: val_loss did not improve from 0.44839
    Epoch 58/200
    4000/4000 [==============================] - 3s 873us/step - loss: 0.4487 - acc: 0.7938 - val_loss: 0.4487 - val_acc: 0.7935

    Epoch 00058: val_loss did not improve from 0.44839
    Epoch 00058: early stopping

    <tensorflow.python.keras.callbacks.History at 0x163ad2110>

If you were to actually run the training, you would be able to visualize its history. Keras saves the entire training history, keeping track of whatever metric you specify (here accuracy and loss).

```python
baseline_model
```

    <tensorflow.python.keras.engine.functional.Functional at 0x16063eed0>

```python
history = baseline_model.history.history
history.keys()
```

    dict_keys(['loss', 'acc', 'val_loss', 'val_acc'])

```python
# accuracy plot
plt.plot(100 * np.array(history['acc']), label='training')
plt.plot(100 * np.array(history['val_acc']), label='validation')
plt.xlim(0)
plt.xlabel('epoch')
plt.ylabel('accuracy %')
plt.legend(loc='lower right', fontsize=20)
plt.show()
```

```python
# loss plot
plt.plot(100 * np.array(history['loss']), label='training')
plt.plot(100 * np.array(history['val_loss']), label='validation')
plt.xlim(0)
plt.xlabel('epoch')
plt.ylabel('loss')
plt.legend(loc='upper right', fontsize=20)
# the line indicate the epoch corresponding to the best performance on the validation set
# plt.vlines(np.argmin(history['val_loss']), 45, 56, linestyle='dashed', linewidth=0.5)
plt.show()
```

```python
print('Loss estimate on unseen examples (from validation set) = {0:.3f}'.format(np.min(history['val_loss'])))
```

    Loss estimate on unseen examples (from validation set) = 0.448

#### Evaluate on test set

```python
baseline_model.load_weights('./models/baseline-model.h5')
```

```python
yhat_baseline = baseline_model.predict(features_test, batch_size=512)
```

```python
bins = np.linspace(0, 1, 20)
_ = plt.hist(yhat_baseline[labels_test==1],
         histtype='stepfilled', alpha=0.5, color='red', label=r"Signal ($W' \rightarrow WZ$)", bins=bins)
_ = plt.hist(yhat_baseline[labels_test==0],
         histtype='stepfilled', alpha=0.5, color='blue', label=r'Background (QCD dijets)', bins=bins)
plt.legend(loc='upper center')
plt.xlabel('P(signal) assigned by the baseline model')
```

## Convolutional Neural Network

We can now instead try to learn a model directly on the pixel space, instead of summarizing the information into engineered features such as mass, tau_21 and delta_R.

```python
from keras.layers import Conv2D, Flatten, LeakyReLU
```

```python
# add channel dimension (1 for grayscale)
images_train = np.expand_dims(images_train, -1)
images_test = np.expand_dims(images_test, -1)
images_val = np.expand_dims(images_val, -1)
```

```python
x = Input(shape=(images_train.shape[1:]))
h = Conv2D(32, kernel_size=7, strides=1)(x)
h = LeakyReLU()(h)
h = Dropout(0.2)(h)

h = Conv2D(64, kernel_size=7, strides=1)(h)
h = LeakyReLU()(h)
h = Dropout(0.2)(h)

h = Conv2D(128, kernel_size=5, strides=1)(h)
h = LeakyReLU()(h)
h = Dropout(0.2)(h)

h = Conv2D(256, kernel_size=5, strides=1)(h)
h = LeakyReLU()(h)
h = Flatten()(h)
h = Dropout(0.2)(h)
y = Dense(1, activation='sigmoid')(h)
```

```python
cnn_model = Model(x, y)
cnn_model.compile('adam', 'binary_crossentropy', metrics=['acc'])
```

```python
cnn_model.summary()
```

    Model: "model_2"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    input_2 (InputLayer)         [(None, 25, 25, 1)]       0
    _________________________________________________________________
    conv2d (Conv2D)              (None, 19, 19, 32)        1600
    _________________________________________________________________
    leaky_re_lu (LeakyReLU)      (None, 19, 19, 32)        0
    _________________________________________________________________
    dropout (Dropout)            (None, 19, 19, 32)        0
    _________________________________________________________________
    conv2d_1 (Conv2D)            (None, 13, 13, 64)        100416
    _________________________________________________________________
    leaky_re_lu_1 (LeakyReLU)    (None, 13, 13, 64)        0
    _________________________________________________________________
    dropout_1 (Dropout)          (None, 13, 13, 64)        0
    _________________________________________________________________
    conv2d_2 (Conv2D)            (None, 9, 9, 128)         204928
    _________________________________________________________________
    leaky_re_lu_2 (LeakyReLU)    (None, 9, 9, 128)         0
    _________________________________________________________________
    dropout_2 (Dropout)          (None, 9, 9, 128)         0
    _________________________________________________________________
    conv2d_3 (Conv2D)            (None, 5, 5, 256)         819456
    _________________________________________________________________
    leaky_re_lu_3 (LeakyReLU)    (None, 5, 5, 256)         0
    _________________________________________________________________
    flatten (Flatten)            (None, 6400)              0
    _________________________________________________________________
    dropout_3 (Dropout)          (None, 6400)              0
    _________________________________________________________________
    dense_4 (Dense)              (None, 1)                 6401
    =================================================================
    Total params: 1,132,801
    Trainable params: 1,132,801
    Non-trainable params: 0
    _________________________________________________________________

Feel free to try to train it at home! For this tutorial, we will just load in pre-trained weights.

```python
cnn_model.fit(
    images_train, labels_train,
    epochs=100,
    batch_size=512,
    validation_data=(images_val, labels_val),
    callbacks=[
        EarlyStopping(verbose=True, patience=30, monitor='val_loss'),
        ModelCheckpoint('./models/cnn-model.h5', monitor='val_loss',
                        verbose=True, save_best_only=True)
    ]
)
```

```python
cnn_history = cnn_model.history.history
```

```python
# accuracy plot
plt.plot(100 * np.array(cnn_history['acc']), label='training')
plt.plot(100 * np.array(cnn_history['val_acc']), label='validation')
plt.xlim(0)
plt.xlabel('epoch')
plt.ylabel('accuracy %')
plt.legend(loc='lower right', fontsize=20)
plt.show()
```

```python
# loss plot
plt.plot(100 * np.array(cnn_history['loss']), label='training')
plt.plot(100 * np.array(cnn_history['val_loss']), label='validation')
plt.xlim(0)
plt.xlabel('epoch')
plt.ylabel('loss')
plt.legend(loc='upper right', fontsize=20)
# the line indicate the epoch corresponding to the best performance on the validation set
plt.vlines(np.argmin(cnn_history['val_loss']), 43, 56, linestyle='dashed', linewidth=0.5)
plt.show()
```

```python
print 'Loss estimate on unseen examples (from validation set) = {0:.3f}'.format(np.min(cnn_history['val_loss']))
```

```python
cnn_model.load_weights('models/cnn-model.h5')
yhat_cnn = cnn_model.predict(images_test, batch_size=512, verbose=True)
```

## Locally-Connected Neural Network

```python
from keras.layers import LocallyConnected2D , MaxPool2D, Flatten
```

```python
x = Input(shape=(images_train.shape[1:]))
h = LocallyConnected2D(32, kernel_size=9, strides=2)(x)
h = LeakyReLU()(h)

h = Dropout(0.2)(h)
h = LocallyConnected2D(32, kernel_size=5, strides=1)(h)
h = LeakyReLU()(h)

h = Dropout(0.2)(h)
h = LocallyConnected2D(64, kernel_size=3, strides=1)(h)
h = LeakyReLU()(h)

h = Dropout(0.2)(h)
h = LocallyConnected2D(64, kernel_size=3, strides=1)(h)
h = LeakyReLU()(h)

h = Flatten()(h)
h = Dropout(0.2)(h)
y = Dense(1, activation='sigmoid')(h)
```

```python
lcn_model = Model(x, y)
lcn_model.compile('adam', 'binary_crossentropy', metrics=['acc'])
```

```python
lcn_model.summary()
```

Feel free to try to train it at home! For this tutorial, we will just load in pre-trained weights.

```python
# lcn_model.fit(
#     images_train, labels_train,
#     epochs=100,
#     batch_size=256,
#     validation_data=(images_val, labels_val),
#     callbacks=[
#         EarlyStopping(verbose=True, patience=30, monitor='val_loss'),
#         ModelCheckpoint('./models/lcn-model.h5', monitor='val_loss',
#                         verbose=True, save_best_only=True)
#     ]
# )
```

```python
lcn_history = lcn_model.history.history
```

```python
# accuracy plot
plt.plot(100 * np.array(lcn_history['acc']), label='training')
plt.plot(100 * np.array(lcn_history['val_acc']), label='validation')
plt.xlim(0)
plt.xlabel('epoch')
plt.ylabel('accuracy %')
plt.legend(loc='lower right', fontsize=20)
plt.show()
```

```python
# loss plot
plt.plot(100 * np.array(lcn_history['loss']), label='training')
plt.plot(100 * np.array(lcn_history['val_loss']), label='validation')
plt.xlim(0)
plt.xlabel('epoch')
plt.ylabel('loss')
plt.legend(loc='upper right', fontsize=20)
# the line indicate the epoch corresponding to the best performance on the validation set
plt.vlines(np.argmin(lcn_history['val_loss']), 43, 56, linestyle='dashed', linewidth=0.5)
plt.show()
```

```python
print 'Loss estimate on unseen examples (from validation set) = {0:.3f}'.format(np.min(lcn_history['val_loss']))
```

```python
lcn_model.load_weights('models/lcn-model.h5')
yhat_lcn = lcn_model.predict(images_test, batch_size=512)
```

## Fully-Connected network

```python
x = Input(shape=(images_train.shape[1:]))
h = Flatten()(x)
h = Dense(25 ** 2, kernel_initializer='he_normal')(h)
h = LeakyReLU()(h)

h = Dropout(0.2)(h)
h = Dense(512, kernel_initializer='he_normal')(h)
h = LeakyReLU()(h)

h = Dropout(0.2)(h)
h = Dense(256, kernel_initializer='he_normal')(h)
h = LeakyReLU()(h)

h = Dropout(0.2)(h)
h = Dense(128, kernel_initializer='he_normal')(h)
h = LeakyReLU()(h)

h = Dropout(0.2)(h)
y = Dense(1, activation='sigmoid')(h)
```

```python
dense_model = Model(x, y)
dense_model.compile('adam', 'binary_crossentropy', metrics=['acc'])
```

```python
dense_model.fit(
    images_train, labels_train,
    epochs=100,
    batch_size=256,
    validation_data=(images_val, labels_val),
    callbacks=[
        EarlyStopping(verbose=True, patience=30, monitor='val_loss'),
        ModelCheckpoint('./models/dense-model.h5', monitor='val_loss',
                        verbose=True, save_best_only=True)
    ]
)
```

```python
dense_history = dense_model.history.history
```

```python
# accuracy plot
plt.plot(100 * np.array(dense_history['acc']), label='training')
plt.plot(100 * np.array(dense_history['val_acc']), label='validation')
plt.xlim(0)
plt.xlabel('epoch')
plt.ylabel('accuracy %')
plt.legend(loc='lower right', fontsize=20)
plt.show()
```

```python
# loss plot
plt.plot(100 * np.array(dense_history['loss']), label='training')
plt.plot(100 * np.array(dense_history['val_loss']), label='validation')
plt.xlim(0)
plt.xlabel('epoch')
plt.ylabel('loss')
plt.legend(loc='upper right', fontsize=20)
# the line indicate the epoch corresponding to the best performance on the validation set
plt.vlines(np.argmin(dense_history['val_loss']), 43, 56, linestyle='dashed', linewidth=0.5)
plt.show()
```

```python
print 'Loss estimate on unseen examples (from validation set) = {0:.3f}'.format(np.min(dense_history['val_loss']))
```

```python
dense_model.load_weights('models/dense-model.h5')
yhat_dense = dense_model.predict(images_test, batch_size=512, verbose=True)
```

## Plot ROC Curves

A standard way to visualize the tradeoff between low false positive rate (FPR) and high true positive rate (TPR) is by plotting them on a ROC curve. In Physics, people like to plot the TPR on the x-axis, and 1/FPR on the y-axis.

```python
from sklearn.metrics import roc_curve
```

```python
fpr_cnn, tpr_cnn, _ = roc_curve(labels_test, yhat_cnn)
fpr_lcn, tpr_lcn, _ = roc_curve(labels_test, yhat_lcn)
fpr_dense, tpr_dense, _ = roc_curve(labels_test, yhat_dense)
fpr_baseline, tpr_baseline, _ = roc_curve(labels_test, yhat_baseline)
```

```python
plt.figure(figsize=(10,10))
plt.grid(b = True, which = 'minor')
plt.grid(b = True, which = 'major')
_ = plt.plot(tpr_cnn, 1./fpr_cnn, label='CNN')
_ = plt.plot(tpr_lcn, 1./fpr_lcn, label='LCN')
_ = plt.plot(tpr_dense, 1./fpr_dense, label='FCN')
_ = plt.plot(tpr_baseline, 1./fpr_baseline, label='Baseline')
plt.legend()
plt.xlim((0.1, 0.9))
plt.ylim((1, 1000))
plt.yscale('log')
```

## What is the network learning?

```python
def get_correlations(images, disc_output):
    '''
    calculate linear correlation between each pixel and the output of the classifier
    to see what pixels are more indicative of a specific class.
    '''
    import pandas as pd
    # -- find the total number of pixels per image, here 25 x 25
    n_pixels = np.prod(images.shape[1:3])

    # -- add the pixels as columns to a dataframe
    df = pd.DataFrame(
        {i : np.squeeze(images).reshape(-1, n_pixels)[:, i] for i in range(n_pixels)}
    )
    # -- add a column to the end of the dataframe for the discriminator's output
    df['disc_output'] = disc_output
    # -- pandas offers an easy solution to calculate correlations
    # (even though it's slow because it also calculates the correlation between each pixel and every other pixel)
    correlations = df.corr().values[:-1, -1]
    return correlations

def plot_correlations(correlations, extent, title='', img_dim=(25, 25), cmap=plt.cm.seismic):
    '''
    call the function about and then plot the correlations in image format
    '''
    max_mag = max(
        abs(np.min(correlations[np.isfinite(correlations)])),
        abs(np.max(correlations[np.isfinite(correlations)])),
    ) # highest correlation value (abs value), to make the plot look nice and on a reasonable scale

    f, ax = plt.subplots(figsize=(6, 6))
    im = ax.imshow(
        correlations.reshape(img_dim),
        interpolation='nearest',
        norm=Normalize(vmin=-max_mag, vmax=max_mag),
        extent=extent,
        cmap=cmap
    )

    plt.colorbar(im, fraction=0.05, pad=0.05)
    plt.xlabel(r'[Transformed] Pseudorapidity $(\eta)$')
    plt.ylabel(r'[Transformed] Azimuthal Angle $(\phi)$')
    plt.title(title)
#     plt.savefig(os.path.join('..', outdir, outname))
```

You can now visualize what each network picks up on, at least to first order. These correlation plots tells us whether a specific pixel being strongly activated is a good indicator of that jet image belonging to a class or the other. Red represents the signal (boosted W from W'-->WZ), blue represents the background (QCD).

```python
plot_correlations(
    get_correlations(images_test[:10000], yhat_cnn[:10000]),
    extent=[-1.25, 1.25, -1.25, 1.25],
    title='Correlation between pixels \n and the CNN prediction'
)
```

```python
plot_correlations(
    get_correlations(images_test[:10000], yhat_lcn[:10000]),
    extent=[-1.25, 1.25, -1.25, 1.25],
    title='Correlation between pixels \n and the LCN prediction'
)
```

```python
plot_correlations(
    get_correlations(images_test[:10000], yhat_dense[:10000]),
    extent=[-1.25, 1.25, -1.25, 1.25],
    title='Correlation between pixels \n and the FCN prediction'
)
```

```python
plot_correlations(
    get_correlations(images_test[:10000], yhat_baseline[:10000]),
    extent=[-1.25, 1.25, -1.25, 1.25],
    title='Correlation between pixels \n and the Baseline prediction'
)
```

You can also look at how the output of each classifier is correlated with known quantities that are known to be discriminative, such as the ones used in the baseline classifier above (mass, tau_21, delta_R). This will inform us as to whether the network has 'learned' to internally calculate a representation that is close to these variables, thus eliminating our need to come up with these features ourselves.

```python
def plot_output_vs_kin(kin, output, xlabel, ylabel, nbins=30):
    '''
    Plot one output of the discriminator network vs. one of the 1D physics variables that describe jets
    Args:
    -----
        kin : numpy array, kinematic property (such as mass or pT) associated with each image. I.e.: discrete_mass(np.squeeze(generated_images))
        output : numpy array, one of the 2 outputs of the discriminator, evaluated on the same images that `kin` refers to
        xlabel : string, x-axis label that describes the meaning of `kin`
        ylabel : string, y-axis label that describes the meaning og `output`
        outname : name of the output file, to be placed in ../plots/
        nbins : (default = 30) number of bins to use to represent the distributions in a discretized way

    '''
    # set the colormap
    plt.set_cmap('jet')
    # draw a 2d histogram of the discriminator's output versus the kinematic variable of choice (mass, pT, etc.)
    h, binx, biny, _ = plt.hist2d(kin, output.reshape(-1,), bins=nbins)
    plt.clf() # we don't want to plot this 2D histogram, we want to normalize it per bin first

    # normalize the histogram such that the entries in each column add up to 1, such that the intensity
    # of each corresponds to the percentage of the jets in a given mass (or pT) bin that get assigned a p
    for i in range(nbins):
        h[i, :] = h[i, :] / float(np.sum(h[i, :]))
    # plot the normalized histogram as an image
    f, ax2 = plt.subplots(figsize=(6, 6))
    im = ax2.imshow(
        np.flipud(h.T),
        interpolation='nearest',
        norm=LogNorm(),
        extent=[binx.min(), binx.max(), biny.min(), biny.max()],
        aspect="auto"
    )
    plt.xlabel(xlabel)
    plt.ylabel(ylabel)
    # add a custom colorbar
    cax = f.add_axes([0.93, 0.1, 0.03, 0.8])
    plt.colorbar(im, cax = cax)
    plt.set_cmap('viridis')
```

### Mass

```python
plot_output_vs_kin(
    scaler.inverse_transform(features_test)[:, 0], # mass
    yhat_cnn,
    xlabel='Discrete jet image mass (GeV)',
    ylabel='P(signal)',
)
```

```python
plot_output_vs_kin(
    scaler.inverse_transform(features_test)[:, 0], # mass
    yhat_lcn,
    xlabel='Discrete jet image mass (GeV)',
    ylabel='P(signal)',
)
```

```python
plot_output_vs_kin(
    scaler.inverse_transform(features_test)[:, 0], # mass
    yhat_dense,
    xlabel='Discrete jet image mass (GeV)',
    ylabel='P(signal)',
)
```

```python
plot_output_vs_kin(
    scaler.inverse_transform(features_test)[:, 0], # mass
    yhat_baseline,
    xlabel='Discrete jet image mass (GeV)',
    ylabel='P(signal)',
)
```

### Tau_21

```python
plot_output_vs_kin(
    scaler.inverse_transform(features_test)[:, 1], # tau21
    yhat_cnn,
    xlabel=r'Discrete jet $\tau_{21}$',
    ylabel='P(signal)',
)
```

```python
plot_output_vs_kin(
    scaler.inverse_transform(features_test)[:, 1], # tau21
    yhat_lcn,
    xlabel=r'Discrete jet $\tau_{21}$',
    ylabel='P(signal)',
)
```

```python
plot_output_vs_kin(
    scaler.inverse_transform(features_test)[:, 1], # tau21
    yhat_dense,
    xlabel=r'Discrete jet $\tau_{21}$',
    ylabel='P(signal)',
)
```

```python
plot_output_vs_kin(
    scaler.inverse_transform(features_test)[:, 1], # tau21
    yhat_baseline,
    xlabel=r'Discrete jet $\tau_{21}$',
    ylabel='P(signal)',
)
```

### Delta R

```python
plot_output_vs_kin(
    scaler.inverse_transform(features_test)[:, 2], # deltaR
    yhat_cnn,
    xlabel=r'Discrete jet $\Delta R$',
    ylabel='P(signal)',
)
```

```python
plot_output_vs_kin(
    scaler.inverse_transform(features_test)[:, 2], # deltaR
    yhat_lcn,
    xlabel=r'Discrete jet $\Delta R$',
    ylabel='P(signal)',
)
```

```python
plot_output_vs_kin(
    scaler.inverse_transform(features_test)[:, 2], # deltaR
    yhat_dense,
    xlabel=r'Discrete jet $\Delta R$',
    ylabel='P(signal)',
)
```

```python
plot_output_vs_kin(
    scaler.inverse_transform(features_test)[:, 2], # deltaR
    yhat_baseline,
    xlabel=r'Discrete jet $\Delta R$',
    ylabel='P(signal)',
)
```

Sometimes, however, it is not a good idea to have your network learn the mass of a jet and use that to classify jets. In that case, in fact, the network will successfully suppress all jets outside of the signal-like mass window and sculpt the mass of the background to look like a signal peak.

What we would like to be able to do, instead, is to have a classifier that is insensitive to mass, and that reduced the background across the entire mass spectrum.

For reference, see: C. Shimmin et al., Decorrelated Jet Substructure Tagging using Adversarial Neural Networks.

# Training a GAN (WGAN-GP) on jet images

WGAN-GP = a type of GAN that minimizes the Wasserstein distance between the target and generated distributions and enforces the Lipschitz contraint by penalizing the norm of the gradient instead of clipping weights.

```python
from keras.layers import Input, Dense, Reshape, Flatten
from keras.layers.merge import _Merge
from keras.layers.convolutional import Convolution2D, Conv2DTranspose
from keras.layers.normalization import BatchNormalization
from keras.layers.advanced_activations import LeakyReLU
from keras.optimizers import Adam
from keras import backend as K
```

```python
BATCH_SIZE = 100

# The training ratio is the number of discriminator updates per generator
# update. The paper uses 5.
TRAINING_RATIO = 5
GRADIENT_PENALTY_WEIGHT = 10  # As per the paper
```

Define new custom loss functions not included among the standard keras ones:

```python
def wasserstein_loss(y_true, y_pred):
    return K.mean(y_true * y_pred)

def gradient_penalty_loss(y_true, y_pred, averaged_samples,
                          gradient_penalty_weight):
    gradients = K.gradients(K.sum(y_pred), averaged_samples)
    gradient_l2_norm = K.sqrt(K.sum(K.square(gradients)))
    gradient_penalty = gradient_penalty_weight * K.square(1 - gradient_l2_norm)
    return gradient_penalty
```

```python
def make_generator():
    """Creates a generator model that takes a 100-dimensional latent prior and
    converts to size 25 x 25 x 1"""

    z = Input(shape=(100, ))

    x = Dense(1024, input_dim=100)(z)
    x = LeakyReLU()(x)

    x = Dense(128 * 7 * 7)(x)
    x = BatchNormalization()(x)
    x = LeakyReLU()(x)

    x = Reshape((7, 7, 128))(x)

    x = Conv2DTranspose(128, (5, 5), strides=2, padding='same')(x)
    x = BatchNormalization(axis=-1)(x)
    x = LeakyReLU()(x)

    x = Convolution2D(64, (5, 5), padding='same')(x)
    x = BatchNormalization(axis=-1)(x)
    x = LeakyReLU()(x)

    x = Conv2DTranspose(64, (5, 5), strides=2, padding='same')(x)
    x = BatchNormalization(axis=-1)(x)
    x = LeakyReLU()(x)

    y = Convolution2D(1, (4, 4), padding='valid', activation='relu')(x)

    return Model(z, y)
```

```python
def make_discriminator():

    x = Input(shape=(25, 25, 1))

    h = Convolution2D(64, (5, 5), padding='same')(x)
    h = LeakyReLU()(h)

    h = Convolution2D(128, (5, 5), kernel_initializer='he_normal',
                      strides=2)(h)
    h = LeakyReLU()(h)

    h = Convolution2D(256, (5, 5), kernel_initializer='he_normal',
                      padding='same', strides=2)(h)
    h = LeakyReLU()(h)

    h = Flatten()(h)
    h = Dense(1024, kernel_initializer='he_normal')(h)
    h = LeakyReLU()(h)

    y = Dense(1, kernel_initializer='he_normal')(h)

    return Model(x, y)
```

Build individual models:

```python
generator = make_generator()
discriminator = make_discriminator()
```

Build connected model with only generator trainable:

```python
discriminator.trainable = False
z = Input(shape=(100, ))
generator_model = Model(z, discriminator(generator(z)))
# We use the Adam paramaters from Gulrajani et al.
generator_model.compile(optimizer=Adam(0.0001, beta_1=0.5, beta_2=0.9),
                        loss=wasserstein_loss)
```

Build connected model with only discriminator trainable:

```python
discriminator.trainable = True
generator.trainable = False
```

```python
class RandomWeightedAverage(_Merge): # used for gradient norm penalty

    def _merge_function(self, inputs):
        weights = K.random_uniform((K.shape(inputs[0])[0], 1, 1, 1))
        return (weights * inputs[0]) + ((1 - weights) * inputs[1])
```

```python
real_samples = Input(shape=(25, 25, 1))
z = Input(shape=(100,))
fake_samples = generator(z)

critic_out_fake = discriminator(fake_samples)
critic_out_real = discriminator(real_samples)

# generate weighted-averages of real and generated
# samples, to use for the gradient norm penalty.
averaged_samples = RandomWeightedAverage()([real_samples, fake_samples])

# running them thru critic to get the gradient norm for the GP loss.
averaged_samples_out = discriminator(averaged_samples)

# The gradient penalty loss function requires the input averaged samples
def gp_loss(y_true, y_pred):
    return gradient_penalty_loss(
        y_true, y_pred,
        averaged_samples=averaged_samples,
        gradient_penalty_weight=GRADIENT_PENALTY_WEIGHT
    )

discriminator_model = Model(
    inputs=[real_samples, z],
    outputs=[critic_out_real, critic_out_fake, averaged_samples_out]
)

# We use the Adam paramaters from Gulrajani et al.
discriminator_model.compile(
    optimizer=Adam(0.0001, beta_1=0.5, beta_2=0.9),
    loss=[wasserstein_loss, wasserstein_loss, gp_loss]
)
```

```python
# positive_y is the label vector for real samples, with value 1.
# negative_y is the label vector for generated samples, with value -1.
# dummy_y vector is passed to the gradient_penalty loss function and is
# not used.

positive_y = np.ones((BATCH_SIZE, 1), dtype=np.float32)
negative_y = -positive_y
dummy_y = np.zeros((BATCH_SIZE, 1), dtype=np.float32)
```

```python
# do a little bit of scaling for stability
X_train = np.expand_dims(np.squeeze(images_train[:30000]) / 100, -1)
```

```python
overall_disc_loss = []

for epoch in range(200): # train for 200 iterations

    # at each epoch, shuffle the training set to get new samples
    np.random.shuffle(X_train)
    print("Epoch: ", epoch)
    print("Number of batches: ", int(X_train.shape[0] // BATCH_SIZE))
    discriminator_loss = []
    generator_loss = []

    # we'll need this many samples per critic update
    critic_nb_samples = BATCH_SIZE * TRAINING_RATIO

    # loop through batches
    for i in range(int(X_train.shape[0] // (BATCH_SIZE * TRAINING_RATIO))):
        X_critic = X_train[i * critic_nb_samples:(i + 1) * critic_nb_samples]

        # critic gets trained 5 times more per iteration than the generator
        for j in range(TRAINING_RATIO):
            X_minibatch = X_critic[j * BATCH_SIZE:(j + 1) * BATCH_SIZE]

            # generate new input noise
            noise = np.random.rand(BATCH_SIZE, 100).astype(np.float32)

            # train the discriminator (or critic)
            disc_loss = discriminator_model.train_on_batch(
                [X_minibatch, noise],
                [positive_y, negative_y, dummy_y]
            )
            discriminator_loss.append(disc_loss)

        critic_score = np.array(discriminator_loss)[:, 0]
        if i % 10 == 0:
            print('critic score =', critic_score.mean())
        overall_disc_loss.extend(critic_score.tolist())
        # train the generator
        gen_loss = generator_model.train_on_batch(
            np.random.rand(BATCH_SIZE, 100),
            positive_y
        )
        generator_loss.append(gen_loss)
```

```python
# discriminator.save_weights('./models/wgan-discriminator.h5')
# generator.save_weights('./models/wgan-generator.h5')
```

Load pre-trained wights:

```python
discriminator.load_weights('./models/wgan-discriminator.h5')
generator.load_weights('./models/wgan-generator.h5')
```

Now that we have a trained GAN, we can see if it actually works and what it produces. We can now get rid of the critic (discriminator) and focus only on the part that we really care about, the generator. Let's execute a forward pass on the generator.

```python
# input noise that will be transformed into jet images
noise = np.random.rand(1000, 100).astype(np.float32)
```

```python
# produce some jet images from the generator!
fake_jets = generator.predict(noise, batch_size=BATCH_SIZE, verbose=True)

# rescale energies and remove redundant dimension for grayscale channel
fake_jets = np.squeeze(fake_jets * 100)
```

Let's look at some GAN-generated jet images!

```python
plot_jet_image(fake_jets.mean(0))
```

We can also look at the difference between the average generated image and the average real image to identify parts of the image that are not well-captured by the GAN and might need improvement. Green pixels are more strongly activated, on average, in fake images, while purple pixels are more strongly activated in background images.

```python
plot_diff_jet_image(fake_jets.mean(0) - images.mean(0), cmap='PRGn')
```

There are multiple ways of imporving this training, such as conditioning on class or attributes, using minibatch discrimination, enforcing physical constraints, etc.


```python
import sys
sys.path.append('..')
import torch
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.preprocessing import LabelEncoder, MinMaxScaler
from sympy import simplify_logic

from lens.utils.relu_nn import get_reduced_model, prune_features
from lens import logic
from lens.utils.base import collect_parameters

torch.manual_seed(0)
np.random.seed(0)
```

```python
gene_expression_matrix = pd.read_csv('w_1/data_0.csv', index_col=None, header=None)
labels = pd.read_csv('w_1/tempLabels_W-1.csv', index_col=None, header=None)
```

```python
gene_expression_matrix
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      ...
      28392
      28393
      28394
      28395
      28396
      28397
      28398
      28399
      28400
      28401

      0
      14.622486
      11.162004
      3.320000
      3.320000
      3.320000
      12.788433
      6.143456
      3.320000
      4.876620
      3.320000
      ...
      3.32
      3.320000
      3.885589
      3.914260
      3.320000
      3.32
      3.320000
      4.465420
      3.320000
      4.973620

      1
      14.398743
      11.000080
      3.320000
      3.320000
      3.320000
      12.845914
      6.147482
      3.320000
      4.484223
      3.320000
      ...
      3.32
      3.575025
      4.236519
      4.047825
      3.320000
      3.32
      4.176269
      3.320000
      4.553796
      4.967418

      2
      14.692079
      11.100175
      3.320000
      4.171535
      3.320000
      12.712544
      5.583210
      3.320000
      3.478171
      3.320000
      ...
      3.32
      3.320000
      3.992331
      4.865538
      3.320000
      3.32
      3.488281
      3.406285
      3.320000
      6.676063

      3
      14.613382
      11.023209
      3.320000
      3.320000
      3.320000
      12.750496
      5.688023
      3.320000
      4.464426
      3.320000
      ...
      3.32
      3.855643
      3.320000
      4.905350
      3.320000
      3.32
      4.158393
      4.433457
      3.874214
      5.981160

      4
      14.482065
      10.989851
      3.320000
      3.992726
      4.574745
      12.878702
      6.195418
      4.177962
      3.872567
      3.320000
      ...
      3.32
      3.320000
      3.320000
      4.879493
      3.320000
      3.32
      4.571869
      3.320000
      4.982136
      6.145585

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      56
      14.565031
      11.699843
      3.320000
      3.320000
      3.320000
      12.789212
      6.504027
      3.320000
      3.320000
      6.182912
      ...
      3.32
      3.320000
      3.320000
      4.338456
      3.771718
      3.32
      3.320000
      3.320000
      3.320000
      6.100644

      57
      14.624502
      11.918757
      3.320000
      4.292406
      3.430485
      10.728709
      6.197159
      3.320000
      4.089918
      5.201608
      ...
      3.32
      3.320000
      3.320000
      4.790134
      3.320000
      3.32
      4.985474
      4.444057
      3.580523
      6.301926

      58
      14.585190
      11.090112
      3.320000
      3.674768
      3.320000
      12.877485
      6.326960
      3.320000
      3.320000
      3.320000
      ...
      3.32
      3.320000
      3.320000
      4.547342
      3.320000
      3.32
      4.064473
      3.320000
      4.254152
      5.964505

      59
      14.449554
      10.805855
      3.320000
      3.320000
      3.320000
      12.660038
      6.261395
      3.320000
      4.125096
      3.320000
      ...
      3.32
      3.784260
      3.644823
      4.546974
      3.427441
      3.32
      4.666265
      3.888525
      3.765754
      5.452018

      60
      14.439020
      11.080826
      4.152651
      3.320000
      3.320000
      12.187504
      6.154358
      3.320000
      4.247009
      3.320000
      ...
      3.32
      3.671102
      3.519157
      4.771458
      3.320000
      3.32
      5.115644
      3.320000
      3.410361
      6.449961

61 rows × 28402 columns

```python
labels
```

      0

      0
      diagnosis: healthy control

      1
      diagnosis: healthy control

      2
      diagnosis: healthy control

      3
      diagnosis: healthy control

      4
      diagnosis: healthy control

      ...
      ...

      56
      omalizumab responder status: Responder

      57
      omalizumab responder status: Responder

      58
      omalizumab responder status: Responder

      59
      omalizumab responder status: Responder

      60
      omalizumab responder status: Responder

61 rows × 1 columns

```python
encoder = LabelEncoder()
labels_encoded = encoder.fit_transform(labels.values)
labels_encoded_noncontrols = labels_encoded[labels_encoded!=0] - 1

data_controls = gene_expression_matrix[labels_encoded==0]
data = gene_expression_matrix[labels_encoded!=0]

gene_signature = data_controls.mean(axis=0)
data_scaled = data - gene_signature

scaler = MinMaxScaler((0, 1))
scaler.fit(data_scaled)
data_normalized = scaler.transform(data_scaled)

x_train = torch.FloatTensor(data_normalized)
y_train = torch.FloatTensor(labels_encoded_noncontrols).unsqueeze(1)
print(x_train.shape)
print(y_train.shape)
```

    /home/pietro/anaconda3/envs/dev/lib/python3.7/site-packages/sklearn/preprocessing/_label.py:251: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
      y = column_or_1d(y, warn=True)

    torch.Size([40, 28402])
    torch.Size([40, 1])

```python
layers = [
    torch.nn.Linear(x_train.size(1), 10),
    torch.nn.ReLU(),
    torch.nn.Linear(10, 5),
    torch.nn.ReLU(),
    torch.nn.Linear(5, 1),
    torch.nn.Sigmoid(),
]
model = torch.nn.Sequential(*layers)

optimizer = torch.optim.Adam(model.parameters(), lr=0.0001)
model.train()
need_pruning = True
for epoch in range(13000):
    # forward pass
    optimizer.zero_grad()
    y_pred = model(x_train)
    # Compute Loss
    loss = torch.nn.functional.mse_loss(y_pred, y_train)

    for module in model.children():
        if isinstance(module, torch.nn.Linear):
            loss += 0.007 * torch.norm(module.weight, 1)

    # backward pass
    loss.backward()
    optimizer.step()

    # compute accuracy
    if epoch % 1000 == 0:
        y_pred_d = (y_pred > 0.5)
        accuracy = (y_pred_d.eq(y_train).sum(dim=1) == y_train.size(1)).sum().item() / y_train.size(0)
        print(f'Epoch {epoch}: train accuracy: {accuracy:.4f}')

    if epoch > 5000 and need_pruning:
        prune_features(model)
        need_pruning = False
```

    Epoch 0: train accuracy: 0.7500
    Epoch 1000: train accuracy: 0.7500
    Epoch 2000: train accuracy: 0.7500
    Epoch 3000: train accuracy: 0.7500
    Epoch 4000: train accuracy: 0.9750
    Epoch 5000: train accuracy: 1.0000
    Epoch 6000: train accuracy: 0.8500
    Epoch 7000: train accuracy: 0.9500
    Epoch 8000: train accuracy: 0.9750
    Epoch 9000: train accuracy: 0.9750
    Epoch 10000: train accuracy: 1.0000
    Epoch 11000: train accuracy: 1.0000
    Epoch 12000: train accuracy: 1.0000

# Combine local explanations

```python
global_explanation, predictions, counter = logic.combine_local_explanations(model, x_train, y_train,
                                                                          topk_explanations=3)

ynp = y_train.detach().numpy()[:, 0]
accuracy = np.sum(predictions == ynp) / len(ynp)
print(f'Accuracy of when using the formula "{global_explanation}": {accuracy:.4f}')
```

    Accuracy of when using the formula "(feature0000006749 & feature0000026329) | (feature0000010382 & feature0000013710 & feature0000021119 & feature0000025379 & feature0000026329 & feature0000027097 & ~feature0000013464) | (feature0000010382 & feature0000021119 & feature0000026329 & ~feature0000013464 & ~feature0000013710 & ~feature0000025379 & ~feature0000027097)": 0.5500

```python
w, b = collect_parameters(model)
feature_weights = w[0]
feature_used_bool = np.sum(np.abs(feature_weights), axis=0) > 0
feature_used = np.nonzero(feature_used_bool)[0]
len(feature_used)
```

    10

```python
counter
```

    Counter({'~feature0000003625 & feature0000005394 & ~feature0000006749 & ~feature0000010382 & ~feature0000013464 & feature0000013710 & feature0000021119 & feature0000022371 & ~feature0000025379 & feature0000026329': 1,
             'feature0000003625 & feature0000005394 & ~feature0000006749 & feature0000010382 & ~feature0000013464 & feature0000013710 & feature0000021119 & ~feature0000022371 & feature0000025379 & feature0000026329': 1,
             '~feature0000003625 & ~feature0000005394 & feature0000006749 & ~feature0000010382 & ~feature0000013464 & feature0000013710 & feature0000021119 & feature0000022371 & ~feature0000025379 & feature0000026329': 1,
             '~feature0000003625 & feature0000005394 & ~feature0000006749 & feature0000010382 & ~feature0000013464 & feature0000013710 & ~feature0000021119 & ~feature0000022371 & ~feature0000025379 & ~feature0000026329': 1,
             'feature0000003625 & ~feature0000005394 & feature0000006749 & feature0000010382 & ~feature0000013464 & ~feature0000013710 & ~feature0000021119 & ~feature0000022371 & feature0000025379 & feature0000026329': 1,
             'feature0000003625 & ~feature0000005394 & feature0000006749 & ~feature0000010382 & feature0000013464 & feature0000013710 & ~feature0000021119 & ~feature0000022371 & ~feature0000025379 & ~feature0000026329': 1,
             'feature0000003625 & feature0000005394 & ~feature0000006749 & feature0000010382 & ~feature0000013464 & ~feature0000013710 & feature0000021119 & ~feature0000022371 & feature0000025379 & feature0000026329': 1,
             'feature0000003625 & feature0000005394 & feature0000006749 & ~feature0000010382 & ~feature0000013464 & ~feature0000013710 & feature0000021119 & feature0000022371 & feature0000025379 & feature0000026329': 1,
             '~feature0000003625 & feature0000005394 & feature0000006749 & feature0000010382 & ~feature0000013464 & feature0000013710 & feature0000021119 & feature0000022371 & ~feature0000025379 & ~feature0000026329': 1,
             '~feature0000003625 & ~feature0000005394 & feature0000006749 & ~feature0000010382 & ~feature0000013464 & feature0000013710 & ~feature0000021119 & ~feature0000022371 & ~feature0000025379 & ~feature0000026329': 1,
             '~feature0000003625 & feature0000005394 & ~feature0000006749 & feature0000010382 & ~feature0000013464 & ~feature0000013710 & feature0000021119 & ~feature0000022371 & ~feature0000025379 & feature0000026329': 1,
             'feature0000003625 & feature0000005394 & ~feature0000006749 & feature0000010382 & ~feature0000013464 & feature0000013710 & ~feature0000021119 & ~feature0000022371 & ~feature0000025379 & ~feature0000026329': 1,
             '~feature0000003625 & ~feature0000005394 & feature0000006749 & feature0000010382 & ~feature0000013464 & ~feature0000013710 & ~feature0000021119 & feature0000022371 & ~feature0000025379 & ~feature0000026329': 1,
             'feature0000013710 & feature0000022371 & feature0000025379': 1,
             '~feature0000003625 & ~feature0000005394 & ~feature0000006749 & feature0000010382 & ~feature0000013464 & ~feature0000013710 & feature0000021119 & feature0000022371 & feature0000025379 & feature0000026329': 1,
             '~feature0000003625 & ~feature0000005394 & ~feature0000006749 & ~feature0000010382 & ~feature0000013464 & feature0000013710 & feature0000021119 & feature0000022371 & feature0000025379 & feature0000026329': 1,
             '~feature0000003625 & ~feature0000005394 & ~feature0000006749 & feature0000010382 & ~feature0000013464 & feature0000013710 & feature0000021119 & ~feature0000022371 & ~feature0000025379 & feature0000026329': 2,
             '~feature0000003625 & ~feature0000005394 & ~feature0000006749 & feature0000010382 & ~feature0000013464 & feature0000013710 & feature0000021119 & feature0000022371 & feature0000025379 & feature0000026329': 1,
             'feature0000003625 & feature0000005394 & feature0000006749 & ~feature0000010382 & ~feature0000013464 & ~feature0000013710 & feature0000021119 & ~feature0000022371 & ~feature0000025379 & feature0000026329': 2,
             'feature0000003625 & ~feature0000005394 & ~feature0000006749 & ~feature0000010382 & ~feature0000013464 & feature0000013710 & ~feature0000021119 & feature0000022371 & feature0000025379 & feature0000026329': 1,
             'feature0000003625 & feature0000022371 & feature0000025379': 1,
             'feature0000003625 & ~feature0000005394 & feature0000006749 & feature0000010382 & ~feature0000013464 & ~feature0000013710 & feature0000021119 & feature0000022371 & ~feature0000025379 & feature0000026329': 1,
             '~feature0000003625 & feature0000005394 & ~feature0000006749 & feature0000010382 & ~feature0000013464 & feature0000013710 & feature0000021119 & ~feature0000022371 & ~feature0000025379 & feature0000026329': 2,
             'feature0000003625 & ~feature0000005394 & ~feature0000006749 & ~feature0000010382 & ~feature0000013464 & feature0000013710 & feature0000021119 & ~feature0000022371 & ~feature0000025379 & feature0000026329': 1,
             'feature0000003625 & feature0000005394 & ~feature0000006749 & feature0000010382 & ~feature0000013464 & feature0000013710 & feature0000022371': 1,
             '~feature0000003625 & feature0000005394 & ~feature0000006749 & feature0000010382 & ~feature0000013464 & feature0000013710 & ~feature0000021119 & ~feature0000022371 & feature0000025379 & ~feature0000026329': 1,
             '~feature0000003625 & ~feature0000005394 & ~feature0000006749 & feature0000010382 & ~feature0000013464 & ~feature0000013710 & feature0000021119 & ~feature0000022371 & ~feature0000025379 & feature0000026329': 1})

```python
pd.DataFrame({
    'predictions': predictions.ravel(),
    'labels': y_train.detach().numpy().ravel(),
})
```

      predictions
      labels

      0
      False
      0.0

      1
      False
      0.0

      2
      False
      0.0

      3
      False
      0.0

      4
      False
      0.0

      5
      False
      0.0

      6
      False
      0.0

      7
      False
      0.0

      8
      False
      0.0

      9
      False
      0.0

      10
      True
      1.0

      11
      True
      1.0

      12
      True
      1.0

      13
      True
      1.0

      14
      True
      1.0

      15
      True
      1.0

      16
      True
      1.0

      17
      True
      1.0

      18
      True
      1.0

      19
      True
      1.0

      20
      True
      1.0

      21
      True
      1.0

      22
      True
      1.0

      23
      True
      1.0

      24
      True
      1.0

      25
      True
      1.0

      26
      True
      1.0

      27
      True
      1.0

      28
      True
      1.0

      29
      True
      1.0

      30
      True
      1.0

      31
      True
      1.0

      32
      True
      1.0

      33
      True
      1.0

      34
      True
      1.0

      35
      True
      1.0

      36
      True
      1.0

      37
      True
      1.0

      38
      True
      1.0

      39
      True
      1.0


## [mlcourse.ai](https://mlcourse.ai) – Open Machine Learning Course

Author: [Egor Polusmak](https://www.linkedin.com/in/egor-polusmak/). Translated and edited by [Yuanyuan Pao](https://www.linkedin.com/in/yuanyuanpao/). This material is subject to the terms and conditions of the [Creative Commons CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) license. Free use is permitted for any non-commercial purpose.

# Topic 9. Time series analysis in Python
## Part 2. Predicting the future with Facebook Prophet

Time series forecasting finds wide application in data analytics. These are only some of the conceivable predictions of future trends that might be useful:
- The number of servers that an online service will need next year.
- The demand for a grocery product at a supermarket on a given day.
- The tomorrow closing price of a tradable financial asset.

For another example, we can make a prediction of some team's performance and then use it as a baseline: first to set goals for the team, and then to measure the actual team performance relative to the baseline.

There are quite a few different methods to predict future trends, for example, [ARIMA](https://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average), [ARCH](https://en.wikipedia.org/wiki/Autoregressive_conditional_heteroskedasticity), [regressive models](https://en.wikipedia.org/wiki/Autoregressive_model), [neural networks](https://medium.com/machine-learning-world/neural-networks-for-algorithmic-trading-1-2-correct-time-series-forecasting-backtesting-9776bfd9e589).

In this article, we will look at [Prophet](https://facebook.github.io/prophet/), a library for time series forecasting released by Facebook and open-sourced on February 23, 2017. We will also try it out in the problem of predicting the daily number of posts published on Medium.

## Article outline

1. Introduction
2. The Prophet Forecasting Model
3. Practice with Prophet
    * 3.1 Installation in Python
    * 3.2 Dataset
    * 3.3 Exploratory visual analysis
    * 3.4 Making a forecast
    * 3.5 Forecast quality evaluation
    * 3.6 Visualization
4. Box-Cox Transformation
5. Summary
6. References

## 1. Introduction

According to the [article](https://research.fb.com/prophet-forecasting-at-scale/) on Facebook Research, Prophet was initially developed for the purpose of creating high quality business forecasts. This library tries to address the following difficulties common to many business time series:
- Seasonal effects caused by human behavior: weekly, monthly and yearly cycles, dips and peaks on public holidays.
- Changes in trend due to new products and market events.
- Outliers.

The authors claim that, even with the default settings, in many cases, their library produces forecasts as accurate as those delivered by experienced analysts.

Moreover, Prophet has a number of intuitive and easily interpretable customizations that allow gradually improving the quality of the forecasting model. What is especially important, these paramaters are quite comprehensible even for non-experts in time series analysis, which is a field of data science requiring certain skill and experience.

By the way, the original article is called "Forecasting at Scale", but it is not about the scale in the "usual" sense, that is addressing computational and infrastructure problems of a large number of working programs. According to the authors, Prophet should scale well in the following 3 areas:
- Accessibility to a wide audience of analysts, possibly without profound expertise in time series.
- Applicability to a wide range of distinct forecasting problems.
- Automated performance estimation of a large number of forecasts including flagging of potential problems for their subsequent inspection by the analyst.

## 2. The Prophet Forecasting Model

Now, let's take a closer look at how Prophet works. In its essence, this library utilizes the [additive regression model](https://en.wikipedia.org/wiki/Additive_model) $y(t)$ comprising the following components:

$$y(t) = g(t) + s(t) + h(t) + \epsilon_{t},$$

where:
* Trend $g(t)$ models non-periodic changes.
* Seasonality $s(t)$ represents periodic changes.
* Holidays component $h(t)$ contributes information about holidays and events.

Below, we will consider some important properties of these model components.

### Trend

The Prophet library implements two possible trend models for $g(t)$.

The first one is called *Nonlinear, Saturating Growth*. It is represented in the form of the [logistic growth model](https://en.wikipedia.org/wiki/Logistic_function):

$$g(t) = \frac{C}{1+e^{-k(t - m)}},$$

where:
* $C$ is the carrying capacity (that is the curve's maximum value).
* $k$ is the growth rate (which represents "the steepness" of the curve).
* $m$ is an offset parameter.

This logistic equation allows modelling non-linear growth with saturation, that is when the growth rate of a value decreases with its growth. One of the typical examples would be representing the growth of the audience of an application or a website.

Actually, $C$ and $k$ are not necessarily constants and may vary over time. Prophet supports both automatic and manual tuning of their variability. The library can itself choose optimal points of trend changes by fitting the supplied historical data.

Also, Prophet allows analysts to manually set changepoints of the growth rate and capacity values at different points in time. For instance, analysts may have insights about dates of past releases that prominently influenced some key product indicators.

The second trend model is a simple *Piecewise Linear Model* with a constant rate of growth. It is best suited for problems without saturating growth.

### Seasonality

The seasonal component $s(t)$ provides a flexible model of periodic changes due to weekly and yearly seasonality.

Weekly seasonal data is modeled with dummy variables. Six new variables are added: `monday`, `tuesday`, `wednesday`, `thursday`, `friday`, `saturday`, which take values 0 or 1 depending on the day of the week. The feature `sunday` is not added because it would be a linear combination of the other days of the week, and this fact would have an adverse effect on the model.

Yearly seasonality model in Prophet relies on Fourier series.

Since [version 0.2](https://github.com/facebook/prophet) you can also use *sub-daily time series* and make *sub-daily forecasts* as well as employ the new *daily seasonality* feature.

### Holidays and Events

The component $h(t)$ represents predictable abnormal days of the year including those on irregular schedules, e.g., Black Fridays.

To utilize this feature, the analyst needs to provide a custom list of events.

### Error

The error term $\epsilon(t)$ represents information that was not reflected in the model. Usually it is modeled as normally distributed noise.

### Prophet Benchmarking

For a detailed description of the model and algorithms behind Prophet refer to the paper ["Forecasting at scale"](https://peerj.com/preprints/3190/) by Sean J. Taylor and Benjamin Letham.

The authors also compared their library with several other methods for time series forecasting. They used [Mean Absolute Percentage Error (MAPE)](https://en.wikipedia.org/wiki/Mean_absolute_percentage_error) as a measure of prediction accuracy. In this research, Prophet has shown substantially lower forecasting error than the other models.

Let's look closer at how the forcasting quality was measured in the article. To do this, we will need the formula of Mean Absolute Percentage Error.

Let $y_{i}$ be the *actual (historical) value* and $\hat{y}_{i}$ be the *forecast value* given by our model.

Then $e_{i} = y_{i} - \hat{y}_{i}$ is the *forecast error* and $p_{i} =\frac{\displaystyle e_{i}}{\displaystyle y_{i}}$ is the *relative forecast error*.

We define

$$MAPE = mean\big(\left |p_{i} \right |\big)$$

MAPE is widely used as a measure of prediction accuracy because it expresses error as a percentage and thus can be used in model evaluations on different datasets.

In addition, when evaluating a forecasting algorithm, it may prove useful to calculate [MAE (Mean Absolute Error)](https://en.wikipedia.org/wiki/Mean_absolute_error) in order to have a picture of errors in absolute numbers. Using previously defined components, its equation will be

$$MAE = mean\big(\left |e_{i}\right |\big)$$

A few words about the algorithms that Prophet was compared with. Most of them are quite simple and often are used as a baseline for other models:
* `naive` is a simplistic forecasting approach where we predict all future values relying solely on the observation at the last available point of time.
* `snaive` (seasonal naive) is a model that makes constant predictions taking into account information about seasonality. For instance, in the case of weekly seasonal data for each future Monday, we would predict the value from the last Monday, and for all future Tuesdays we would use the value from the last Tuesday and so on.
* `mean` uses the averaged value of data as a forecast.
* `arima` stands for *Autoregressive Integrated Moving Average*, see [Wikipedia](https://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average) for details.
* `ets` stands for *Exponential Smoothing*, see [Wikipedia](https://en.wikipedia.org/wiki/Exponential_smoothing) for more.

## 3. Practice with Facebook Prophet

### 3.1 Installation in Python

First, you need to install the library. Prophet is available for Python and R. The choice will depend on your personal preferences and project requirements. Further in this article we will use Python.

In Python you can install Prophet using PyPI:
```
$ pip install fbprophet
```

In R you can find the corresponing CRAN package. Refer to the [documentation](https://facebookincubator.github.io/prophet/docs/installation.html) for details.

Let's import the modules that we will need, and initialize our environment:

```python
import warnings

warnings.filterwarnings("ignore")

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import statsmodels.api as sm
from scipy import stats

%matplotlib inline
```

### 3.2 Dataset

We will predict the daily number of posts published on [Medium](https://medium.com/).

First, we load our dataset.

```python
df = pd.read_csv("../../data/medium_posts.csv.zip", sep="\t")
```

Next, we leave out all columns except `published` and `url`. The former corresonds to the time dimension while the latter uniquely identifies a post by its URL. Along the way we get rid of possible duplicates and missing values in the data:

```python
df = df[["published", "url"]].dropna().drop_duplicates()
```

Next, we need to convert `published` to the datetime format because by default `pandas` treats this field as string-valued.

```python
df["published"] = pd.to_datetime(df["published"])
```

Let's sort the dataframe by time and take a look at what we've got:

```python
df.sort_values(by=["published"]).head(n=3)
```

Medium's public release date was August 15, 2012. But, as you can see from the data above, there are at least several rows with much earlier publication dates. They have somehow turned up in our dataset, but they are hardly legitimate ones. We will just trim our time series to keep only those rows that fall onto the period from August 15, 2012 to June 25, 2017:

```python
df = df[
    (df["published"] > "2012-08-15") & (df["published"] < "2017-06-26")
].sort_values(by=["published"])
df.head(n=3)
```

```python
df.tail(n=3)
```

As we are going to predict the number of published posts, we will aggregate and count unique posts at each given point in time. We will name the corresponding new column `posts`:

```python
aggr_df = df.groupby("published")[["url"]].count()
aggr_df.columns = ["posts"]
```

In this practice, we are interested in the number of posts **a day**. But at this moment all our data is divided into irregular time intervals that are less than a day. This is called a *sub-daily time series*. To see it, let's print out the first 3 rows:

```python
aggr_df.head(n=3)
```

To fix this, we need to aggregate the post counts by "bins" of a date size. In time series analysis, this process is referred to as *resampling*. And if we *reduce* the sampling rate of data it is often called *downsampling*.

Luckily, `pandas` has a built-in functionality for this task. We will resample our time index down to 1-day bins:

```python
daily_df = aggr_df.resample("D").apply(sum)
daily_df.head(n=3)
```

### 3.3 Exploratory visual analysis

As always, it may be helpful and instructive to look at a graphical representation of your data.

We will create a time series plot for the whole time range. Displaying data over such a long period of time can give clues about seasonality and conspicuous abnormal deviations.

First, we import and initialize the `Plotly` library, which allows creating beautiful interactive plots:

```python
from plotly import graph_objs as go
from plotly.offline import init_notebook_mode, iplot

# Initialize plotly
init_notebook_mode(connected=True)
```

We also define a helper function, which will plot our dataframes throughout the article:

```python
def plotly_df(df, title=""):
    """Visualize all the dataframe columns as line plots."""
    common_kw = dict(x=df.index, mode="lines")
    data = [go.Scatter(y=df[c], name=c, **common_kw) for c in df.columns]
    layout = dict(title=title)
    fig = dict(data=data, layout=layout)
    iplot(fig, show_link=False)
```

Let's try and plot our dataset *as is*:

```python
plotly_df(daily_df, title="Posts on Medium (daily)")
```

High-frequency data can be rather difficult to analyze. Even with the ability to zoom in provided by `Plotly`, it is hard to infer anything meaningful from this chart apart from the prominent upward and accelerating trend.

To reduce the noise, we will resample the post counts down to weekly bins. Besides *binning*, other possible techniques of noise reduction include [Moving-Average Smoothing](https://en.wikipedia.org/wiki/Moving_average) and [Exponential Smoothing](https://en.wikipedia.org/wiki/Exponential_smoothing), among others.

We save our downsampled dataframe in a separate variable because further in this practice we will work only with daily series:

```python
weekly_df = daily_df.resample("W").apply(sum)
```

Finally, we plot the result:

```python
plotly_df(weekly_df, title="Posts on Medium (weekly)")
```

This downsampled chart proves to be somewhat better for an analyst's perception.

One of the most useful functions that `Plotly` provides is the ability to quickly dive into different periods of timeline in order to better understand the data and find visual clues about possbile trends, periodic and irregular effects.

For example, zooming-in on a couple of consecutive years shows us time points corresponding to Christmas holidays, which greatly influence human behaviors.

Now, we're going to omit the first few years of observations, up to 2015. First, they won't contribute much into the forecast quality in 2017. Second, these first years, having very low number of posts per day, are likely to increase noise in our predictions, as the model would be forced to fit this abnormal historical data along with more relevant and indicative data from the recent years.

```python
daily_df = daily_df.loc[daily_df.index >= "2015-01-01"]
daily_df.head(n=3)
```

To sum up, from visual analysis we can see that our dataset is non-stationary with a prominent growing trend. It also demonstrates weekly and yearly seasonality and a number of abnormal days in each year.

### 3.4 Making a forecast

Prophet's API is very similar to the one you can find in `sklearn`. First we create a model, then call the method `fit`, and, finally, make a forecast. The input to the method `fit` is a `DataFrame` with two columns:
* `ds` (datestamp) must be of type `date` or `datetime`.
* `y` is a numeric value we want to predict.

To get started, we'll import the library and mute unimportant diagnostic messages:

```python
import logging

from fbprophet import Prophet

logging.getLogger().setLevel(logging.ERROR)
```

Let's convert our dataframe to the format required by Prophet:

```python
df = daily_df.reset_index()
df.columns = ["ds", "y"]
# converting timezones (issue https://github.com/facebook/prophet/issues/831)
df["ds"] = df["ds"].dt.tz_convert(None)
df.tail(n=3)
```

The authors of the library generally advise to make predictions based on at least several months, ideally, more than a year of historical data. Luckily, in our case we have more than a couple of years of data to fit the model.

To measure the quality of our forecast, we need to split our dataset into the *historical part*, which is the first and biggest slice of our data, and the *prediction part*, which will be located at the end of the timeline. We will remove the last month from the dataset in order to use it later as a prediction target:

```python
prediction_size = 30
train_df = df[:-prediction_size]
train_df.tail(n=3)
```

Now we need to create a new `Prophet` object. Here we can pass the parameters of the model into the constructor. But in this article we will use the defaults. Then we train our model by invoking its `fit` method on our training dataset:

```python
m = Prophet()
m.fit(train_df);
```

Using the helper method `Prophet.make_future_dataframe`, we create a dataframe which will contain all dates from the history and also extend into the future for those 30 days that we left out before.

```python
future = m.make_future_dataframe(periods=prediction_size)
future.tail(n=3)
```

We predict values with `Prophet` by passing in the dates for which we want to create a forecast. If we also supply the historical dates (as in our case), then in addition to the prediction we will get an in-sample fit for the history. Let's call the model's `predict` method with our `future` dataframe as an input:

```python
forecast = m.predict(future)
forecast.tail(n=3)
```

In the resulting dataframe you can see many columns characterizing the prediction, including trend and seasonality components as well as their confidence intervals. The forecast itself is stored in the `yhat` column.

The Prophet library has its own built-in tools for visualization that enable us to quickly evaluate the result.

First, there is a method called `Prophet.plot` that plots all the points from the forecast:

```python
m.plot(forecast);
```

This chart doesn't look very informative. The only definitive conclusion that we can draw here is that the model treated many of the data points as outliers.

The second function `Prophet.plot_components` might be much more useful in our case. It allows us to observe different components of the model separately: trend, yearly and weekly seasonality. In addition, if you supply information about holidays and events to your model, they will also be shown in this plot.

Let's try it out:

```python
m.plot_components(forecast);
```

As you can see from the trend graph, Prophet did a good job by fitting the accelerated growth of new posts at the end of 2016. The graph of weekly seasonality leads to the conclusion that usually there are less new posts on Saturdays and Sundays than on the other days of the week. In the yearly seasonality graph there is a prominent dip on Christmas Day.

### 3.5 Forecast quality evaluation

Let's evaluate the quality of the algorithm by calculating the error metrics for the last 30 days that we predicted. For this, we will need the observations $y_i$ and the corresponding predicted values $\hat{y}_i$.

Let's look into the object `forecast` that the library created for us:

```python
print(", ".join(forecast.columns))
```

We can see that this dataframe contains all the information we need except for the historical values. We need to join the `forecast` object with the actual values `y` from the original dataset `df`. For this we will define a helper function that we will reuse later:

```python
def make_comparison_dataframe(historical, forecast):
    """Join the history with the forecast.

       The resulting dataset will contain columns 'yhat', 'yhat_lower', 'yhat_upper' and 'y'.
    """
    return forecast.set_index("ds")[["yhat", "yhat_lower", "yhat_upper"]].join(
        historical.set_index("ds")
    )
```

Let's apply this function to our last forecast:

```python
cmp_df = make_comparison_dataframe(df, forecast)
cmp_df.tail(n=3)
```

We are also going to define a helper function that we will use to gauge the quality of our forecasting with MAPE and MAE error measures:

```python
def calculate_forecast_errors(df, prediction_size):
    """Calculate MAPE and MAE of the forecast.

       Args:
           df: joined dataset with 'y' and 'yhat' columns.
           prediction_size: number of days at the end to predict.
    """

    # Make a copy
    df = df.copy()

    # Now we calculate the values of e_i and p_i according to the formulas given in the article above.
    df["e"] = df["y"] - df["yhat"]
    df["p"] = 100 * df["e"] / df["y"]

    # Recall that we held out the values of the last `prediction_size` days
    # in order to predict them and measure the quality of the model.

    # Now cut out the part of the data which we made our prediction for.
    predicted_part = df[-prediction_size:]

    # Define the function that averages absolute error values over the predicted part.
    error_mean = lambda error_name: np.mean(np.abs(predicted_part[error_name]))

    # Now we can calculate MAPE and MAE and return the resulting dictionary of errors.
    return {"MAPE": error_mean("p"), "MAE": error_mean("e")}
```

Let's use our function:

```python
for err_name, err_value in calculate_forecast_errors(cmp_df, prediction_size).items():
    print(err_name, err_value)
```

As a result, the relative error of our forecast (MAPE) is about 22.6%, and on average our model is wrong by ~70 posts (MAE).

### 3.6 Visualization

Let's create our own visualization of the model built by Prophet. It will comprise the actual values, forecast and confidence intervals.

First, we will plot the data for a shorter period of time to make the data points easier to distinguish. Second, we will show the model performance only for the period that we predicted, that is the last 30 days. It seems that these two measures should give us a more legible plot.

Third, we will use `Plotly` to make our chart interactive, which is great for exploring.

We will define a custom helper function `show_forecast` and call it (for more on how it works please refer to the comments in the code and the [documentation](https://plot.ly/python/)):

```python
def show_forecast(cmp_df, num_predictions, num_values, title):
    """Visualize the forecast."""

    def create_go(name, column, num, **kwargs):
        points = cmp_df.tail(num)
        args = dict(name=name, x=points.index, y=points[column], mode="lines")
        args.update(kwargs)
        return go.Scatter(**args)

    lower_bound = create_go(
        "Lower Bound",
        "yhat_lower",
        num_predictions,
        line=dict(width=0),
        marker=dict(color="gray"),
    )
    upper_bound = create_go(
        "Upper Bound",
        "yhat_upper",
        num_predictions,
        line=dict(width=0),
        marker=dict(color="gray"),
        fillcolor="rgba(68, 68, 68, 0.3)",
        fill="tonexty",
    )
    forecast = create_go(
        "Forecast", "yhat", num_predictions, line=dict(color="rgb(31, 119, 180)")
    )
    actual = create_go("Actual", "y", num_values, marker=dict(color="red"))

    # In this case the order of the series is important because of the filling
    data = [lower_bound, upper_bound, forecast, actual]

    layout = go.Layout(yaxis=dict(title="Posts"), title=title, showlegend=False)
    fig = go.Figure(data=data, layout=layout)
    iplot(fig, show_link=False)

show_forecast(cmp_df, prediction_size, 100, "New posts on Medium")
```

At first glance, the prediction of the mean values by our model seems to be sensible. The high value of MAPE that we got above may be explained by the fact that the model failed to catch on to increasing peak-to-peak amplitude of weakly seasonality.

Also, we can conclude from the graph above that many of the actual values lie outside the confidence interval. Prophet may not be suitable for time series with unstable variance, at least when the default settings are used. We will try to fix this by applying a transform to our data.

## 4. Box-Cox Transformation

So far we have used Prophet with the default settings and the original data. We will leave the parameters of the model alone. But despite this we still have some room for improvement. In this section, we will apply the [Box–Cox transformation](http://onlinestatbook.com/2/transformations/box-cox.html) to our original series. Let's see where it will lead us.

A few words about this transformation. This is a monotonic data transformation that can be used to stabilize variance. We will use the one-parameter Box–Cox transformation, which is defined by the following expression:

$$
\begin{equation}
  boxcox^{(\lambda)}(y_{i}) = \begin{cases}
    \frac{\displaystyle y_{i}^{\lambda} - 1}{\displaystyle \lambda} &, \text{if $\lambda \neq 0$}.\\
    ln(y_{i}) &, \text{if $\lambda = 0$}.
  \end{cases}
\end{equation}
$$

We will need to implement the inverse of this function in order to be able to restore the original data scale. It is easy to see that the inverse is defined as:

$$
\begin{equation}
  invboxcox^{(\lambda)}(y_{i}) = \begin{cases}
    e^{\left (\frac{\displaystyle ln(\lambda y_{i} + 1)}{\displaystyle \lambda} \right )} &, \text{if $\lambda \neq 0$}.\\
    e^{y_{i}} &, \text{if $\lambda = 0$}.
  \end{cases}
\end{equation}
$$

The corresponding function in Python is implemented as follows:

```python
def inverse_boxcox(y, lambda_):
    return np.exp(y) if lambda_ == 0 else np.exp(np.log(lambda_ * y + 1) / lambda_)
```

First, we prepare our dataset by setting its index:

```python
train_df2 = train_df.copy().set_index("ds")
```

Then, we apply the function `stats.boxcox` from `Scipy`, which applies the Box–Cox transformation. In our case it will return two values. The first one is the transformed series and the second one is the found value of $\lambda$ that is optimal in terms of the maximum log-likelihood:

```python
train_df2["y"], lambda_prophet = stats.boxcox(train_df2["y"])
train_df2.reset_index(inplace=True)
```

We create a new `Prophet` model and repeat the fit-predict cycle that we have already done above:

```python
m2 = Prophet()
m2.fit(train_df2)
future2 = m2.make_future_dataframe(periods=prediction_size)
forecast2 = m2.predict(future2)
```

At this point, we need to revert the Box–Cox transformation with our inverse function and the known value of $\lambda$:

```python
for column in ["yhat", "yhat_lower", "yhat_upper"]:
    forecast2[column] = inverse_boxcox(forecast2[column], lambda_prophet)
```

Here we will reuse our tools for making the comparison dataframe and calculating the errors:

```python
cmp_df2 = make_comparison_dataframe(df, forecast2)
for err_name, err_value in calculate_forecast_errors(cmp_df2, prediction_size).items():
    print(err_name, err_value)
```

So, we can definitely state an increase in the quality of the model.

Finally, let's plot our previous performance with the latest results side-by-side. Note that we use `prediction_size` for the third parameter in order to zoom in on the interval being predicted:

```python
show_forecast(cmp_df, prediction_size, 100, "No transformations")
show_forecast(cmp_df2, prediction_size, 100, "Box–Cox transformation")
```

We see that the forecast of weekly changes in the second graph is much closer to the real values now.

## 5. Summary

We have taken a look at *Prophet*, an open-source forecasting library that is specifically targeted at business time series. We have also done some hands-on practice in time series prediction.

As we have seen, the Prophet library does not make wonders, and its predictions out-of-box are not [ideal](https://en.wikipedia.org/wiki/No_free_lunch_in_search_and_optimization). It is still up to the data scientist to explore the forecast results, tune model parameters and transform data when necessary.

However, this library is user-friendly and easily customizable. The sole ability to take into account abnormal days that are known to the analyst beforehand might make a difference in some cases.

All in all, the Prophet library is worth being a part of your analytical toolbox.

## 6. References

- Official [Prophet repository](https://github.com/facebookincubator/prophet) on GitHub.
- Official [Prophet documentation](https://facebookincubator.github.io/prophet/docs/quick_start.html).
- Sean J. Taylor, Benjamin Letham ["Forecasting at scale"](https://facebookincubator.github.io/prophet/static/prophet_paper_20170113.pdf) — scientific paper explaining the algorithm which lays the foundation of `Prophet`.
- [Forecasting Website Traffic Using Facebook’s Prophet Library](http://pbpython.com/prophet-overview.html) — `Prophet` overview with an example of website traffic forecasting.
- Rob J. Hyndman, George Athanasopoulos ["Forecasting: principles and practice"](https://www.otexts.org/fpp) – a very good online book about time series forecasting.