## Variational Autoencoder

A variational autoencoder is a generative neural network model consisting of an encoder, a sampling layer and a decoder. Autoencoders are generally used to learn a latent representation of data by creating an information bottleneck through a pyramid-like structure of the encoder where each layers has fewer degrees of freedom compared to the previous layer. This results in dimensionality reduction of the input. The decoder on the other hand takes a compressed latent representation and reconstructs the original input. In order to make an autoencoder generative, we learn a variational approximation of the hidden layer.



Autoencoders are data driven and application specific, for example an autoencoder trained on digits will not perform well on images of faces. Autoencoders result in lossy compression. They are useful in data denoising and dimensionality reduction. Training variational autoencoders consists of optimizing a custom loss function consisting of reconstruction loss and regularization loss:

\begin{equation}
    \min H(p,q) + KL(z||pz)
\end{equation}

where $H(p,q)$ is the cross-entropy reconstruction loss defined as $-\sum_i p_i \log q_i$ where $p$ is the ground truth distribution and $q$ is the predicted distribution and $KL(q||z)$ is the regularization loss that constraints our variational approximation $z$ to be close to our prior $pz \sim N(0,1)$ which is assumed to be a standard normal in our example. The regularization loss forces similar inputs to be close in the latent space. In variational autoencoder, we learn the variational approximation parameters ($\mu$ and $\log \sigma$) using the neural network. Thus, we can construct our approximate distribution $z$ as follows:

\begin{equation}
    z = \mu + \exp(\log \sigma) \times N(0,1)
\end{equation}

As we'll see in the following example, we'll be able to generate an image manifold by sampling from our distribution $z \sim N(\mu, \sigma^2)$ above and feeding the samples through a decoder to generate an image.


```python
%matplotlib inline
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt

import math
from scipy.stats import norm

import keras
from keras.layers import Input, Dense, Lambda, Layer
from keras.models import Model
from keras import backend as K
from keras import metrics
from keras import optimizers
from keras.datasets import mnist

from keras.callbacks import ModelCheckpoint
from keras.callbacks import TensorBoard
from keras.callbacks import LearningRateScheduler 
from keras.callbacks import EarlyStopping

from keras.utils import plot_model
```

    Using TensorFlow backend.


    Couldn't import dot_parser, loading of dot files will not be possible.


In training variational autoencoder, we'll use a learning rate schedule that divides the initial learning rate by 2 every fixed number of epochs, we'll also use a callback to monitor the learning rate.


```python
DATA_PATH = '/data/vision/fisher/data1/vsmolyakov/vae/'

def step_decay(epoch):
    lr_init = 0.001
    drop = 0.5
    epochs_drop = 16.0
    lr_new = lr_init * math.pow(drop, math.floor((1+epoch)/epochs_drop))
    return lr_new

class LR_hist(keras.callbacks.Callback):
    def on_train_begin(self, logs={}):
        self.losses = []
        self.lr = []
    def on_epoch_end(self, batch, logs={}):
        self.losses.append(logs.get('loss'))
        self.lr.append(step_decay(len(self.losses)))
```

Let's load the data and define training and model parameters:


```python
#load data
print "loading data..."
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train = x_train.astype('float32') / 255.0
x_test = x_test.astype('float32') / 255.0
x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))
x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))

#training params
batch_size = 100
num_epochs = 128 

#model params
original_dim = 784
intermediate_dim = 256
latent_dim = 2
epsilon_std = 1.0
```

    loading data...


We'll implement the sampling layer using a Lambda layer in Keras.


```python
#sampling layer
def sampling(args):
    z_mean, z_log_sigma = args
    epsilon = K.random_normal(shape=(K.shape(z_mean)[0], latent_dim), mean=0.0, stddev=epsilon_std)
    return z_mean + K.exp(z_log_sigma) * epsilon
```

Let's define the variational autoencoder architecture:


```python
#VAE architecture
#encoder
x = Input(shape=(original_dim,))
h = Dense(intermediate_dim, activation='relu')(x)
z_mean = Dense(latent_dim)(h)
z_log_sigma = Dense(latent_dim)(h)

#sampling
z = Lambda(sampling, output_shape=(latent_dim,))([z_mean, z_log_sigma])

#decoder
decoder_h = Dense(intermediate_dim, activation='relu')
decoder_mean = Dense(original_dim, activation='sigmoid')
h_decoded = decoder_h(z)
x_decoded_mean = decoder_mean(h_decoded)
```

Because our loss function consists of two terms, we have to define a custom loss layer:


```python
#custom loss layer
class CustomVariationalLayer(Layer):
    def __init__(self, **kwargs):
        self.is_placeholder = True
        super(CustomVariationalLayer, self).__init__(**kwargs)

    def vae_loss(self, x, x_decoded_mean):
        xent_loss = original_dim * metrics.binary_crossentropy(x, x_decoded_mean)
        kl_loss = - 0.5 * K.sum(1 + z_log_sigma - K.square(z_mean) - K.exp(z_log_sigma), axis=-1)
        return K.mean(xent_loss + kl_loss)

    def call(self, inputs):
        x = inputs[0]
        x_decoded_mean = inputs[1]
        loss = self.vae_loss(x, x_decoded_mean)
        self.add_loss(loss, inputs=inputs)
        return x

```

We can finally compile our variational autoencoder model using Adam optimizer:


```python
y = CustomVariationalLayer()([x, x_decoded_mean])
vae = Model(x, y)

adam = optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)
vae.compile(loss=None, optimizer=adam, metrics=['accuracy'])
vae.summary()
```

    ____________________________________________________________________________________________________
    Layer (type)                     Output Shape          Param #     Connected to                     
    ====================================================================================================
    input_1 (InputLayer)             (None, 784)           0                                            
    ____________________________________________________________________________________________________
    dense_1 (Dense)                  (None, 256)           200960      input_1[0][0]                    
    ____________________________________________________________________________________________________
    dense_2 (Dense)                  (None, 2)             514         dense_1[0][0]                    
    ____________________________________________________________________________________________________
    dense_3 (Dense)                  (None, 2)             514         dense_1[0][0]                    
    ____________________________________________________________________________________________________
    lambda_1 (Lambda)                (None, 2)             0           dense_2[0][0]                    
                                                                       dense_3[0][0]                    
    ____________________________________________________________________________________________________
    dense_4 (Dense)                  (None, 256)           768         lambda_1[0][0]                   
    ____________________________________________________________________________________________________
    dense_5 (Dense)                  (None, 784)           201488      dense_4[0][0]                    
    ____________________________________________________________________________________________________
    custom_variational_layer_1 (Cust [(None, 784), (None,  0           input_1[0][0]                    
                                                                       dense_5[0][0]                    
    ====================================================================================================
    Total params: 404,244
    Trainable params: 404,244
    Non-trainable params: 0
    ____________________________________________________________________________________________________


    /usr/local/lib/python2.7/dist-packages/ipykernel_launcher.py:5: UserWarning: Output "custom_variational_layer_1" missing from loss dictionary. We assume this was done on purpose, and we will not be expecting any data to be passed to "custom_variational_layer_1" during training.
      """



```python
#define callbacks
file_name = DATA_PATH + 'weights-checkpoint.hdf5'
checkpoint = ModelCheckpoint(file_name, monitor='loss', verbose=1, save_best_only=True, mode='min')
tensor_board = TensorBoard(log_dir='./logs', write_graph=True)
hist_lr = LR_hist()
reduce_lr = LearningRateScheduler(step_decay) 
early_stopping = EarlyStopping(monitor='loss', min_delta=0.01, patience=64, verbose=1)
callbacks_list = [checkpoint, tensor_board, hist_lr, reduce_lr, early_stopping]

```


```python
print "training vae model..."
hist = vae.fit(x_train, shuffle=True, epochs=num_epochs, batch_size=batch_size, callbacks=callbacks_list, validation_data=(x_test, None), verbose=2)
```

    training vae model...
    Train on 60000 samples, validate on 10000 samples
    Epoch 1/128
    Epoch 00000: loss improved from inf to 207.46185, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 207.4619 - val_loss: 181.7194
    Epoch 2/128
    Epoch 00001: loss improved from 207.46185 to 175.27829, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 175.2783 - val_loss: 170.4656
    Epoch 3/128
    Epoch 00002: loss improved from 175.27829 to 168.62863, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 168.6286 - val_loss: 166.0275
    Epoch 4/128
    Epoch 00003: loss improved from 168.62863 to 165.28341, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 165.2834 - val_loss: 163.8910
    Epoch 5/128
    Epoch 00004: loss improved from 165.28341 to 163.33535, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 163.3354 - val_loss: 162.1486
    Epoch 6/128
    Epoch 00005: loss improved from 163.33535 to 161.73204, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 161.7320 - val_loss: 160.7561
    Epoch 7/128
    Epoch 00006: loss improved from 161.73204 to 160.28968, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 160.2897 - val_loss: 159.2922
    Epoch 8/128
    Epoch 00007: loss improved from 160.28968 to 158.86565, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 158.8656 - val_loss: 158.5544
    Epoch 9/128
    Epoch 00008: loss improved from 158.86565 to 157.58157, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 157.5816 - val_loss: 156.9721
    Epoch 10/128
    Epoch 00009: loss improved from 157.58157 to 156.33295, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 156.3329 - val_loss: 155.6949
    Epoch 11/128
    Epoch 00010: loss improved from 156.33295 to 155.37949, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 155.3795 - val_loss: 155.2493
    Epoch 12/128
    Epoch 00011: loss improved from 155.37949 to 154.52507, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 154.5251 - val_loss: 154.0871
    Epoch 13/128
    Epoch 00012: loss improved from 154.52507 to 153.86931, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 153.8693 - val_loss: 153.7534
    Epoch 14/128
    Epoch 00013: loss improved from 153.86931 to 153.30271, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 153.3027 - val_loss: 153.2158
    Epoch 15/128
    Epoch 00014: loss improved from 153.30271 to 152.72257, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 152.7226 - val_loss: 152.9466
    Epoch 16/128
    Epoch 00015: loss improved from 152.72257 to 151.76216, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 151.7622 - val_loss: 152.1998
    Epoch 17/128
    Epoch 00016: loss improved from 151.76216 to 151.46707, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 151.4671 - val_loss: 152.0505
    Epoch 18/128
    Epoch 00017: loss improved from 151.46707 to 151.21640, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 151.2164 - val_loss: 151.9097
    Epoch 19/128
    Epoch 00018: loss improved from 151.21640 to 150.98383, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 150.9838 - val_loss: 151.8249
    Epoch 20/128
    Epoch 00019: loss improved from 150.98383 to 150.73573, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 150.7357 - val_loss: 151.6568
    Epoch 21/128
    Epoch 00020: loss improved from 150.73573 to 150.51922, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 150.5192 - val_loss: 151.6240
    Epoch 22/128
    Epoch 00021: loss improved from 150.51922 to 150.30904, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 150.3090 - val_loss: 151.5045
    Epoch 23/128
    Epoch 00022: loss improved from 150.30904 to 150.11874, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 150.1187 - val_loss: 151.5699
    Epoch 24/128
    Epoch 00023: loss improved from 150.11874 to 149.93569, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 149.9357 - val_loss: 151.2348
    Epoch 25/128
    Epoch 00024: loss improved from 149.93569 to 149.73812, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 149.7381 - val_loss: 151.1765
    Epoch 26/128
    Epoch 00025: loss improved from 149.73812 to 149.57059, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 149.5706 - val_loss: 151.0119
    Epoch 27/128
    Epoch 00026: loss improved from 149.57059 to 149.39235, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 149.3923 - val_loss: 151.0611
    Epoch 28/128
    Epoch 00027: loss improved from 149.39235 to 149.21818, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 149.2182 - val_loss: 150.7763
    Epoch 29/128
    Epoch 00028: loss improved from 149.21818 to 149.08684, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 149.0868 - val_loss: 150.8500
    Epoch 30/128
    Epoch 00029: loss improved from 149.08684 to 148.90517, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 148.9052 - val_loss: 150.8162
    Epoch 31/128
    Epoch 00030: loss improved from 148.90517 to 148.76809, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 148.7681 - val_loss: 150.8209
    Epoch 32/128
    Epoch 00031: loss improved from 148.76809 to 148.26388, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 148.2639 - val_loss: 150.3329
    Epoch 33/128
    Epoch 00032: loss improved from 148.26388 to 148.16450, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 148.1645 - val_loss: 150.4133
    Epoch 34/128
    Epoch 00033: loss improved from 148.16450 to 148.07045, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 148.0705 - val_loss: 150.2414
    Epoch 35/128
    Epoch 00034: loss improved from 148.07045 to 147.99750, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 147.9975 - val_loss: 150.2835
    Epoch 36/128
    Epoch 00035: loss improved from 147.99750 to 147.91649, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 147.9165 - val_loss: 150.2635
    Epoch 37/128
    Epoch 00036: loss improved from 147.91649 to 147.84790, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 147.8479 - val_loss: 150.2183
    Epoch 38/128
    Epoch 00037: loss improved from 147.84790 to 147.75718, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 147.7572 - val_loss: 150.2958
    Epoch 39/128
    Epoch 00038: loss improved from 147.75718 to 147.68742, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 147.6874 - val_loss: 150.3576
    Epoch 40/128
    Epoch 00039: loss improved from 147.68742 to 147.61803, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 147.6180 - val_loss: 150.1109
    Epoch 41/128
    Epoch 00040: loss improved from 147.61803 to 147.53446, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 147.5345 - val_loss: 150.1735
    Epoch 42/128
    Epoch 00041: loss improved from 147.53446 to 147.47079, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 147.4708 - val_loss: 150.1849
    Epoch 43/128
    Epoch 00042: loss improved from 147.47079 to 147.38060, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    10s - loss: 147.3806 - val_loss: 150.1782
    Epoch 44/128
    Epoch 00043: loss improved from 147.38060 to 147.32383, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    10s - loss: 147.3238 - val_loss: 150.2023
    Epoch 45/128
    Epoch 00044: loss improved from 147.32383 to 147.23673, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    10s - loss: 147.2367 - val_loss: 150.0926
    Epoch 46/128
    Epoch 00045: loss improved from 147.23673 to 147.17098, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 147.1710 - val_loss: 150.0273
    Epoch 47/128
    Epoch 00046: loss improved from 147.17098 to 147.11243, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 147.1124 - val_loss: 149.9769
    Epoch 48/128
    Epoch 00047: loss improved from 147.11243 to 146.86245, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 146.8624 - val_loss: 149.8431
    Epoch 49/128
    Epoch 00048: loss improved from 146.86245 to 146.81145, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 146.8115 - val_loss: 149.8324
    Epoch 50/128
    Epoch 00049: loss improved from 146.81145 to 146.78653, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 146.7865 - val_loss: 149.8825
    Epoch 51/128
    Epoch 00050: loss improved from 146.78653 to 146.72616, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    10s - loss: 146.7262 - val_loss: 149.8740
    Epoch 52/128
    Epoch 00051: loss improved from 146.72616 to 146.70117, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 146.7012 - val_loss: 149.8225
    Epoch 53/128
    Epoch 00052: loss improved from 146.70117 to 146.66899, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 146.6690 - val_loss: 149.8604
    Epoch 54/128
    Epoch 00053: loss improved from 146.66899 to 146.62405, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 146.6241 - val_loss: 149.8451
    Epoch 55/128
    Epoch 00054: loss improved from 146.62405 to 146.57467, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 146.5747 - val_loss: 149.8275
    Epoch 56/128
    Epoch 00055: loss improved from 146.57467 to 146.54511, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 146.5451 - val_loss: 149.8263
    Epoch 57/128
    Epoch 00056: loss improved from 146.54511 to 146.51676, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 146.5168 - val_loss: 149.8087
    Epoch 58/128
    Epoch 00057: loss improved from 146.51676 to 146.49652, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    10s - loss: 146.4965 - val_loss: 149.8314
    Epoch 59/128
    Epoch 00058: loss improved from 146.49652 to 146.44952, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    10s - loss: 146.4495 - val_loss: 149.8319
    Epoch 60/128
    Epoch 00059: loss improved from 146.44952 to 146.41819, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 146.4182 - val_loss: 149.8506
    Epoch 61/128
    Epoch 00060: loss improved from 146.41819 to 146.38260, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 146.3826 - val_loss: 149.7496
    Epoch 62/128
    Epoch 00061: loss improved from 146.38260 to 146.36248, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 146.3625 - val_loss: 149.8067
    Epoch 63/128
    Epoch 00062: loss improved from 146.36248 to 146.31845, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    10s - loss: 146.3185 - val_loss: 149.7689
    Epoch 64/128
    Epoch 00063: loss improved from 146.31845 to 146.19048, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 146.1905 - val_loss: 149.7059
    Epoch 65/128
    Epoch 00064: loss improved from 146.19048 to 146.15993, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 146.1599 - val_loss: 149.6759
    Epoch 66/128
    Epoch 00065: loss improved from 146.15993 to 146.15078, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 146.1508 - val_loss: 149.7215
    Epoch 67/128
    Epoch 00066: loss improved from 146.15078 to 146.12988, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 146.1299 - val_loss: 149.6360
    Epoch 68/128
    Epoch 00067: loss improved from 146.12988 to 146.11893, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 146.1189 - val_loss: 149.6864
    Epoch 69/128
    Epoch 00068: loss improved from 146.11893 to 146.08901, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    10s - loss: 146.0890 - val_loss: 149.7056
    Epoch 70/128
    Epoch 00069: loss improved from 146.08901 to 146.08488, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 146.0849 - val_loss: 149.7462
    Epoch 71/128
    Epoch 00070: loss improved from 146.08488 to 146.05973, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 146.0597 - val_loss: 149.6856
    Epoch 72/128
    Epoch 00071: loss improved from 146.05973 to 146.04602, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 146.0460 - val_loss: 149.6931
    Epoch 73/128
    Epoch 00072: loss improved from 146.04602 to 146.02700, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 146.0270 - val_loss: 149.7390
    Epoch 74/128
    Epoch 00073: loss improved from 146.02700 to 145.99736, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    10s - loss: 145.9974 - val_loss: 149.7076
    Epoch 75/128
    Epoch 00074: loss improved from 145.99736 to 145.99558, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    10s - loss: 145.9956 - val_loss: 149.7026
    Epoch 76/128
    Epoch 00075: loss improved from 145.99558 to 145.99270, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    10s - loss: 145.9927 - val_loss: 149.6258
    Epoch 77/128
    Epoch 00076: loss improved from 145.99270 to 145.94872, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    10s - loss: 145.9487 - val_loss: 149.6698
    Epoch 78/128
    Epoch 00077: loss did not improve
    9s - loss: 145.9548 - val_loss: 149.6526
    Epoch 79/128
    Epoch 00078: loss improved from 145.94872 to 145.92872, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    10s - loss: 145.9287 - val_loss: 149.6328
    Epoch 80/128
    Epoch 00079: loss improved from 145.92872 to 145.85481, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    10s - loss: 145.8548 - val_loss: 149.6230
    Epoch 81/128
    Epoch 00080: loss improved from 145.85481 to 145.84402, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    10s - loss: 145.8440 - val_loss: 149.6277
    Epoch 82/128
    Epoch 00081: loss improved from 145.84402 to 145.84301, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 145.8430 - val_loss: 149.6380
    Epoch 83/128
    Epoch 00082: loss improved from 145.84301 to 145.82151, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 145.8215 - val_loss: 149.6593
    Epoch 84/128
    Epoch 00083: loss did not improve
    9s - loss: 145.8283 - val_loss: 149.6599
    Epoch 85/128
    Epoch 00084: loss did not improve
    9s - loss: 145.8226 - val_loss: 149.6392
    Epoch 86/128
    Epoch 00085: loss improved from 145.82151 to 145.79993, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    10s - loss: 145.7999 - val_loss: 149.6925
    Epoch 87/128
    Epoch 00086: loss did not improve
    9s - loss: 145.8020 - val_loss: 149.6411
    Epoch 88/128
    Epoch 00087: loss improved from 145.79993 to 145.78678, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    10s - loss: 145.7868 - val_loss: 149.6104
    Epoch 89/128
    Epoch 00088: loss improved from 145.78678 to 145.77177, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    10s - loss: 145.7718 - val_loss: 149.6495
    Epoch 90/128
    Epoch 00089: loss did not improve
    9s - loss: 145.7738 - val_loss: 149.5923
    Epoch 91/128
    Epoch 00090: loss improved from 145.77177 to 145.76849, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    10s - loss: 145.7685 - val_loss: 149.5719
    Epoch 92/128
    Epoch 00091: loss improved from 145.76849 to 145.74702, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    10s - loss: 145.7470 - val_loss: 149.6446
    Epoch 93/128
    Epoch 00092: loss did not improve
    9s - loss: 145.7517 - val_loss: 149.5809
    Epoch 94/128
    Epoch 00093: loss improved from 145.74702 to 145.72771, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    11s - loss: 145.7277 - val_loss: 149.6039
    Epoch 95/128
    Epoch 00094: loss improved from 145.72771 to 145.72395, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    10s - loss: 145.7240 - val_loss: 149.6169
    Epoch 96/128
    Epoch 00095: loss improved from 145.72395 to 145.68434, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    10s - loss: 145.6843 - val_loss: 149.5976
    Epoch 97/128
    Epoch 00096: loss did not improve
    9s - loss: 145.6873 - val_loss: 149.6068
    Epoch 98/128
    Epoch 00097: loss improved from 145.68434 to 145.67932, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    10s - loss: 145.6793 - val_loss: 149.6142
    Epoch 99/128
    Epoch 00098: loss improved from 145.67932 to 145.66919, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    10s - loss: 145.6692 - val_loss: 149.5717
    Epoch 100/128
    Epoch 00099: loss did not improve
    9s - loss: 145.6815 - val_loss: 149.6050
    Epoch 101/128
    Epoch 00100: loss improved from 145.66919 to 145.65051, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    10s - loss: 145.6505 - val_loss: 149.5857
    Epoch 102/128
    Epoch 00101: loss did not improve
    9s - loss: 145.6612 - val_loss: 149.5564
    Epoch 103/128
    Epoch 00102: loss did not improve
    9s - loss: 145.6711 - val_loss: 149.6020
    Epoch 104/128
    Epoch 00103: loss did not improve
    9s - loss: 145.6557 - val_loss: 149.5986
    Epoch 105/128
    Epoch 00104: loss improved from 145.65051 to 145.64327, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 145.6433 - val_loss: 149.5855
    Epoch 106/128
    Epoch 00105: loss did not improve
    9s - loss: 145.6450 - val_loss: 149.5933
    Epoch 107/128
    Epoch 00106: loss improved from 145.64327 to 145.63598, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 145.6360 - val_loss: 149.5353
    Epoch 108/128
    Epoch 00107: loss improved from 145.63598 to 145.63409, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 145.6341 - val_loss: 149.6438
    Epoch 109/128
    Epoch 00108: loss improved from 145.63409 to 145.63216, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 145.6322 - val_loss: 149.5546
    Epoch 110/128
    Epoch 00109: loss did not improve
    9s - loss: 145.6322 - val_loss: 149.5500
    Epoch 111/128
    Epoch 00110: loss improved from 145.63216 to 145.63096, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    10s - loss: 145.6310 - val_loss: 149.6161
    Epoch 112/128
    Epoch 00111: loss improved from 145.63096 to 145.60660, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    10s - loss: 145.6066 - val_loss: 149.5684
    Epoch 113/128
    Epoch 00112: loss improved from 145.60660 to 145.60015, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 145.6002 - val_loss: 149.5835
    Epoch 114/128
    Epoch 00113: loss improved from 145.60015 to 145.59763, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 145.5976 - val_loss: 149.5670
    Epoch 115/128
    Epoch 00114: loss improved from 145.59763 to 145.59681, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 145.5968 - val_loss: 149.5997
    Epoch 116/128
    Epoch 00115: loss did not improve
    9s - loss: 145.5982 - val_loss: 149.5843
    Epoch 117/128
    Epoch 00116: loss did not improve
    9s - loss: 145.6055 - val_loss: 149.5793
    Epoch 118/128
    Epoch 00117: loss improved from 145.59681 to 145.58814, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 145.5881 - val_loss: 149.6058
    Epoch 119/128
    Epoch 00118: loss did not improve
    9s - loss: 145.5951 - val_loss: 149.6007
    Epoch 120/128
    Epoch 00119: loss did not improve
    9s - loss: 145.5903 - val_loss: 149.5758
    Epoch 121/128
    Epoch 00120: loss did not improve
    9s - loss: 145.5914 - val_loss: 149.5590
    Epoch 122/128
    Epoch 00121: loss improved from 145.58814 to 145.58057, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 145.5806 - val_loss: 149.5911
    Epoch 123/128
    Epoch 00122: loss improved from 145.58057 to 145.58024, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 145.5802 - val_loss: 149.5867
    Epoch 124/128
    Epoch 00123: loss did not improve
    9s - loss: 145.5820 - val_loss: 149.5984
    Epoch 125/128
    Epoch 00124: loss did not improve
    9s - loss: 145.5804 - val_loss: 149.5401
    Epoch 126/128
    Epoch 00125: loss improved from 145.58024 to 145.56877, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 145.5688 - val_loss: 149.5702
    Epoch 127/128
    Epoch 00126: loss did not improve
    9s - loss: 145.5734 - val_loss: 149.6175
    Epoch 128/128
    Epoch 00127: loss improved from 145.56877 to 145.56005, saving model to /data/vision/fisher/data1/vsmolyakov/vae/weights-checkpoint.hdf5
    9s - loss: 145.5601 - val_loss: 149.5803


Let's save the model and final weights, we can then load the saved model as follows:


```python
vae.save(DATA_PATH + 'final_model.h5', overwrite=True)
vae.save_weights(DATA_PATH + 'final_weights.h5', overwrite=True)

#load saved model
#vae = load_model(DATA_PATH + 'final_model.h5')
```


```python
encoder = Model(x, z_mean)

#generative model
decoder_input = Input(shape=(latent_dim,))
_h_decoded = decoder_h(decoder_input)
_x_decoded_mean = decoder_mean(_h_decoded)
generator = Model(decoder_input, _x_decoded_mean)
```

Let's visualize our training loss and the learning rate:


```python
#generate plots
plt.figure()
plt.plot(hist.history['loss'], lw=2.0, label='Adam')
plt.title('VAE model')
plt.xlabel('Epochs')
plt.ylabel('Training Loss')
plt.legend()
plt.savefig('./figures/vae_training_loss.png')

```


```python
plt.figure()
plt.plot(hist_lr.lr, lw=2.0, label='learning rate')
plt.title('VAE model')
plt.xlabel('Epochs')
plt.ylabel('Learning Rate')
plt.legend()
plt.savefig('./figures/vae_learning_rate.png')

```

We see a staircase pattern in the learning rate caused by a division by 2 of the initial learning rate every 16 epochs as desired.

Finally, we can scan the latent space to generate a manifold of images learned by the variational autoencoder. Since we know that our variational approximation $z \sim N(\mu, \sigma^2)$ is close to $N(0,1)$ due to the regularization loss, we can define a grid of dimension $h=2$ centered at zero and for each sample in that grid generate an image by feeding our samples through a decoder.


```python
n = 15 #number of samples
digit_size = 28
figure = np.zeros((digit_size * n, digit_size * n))
grid_x = norm.ppf(np.linspace(0.05, 0.95, n))
grid_y = norm.ppf(np.linspace(0.05, 0.95, n))

for i, yi in enumerate(grid_x):
    for j, xi in enumerate(grid_y):
        z_sample = np.array([[xi, yi]])
        x_decoded = generator.predict(z_sample)
        digit = x_decoded[0].reshape(digit_size, digit_size)
        figure[i * digit_size: (i+1) * digit_size,
               j * digit_size: (j+1) * digit_size] = digit

```


```python
plt.figure()
plt.imshow(figure)
plt.savefig('./figures/vae_manifold.png')
```

## References
[1] D. Kingma et al., "Auto-Encoding Variational Bayes", ICLR 2014  
[2] F. Chollet, "Keras Blog", https://blog.keras.io/

ae7de1efe37dfdd190f456f9749fbab18e62018d
ipynb
Jupyter Notebook
chp02/keras_vae.ipynb
gerket/experiments_with_python
5dd6dbd69deaaa318bfa7d2c3c9f7fae6220c460
MIT
2017-08-22T13:14:54.000Z
2022-03-28T17:56:59.000Z
chp02/keras_vae.ipynb
gerket/experiments_with_python
5dd6dbd69deaaa318bfa7d2c3c9f7fae6220c460
MIT
2017-07-31T00:52:36.000Z
2018-10-01T14:29:51.000Z
chp02/keras_vae.ipynb
gerket/experiments_with_python
5dd6dbd69deaaa318bfa7d2c3c9f7fae6220c460
MIT
2017-08-23T08:08:32.000Z
2022-03-09T07:04:01.000Z
Qwen/Qwen-72B
1. YES
2. YES