## Visualizing Convolutional Neural Networks and Neural Style Transfer

July 2019 <br>
**Author:** Matthew Stewart


```python
#RUN THIS CELL 
import requests
from IPython.core.display import HTML
styles = requests.get("https://raw.githubusercontent.com/Harvard-IACS/2019-CS109B/master/content/styles/cs109.css").text
HTML(styles)
```




<style>
blockquote { background: #AEDE94; }
h1 { 
    padding-top: 25px;
    padding-bottom: 25px;
    text-align: left; 
    padding-left: 10px;
    background-color: #DDDDDD; 
    color: black;
}
h2 { 
    padding-top: 10px;
    padding-bottom: 10px;
    text-align: left; 
    padding-left: 5px;
    background-color: #EEEEEE; 
    color: black;
}

div.exercise {
	background-color: #ffcccc;
	border-color: #E9967A; 	
	border-left: 5px solid #800080; 
	padding: 0.5em;
}
div.discussion {
	background-color: #ccffcc;
	border-color: #88E97A;
	border-left: 5px solid #0A8000; 
	padding: 0.5em;
}
div.theme {
	background-color: #DDDDDD;
	border-color: #E9967A; 	
	border-left: 5px solid #800080; 
	padding: 0.5em;
	font-size: 18pt;
}
div.gc { 
	background-color: #AEDE94;
	border-color: #E9967A; 	 
	border-left: 5px solid #800080; 
	padding: 0.5em;
	font-size: 12pt;
}
p.q1 { 
    padding-top: 5px;
    padding-bottom: 5px;
    text-align: left; 
    padding-left: 5px;
    background-color: #EEEEEE; 
    color: black;
}
header {
   padding-top: 35px;
    padding-bottom: 35px;
    text-align: left; 
    padding-left: 10px;
    background-color: #DDDDDD; 
    color: black;
}
</style>






```python
import time
import numpy as np

from keras import backend as K
from keras.applications import vgg16, vgg19
from keras.preprocessing.image import load_img

from scipy.misc import imsave
from scipy.optimize import fmin_l_bfgs_b

# preprocessing
from utils import preprocess_image, deprocess_image

%matplotlib inline
```

### Part 1: Content loss

We can generate an image that combines the content and style of a pair with a loss function that incorporates this information. This is achieved with two terms, one that mimics the specific ativations of a certain layer for the content image, and a second term that mimics the style. The variable to optimize in the loss function will be a generated image that aims to minimize the proposed cost. Note that to optimize this function, we will perform gradient descent __on the pixel values__, rather than on the neural network weights.

We will load a trained neural network called VGG-16 proposed in [1](https://arxiv.org/pdf/1409.1556.pdf), who secured the first and second place in the localisation and classification tracks of ImageNet Challenge in 2014, respectively. This network has been trained to discriminate over 1000 classes over more than a million images. We will use the activation values obtained for an image of interest to represent the content and styles. In order to do so, we will feed-forward the image of interest and observe it's activation values at the indicated layer.

The content loss function measures how much the feature map of the generated image differs from the feature map of the source image. We will only consider a single layer to represent the contents of an image. The authors of this technique indicated they obtained better results when doing so. We denote the feature maps for layer $l$ with $a^{[l]} \in \mathbb{R}^{n_H^{[l]} \times n_W^{[l]} \times n_C^{[l]}}$. Parameter $n_C^{[l]}$ is the number of filters/channels in layer $l$, $n_H^{[l]}$ and $n_W^{[l]}$ are the height and width.

The content loss is then given by:
\begin{equation}
    J^{[l]}_C = \big\Vert a^{[l](G)} - a^{[l](C)} \big\Vert^2_{\mathcal{F}},
\end{equation}
where $a^{[l](G)}$ refers to the layer's activation values of the generated image, and $a^{[l](C)}$ to those of the content image.

<div class="exercise"> <b> Part 1: Content loss</b> </div>
    
Implement funtion `feature_reconstruction_loss` that computes the loss of two feature inputs. You will need to use [keras backend functions](https://keras.io/backend/#backend-functions) to complete the exercise.


```python
def feature_reconstruction_loss(base, output):
    """
    Compute the content loss for style transfer.
    
    Inputs:
    - output: features of the generated image, Tensor with shape [height, width, channels]
    - base: features of the content image, Tensor with shape [height, width, channels]
    
    Returns:
    - scalar content loss
    """
    # YOUR CODE GOES HERE    
    return K.sum(K.square(output - base))    
```

Test your implementation:


```python
np.random.seed(1)
base = np.random.randn(10,10,3)
output = np.random.randn(10,10,3)
a = K.constant(base)
b = K.constant(output)
test = feature_reconstruction_loss(a, b)
print('Result:          ', K.eval(test))
print('Expected result: ', 605.62195)
```

### Part 2: Style loss

The style measures the similarity among filters in a set of layers. In order to compute that similarity, we will compute the Gram matrix of the activation values for the style layers, i.e., $a^{[l]}$ for some set $\mathcal{L}$. The Gram matrix is related to the empirical covariance matrix, and therefore, reflects the statistics of the activation values.

Given a feature map $a^{[l]}$ of shape $(n_H^{[l]}, n_W^{[l]}, n_C^{[l]})$, the Gram matrix has shape $(n_C^{[l]}, n_C^{[l]})$ and its elements are given by:
\begin{equation*}
    G^{[l]}_{k k'} = \sum_{i=1}^{n_H^{[l]}} \sum_{j=1}^{n_W^{[l]}} a^{[l]}_{ijk} a^{[l]}_{ijk'}.
\end{equation*}
The output is a 2-D matrix which approximately measures the cross-correlation among different filters for a given layer. This in essence constitutes the style of a layer.

<div class="exercise"> <b> Part 2: Computing the Gram matrix</b> </div>

We implement a function that computes the Gram matrix of a given keras tensor. This can be accomplished efficiently if $x$ is reshaped as a tensor of shape ($n_C^{[l]} \times n_H^{[l]} n_W^{[l]}$) and then you compute the outer product of this matrix with itself. We need to use [keras backend functions](https://keras.io/backend/#backend-functions) for this.


```python
def gram_matrix(x):
    """
    Computes the outer-product of the input tensor x.

    Input:
    - x: input tensor of shape (H, W, C)

    Returns:
    - tensor of shape (C, C) corresponding to the Gram matrix of
    the input image.
    """
    # YOUR CODE GOES HERE   
    features = K.batch_flatten(K.permute_dimensions(x, (2, 0, 1)))
    return K.dot(features, K.transpose(features))
```

Test your implementation:


```python
np.random.seed(1)
x_np = np.random.randn(10,10,3)
x = K.constant(x_np)
test = gram_matrix(x)
print('Result:\n', K.eval(test))
print('Expected:\n', np.array([[99.75723, -9.96186, -1.4740534], [-9.96186, 86.854324, -4.141108 ], [-1.4740534, -4.141108, 82.30106  ]]))
```

### Part 3: Style loss: layer's loss

Now we can tackle the style loss. For a given layer $l$, the style loss is defined as follows:
\begin{equation*}
    J^{[l]}_S = \frac{1}{4 (n^{[l]}_W n^{[l]}_H)^2} \Big\Vert G^{[l](S)} - G^{[l](G)}\Big\Vert^2_{\mathcal{F}}.
\end{equation*}

In practice we compute the style loss at a set of layers $\mathcal{L}$ rather than just a single layer $l$; then the total style loss is the sum of style losses at each layer:

$$J_S = \sum_{l \in \mathcal{L}} \lambda_l J^{[l]}_S$$
where $\lambda_l$ corresponds to a weighting parameter.

<div class="exercise"> <b> Part 3: Computing the layer's loss</b> </div>

Implement `style_reconstruction_loss` that computes the loss for a given layer $l$. We again need to use [keras backend functions](https://keras.io/backend/#backend-functions) for this.


```python
def style_reconstruction_loss(base, output):
    """
    Computes the style reconstruction loss. It encourages the output img 
    to have same stylistic features as style image.
    
    Inputs:
    - base: features at given layer of the style image.
    - output: features of the same length as base of the generated image.
      
    Returns:
    - style_loss: scalar style loss
    """
    # YOUR CODE GOES HERE 
    H, W = int(base.shape[0]), int(base.shape[1])
    gram_base = gram_matrix(base)
    gram_output = gram_matrix(output)
    factor = 1.0 / float((2*H*W)**2)
    out = factor * K.sum(K.square(gram_output - gram_base))
    return out
```

Test your implementation:


```python
np.random.seed(1)
x = np.random.randn(10,10,3)
y = np.random.randn(10,10,3)
a = K.constant(x)
b = K.constant(y)
test = style_reconstruction_loss(a, b)
print('Result:  ', K.eval(test))
print('Expected:', 0.09799164)
```

### Part 4: Total-variation regularization
We will also encourage smoothness in the image using a total-variation regularizer. This penalty term will reduce variation among the neighboring pixel values.

The following expression constitues the regularization penalty over all pairs that are next to each other horizontally or vertically. The expression is independent among different RGB channels.
\begin{equation*}
    J_{tv} = \sum_{c=1}^3\sum_{i=1}^{n^{[l]}_H-1} \sum_{j=1}^{n^{[l]}_W-1} \left( (x_{i,j+1, c} - x_{i,j,c})^2 + (x_{i+1, j,c} - x_{i,j,c})^2  \right)
\end{equation*}

<div class="exercise"> <b> Part 4: Total-variation regularization</b> </div>

In the next cell, fill in the definition for the TV loss term.

__Remark:__ $x$ has dimension $(1, n_H^{[l]}, n_W^{[l]}, n_C^{[l]})$, which is different from the 3D-tensors we used before.


```python
def total_variation_loss(x):
    """
    Total variational loss. Encourages spatial smoothness 
    in the output image.
    
    Inputs:
    - x: image with pixels, has shape 1 x H x W x C.
      
    Returns:
    - total variation loss, a scalar number.
    """
    # YOUR CODE GOES HERE 
    a = K.square(x[:, :-1, :-1, :] - x[:, 1:, :-1, :])
    b = K.square(x[:, :-1, :-1, :] - x[:, :-1, 1:, :])
    return K.sum(a + b)
```

Test your implementation. If you do not get exact results but similar, you may still have a correct implementation. The goal is that you write a smoother for neighboring pixels.


```python
np.random.seed(1)
x_np = np.random.randn(1,10,10,3)
x = K.constant(x_np)
test = total_variation_loss(x)
print('Result:  ', K.eval(test))
print('Expected:', 937.0538)
```

### Part 5: Style transfer

We now put it all together and generate some images! The `style_transfer` function below combines all the losses you coded up above and optimizes for an image that minimizes the total loss. Read the code and comments to understand the procedure.


```python
def style_transfer(base_img_path, style_img_path, output_img_path, convnet='vgg16', 
        content_weight=3e-2, style_weights=(20000, 500, 12, 1, 1), tv_weight=5e-2, content_layer='block4_conv2', 
        style_layers=['block1_conv1', 'block2_conv1', 'block3_conv1', 'block4_conv1', 'block5_conv1'], iterations=50):
    
    print('\nInitializing Neural Style model...')

    # Determine the image sizes. Fix the output size from the content image.
    print('\n\tResizing images...')
    width, height = load_img(base_img_path).size
    new_dims = (height, width)

    # Preprocess content and style images. Resizes the style image if needed.
    content_img = K.variable(preprocess_image(base_img_path, new_dims))
    style_img = K.variable(preprocess_image(style_img_path, new_dims))

    # Create an output placeholder with desired shape.
    # It will correspond to the generated image after minimizing the loss function.
    output_img = K.placeholder((1, height, width, 3))
    
    # Sanity check on dimensions
    print("\tSize of content image is: {}".format(K.int_shape(content_img)))
    print("\tSize of style image is: {}".format(K.int_shape(style_img)))
    print("\tSize of output image is: {}".format(K.int_shape(output_img)))

    # Combine the 3 images into a single Keras tensor, for ease of manipulation
    # The first dimension of a tensor identifies the example/input.
    input_img = K.concatenate([content_img, style_img, output_img], axis=0)

    # Initialize the vgg16 model
    print('\tLoading {} model'.format(convnet.upper()))

    if convnet == 'vgg16':
        model = vgg16.VGG16(input_tensor=input_img, weights='imagenet', include_top=False)
    else:
        model = vgg19.VGG19(input_tensor=input_img, weights='imagenet', include_top=False)
        
    print('\tComputing losses...')
    # Get the symbolic outputs of each "key" layer (they have unique names).
    # The dictionary outputs an evaluation when the model is fed an input.
    outputs_dict = dict([(layer.name, layer.output) for layer in model.layers])

    # Extract features from the content layer
    content_features = outputs_dict[content_layer]

    # Extract the activations of the base image and the output image
    base_image_features = content_features[0, :, :, :]  # 0 corresponds to base
    combination_features = content_features[2, :, :, :] # 2 coresponds to output

    # Calculate the feature reconstruction loss
    content_loss = content_weight * feature_reconstruction_loss(base_image_features, combination_features)

    # For each style layer compute style loss
    # The total style loss is the weighted sum of those losses
    temp_style_loss = K.variable(0.0)       # we update this variable in the loop
    weight = 1.0 / float(len(style_layers))
    
    for i, layer in enumerate(style_layers):
        # extract features of given layer
        style_features = outputs_dict[layer]
        # from those features, extract style and output activations
        style_image_features = style_features[1, :, :, :]   # 1 corresponds to style image
        output_style_features = style_features[2, :, :, :]  # 2 coresponds to generated image
        temp_style_loss += style_weights[i] * weight * \
                    style_reconstruction_loss(style_image_features, output_style_features)
    style_loss = temp_style_loss

    # Compute total variational loss.
    tv_loss = tv_weight * total_variation_loss(output_img)

    # Composite loss
    total_loss = content_loss + style_loss + tv_loss
    
    # Compute gradients of output img with respect to total_loss
    print('\tComputing gradients...')
    grads = K.gradients(total_loss, output_img)
    
    outputs = [total_loss] + grads
    loss_and_grads = K.function([output_img], outputs)  
    
    # Initialize the generated image from random noise
    x = np.random.uniform(0, 255, (1, height, width, 3)) - 128.
    
    # Loss function that takes a vectorized input image, for the solver
    def loss(x):
        x = x.reshape((1, height, width, 3))   # reshape
        return loss_and_grads([x])[0]
    
    # Gradient function that takes a vectorized input image, for the solver
    def grads(x):
        x = x.reshape((1, height, width, 3))   # reshape
        return loss_and_grads([x])[1].flatten().astype('float64')
    
    # Fit over the total iterations
    for i in range(iterations+1):
        print('\n\tIteration: {}'.format(i+1))

        toc = time.time()
        x, min_val, info = fmin_l_bfgs_b(loss, x.flatten(), fprime=grads, maxfun=20)

        # save current generated image
        if i%10 == 0:
            img = deprocess_image(x.copy(), height, width)
            fname = output_img_path + '_at_iteration_%d.png' % (i)
            imsave(fname, img)
            print('\t\tImage saved as', fname)

        tic = time.time()

        print('\t\tLoss: {:.2e}, Time: {} seconds'.format(float(min_val), float(tic-toc)))
```

<div class="exercise"> <b> Part 5: Generate pictures!</b> </div>

Find style and content images under `images/inputs/`.

* The `base_img_path` is the filename of content image.
* The `style_img_path` is the filename of style image.
* The `output_img_path` is the filename of generated image.
* The `convnet` is for the neural network weights, VGG-16 or VGG-19.
* The `content_layer` specifies which layer to use for content loss.
* The `content_weight` weights the content loss in the overall composite loss function. Increasing the value of this parameter will make the final image look more realistic (closer to the original content).
* `style_layers` specifies a list of which layers to use for the style loss. 
* `style_weights` specifies a list of weights to use for each layer in style_layers (each of which will contribute a term to the overall style loss). We generally use higher weights for the earlier style layers because they describe more local/smaller scale features, which are more important to texture than features over larger receptive fields. In general, increasing these weights will make the resulting image look less like the original content and more distorted towards the appearance of the style image.
* `tv_weight` specifies the weighting of total variation regularization in the overall loss function. Increasing this value makes the resulting image look smoother and less jagged, at the cost of lower fidelity to style and content. 

**CAUTION:** The script saves an image every 10 iterations.

### Great wave of Kanagawa + Chicago


```python
params = {
'base_img_path' : 'images/inputs/chicago.jpg', 
'style_img_path' : 'images/inputs/great_wave_of_kanagawa.jpg', 
'output_img_path' : 'images/results/wave_chicago', 
'convnet' : 'vgg16', 
'content_weight' : 500, 
'style_weights' : (10, 10, 50, 10, 10),
'tv_weight' : 200, 
'content_layer' : 'block4_conv2', 
'style_layers' : ['block1_conv1',
                  'block2_conv1',
                  'block3_conv1', 
                  'block4_conv1', 
                  'block5_conv1'], 
'iterations' : 50
}

style_transfer(**params)
```

### Starry night + Tübingen


```python
params = {
'base_img_path' : 'images/inputs/tubingen.jpg', 
'style_img_path' : 'images/inputs/starry_night.jpg', 
'output_img_path' : 'images/results/starry_tubingen', 
'convnet' : 'vgg16', 
'content_weight' : 100, 
'style_weights' : (1000, 100, 12, 1, 1),
'tv_weight' : 200, 
'content_layer' : 'block4_conv2', 
'style_layers' : ['block1_conv1',
                  'block2_conv1',
                  'block3_conv1', 
                  'block4_conv1', 
                  'block5_conv1'], 
'iterations' : 50
}

style_transfer(**params)
```

### Acknowledgments

- The implementation uses code from Francois Chollet's neural style transfer.
- The implementation uses code from Kevin Zakka's neural style transfer, under MIT license.
- The hierarchy borrows from Giuseppe Bonaccorso's gist, under MIT license.

d125c43f026017917a82b2a7f83e1a2fdf1bac58
ipynb
Jupyter Notebook
Neural-Style-Transfer/Neural-Style-Transfer.ipynb
victorwu89/Neural-Networks
6de5378701e5f8bac3be92ebf41ce778162a3d34
MIT
2019-06-18T07:32:23.000Z
2022-01-18T07:53:08.000Z
Neural-Style-Transfer/Neural-Style-Transfer.ipynb
victorwu89/Neural-Networks
6de5378701e5f8bac3be92ebf41ce778162a3d34
MIT
Neural-Style-Transfer/Neural-Style-Transfer.ipynb
victorwu89/Neural-Networks
6de5378701e5f8bac3be92ebf41ce778162a3d34
MIT
2019-06-18T13:33:44.000Z
2022-03-15T13:16:10.000Z
Qwen/Qwen-72B
1. YES
2. YES