## Exercise 1 (4 points): 1D CNN [Pen and Paper]

In this assignment, we want to understand convolutional neural networks (CNNs) and we start by applying them to a one-dimensional input so that we get used to the basic concepts. An implementation applied to images follows in Exercise 2.

In the context of this exercise, the output 𝑦𝑘 of the 𝑘-th neuron in the 𝑗-th feature map of a convolutional layer is calculated as <br/>

\begin{equation}y_{k,j} = f \left(\sum_{i=1}^M \sum_{c=1}^C w_{i, c; j}⋅ x_{i+k-1, c} + b_j\right)\end{equation}

with the weights 𝑤𝑖,𝑐;𝑗, the input 𝑥𝑖,𝑐, the bias 𝑏𝑗 and the transfer function 𝑓 (𝑥). 𝑀 denotes the filter length and 𝐶 the number of channels this filter pools from. Note: Equation 1 does not include the concepts of padding and stride sizes used below (cf. TensorFlow references for further details about these parameters).

Suppose that our CNN is defined as follows:
    
    I An input of length 10 consisting of 𝐶 = 2 channels.
    
    II One convolutional layer with 3 feature maps, a filter length of 𝑀 = 5, a stride of 1 and the padding set to valid. As activation function, rectified linear units, i.e. 𝑓 (𝑥) = max(𝑥, 0), are used.
    
    III A max-pooling layer with a pool size of 3, a stride of 2 and the padding set to same.
    
    IV Two output neurons with a linear activation function (the output of the pooling layer is flattened before it is passed on to the output layer).
    
Let’s use this network to perform some basic computations.
1. The upper part of the input matrix 𝑋 is given as follows
\begin{pmatrix}
x_{1, 1} & 0\\
1 & -1\\
2 & 0\\
0 & 0\\
1 & 1\\
\vdots & \vdots
\end{pmatrix}
We now consider the first filter (𝑗 = 1) of the convolutional layer.
    
    a) What is the dimension of the weight matrix 𝑊 (filter size)?

    b) Suppose that all weights which pool from the first channel are set to $w_{i,1;1} = 1$, the weights for the second channel are set to $w_{i,2;1} = 2$ and the bias is set to $b_1 = 0$. The output of the first neuron is $y_{1;1} = 5$. Calculate the missing input value $x_{1,1}$.

2. What is the difference between a one-dimensional convolutional layer (temporal convolution) which pools from multiple channels (as implemented by Equation 1) compared to a two-dimensional CNN layer (e.g. with a filter size of 5 × 2)?

3. For each layer in our network, fill out Table 1 by deciding about

    • the dimensions of the layer’s output tensor (e.g. output matrix of the convolutional layer).

    • the number of weights per layer (if any).

    • the number of biases per layer (if any)

4. Suppose we used a standard hidden layer with 10 neurons instead of the convolutional and pooling layers (the output neurons are still used), how does the total number of weights and biases in the network change (the input is flattened before it is passed on to the hidden layer)?

## Exercise 2 (6 points): 2D CNN

The convolution operation is often used in computer vision domain and it is hence not surprising that CNNs are especially helpful in image processing tasks. We now want to apply a CNN to a small image classification task based on the STL-10 dataset [1]. More concretely, we are using two classes with 500 images each and train a network to distinguish between them.

Our main goal is to visualize how the network processes the images, i.e. highlight the activations of the layers. Therefore, we disregard other aspects like the validation against a test set. Further, we are also going to cover some new TensorFlow aspects, namely datasets and iterators2 as a novel way to feed data to the network.

Please use the file CNN2D.ipynb attached to this exercise as the basis for your implementation. For the test script, please pay also attention to Table 2.

1. Load the images and the corresponding labels from the attached stl10.pickle file

2. It would not be very practical to visualize the network activations for all images so we only use a few examples. Create a subset of the images (and labels) which contains these example images. Make sure that you use at least one instance per class.

3. We need to feed the images and the corresponding labels to the network and we want to use datasets and iterators for this purpose.

    a) Define a training dataset which iterates over all images. Shuffle the dataset (set a buffer size which equals the total number of training instances) and then apply the specified batch size.

    b) Define another dataset for the example images selected previously. Batch over all instances at once.

    c) Create a reinitializable iterator and define an initialization operation for each dataset.

    d) To access the data, extract tensors for the images and labels from the iterator (iter ator.get_next())

4. We now have all ingredients together to build our network. Use the defined parameters whenever it is suitable.

    a) Create a hierarchy of subsequent convolutional and pooling layer pairs (n_cnn_la yers in total). That is, first process the input via a (two-dimensional) convolutional layer and then apply an average (two-dimensional) polling layer. Store the CNN layers so that you can later easily calculate the network activations for the example images.

    b) Reshape the output of the last pooling layer to a one-dimensional vector. Note that you still need a two-dimensional tensor with the first dimension denoting the batches.

    c) Use a final (linear) layer which calculates the logits for each class. 
    
    d) Define a training operation based on the cross-entropy loss function (including regularization) and the Adam optimizer.

    e) Create a tensor which calculates the accuracy over the current batch

5. The next step is to train and evaluate the network. We also want to calculate (and store) the activation images whenever the state of the network changes. Note that the training can take a few minutes.

    a) Calculate the activation images for the initial state of the network (after initializing the variables).

    b) Train the network over multiple epochs. With each training step (one batch), calculate the loss and the accuracy and average all values together so that one loss and one accuracy value per epoch remains3. Keep track of these values.

    c) After the training phase of each epoch, calculate the activation images again.

6. Plot the loss and the accuracy values over the epochs.

7. Plot your selected images.

8. Create an animation (interact) for the activation images which lets you select the image, the epoch and the layer. For each configuration, the output of the four filters should be shown. Set explicit values for the data range when using the imshow() function (vmin and vmax) which correspond to the boundaries of the tanh(𝑥) function. An example can be seen in Figure 1.

b1f40228504451a49090693fc7740f704fcd4862
ipynb
Jupyter Notebook
12_CNN/CNN.ipynb
jhinga-la-la/pattern-recognition-course
7ad4f70b2c427f3c37f59f47768b90371873823c
Apache-2.0
12_CNN/CNN.ipynb
jhinga-la-la/pattern-recognition-course
7ad4f70b2c427f3c37f59f47768b90371873823c
Apache-2.0
12_CNN/CNN.ipynb
jhinga-la-la/pattern-recognition-course
7ad4f70b2c427f3c37f59f47768b90371873823c
Apache-2.0
Qwen/Qwen-72B
1. YES
2. YES