Histograms of Oriented Gradients (HOG)

Introduction

正如我们在ORB算法中看到的那样，我们可以使用图像中的关键点来进行基于关键点的匹配，以此来检测图像中的对象。如果你想检测具有许多不受背景影响的一致内部特征的对象，这些类型的算法就会非常有用。比如，这些算法适用于人脸检测，因为人脸具有许多不受图像背景影响的一致内部特征，例如眼睛、鼻子和嘴巴。然而，如果要尝试进行更一般的对象识别（例如，图像中的行人检测），这些类型的算法就不会那么有用了。原因是行人与人脸不同，没有与人脸一样的内部特征，毕竟每个行人的体形和走路风格都不同（见图1）。这意味着每个行人都有不同的内部特征，因此我们需要一些可以用于更一般地描述行人的方法。

Fig. 1. - Pedestrians.

其中，有一种方法是尝试通过轮廓来检测行人。通过轮廓（边界）可以检测到图像中的对象，但这种方法非常具有难度，因为我们必须处理背景和前景之间的对比，而完成这一点是相当困难的。例如，假设你想要检测在白色建筑物前行走的图像中的行人，并且她穿着白色外套和黑色裤子（见图2）。我们可以在图2中看到，由于图像的背景大多是白色的，因此黑色裤子具有非常高的对比度，但是白色外套具有非常低的对比度。在这种情况下，非常容易就可以检测到裤子的边缘，但要检测外套的边缘，就会非常困难。在这个时候，就需要使用**HOG** 了。HOG代表**方向梯度直方图** ，它是由Navneet Dalal和Bill Triggs于2005年首次提出的。

Fig. 2. - High and Low Contrast.

HOG算法的工作原理是通过创建图像中梯度方向分布的直方图，然后以非常特殊的方式对它们进行归一化。即使在对比度非常低的情况下，这种特殊的归一化也使得HOG在检测物体边缘方面非常有效。这些归一化的直方图会被放在一个特征向量中，称为HOG描述子。HOG可用于训练机器学习算法，如支持向量机（SVM），使其根据它们的边界（边缘）检测图像中的对象 。由于其巨大的成功和可靠性，HOG已成为用于对象检测的计算机视觉中最广泛使用的算法之一。

在这个notebook中，你将要学习的内容如下：

* HOG算法的工作原理
* 如何使用OpenCV创建HOG描述子
* 如何可视化HOG描述子。

# HOG算法

顾名思义，HOG算法是一种基于从图像梯度的方向创建直方图的算法。 HOG算法是通过一系列步骤实现的，具体如下：

1. 对于某个特定对象的图像，设置一个可以覆盖图像中整个对象的检测窗口或者你感兴趣的区域（参见图3）。

2. 计算检测窗口中每个像素的梯度的大小和方向。

3. 将检测窗口划分为连接的像素*单元格* ，其中，所有单元格的大小相同（参见图3）。该单元格的大小是一个自由参数，通常选择它的目的在于匹配想要检测的特征的范围。例如，在64×128像素检测窗口中，6至8像素宽的正方形单元格适合于检测人体肢体。

4. 为每个单元格创建直方图，首先将每个单元格中所有像素的渐变方向分组为特定数量的方向（角度）区间（bin）；然后将每个角度区间中梯度的梯度大小相加（见图3）。直方图中的区间数是一个自由参数，通常设置为9个角度区间。

5. 将相邻的单元格分组成*块*（见图3）。每个块中的单元格数是一个自由参数，所有块必须具有相同的大小。每个块之间的距离（称为步长）是一个自由参数，但通常设置为块大小的一半，在这种情况下，你将会获得重叠块（*请查看下面的视频*）。这样，HOG算法会根据经验更好地与重叠块一起运行。

6. 使用每个块中包含的单元格来归一化该块中的单元格直方图（参见图3）。如果你有重叠块，这说明大多数单元格将根据不同的块进行归一化（*请查看下面的视频*）。因此，相同的单元格可以具有几种不同的归一化。

7. 将所有块中的所有归一化直方图收集到称为HOG描述子的单个特征向量中。

8. 使用来自相同类型对象的大量图像的结果HOG描述子来训练机器学习算法（例如SVM），进而检测图像中那些类型的对象。 例如，你可以使用来自行人的大量图像的HOG描述子来训练SVM，进而检测图像中的行人。其中，训练是通过你想要在图像中检测到的对象的正反面示例完成的。

9. 训练完SVM之后，你需要使用滑动窗口方法来尝试检测和定位图像中的对象。 要想检测图像中的对象，你需要找到看起来类似于SVM所学习的HOG类型的图像部分。

Fig. 3. - HOG Diagram.

Vid. 1. - HOG Animation.

# 为什么HOG算法如此有效

如上所述，HOG通过在称为*单元格*的图像的局部部分中添加特定方向的梯度的大小来创建直方图。这样做，我们保证了更强的梯度会对其各自的角度区间的大小产生更多影响，同时由噪声导致的较弱与随机定向梯度的影响被最小化。总之，直方图会以这种方式告诉我们每个单元格的主导梯度方向。

### 处理对比

现在，由于局部照明的变化以及背景和前景之间的对比度，主导方向的梯度差异会非常大。

为了弄清楚背景与前景对比度差异，HOG算法尝试在本地检测边缘。为了做到这一点，它定义了一组称为**块**的单元格，并使用这个本地单元格组对该直方图进行归一化。通过局部归一化，HOG算法可以非常可靠地检测每个块中的边缘。这就叫做**块归一化**。

除了使用块归一化之外，HOG算法还使用重叠块来提高其性能。通过使用重叠块，每个单元格向最终HOG描述子贡献若干独立分量，其中每个分量对应于相对于不同块归一化的单元格。这似乎是多余的，但是，经验证明，通过相对于不同的本地块对每个单元格进行几次归一化，HOG算法的性能会显著提高。

### 加载图像并导入资源

要创建HOG描述子，第一步是将所需的包加载到Python中并加载图像。

我们首先使用OpenCV加载三角形图块的图像。因为，`cv2.imread()`函数会将图像加载为BGR，而我们需要将图像转换为RGB，这样我们就可以使用正确的颜色显示它。与之前一样，我们会将BGR图像转换为灰度图像进行分析。

```python
import cv2
import numpy as np
import matplotlib.pyplot as plt

# Set the default figure size
plt.rcParams['figure.figsize'] = [17.0, 7.0]

# Load the image
image = cv2.imread('./images/triangle_tile.jpeg')

# Convert the original image to RGB
original_image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

# Convert the original image to gray scale
gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

# Print the shape of the original and gray scale images
print('The original image has shape: ', original_image.shape)
print('The gray scale image has shape: ', gray_image.shape)

# Display the images
plt.subplot(121)
plt.imshow(original_image)
plt.title('Original Image')
plt.subplot(122)
plt.imshow(gray_image, cmap='gray')
plt.title('Gray Scale Image')
plt.show()
```

    The original image has shape:  (250, 250, 3)
    The gray scale image has shape:  (250, 250)

    <matplotlib.figure.Figure at 0x7f4b49d45e80>

# 创建HOG描述子

我们将使用OpenCV的`HOGDescriptor`类来创建HOG描述子， 使用`HOGDescriptor()`函数设置HOG描述子的参数。`HOGDescriptor()`函数的参数及其默认值如下：

`cv2.HOGDescriptor(win_size = (64, 128),
                  block_size = (16, 16),
                  block_stride = (8, 8),
                  cell_size = (8, 8),
                  nbins = 9,
                  win_sigma = DEFAULT_WIN_SIGMA,
                  threshold_L2hys = 0.2,
                  gamma_correction = true,
                  nlevels = DEFAULT_NLEVELS)`

参数：

* **win_size** – *Size*
  检测窗口的大小（以像素为单位）（*高度，宽度*）。定义一个感兴趣的区域。它必须是单元格尺寸的整数倍。

* **block_size** – *Size*
  块大小（以像素为单位）（*宽度，高度*）。定义每个块中有多少个单元格。它必须是单元格尺寸的整数倍，并且必须小于检测窗口。块越小，你将获得的细节越详细。

* **block_stride** – *Size*
  以像素为单位块步长（*水平，垂直*）。它必须是单元格尺寸的整数倍。 `block_stride`定义了相邻块之间的距离，例如，水平8个像素和垂直8个像素。`block_stride`越长，算法就会运行得越快（因为评估的块会越少），但算法也可能不再运行。

* **cell_size** – *Size*
  单元格大小，以像素为单位（*宽度，高度*）。确定格单元的大小。单元格越小，你将获得的细节越详细。

* **nbins** – *int*
  直方图的区间数。确定用于制作直方图的角度区间数量。使用的区间越多，就可以获取更多的渐变方向。 HOG使用无符号渐变，因此角度区间的值将介于0和180度之间。

* **win_sigma** – *double*
  高斯平滑窗口参数。通过在计算直方图之前对每个像素应用高斯空间窗口来平滑块边缘附近的像素，从而改善HOG算法的性能。

* **threshold_L2hys** – *double*
   L2-Hys（Lowe式限幅L2范数）归一化方法收缩率。 L2-Hys方法用于归一化块，它由L2范数和剪切以及重归一化组成。限幅会将每个块的描述子向量的最大值限制为具有给定阈值的值（默认为0.2）。限幅之后，如*IJCV*，60（2）：91-110,2004中所述，重新归一化描述子矢量。

* **gamma_correction** – *bool*
  用于指定是否需要伽马校正预处理的标志。执行伽马校正可以在一定程度上提高HOG算法的性能。

* **nlevels** – *int*
  检测窗口增加的最大数量。

我们可以看到，`cv2.HOGDescriptor()`函数支持各种参数。前几个参数（`block_size, block_stride, cell_size`以及`nbins`）是最有可能更改的参数。其他参数都可以放心地保留其默认值，这样你会获得很不错的结果。

在下面的代码中，我们将使用 `cv2.HOGDescriptor()`函数来设置单元格大小、块大小、块步长以及HOG描述子的直方图区间数。然后，我们将使用`.compute(image)`方法计算给定`image`的HOG描述子（特征向量）。

```python
# Specify the parameters for our HOG descriptor

# Cell Size in pixels (width, height). Must be smaller than the size of the detection window
# and must be chosen so that the resulting Block Size is smaller than the detection window.
cell_size = (6, 6)

# Number of cells per block in each direction (x, y). Must be chosen so that the resulting
# Block Size is smaller than the detection window
num_cells_per_block = (2, 2)

# Block Size in pixels (width, height). Must be an integer multiple of Cell Size.
# The Block Size must be smaller than the detection window
block_size = (num_cells_per_block[0] * cell_size[0],
              num_cells_per_block[1] * cell_size[1])

# Calculate the number of cells that fit in our image in the x and y directions
x_cells = gray_image.shape[1] // cell_size[0]
y_cells = gray_image.shape[0] // cell_size[1]

# Horizontal distance between blocks in units of Cell Size. Must be an integer and it must
# be set such that (x_cells - num_cells_per_block[0]) / h_stride = integer.
h_stride = 1

# Vertical distance between blocks in units of Cell Size. Must be an integer and it must
# be set such that (y_cells - num_cells_per_block[1]) / v_stride = integer.
v_stride = 1

# Block Stride in pixels (horizantal, vertical). Must be an integer multiple of Cell Size
block_stride = (cell_size[0] * h_stride, cell_size[1] * v_stride)

# Number of gradient orientation bins
num_bins = 9

# Specify the size of the detection window (Region of Interest) in pixels (width, height).
# It must be an integer multiple of Cell Size and it must cover the entire image. Because
# the detection window must be an integer multiple of cell size, depending on the size of
# your cells, the resulting detection window might be slightly smaller than the image.
# This is perfectly ok.
win_size = (x_cells * cell_size[0] , y_cells * cell_size[1])

# Print the shape of the gray scale image for reference
print('\nThe gray scale image has shape: ', gray_image.shape)
print()

# Print the parameters of our HOG descriptor
print('HOG Descriptor Parameters:\n')
print('Window Size:', win_size)
print('Cell Size:', cell_size)
print('Block Size:', block_size)
print('Block Stride:', block_stride)
print('Number of Bins:', num_bins)
print()

# Set the parameters of the HOG descriptor using the variables defined above
hog = cv2.HOGDescriptor(win_size, block_size, block_stride, cell_size, num_bins)

# Compute the HOG Descriptor for the gray scale image
hog_descriptor = hog.compute(gray_image)
```

#  HOG描述子中的元素数量

由此得到的HOG描述子（特征向量）包含来自在一个长向量中连接的检测窗口中的所有块的所有单元格的归一化直方图。因此，HOG特征向量的大小将由检测窗口中的块总数乘以每个块的单元格数乘以定向区间的数量来得出，公式如下：

\begin{equation}
\mbox{total_elements} = (\mbox{total_number_of_blocks})\mbox{ } \times \mbox{ } (\mbox{number_cells_per_block})\mbox{ } \times \mbox{ } (\mbox{number_of_bins})
\end{equation}

如果没有重叠块（*即* ` block_stride`等于`block_size`），则可以通过将检测窗口的大小除以块大小来计算块的总数。但是，在一般情况下，必须考虑到我们有重叠块的事实。要想知道一般情况下的块总数（*即* 对于任何`block_stride`和`block_size`），可以使用下面给出的公式：

\begin{equation}
\mbox{Total}_i = \left( \frac{\mbox{block_size}_i}{\mbox{block_stride}_i} \right)\left( \frac{\mbox{window_size}_i}{\mbox{block_size}_i} \right) - \left [\left( \frac{\mbox{block_size}_i}{\mbox{block_stride}_i} \right) - 1 \right]; \mbox{  for  } i = x,y
\end{equation}

其中， Total$_x$ ，是沿检测窗口宽度的块总数， Total$_y$，是沿着检测窗口高度的块总数。这个  Total$_x$ 和 Total$_y$公式考虑了重叠产生的额外块。在计算 Total$_x$ 和 Total$_y$之后，我们可以通过 Total$_x$ 乘以 Total$_y$来获取检测窗口中的块总数。其实，上述公式可以进行相当大的简化，因为`block_size`，`block_stride`和`window_size`都是根据`cell_size`定义的。通过进行适当的替换和删除，上述公式简化为：

\begin{equation}
\mbox{Total}_i = \left(\frac{\mbox{cells}_i - \mbox{num_cells_per_block}_i}{N_i}\right) + 1\mbox{  }; \mbox{  for  } i = x,y
\end{equation}

其中， cells$_x$是沿检测窗口宽度的单元格总数， cells$_y$，是沿检测窗口高度的单元格总数。  $N_x$是以`cell_size`为单位的水平块步长，$N_y$是以cell_size`cell_size`为单位的垂直块步长。

接下来，让我们计算一下HOG特征向量的元素数量应该是多少，并检查它是否与上面计算的HOG描述子的形状相匹配。

```python
# Calculate the total number of blocks along the width of the detection window
tot_bx = np.uint32(((x_cells - num_cells_per_block[0]) / h_stride) + 1)

# Calculate the total number of blocks along the height of the detection window
tot_by = np.uint32(((y_cells - num_cells_per_block[1]) / v_stride) + 1)

# Calculate the total number of elements in the feature vector
tot_els = (tot_bx) * (tot_by) * num_cells_per_block[0] * num_cells_per_block[1] * num_bins

# Print the total number of elements the HOG feature vector should have
print('\nThe total number of elements in the HOG Feature Vector should be: ',
      tot_bx, 'x',
      tot_by, 'x',
      num_cells_per_block[0], 'x',
      num_cells_per_block[1], 'x',
      num_bins, '=',
      tot_els)

# Print the shape of the HOG Descriptor to see that it matches the above
print('\nThe HOG Descriptor has shape:', hog_descriptor.shape)
print()
```

# 可视化HOG描述子

我们可以通过将与每个单元格相关联的直方图绘制为矢量集合来可视化HOG描述子。为此，我们将直方图中的每个区间绘制为单个向量，其大小由区间的高度而定，其方向由与其关联的角度区间而定。由于任何给定的单元格，可能有多个与之关联的直方图。由于重叠块的存在，我们将选择平均分配每个单元格的所有直方图，这样便于为每个单元格生成单个直方图。

要可视化HOG描述子，OpenCV没有简单的方法可用，因此我们必须先进行一些操作才能使其可视化。这要从改变HOG描述子开始，这样可以使我们的计算更容易。然后，我们将计算每个单元格的平均直方图，最后我们将直方图区间转换为矢量。只要有了矢量，就可以为图像中的每个单元格绘制相应的矢量图。

下面的代码会生成一个交互式图，你可以与之进行交互。该图包含：
* 灰度图像，
* HOG描述子（特征向量），
* HOG描述子的放大部分，
* 所选单元格的直方图。

**你可以通过灰度图像或HOG描述子图像上的任意位置来选择特定单元格。** 单击任一图像后，将出现一个*洋红色*矩形，它会显示你选择的单元。缩放窗口将显示所选单元格周围的HOG描述子的一个放大版本，而直方图将显示所选单元格的相应直方图。交互式窗口底部还有按钮，你可以选择其他功能，例如平移，并根据需要提供了保存图形的选项。点击Home按钮，会将图形返回到其默认值。

**注意**: 如果你在优达学城工作区中运行此notebook，则交互式图大约会有2秒的延迟。这就是说，如果单击图像进行放大，则交互图的刷新将需要大约2秒钟。

```python
%matplotlib notebook

import copy
import matplotlib.patches as patches

# Set the default figure size
plt.rcParams['figure.figsize'] = [9.8, 9]

# Reshape the feature vector to [blocks_y, blocks_x, num_cells_per_block_x, num_cells_per_block_y, num_bins].
# The blocks_x and blocks_y will be transposed so that the first index (blocks_y) referes to the row number
# and the second index to the column number. This will be useful later when we plot the feature vector, so
# that the feature vector indexing matches the image indexing.
hog_descriptor_reshaped = hog_descriptor.reshape(tot_bx,
                                                 tot_by,
                                                 num_cells_per_block[0],
                                                 num_cells_per_block[1],
                                                 num_bins).transpose((1, 0, 2, 3, 4))

# Print the shape of the feature vector for reference
print('The feature vector has shape:', hog_descriptor.shape)

# Print the reshaped feature vector
print('The reshaped feature vector has shape:', hog_descriptor_reshaped.shape)

# Create an array that will hold the average gradients for each cell
ave_grad = np.zeros((y_cells, x_cells, num_bins))

# Print the shape of the ave_grad array for reference
print('The average gradient array has shape: ', ave_grad.shape)

# Create an array that will count the number of histograms per cell
hist_counter = np.zeros((y_cells, x_cells, 1))

# Add up all the histograms for each cell and count the number of histograms per cell
for i in range (num_cells_per_block[0]):
    for j in range(num_cells_per_block[1]):
        ave_grad[i:tot_by + i,
                 j:tot_bx + j] += hog_descriptor_reshaped[:, :, i, j, :]

        hist_counter[i:tot_by + i,
                     j:tot_bx + j] += 1

# Calculate the average gradient for each cell
ave_grad /= hist_counter

# Calculate the total number of vectors we have in all the cells.
len_vecs = ave_grad.shape[0] * ave_grad.shape[1] * ave_grad.shape[2]

# Create an array that has num_bins equally spaced between 0 and 180 degress in radians.
deg = np.linspace(0, np.pi, num_bins, endpoint = False)

# Each cell will have a histogram with num_bins. For each cell, plot each bin as a vector (with its magnitude
# equal to the height of the bin in the histogram, and its angle corresponding to the bin in the histogram).
# To do this, create rank 1 arrays that will hold the (x,y)-coordinate of all the vectors in all the cells in the
# image. Also, create the rank 1 arrays that will hold all the (U,V)-components of all the vectors in all the
# cells in the image. Create the arrays that will hold all the vector positons and components.
U = np.zeros((len_vecs))
V = np.zeros((len_vecs))
X = np.zeros((len_vecs))
Y = np.zeros((len_vecs))

# Set the counter to zero
counter = 0

# Use the cosine and sine functions to calculate the vector components (U,V) from their maginitudes. Remember the
# cosine and sine functions take angles in radians. Calculate the vector positions and magnitudes from the
# average gradient array
for i in range(ave_grad.shape[0]):
    for j in range(ave_grad.shape[1]):
        for k in range(ave_grad.shape[2]):
            U[counter] = ave_grad[i,j,k] * np.cos(deg[k])
            V[counter] = ave_grad[i,j,k] * np.sin(deg[k])

            X[counter] = (cell_size[0] / 2) + (cell_size[0] * i)
            Y[counter] = (cell_size[1] / 2) + (cell_size[1] * j)

            counter = counter + 1

# Create the bins in degress to plot our histogram.
angle_axis = np.linspace(0, 180, num_bins, endpoint = False)
angle_axis += ((angle_axis[1] - angle_axis[0]) / 2)

# Create a figure with 4 subplots arranged in 2 x 2
fig, ((a,b),(c,d)) = plt.subplots(2,2)

# Set the title of each subplot
a.set(title = 'Gray Scale Image\n(Click to Zoom)')
b.set(title = 'HOG Descriptor\n(Click to Zoom)')
c.set(title = 'Zoom Window', xlim = (0, 18), ylim = (0, 18), autoscale_on = False)
d.set(title = 'Histogram of Gradients')

# Plot the gray scale image
a.imshow(gray_image, cmap = 'gray')
a.set_aspect(aspect = 1)

# Plot the feature vector (HOG Descriptor)
b.quiver(Y, X, U, V, color = 'white', headwidth = 0, headlength = 0, scale_units = 'inches', scale = 5)
b.invert_yaxis()
b.set_aspect(aspect = 1)
b.set_facecolor('black')

# Define function for interactive zoom
def onpress(event):

    #Unless the left mouse button is pressed do nothing
    if event.button != 1:
        return

    # Only accept clicks for subplots a and b
    if event.inaxes in [a, b]:

        # Get mouse click coordinates
        x, y = event.xdata, event.ydata

        # Select the cell closest to the mouse click coordinates
        cell_num_x = np.uint32(x / cell_size[0])
        cell_num_y = np.uint32(y / cell_size[1])

        # Set the edge coordinates of the rectangle patch
        edgex = x - (x % cell_size[0])
        edgey = y - (y % cell_size[1])

        # Create a rectangle patch that matches the the cell selected above
        rect = patches.Rectangle((edgex, edgey),
                                  cell_size[0], cell_size[1],
                                  linewidth = 1,
                                  edgecolor = 'magenta',
                                  facecolor='none')

        # A single patch can only be used in a single plot. Create copies
        # of the patch to use in the other subplots
        rect2 = copy.copy(rect)
        rect3 = copy.copy(rect)

        # Update all subplots
        a.clear()
        a.set(title = 'Gray Scale Image\n(Click to Zoom)')
        a.imshow(gray_image, cmap = 'gray')
        a.set_aspect(aspect = 1)
        a.add_patch(rect)

        b.clear()
        b.set(title = 'HOG Descriptor\n(Click to Zoom)')
        b.quiver(Y, X, U, V, color = 'white', headwidth = 0, headlength = 0, scale_units = 'inches', scale = 5)
        b.invert_yaxis()
        b.set_aspect(aspect = 1)
        b.set_facecolor('black')
        b.add_patch(rect2)

        c.clear()
        c.set(title = 'Zoom Window')
        c.quiver(Y, X, U, V, color = 'white', headwidth = 0, headlength = 0, scale_units = 'inches', scale = 1)
        c.set_xlim(edgex - cell_size[0], edgex + (2 * cell_size[0]))
        c.set_ylim(edgey - cell_size[1], edgey + (2 * cell_size[1]))
        c.invert_yaxis()
        c.set_aspect(aspect = 1)
        c.set_facecolor('black')
        c.add_patch(rect3)

        d.clear()
        d.set(title = 'Histogram of Gradients')
        d.grid()
        d.set_xlim(0, 180)
        d.set_xticks(angle_axis)
        d.set_xlabel('Angle')
        d.bar(angle_axis,
              ave_grad[cell_num_y, cell_num_x, :],
              180 // num_bins,
              align = 'center',
              alpha = 0.5,
              linewidth = 1.2,
              edgecolor = 'k')

        fig.canvas.draw()

# Create a connection between the figure and the mouse click
fig.canvas.mpl_connect('button_press_event', onpress)
plt.show()
```

# 了解直方图

下面，让我们看一下上图的几个快照，看看所选单元格的直方图是否有意义。首先，看一个在三角形内部而不是边缘附近的一个单元格：

Fig. 4. - Histograms Inside a Triangle.

在这个示例中，由于三角形几乎都是相同的颜色，因此在所选单元格中不应存在任何主导梯度。我们在缩放窗口和直方图中也可以清楚地看到，情况确实如此。梯度有很多，但没有一个在另一个上占明显的主导地位。

现在，让我们看一下靠近水平边缘的一个单元格：

Fig. 5. - Histograms Near a Horizontal Edge.

请牢记，边缘是图像中强度变化很突然的一个区域。在这些区域，我们将在某个特定方向上具有高强度梯度。这正是我们在所选单元格的相应直方图和缩放窗口中看到的内容。在缩放窗口中，我们可以看到主导梯度呈几乎90度的向上趋势，因为这是强度急剧变化的方向。因此，我们应该期望直方图中的90度区域比其他区域更强。实际上，这就是我们所看到的。

接下来，让我们看一下靠近垂直边缘的一个单元格：

Fig. 6. - Histograms Near a Vertical Edge.

在这个示例中，我们希望该单元格中的主导梯度是水平的，接近180度，因为这是强度急剧变化的方向。因此，我们应该期望直方图中的170度区域会强烈支配其他区域。实际上，这就是我们在直方图中看到的，但我们也看到该单元格中还有另一个显性梯度，即10度区间中的梯度。原因在于HOG算法使用无符号梯度，这意味着0度和180度被认为是相同的。因此，当创建直方图时，160度和180度之间的角度与10度区间和170度区间成比例。这就会导致在垂直边缘附近的单元格中，存在两个主导梯度，而不是只有一个。

最后，让我们看一下靠近对角线边缘的一个单元格。

Fig. 7. - Histograms Near a Diagonal Edge.

要理解我们所看到的东西，首先，我们要知道，梯度有一个x分量和一个y分量，就像矢量一样。因此，一个梯度的最终方向将由其分量的矢量和而定。出于这个原因，在垂直边缘上，梯度都是水平的，因为它们只有一个x分量，如图4所示。而在水平边缘上，梯度都是垂直的，因为它们只有一个y分量，如图3所示。因此，在对角线边缘，梯度将会是对角线，因为x和y分量都是非零。由于图像中的对角线边缘接近45度，我们应该期望在50度区间中看到一个主导的梯度方向。实际上，这就是我们在直方图中看到的，但是，如图4所示，我们看到有两个主导梯度而不是一个。其原因在于，创建直方图时，靠近区间边缘的角度成比例地给相邻区间提供贡献。例如，角度为40度的梯度位于30度和50度区间的中间。因此，梯度的大小被均匀地分成30度和50度的区间。这就会导致在对角线边缘附近的单元中，存在两个主导梯度，而不是只有一个。

现在你已经了解了HOG的实现方式，接下来，在工作区中找到名为*Examples*的notebook。你可以在那里为各种图像的HOG描述子设置自己的参数。祝你学得开心哦！


[Table of Contents](./table_of_contents.ipynb)

# Designing Nonlinear Kalman Filters

```python
%matplotlib inline
```

```python
#format the book
import book_format
book_format.set_style()
```

<style>
.output_wrapper, .output {
    height:auto !important;
    max-height:100000px;
}
.output_scroll {
    box-shadow:none !important;
    webkit-box-shadow:none !important;
}
</style>

## Introduction

** Author's note: I was initially planning to have a design nonlinear chapter that compares various approaches. This may or may not happen, but for now this chapter has no useful content and I suggest not reading it. **

We see that the Kalman filter reasonably tracks the ball. However, as already explained, this is a silly example; we can predict trajectories in a vacuum with arbitrary precision; using a Kalman filter in this example is a needless complication.

### Kalman Filter with Air Drag

I will dispense with the step 1, step 2, type approach and proceed in a more natural style that you would use in a non-toy engineering problem. We have already developed a Kalman filter that does excellently at tracking a ball in a vacuum, but that does not incorporate the effects of air drag into the model. We know that the process model is implemented with $\textbf{F}$, so we will turn our attention to that immediately.

Notionally, the computation that $\textbf{F}$ computes is

$$x' = Fx$$

With no air drag, we had

$$
\mathbf{F} = \begin{bmatrix}
1 & \Delta t & 0 & 0 & 0 \\
0 & 1 & 0 & 0 & 0 \\
0 & 0 & 1 & \Delta t & \frac{1}{2}{\Delta t}^2 \\
0 & 0 & 0 & 1 & \Delta t \\
0 & 0 & 0 & 0 & 1
\end{bmatrix}
$$

which corresponds to the equations

$$
\begin{aligned}
x &= x + v_x \Delta t \\
v_x &= v_x \\
\\
y &= y + v_y \Delta t + \frac{a_y}{2} {\Delta t}^2 \\
v_y &= v_y + a_y \Delta t \\
a_y &= a_y
\end{aligned}
$$

From the section above we know that our new Euler equations must be

$$
\begin{aligned}
x &= x + v_x \Delta t \\
v_x &= v_x \\
\\
y &= y + v_y \Delta t + \frac{a_y}{2} {\Delta t}^2 \\
v_y &= v_y + a_y \Delta t \\
a_y &= a_y
\end{aligned}
$$

## Realistic 2D Position Sensors

The position sensor in the last example are not very realistic. In general there is no 'raw' sensor that provides (x,y) coordinates. We have GPS, but GPS already uses a Kalman filter to create a filtered output; we should not be able to improve the signal by passing it through another Kalman filter unless we incorporate additional sensors to provide additional information. We will tackle that problem later.

Consider the following set up. In an open field we put two transmitters at a known location, each transmitting a signal that we can detect. We process the signal and determine  how far we are from that signal, with some noise. First, let's look at a visual depiction of that.

```python
import matplotlib.pyplot as plt

circle1=plt.Circle((-4, 0), 5, color='#004080',
                   fill=False, linewidth=20, alpha=.7)
circle2=plt.Circle((4, 0), 5, color='#E24A33',
                   fill=False, linewidth=5, alpha=.7)

fig = plt.gcf()
ax = fig.gca()

plt.axis('equal')
plt.xlim((-10, 10))
plt.ylim((-10, 10))

plt.plot ([-4, 0], [0, 3], c='#004080')
plt.plot ([4, 0], [0, 3], c='#E24A33')
plt.text(-4, -.5, "A", fontsize=16, horizontalalignment='center')
plt.text(4, -.5, "B", fontsize=16, horizontalalignment='center')

ax.add_artist(circle1)
ax.add_artist(circle2)
plt.show()
```

Here I have attempted to show transmitter A, drawn in red, at (-4,0) and a second one B, drawn in blue, at (4,0). The red and blue circles show the range from the transmitters to the robot, with the width illustrating the effect of the $1\sigma$ angular error for each transmitter. Here I have given the blue transmitter more error than the red one. The most probable position for the robot is where the two circles intersect, which I have depicted with the red and blue lines. You will object that we have two intersections, not one, but we will see how we deal with that when we design the measurement function.

This is a very common sensor set up. Aircraft still use this system to navigate, where it is called DME (Distance Measuring Equipment). Today GPS is a much more common navigation system, but I have worked on an aircraft where we integrated sensors like this into our filter along with the GPS, INS, altimeters, etc. We will tackle what is called *multi-sensor fusion* later; for now we will just address this simple configuration.

The first step is to design our state variables. We will assume that the robot is traveling in a straight direction with constant velocity. This is unlikely to be true for a long period of time, but is acceptable for short periods of time. This does not differ from the previous problem - we will want to track the values for the robot's position and velocity. Hence,

$$\mathbf{x} =
\begin{bmatrix}x\\v_x\\y\\v_y\end{bmatrix}$$

The next step is to design the state transition function. This also will be the same as the previous problem, so without further ado,

$$
\mathbf{x}' = \begin{bmatrix}1& \Delta t& 0& 0\\0& 1& 0& 0\\0& 0& 1& \Delta t\\ 0& 0& 0& 1\end{bmatrix}\mathbf{x}$$

The next step is to design the control inputs. We have none, so we set ${\mathbf{B}}=0$.

The next step is to design the measurement function $\mathbf{z} = \mathbf{Hx}$. We can model the measurement using the Pythagorean theorem.

$$
z_a = \sqrt{(x-x_A)^2 + (y-y_A)^2} + v_a\\[1em]
z_b = \sqrt{(x-x_B])^2 + (y-y_B)^2} + v_b
$$

where $v_a$ and $v_b$ are white noise.

We see an immediate problem. The Kalman filter is designed for linear equations, and this is obviously nonlinear. In the next chapters we will look at several ways to handle nonlinear problems in a robust way, but for now we will do something simpler. If we know the approximate position of the robot than we can linearize these equations around that point. I could develop the generalized mathematics for this technique now, but instead let me just present the worked example to give context to that development.

Instead of computing $\mathbf{H}$ we will compute the partial derivative of $\mathbf{H}$ with respect to the robot's position $\mathbf{x}$. You are probably familiar with the concept of partial derivative, but if not, it just means how $\mathbf{H}$ changes with respect to the robot's position. It is computed as the partial derivative of $\mathbf{H}$ as follows:

$$\frac{\partial \mathbf{h}}{\partial \mathbf{x}} =
\begin{bmatrix}
\frac{\partial h_1}{\partial x_1} & \frac{\partial h_1}{\partial x_2} &\dots \\
\frac{\partial h_2}{\partial x_1} & \frac{\partial h_2}{\partial x_2} &\dots \\
\vdots & \vdots
\end{bmatrix}
$$

Let's work the first partial derivative. We want to find

$$\frac{\partial }{\partial x} \sqrt{(x-x_A)^2 + (y-y_A)^2}
$$

Which we compute as

$$
\begin{aligned}
\frac{\partial h_1}{\partial x} &= ((x-x_A)^2 + (y-y_A)^2))^\frac{1}{2} \\
&= \frac{1}{2}\times 2(x-x_a)\times ((x-x_A)^2 + (y-y_A)^2))^{-\frac{1}{2}} \\
&= \frac{x_r - x_A}{\sqrt{(x_r-x_A)^2 + (y_r-y_A)^2}}
\end{aligned}
$$

We continue this computation for the partial derivatives of the two distance equations with respect to $x$, $y$, $dx$ and $dy$, yielding

$$\frac{\partial\mathbf{h}}{\partial\mathbf{x}}=
\begin{bmatrix}
\frac{x_r - x_A}{\sqrt{(x_r-x_A)^2 + (y_r-y_A)^2}} & 0 &
\frac{y_r - y_A}{\sqrt{(x_r-x_A)^2 + (y_r-y_A)^2}} & 0 \\
\frac{x_r - x_B}{\sqrt{(x_r-x_B)^2 + (y_r-y_B)^2}} & 0 &
\frac{y_r - y_B}{\sqrt{(x_r-x_B)^2 + (y_r-y_B)^2}} & 0 \\
\end{bmatrix}
$$

That is pretty painful, and these are very simple equations. Computing the Jacobian can be extremely difficult or even impossible for more complicated systems. However, there is an easy way to get Python to do the work for you by using the SymPy module [1]. SymPy is a Python library for symbolic mathematics. The full scope of its abilities are beyond this book, but it can perform algebra, integrate and differentiate equations, find solutions to differential equations, and much more. We will use it to compute our Jacobian!

First, a simple example. We will import SymPy, initialize its pretty print functionality (which will print equations using LaTeX). We will then declare a symbol for NumPy to use.

```python
import sympy
from sympy import init_printing
init_printing(use_latex='mathjax')

phi, x = sympy.symbols('\phi, x')
phi
```

$\displaystyle \phi$

Notice how we use a latex expression for the symbol `phi`. This is not necessary, but if you do it will render as LaTeX when output. Now let's do some math. What is the derivative of $\sqrt{\phi}$?

```python
sympy.diff('sqrt(phi)')
```

$\displaystyle \frac{1}{2 \sqrt{\phi}}$

We can factor equations.

```python
sympy.factor('phi**3 -phi**2 + phi - 1')
```

$\displaystyle \left(\phi - 1\right) \left(\phi^{2} + 1\right)$

SymPy has a remarkable list of features, and as much as I enjoy exercising its features we cannot cover them all here. Instead, let's compute our Jacobian.

```python
from sympy import symbols, Matrix
phi = symbols('\phi')
phi

x, y, xa, xb, ya, yb, dx, dy = symbols('x y x_a x_b y_a y_b dx dy')

H = Matrix([[sympy.sqrt((x-xa)**2 + (y-ya)**2)],
            [sympy.sqrt((x-xb)**2 + (y-yb)**2)]])

state = Matrix([x, dx, y, dy])
H.jacobian(state)
```

$\displaystyle \left[\begin{matrix}\frac{x - x_{a}}{\sqrt{\left(x - x_{a}\right)^{2} + \left(y - y_{a}\right)^{2}}} & 0 & \frac{y - y_{a}}{\sqrt{\left(x - x_{a}\right)^{2} + \left(y - y_{a}\right)^{2}}} & 0\\\frac{x - x_{b}}{\sqrt{\left(x - x_{b}\right)^{2} + \left(y - y_{b}\right)^{2}}} & 0 & \frac{y - y_{b}}{\sqrt{\left(x - x_{b}\right)^{2} + \left(y - y_{b}\right)^{2}}} & 0\end{matrix}\right]$

In a nutshell, the entry (0,0) contains the difference between the x coordinate of the robot and transmitter A's x coordinate divided by the distance between the robot and A. (2,0) contains the same, except for the y coordinates of the robot and transmitters. The bottom row contains the same computations, except for transmitter B. The 0 entries account for the velocity components of the state variables; naturally the range does not provide us with velocity.

The values in this matrix change as the robot's position changes, so this is no longer a constant; we will have to recompute it for every time step of the filter.

If you look at this you may realize that this is just a computation of x/dist and y/dist, so we can switch this to a trigonometic form with no loss of generality:

$$\frac{\partial\mathbf{h}}{\partial\mathbf{x}}=
\begin{bmatrix}
-\cos{\theta_A} & 0 & -\sin{\theta_A} & 0 \\
-\cos{\theta_B} & 0 & -\sin{\theta_B} & 0
\end{bmatrix}
$$

However, this raises a huge problem. We are no longer computing $\mathbf{H}$, but $\Delta\mathbf{H}$, the change of $\mathbf{H}$. If we passed this into our Kalman filter without altering the rest of the design the output would be nonsense. Recall, for example, that we multiply $\mathbf{Hx}$ to generate the measurements that would result from the given estimate of $\mathbf{x}$ But now that $\mathbf{H}$ is linearized around our position it contains the *change* in the measurement function.

We are forced, therefore, to use the *change* in $\mathbf{x}$ for our state variables. So we have to go back and redesign our state variables.

>Please note this is a completely normal occurrence in designing Kalman filters. The textbooks present examples like this as *fait accompli*, as if it is trivially obvious that the state variables needed to be velocities, not positions. Perhaps once you do enough of these problems it would be trivially obvious, but at that point why are you reading a textbook? I find myself reading through a presentation multiple times, trying to figure out why they made a choice, finally to realize that it is because of the consequences of something on the next page. My presentation is longer, but it reflects what actually happens when you design a filter. You make what seem reasonable design choices, and as you move forward you discover properties that require you to recast your earlier steps. As a result, I am going to somewhat abandon my **step 1**, **step 2**, etc.,  approach, since so many real problems are not quite that straightforward.

If our state variables contain the velocities of the robot and not the position then how do we track where the robot is? We can't. Kalman filters that are linearized in this fashion use what is called a *nominal trajectory* - i.e. you assume a position and track direction, and then apply the changes in velocity and acceleration to compute the changes in that trajectory. How could it be otherwise? Recall the graphic showing the intersection of the two range circles - there are two areas of intersection. Think of what this would look like if the two transmitters were very close to each other - the intersections would be two very long crescent shapes. This Kalman filter, as designed, has no way of knowing your true position from only distance measurements to the transmitters. Perhaps your mind is already leaping to ways of working around this problem. If so, stay engaged, as later sections and chapters will provide you with these techniques. Presenting the full solution all at once leads to more confusion than insight, in my opinion.

So let's redesign our *state transition function*. We are assuming constant velocity and no acceleration, giving state equations of
$$
\dot{x}' = \dot{x} \\
\ddot{x}' = 0 \\
\dot{y}' = \dot{y} \\
\dot{y}' = 0$$

This gives us the the *state transition function* of

$$
\mathbf{F} = \begin{bmatrix}0 &1 & 0& 0\\0& 0& 0& 0\\0& 0& 0& 1\\ 0& 0& 0& 0\end{bmatrix}$$

A final complication comes from the measurements that we pass in. $\mathbf{Hx}$ is now computing the *change* in the measurement from our nominal position, so the measurement that we pass in needs to be not the range to A and B, but the *change* in range from our measured range to our nominal position.

There is a lot here to take in, so let's work through the code bit by bit. First we will define a function to compute $\frac{\partial\mathbf{h}}{\partial\mathbf{x}}$ for each time step.

```python
from math import sin, cos, atan2

def H_of(pos, pos_A, pos_B):
    """ Given the position of our object at 'pos' in 2D, and two
    transmitters A and B at positions 'pos_A' and 'pos_B', return
    the partial derivative of H
    """

    theta_a = atan2(pos_a[1] - pos[1], pos_a[0] - pos[0])
    theta_b = atan2(pos_b[1] - pos[1], pos_b[0] - pos[0])

    return np.array([[0, -cos(theta_a), 0, -sin(theta_a)],
                     [0, -cos(theta_b), 0, -sin(theta_b)]])
```

Now we need to create our simulated sensor.

```python
from numpy.random import randn

class DMESensor(object):
    def __init__(self, pos_a, pos_b, noise_factor=1.0):
        self.A = pos_a
        self.B = pos_b
        self.noise_factor = noise_factor

    def range_of(self, pos):
        """ returns tuple containing noisy range data to A and B
        given a position 'pos'
        """

        ra = math.sqrt((self.A[0] - pos[0])**2 + (self.A[1] - pos[1])**2)
        rb = math.sqrt((self.B[0] - pos[0])**2 + (self.B[1] - pos[1])**2)

        return (ra + randn()*self.noise_factor,
                rb + randn()*self.noise_factor)
```

Finally, we are ready for the Kalman filter code. I will position the transmitters at x=-100 and 100, both with y=-20. This gives me enough space to get good triangulation from both as the robot moves. I will start the robot at (0,0) and move by (1,1) each time step.

```python
import kf_book.book_plots as bp
from filterpy.kalman import KalmanFilter
import math
import numpy as np

pos_a = (100, -20)
pos_b = (-100, -20)

f1 = KalmanFilter(dim_x=4, dim_z=2)

f1.F = np.array ([[0, 1, 0, 0],
                  [0, 0, 0, 0],
                  [0, 0, 0, 1],
                  [0, 0, 0, 0]], dtype=float)

f1.R *= 1.
f1.Q *= .1

f1.x = np.array([[1, 0, 1, 0]], dtype=float).T
f1.P = np.eye(4) * 5.

# initialize storage and other variables for the run
count = 30
xs, ys = [], []
pxs, pys = [], []

# create the simulated sensor
d = DMESensor(pos_a, pos_b, noise_factor=3.)

# pos will contain our nominal position since the filter does not
# maintain position.
pos = [0, 0]

for i in range(count):
    # move (1,1) each step, so just use i
    pos = [i, i]

    # compute the difference in range between the nominal track
    # and measured ranges
    ra,rb = d.range_of(pos)
    rx,ry = d.range_of((pos[0] + f1.x[0, 0], pos[1] + f1.x[2, 0]))
    z = np.array([[ra - rx], [rb - ry]])

    # compute linearized H for this time step
    f1.H = H_of (pos, pos_a, pos_b)

    # store stuff so we can plot it later
    xs.append(f1.x[0, 0]+i)
    ys.append(f1.x[2, 0]+i)
    pxs.append(pos[0])
    pys.append(pos[1])

    # perform the Kalman filter steps
    f1.predict()
    f1.update(z)

bp.plot_filter(xs, ys)
bp.plot_track(pxs, pys)
plt.legend(loc=2)
plt.show()
```

## Linearizing the Kalman Filter

Now that we have seen an example of linearizing the Kalman filter we are in a position to better understand the math.

We start by assuming some function $\mathbf f$

## Example: A falling Ball

**author's note: ignore this section for now. **

In the **Designing Kalman Filters** chapter I first considered tracking a ball in a vacuum, and then in the atmosphere. The Kalman filter performed very well for vacuum, but diverged from the ball's path in the atmosphere. Let us look at the output; to avoid littering this chapter with code from that chapter I have placed it all in the file `ekf_internal.py'.

```python
import kf_book.ekf_internal as ekf
ekf.plot_ball()
```

We can artificially force the Kalman filter to track the ball by making $Q$ large. That would cause the filter to mistrust its prediction, and scale the kalman gain $K$ to strongly favor the measurments. However, this is not a valid approach. If the Kalman filter is correctly predicting the process we should not 'lie' to the filter by telling it there are process errors that do not exist. We may get away with that for some problems, in some conditions, but in general the Kalman filter's performance will be substandard.

Recall from the **Designing Kalman Filters** chapter that the acceleration is

$$a_x = (0.0039 + \frac{0.0058}{1+\exp{[(v-35)/5]}})*v*v_x \\
a_y = (0.0039 + \frac{0.0058}{1+\exp{[(v-35)/5]}})*v*v_y- g
$$

These equations will be *very* unpleasant to work with while we develop this subject, so for now I will retreat to a simpler one dimensional problem using this simplified equation for acceleration that does not take the nonlinearity of the drag coefficient into account:

$$\begin{aligned}
\ddot{y} &= \frac{0.0034ge^{-y/20000}\dot{y}^2}{2\beta} - g \\
\ddot{x} &= \frac{0.0034ge^{-x/20000}\dot{x}^2}{2\beta}
\end{aligned}$$

Here $\beta$ is the ballistic coefficient, where a high number indicates a low drag.

This is still nonlinear, so we need to linearize this equation at the current state point. If our state is position and velocity, we need an equation for some arbitrarily small change in $\mathbf{x}$, like so:

$$ \begin{bmatrix}\Delta \dot{x} \\ \Delta \ddot{x} \\ \Delta \dot{y} \\ \Delta \ddot{y}\end{bmatrix} =
\large\begin{bmatrix}
\frac{\partial \dot{x}}{\partial x} &
\frac{\partial \dot{x}}{\partial \dot{x}} &
\frac{\partial \dot{x}}{\partial y} &
\frac{\partial \dot{x}}{\partial \dot{y}} \\
\frac{\partial \ddot{x}}{\partial x} &
\frac{\partial \ddot{x}}{\partial \dot{x}}&
\frac{\partial \ddot{x}}{\partial y}&
\frac{\partial \dot{x}}{\partial \dot{y}}\\
\frac{\partial \dot{y}}{\partial x} &
\frac{\partial \dot{y}}{\partial \dot{x}} &
\frac{\partial \dot{y}}{\partial y} &
\frac{\partial \dot{y}}{\partial \dot{y}} \\
\frac{\partial \ddot{y}}{\partial x} &
\frac{\partial \ddot{y}}{\partial \dot{x}}&
\frac{\partial \ddot{y}}{\partial y}&
\frac{\partial \dot{y}}{\partial \dot{y}}
\end{bmatrix}\normalsize
\begin{bmatrix}\Delta x \\ \Delta \dot{x} \\ \Delta \dot{y} \\ \Delta \ddot{y}\end{bmatrix}$$

The equations do not contain both an x and a y, so any partial derivative with both in it must be equal to zero. We also know that $\large\frac{\partial \dot{x}}{\partial x}\normalsize = 0$ and that  $\large\frac{\partial \dot{x}}{\partial \dot{x}}\normalsize = 1$, so our matrix ends up being

$$\mathbf{F} = \begin{bmatrix}0&1&0&0 \\
\frac{0.0034e^{-x/22000}\dot{x}^2g}{44000\beta}&0&0&0
\end{bmatrix}$$

$$\begin{aligned}\ddot{x} &= -\frac{1}{2}C_d\rho A \dot{x}\\
\ddot{y} &= -\frac{1}{2}C_d\rho A \dot{y}-g\end{aligned}$$

```python
from sympy.abc import *
from sympy import *

init_printing(pretty_print=True, use_latex='mathjax')

x1 = (0.0034*g*exp(-x/22000)*((x)**2))/(2*b) - g

x2 = (a*g*exp(-x/c)*(Derivative(x)**2))/(2*b) - g

#pprint(x1)
#pprint(Derivative(x)*Derivative(x,n=2))
#pprint(diff(x2, x))
```

** orphan text
This approach has many issues. First, of course, is the fact that the linearization does not produce an exact answer. More importantly, we are not linearizing the actual path, but our filter's estimation of the path. We linearize the estimation because it is statistically likely to be correct; but of course it is not required to be. So if the filter's output is bad that will cause us to linearize an incorrect estimate, which will almost certainly lead to an even worse estimate. In these cases the filter will quickly diverge. This is where the 'black art' of Kalman filter comes in. We are trying to linearize an estimate, and there is no guarantee that the filter will be stable. A vast amount of the literature on Kalman filters is devoted to this problem. Another issue is that we need to linearize the system using analytic methods. It may be difficult or impossible to find an analytic solution to some problems. In other cases we may be able to find the linearization, but the computation is very expensive. **

## References

[1] http://sympy.org


# Random Forest + LIBRAS

This notebook covers a Python-based solution to be compared to the experiments 1 to 4 presented in .

The presented solution consists in reproducing the feature vectors for each of the experiments utilizing (x, y) points in , applying a RandomForest classifier, and lastly, comparing the resulting performance to the ones obtained by SVM and k-NN approaches.

## Context

\# TO DO

## Dataset

\# TO DO

## Experiments

\# TO DO

```python
import glob as gl
import numpy as np
from scipy.io import loadmat
from typing import List
import pandas as pd
```

```python
def labelname(file_name):  # Extract labels from filenames
    label = file_name.replace("data/fundamental/sample", "").lower()
    label = label.replace(".mat", "").lower()
    label = label.split('-', 1)[-1]
    return label

class Signal:  # Signal representation containing x and y coordinates and corresponding label
    def __init__(self, x, y, label):
        self.x = x
        self.y = y
        self.label = label
```

### Experiment 1

The first experiment utilizes each signal in its raw form.

```python
def sigvalues(sig):  # EX.1: raw data
    updsig = np.zeros((np.shape(sig.x)[0], np.shape(sig.x)[1] * 2))
    updsig[:, ::2] = sig.x
    updsig[:, 1::2] = sig.y

    return updsig  # Updated signal
```

### Experiment 2

The second experiment consists of applying z-normalization to each sample. The updated coordinates are:
\begin{align}
x_{\mathcal{N(0,1)}}=\frac{x-\bar{x}}{\sigma(x)} \\
y_{\mathcal{N(0,1)}}=\frac{y-\bar{y}}{\sigma(y)}
\end{align}

```python
def sigvalues(sig):  # EX.2: z-norm
    updsig = np.zeros((np.shape(sig.x)[0], np.shape(sig.x)[1] * 2))
    for idx, x in enumerate(sig.x):
        sig.x[idx] = np.divide((np.transpose(x) - np.mean(x)), np.std(x))
    for idx, y in enumerate(sig.y):
        sig.y[idx] = np.divide((np.transpose(y) - np.mean(y)), np.std(y))

    updsig[:, ::2] = sig.x
    updsig[:, 1::2] = sig.y

    return updsig  # Updated signal
```

### Experiment 3

The third experiment consists of normalizing each signal by its 1st frame centroid, as follows:
\begin{align}
\tilde{x}_{P,f}=x_{P,f}-\bar{x}_{1} \\
\tilde{y}_{P,f}=y_{P,f}-\bar{y}_{1}
\end{align}

```python
def sigvalues(sig): # EX.3: norm by 1st frame centroid
    frame_x = np.split(sig.x, 5, axis=1)  # x-coordinates by frame
    frame_y = np.split(sig.y, 5, axis=1)  # y-coordinates by frame
    cent_x, cent_y = (np.mean(frame_x[0], axis=1), np.mean(frame_y[0], axis=1))  # first frame centroid of each recording

    updsig = np.zeros((np.shape(sig.x)[0], np.shape(sig.x)[1] * 2))
    for idx, x in enumerate(sig.x):
        sig.x[idx] = x - cent_x[idx]
    for idx, y in enumerate(sig.y):
        sig.y[idx] = y - cent_y[idx]

    updsig[:, ::2] = sig.x
    updsig[:, 1::2] = sig.y

    return updsig  # Updated signal
```

### Experiment 4

The fourth experiment consists of normalizing each signal by its current frame centroid, as follows:
\begin{align}
\tilde{x}_{P,f}=x_{P,f}-\bar{x}_{f} \\
\tilde{y}_{P,f}=y_{P,f}-\bar{y}_{f}
\end{align}

```python
def sigvalues(sig):  # EX.4
    frame_x = np.split(sig.x, 5, axis=1)  # x-coordinates by frame
    frame_y = np.split(sig.y, 5, axis=1)  # y-coordinates by frame
    cent_x, cent_y = (np.mean(frame_x, axis=2), np.mean(frame_y, axis=2))  # centroids of each recording

    nframes, nrecs, idx = np.shape(frame_x)
    updsig = np.zeros((np.shape(sig.x)[0], np.shape(sig.x)[1] * 2))
    for fx in range(nframes):
        frame_x[fx] = np.transpose(np.transpose(frame_x[fx]) - cent_x[fx])
    for fy in range(nframes):
        frame_y[fy] = np.transpose(np.transpose(frame_y[fy]) - cent_y[fy])

    updsig[:, ::2] = np.hstack(frame_x)
    updsig[:, 1::2] = np.hstack(frame_y)

    return updsig  # Updated signal
```

## Classification

For means of comparison, this solution utilizes the Random Forest algorithm for the classification task. Tuning is done on every iteration utilizing GridSearchCV.

```python
files = gl.glob("data/fundamental/*.mat")  # type: list

signals = []  # type: List[Signal]

for f in files:
    data = loadmat(f).get('pontosSinal')
    signals.append(Signal(data[:, ::2], data[:, 1::2], labelname(f)))

n_signs = len(signals)
n_recs, n_x = np.shape(signals[0].x)  # Number of recordings and number of features

signals_feat = []  # Updated signals, according to each experiment
signals_labels = []
labels_dict = {'angry': 1, 'disgusted': 2, 'fearful': 3, 'happy': 4, 'sad': 5, 'surprised': 6,
               'neutral': 7, 'suckedcheeks': 8, 'inflatedcheeks': 9}  # Dictionary of signals' labels, for reference
for s in signals:
    signals_feat.append(sigvalues(s))
    signals_labels.append([labels_dict[s.label]] * n_recs)

sig_features = np.reshape(signals_feat, (n_signs * n_recs, n_x * 2))
sig_labels = np.reshape(signals_labels, (n_signs * n_recs,))
```

```python
from sklearn.model_selection import StratifiedShuffleSplit
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import RandomizedSearchCV
# from sklearn.model_selection import GridSearchCV
from sklearn.metrics import classification_report
from pprint import pprint
```

\# TO DO: PARAMETER SELECTION

```python
# Number of trees in random forest
n_estimators = [int(x) for x in np.linspace(start=800, stop=2000, num=10)]
# Number of features to consider at every split
max_features = ['log2', 'sqrt', 0.3]
# Maximum number of levels in tree
max_depth = [int(x) for x in np.linspace(10, 80, num=10)]
max_depth.append(None)
# Minimum number of samples required to split a node
min_samples_split = [3, 5, 7]
# Minimum number of samples required at each leaf node
min_samples_leaf = [2, 3, 4]
# Method of selecting samples for training each tree
bootstrap = [True, False]
# Criterion
criterion = ['gini', 'entropy']
# Create the random grid
random_grid = {'n_estimators': n_estimators,
               'max_features': max_features,
               'max_depth': max_depth,
               'min_samples_split': min_samples_split,
               'min_samples_leaf': min_samples_leaf,
               'bootstrap': bootstrap,
               'criterion': criterion
              }
```

\# TO DO

```python
# cv = 5
niter = 30
results = []
train_acc = []
test_acc = []
train_report =[]
class_report = []
selected_params = []
cm = [] # confusion matrix
feature_importance = []
col_names = range(1,1211)

for i in range(niter):
    print("Iteration:: ", i)
    sss = StratifiedShuffleSplit(n_splits=3, test_size=0.2, random_state=42)
    sss.get_n_splits(sig_features, sig_labels)

    for train_index, test_index in sss.split(sig_features, sig_labels):
        train_x, test_x = sig_features[train_index], sig_features[test_index]
        train_y, test_y = sig_labels[train_index], sig_labels[test_index]

    rf = RandomForestClassifier()
    rf_random = RandomizedSearchCV(estimator=rf, param_distributions=random_grid, n_iter=500, cv=3, verbose=1, n_jobs=-1)

    rf_random.fit(train_x, train_y)
    predictions = rf_random.predict(test_x)

    selected_params.append(rf_random.best_params_)
    train_acc.append(accuracy_score(train_y, rf_random.predict(train_x)))
    test_acc.append(accuracy_score(test_y, predictions))
    cm.append(pd.crosstab(test_y, predictions, rownames=['True'], colnames=['Predicted'], margins=True))
    class_report.append(classification_report(test_y, predictions, target_names=list(labels_dict.keys())))
    feature_importance.append(pd.DataFrame(data=sorted(zip(map(lambda x: round(x, 4),
                                                               rf_random.best_estimator_.feature_importances_),
                                                           col_names), reverse=True)))
```

    Iteration::  0
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:  2.5min
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  7.7min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed: 17.2min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 30.4min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 48.4min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 60.2min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

    Iteration::  1
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:  1.3min
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  7.1min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed: 15.1min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 27.5min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 45.2min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 53.3min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

    Iteration::  2
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:  2.1min
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  7.2min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed: 16.4min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 28.4min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 46.6min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 57.6min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

    Iteration::  3
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:  2.1min
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  8.0min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed: 16.5min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 30.1min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 47.7min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 58.7min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

    Iteration::  4
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:  1.6min
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  7.9min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed: 16.4min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 30.2min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 49.0min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 60.7min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

    Iteration::  5
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:  1.4min
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  7.8min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed: 17.2min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 29.8min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 49.1min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 60.6min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

    Iteration::  6
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:  1.3min
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  5.8min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed: 16.3min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 29.4min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 47.1min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 56.6min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

    Iteration::  7
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:  1.9min
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  7.2min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed: 17.2min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 30.0min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 45.7min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 55.0min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

    Iteration::  8
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:  1.1min
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  6.3min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed: 16.4min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 29.2min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 47.3min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 56.4min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

    Iteration::  9
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:  1.0min
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  5.8min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed: 16.3min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 30.6min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 48.8min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 57.8min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

    Iteration::  10
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:  1.2min
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  8.3min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed: 17.8min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 31.7min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 46.9min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 57.9min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

    Iteration::  11
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:  1.6min
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  7.6min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed: 17.6min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 31.1min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 48.0min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 58.4min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

    Iteration::  12
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:  1.8min
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  6.4min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed: 15.6min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 31.2min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 47.5min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 56.0min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

    Iteration::  13
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:  2.1min
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  7.7min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed: 16.1min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 23.0min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 32.3min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 37.1min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

    Iteration::  14
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:  1.4min
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  4.7min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed:  8.6min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 14.8min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 23.6min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 29.3min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

    Iteration::  15
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:   36.9s
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  3.7min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed:  9.5min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 16.0min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 23.8min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 29.4min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

    Iteration::  16
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:   39.0s
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  4.3min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed:  8.7min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 14.5min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 22.2min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 26.3min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

    Iteration::  17
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:   49.0s
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  3.6min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed:  8.5min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 14.7min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 21.4min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 26.2min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

    Iteration::  18
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:  1.1min
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  4.3min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed:  9.4min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 16.6min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 25.8min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 30.9min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

    Iteration::  19
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:  1.2min
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  4.3min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed: 12.8min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 25.3min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 44.9min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 54.8min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

    Iteration::  20
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:  1.3min
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  7.1min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed: 16.6min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 27.2min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 45.0min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 55.6min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

    Iteration::  21
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:  2.0min
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  7.6min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed: 17.4min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 32.2min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 49.2min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 58.5min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

    Iteration::  22
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:   46.0s
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  6.9min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed: 17.0min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 29.7min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 47.0min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 57.7min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

    Iteration::  23
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:  1.7min
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  8.0min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed: 17.7min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 30.4min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 50.0min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 59.3min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

    Iteration::  24
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:  1.7min
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  7.3min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed: 17.7min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 29.1min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 45.3min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 55.0min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

    Iteration::  25
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:  1.8min
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  7.8min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed: 16.4min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 29.9min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 45.9min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 57.8min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

    Iteration::  26
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:  2.0min
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  7.8min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed: 20.1min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 32.0min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 45.2min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 54.0min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

    Iteration::  27
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:  1.5min
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  7.2min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed: 17.0min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 31.2min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 47.8min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 57.0min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

    Iteration::  28
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:  1.4min
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  7.2min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed: 15.1min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 27.4min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 46.1min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 57.2min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

    Iteration::  29
    Fitting 3 folds for each of 500 candidates, totalling 1500 fits

    [Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:  1.8min
    [Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  7.9min
    [Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed: 16.2min
    [Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 30.1min
    [Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 45.5min
    [Parallel(n_jobs=-1)]: Done 1500 out of 1500 | elapsed: 55.2min finished
    C:\Users\rubiarg\Anaconda3\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py:1428: UserWarning: labels size, 6, does not match size of target_names, 9
      .format(len(labels), len(target_names))

## Results

```python
from functools import reduce

conf_matrix = reduce(lambda x, y: x.add(y, fill_value=0), cm)

np.savetxt("results/thesis/raw-trainacc.csv",train_acc,delimiter=",")
np.savetxt("results/thesis/raw-testacc.csv",test_acc,delimiter=",")
np.savetxt("results/thesis/raw-selected_params.txt",selected_params,fmt='%s',delimiter='\n')
np.savetxt("results/thesis/raw-classreport.txt",class_report,fmt='%s',delimiter='\n')
conf_matrix.to_csv("results/thesis/raw-cm.csv")

import pickle
with open('results/thesis/raw-featimportance.pickle', 'wb') as fp:
    for feat in feature_importance:
        pickle.dump(feat, fp)
```

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib notebook
acc = np.transpose([train_acc, test_acc])
acc = pd.DataFrame(data=acc, columns=['train_acc', 'test_acc'])
boxplot = acc.boxplot(column=['train_acc', 'test_acc'])
plt.show()
```

    <IPython.core.display.Javascript object>

```python
acc.describe()
```

      train_acc
      test_acc

      count
      30.000000
      3.000000e+01

      mean
      0.999583
      8.000000e-01

      std
      0.002282
      2.258405e-16

      min
      0.987500
      8.000000e-01

      25%
      1.000000
      8.000000e-01

      50%
      1.000000
      8.000000e-01

      75%
      1.000000
      8.000000e-01

      max
      1.000000
      8.000000e-01

# References