# Tutorial 2: Facial recognition using modern convnets
**Week 2, Day 2: Modern Convnets**

**By Neuromatch Academy**

__Content creators:__ Laura Pede, Richard Vogg, Marissa Weis, Timo Lüddecke, Alexander Ecker (based on an initial version by Ben Heil)

__Content reviewers:__ Arush Tagade, Polina Turishcheva, Yu-Fang Yang, Bettina Hein, Melvin Selim Atay

__Content editors:__ Roberto Guidotti, Spiros Chavlis

__Production editors:__ Anoop Kulkarni, Roberto Guidotti, Cary Murray, Spiros Chavlis

**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

---
# Tutorial Objectives

In this tutorial you will learn about:

1. An application of modern CNNs in facial recognition.
2. Ethical aspects of facial recognition.

```python
# @title Tutorial slides

# @markdown These are the slides for the videos in this tutorial
from IPython.display import IFrame
IFrame(src=f"https://mfr.ca-1.osf.io/render?url=https://osf.io/4r2dp/?direct%26mode=render%26action=download%26mode=render", width=854, height=480)
```

---
# Setup

```python
# @title Install dependencies
# @markdown Install facenet - a model used to do facial recognition
!pip install facenet-pytorch --quiet
!pip install pillow --quiet
```

```python
# Imports
import glob
import torch

import numpy as np
import sklearn.decomposition
import matplotlib.pyplot as plt

from PIL import Image

from torchvision import transforms
from torchvision.utils import make_grid
from torchvision.datasets import ImageFolder

from facenet_pytorch import MTCNN, InceptionResnetV1
```

```python
# @title Set random seed

# @markdown Executing `set_seed(seed=seed)` you are setting the seed

# for DL its critical to set the random seed so that students can have a
# baseline to compare their results to expected results.
# Read more here: https://pytorch.org/docs/stable/notes/randomness.html

# Call `set_seed` function in the exercises to ensure reproducibility.
import random
import torch

def set_seed(seed=None, seed_torch=True):
  if seed is None:
    seed = np.random.choice(2 ** 32)
  random.seed(seed)
  np.random.seed(seed)
  if seed_torch:
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True

  print(f'Random seed {seed} has been set.')

# In case that `DataLoader` is used
def seed_worker(worker_id):
  worker_seed = torch.initial_seed() % 2**32
  np.random.seed(worker_seed)
  random.seed(worker_seed)
```

```python
# @title Set device (GPU or CPU). Execute `set_device()`
# especially if torch modules used.

# inform the user if the notebook uses GPU or CPU.

def set_device():
  device = "cuda" if torch.cuda.is_available() else "cpu"
  if device != "cuda":
    print("WARNING: For this notebook to perform best, "
        "if possible, in the menu under `Runtime` -> "
        "`Change runtime type.`  select `GPU` ")
  else:
    print("GPU is enabled in this notebook.")

  return device
```

```python
SEED = 2021
set_seed(seed=SEED)
DEVICE = set_device()
```

---
# Section 1: Face Recognition

## Section 1.1: Download and prepare the data

```python
# @title Download Data

!rm -rf cis_522_data/
!git clone --quiet https://github.com/ben-heil/cis_522_data.git
!tar -xzf cis_522_data/archive.tar.gz
!tar -xzf cis_522_data/faces.tar.gz
```

```python
# @title Video 1: Face Recognition using CNNs
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV17B4y1K7WV", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"jJqEv8hpRa4", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

One application of large CNNs is **facial recognition**. The problem formulation in facial recognition is a little different from the image classification we've seen so far. In facial recognition, we don't want to have a fixed number of individuals that the model can learn. If that were the case then to learn a new person it would be necessary to modify the output portion of the architecture and retrain to account for the new person.

Instead, we train a model to learn an **embedding** where images from the same individual are close to each other in an embedded space, and images corresponding to different people are far apart. When the model is trained, it takes as input an image and outputs an embedding vector corresponding to the image.

To achieve this, facial recognitions typically use a **triplet loss** that compares two images from the same individual (i.e., "anchor" and "positive" images) and a negative image from a different individual (i.e., "negative" image). The loss requires the distance between the anchor and negative points to be greater than a margin $\alpha$ + the distance between the anchor and positive points.

## Section 1.2: View and transform the data

A well-trained facial recognition system should be able to map different images of the same individual relatively close together. We will load 15 images of three individuals (maybe you know them - then you can see that your brain is quite well in facial recognition).

After viewing the images, we will transform them: MTCNN ([github repo](https://github.com/ipazc/mtcnn)) detects the face and crops the image around the face. Then we stack all the images together in a tensor.

```python
# @title Display Images
# @markdown Here are the source images of Bruce Lee, Neil Patrick Harris, and Pam Grier
train_transform = transforms.Compose((transforms.Resize((256, 256)),
                                      transforms.ToTensor()))

face_dataset = ImageFolder('faces', transform=train_transform)

image_count = len(face_dataset)

face_loader = torch.utils.data.DataLoader(face_dataset,
                                          batch_size=45,
                                          shuffle=False)

dataiter = iter(face_loader)
images, labels = dataiter.next()

# show images
plt.figure(figsize=(15, 15))
plt.imshow(make_grid(images, nrow=15).permute(1, 2, 0))
plt.axis('off')
plt.show()
```

```python
# @title Image Preprocessing Function
def process_images(image_dir: str, size=256):
  """
  This function returns two tensors for the given image dir: one usable for inputting into the
  facenet model, and one that is [0,1] scaled for visualizing

  Parameters:
    image_dir: The glob corresponding to images in a directory

  Returns:
    model_tensor: A image_count x channels x height x width tensor scaled to between -1 and 1,
                  with the faces detected and cropped to the center using mtcnn
    display_tensor: A transformed version of the model tensor scaled to between 0 and 1
  """
  mtcnn = MTCNN(image_size=size, margin=32)
  images = []
  for img_path in glob.glob(image_dir):
    img = Image.open(img_path)
    # Normalize and crop image
    img_cropped = mtcnn(img)
    images.append(img_cropped)

  model_tensor = torch.stack(images)
  display_tensor = model_tensor / (model_tensor.max() * 2)
  display_tensor += .5

  return model_tensor, display_tensor
```

Now that we have our images loaded, we need to preprocess them. To make the images easier for the network to learn, we crop them to include just faces.

```python
bruce_tensor, bruce_display = process_images('faces/bruce/*.jpg')
neil_tensor, neil_display = process_images('faces/neil/*.jpg')
pam_tensor, pam_display = process_images('faces/pam/*.jpg')

tensor_to_display = torch.cat((bruce_display, neil_display, pam_display))

plt.figure(figsize=(15, 15))
plt.imshow(make_grid(tensor_to_display, nrow=15).permute(1, 2, 0))
plt.axis('off')
plt.show()
```

## Section 1.3: Embedding with a pretrained network

We load a pretrained facial recognition model called [FaceNet](https://github.com/timesler/facenet-pytorch). It was trained on the [VGGFace2](https://github.com/ox-vgg/vgg_face2) dataset which contains 3.31 million images of 9131 individuals.

We use the pretrained model to calculate embeddings for all of our input images.

```python
resnet = InceptionResnetV1(pretrained='vggface2').eval().to(DEVICE)
```

```python
# Calculate embedding
resnet.classify = False
bruce_embeddings = resnet(bruce_tensor.to(DEVICE))
neil_embeddings = resnet(neil_tensor.to(DEVICE))
pam_embeddings = resnet(pam_tensor.to(DEVICE))
```

### Think! 1.3: Embedding vectors

We want to understand what happens when the model receives an image and returns the corresponding embedding vector.

- What are the height, width and number of channels of one input image?
- What are the dimensions of one stack of images (e.g. bruce_tensor)?
- What are the dimensions of the corresponding embedding (e.g. bruce_embeddings)?
- What would be the dimensions of the embedding of one input image?

**Hints:**
- You can double click on a variable name and hover over it to see the dimensions of tensors.
- You do not have to answer the questions in the order they are asked.

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W2D2_ModernConvnets/solutions/W2D2_Tutorial2_Solution_bbe072ff.py)

We cannot show 512-dimensional vectors visually, but using **Principal Component Analysis (PCA)** we can project the 512 dimensions onto a 2-dimensional space while preserving the maximum amount of data variation possible. This is just a visual aid for us to understand the concept. Note that if you would like to do any calculation, like distances between two images, this would be done with the whole 512-dimensional embedding vectors.

```python
embedding_tensor = torch.cat((bruce_embeddings,
                              neil_embeddings,
                              pam_embeddings)).to(device='cpu')

pca = sklearn.decomposition.PCA(n_components=2)
pca_tensor = pca.fit_transform(embedding_tensor.detach().cpu().numpy())
```

```python
num = 15
categs = 3
colors = ['blue', 'orange', 'magenta']
labels = ['Bruce Lee', 'Neil Patrick Harris', 'Pam Grier']
markers = ['o', 'x', 's']
plt.figure(figsize=(8, 8))
for i in range(categs):
   plt.scatter(pca_tensor[i*num:(i+1)*num, 0],
               pca_tensor[i*num:(i+1)*num, 1],
               c=colors[i],
               marker=markers[i], label=labels[i])
plt.legend()
plt.title('PCA Representation of the Image Embeddings')
plt.xlabel('PC 1')
plt.ylabel('PC 2')
plt.show()
```

Great! The images corresponding to each individual are separated from each other in the embedding space!

If Neil Patrick Harris wants to unlock his phone with facial recognition, the phone takes the image from the camera, calculates the embedding and checks if it is close to the registered embeddings corresponding to Neil Patrick Harris.

---
# Section 2: Ethics – bias/discrimination due to pre-training datasets
Popular facial recognition datasets like VGGFace2 and CASIA-WebFace consist primarily of caucasian faces.
As a result, even state of the art facial recognition models [substantially underperform](https://openaccess.thecvf.com/content_ICCV_2019/papers/Wang_Racial_Faces_in_the_Wild_Reducing_Racial_Bias_by_Information_ICCV_2019_paper.pdf) when attempting to recognize faces of other races.

Given the implications that poor model performance can have in fields like security and criminal justice, it's very important to be aware of these limitations if you're going to be building facial recognition systems.

In this example we will work with a small subset from the [UTKFace](https://susanqq.github.io/UTKFace/) dataset with 49 pictures of black women and 49 picture of white women. We will use the same pretrained model as in Section 8 of Tutorial 1, see and discuss the consequences of the model being trained on an imbalanced dataset.

```python
# @title Video 2: Ethical aspects
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1Jo4y1Q7K3", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"vYilJV3PqUM", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

## Section 2.1: Download the Data

```python
# @title Run this cell to get the data and free ram by deleting old variables

!rm -rf cis_522_data/

if 'embedding_tesnor' in locals():
  del embedding_tensor
if 'pca_tesnor' in locals():
  del pca_tensor
if 'tensor_to_display' in locals():
  del tensor_to_display
if 'bruce_embeddings' in locals():
  del bruce_embeddings
if 'neil_embeddings' in locals():
  del neil_embeddings
if 'pam_embeddings' in locals():
  del pam_embeddings

# remove directories, if previously run.
!rm -rf __MACOSX/
!rm -rf face_sample/
!rm -rf face_sample2/
!git clone --quiet https://github.com/richardvogg/face_sample.git
!unzip -q face_sample/face_sample2.zip
```

## Section 2.2: Load, view and transform the data

```python
black_female_tensor, black_female_display = process_images('face_sample2/??_1_1_*.jpg', size=150)
white_female_tensor, white_female_display = process_images('face_sample2/??_1_0_*.jpg', size=150)
```

We can check the dimensions of these tensors and see that for each group we have images of size 150x150 and three channels (RGB) of 49 individuals.

**Note:** Originally, the size of images was 200x200, but due to RAM resources, we have reduce it. You can change it back, i.e., `size=200`.

```python
print(white_female_tensor.shape)
print(black_female_tensor.shape)
```

```python
# @title Visualize some example faces
tensor_to_display = torch.cat((white_female_display[:15],
                               black_female_display[:15]))

plt.figure(figsize=(12, 12))
plt.imshow(make_grid(tensor_to_display, nrow = 15).permute(1, 2, 0))
plt.axis('off')
plt.show()
```

## Section 2.3: Calculate embeddings

We use the same pretrained facial recognition network as in section 8 to calculate embeddings. If you have memory issues running this part, go to Edit > Notebook settings and check if GPU is selected as Hardware accelerator. If this does not help you can restart the notebook, go to Runtime -> Restart runtime.

```python
resnet.classify = False
black_female_embeddings = resnet(black_female_tensor.to(DEVICE))
white_female_embeddings = resnet(white_female_tensor.to(DEVICE))
```

We will use the embeddings to show that the model was trained on an imbalanced dataset. For this, we are going to calculate a distance matrix of all combinations of images, like in this small example with $n=3$ (in our case $n=98$).

Calculate the distance between each pair of image embeddings in our tensor and visualize all the distances. Remember that two embeddings are vectors and the distance between two vectors is the Euclidean distance.

```python
# @title Function to calculate pairwise distances
# @markdown `torch.cdist` is used
def calculate_pairwise_distances(embedding_tensor):
  """
  This function calculates the distance between each pair of image embeddings
  in a tensor using the `torch.cdist`.

  Parameters:
    embedding_tensor : torch.Tensor
      A num_images x embedding_dimension tensor

  Returns:
    distances : torch.Tensor
      A num_images x num_images tensor containing the pairwise distances between
      each to image embedding
  """

  distances = torch.cdist(embedding_tensor, embedding_tensor)

  return distances
```

```python
# @title Visualize the distances

embedding_tensor = torch.cat((black_female_embeddings,
                              white_female_embeddings)).to(device='cpu')

distances = calculate_pairwise_distances(embedding_tensor)

plt.figure(figsize=(8, 8))
plt.imshow(distances.detach().cpu().numpy())
plt.annotate('Black female', (2, -0.5), fontsize=20, va='bottom')
plt.annotate('White female', (52, -0.5), fontsize=20, va='bottom')
plt.annotate('Black female', (-0.5, 45), fontsize=20, rotation=90, ha='right')
plt.annotate('White female', (-0.5, 90), fontsize=20, rotation=90, ha='right')
cbar = plt.colorbar()
cbar.set_label('Distance', fontsize=16)
plt.axis('off')
plt.show()
```

## Exercise 2.1

What do you observe? The faces of which group are more similar to each other for the Face Detection algorithm?

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W2D2_ModernConvnets/solutions/W2D2_Tutorial2_Solution_866d0881.py)

## Exercise 2.2
- What does it mean in real life applications that the distance is smaller between the embeddings of one group?
- Can you come up with example situations/applications where this has a negative impact?
- What could you do to avoid these problems?

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W2D2_ModernConvnets/solutions/W2D2_Tutorial2_Solution_b92b2f40.py)

Lastly, to show the importance of the dataset which you use to pretrain your model, look at how much space white men and women take in different embeddings. FairFace is a dataset which is specifically created with completely balanced classes. The blue dots in all visualizations are white male and white female.

Adopted from [Kärkkäinen and Joo, 2019, arXiv](https://arxiv.org/abs/1908.04913)

---
# Summary

In this tutorial we have learned how to apply a modern convnet in real application such as facial recognition. However, as the state-of-the-art tools for facial recognition are trained mostly with caucasian faces, they fail or they perform much worst when they have to deal with faces from other races.

```python
# @title Video 3: Summary and Outlook
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1So4y1D7Ev", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"sjj0-7i6XfE", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

---
# Bonus : Within Sum of Squares

We can try to put this observation in numbers. For this we work with the embeddings.
We want to calculate the centroid of each group, which is the average of the 49 embeddings of the group. As each embedding vector has a dimension of 512, the centroid will also have this dimension.

Now we can calculate how far away the observations $x$ of each group $S_i$ are from the centroid $\mu_i$. This concept is known as Within Sum of Squares (WSS) from cluster analysis.

\begin{equation}
\text{WSS} = \sum_{x\in S_i} ||x - \mu_i||^2
\end{equation}

where $|| \cdot ||$ is the Euclidean norm.

The Within Sum of Squares (WSS) is a number which measures this variability of a group in the embedding space. If all embeddings of one group were very close to each other, the WSS would be very small. In our case we see that the WSS for the black females is much smaller than for the white females. This means that it is much harder for the model to distinguish two black females than to distinguish two white females. The WSS complements the observation from the distance matrix, where we observed overall smaller pairwise distances between black females.

```python
# @title Function to calculate WSS

def wss(group):
  """
  This function returns the sum of squared distances of the N vectors of a
  group tensor (N x K) to its centroid (1 x K).

  Args:
    group: A image_count x embedding_size tensor

  Returns:
    sum_sq: A 1x1 tensor with the sum of squared distances.

  Hints:
    - to calculate the centroid, torch.mean() will be of use.
    - We need the mean of the N=49 observations. If our input tensor is of size
      N x K, we expect the centroid to be of dimensions 1 x K.
      Use the axis argument within torch.mean
    """

  centroid = torch.mean(group, axis=0)
  distance = torch.linalg.norm(group - centroid.view(1, -1), axis=1)
  sum_sq = torch.sum(distance**2)
  return sum_sq
```

```python
# @markdown Let's calculate the WSS for the two groups of our example.

print(f"Black female embedding WSS: {np.round(wss(black_female_embeddings).item(), 2)}")
print(f"White female embedding WSS: {np.round(wss(white_female_embeddings).item(), 2)}")
```


# Molecular Dynamics: Lab 1

In part based on [Fortran code from Furio Ercolessi](http://www.fisica.uniud.it/~ercolessi/md/f90/).

```python
from IPython.core.display import HTML
css_file = 'https://raw.githubusercontent.com/ngcm/training-public/master/ipython_notebook_styles/ngcmstyle.css'
HTML(url=css_file)
```

<link href='http://fonts.googleapis.com/css?family=Open+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 1000px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1200px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Open Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:900px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Arvo', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Arvo', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Arvo', serif;
    margin-top:16px;
	font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Arvo', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Arvo', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
        font-family: "PT Mono";
        font-size: 90%;
}

</style>

## Basics of Molecular Dynamics

```python
%matplotlib inline
import numpy
from matplotlib import pyplot
from mpl_toolkits.mplot3d.axes3d import Axes3D
from matplotlib import rcParams
rcParams['font.family'] = 'serif'
rcParams['font.size'] = 16
rcParams['figure.figsize'] = (12,6)
```

The basic idea of molecular dynamics is to follow the location of the particles in time using Newton's laws. Each particle has a location $\vec{X}_i$ which obeys

$$
\begin{equation}
  m_i \frac{d^2}{d t^2} \vec{X}_i = m_i \vec{A}_i = -\nabla V \left( \vec{X}_1, \dots, \vec{X}_N \right).
\end{equation}
$$

Most of the time we will work in computational coordinates, rescaled by a reference length $L$, so that the particles obey

$$
\begin{equation}
  \frac{d^2}{d t^2} \vec{x}_i = \vec{a}_i = -\nabla V \left( \vec{x}_1, \dots, \vec{x}_N \right).
\end{equation}
$$

This gives us a computational domain, once translated to the centre-of-mass, of $\vec{x} \in [-0.5, 0.5]^3$.

The simplest possibility is that the interaction potential $V$ is the sum of pairwise interactions,

$$
\begin{equation}
  V \left( \vec{x}_1, \dots, \vec{x}_N \right) = \sum_i \sum_{j>i} \phi \left( L \left| \vec{x}_i - \vec{x}_j \right| \right).
\end{equation}
$$

The most commonly used pairwise interaction potential is the *Lennard-Jones* potential

$$
\begin{equation}
  \phi(r) = 4 \varepsilon \left[ \left( \frac{\sigma}{r} \right)^{12} - \left( \frac{\sigma}{r} \right)^{6} \right].
\end{equation}
$$

To stop this potential extending for infinite distance a cutoff is used; $\phi(r) \to \phi(r) - \phi(R_c)$, where $R_c$ is a constant (here $2.5$ when $\sigma = 1 = \varepsilon = M_i$).

As, in this case,

$$
\begin{equation}
  \frac{d}{dr} \phi(r) = 24 \left[ 2 \left( \frac{1}{r} \right)^{14} - \left( \frac{1}{r} \right)^{8} \right],
\end{equation}
$$

we have

$$
\begin{equation}
  \vec{a}_i = -\left. \nabla V \right|_{i} = \sum_{j>i}  (\vec{x}_i - \vec{x}_j) \, \frac{d}{dr} \phi(r) - \sum_{j<i}  (\vec{x}_j - \vec{x}_i) \, \frac{d}{dr} \phi(r).
\end{equation}
$$

Note that Newton's third law means that the force on particle $i$ from particle $j$ must be exactly opposite to the force on particle $j$ from particle $i$. So the algorithm can be implemented by setting $\vec{a}_i = 0$ and, for each particle $i$, and for all $j > i$, setting *both*

1. $\vec{a}_i = \vec{a}_i + (\vec{x}_i - \vec{x}_j) \, \frac{d}{dr} \phi(r)$,
2. $\vec{a}_j = \vec{a}_j - (\vec{x}_i - \vec{x}_j) \, \frac{d}{dr} \phi(r)$.

Implement an algorithm to calculate the acceleration given the positions.

```python

```

### Time evolution

The problem with time integrating is that most numerical algorithms do not conserve the total energy of the system, which can be be crucial. Certain algorithms, such as the [velocity Verlet](http://en.wikipedia.org/wiki/Verlet_integration#Velocity_Verlet) algorithm, do conserve total energy.

#### Velocity Verlet

Given initial positions $\vec{x}$ and velocities $\vec{v}$ of each particle, the algorithm can be written as

1. $x^{n+1} = \vec{x}^n + \Delta t \, \vec{v}^n + \tfrac{1}{2} \Delta t^2 \, \vec{a}^n$
2. $\vec{v}^{*} = \vec{v}^n + \tfrac{1}{2} \Delta t \, \vec{a}^n$;
3. Derive $\vec{a}^{n+1}$ from the interaction potential using $\vec{x}^{n+1}$;
4. $\vec{v}^{n+1} = \vec{v}^{*} + \tfrac{1}{2} \Delta t \vec{a}^{n+1}$.

Implement this algorithm.

```python

```

### Periodic boundaries

If we just evolved the particles, they could fly off to infinity. Instead it is typical to work inside a box with periodic boundaries, so that there is an "infinite lattice" of particles.

Implement a function that resets the position of particles that leave the given domain; the velocities and particles should remain unchanged.

```python

```

### What to compute

The fundamental quantity is the location of the particles. However, the thing to analyze will be average properties of the particles; their mean temperature, pressure and energies.

Implement a function, given the particle locations $\vec{x}$ to compute the temperature, where

$$
\begin{align}
%  E_{\text{potential},\ i} &= \sum_{j /ne i} \phi \left( L \left| \vec{x}_i - \vec{x}_j \right| \right), \\
  E_{\text{kinetic},\ i} &= \tfrac{1}{2} L^2 \left| \vec{v}_i \right|^2, \\
  T & = \frac{2}{3 N} \sum_{i=1}^N E_{\text{kinetic},\ i}.
\end{align}
$$

```python

```

### Run in a box

Fix the domain to be $\vec{x} \in [0, 6.1984]^3$. Fix the timestep to be $0.005$ and take 100 timesteps, given the initial data prescribed in `input.dat` (see `input.dat` on Blackboard).

```python

```

Then plot some basic quantities, such as

1. The temperature as a function of time;
2. The trajectory of two particles from the middle of the domain (eg particles 100 and 101) as a function of time (using `plot3D`);
3. The trajectories of all the particles.

```python

```

FEEG6016 Simulation and Modelling/03-Molecular-Dynamics-Lab-1.ipynb


```python
# Erasmus+ ICCT project (2018-1-SI01-KA203-047081)

# Toggle cell visibility

from IPython.display import HTML
tag = HTML('''
Toggle cell visibility <a href="javascript:code_toggle()">here</a>.''')
display(tag)

# Hide the code completely

# from IPython.display import HTML
# tag = HTML('''<style>
# div.input {
#     display:none;
# }
# </style>''')
# display(tag)
```

Toggle cell visibility here.

```python
%matplotlib notebook
import scipy.signal as signal
import matplotlib.pyplot as plt
from ipywidgets import widgets
from ipywidgets import interact
import numpy as np
import sympy as sym
```

## Controller PID - risposta nel tempo

Un algoritmo di controllo proporzionale-integrale-derivativo (PID) è di gran lunga l'algoritmo di controllo più comune. La sua funzione di trasferimento è

\begin{equation}
    P(s)=K_p \cdot \left( 1 + \frac{1}{T_i s} + T_d s \right).
\end{equation}

È costituito dalla somma dei canali proporzionale, integrale e derivativo. Non tutti devono essere presenti, vengono utilizzati anche algoritmi di controllo P, PI o PD. In questo esempio viene mostrata la risposta di un controller P, PI, PD o PID al gradino unitario, all'impulso unitario, alla rampa unitaria o ad un ingresso sinusoidale.

---

### Come usare questo notebook?
1. Alterna tra *funzione gradino*, *funzione impulso*, *funzione rampa* e *funzione seno* per selezionare il segnale di ingresso.
2. Clicca il pulsante *P*, *PI*, *PD* o *PID* per selezionare tra proporzionale, proporzionale-integrale, proporzionale-derivativo o proporzionale-integrale-derivativo.
3. Sposta gli sliders per modificare i valori dei coefficienti proporzionale ($K_p$), integrale ($T_i$) e derivativo ($T_d$).
4. Sposta lo slider $t_{max}$ per modificare il valore massimo del tempo sull'asse x.

```python
a = 0.1

# make figure
fig = plt.figure(figsize=(9.8, 5),num='Controllore PID')
# add axes
ax = fig.add_subplot(111)
ax.grid(which='both', axis='both', color='lightgray')
ax.set_title('Risposta')
# plot step function and responses (initalisation)
input_plot, = ax.plot([],[],'C0', linewidth=1,label='input')
response_plot, = ax.plot([],[], 'C1', linewidth=2,label='output')
ax.axhline(linewidth=.5, color='k')
ax.axvline(linewidth=.5, color='k')
ax.legend()

ax.set_xlabel('$t$ [s]')
ax.set_ylabel('input, output')
plt.show()

P, I, D, s = sym.symbols('P, I, D, s')

input_type = 'gradino' #input function
Time_span = 10 # max time on x-axis plot

#initialize global variables
KP = 1.
TI = 1.
TD = 1.
num = []
den = []

def update_plot():
    global num, den, input_type, Time_span
    num_temp = [float(i.subs(P,KP).subs(I,TI).subs(D,TD)) for i in num]
    den_temp = [float(i.subs(P,KP).subs(I,TI).subs(D,TD)) for i in den]

    system = signal.TransferFunction(num_temp, den_temp)

    #time, response = signal.step(system) #only for setting time borders (for nicer plot. could also calculate dominant frequency)
    #time = np.linspace(0,time[-1],1000)
    time = np.linspace(0, Time_span, 600)

    if input_type == 'gradino':
        u = np.ones_like(time)
        u = np.concatenate((np.array([0]),u))
        time, response = signal.step(system, T=time)
        time = np.concatenate((np.array([0]), time))
        response = np.concatenate((np.array([0]), response))
    elif input_type == 'impulso':
        u = np.zeros_like(time)
        u = np.concatenate((np.array([10]), u))
        time, response = signal.impulse(system, T=time)
        time = np.concatenate((np.array([0]), time))
        response = np.concatenate((np.array([0]), response))
    elif input_type == 'sinusoide':
        u = np.sin(time*2*np.pi)
        time, response, _ = signal.lsim(system, U=u, T=time)
    elif input_type == 'rampa':
        u = time
        time, response, _ = signal.lsim(system, U=u, T=time)
    else:
        raise Exception("Errore nel programma. Fai ripartire la simulazione.")

    response_plot.set_data(time, response)
    input_plot.set_data(time, u)
    ax.set_ylim([min([np.min(u), min(response),-.1]),min(100,max([max(response)*1.05, 1, 1.05*np.max(u[1:])]))])
    ax.set_xlim([-0.1,max(time)])
    plt.show()

def transfer_func(controller_type):
    global num, den
    proportional = P
    integral = P/(I*s)
    differential = P*D*s/(a*D*s+1)
    if controller_type =='P':
        controller_func = proportional
        Kp_widget.disabled=False
        Ti_widget.disabled=True
        Td_widget.disabled=True
    elif controller_type =='PI':
        controller_func = proportional+integral
        Kp_widget.disabled=False
        Ti_widget.disabled=False
        Td_widget.disabled=True
    elif controller_type == 'PD':
        controller_func = proportional+differential
        Kp_widget.disabled=False
        Ti_widget.disabled=True
        Td_widget.disabled=False
    else:
        controller_func = proportional+integral+differential
        Kp_widget.disabled=False
        Ti_widget.disabled=False
        Td_widget.disabled=False
    system_func = controller_func

    num = [sym.fraction(system_func.factor())[0].expand().coeff(s, i) for i in reversed(range(1+sym.degree(sym.fraction(system_func.factor())[0], gen=s)))]
    den = [sym.fraction(system_func.factor())[1].expand().coeff(s, i) for i in reversed(range(1+sym.degree(sym.fraction(system_func.factor())[1], gen=s)))]
    update_plot()

def func(Kp, Ti, Td, time_span):
    global KP, TI, TD, Time_span
    KP = Kp
    TI = Ti
    TD = Td
    Time_span = time_span
    update_plot()

style = {'description_width': 'initial'}

def buttons_controller_clicked(event):
    controller = buttons_controller.options[buttons_controller.index]
    transfer_func(controller)
buttons_controller = widgets.ToggleButtons(
    options=['P', 'PI', 'PD', 'PID'],
    description='Seleziona il tipo di controller:',
    disabled=False,
    style=style)
buttons_controller.observe(buttons_controller_clicked)

def buttons_input_clicked(event):
    global input_type
    input_type = buttons_input.options[buttons_input.index]
    update_plot()
buttons_input = widgets.ToggleButtons(
    options=['gradino','impulso', 'rampa', 'sinusoide'],
    description='Seleziona l\'input:',
    disabled=False,
    style=style)
buttons_input.observe(buttons_input_clicked)

Kp_widget = widgets.IntSlider(value=20,min=1,max=100,step=1,description=r'\(K_p \)',
    disabled=False,continuous_update=True,orientation='horizontal',readout=True,readout_format='.1d')
Ti_widget = widgets.FloatSlider(value=.1,min=0.001,max=3.,step=0.001,description=r'\(T_{i} \)',
    disabled=False,continuous_update=True,orientation='horizontal',readout=True,readout_format='.3f')
Td_widget = widgets.FloatSlider(value=.1,min=0.001,max=3.,step=0.001,description=r'\(T_{d} \)',
    disabled=False,continuous_update=True,orientation='horizontal',readout=True,readout_format='.3f')

time_span_widget = widgets.FloatSlider(value=10.,min=.5,max=50.,step=0.1,description=r'\(t_{max} \)',
    disabled=False,continuous_update=True,orientation='horizontal',readout=True,readout_format='.1f')

transfer_func('P')

display(buttons_input)
display(buttons_controller)

interact(func, Kp=Kp_widget, Ti=Ti_widget, Td=Td_widget, time_span=time_span_widget);
```

    <IPython.core.display.Javascript object>

    ToggleButtons(description="Seleziona l'input:", options=('gradino', 'impulso', 'rampa', 'sinusoide'), style=To…

    ToggleButtons(description='Seleziona il tipo di controller:', options=('P', 'PI', 'PD', 'PID'), style=ToggleBu…

    interactive(children=(IntSlider(value=20, description='\\(K_p \\)', min=1, readout_format='.1d'), FloatSlider(…


# PyTorch tutorial の内容について(2)

- date: 2020-0722
- author: 浅川伸一

[https://github.com/pytorch/tutorials/tree/master/beginner_source/nlp](https://github.com/pytorch/tutorials/tree/master/beginner_source/nlp) を見ると
PyTorch で 自然言語処理を行う場合のチュートリアルは以下とおりである

# Deep Learning for NLP with Pytorch

1. [pytorch_tutorial.py](https://github.com/pytorch/tutorials/blob/master/beginner_source/nlp/pytorch_tutorial.py):
	[PyTorch 入門 Introduction to PyTorch](https://pytorch.org/tutorials/beginner/nlp/pytorch_tutorial.html)

2. [deep_learning_tutorial.py](https://github.com/pytorch/tutorials/blob/master/beginner_source/nlp/deep_learning_tutorial.py):
	[PyTorch による深層学習 Deep Learning with PyTorch](https://pytorch.org/tutorials/beginner/nlp/deep_learning_tutorial.html)

3. [word_embeddings_tutorial.py](https://github.com/pytorch/tutorials/blob/master/beginner_source/nlp/word_embeddings_tutorial.py):
	[単語埋め込み:語彙的意味の符号化 Word Embeddings: Encoding Lexical Semantics](https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html)

4. [sequence_models_tutorial.py]((https://github.com/pytorch/tutorials/blob/master/beginner_source/nlp/sequence_models_tutorial.py):
	[系列モデルと LSTM Sequence Models and Long-Short Term Memory Networks](https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html)

5. [advanced_tutorial.py]((https://github.com/pytorch/tutorials/blob/master/beginner_source/nlp/advanced_tutorial.py):
	[動的意思決定と双方向 LSTM 条件付き確率場 Advanced: Making Dynamic Decisions and the Bi-LSTM CRF](https://pytorch.org/tutorials/beginner/nlp/advanced_tutorial.html)

以下では，このうちの 2 について解説している。

```python
%matplotlib inline
```

PyTorch によるディープラーニング
********************************

## ディープラーニングのブルディンクブロック: アフィン変換，非線形性，目的関数

ディープラーニングは、線形性と非線形性を巧妙な方法で組み合わせることで構成されています。
非線形性を導入することで、強力なモデルを作ることができます。
このセクションでは、これらのコアコンポーネントで遊び、目的関数を構成し、モデルがどのように訓練されるかを見ていきます。

### アフィン写像

ディープラーニングの中核を成すものの一つがアフィン写像であり、次式の関数 $f(x)$ で表します。

\begin{align}
f(x) = Ax + b
\end{align}

行列 $A$ とベクトル $x, b$ を求めます。ここで学習するパラメータは $A$ と $b$ です。
$b$ はバイアスとして使用されます。

PyTorch や他のほとんどのディープラーニングフレームワークでは 従来の線形代数とは少し違うことをしています。
それは、入力の列の代わりに 行を写像します。
つまり 以下の出力の $i$'番目の行は $A$ の下の入力の $i$'th 行にバイアス項を加えたものです。
下の例を見てください。

```python
# Author: Robert Guthrie

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

torch.manual_seed(1)
```

    <torch._C.Generator at 0x7fa696ff0120>

```python
lin = nn.Linear(5, 3)  # 5 次のベクトルから 3 次のベクトルへの線形写像 ``nn.Linear`` #  maps from R^5 to R^3, parameters A, b
# データは 2 行 5 列 すなわち 5 次元のベクトルからなるデータが 2 つ # data is 2x5.  A maps from 5 to 3... can we map "data" under A?
data = torch.randn(2, 5)
print(lin(data))  # 実際には 行列積
```

    tensor([[ 0.1755, -0.3268, -0.5069],
            [-0.6602,  0.2260,  0.1089]], grad_fn=<AddmmBackward>)

### 非線形性

次の事実に注意してください。2 つのアフィン写像 $f(x) = Ax + b$ と $g(x) = Cx + d$ があるとします。
$f(g(x))$ は何でしょうか？

\begin{align}f(g(x)) = A(Cx + d) + b = ACx + (Ad + b)\end{align}

$A$ と $C$ は行列であり，$Ad + b$ はベクトルです。
アフィン写像の合成はアフィン写像であることがわかります。

このことから，ニューラルネットワークをアフィン写像の幾重にも重ねた，ディープニューラルネットワークモデル
を作成する場合，
単一のアフィン写像による成績以上の性能は得られないことを意味します。

逆に，アフィン写像の間に非線形性を導入するとにより，より強力なモデルを構築することができます。

いくつかのコアとなる非線形性があります。
$\tanh(x)$, $\sigma(x)$ (シグモイド関数), $\text{ReLU}(x)$ が一般的です。
他にも数多非線形変換がありうるのに，なぜこれらが用いられるのか疑問に思うかも知れません。
理由は勾配計算が容易だからです。勾配の計算はニューラルネットワークの学習には必須です。

例えば

\begin{align}\frac{d\sigma}{dx} = \sigma(x)(1 - \sigma(x))\end{align}

昔からニューラルネットワークをご存知の方は，シグモイド関数 $\sigma(x)$ がデフォルトであると思われていることでしょう。ですが，最近ではシグモイド関数は用いられません。理由は勾配消失問題に対処するのが難しいからです。
このため，現在では $\tanh(x)$ や ReLU (整流線型ユニット) が用いられます。

```python
# pytorch では ほとんどの非線形活性化関数は torch.functional で定義されています (F としてインポートしています)
# 非線形型は一般的にアフィン写像のような パラメータを持たちません。
# すなわち 学習中に更新されるパラメータを持ちません。
data = torch.randn(2, 2)
print(data)
print(F.relu(data))
```

    tensor([[-0.5404, -2.2102],
            [ 2.1130, -0.0040]])
    tensor([[0.0000, 0.0000],
            [2.1130, 0.0000]])

### ソフトマックスと確率

関数 $\text{Softmax}(x)$ も単なる非線形活性化関数の一つです。
ですが，通常ネットワーク内で最終層で行われる操作であるという点で特殊です。
ソフトマックス関数 は実数のベクトルを引数として取り，
各ニューロン (ユニット) の確率分布を返すからです。
ソフトマックス関数の定義は次のようになります。
$x$ を実ベクトルとする (正でも負でも何でもよく，制約はない)。
$text{Softmax}(x)$ の i 番目の成分は以下になります:

\begin{align}\frac{\exp(x_i)}{\sum_j \exp(x_j)}\end{align}

出力値が確率分布をなす (各要素は非負で，かつ，全要素の総和が 1 ) ことは，明らかに重要です。
入力に要素ごとの 指数化演算子を適用してすべてを非負にして，、正規化定数で除算すると考えることもできます。

---

### 訳注:
Python で頻用される科学計算ライブラリ `scipy.special` には `scipy.special.logsumexp()` という関数が用意されている。
上記ソフトマックス関数を計算する際に，分母 $\sum\exp(x_j)$ は 指数変換した各値を総和する (`sumexp` する) ことを意味し，
確率密度関数の対数尤度を計算する場合には，この `sumexp` を対数変換するので `logsumexp` と命名された。
`logsumexp()` 関数を用いてソフトマックス関数を定義すれば以下のようになる:

\begin{align}\log\text{softmax}(x_i) = x_i - \text{logsumexp(x)}\end{align}

すなわちソフトマックス関数の対数を計算するときには，各値から `logsumexp(x)` を減じるだけになるので，
計算が極端にかんたんになる。
筆者の知る限り，Fortran 時代から logsumexp ルーチンは存在する。もはやその起源や命名者を特定できないように見受けられる。
実際，以下のコードは同じ値を与える。

```python
import numpy as np
from scipy.special import softmax, logsumexp

x1, x2, x3 = 1., 2., 3.
print(np.log(softmax([x1, x2, x3]))[0])
print( x1 - logsumexp([x1, x2, x3]))
```

```python
# ソフトマックス関数 Softmax は torch.nn.functional でも定義されている
data = torch.randn(5)
print(data)
print(F.softmax(data, dim=0))
print(F.softmax(data, dim=0).sum())  # 確率分布であるから総和は 1 となる
print(F.log_softmax(data, dim=0))  # ソフトマックス関数の対数変換
```

    tensor([ 1.3800, -1.3505,  0.3455,  0.5046,  1.8213])
    tensor([0.2948, 0.0192, 0.1048, 0.1228, 0.4584])
    tensor(1.)
    tensor([-1.2214, -3.9519, -2.2560, -2.0969, -0.7801])

### 目的関数

目的関数は学習に用いられ，ネットワークがその値を最小化されるよう訓練される関数です
(*損失関数* または *コスト関数* とも呼ばれます)。
これは，最初に 訓練インスタンス を選択し，ニューラルネットワークを実行して 出力関数の損失値を計算することで進行します。
その後，モデルのパラメータは，損失関数の微分を取ることで更新されます。
直感的には，モデルがその答えに完全に自信を持っていて，その答えが誤っている場合，損失関数の値は大きくなります。
反対に，その答えに非常に自信を持っていて，かつ，その答えが正しければ、損失値は小さくなります。

学習事例の損失関数を最小化するという考え方は，ネットワークの一般化に関与します。
訓練データセット，とテストデータセット，を分割するのは，訓練環境での未見のデータでの損失が小さいことを期待するためです。
損失関数の例としては、*負の対数尤度損失* があります。
これは多クラス分類の非常に一般的な目的関数です。
教師付き多クラス分類では，正しい出力の負の対数確率を最小にするようにネットワークを訓練することを意味します (正しい出力に対する対数確率を最大にする)。

(訳注: 目的関数 ojbective function は，損失関数 loss function は，ほぼ同義で使われます。
Goodfellow の教科書では objective, loss, goal, error function などは同義としてで扱う書かれています。)

## 最適化と訓練

So what we can compute a loss function for an instance?
What do we do with that? We saw earlier that Tensors know how to compute gradients with respect to the things that were used to compute it.
Well, since our loss is an Tensor, we can compute gradients with respect to all of the parameters used to compute it!
Then we can perform standard gradient updates.
Let $\theta$ be our parameters, $L(\theta)$ the loss function, and $\eta$ a positive learning rate. Then:

ある事例 (データ) の損失関数を実際どう計算するのでしょうか？
先ほど，PyTorch のテンソルは計算に使われたすべてについての勾配計算に関する情報を知っていると説明しました。
損失関数は PyTorch テンソル なので，計算に使われたすべてのパラメータに関する勾配を計算することができます。
標準的な勾配降下法によりパラメータ更新が可能です。
$\theta$ をパラメータとし，損失関数を $L(\theta)$，正の学習率を $\eta$ とすれば次式を得ます:

\begin{align}\theta^{(t+1)} = \theta^{(t)} - \eta \nabla_\theta L(\theta)\end{align}

このバニラ勾配降下法 による更新だけでなく，それ以上のことをしようとする試みには，膨大なアルゴリズムの研究が集積されていてり，活発な研究が行われています。
(訳注: 何の飾り付けのない素のアルゴリズムという意味でしばしば `バニラ` vanilla という形容詞を用います。
何のフレーバーもつかない素のアイスクリームをバニラアイスクリームと呼ぶように，
例えば，教科書に記載されているような素の誤差逆伝播法のことを `バニラバックプロパゲーション` などと呼ぶ習慣があります)
バニラではない，多くの発展的なアルゴリズムでは，訓練時に起こっていることに基づいて学習率を変化させます。
本当に興味がない限り，これらのアルゴリズムが具体的に何をしているのかを気にする必要はありません。
Torch では `torch.optim` パッケージで多くのアルゴリズムを提供しています。
それらアルゴリズムはすべて完全に透過的です。すなわち，
最も単純な勾配更新法を使うこととは，より複雑なアルゴリズムと同じく扱うことができます。
ネットワークのパフォーマンスを最適化するには，さまざまな更新アルゴリズムや更新アルゴリズムのパラメータ (初期学習率の違いなど) を試すことが重要です。
多くの場合， `バニラ SGD` を `Adam` や `RMSProp` のような最適化手法に置き換えるだけで，パフォーマンスが顕著に向上します。

（訳注: バニラ SGD ではハイパーパラメータ $\eta$ が固定です。
一方 `RMSprop` 以降のアルゴリズムでは，各パラメータごとに異なる $\eta$ を採用します。
また，2 次微分 **ヘッセ行列** を用いるニュートン法の近似として，ヘッセ行列を近似式で置き換えた
準ニュートン法のニューラルネットワーク的実装が `Adam` です。ニュートン法からの類推で明らかなように
Adam では学習係数 $\eta$ をヘシアンに応じて調整します。
）

# PyTorch によるネットワークコンポネントの作成

NLP の話に移る前に，アフィン写像と非線形性だけを使って PyTorch でネットワークを構築してみましょう。
加えて，PyTorch の負の対数尤度を使って損失関数を計算し，誤差逆伝播法 (バックプロパゲーション) でパラメータを更新する方法も見てみましょう。

すべてのネットワークコンポーネントは `nn.Module` を継承し，`forward()` メソッドをオーバーライドする必要があります。
ボイラープレート (訳注: いわゆるテンプレート版あるいはプロトタイプ，オブジェクト指向でのクラス定義的な意味合いです) に関する限りでは、これくらいです。
`nn.Module` を継承することで コンポーネントに関数 (機能) を提供します。
例えば 訓練可能なパラメータを管理したり  ``.to(device)`` メソッドを使って CPU と GPU を入れ替えたりすることができます。

疎な bag-of-words 表現を受け取り，2 つのラベルの確率分布を出力するネットワークの注釈付きの例を書いてみましょう。
「英語」と「スペイン語」の 2 つのラベルの確率を出力するネットワークの例を書いてみましょう。
このモデルは単なるロジスティック回帰です。

### 例: Bog-of-Words によるロジスティック回帰分類器

モデルは 疎な BoW 表現をラベルの対数確率にマッピングするものです。
語彙内の各単語にインデックスを割り当てます。例えば，全語彙が "hello" と "world" の 2 つの単語で成り立っているとします。
それぞれ 0 と 1 のインデックスを持つとします。
文 "hello hello hello hello hello hello " の BoW ベクトルは次のようになります。

\begin{align}\left[ 4, 0 \right]\end{align}

同様に，"hello, workd world hello" の BoW ベクトルは次のようになります:

\begin{align}\left[ 2, 2 \right]\end{align}

一般には，次のような表記となります:

\begin{align}\left[ \text{Count}(\text{hello}), \text{Count}(\text{world}) \right]\end{align}

この BOW ベクトルを $x$ と表記します。ネットワークの出力は以下のとおり:

\begin{align}\log \text{Softmax}(Ax + b)\end{align}

すなわち入力信号を アフィン写像に通して，その値に対して log softmax を行います。

```python
data = [("me gusta comer en la cafeteria".split(), "SPANISH"),
        ("Give it to me".split(), "ENGLISH"),
        ("No creo que sea una buena idea".split(), "SPANISH"),
        ("No it is not a good idea to get lost at sea".split(), "ENGLISH")]

test_data = [("Yo creo que si".split(), "SPANISH"),
             ("it is lost on me".split(), "ENGLISH")]

# word_to_ix は語彙中の各単語をとユニークな整数に変換します。
word_to_ix = {}
for sent, _ in data + test_data:
    for word in sent:
        if word not in word_to_ix:
            word_to_ix[word] = len(word_to_ix)
print(word_to_ix)

VOCAB_SIZE = len(word_to_ix)
NUM_LABELS = 2

class BoWClassifier(nn.Module):  # inheriting from nn.Module!

    def __init__(self, num_labels, vocab_size):
        # nn.Module の init 関数を呼び出します。文法で混乱しないように。
        # nn.Module での初期化が行われます
        super(BoWClassifier, self).__init__()

        # 必要なパラメータの定義。アフィン写像のパラメータ A と b
        # Torch には nn.Linear() が定義されており，これがアフィン写像を提供します
        # 入力次元が vocab_size であり，出力が num_labels である理由を理解してください。
        self.linear = nn.Linear(vocab_size, num_labels)

        # 非線形関数 log softmax にはパラメータがないことに注意

    def forward(self, bow_vec):
        # 入力を線形層に通し，さらに log_softmax に通します。
        # 多くの非線形関数ややその他の関数は torch.nn.functional にあります
        return F.log_softmax(self.linear(bow_vec), dim=1)

def make_bow_vector(sentence, word_to_ix):
    vec = torch.zeros(len(word_to_ix))
    for word in sentence:
        vec[word_to_ix[word]] += 1
    return vec.view(1, -1)

def make_target(label, label_to_ix):
    return torch.LongTensor([label_to_ix[label]])

model = BoWClassifier(NUM_LABELS, VOCAB_SIZE)

# モデルはパラメータを知っています。 以下の最初の出力は A，2番目の出力は bです。
# モジュールの __init__ 関数でクラス変数にコンポーネントを代入するときはいつも
# self.linear = nn.Linear(...) 行で行われていました。
# PyTorch 開発者による Python 魔法により，このモジュール (BoWClassifier) は
# nn.Linear のパラメータの知識を保持しています
for param in model.parameters():
    print(param)

# このモデルを走らせるためには，BoW ベクトルを通します
# ここでは訓練する必要がないので，以下では torch.no_grad() で warp します
with torch.no_grad():
    sample = data[0]
    bow_vector = make_bow_vector(sample[0], word_to_ix)
    log_probs = model(bow_vector)
    print(log_probs)
```

    {'me': 0, 'gusta': 1, 'comer': 2, 'en': 3, 'la': 4, 'cafeteria': 5, 'Give': 6, 'it': 7, 'to': 8, 'No': 9, 'creo': 10, 'que': 11, 'sea': 12, 'una': 13, 'buena': 14, 'idea': 15, 'is': 16, 'not': 17, 'a': 18, 'good': 19, 'get': 20, 'lost': 21, 'at': 22, 'Yo': 23, 'si': 24, 'on': 25}
    Parameter containing:
    tensor([[ 0.1194,  0.0609, -0.1268,  0.1274,  0.1191,  0.1739, -0.1099, -0.0323,
             -0.0038,  0.0286, -0.1488, -0.1392,  0.1067, -0.0460,  0.0958,  0.0112,
              0.0644,  0.0431,  0.0713,  0.0972, -0.1816,  0.0987, -0.1379, -0.1480,
              0.0119, -0.0334],
            [ 0.1152, -0.1136, -0.1743,  0.1427, -0.0291,  0.1103,  0.0630, -0.1471,
              0.0394,  0.0471, -0.1313, -0.0931,  0.0669,  0.0351, -0.0834, -0.0594,
              0.1796, -0.0363,  0.1106,  0.0849, -0.1268, -0.1668,  0.1882,  0.0102,
              0.1344,  0.0406]], requires_grad=True)
    Parameter containing:
    tensor([0.0631, 0.1465], requires_grad=True)
    tensor([[-0.5378, -0.8771]])

上の値のうち、どれが ENGLISH の対数確率に対応し，どれが SPANISH の対数確率に対応するのでしょうか？
これまで定義したことはありませんでしたが，もし学習したいのであれば定義する必要があります。

```python
label_to_ix = {"SPANISH": 0, "ENGLISH": 1}
```

以下は訓練です。

訓練には，インスタンスを通過させて対数確率を取得し，損失関数を計算し，損失関数の勾配を計算し，勾配ステップでパラメータを更新します。
損失関数は Torch の nn パッケージ内で提供されています。
`nn.NLLLoss()` は求める負の対数尤度損失です。
 (訳注: NLL とは Negative Log Loss 関数，負の対数尤度を意味します)
また 最適化関数も torch.optim で定義されています。ここでは SGD を使用します。

NLL 損失への *input* は対数確率のベクトルとターゲットラベルであることに注意してください。
この関数は 対数確率を計算してくれません。
これが，このネットワークの最後の層が log softmax である理由です。
損失関数 nn.CrossEntropyLoss() は NLLLoss() と同じですが、log softmax を計算してくれます。

```python
# 訓練前にテストデータを評価します。訓練前後での変化を見るためです
with torch.no_grad():
    for instance, label in test_data:
        bow_vec = make_bow_vector(instance, word_to_ix)
        log_probs = model(bow_vec)
        print(log_probs)

# "creo" に対応する行列の列を印字
print(next(model.parameters())[:, word_to_ix["creo"]])

loss_function = nn.NLLLoss()
optimizer = optim.SGD(model.parameters(), lr=0.1)

# 学習データを何度か通したいと思うでしょう.
# 100 は実際のデータセットよりもはるかに大きいです。
# ですが 実際のデータセットは 2 つ以上のインスタンスを持っています。
# 通常 5 から 30 エポックの間のどこかが妥当です。
for epoch in range(100):
    for instance, label in data:
        # ステップ 1. PyTorch には勾配が蓄積されていることを覚えておいてください
        # 各インスタンスの前にその勾配を ゼロ消去する必要があります
        model.zero_grad()

        # Step 2. Make our BOW vector and also we must wrap the target in a
        # Tensor as an integer. For example, if the target is SPANISH, then
        # we wrap the integer 0. The loss function then knows that the 0th
        # element of the log probabilities is the log probability
        # corresponding to SPANISH

        bow_vec = make_bow_vector(instance, word_to_ix)
        target = make_target(label, label_to_ix)

        # Step 3. Run our forward pass.
        log_probs = model(bow_vec)

        # Step 4. Compute the loss, gradients, and update the parameters by
        # calling optimizer.step()
        loss = loss_function(log_probs, target)
        loss.backward()
        optimizer.step()

with torch.no_grad():
    for instance, label in test_data:
        bow_vec = make_bow_vector(instance, word_to_ix)
        log_probs = model(bow_vec)
        print(log_probs)

# Index corresponding to Spanish goes up, English goes down!
print(next(model.parameters())[:, word_to_ix["creo"]])
```

    tensor([[-0.9297, -0.5020]])
    tensor([[-0.6388, -0.7506]])
    tensor([-0.1488, -0.1313], grad_fn=<SelectBackward>)
    tensor([[-0.2093, -1.6669]])
    tensor([[-2.5330, -0.0828]])
    tensor([ 0.2803, -0.5605], grad_fn=<SelectBackward>)

正解が出ました!
最初の例では，スペイン語の対数確率がはるかに高いことがわかります。
英語では，テストデータの 2 番目の方が当然のように英語の方が高いです。

これで PyTorch コンポーネントを作って，そこにいくつかのデータを渡して，勾配更新を行う方法がわかりました。
これで ディープ NLP が提供するものをさらに深く掘り下げる準備が整いました。


# Chapter 2

## Notes

### Use of "frequentist"

In this chapter, "frequentist" is often equated with hypothesis testing based on P-values.
However, that's a bit misleading.
Fundamentally, there are 2 branches statistics (frequentist and Bayesian), because we have two things we can condition on, the model or the data.
For a nice explanation of this grounded in decision theory, check out the first 20 minutes of this lecture by Michael Jordan (no, not "air" Jordan, more like "err" Jordan):

http://videolectures.net/mlss09uk_jordan_bfway/

If your browser has trouble with this site, here's a YouTube version with no slides:

https://www.youtube.com/watch?v=HUAE26lNDuE

You can approach hypothesis testing from a frequentist or Bayesian approach.
We tend to teach the frequentist approach (P-values and confidence intervals) to hypothesis testing, but they are not equivalent.

### Comparing Bayes factors to AIC

At the end of the third paragraph of Section 2.4c, when comparing model choice using AIC and Bayes factors, it reads "If all parameters can be estimated with precision, results from both approaches should be similar."

This is not always true. If the data are very informative, we can expect that our posterior probability densities of the parameters will be precise and robust to prior assumptions within each model. However, when comparing posterior probabilities across models, it is the average (marginal) likelihood of each model that updates prior probabilities. These average likelihoods are weighted by the prior. So, even if we have very informative data and precise parameter estimates within each model, the model posterior probabilities can still be very sensitive to the prior distributions on the parameters of each model.

Long story short, if we put enough prior weight on unreasonable parameter values, we can "sink" the marginal likelihood of any model. This is responsible for what is known as [Lindley's Paradox](https://en.wikipedia.org/wiki/Lindley%27s_paradox).

This doesn't mean that Bayesian model comparison is bad. It simply means that the priors we place on each parameter within each model serve as a "penalty" for that parameter, so we have to choose those priors carefully.

## Errata

### Equation 2.7

\begin{equation}
    \Delta = -2 \ln \frac{\textrm{ML}_1}{\textrm{ML}_2} = -2 (\ln \textrm{ML}_1 - \ln \textrm{ML}_2)
\end{equation}

### Equation 2.21

"H" should be "D" in the last equation. Also $p_h$ should be $p_H$:

\begin{equation}
    Pr(H|D) = \frac{P(D | p_H, N) f(p_H)}{\int_{0}^{1} P(D | p_H, N) f(p_H) dp_H}
\end{equation}

### MCMC 3b

The proposal density is inverted. It should be:

\begin{equation}
    R_{\textrm{proposal}} = \frac{Q(p | p^{\prime})}{Q(p^{\prime} | p)}
\end{equation}

The way I remember this: If the proposing the new state is super likely, I need to penalize the acceptance ratio for it, so it needs to go in the denominator.

### Last paragraph of Section 2.4c

"arrogance sampling" should be "importance sampling." Though, I must say, I am now partial to arrogance sampling.

# Chapter 3

## Notes

Paragraph 2 of Section 3.3a reads (in reference to Brownian motion):

"This model assumes that the number of alleles is so large that there is effectively no chance of mutations happening to the same allele more than once..."

This is a bit confusing. At any given time, the model does not assume there must be a huge number of alleles. It only assumes that every mutation creates a new allele (it can't create an allele that has already been seen). So, as time approaches infinity, the total number of alleles observed (including the ones lost to drift, which is most of them) will approach infinity.

## Errata

### Equation 3.17
This distribution should not be for the expectation of $\bar{z}(t)$. It should be:

\begin{equation}
    \bar{z}(t) \sim N(\bar{z}(0), \sigma_{B}^{2} t_1)
\end{equation}

# Chapter 4

## Errata

### In the last paragraph of Section 4.2

$e^{2.68} = 2.3$ should be $e^{0.85} = 2.3$

### Second sentence after Equation 4.9

"Equation 4.8" should be "Equation 4.9," so that it reads:

"Equation 4.9 is exactly identical to the estimated rate fo evolution calculated using the average squared independent contract..."