# Anomaly Detection on the MNIST Dataset
### By Shashank Swaminathan and Chris Lee
## Introduction

Our project was centered around anomaly detection, a method of detecting outliers in a given dataset. It is especially useful in situations where early detection of items differing from the norm is important. For example, in food production facilities, broken conveyor belts could lead to contamination. However, detecting problems early is difficult, as there are not many signs of issues prior to the belt completely breaks. With anomaly detection, the defects could be found earlier on, reducing the risk of contamination.

Anomaly detection can be used in a multitude of scenarios, from finding anomalies in astronomical data to recognition of oddities in medical images. We chose to focus on the image recognition aspect of anomaly detection - specifically on the MNIST dataset. This was for multiple reasons - ease of access to data, the amount of data available, and the relative uniformity of the training data (most arabic numerals look similar to each other, even with moderately bad handwriting). Our goal was to be able to identify if a provided image of a number was anomalous - far off from the standard form of a number.

We approached this problem in two ways, using PCA and Autoencoders. In fact, both methods come from the same angle, and so first we'll address how the overall anomaly detection algorithm works.

## How do we detect anomalies?
A good way to detect anomalies is to simply compare the input image against a "average image" for the given type, i.e. when checking if a given number is a "7", we compare the input image against an average image for a seven. This method is quite useful, but has a significant drawback - we need to know what the input image is trying to represent ahead of time. If the image is already misidentified, then we might improperly identify anomalies when there are none. Even worse, we might miss anomalies. Hence, we look towards the second method, the one that we adopted.

In this method, we will train a model of some kind on a given set of good training images. The model will identify major features within all the images. We'll then use this model to try and recreate a given input image, using those identified major features. The differences between the input image and the recreated image will then represent parts of the image that aren't common to the given dataset - the anomalies in the image. We'll use these anomalies (the loss between the input image and the recreated image) to determine if the image is an anomaly.

Finally, we'll use the assumption that given a set of good training data, the losses between input and recreated images will form a standard distribution. We can then put a threshold at the higher end of this distribution - if the loss between a given input and its recreation is higher than this bar, we'll consider it an anomaly.

There are some points to make about this method:
 - First, the type of training data. Since we are training a model to identify key features within the training data, the classes of data we are training on should all be related, so that the key features across images are also similar. Training across a bunch of single digit numbers would satisfy this - since they are all single digits, and of the same style. However, applying this method on inconsistent datasets, like one of cars and cats would definitely not be appropriate.
 - The second point is that we are making a pretty big assumption when stating that the losses we'll find will become something like a standard distribution curve. While this was not disproven by the training that we did, it is important to remember that this might not always hold, and that it is a point of caution.

## Implementation of Anomaly Detection Algorithm
Now, we move onto discussing how we implemented the algorithm. Just to reiterate, here are the steps we need to implement for our anomaly detection to succeed:
 - Train a model to identify the key features in a set of training images.
 - Use this model to reconstruct a set of training images, and find the losses between input images and reconstructed images.
 - Find a suitable upper limit on the loss values based on the distribution of losses from training.
 - For a given input image, find the loss between input and the recreation. If above the threshold, it is an anomaly.

The first few steps - training a model to identify key features and reconstruct images - fall within the category of **image compression**. In image compression, we identify key features of the input data, and then try to compress the data into a lower dimension by only using these key features. This is essentially the same process as recreating an image from a set of key features. Only here, after determining the key features and stripping it down to a construct of those key features (reducing the image's dimensionality to only these key features), we recreate the image based on the key features (transform the compressed image back to the original dimension).

There are a few methods to do image compression - two common methods are using **Principal Component Analysis (PCA)** and using **Autoencoders**. In PCA, we use the covariance matrix of the data to determine a set of "basis vectors" that represent the input data's key features. Autoencoders are completely different - to be concise, they are unsupervised neural nets that iteratively learn how to compress the given data in the best way possible (reducing the loss of information before and after compression).

PCA is a linear process - it takes the input data, finds the covariance matrix, and then finds a set of key feature vectors such that a linear combination of those vectors can approximate the input data. It does a linear mapping between the data and its key features. This is good as an approximation of the data - it is does the job to a good degree, and is quite fast. However, it is quite poor at identifying non-linear relationships. The more complex the input data, the worse it is at identifying feature relationships. In a situation such as anomaly detection, where misidentified cases can have serious consequences, this may not be useful. It's also important to note here that the PCA method is not one that involves neural networks, and is essentially a linear model.

Autoencoders are a very nonlinear process. They follow the standard mechanisms of an ANN - when used in image processing, they include convolutional layers, pooling layers, ReLU layers, the whole deal. Because they also use gradient descent to optimize its encoding process (the compression process), it can compress data quite well. However, because it is a neural net, it also falls to the trappings of neural nets. One particularly problematic point that shows up while using it in anomaly detection is overfitting. Autoencoders tend to overfit to its input data - which is understandable, given that it is incentivized to reduce losses in compression for its training data. However, overfitting means that the encoder is not only extracting key features, but also common anomalous patterns in the input data, which is troublesome.

Because both have drawbacks, we decided to use both approaches to anomaly detection. First we implemented the PCA approach, and then the Autoencoder approach. Since it is hard to get a good set of 'anomalous data', we don't have an actual accuracy score of how good the detector is. Instead, we were able to provide example cases of the detector in action, for both PCA and Autoencoding.

We didn't put docstrings in our functions since we're already documenting the work using the text option of the notebook. Also, the reason we spend some time discussing the PCA approach in detail, though it is does not involve a neural network, is to provide some basis intuition on what we are doing for anomaly detection. The process we use for PCA will be extended for the Autoencoder process.

### Using PCA

First up, we download our data. Because `sklearn` has a lot of cool PCA-related functions, we used `sklearn` for the PCA method.


```
# from sklearn.datasets import fetch_mldata
# mnist = fetch_mldata('MNIST original')
import matplotlib.pyplot as plt
import numpy as np # we always love numpy
import time
import seaborn as sns

from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.metrics import mean_squared_error
from sklearn.datasets import fetch_openml

# Load data from https://www.openml.org/d/554
mnist = fetch_openml('mnist_784', version=1, return_X_y=False)

from sklearn.model_selection import train_test_split
# test_size: what proportion of original data is used for test set
train_img, test_img, train_lbl, test_lbl = train_test_split(mnist.data,
                                                            mnist.target,
                                                            test_size=1/7.0,
                                                            random_state=0)
```

Next, we define the helper functions used for the PCA algorithm: 
 - First up is the ever-useful image display function, useful for validating if the image data we have makes sense.
 - Second is the preprocessing function. Because PCA is a linearly based process, larger base values will scale the losses up, and might skew the results. Hence, we center our data around the mean of the data, and normalize all the values.
 - The third function is the actual PCA function. Here, we make use of `sklearn`'s PCA function to create a PCA model for us.
 - The fourth function will take a set of input training images, use the PCA model to reconstruct approximations of those images, and then find the losses between each image in the input dataset.
 - This is a useful way of inspecting the losses between input images and reconstructed images. We especially use this function to check if the losses appear to fall in a standard distribution curve.


```
def disp_image_pca(image):
    # need to reorder the tensor dimensions to work properly with imshow
    plt.imshow(image, 'gray')
    plt.axis('off')
    plt.show()

def preprocess_img_pca(train_img, test_img):
  scaler = StandardScaler()# Fit on training set only.
  scaler.fit(train_img)# Apply transform to both the training set and the test set.
  train_img = scaler.transform(train_img)
  test_img = scaler.transform(test_img)
  return train_img, test_img

def my_pca(train_img, amt):
  mpca = PCA(amt)
  mpca.fit(train_img)
  return mpca

def get_losses_pca(imgs, pca):
  post_encoding = mpca.inverse_transform(mpca.transform(imgs))
  losses = []
  for i in np.arange(imgs.shape[0]):
    losses.append(mean_squared_error(imgs[i,:], post_encoding[i,:]))
  return losses

def plot_losses_pca(losses, set_name):
  sns.distplot(losses, kde=False)
  plt.title("Loss Distribution: " + set_name)
  plt.xlabel("Loss Value")
  plt.ylabel("Quanitity")
  plt.show()
```

Just to inspect the loaded data, we'll use the display image function to see what we're looking at.


```
disp_image_pca(train_img[2,:].reshape(28,28))
```

Now we begin actually training our PCA model. We preprocess our data, creating mean-centered and normalized versions of the training and test datasets. We then train a PCA model on the training images. The way the PCA model works is that it preserves a certain percent of variation within the dataset - here, we've selected for 90% of variance within the dataset. The resultant number of dimensions that it retains after compressions is shown as the output of `mpca.n_components`.


```
n_train_img, n_test_img = preprocess_img_pca(train_img, test_img)
mpca = my_pca(n_train_img, 0.90)
mpca.n_components_
```

With the PCA model trained, it's time to evaluate the losses across the data! We evaluate the losses across both the training and test datasets. This way, we can do three things at once:
 - We can see the losses across the training and test sets, between the input and reconstructed images.
 - We can see if the losses across the datasets fall into a standard distribution curve (thereby validating our initial assumption).
 - We can finally see if the curves for the training and test set are similar - if they are, that means that the PCA model is not overfit for the training set, as compared to the test set.

It is worth pointing out that of the losses graphed, there is a certain amount cut from the graph - losses that were greater than 0.5. This was because there were a small amount of loss values that were significant outliers, enough to warp the graph display. In order to preserve the view of the graph, we select only for the losses within a reasonable range (and ignore the outliers). This is a reasonable choice, since there weren't many outliers to begin with - for example, in the training dataset, there were only around 2000 outliers, as compared to 60000 images.


```
# Training set
all_losses_train = np.array(get_losses_pca(n_train_img, mpca))
plot_losses_pca(all_losses_train[np.where(all_losses_train < 0.5)], "Training Set")

# Testing set
all_losses_test = np.array(get_losses_pca(n_test_img, mpca))
plot_losses_pca(all_losses_test[np.where(all_losses_test < 0.5)], "Test Set")
```

From the above, we can see that all three of the points we were looking for have been found. The losses have been evaluated; the curves do appear to be like standard distribution curves; and the curves between the training and test sets appear to be the same. This means the model has performed as we expected.

From the plot of losses, we can infer a reasonable threshold to consider an input an anomaly. We based our threshold off of z-score analysis, and placed a cutoff at 0.15. With that defined, we can create an 'anomaly identifier' function, that would guess if the image is an anomaly. 


```
def is_anomaly_pca(img, pca = mpca, threshold = 0.15):
  loss = get_losses_pca(img, mpca)
  if (loss[0] > threshold):
    print("This is an anomaly with loss: %s" % loss[0])
  return loss[0]
```

Though the function is done, we still lack true anomalous data. To help create some, we defined a couple of helper functions that would essentially mash images together and force create an anomalous image. In addition to that, we added a helper function that would plot both the image and the reconstruction, to show the effects of the PCA based analysis.


```
# Combines images from dataset to create anomalies.
# Assumes input images of shape (1, 784)
def make_anomaly_pca(img1, img2):
    return np.max((img1, img2), axis=0)

#Plots an input and output of the model
def show_in_out(img, pca_model):
    # Plots anomaly
    plt.title("Input")
    disp_image_pca(img.reshape(28,28))

    # Passes image into Model
    img = img.reshape(1,img.shape[0])
    encoded = pca_model.inverse_transform(pca_model.transform(img))

    # Plots output
    plt.title("Output")
    disp_image_pca(encoded.reshape(28,28))

    loss = is_anomaly_pca(img)
    if loss < 0.15:
      print("Anomaly not detected.")
    print(F"LOSS: {loss}")
```

With those defined, we'll now present two examples of anomaly detection using PCA - one where an anomaly was not detected, and one where an anomaly was detected.

#### No Anomaly Detected
Here, the image is clearly very poor - it would almost certainly be considered an actual anomaly. However, the detector did not register this as an anomaly. This could be attributed to multiple things, but we think the most likely explanation is because PCA cannot handle non-linear relationships well, and so was unable to deal with such a messy image well.


```
anomaly_img = make_anomaly_pca(n_test_img[1,:], n_test_img[0,:])
show_in_out(anomaly_img, mpca)
```

#### Anomaly Detected
Here, an anomaly is detected. However, in order to do so, we had to mash multiple images together, and thereby accumulate a bunch of error before getting the final 'anomalous' image. This clearly shows the limitation of PCA - while it can definitely do reconstruction to a certain degree, it fails at identifying anything than the most anomalous images.


```
anomaly_img_1 = make_anomaly_pca(n_test_img[1,:], n_test_img[0,:])
anomaly_img_2 = make_anomaly_pca(n_test_img[2,:], anomaly_img_1)
anomaly_img_3 = make_anomaly_pca(n_test_img[3,:], anomaly_img_2)
anomaly_img_4 = make_anomaly_pca(n_test_img[4,:], anomaly_img_3)
anomaly_img_5 = make_anomaly_pca(n_test_img[5,:], anomaly_img_4)
show_in_out(anomaly_img_5, mpca)
```

With that, we have covered the PCA approach to anomaly detection. Note that this did not involve any neural networks! Instead, we focused on using a linear relationship between image data and variance. However, we also realized that there were quite a few limitations with this method. This in turn motivates the next section - using an Autoencoder.

### Using an Autoencoder
To reiterate what an Autoencoder based approach is, it is a nonlinear process involving reinforcement learning that trains an unsupervised neural network model to encode and decode data to lower dimensions while minizing information loss. We will use this encoding process the same way we did with PCA - to isolate key features of the dataset, and then use the difference between the input and reconstructed image to determine the presence of an anomaly.

In this approach, we will be using PyTorch to do our neural network implementation. Because the model is quite massive, we mount a Google Drive account to the system, in order to save the model parameters for future sessions.

The first step, as always, is to import the necessary libraries and the actual data we will be using.


```
# imports all necessary libraries

!pip install torchviz

import torch
import torchvision
from torch import nn
from torch.autograd import Variable
from torch.utils.data import DataLoader
from torchviz import make_dot
from torchvision import transforms
from torchvision.utils import save_image
from torchvision.datasets import MNIST

import time
import seaborn as sns
import numpy as np
import matplotlib.pyplot as plt

from google.colab import drive
drive.mount('/content/gdrive')
```

    Requirement already satisfied: torchviz in /usr/local/lib/python3.6/dist-packages (0.0.1)
    Requirement already satisfied: graphviz in /usr/local/lib/python3.6/dist-packages (from torchviz) (0.10.1)
    Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (from torchviz) (1.2.0)
    Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torch->torchviz) (1.16.5)
    Drive already mounted at /content/gdrive; to attempt to forcibly remount, call drive.mount("/content/gdrive", force_remount=True).


### Helper Functions
The functions below help to visualize different parts of the model. 'to_image' converts an output of the model into an image easily shown using 'disp_model', which uses matplotlib to plot and show an image. 'visualize_network' is a function that shows the structure of the model used.


```
def to_img(x):
    """Converts x into an image compatible with matplotlib"""
    x = 0.5 * (x + 1)
    x = x.clamp(0, 1)
    x = x.view(x.size(0), 1, 28, 28)
    return x

def disp_image(image, predicted=None):
    
    plt.imshow(image, 'gray')
    plt.axis('off')
    plt.show()

def visualize_network(net, device='cuda', image_dims=(1,28,28)):
    # Visualize the architecture of the model
    # We need to give the net a fake input for this library to visualize the architecture
    fake_input = Variable(torch.zeros((1,image_dims[0], image_dims[1], image_dims[2]))).to(device)
    outputs = net(fake_input)
    # Plot the DAG (Directed Acyclic Graph) of the model
    return make_dot(outputs, dict(net.named_parameters()))
```

### Hyperparameters
After some testing, we found that 20 epochs with a batch size of 128 and a learning rate of 0.001, our model became reasonably accurate.


```
epochs = 20
batch_size = 128 
learning_rate = 1e-3
```

### Loading the Data


```
img_transform = transforms.Compose([
    transforms.ToTensor(), # Transforms data to tensor
    transforms.Normalize((.5,), (.5,)) # Normalizes data
])

# Loads data and formats based on transform above
dataset = MNIST('./data', download=True, transform=img_transform) 

# Creates a dataloader for pulling random sets of data for training
dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=True)
```

### The Model
The model below is an autoencoder, a type of unsupervised learning model. It has two primary parts: the encoder and decoder. The encoder learns non-linear patterns for reducing the dimensionality of the image, and the decoder learns how to convert an encoded image back to a similar image to the original input. The layer structure of the model was based off the code seen here: https://github.com/L1aoXingyu/pytorch-beginner/blob/master/08-AutoEncoder/conv_autoencoder.py. 


```
class Autoencoder(nn.Module):
    def __init__(self):
        super(Autoencoder, self).__init__()
        self.encoder = nn.Sequential(
            nn.Conv2d(1, 16, 3, stride=3, padding=1), # Convolution 1 input, 16 outputs, 3x3 kernel
            nn.ReLU(True), 
            nn.MaxPool2d(2, stride=2), # Maxpooling 2x2 kernel
            nn.Conv2d(16, 8, 3, stride=2, padding=1), # Convolution 16 inputs, 8 outputs, 3x3 kernel
            nn.ReLU(True),
            nn.MaxPool2d(2, stride=1) # Maxpooling 2x2 kernel
        )

        # The decoder reverses the steps accomplished in the encoder.
        self.decoder = nn.Sequential(
            nn.ConvTranspose2d(8, 16, 3, stride=2),
            nn.ReLU(True),
            nn.ConvTranspose2d(16, 8, 5, stride=3, padding=1),
            nn.ReLU(True),
            nn.ConvTranspose2d(8, 1, 2, stride=2, padding=1),
            nn.Tanh()
        )

    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x
```

### Training the Model
The model was trained over 20 epochs using a batch size of 128 and learning rate of 0.001. Any more epochs only seemed to overfit the data and only minimally decrease the loss. Because we have no labels, our loss was calculated as the mean squared error between the input and output image of the model. By optimizing this value, the autoencoder could learn to encode and decode the data.


```
def train_model(model, num_epochs, find_loss, model_save='autoencoder.pt'):
    batch_loss = []
    batch_num = []
    curr_batch = 0

    print_every = 16
    
    # Attempts to load model from google drive
    try: 
        path = F"/content/gdrive/My Drive/{model_save}"
        model.load_state_dict(torch.load(path))
        print("Model Loaded")
    
    # If model is not found in drive, model is trained
    except:
        print("Model Training")

        # Iterates through num_epochs epochs
        for epoch in range(num_epochs):
            # Total loss of a particular batch
            running_loss = 0.0

            # The start time of an epoch
            start_time = time.time()

            # Iterates through a batch
            for i, data in enumerate(dataloader, 0):
                # Loads image and converts is to a variable in graphics memory
                img, _ = data
                img = Variable(img).cuda()
                # ===================forward=====================
                # Plugs input into the model and finds loss
                output = model(img)
                loss = find_loss(output, img)
                running_loss += loss.data.item()
                # ===================backward====================
                # Backpropagates values to tweak weights
                optimizer.zero_grad()
                loss.backward()
                optimizer.step()

                # Prints epoch every print_every and appends loss data to list
                if (i % print_every) == print_every-1:
                    print("Epoch {}, Iteration {}\t train_loss: {:.2f} took: {:.2f}s".format(
                          epoch + 1, i+1, running_loss / print_every, time.time() - start_time))
                    batch_loss.append(running_loss / print_every)
                    batch_num.append(curr_batch)
                    running_loss = 0.0

                curr_batch+=1
            # ===================log========================
            # Prints when epoch is finished
            print('\nEpoch [{}/{}], loss:{:.4f}'
                .format(epoch+1, num_epochs, loss.data))
            
        # Saves model to 
        model_save_name = 'autoencoder.pt'
        path = F"/content/gdrive/My Drive/{model_save_name}" 
        torch.save(model.state_dict(), path)

    return batch_loss, batch_num
```


```
model = Autoencoder().cuda()
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate,
                             weight_decay=1e-5)

visualize_network(model)
```




    

    




```
batch_loss, batch_num = train_model(model, epochs, criterion)
```

    Model Training
    Epoch 1, Iteration 16	 train_loss: 0.75 took: 0.51s
    Epoch 1, Iteration 32	 train_loss: 0.67 took: 1.01s
    Epoch 1, Iteration 48	 train_loss: 0.52 took: 1.49s
    Epoch 1, Iteration 64	 train_loss: 0.43 took: 1.94s
    Epoch 1, Iteration 80	 train_loss: 0.40 took: 2.41s
    Epoch 1, Iteration 96	 train_loss: 0.38 took: 2.89s
    Epoch 1, Iteration 112	 train_loss: 0.37 took: 3.36s
    Epoch 1, Iteration 128	 train_loss: 0.36 took: 3.82s
    Epoch 1, Iteration 144	 train_loss: 0.35 took: 4.31s
    Epoch 1, Iteration 160	 train_loss: 0.35 took: 4.76s
    Epoch 1, Iteration 176	 train_loss: 0.34 took: 5.23s
    Epoch 1, Iteration 192	 train_loss: 0.33 took: 5.71s
    Epoch 1, Iteration 208	 train_loss: 0.33 took: 6.24s
    Epoch 1, Iteration 224	 train_loss: 0.33 took: 6.71s
    Epoch 1, Iteration 240	 train_loss: 0.33 took: 7.19s
    Epoch 1, Iteration 256	 train_loss: 0.33 took: 7.64s
    Epoch 1, Iteration 272	 train_loss: 0.32 took: 8.10s
    Epoch 1, Iteration 288	 train_loss: 0.32 took: 8.59s
    Epoch 1, Iteration 304	 train_loss: 0.32 took: 9.05s
    Epoch 1, Iteration 320	 train_loss: 0.32 took: 9.53s
    Epoch 1, Iteration 336	 train_loss: 0.32 took: 10.00s
    Epoch 1, Iteration 352	 train_loss: 0.31 took: 10.49s
    Epoch 1, Iteration 368	 train_loss: 0.31 took: 10.96s
    Epoch 1, Iteration 384	 train_loss: 0.31 took: 11.42s
    Epoch 1, Iteration 400	 train_loss: 0.31 took: 11.91s
    Epoch 1, Iteration 416	 train_loss: 0.31 took: 12.39s
    Epoch 1, Iteration 432	 train_loss: 0.30 took: 12.84s
    Epoch 1, Iteration 448	 train_loss: 0.30 took: 13.30s
    Epoch 1, Iteration 464	 train_loss: 0.29 took: 13.78s
    
    Epoch [1/20], loss:0.2851
    Epoch 2, Iteration 16	 train_loss: 0.28 took: 0.48s
    Epoch 2, Iteration 32	 train_loss: 0.26 took: 0.94s
    Epoch 2, Iteration 48	 train_loss: 0.25 took: 1.41s
    Epoch 2, Iteration 64	 train_loss: 0.24 took: 1.89s
    Epoch 2, Iteration 80	 train_loss: 0.23 took: 2.38s
    Epoch 2, Iteration 96	 train_loss: 0.22 took: 2.84s
    Epoch 2, Iteration 112	 train_loss: 0.22 took: 3.29s
    Epoch 2, Iteration 128	 train_loss: 0.22 took: 3.80s
    Epoch 2, Iteration 144	 train_loss: 0.21 took: 4.29s
    Epoch 2, Iteration 160	 train_loss: 0.21 took: 4.74s
    Epoch 2, Iteration 176	 train_loss: 0.20 took: 5.20s
    Epoch 2, Iteration 192	 train_loss: 0.20 took: 5.66s
    Epoch 2, Iteration 208	 train_loss: 0.20 took: 6.12s
    Epoch 2, Iteration 224	 train_loss: 0.20 took: 6.61s
    Epoch 2, Iteration 240	 train_loss: 0.19 took: 7.09s
    Epoch 2, Iteration 256	 train_loss: 0.19 took: 7.57s
    Epoch 2, Iteration 272	 train_loss: 0.19 took: 8.05s
    Epoch 2, Iteration 288	 train_loss: 0.19 took: 8.51s
    Epoch 2, Iteration 304	 train_loss: 0.19 took: 8.98s
    Epoch 2, Iteration 320	 train_loss: 0.19 took: 9.46s
    Epoch 2, Iteration 336	 train_loss: 0.18 took: 9.94s
    Epoch 2, Iteration 352	 train_loss: 0.18 took: 10.41s
    Epoch 2, Iteration 368	 train_loss: 0.18 took: 10.86s
    Epoch 2, Iteration 384	 train_loss: 0.18 took: 11.32s
    Epoch 2, Iteration 400	 train_loss: 0.18 took: 11.80s
    Epoch 2, Iteration 416	 train_loss: 0.18 took: 12.29s
    Epoch 2, Iteration 432	 train_loss: 0.18 took: 12.77s
    Epoch 2, Iteration 448	 train_loss: 0.18 took: 13.24s
    Epoch 2, Iteration 464	 train_loss: 0.18 took: 13.71s
    
    Epoch [2/20], loss:0.1749
    Epoch 3, Iteration 16	 train_loss: 0.17 took: 0.48s
    Epoch 3, Iteration 32	 train_loss: 0.17 took: 0.97s
    Epoch 3, Iteration 48	 train_loss: 0.17 took: 1.44s
    Epoch 3, Iteration 64	 train_loss: 0.17 took: 1.89s
    Epoch 3, Iteration 80	 train_loss: 0.17 took: 2.37s
    Epoch 3, Iteration 96	 train_loss: 0.17 took: 2.82s
    Epoch 3, Iteration 112	 train_loss: 0.17 took: 3.30s
    Epoch 3, Iteration 128	 train_loss: 0.17 took: 3.74s
    Epoch 3, Iteration 144	 train_loss: 0.17 took: 4.21s
    Epoch 3, Iteration 160	 train_loss: 0.16 took: 4.66s
    Epoch 3, Iteration 176	 train_loss: 0.16 took: 5.12s
    Epoch 3, Iteration 192	 train_loss: 0.16 took: 5.56s
    Epoch 3, Iteration 208	 train_loss: 0.16 took: 6.02s
    Epoch 3, Iteration 224	 train_loss: 0.16 took: 6.50s
    Epoch 3, Iteration 240	 train_loss: 0.16 took: 6.96s
    Epoch 3, Iteration 256	 train_loss: 0.16 took: 7.45s
    Epoch 3, Iteration 272	 train_loss: 0.16 took: 7.92s
    Epoch 3, Iteration 288	 train_loss: 0.15 took: 8.41s
    Epoch 3, Iteration 304	 train_loss: 0.15 took: 8.91s
    Epoch 3, Iteration 320	 train_loss: 0.15 took: 9.38s
    Epoch 3, Iteration 336	 train_loss: 0.15 took: 9.85s
    Epoch 3, Iteration 352	 train_loss: 0.15 took: 10.35s
    Epoch 3, Iteration 368	 train_loss: 0.15 took: 10.81s
    Epoch 3, Iteration 384	 train_loss: 0.15 took: 11.29s
    Epoch 3, Iteration 400	 train_loss: 0.15 took: 11.74s
    Epoch 3, Iteration 416	 train_loss: 0.15 took: 12.21s
    Epoch 3, Iteration 432	 train_loss: 0.15 took: 12.68s
    Epoch 3, Iteration 448	 train_loss: 0.15 took: 13.16s
    Epoch 3, Iteration 464	 train_loss: 0.15 took: 13.64s
    
    Epoch [3/20], loss:0.1478
    Epoch 4, Iteration 16	 train_loss: 0.15 took: 0.48s
    Epoch 4, Iteration 32	 train_loss: 0.15 took: 0.96s
    Epoch 4, Iteration 48	 train_loss: 0.15 took: 1.43s
    Epoch 4, Iteration 64	 train_loss: 0.15 took: 1.90s
    Epoch 4, Iteration 80	 train_loss: 0.15 took: 2.35s
    Epoch 4, Iteration 96	 train_loss: 0.15 took: 2.85s
    Epoch 4, Iteration 112	 train_loss: 0.14 took: 3.32s
    Epoch 4, Iteration 128	 train_loss: 0.15 took: 3.83s
    Epoch 4, Iteration 144	 train_loss: 0.14 took: 4.30s
    Epoch 4, Iteration 160	 train_loss: 0.14 took: 4.77s
    Epoch 4, Iteration 176	 train_loss: 0.14 took: 5.25s
    Epoch 4, Iteration 192	 train_loss: 0.14 took: 5.72s
    Epoch 4, Iteration 208	 train_loss: 0.14 took: 6.17s
    Epoch 4, Iteration 224	 train_loss: 0.14 took: 6.65s
    Epoch 4, Iteration 240	 train_loss: 0.14 took: 7.12s
    Epoch 4, Iteration 256	 train_loss: 0.14 took: 7.59s
    Epoch 4, Iteration 272	 train_loss: 0.15 took: 8.05s
    Epoch 4, Iteration 288	 train_loss: 0.14 took: 8.50s
    Epoch 4, Iteration 304	 train_loss: 0.14 took: 8.98s
    Epoch 4, Iteration 320	 train_loss: 0.14 took: 9.45s
    Epoch 4, Iteration 336	 train_loss: 0.14 took: 9.93s
    Epoch 4, Iteration 352	 train_loss: 0.14 took: 10.39s
    Epoch 4, Iteration 368	 train_loss: 0.14 took: 10.87s
    Epoch 4, Iteration 384	 train_loss: 0.14 took: 11.35s
    Epoch 4, Iteration 400	 train_loss: 0.14 took: 11.82s
    Epoch 4, Iteration 416	 train_loss: 0.14 took: 12.27s
    Epoch 4, Iteration 432	 train_loss: 0.14 took: 12.74s
    Epoch 4, Iteration 448	 train_loss: 0.14 took: 13.19s
    Epoch 4, Iteration 464	 train_loss: 0.14 took: 13.64s
    
    Epoch [4/20], loss:0.1456
    Epoch 5, Iteration 16	 train_loss: 0.14 took: 0.46s
    Epoch 5, Iteration 32	 train_loss: 0.14 took: 0.94s
    Epoch 5, Iteration 48	 train_loss: 0.14 took: 1.44s
    Epoch 5, Iteration 64	 train_loss: 0.14 took: 1.91s
    Epoch 5, Iteration 80	 train_loss: 0.14 took: 2.40s
    Epoch 5, Iteration 96	 train_loss: 0.14 took: 2.87s
    Epoch 5, Iteration 112	 train_loss: 0.14 took: 3.37s
    Epoch 5, Iteration 128	 train_loss: 0.14 took: 3.83s
    Epoch 5, Iteration 144	 train_loss: 0.14 took: 4.31s
    Epoch 5, Iteration 160	 train_loss: 0.14 took: 4.77s
    Epoch 5, Iteration 176	 train_loss: 0.14 took: 5.25s
    Epoch 5, Iteration 192	 train_loss: 0.14 took: 5.73s
    Epoch 5, Iteration 208	 train_loss: 0.14 took: 6.18s
    Epoch 5, Iteration 224	 train_loss: 0.14 took: 6.65s
    Epoch 5, Iteration 240	 train_loss: 0.14 took: 7.10s
    Epoch 5, Iteration 256	 train_loss: 0.14 took: 7.55s
    Epoch 5, Iteration 272	 train_loss: 0.14 took: 8.01s
    Epoch 5, Iteration 288	 train_loss: 0.14 took: 8.51s
    Epoch 5, Iteration 304	 train_loss: 0.14 took: 8.99s
    Epoch 5, Iteration 320	 train_loss: 0.14 took: 9.45s
    Epoch 5, Iteration 336	 train_loss: 0.13 took: 9.90s
    Epoch 5, Iteration 352	 train_loss: 0.14 took: 10.36s
    Epoch 5, Iteration 368	 train_loss: 0.14 took: 10.81s
    Epoch 5, Iteration 384	 train_loss: 0.14 took: 11.27s
    Epoch 5, Iteration 400	 train_loss: 0.14 took: 11.76s
    Epoch 5, Iteration 416	 train_loss: 0.14 took: 12.23s
    Epoch 5, Iteration 432	 train_loss: 0.13 took: 12.70s
    Epoch 5, Iteration 448	 train_loss: 0.13 took: 13.15s
    Epoch 5, Iteration 464	 train_loss: 0.14 took: 13.61s
    
    Epoch [5/20], loss:0.1359
    Epoch 6, Iteration 16	 train_loss: 0.14 took: 0.48s
    Epoch 6, Iteration 32	 train_loss: 0.14 took: 0.96s
    Epoch 6, Iteration 48	 train_loss: 0.13 took: 1.42s
    Epoch 6, Iteration 64	 train_loss: 0.13 took: 1.92s
    Epoch 6, Iteration 80	 train_loss: 0.14 took: 2.37s
    Epoch 6, Iteration 96	 train_loss: 0.14 took: 2.84s
    Epoch 6, Iteration 112	 train_loss: 0.13 took: 3.34s
    Epoch 6, Iteration 128	 train_loss: 0.13 took: 3.81s
    Epoch 6, Iteration 144	 train_loss: 0.14 took: 4.29s
    Epoch 6, Iteration 160	 train_loss: 0.13 took: 4.76s
    Epoch 6, Iteration 176	 train_loss: 0.13 took: 5.26s
    Epoch 6, Iteration 192	 train_loss: 0.13 took: 5.73s
    Epoch 6, Iteration 208	 train_loss: 0.14 took: 6.20s
    Epoch 6, Iteration 224	 train_loss: 0.13 took: 6.65s
    Epoch 6, Iteration 240	 train_loss: 0.13 took: 7.13s
    Epoch 6, Iteration 256	 train_loss: 0.14 took: 7.60s
    Epoch 6, Iteration 272	 train_loss: 0.13 took: 8.11s
    Epoch 6, Iteration 288	 train_loss: 0.13 took: 8.59s
    Epoch 6, Iteration 304	 train_loss: 0.13 took: 9.07s
    Epoch 6, Iteration 320	 train_loss: 0.13 took: 9.55s
    Epoch 6, Iteration 336	 train_loss: 0.13 took: 10.03s
    Epoch 6, Iteration 352	 train_loss: 0.13 took: 10.49s
    Epoch 6, Iteration 368	 train_loss: 0.13 took: 10.98s
    Epoch 6, Iteration 384	 train_loss: 0.13 took: 11.49s
    Epoch 6, Iteration 400	 train_loss: 0.13 took: 11.99s
    Epoch 6, Iteration 416	 train_loss: 0.13 took: 12.48s
    Epoch 6, Iteration 432	 train_loss: 0.13 took: 12.95s
    Epoch 6, Iteration 448	 train_loss: 0.13 took: 13.41s
    Epoch 6, Iteration 464	 train_loss: 0.13 took: 13.87s
    
    Epoch [6/20], loss:0.1376
    Epoch 7, Iteration 16	 train_loss: 0.13 took: 0.47s
    Epoch 7, Iteration 32	 train_loss: 0.13 took: 0.97s
    Epoch 7, Iteration 48	 train_loss: 0.13 took: 1.42s
    Epoch 7, Iteration 64	 train_loss: 0.13 took: 1.87s
    Epoch 7, Iteration 80	 train_loss: 0.13 took: 2.36s
    Epoch 7, Iteration 96	 train_loss: 0.13 took: 2.84s
    Epoch 7, Iteration 112	 train_loss: 0.13 took: 3.30s
    Epoch 7, Iteration 128	 train_loss: 0.13 took: 3.75s
    Epoch 7, Iteration 144	 train_loss: 0.13 took: 4.28s
    Epoch 7, Iteration 160	 train_loss: 0.13 took: 4.75s
    Epoch 7, Iteration 176	 train_loss: 0.13 took: 5.23s
    Epoch 7, Iteration 192	 train_loss: 0.13 took: 5.70s
    Epoch 7, Iteration 208	 train_loss: 0.13 took: 6.19s
    Epoch 7, Iteration 224	 train_loss: 0.13 took: 6.65s
    Epoch 7, Iteration 240	 train_loss: 0.13 took: 7.13s
    Epoch 7, Iteration 256	 train_loss: 0.13 took: 7.62s
    Epoch 7, Iteration 272	 train_loss: 0.13 took: 8.10s
    Epoch 7, Iteration 288	 train_loss: 0.13 took: 8.56s
    Epoch 7, Iteration 304	 train_loss: 0.13 took: 9.02s
    Epoch 7, Iteration 320	 train_loss: 0.13 took: 9.49s
    Epoch 7, Iteration 336	 train_loss: 0.13 took: 9.97s
    Epoch 7, Iteration 352	 train_loss: 0.13 took: 10.44s
    Epoch 7, Iteration 368	 train_loss: 0.13 took: 10.92s
    Epoch 7, Iteration 384	 train_loss: 0.13 took: 11.40s
    Epoch 7, Iteration 400	 train_loss: 0.13 took: 11.87s
    Epoch 7, Iteration 416	 train_loss: 0.13 took: 12.35s
    Epoch 7, Iteration 432	 train_loss: 0.13 took: 12.84s
    Epoch 7, Iteration 448	 train_loss: 0.13 took: 13.33s
    Epoch 7, Iteration 464	 train_loss: 0.13 took: 13.81s
    
    Epoch [7/20], loss:0.1277
    Epoch 8, Iteration 16	 train_loss: 0.13 took: 0.50s
    Epoch 8, Iteration 32	 train_loss: 0.13 took: 0.98s
    Epoch 8, Iteration 48	 train_loss: 0.13 took: 1.44s
    Epoch 8, Iteration 64	 train_loss: 0.13 took: 1.90s
    Epoch 8, Iteration 80	 train_loss: 0.13 took: 2.38s
    Epoch 8, Iteration 96	 train_loss: 0.13 took: 2.86s
    Epoch 8, Iteration 112	 train_loss: 0.13 took: 3.35s
    Epoch 8, Iteration 128	 train_loss: 0.13 took: 3.83s
    Epoch 8, Iteration 144	 train_loss: 0.13 took: 4.30s
    Epoch 8, Iteration 160	 train_loss: 0.13 took: 4.78s
    Epoch 8, Iteration 176	 train_loss: 0.13 took: 5.25s
    Epoch 8, Iteration 192	 train_loss: 0.13 took: 5.71s
    Epoch 8, Iteration 208	 train_loss: 0.13 took: 6.17s
    Epoch 8, Iteration 224	 train_loss: 0.13 took: 6.65s
    Epoch 8, Iteration 240	 train_loss: 0.13 took: 7.11s
    Epoch 8, Iteration 256	 train_loss: 0.13 took: 7.56s
    Epoch 8, Iteration 272	 train_loss: 0.13 took: 8.04s
    Epoch 8, Iteration 288	 train_loss: 0.13 took: 8.49s
    Epoch 8, Iteration 304	 train_loss: 0.13 took: 8.94s
    Epoch 8, Iteration 320	 train_loss: 0.13 took: 9.42s
    Epoch 8, Iteration 336	 train_loss: 0.13 took: 9.91s
    Epoch 8, Iteration 352	 train_loss: 0.13 took: 10.40s
    Epoch 8, Iteration 368	 train_loss: 0.13 took: 10.88s
    Epoch 8, Iteration 384	 train_loss: 0.13 took: 11.35s
    Epoch 8, Iteration 400	 train_loss: 0.13 took: 11.85s
    Epoch 8, Iteration 416	 train_loss: 0.13 took: 12.32s
    Epoch 8, Iteration 432	 train_loss: 0.13 took: 12.80s
    Epoch 8, Iteration 448	 train_loss: 0.13 took: 13.27s
    Epoch 8, Iteration 464	 train_loss: 0.13 took: 13.72s
    
    Epoch [8/20], loss:0.1196
    Epoch 9, Iteration 16	 train_loss: 0.13 took: 0.49s
    Epoch 9, Iteration 32	 train_loss: 0.13 took: 0.97s
    Epoch 9, Iteration 48	 train_loss: 0.13 took: 1.43s
    Epoch 9, Iteration 64	 train_loss: 0.13 took: 1.89s
    Epoch 9, Iteration 80	 train_loss: 0.13 took: 2.35s
    Epoch 9, Iteration 96	 train_loss: 0.13 took: 2.79s
    Epoch 9, Iteration 112	 train_loss: 0.13 took: 3.25s
    Epoch 9, Iteration 128	 train_loss: 0.13 took: 3.70s
    Epoch 9, Iteration 144	 train_loss: 0.13 took: 4.18s
    Epoch 9, Iteration 160	 train_loss: 0.13 took: 4.63s
    Epoch 9, Iteration 176	 train_loss: 0.13 took: 5.09s
    Epoch 9, Iteration 192	 train_loss: 0.13 took: 5.54s
    Epoch 9, Iteration 208	 train_loss: 0.13 took: 6.01s
    Epoch 9, Iteration 224	 train_loss: 0.13 took: 6.47s
    Epoch 9, Iteration 240	 train_loss: 0.13 took: 6.94s
    Epoch 9, Iteration 256	 train_loss: 0.13 took: 7.40s
    Epoch 9, Iteration 272	 train_loss: 0.13 took: 7.86s
    Epoch 9, Iteration 288	 train_loss: 0.13 took: 8.32s
    Epoch 9, Iteration 304	 train_loss: 0.13 took: 8.77s
    Epoch 9, Iteration 320	 train_loss: 0.13 took: 9.23s
    Epoch 9, Iteration 336	 train_loss: 0.13 took: 9.67s
    Epoch 9, Iteration 352	 train_loss: 0.13 took: 10.13s
    Epoch 9, Iteration 368	 train_loss: 0.12 took: 10.61s
    Epoch 9, Iteration 384	 train_loss: 0.13 took: 11.07s
    Epoch 9, Iteration 400	 train_loss: 0.13 took: 11.54s
    Epoch 9, Iteration 416	 train_loss: 0.13 took: 11.99s
    Epoch 9, Iteration 432	 train_loss: 0.13 took: 12.47s
    Epoch 9, Iteration 448	 train_loss: 0.13 took: 12.93s
    Epoch 9, Iteration 464	 train_loss: 0.12 took: 13.45s
    
    Epoch [9/20], loss:0.1304
    Epoch 10, Iteration 16	 train_loss: 0.13 took: 0.47s
    Epoch 10, Iteration 32	 train_loss: 0.13 took: 0.93s
    Epoch 10, Iteration 48	 train_loss: 0.13 took: 1.38s
    Epoch 10, Iteration 64	 train_loss: 0.12 took: 1.85s
    Epoch 10, Iteration 80	 train_loss: 0.12 took: 2.30s
    Epoch 10, Iteration 96	 train_loss: 0.12 took: 2.79s
    Epoch 10, Iteration 112	 train_loss: 0.12 took: 3.26s
    Epoch 10, Iteration 128	 train_loss: 0.13 took: 3.72s
    Epoch 10, Iteration 144	 train_loss: 0.13 took: 4.21s
    Epoch 10, Iteration 160	 train_loss: 0.13 took: 4.67s
    Epoch 10, Iteration 176	 train_loss: 0.13 took: 5.17s
    Epoch 10, Iteration 192	 train_loss: 0.13 took: 5.65s
    Epoch 10, Iteration 208	 train_loss: 0.12 took: 6.13s
    Epoch 10, Iteration 224	 train_loss: 0.13 took: 6.61s
    Epoch 10, Iteration 240	 train_loss: 0.13 took: 7.10s
    Epoch 10, Iteration 256	 train_loss: 0.12 took: 7.61s
    Epoch 10, Iteration 272	 train_loss: 0.12 took: 8.10s
    Epoch 10, Iteration 288	 train_loss: 0.13 took: 8.59s
    Epoch 10, Iteration 304	 train_loss: 0.12 took: 9.09s
    Epoch 10, Iteration 320	 train_loss: 0.13 took: 9.58s
    Epoch 10, Iteration 336	 train_loss: 0.12 took: 10.05s
    Epoch 10, Iteration 352	 train_loss: 0.13 took: 10.51s
    Epoch 10, Iteration 368	 train_loss: 0.13 took: 11.02s
    Epoch 10, Iteration 384	 train_loss: 0.13 took: 11.51s
    Epoch 10, Iteration 400	 train_loss: 0.12 took: 12.00s
    Epoch 10, Iteration 416	 train_loss: 0.12 took: 12.48s
    Epoch 10, Iteration 432	 train_loss: 0.13 took: 12.96s
    Epoch 10, Iteration 448	 train_loss: 0.13 took: 13.48s
    Epoch 10, Iteration 464	 train_loss: 0.12 took: 13.98s
    
    Epoch [10/20], loss:0.1279
    Epoch 11, Iteration 16	 train_loss: 0.13 took: 0.49s
    Epoch 11, Iteration 32	 train_loss: 0.12 took: 0.98s
    Epoch 11, Iteration 48	 train_loss: 0.13 took: 1.46s
    Epoch 11, Iteration 64	 train_loss: 0.12 took: 1.91s
    Epoch 11, Iteration 80	 train_loss: 0.13 took: 2.38s
    Epoch 11, Iteration 96	 train_loss: 0.12 took: 2.87s
    Epoch 11, Iteration 112	 train_loss: 0.12 took: 3.35s
    Epoch 11, Iteration 128	 train_loss: 0.12 took: 3.83s
    Epoch 11, Iteration 144	 train_loss: 0.13 took: 4.31s
    Epoch 11, Iteration 160	 train_loss: 0.13 took: 4.78s
    Epoch 11, Iteration 176	 train_loss: 0.12 took: 5.26s
    Epoch 11, Iteration 192	 train_loss: 0.12 took: 5.76s
    Epoch 11, Iteration 208	 train_loss: 0.13 took: 6.24s
    Epoch 11, Iteration 224	 train_loss: 0.12 took: 6.70s
    Epoch 11, Iteration 240	 train_loss: 0.13 took: 7.18s
    Epoch 11, Iteration 256	 train_loss: 0.12 took: 7.66s
    Epoch 11, Iteration 272	 train_loss: 0.12 took: 8.14s
    Epoch 11, Iteration 288	 train_loss: 0.12 took: 8.62s
    Epoch 11, Iteration 304	 train_loss: 0.12 took: 9.09s
    Epoch 11, Iteration 320	 train_loss: 0.12 took: 9.59s
    Epoch 11, Iteration 336	 train_loss: 0.12 took: 10.04s
    Epoch 11, Iteration 352	 train_loss: 0.12 took: 10.52s
    Epoch 11, Iteration 368	 train_loss: 0.12 took: 10.99s
    Epoch 11, Iteration 384	 train_loss: 0.13 took: 11.48s
    Epoch 11, Iteration 400	 train_loss: 0.12 took: 11.92s
    Epoch 11, Iteration 416	 train_loss: 0.12 took: 12.39s
    Epoch 11, Iteration 432	 train_loss: 0.12 took: 12.86s
    Epoch 11, Iteration 448	 train_loss: 0.12 took: 13.35s
    Epoch 11, Iteration 464	 train_loss: 0.12 took: 13.83s
    
    Epoch [11/20], loss:0.1313
    Epoch 12, Iteration 16	 train_loss: 0.12 took: 0.47s
    Epoch 12, Iteration 32	 train_loss: 0.12 took: 0.93s
    Epoch 12, Iteration 48	 train_loss: 0.12 took: 1.39s
    Epoch 12, Iteration 64	 train_loss: 0.13 took: 1.86s
    Epoch 12, Iteration 80	 train_loss: 0.13 took: 2.33s
    Epoch 12, Iteration 96	 train_loss: 0.12 took: 2.82s
    Epoch 12, Iteration 112	 train_loss: 0.12 took: 3.27s
    Epoch 12, Iteration 128	 train_loss: 0.12 took: 3.74s
    Epoch 12, Iteration 144	 train_loss: 0.12 took: 4.19s
    Epoch 12, Iteration 160	 train_loss: 0.12 took: 4.70s
    Epoch 12, Iteration 176	 train_loss: 0.12 took: 5.17s
    Epoch 12, Iteration 192	 train_loss: 0.12 took: 5.66s
    Epoch 12, Iteration 208	 train_loss: 0.12 took: 6.11s
    Epoch 12, Iteration 224	 train_loss: 0.12 took: 6.60s
    Epoch 12, Iteration 240	 train_loss: 0.12 took: 7.07s
    Epoch 12, Iteration 256	 train_loss: 0.12 took: 7.56s
    Epoch 12, Iteration 272	 train_loss: 0.12 took: 8.02s
    Epoch 12, Iteration 288	 train_loss: 0.12 took: 8.49s
    Epoch 12, Iteration 304	 train_loss: 0.12 took: 8.96s
    Epoch 12, Iteration 320	 train_loss: 0.12 took: 9.41s
    Epoch 12, Iteration 336	 train_loss: 0.12 took: 9.88s
    Epoch 12, Iteration 352	 train_loss: 0.12 took: 10.39s
    Epoch 12, Iteration 368	 train_loss: 0.12 took: 10.85s
    Epoch 12, Iteration 384	 train_loss: 0.12 took: 11.30s
    Epoch 12, Iteration 400	 train_loss: 0.12 took: 11.80s
    Epoch 12, Iteration 416	 train_loss: 0.12 took: 12.26s
    Epoch 12, Iteration 432	 train_loss: 0.12 took: 12.75s
    Epoch 12, Iteration 448	 train_loss: 0.12 took: 13.22s
    Epoch 12, Iteration 464	 train_loss: 0.12 took: 13.69s
    
    Epoch [12/20], loss:0.1206
    Epoch 13, Iteration 16	 train_loss: 0.12 took: 0.45s
    Epoch 13, Iteration 32	 train_loss: 0.12 took: 0.94s
    Epoch 13, Iteration 48	 train_loss: 0.12 took: 1.50s
    Epoch 13, Iteration 64	 train_loss: 0.12 took: 2.03s
    Epoch 13, Iteration 80	 train_loss: 0.12 took: 2.49s
    Epoch 13, Iteration 96	 train_loss: 0.12 took: 2.98s
    Epoch 13, Iteration 112	 train_loss: 0.12 took: 3.46s
    Epoch 13, Iteration 128	 train_loss: 0.12 took: 3.96s
    Epoch 13, Iteration 144	 train_loss: 0.12 took: 4.45s
    Epoch 13, Iteration 160	 train_loss: 0.12 took: 4.93s
    Epoch 13, Iteration 176	 train_loss: 0.12 took: 5.41s
    Epoch 13, Iteration 192	 train_loss: 0.12 took: 5.86s
    Epoch 13, Iteration 208	 train_loss: 0.12 took: 6.31s
    Epoch 13, Iteration 224	 train_loss: 0.12 took: 6.79s
    Epoch 13, Iteration 240	 train_loss: 0.12 took: 7.26s
    Epoch 13, Iteration 256	 train_loss: 0.12 took: 7.75s
    Epoch 13, Iteration 272	 train_loss: 0.12 took: 8.29s
    Epoch 13, Iteration 288	 train_loss: 0.12 took: 8.76s
    Epoch 13, Iteration 304	 train_loss: 0.12 took: 9.24s
    Epoch 13, Iteration 320	 train_loss: 0.12 took: 9.73s
    Epoch 13, Iteration 336	 train_loss: 0.12 took: 10.21s
    Epoch 13, Iteration 352	 train_loss: 0.12 took: 10.67s
    Epoch 13, Iteration 368	 train_loss: 0.12 took: 11.13s
    Epoch 13, Iteration 384	 train_loss: 0.12 took: 11.57s
    Epoch 13, Iteration 400	 train_loss: 0.12 took: 12.04s
    Epoch 13, Iteration 416	 train_loss: 0.12 took: 12.50s
    Epoch 13, Iteration 432	 train_loss: 0.12 took: 12.99s
    Epoch 13, Iteration 448	 train_loss: 0.12 took: 13.46s
    Epoch 13, Iteration 464	 train_loss: 0.12 took: 13.92s
    
    Epoch [13/20], loss:0.1162
    Epoch 14, Iteration 16	 train_loss: 0.12 took: 0.54s
    Epoch 14, Iteration 32	 train_loss: 0.12 took: 1.01s
    Epoch 14, Iteration 48	 train_loss: 0.12 took: 1.48s
    Epoch 14, Iteration 64	 train_loss: 0.12 took: 1.98s
    Epoch 14, Iteration 80	 train_loss: 0.12 took: 2.44s
    Epoch 14, Iteration 96	 train_loss: 0.12 took: 2.92s
    Epoch 14, Iteration 112	 train_loss: 0.12 took: 3.43s
    Epoch 14, Iteration 128	 train_loss: 0.12 took: 3.94s
    Epoch 14, Iteration 144	 train_loss: 0.12 took: 4.42s
    Epoch 14, Iteration 160	 train_loss: 0.12 took: 4.90s
    Epoch 14, Iteration 176	 train_loss: 0.12 took: 5.38s
    Epoch 14, Iteration 192	 train_loss: 0.12 took: 5.85s
    Epoch 14, Iteration 208	 train_loss: 0.12 took: 6.31s
    Epoch 14, Iteration 224	 train_loss: 0.12 took: 6.78s
    Epoch 14, Iteration 240	 train_loss: 0.12 took: 7.22s
    Epoch 14, Iteration 256	 train_loss: 0.12 took: 7.71s
    Epoch 14, Iteration 272	 train_loss: 0.12 took: 8.19s
    Epoch 14, Iteration 288	 train_loss: 0.12 took: 8.66s
    Epoch 14, Iteration 304	 train_loss: 0.12 took: 9.12s
    Epoch 14, Iteration 320	 train_loss: 0.12 took: 9.59s
    Epoch 14, Iteration 336	 train_loss: 0.12 took: 10.03s
    Epoch 14, Iteration 352	 train_loss: 0.12 took: 10.50s
    Epoch 14, Iteration 368	 train_loss: 0.12 took: 10.96s
    Epoch 14, Iteration 384	 train_loss: 0.12 took: 11.42s
    Epoch 14, Iteration 400	 train_loss: 0.12 took: 11.87s
    Epoch 14, Iteration 416	 train_loss: 0.12 took: 12.33s
    Epoch 14, Iteration 432	 train_loss: 0.12 took: 12.80s
    Epoch 14, Iteration 448	 train_loss: 0.12 took: 13.25s
    Epoch 14, Iteration 464	 train_loss: 0.12 took: 13.73s
    
    Epoch [14/20], loss:0.1277
    Epoch 15, Iteration 16	 train_loss: 0.12 took: 0.48s
    Epoch 15, Iteration 32	 train_loss: 0.12 took: 0.95s
    Epoch 15, Iteration 48	 train_loss: 0.12 took: 1.39s
    Epoch 15, Iteration 64	 train_loss: 0.12 took: 1.88s
    Epoch 15, Iteration 80	 train_loss: 0.12 took: 2.34s
    Epoch 15, Iteration 96	 train_loss: 0.12 took: 2.82s
    Epoch 15, Iteration 112	 train_loss: 0.12 took: 3.27s
    Epoch 15, Iteration 128	 train_loss: 0.12 took: 3.73s
    Epoch 15, Iteration 144	 train_loss: 0.12 took: 4.18s
    Epoch 15, Iteration 160	 train_loss: 0.12 took: 4.67s
    Epoch 15, Iteration 176	 train_loss: 0.12 took: 5.16s
    Epoch 15, Iteration 192	 train_loss: 0.12 took: 5.63s
    Epoch 15, Iteration 208	 train_loss: 0.12 took: 6.11s
    Epoch 15, Iteration 224	 train_loss: 0.12 took: 6.59s
    Epoch 15, Iteration 240	 train_loss: 0.12 took: 7.10s
    Epoch 15, Iteration 256	 train_loss: 0.12 took: 7.58s
    Epoch 15, Iteration 272	 train_loss: 0.12 took: 8.08s
    Epoch 15, Iteration 288	 train_loss: 0.12 took: 8.56s
    Epoch 15, Iteration 304	 train_loss: 0.12 took: 9.04s
    Epoch 15, Iteration 320	 train_loss: 0.12 took: 9.51s
    Epoch 15, Iteration 336	 train_loss: 0.12 took: 9.99s
    Epoch 15, Iteration 352	 train_loss: 0.12 took: 10.45s
    Epoch 15, Iteration 368	 train_loss: 0.12 took: 10.94s
    Epoch 15, Iteration 384	 train_loss: 0.12 took: 11.41s
    Epoch 15, Iteration 400	 train_loss: 0.12 took: 11.88s
    Epoch 15, Iteration 416	 train_loss: 0.12 took: 12.35s
    Epoch 15, Iteration 432	 train_loss: 0.12 took: 12.83s
    Epoch 15, Iteration 448	 train_loss: 0.12 took: 13.33s
    Epoch 15, Iteration 464	 train_loss: 0.12 took: 13.80s
    
    Epoch [15/20], loss:0.1163
    Epoch 16, Iteration 16	 train_loss: 0.12 took: 0.51s
    Epoch 16, Iteration 32	 train_loss: 0.12 took: 0.97s
    Epoch 16, Iteration 48	 train_loss: 0.12 took: 1.48s
    Epoch 16, Iteration 64	 train_loss: 0.12 took: 1.97s
    Epoch 16, Iteration 80	 train_loss: 0.12 took: 2.45s
    Epoch 16, Iteration 96	 train_loss: 0.12 took: 2.92s
    Epoch 16, Iteration 112	 train_loss: 0.12 took: 3.41s
    Epoch 16, Iteration 128	 train_loss: 0.12 took: 3.89s
    Epoch 16, Iteration 144	 train_loss: 0.12 took: 4.36s
    Epoch 16, Iteration 160	 train_loss: 0.12 took: 4.82s
    Epoch 16, Iteration 176	 train_loss: 0.12 took: 5.29s
    Epoch 16, Iteration 192	 train_loss: 0.12 took: 5.78s
    Epoch 16, Iteration 208	 train_loss: 0.12 took: 6.26s
    Epoch 16, Iteration 224	 train_loss: 0.12 took: 6.73s
    Epoch 16, Iteration 240	 train_loss: 0.12 took: 7.20s
    Epoch 16, Iteration 256	 train_loss: 0.12 took: 7.69s
    Epoch 16, Iteration 272	 train_loss: 0.12 took: 8.15s
    Epoch 16, Iteration 288	 train_loss: 0.12 took: 8.62s
    Epoch 16, Iteration 304	 train_loss: 0.12 took: 9.10s
    Epoch 16, Iteration 320	 train_loss: 0.12 took: 9.58s
    Epoch 16, Iteration 336	 train_loss: 0.12 took: 10.04s
    Epoch 16, Iteration 352	 train_loss: 0.12 took: 10.51s
    Epoch 16, Iteration 368	 train_loss: 0.12 took: 11.02s
    Epoch 16, Iteration 384	 train_loss: 0.12 took: 11.50s
    Epoch 16, Iteration 400	 train_loss: 0.12 took: 11.98s
    Epoch 16, Iteration 416	 train_loss: 0.12 took: 12.46s
    Epoch 16, Iteration 432	 train_loss: 0.12 took: 12.93s
    Epoch 16, Iteration 448	 train_loss: 0.12 took: 13.41s
    Epoch 16, Iteration 464	 train_loss: 0.12 took: 13.89s
    
    Epoch [16/20], loss:0.1290
    Epoch 17, Iteration 16	 train_loss: 0.12 took: 0.48s
    Epoch 17, Iteration 32	 train_loss: 0.12 took: 0.97s
    Epoch 17, Iteration 48	 train_loss: 0.12 took: 1.46s
    Epoch 17, Iteration 64	 train_loss: 0.12 took: 1.95s
    Epoch 17, Iteration 80	 train_loss: 0.12 took: 2.44s
    Epoch 17, Iteration 96	 train_loss: 0.12 took: 2.89s
    Epoch 17, Iteration 112	 train_loss: 0.12 took: 3.33s
    Epoch 17, Iteration 128	 train_loss: 0.12 took: 3.79s
    Epoch 17, Iteration 144	 train_loss: 0.12 took: 4.24s
    Epoch 17, Iteration 160	 train_loss: 0.12 took: 4.70s
    Epoch 17, Iteration 176	 train_loss: 0.12 took: 5.15s
    Epoch 17, Iteration 192	 train_loss: 0.12 took: 5.64s
    Epoch 17, Iteration 208	 train_loss: 0.12 took: 6.08s
    Epoch 17, Iteration 224	 train_loss: 0.12 took: 6.55s
    Epoch 17, Iteration 240	 train_loss: 0.12 took: 7.05s
    Epoch 17, Iteration 256	 train_loss: 0.12 took: 7.51s
    Epoch 17, Iteration 272	 train_loss: 0.12 took: 7.97s
    Epoch 17, Iteration 288	 train_loss: 0.12 took: 8.42s
    Epoch 17, Iteration 304	 train_loss: 0.12 took: 8.90s
    Epoch 17, Iteration 320	 train_loss: 0.12 took: 9.36s
    Epoch 17, Iteration 336	 train_loss: 0.12 took: 9.82s
    Epoch 17, Iteration 352	 train_loss: 0.12 took: 10.27s
    Epoch 17, Iteration 368	 train_loss: 0.12 took: 10.75s
    Epoch 17, Iteration 384	 train_loss: 0.12 took: 11.21s
    Epoch 17, Iteration 400	 train_loss: 0.12 took: 11.70s
    Epoch 17, Iteration 416	 train_loss: 0.12 took: 12.19s
    Epoch 17, Iteration 432	 train_loss: 0.12 took: 12.68s
    Epoch 17, Iteration 448	 train_loss: 0.12 took: 13.16s
    Epoch 17, Iteration 464	 train_loss: 0.12 took: 13.64s
    
    Epoch [17/20], loss:0.1098
    Epoch 18, Iteration 16	 train_loss: 0.12 took: 0.48s
    Epoch 18, Iteration 32	 train_loss: 0.12 took: 0.96s
    Epoch 18, Iteration 48	 train_loss: 0.12 took: 1.43s
    Epoch 18, Iteration 64	 train_loss: 0.12 took: 1.91s
    Epoch 18, Iteration 80	 train_loss: 0.12 took: 2.38s
    Epoch 18, Iteration 96	 train_loss: 0.12 took: 2.86s
    Epoch 18, Iteration 112	 train_loss: 0.12 took: 3.35s
    Epoch 18, Iteration 128	 train_loss: 0.12 took: 3.79s
    Epoch 18, Iteration 144	 train_loss: 0.12 took: 4.27s
    Epoch 18, Iteration 160	 train_loss: 0.12 took: 4.72s
    Epoch 18, Iteration 176	 train_loss: 0.12 took: 5.19s
    Epoch 18, Iteration 192	 train_loss: 0.12 took: 5.64s
    Epoch 18, Iteration 208	 train_loss: 0.12 took: 6.11s
    Epoch 18, Iteration 224	 train_loss: 0.12 took: 6.56s
    Epoch 18, Iteration 240	 train_loss: 0.12 took: 7.01s
    Epoch 18, Iteration 256	 train_loss: 0.12 took: 7.46s
    Epoch 18, Iteration 272	 train_loss: 0.12 took: 7.94s
    Epoch 18, Iteration 288	 train_loss: 0.12 took: 8.40s
    Epoch 18, Iteration 304	 train_loss: 0.11 took: 8.87s
    Epoch 18, Iteration 320	 train_loss: 0.12 took: 9.33s
    Epoch 18, Iteration 336	 train_loss: 0.12 took: 9.78s
    Epoch 18, Iteration 352	 train_loss: 0.12 took: 10.26s
    Epoch 18, Iteration 368	 train_loss: 0.11 took: 10.71s
    Epoch 18, Iteration 384	 train_loss: 0.12 took: 11.19s
    Epoch 18, Iteration 400	 train_loss: 0.12 took: 11.68s
    Epoch 18, Iteration 416	 train_loss: 0.12 took: 12.15s
    Epoch 18, Iteration 432	 train_loss: 0.12 took: 12.63s
    Epoch 18, Iteration 448	 train_loss: 0.12 took: 13.11s
    Epoch 18, Iteration 464	 train_loss: 0.12 took: 13.63s
    
    Epoch [18/20], loss:0.1211
    Epoch 19, Iteration 16	 train_loss: 0.12 took: 0.47s
    Epoch 19, Iteration 32	 train_loss: 0.12 took: 0.93s
    Epoch 19, Iteration 48	 train_loss: 0.12 took: 1.39s
    Epoch 19, Iteration 64	 train_loss: 0.12 took: 1.85s
    Epoch 19, Iteration 80	 train_loss: 0.12 took: 2.30s
    Epoch 19, Iteration 96	 train_loss: 0.12 took: 2.76s
    Epoch 19, Iteration 112	 train_loss: 0.12 took: 3.23s
    Epoch 19, Iteration 128	 train_loss: 0.12 took: 3.72s
    Epoch 19, Iteration 144	 train_loss: 0.12 took: 4.20s
    Epoch 19, Iteration 160	 train_loss: 0.12 took: 4.66s
    Epoch 19, Iteration 176	 train_loss: 0.12 took: 5.11s
    Epoch 19, Iteration 192	 train_loss: 0.12 took: 5.57s
    Epoch 19, Iteration 208	 train_loss: 0.12 took: 6.05s
    Epoch 19, Iteration 224	 train_loss: 0.12 took: 6.51s
    Epoch 19, Iteration 240	 train_loss: 0.12 took: 6.97s
    Epoch 19, Iteration 256	 train_loss: 0.12 took: 7.45s
    Epoch 19, Iteration 272	 train_loss: 0.11 took: 7.93s
    Epoch 19, Iteration 288	 train_loss: 0.12 took: 8.40s
    Epoch 19, Iteration 304	 train_loss: 0.11 took: 8.87s
    Epoch 19, Iteration 320	 train_loss: 0.12 took: 9.33s
    Epoch 19, Iteration 336	 train_loss: 0.12 took: 9.81s
    Epoch 19, Iteration 352	 train_loss: 0.11 took: 10.29s
    Epoch 19, Iteration 368	 train_loss: 0.12 took: 10.74s
    Epoch 19, Iteration 384	 train_loss: 0.11 took: 11.22s
    Epoch 19, Iteration 400	 train_loss: 0.11 took: 11.71s
    Epoch 19, Iteration 416	 train_loss: 0.12 took: 12.21s
    Epoch 19, Iteration 432	 train_loss: 0.12 took: 12.67s
    Epoch 19, Iteration 448	 train_loss: 0.12 took: 13.15s
    Epoch 19, Iteration 464	 train_loss: 0.11 took: 13.63s
    
    Epoch [19/20], loss:0.1236
    Epoch 20, Iteration 16	 train_loss: 0.11 took: 0.49s
    Epoch 20, Iteration 32	 train_loss: 0.11 took: 0.94s
    Epoch 20, Iteration 48	 train_loss: 0.12 took: 1.42s
    Epoch 20, Iteration 64	 train_loss: 0.12 took: 1.90s
    Epoch 20, Iteration 80	 train_loss: 0.12 took: 2.38s
    Epoch 20, Iteration 96	 train_loss: 0.12 took: 2.87s
    Epoch 20, Iteration 112	 train_loss: 0.12 took: 3.35s
    Epoch 20, Iteration 128	 train_loss: 0.12 took: 3.83s
    Epoch 20, Iteration 144	 train_loss: 0.12 took: 4.32s
    Epoch 20, Iteration 160	 train_loss: 0.11 took: 4.77s
    Epoch 20, Iteration 176	 train_loss: 0.11 took: 5.22s
    Epoch 20, Iteration 192	 train_loss: 0.11 took: 5.72s
    Epoch 20, Iteration 208	 train_loss: 0.12 took: 6.22s
    Epoch 20, Iteration 224	 train_loss: 0.12 took: 6.70s
    Epoch 20, Iteration 240	 train_loss: 0.12 took: 7.16s
    Epoch 20, Iteration 256	 train_loss: 0.11 took: 7.62s
    Epoch 20, Iteration 272	 train_loss: 0.11 took: 8.06s
    Epoch 20, Iteration 288	 train_loss: 0.11 took: 8.52s
    Epoch 20, Iteration 304	 train_loss: 0.11 took: 8.96s
    Epoch 20, Iteration 320	 train_loss: 0.11 took: 9.43s
    Epoch 20, Iteration 336	 train_loss: 0.12 took: 9.90s
    Epoch 20, Iteration 352	 train_loss: 0.12 took: 10.40s
    Epoch 20, Iteration 368	 train_loss: 0.12 took: 10.87s
    Epoch 20, Iteration 384	 train_loss: 0.11 took: 11.35s
    Epoch 20, Iteration 400	 train_loss: 0.11 took: 11.83s
    Epoch 20, Iteration 416	 train_loss: 0.12 took: 12.30s
    Epoch 20, Iteration 432	 train_loss: 0.12 took: 12.77s
    Epoch 20, Iteration 448	 train_loss: 0.11 took: 13.25s
    Epoch 20, Iteration 464	 train_loss: 0.12 took: 13.71s
    
    Epoch [20/20], loss:0.1182



```
# Plots Loss over batches
plt.plot(batch_num,batch_loss)
plt.title('Loss of Autoencoder')
plt.legend(['train loss', 'validation loss'])
plt.xlabel('Batch number')
plt.ylabel('Loss')
plt.show()
```

### Anomaly or Not?
To determine, whether an image was an anomaly or not, it was necessary to look at the loss distribution of our non-anomalous dataset, and because it appeared like a relatively normal distribution, we decided that using a point's Z score was an anomaly. Z score can be calculated using:

\begin{align}
\ Z = \frac {x + \mu}{\sigma} 
\end{align}

If a Z score of an image is found to be greater than 2.5, then the image is an outlier, which gave resonable accuracy.


```
all_losses = []

# Finds the loss for every image in the database to be plotted as a histogram
for i in range(len(dataset)):
    img, _ = dataset[i]
    img = Variable(img.unsqueeze(1)).cuda()

    output = model(img)

    all_losses.append(criterion(output, img).item())

# Plots Loss distribution
sns.distplot(all_losses, kde=False)
plt.title("Loss Distribution")
plt.xlabel("Loss Value")
plt.ylabel("Quanitity")
plt.show()

# Converts list of losses to Float Tensor type
all_losses = torch.FloatTensor(all_losses)
std = all_losses.std().item() # Finds standard deviation
mean = all_losses.mean().item() # Finds mean

# Finds if an image is an anomaly based on its z score value. 2.5 is the threshold.
def is_anomaly(value, z_thresh = 2.5):
    zscore = (value-mean)/std # Calculates z score for value

    if zscore > z_thresh:
        return True
    return False
```

### Detecting Anomalies
For detecting anomalies, anomalous images had to be created. Using the function below, images in the dataset are combined to create images which clearly deviated from the rest. The show_in_out function plots an image before and after being passed through the model. It also prints its loss and whether or not it is an anomaly.


```
# Combines images from dataset to create anomalies.
def make_anomaly(img1, img2, img_dim = (28,28)):
    # Matrix to store combined image
    output = np.ones((img_dim[0]*img_dim[1],1))

    # Reshapes images for consistency
    img1 = img1.reshape(img_dim[0]*img_dim[1], 1).numpy()
    img2 = img2.reshape(img_dim[0]*img_dim[1], 1).numpy()

    # Iterates through each pixel and selects largest
    for i in range(len(img1)):
        output[i] = max([img1[i], img2[i]])
    
    # Converts numpy array to tensor
    output = torch.from_numpy(output).type(torch.FloatTensor)

    return output

#Plots an input and output of the model
def show_in_out(img, model, loss_func):
    # Plots anomaly
    plt.title("Input")
    disp_image(img.reshape(28,28))

    # Passes image into Model
    img = img.reshape((1,1,28,28))
    img = img.to("cuda")
    output = model(Variable(img).cuda())
    loss = loss_func(output, img).item()

    encoded = to_img(output.cpu().data)

    # Plots output
    plt.title("Output")
    disp_image(encoded.reshape(28,28))

    print(F"LOSS: {loss}")
    print(F"Outlier: {is_anomaly(loss)}")
```

#### Non Anomalous Image
Below is an image from the dataset, non-anomalous, being passed through the model. The input and output are relatively similar, and the image is not classified as an anomaly.


```
# Pulls image from MNist dataset and displays it
img, label = dataset[1]

show_in_out(img, model, criterion)
```

#### Anomalous Image
Below is an anomalous image created by combining different non-anomalous images. After the image was passed through the model, differences were accentuated leading to a greater loss. As a result, the model was able to predict that the image was an anomaly.


```
img1, label = dataset[1]
img2, label = dataset[2]
img3, label = dataset[10]

# Combines images at index 1, 2 and 10 in the dataset
img = make_anomaly(make_anomaly(img1,img2), img3)

show_in_out(img, model, criterion)

```

#### Anomalous Image Inappropriately Classified.
Below is an image created by combining different non-anomalous images. Though the image is considered an anomaly, the model did not classify the image as an anomaly. This is likely because the image is close enough to a badly drawn zero.


```
img1, label = dataset[1]
img2, label = dataset[10]

# Combines images at index 1 and 10 in the dataset
img = make_anomaly(img1,img2)

show_in_out(img, model, criterion)
```

#### Large Anomaly Appropriately Classified.
Below is generated noise passed, which was passed through the model and classified. The input and output are very different, leading to a large loss and ultimately, a classification as an anomaly.


```
# Generates random noisy image
noisy = np.random.random((1,1,28,28))

# Converts noisy image to tensor
noisy = torch.from_numpy(noisy).type(torch.FloatTensor)

show_in_out(noisy, model, criterion)
```

3c1c802641b8de064bdaac9d060c4a47a36c8c5b
ipynb
Jupyter Notebook
M1/AnomalyDetectionProject/Anomaly_Detection_on_MNIST_Chris_Shashank.ipynb
SSModelGit/MachineLearningOlin19
bd9f4a11f57c6151d266db76193ce9d923fbc4c9
MIT
M1/AnomalyDetectionProject/Anomaly_Detection_on_MNIST_Chris_Shashank.ipynb
SSModelGit/MachineLearningOlin19
bd9f4a11f57c6151d266db76193ce9d923fbc4c9
MIT
M1/AnomalyDetectionProject/Anomaly_Detection_on_MNIST_Chris_Shashank.ipynb
SSModelGit/MachineLearningOlin19
bd9f4a11f57c6151d266db76193ce9d923fbc4c9
MIT
Qwen/Qwen-72B
1. YES
2. YES