<a href="https://colab.research.google.com/github/TangJiahui/6.036_Machine_Learning/blob/main/MIT_6_036_Lab_11_RNN.ipynb" target="_parent"></a>

# **MIT 6.036 Spring 2020: Lab 11**

This colab notebook provides code and a framework for sections 2 and 3 of the lab.

## **Setup**

First, download the code distribution for this homework that contains test cases and helper functions.

Run the next code block to download and import the code for this lab.


```
!rm -rf code_for_lab11*
!rm -rf models
!rm -rf *.zip*
!rm -rf __*
!wget --quiet --no-check-certificate https://introml.odl.mit.edu/cat-soop/_static/6.036/labs/lab11/code_for_lab_11.zip
!unzip code_for_lab_11.zip
!mv ./code_for_lab11/* ./
```


```
from sm import *
from util import *
from code_for_lab11 import *
import numpy as np
import _pickle as cPickle
m = cPickle.load(open('models/food_rnn.p', 'rb'))
```

## **Section 2**

### Generating sequences

We are interested in building an RNN model that can predict the next
element in a sequence (this is sometimes referred to as a "language"
model).    The particular form of the model that we will look at is:

<displaymath>
\begin{align}
x_t &= \phi(c_{t-1})\\
s_t &= \tanh(W^{ss} s_{t-1} + W^{sx} x_t + W^{ss}_0)\\
p_t &= \text{softmax}(W^o s_t + W^o_0)
\end{align}
</displaymath>

**Please refer to the lab's prompt for more definitions and details about this section.**

As we just said, you can find more details in the lab's prompt, but here some extra details:

Training is as follows:
<ul>
<li> For each sequence in the input data, it feeds in character <math>t-1</math> from
  the training data and predicts character $t$.  </li>
</ul>

Generation is as follows:
<ul>
<li> If interactive is False: We start with the `start` symbol ('.'), </li>
<li>If interactive is True: We start with the symbol you enter in (which must be present in the training word or be the start symbol), it predicts </li>

The model then predicts a next
  character based on the softmax distribution in the trained model,
  then it feeds that character into the model and repeats until an
  <i>end</i> symbol ('\n') is generated.
</ul>




For each of 

*   "aaaaaaaaaa"
*   "aabaaabbaaaababaabaa"
*   "abcdefghijklmnopqrstuvwxyz"
*   "abcabcabcabcabc"

train an RNN and assess the difficulty of learning each string. You may control *num_hidden* and *num_steps* to facilitate your training in the code below.

**Note**: The output of `test_word` is 100 sequences produced by running the generation process on a trained model.



```
test_word(word="aaaaaaaaaa", interactive=False, num_hidden=5, num_steps=1000, step_size=0.005)
```


```
test_word(word="aabaaabbaaaababaabaa", interactive = False, num_hidden=9, num_steps=15000, step_size=0.005)
```


```
test_word(word="abcdefghijklmnopqrstuvwxyz", num_hidden=5, num_steps=10000, step_size=0.005)
```


```
test_word(word="abcabcabcabcabc", interactive = False, num_hidden=5, num_steps=10000, step_size=0.005)
```

## Problem 2D

The *test_company_names* function uses a file <a href="https://docs.google.com/spreadsheets/d/16wbhwRwu5AP4gydX9buHZj884v48HURaalYtKmdT8r0/edit?usp=sharing">companies.txt</a> of company names for training and generates new names.

Try it in interactive mode; it's more fun. If you have time, experiment with different values of `num_steps`; more steps gives better results. But **be careful not to use too many steps** (no more than the preloaded 20000 steps).

**In running `test_company_names`, you can try with `train=True` where you can train the RNN yourself with this option, but note that this will take a few minutes to run, and you may run out of memory. If this occurs, please restart the kernel under "Runtime"/"Restart runtime". The default is to run  `test_company_names` with `train=False`; which simply loads a pre-trained model.**

Please note that when we train our models, we need to decide what will be our vocabulary size (i.e. how many classes/characters/words our model will learn to predict). Hence, it is possible that when you input a character/word that is out of that vocabulary (depending on how the model is trained -- character-based or word-based), the model crashes (in more sophisticated solutions we could use a `<unknown>` token to represent **tokens out of vocabulary.**)

These models are trained by minimizing NLL.




```
train = False

test_company_names(interactive=False, interactive_top5=False, num_hidden=150, 
                   num_steps=20000, step_size=0.001, train=train)

#Trained with num_hidden=150, num_steps=20000,
#             step_size=.001, split=0
```

## Problem 2E

The *test_food* function uses a file <a href="https://docs.google.com/spreadsheets/d/12eomCfekSdqTlOB4Xwp3n7_FN0SVdVrTDLVUBq-1jyw/edit?usp=sharing">food.txt</a> of recipe names. Try it in interactive mode; it's more fun. Experiment with different values of num_steps if you have time; more steps gives better results. Note the difference between starting with a capital letter versus lower case letter.




```
train = False

test_food(interactive = False, interactive_top5 = False, num_hidden=150, 
          num_steps=20000, step_size=0.001, train=train)

#Trained with num_hidden = 150, num_steps = 20000,
#             step_size = .001, split=0
```

## Problem 2F

Now run one of <code>test_food, test_company_names</code> with <code>interactive_top5=True</code>; play with them for a while. Can you notice the mechanism of how these functions generate their output? 

Use the cell below to run one of <code> test_food, test_company_names</code> with <code>interactive_top5=True</code>!

- What is the mechanism by which these RNNs generate their output? More specifically, what character does the trained RNN seem to output at each location?
- Also, what would be a reasonable criterion for the above-trained RNNs to stop generating more characters? When does it seem to stop? Be prepared to explain this during checkoff.


```
train = False

#test_food(interactive = True, interactive_top5 = True, num_hidden=150, 
#          num_steps=20000, step_size=0.001, train=train)

# Pre trained with split=0, num_hidden = 150,
#                   num_steps = 20000, step_size = .001, 
```


```
train = False

#test_company_names(interactive = True, interactive_top5 = True, num_hidden=150, 
#                  num_steps=20000, step_size=0.001,train=train)

# Pre trained with split=0, num_hidden = 150,
#                   num_steps = 20000, step_size = .001, 
```

6ff3981c757bebe6ab7e54ab2318718ac11001c2
ipynb
Jupyter Notebook
MIT_6_036_Lab11_RNN.ipynb
cobase2010/6.036_Machine_Learning
90ec6867a263f0067277a1e35835ef8436138b6e
MIT
2020-12-20T21:26:01.000Z
2022-03-22T12:18:56.000Z
MIT_6_036_Lab11_RNN.ipynb
cobase2010/6.036_Machine_Learning
90ec6867a263f0067277a1e35835ef8436138b6e
MIT
MIT_6_036_Lab11_RNN.ipynb
cobase2010/6.036_Machine_Learning
90ec6867a263f0067277a1e35835ef8436138b6e
MIT
2020-11-24T07:12:57.000Z
2022-02-16T07:40:45.000Z
Qwen/Qwen-72B
1. YES
2. YES