Natural Language Processing From Scratch

Bruno Gonçalves

bgoncalves@gmail.com

@bgoncalves

# Lesson I - Text representation

In this lesson we will see in some details how we can best represent text in our application. Let's start by importing the modules we will be using:

```python
import string
from collections import Counter
from pprint import pprint
import gzip
import matplotlib.pyplot as plt
import numpy as np

%matplotlib inline
```

We choose a well known nursery rhyme, that has the added distinction of having been the first audio ever recorded, to be the short snippet of text that we will use in our examples:

```python
text = """Mary had a little lamb, little lamb,
    little lamb. Mary had a little lamb
    whose fleece was white as snow.
    And everywhere that Mary went
    Mary went, Mary went. Everywhere
    that Mary went,
    The lamb was sure to go"""
```

## Tokenization

The first step in any analysis is to tokenize the text. What this means is that we will extract all the individual words in the text. For the sake of simplicity, we will assume that our text is well formed and that our words are delimited either by white space or punctuation characters.

```python
def extract_words(text):
    temp = text.split() # Split the text on whitespace
    text_words = []

    for word in temp:
        # Remove any punctuation characters present in the beginning of the word
        while word[0] in string.punctuation:
            word = word[1:]

        # Remove any punctuation characters present in the end of the word
        while word[-1] in string.punctuation:
            word = word[:-1]

        # Append this word into our list of words.
        text_words.append(word.lower())

    return text_words
```

After this step we now have our text represented as an array of individual, lowercase, words:

```python
text_words = extract_words(text)
print(text_words)
```

    ['mary', 'had', 'a', 'little', 'lamb', 'little', 'lamb', 'little', 'lamb', 'mary', 'had', 'a', 'little', 'lamb', 'whose', 'fleece', 'was', 'white', 'as', 'snow', 'and', 'everywhere', 'that', 'mary', 'went', 'mary', 'went', 'mary', 'went', 'everywhere', 'that', 'mary', 'went', 'the', 'lamb', 'was', 'sure', 'to', 'go']

As we saw during the video, this is a wasteful way to represent text. We can be much more efficient by representing each word by a number

```python
word_dict = {}
word_list = []
vocabulary_size = 0
text_tokens = []

for word in text_words:
    # If we are seeing this word for the first time, create an id for it and added it to our word dictionary
    if word not in word_dict:
        word_dict[word] = vocabulary_size
        word_list.append(word)
        vocabulary_size += 1

    # add the token corresponding to the current word to the tokenized text.
    text_tokens.append(word_dict[word])
```

When we were tokenizing our text, we also generated a dictionary **word_dict** that maps words to integers and a **word_list** that maps each integer to the corresponding word.

```python
print("Word list:", word_list, "\n\n Word dictionary:")
pprint(word_dict)
```

    Word list: ['mary', 'had', 'a', 'little', 'lamb', 'whose', 'fleece', 'was', 'white', 'as', 'snow', 'and', 'everywhere', 'that', 'went', 'the', 'sure', 'to', 'go']

     Word dictionary:
    {'a': 2,
     'and': 11,
     'as': 9,
     'everywhere': 12,
     'fleece': 6,
     'go': 18,
     'had': 1,
     'lamb': 4,
     'little': 3,
     'mary': 0,
     'snow': 10,
     'sure': 16,
     'that': 13,
     'the': 15,
     'to': 17,
     'was': 7,
     'went': 14,
     'white': 8,
     'whose': 5}

These two datastructures already proved their usefulness when we converted our text to a list of tokens.

```python
print(text_tokens)
```

    [0, 1, 2, 3, 4, 3, 4, 3, 4, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 0, 14, 0, 14, 0, 14, 12, 13, 0, 14, 15, 4, 7, 16, 17, 18]

Unfortunately, while this representation is convenient for memory reasons it has some severe limitations. Perhaps the most important of which is the fact that computers naturally assume that numbers can be operated on mathematically (by addition, subtraction, etc) in a way that doesn't match our understanding of words.

## One-hot encoding

One typical way of overcoming this difficulty is to represent each word by a one-hot encoded vector where every element is zero except the one corresponding to a specific word.

```python
def one_hot(word, word_dict):
    """
        Generate a one-hot encoded vector corresponding to *word*
    """

    vector = np.zeros(len(word_dict))
    vector[word_dict[word]] = 1

    return vector
```

So, for example, the word "fleece" would be represented by:

```python
fleece_hot = one_hot("fleece", word_dict)
print(fleece_hot)
```

    [0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]

This vector has every element set to zero, except element 6, since:

```python
print(word_dict["fleece"])
fleece_hot[6] == 1
```

    6

    True

## Bag of words

We can now use the one-hot encoded vector for each word to produce a vector representation of our original text, by simply adding up all the one-hot encoded vectors:

```python
text_vector1 = np.zeros(vocabulary_size)

for word in text_words:
    hot_word = one_hot(word, word_dict)
    text_vector1 += hot_word

print(text_vector1)
```

    [6. 2. 2. 4. 5. 1. 1. 2. 1. 1. 1. 1. 2. 2. 4. 1. 1. 1. 1.]

In practice, we can also easily skip the encoding step at the word level by using the *word_dict* defined above:

```python
text_vector = np.zeros(vocabulary_size)

for word in text_words:
    text_vector[word_dict[word]] += 1

print(text_vector)
```

    [6. 2. 2. 4. 5. 1. 1. 2. 1. 1. 1. 1. 2. 2. 4. 1. 1. 1. 1.]

Naturally, this approach is completely equivalent to the previous one and has the added advantage of being more efficient in terms of both speed and memory requirements.

This is known as the __bag of words__ representation of the text. It should be noted that these vectors simply contains the number of times each word appears in our document, so we can easily tell that the word *mary* appears exactly 6 times in our little nursery rhyme.

```python
text_vector[word_dict["mary"]]
```

    6.0

A more pythonic (and efficient) way of producing the same result is to use the standard __Counter__ module:

```python
word_counts = Counter(text_words)
pprint(word_counts)
```

    Counter({'mary': 6,
             'lamb': 5,
             'little': 4,
             'went': 4,
             'that': 2,
             'had': 2,
             'a': 2,
             'everywhere': 2,
             'was': 2,
             'white': 1,
             'snow': 1,
             'sure': 1,
             'to': 1,
             'the': 1,
             'whose': 1,
             'go': 1,
             'as': 1,
             'fleece': 1,
             'and': 1})

From which we can easily generate the __text_vector__ and __word_dict__ data structures:

```python
items = list(word_counts.items())

# Extract word dictionary and vector representation
word_dict2 = dict([[items[i][0], i] for i in range(len(items))])
text_vector2 = [items[i][1] for i in range(len(items))]
```

And let's take a look at them:

```python
print("Text vector:", text_vector2, "\n\nWord dictionary:")
pprint(word_dict2)
```

    Text vector: [1, 2, 4, 5, 1, 2, 1, 4, 6, 1, 1, 1, 1, 1, 1, 2, 1, 2, 2]

    Word dictionary:
    {'a': 15,
     'and': 16,
     'as': 13,
     'everywhere': 17,
     'fleece': 14,
     'go': 12,
     'had': 5,
     'lamb': 3,
     'little': 2,
     'mary': 8,
     'snow': 4,
     'sure': 6,
     'that': 1,
     'the': 10,
     'to': 9,
     'was': 18,
     'went': 7,
     'white': 0,
     'whose': 11}

The results using this approach are slightly different than the previous ones, because the words are mapped to different integer ids but the corresponding values are the same:

```python
for word in word_dict.keys():
    if text_vector[word_dict[word]] != text_vector2[word_dict2[word]]:
        print("Error!")
```

As expected, there are no differences!

## Term Frequency

The bag of words vector representation introduced above relies simply on the frequency of occurence of each word. Following a long tradition of giving fancy names to simple ideas, this is known as __Term Frequency__.

Intuitively, we expect the the frequency with which a given word is mentioned should correspond to the relevance of that word for the piece of text we are considering. For example, **Mary** is a pretty important word in our little nursery rhyme and indeed it is the one that occurs the most often:

```python
sorted(items, key=lambda x:x[1], reverse=True)
```

    [('mary', 6),
     ('lamb', 5),
     ('little', 4),
     ('went', 4),
     ('that', 2),
     ('had', 2),
     ('a', 2),
     ('everywhere', 2),
     ('was', 2),
     ('white', 1),
     ('snow', 1),
     ('sure', 1),
     ('to', 1),
     ('the', 1),
     ('whose', 1),
     ('go', 1),
     ('as', 1),
     ('fleece', 1),
     ('and', 1)]

However, it's hard to draw conclusions from such a small piece of text. Let us consider a significantly larger piece of text, the first 100 MB of the english Wikipedia from: http://mattmahoney.net/dc/textdata. For the sake of convenience, text8.gz has been included in this repository in the **data/** directory. We start by loading it's contents into memory as an array of words:

```python
data = []

for line in gzip.open("data/text8.gz", 'rt'):
    data.extend(line.strip().split())
```

Now let's take a look at the most common words in this large corpus:

```python
counts = Counter(data)

sorted_counts = sorted(list(counts.items()), key=lambda x:x[1], reverse=True)

for word, count in sorted_counts[:10]:
    print(word, count)
```

    the 1061396
    of 593677
    and 416629
    one 411764
    in 372201
    a 325873
    to 316376
    zero 264975
    nine 250430
    two 192644

Surprisingly, we find that the most common words are not particularly meaningful. Indeed, this is a common occurence in Natural Language Processing. The most frequent words are typically auxiliaries required due to gramatical rules.

On the other hand, there is also a large number of words that occur very infrequently as can be easily seen by glancing at the word freqency distribution.

```python
dist = Counter(counts.values())
dist = list(dist.items())
dist.sort(key=lambda x:x[0])
dist = np.array(dist)

norm = np.dot(dist.T[0], dist.T[1])

plt.loglog(dist.T[0], dist.T[1]/norm)
plt.xlabel("count")
plt.ylabel("P(count)")
plt.title("Word frequency distribution")
```

## Stopwords

One common technique to simplify NLP tasks is to remove what are known as Stopwords, words that are very frequent but not meaningful. If we simply remove the most common 100 words, we significantly reduce the amount of data we have to consider while losing little information.

```python
stopwords = set([word for word, count in sorted_counts[:100]])

clean_data = []

for word in data:
    if word not in stopwords:
        clean_data.append(word)

print("Original size:", len(data))
print("Clean size:", len(clean_data))
print("Reduction:", 1-len(clean_data)/len(data))
```

    Original size: 17005207
    Clean size: 9006229
    Reduction: 0.470384041782026

Wow, our dataset size was reduced almost in half!

In practice, we don't simply remove the most common words in our corpus but rather a manually curate list of stopwords. Lists for dozens of languages and applications can easily be found online.

## Term Frequency/Inverse Document Frequency

One way of determining of the relative importance of a word is to see how often it appears across multiple documents. Words that are relevant to a specific topic are more likely to appear in documents about that topic and much less in documents about other topics. On the other hand, less meaningful words (like **the**) will be common across documents about any subject.

To measure the document frequency of a word we will need to have multiple documents. For the sake of simplicity, we will treat each sentence of our nursery rhyme as an individual document:

```python
corpus_text = text.split('.')
corpus_words = []

for document in corpus_text:
    doc_words = extract_words(document)
    corpus_words.append(doc_words)
```

Now our corpus is represented as a list of word lists, where each list is just the word representation of the corresponding sentence:

```python
pprint(corpus_words)
```

    [['mary', 'had', 'a', 'little', 'lamb', 'little', 'lamb', 'little', 'lamb'],
     ['mary',
      'had',
      'a',
      'little',
      'lamb',
      'whose',
      'fleece',
      'was',
      'white',
      'as',
      'snow'],
     ['and', 'everywhere', 'that', 'mary', 'went', 'mary', 'went', 'mary', 'went'],
     ['everywhere',
      'that',
      'mary',
      'went',
      'the',
      'lamb',
      'was',
      'sure',
      'to',
      'go']]

Let us now calculate the number of documents in which each word appears:

```python
document_count = {}

for document in corpus_words:
    word_set = set(document)

    for word in word_set:
        document_count[word] = document_count.get(word, 0) + 1

pprint(document_count)
```

    {'a': 2,
     'and': 1,
     'as': 1,
     'everywhere': 2,
     'fleece': 1,
     'go': 1,
     'had': 2,
     'lamb': 3,
     'little': 2,
     'mary': 4,
     'snow': 1,
     'sure': 1,
     'that': 2,
     'the': 1,
     'to': 1,
     'was': 2,
     'went': 2,
     'white': 1,
     'whose': 1}

As we can see, the word __Mary__ appears in all 4 of our documents, making it useless when it comes to distinguish between the different sentences. On the other hand, words like __white__ which appear in only one document are very discriminative. Using this approach we can define a new quantity, the ___Inverse Document Frequency__ that tells us how frequent a word is across the documents in a specific corpus:

```python
def inv_doc_freq(corpus_words):
    number_docs = len(corpus_words)

    document_count = {}

    for document in corpus_words:
        word_set = set(document)

        for word in word_set:
            document_count[word] = document_count.get(word, 0) + 1

    IDF = {}

    for word in document_count:
        IDF[word] = np.log(number_docs/document_count[word])

    return IDF
```

Where we followed the convention of using the logarithm of the inverse document frequency. This has the numerical advantage of avoiding to have to handle small fractional numbers.

We can easily see that the IDF gives a smaller weight to the most common words and a higher weight to the less frequent:

```python
IDF = inv_doc_freq(corpus_words)

pprint(IDF)
```

    {'a': 0.6931471805599453,
     'and': 1.3862943611198906,
     'as': 1.3862943611198906,
     'everywhere': 0.6931471805599453,
     'fleece': 1.3862943611198906,
     'go': 1.3862943611198906,
     'had': 0.6931471805599453,
     'lamb': 0.28768207245178085,
     'little': 0.6931471805599453,
     'mary': 0.0,
     'snow': 1.3862943611198906,
     'sure': 1.3862943611198906,
     'that': 0.6931471805599453,
     'the': 1.3862943611198906,
     'to': 1.3862943611198906,
     'was': 0.6931471805599453,
     'went': 0.6931471805599453,
     'white': 1.3862943611198906,
     'whose': 1.3862943611198906}

As expected **Mary** has the smallest weight of all words 0, meaning that it is effectively removed from the dataset. You can consider this as a way of implicitly identify and remove stopwords. In case you do want to keep even the words that appear in every document, you can just add a 1. to the argument of the logarithm above:

\begin{equation}
\log\left[1+\frac{N_d}{N_d\left(w\right)}\right]
\end{equation}

When we multiply the term frequency of each word by it's inverse document frequency, we have a good way of quantifying how relevant a word is to understand the meaning of a specific document.

```python
def tf_idf(corpus_words):
    IDF = inv_doc_freq(corpus_words)

    TFIDF = []

    for document in corpus_words:
        TFIDF.append(Counter(document))

    for document in TFIDF:
        for word in document:
            document[word] = document[word]*IDF[word]

    return TFIDF
```

```python
tf_idf(corpus_words)
```

    [Counter({'a': 0.6931471805599453,
              'had': 0.6931471805599453,
              'lamb': 0.8630462173553426,
              'little': 2.0794415416798357,
              'mary': 0.0}),
     Counter({'a': 0.6931471805599453,
              'as': 1.3862943611198906,
              'fleece': 1.3862943611198906,
              'had': 0.6931471805599453,
              'lamb': 0.28768207245178085,
              'little': 0.6931471805599453,
              'mary': 0.0,
              'snow': 1.3862943611198906,
              'was': 0.6931471805599453,
              'white': 1.3862943611198906,
              'whose': 1.3862943611198906}),
     Counter({'and': 1.3862943611198906,
              'everywhere': 0.6931471805599453,
              'mary': 0.0,
              'that': 0.6931471805599453,
              'went': 2.0794415416798357}),
     Counter({'everywhere': 0.6931471805599453,
              'go': 1.3862943611198906,
              'lamb': 0.28768207245178085,
              'mary': 0.0,
              'sure': 1.3862943611198906,
              'that': 0.6931471805599453,
              'the': 1.3862943611198906,
              'to': 1.3862943611198906,
              'was': 0.6931471805599453,
              'went': 0.6931471805599453})]

Now we finally have a vector representation of each of our documents that takes the informational contributions of each word into account. Each of these vectors provides us with a unique representation of each document, in the context (corpus) in which it occurs, making it posssible to define the similarity of two documents, etc.

## Porter Stemmer

There is still, however, one issue with our approach to representing text. Since we treat each word as a unique token and completely independently from all others, for large documents we will end up with many variations of the same word such as verb conjugations, the corresponding adverbs and nouns, etc.

One way around this difficulty is to use stemming algorithm to reduce words to their root (or stem) version. The most famous Stemming algorithm is known as the **Porter Stemmer** and was introduced by Martin Porter in 1980 [Program 14, 130 (1980)](https://dl.acm.org/citation.cfm?id=275705)

The algorithm starts by defining consonants (C) and vowels (V):

```python
V = set('aeiouy')
C = set('bcdfghjklmnpqrstvwxz')
```

The stem of a word is what is left of that word after a speficic ending has been removed. A function to do this is easy to implement:

```python
def get_stem(suffix, word):
    """
        Extract the stem of a word
    """

    if word.lower().endswith(suffix.lower()): # Case insensitive comparison
        return word[:-len(suffix)]

    return None
```

It also defines words (or stems) to be sequences of vowels and consonants of the form:

\begin{equation}
[C](VC)^m[V]
\end{equation}

where $m$ is called the **measure** of the word and [] represent optional sections.

```python
def measure(orig_word):
    """
        Calculate the "measure" m of a word or stem, according to the Porter Stemmer algorthim
    """

    word = orig_word.lower()

    optV = False
    optC = False
    VC = False
    m = 0

    pos = 0

    # We can think of this implementation as a simple finite state machine
    # looks for sequences of vowels or consonants depending of the state
    # in which it's in, while keeping track of how many VC sequences it
    # has encountered.
    # The presence of the optional V and C portions is recorded in the
    # optV and optC booleans.

    # We're at the initial state.
    # gobble up all the optional consonants at the beginning of the word
    while pos < len(word) and word[pos] in C:
        pos += 1
        optC = True

    while pos < len(word):
        # Now we know that the next state must be a vowel
        while pos < len(word) and word[pos] in V:
            pos += 1
            optV = True

        # Followd by a consonant
        while pos < len(word) and word[pos] in C:
            pos += 1
            optV = False

        # If a consonant was found, then we matched VC
        # so we should increment m by one. Otherwise,
        # optV remained true and we simply had a dangling
        # V sequence.
        if not optV:
            m += 1

    return m
```

Let's consider a simple example. The word __crepusculars__ should have measure 4:

[cr] (ep) (usc) (ul) (ars)

and indeed it does.

```python
word = "crepusculars"
print(measure(word))
```

    4

The Porter algorithm sequentially applies a series of transformation rules over a series of 5 steps (step 1 is divided in 3 substeps and step 5 in 2). The rules are only applied if a certain condition is true.

In addition to possibily specifying a requirement on the measure of a word, conditions can make use of different boolean functions as well:

```python
def ends_with(char, stem):
    """
        Checks the ending of the word
    """
    return stem[-1] == char

def double_consonant(stem):
    """
        Checks the ending of a word for a double consonant
    """
    if len(stem) < 2:
        return False

    if stem[-1] in C and stem[-2] == stem[-1]:
        return True

    return False

def contains_vowel(stem):
    """
        Checks if a word contains a vowel or not
    """
    return len(set(stem) & V) > 0
```

Finally, we define a function to apply a specific rule to a word or stem:

```python
def apply_rule(condition, suffix, replacement, word):
    """
        Apply Porter Stemmer rule.
        if "condition" is True replace "suffix" by "replacement" in "word"
    """

    stem = get_stem(suffix, word)

    if stem is not None and condition is True:
        # Remove the suffix
        word = stem

        # Add the replacement suffix, if any
        if replacement is not None:
            word += replacement

    return word
```

Now we can see how rules can be applied. For example, this rule, from step 1b is successfully applied to __pastered__:

```python
word = "plastered"
suffix = "ed"
stem = get_stem(suffix, word)
apply_rule(contains_vowel(stem), suffix, None, word)
```

    'plaster'

While try applying the same rule to **bled** will fail to pass the condition resulting in no change.

```python
word = "bled"
suffix = "ed"
stem = get_stem(suffix, word)
apply_rule(contains_vowel(stem), suffix, None, word)
```

    'bled'

For a more complex example, we have, in Step 4:

```python
word = "adoption"
suffix = "ion"
stem = get_stem(suffix, word)
apply_rule(measure(stem) > 1 and (ends_with("s", stem) or ends_with("t", stem)), suffix, None, word)
```

    'adopt'

In total, the Porter Stemmer algorithm (for the English language) applies several dozen rules (see https://tartarus.org/martin/PorterStemmer/def.txt for a complete list). Implementing all of them is both tedious and error prone, so we abstain from providing a full implementation of the algorithm here. High quality implementations can be found in all major NLP libraries such as [NLTK](http://www.nltk.org/howto/stem.html).

The dificulties of defining matching rules to arbitrary text cannot be fully resolved without the use of Regular Expressions (typically implemented as Finite State Machines like our __measure__ implementation above), a more advanced topic that is beyond the scope of this course.


#Basic Setup

##Installing required packages

For this task, I am using Qiskit along with PyTorch to complete it.

Installing packages:

* Qiskit v 0.18.3
* Pytorch v 1.9.0+cu102

```python
!pip install qiskit
!pip install torch
```

    Collecting qiskit
      Downloading qiskit-0.30.1.tar.gz (13 kB)
    Collecting qiskit-terra==0.18.3
      Downloading qiskit_terra-0.18.3-cp37-cp37m-manylinux2010_x86_64.whl (6.1 MB)
         |████████████████████████████████| 6.1 MB 4.6 MB/s
    Collecting qiskit-aer==0.9.0
      Downloading qiskit_aer-0.9.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (17.9 MB)
         |████████████████████████████████| 17.9 MB 123 kB/s
    Collecting qiskit-ibmq-provider==0.16.0
      Downloading qiskit_ibmq_provider-0.16.0-py3-none-any.whl (235 kB)
         |████████████████████████████████| 235 kB 44.8 MB/s
    Collecting qiskit-ignis==0.6.0
      Downloading qiskit_ignis-0.6.0-py3-none-any.whl (207 kB)
         |████████████████████████████████| 207 kB 71.6 MB/s
    Collecting qiskit-aqua==0.9.5
      Downloading qiskit_aqua-0.9.5-py3-none-any.whl (2.1 MB)
         |████████████████████████████████| 2.1 MB 25.9 MB/s
    Requirement already satisfied: scipy>=1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-aer==0.9.0->qiskit) (1.4.1)
    Collecting pybind11>=2.6
      Downloading pybind11-2.7.1-py2.py3-none-any.whl (200 kB)
         |████████████████████████████████| 200 kB 57.9 MB/s
    Requirement already satisfied: numpy>=1.16.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-aer==0.9.0->qiskit) (1.19.5)
    Requirement already satisfied: psutil>=5 in /usr/local/lib/python3.7/dist-packages (from qiskit-aqua==0.9.5->qiskit) (5.4.8)
    Requirement already satisfied: h5py<3.3.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-aqua==0.9.5->qiskit) (3.1.0)
    Collecting retworkx>=0.8.0
      Downloading retworkx-0.10.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.4 MB)
         |████████████████████████████████| 1.4 MB 46.2 MB/s
    Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from qiskit-aqua==0.9.5->qiskit) (1.1.5)
    Collecting yfinance>=0.1.62
      Downloading yfinance-0.1.63.tar.gz (26 kB)
    Collecting docplex>=2.21.207
      Downloading docplex-2.22.213.tar.gz (634 kB)
         |████████████████████████████████| 634 kB 62.6 MB/s
    Collecting quandl
      Downloading Quandl-3.6.1-py2.py3-none-any.whl (26 kB)
    Requirement already satisfied: fastdtw<=0.3.4 in /usr/local/lib/python3.7/dist-packages (from qiskit-aqua==0.9.5->qiskit) (0.3.4)
    Requirement already satisfied: sympy>=1.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-aqua==0.9.5->qiskit) (1.7.1)
    Requirement already satisfied: setuptools>=40.1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-aqua==0.9.5->qiskit) (57.4.0)
    Requirement already satisfied: scikit-learn>=0.20.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-aqua==0.9.5->qiskit) (0.22.2.post1)
    Collecting dlx<=1.0.4
      Downloading dlx-1.0.4.tar.gz (5.5 kB)
    Collecting requests-ntlm>=1.1.0
      Downloading requests_ntlm-1.1.0-py2.py3-none-any.whl (5.7 kB)
    Requirement already satisfied: python-dateutil>=2.8.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.16.0->qiskit) (2.8.2)
    Collecting websocket-client>=1.0.1
      Downloading websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
         |████████████████████████████████| 52 kB 1.1 MB/s
    Requirement already satisfied: urllib3>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.16.0->qiskit) (1.24.3)
    Requirement already satisfied: requests>=2.19 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.16.0->qiskit) (2.23.0)
    Collecting symengine>0.7
      Downloading symengine-0.8.1-cp37-cp37m-manylinux2010_x86_64.whl (38.2 MB)
         |████████████████████████████████| 38.2 MB 21 kB/s
    Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.18.3->qiskit) (2.6.0)
    Collecting python-constraint>=1.4
      Downloading python-constraint-1.4.0.tar.bz2 (18 kB)
    Collecting fastjsonschema>=2.10
      Downloading fastjsonschema-2.15.1-py3-none-any.whl (21 kB)
    Collecting ply>=3.10
      Downloading ply-3.11-py2.py3-none-any.whl (49 kB)
         |████████████████████████████████| 49 kB 5.6 MB/s
    Collecting tweedledum<2.0,>=1.1
      Downloading tweedledum-1.1.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (943 kB)
         |████████████████████████████████| 943 kB 36.8 MB/s
    Requirement already satisfied: dill>=0.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.18.3->qiskit) (0.3.4)
    Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from docplex>=2.21.207->qiskit-aqua==0.9.5->qiskit) (1.15.0)
    Requirement already satisfied: cached-property in /usr/local/lib/python3.7/dist-packages (from h5py<3.3.0->qiskit-aqua==0.9.5->qiskit) (1.5.2)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.16.0->qiskit) (2.10)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.16.0->qiskit) (3.0.4)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.16.0->qiskit) (2021.5.30)
    Collecting ntlm-auth>=1.0.2
      Downloading ntlm_auth-1.5.0-py2.py3-none-any.whl (29 kB)
    Collecting cryptography>=1.3
      Downloading cryptography-35.0.0-cp36-abi3-manylinux_2_24_x86_64.whl (3.5 MB)
         |████████████████████████████████| 3.5 MB 20.1 MB/s
    Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.7/dist-packages (from cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.16.0->qiskit) (1.14.6)
    Requirement already satisfied: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi>=1.12->cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.16.0->qiskit) (2.20)
    Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.20.0->qiskit-aqua==0.9.5->qiskit) (1.0.1)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy>=1.3->qiskit-aqua==0.9.5->qiskit) (1.2.1)
    Requirement already satisfied: multitasking>=0.0.7 in /usr/local/lib/python3.7/dist-packages (from yfinance>=0.1.62->qiskit-aqua==0.9.5->qiskit) (0.0.9)
    Collecting lxml>=4.5.1
      Downloading lxml-4.6.3-cp37-cp37m-manylinux2014_x86_64.whl (6.3 MB)
         |████████████████████████████████| 6.3 MB 21.0 MB/s
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->qiskit-aqua==0.9.5->qiskit) (2018.9)
    Collecting inflection>=0.3.1
      Downloading inflection-0.5.1-py2.py3-none-any.whl (9.5 kB)
    Requirement already satisfied: more-itertools in /usr/local/lib/python3.7/dist-packages (from quandl->qiskit-aqua==0.9.5->qiskit) (8.10.0)
    Building wheels for collected packages: qiskit, dlx, docplex, python-constraint, yfinance
      Building wheel for qiskit (setup.py) ... done
      Created wheel for qiskit: filename=qiskit-0.30.1-py3-none-any.whl size=11697 sha256=21aa425acb8bcabec1f896b240c31a0e26032e5857035bfc4afedb547782591a
      Stored in directory: /root/.cache/pip/wheels/b1/fe/6d/96c159b80d9cd79bf1e54b71ad4920634549a826236dd1c9ca
      Building wheel for dlx (setup.py) ... done
      Created wheel for dlx: filename=dlx-1.0.4-py3-none-any.whl size=5719 sha256=21db8ae2c21d149e875a01d15826d8da931a469f2b8b4f4b434e1e55ef8c0089
      Stored in directory: /root/.cache/pip/wheels/78/55/c8/dc61e772445a566b7608a476d151e9dcaf4e092b01b0c4bc3c
      Building wheel for docplex (setup.py) ... done
      Created wheel for docplex: filename=docplex-2.22.213-py3-none-any.whl size=696881 sha256=97e479fc92e9177b3d753e6c6bd8bc574736d19795c291744357021b9be1e484
      Stored in directory: /root/.cache/pip/wheels/90/69/6b/1375c68a5b7ff94c40263b151c86f58bd72200bf0c465b5ba3
      Building wheel for python-constraint (setup.py) ... done
      Created wheel for python-constraint: filename=python_constraint-1.4.0-py2.py3-none-any.whl size=24081 sha256=bb451f470362a3c6871992a6423c397c2cc4b0a6046886175c318864dbf5242b
      Stored in directory: /root/.cache/pip/wheels/07/27/db/1222c80eb1e431f3d2199c12569cb1cac60f562a451fe30479
      Building wheel for yfinance (setup.py) ... done
      Created wheel for yfinance: filename=yfinance-0.1.63-py2.py3-none-any.whl size=23918 sha256=c2eeb740497844dee8c4064be232235ac3528a66a4e2c6a99a61bd0584feabb3
      Stored in directory: /root/.cache/pip/wheels/fe/87/8b/7ec24486e001d3926537f5f7801f57a74d181be25b11157983
    Successfully built qiskit dlx docplex python-constraint yfinance
    Installing collected packages: tweedledum, symengine, retworkx, python-constraint, ply, fastjsonschema, qiskit-terra, ntlm-auth, lxml, inflection, cryptography, yfinance, websocket-client, requests-ntlm, quandl, qiskit-ignis, pybind11, docplex, dlx, qiskit-ibmq-provider, qiskit-aqua, qiskit-aer, qiskit
      Attempting uninstall: lxml
        Found existing installation: lxml 4.2.6
        Uninstalling lxml-4.2.6:
          Successfully uninstalled lxml-4.2.6
    Successfully installed cryptography-35.0.0 dlx-1.0.4 docplex-2.22.213 fastjsonschema-2.15.1 inflection-0.5.1 lxml-4.6.3 ntlm-auth-1.5.0 ply-3.11 pybind11-2.7.1 python-constraint-1.4.0 qiskit-0.30.1 qiskit-aer-0.9.0 qiskit-aqua-0.9.5 qiskit-ibmq-provider-0.16.0 qiskit-ignis-0.6.0 qiskit-terra-0.18.3 quandl-3.6.1 requests-ntlm-1.1.0 retworkx-0.10.2 symengine-0.8.1 tweedledum-1.1.1 websocket-client-1.2.1 yfinance-0.1.63
    Requirement already satisfied: torch in /usr/local/lib/python3.7/dist-packages (1.9.0+cu102)
    Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch) (3.7.4.3)

##Importing the required libraries

The libraries to be imported are:

* Qiskit
* Pytorch
* Numpy
* Matplotlib

```python
import qiskit
import numpy as np
import torch
from qiskit import QuantumCircuit, QuantumRegister, ClassicalRegister, Aer
import torch.nn as nn
import torch.optim as optim
import matplotlib.pyplot as plt
```

```python
print(qiskit.__version__)
print(qiskit.__qiskit_version__)
```

    0.18.3
    {'qiskit-terra': '0.18.3', 'qiskit-aer': '0.9.0', 'qiskit-ignis': '0.6.0', 'qiskit-ibmq-provider': '0.16.0', 'qiskit-aqua': '0.9.5', 'qiskit': None, 'qiskit-nature': None, 'qiskit-finance': None, 'qiskit-optimization': None, 'qiskit-machine-learning': None}

```python
print(torch.__version__)
```

    1.9.0+cu102

#Generating Random States

##Creating TrainingData and Label classes to faciliate training

* TrainingData - Creates a dataset of random values of required bit length and size
* Labels - Wrapper class for the predefined output labels

```python
class TrainingData:
  """
  Creates a Training Dataset of random quantum states.

  Args:
      bit_len (int): The number of bits required to represent a state.
      size (int): The size of dataset required for training. Minimum can be 1
                  and maximum (n^2)
      backend (AerBackend): The backend required to run the quantum simulation
                            for random state generation. If not set, the
                            QASM Simulator will be chosen as default.
      shots (int): The number of shots provided to the backend. Default value
                   is 1024.

  Raises:
      TypeError: Raises the error, if the type determined is not valid.
      ValueError: Raises the error, if the size is either less than 1 or
                  greater than n^2.

  """

  def __init__(self, bit_len, size, backend=Aer.get_backend('qasm_simulator'),
               shots=1024):

    if not isinstance(bit_len, int):
      raise TypeError("The length of bits must be an integer instead of {}.".
                      format(type(bit_len).__name__))
    if not isinstance(size, int):
      raise TypeError("The size of dataset must be an integer instead of {}.".
                      format(type(size).__name__))

    if size < 1:
      raise ValueError("The minimum size of dataset must be 1.")
    if size > bit_len**2:
      raise ValueError("The maximum size of dataset can be generated from {} \
      qubits is {}".format(bit_len, bit_len**2))

    self.bit_len = bit_len
    self.backend = backend
    self.shots = shots
    self.size = size
    counts = self.generate_samples(self.bit_len, self.backend, self.shots)

    while len(counts) < self.size:
      #Generates samples again if they less the required size
      counts = self.generate_samples(self.bit_len, self.backend, self.shots)

    #Sort the dictionary based on the count of states
    self.counts = dict(sorted(counts.items(),
                         key=lambda item: item[1]))
    self._samples = list(self.counts.keys())[:self.size]
    self.params = None

  def generate_samples(self, qubits, backend=Aer.get_backend('qasm_simulator'),
               shots=1024):
    """
    Returns the samples and its counts generated by a random circuit.

    Args:
        qubits (int): The number of quantum bits and classical bits to generate
                      samples.
        backend (AerBackend): The backend required to run the quantum simulation
                              for random state generation. If not set, the
                              QASM Simulator will be chosen as default.
        shots (int): The number of shots provided to the backend. Default value
                     is 1024.
    """
    circuit = QuantumCircuit(qubits,qubits)
    for i in range(qubits):
      args = np.random.rand(3)*2*np.pi
      circuit.u(args[0], args[1], args[2], i)
      circuit.measure(i,i)
    result = backend.run(circuit, shots=shots).result()
    counts = result.get_counts()

    return counts

  def to_params(self):
    """
    Returns the dataset converted from states into angles(radians) for quantum
    circuit.

    Returns:
        Ndarray: A 2D array of shape (self.size, self.bit_len) with paramaters
                 to be passed to QNNCircuit for training.
    """
    self.params = np.zeros((self.size,self.bit_len))
    for index in range(self.size):
      sample = self.samples[index]
      for i in range(self.bit_len):
        state = sample[i]
        if state=="1":
          #RY(π/2)|0> = X|0> = |1>
          self.params[index][self.bit_len-i-1] = np.pi/2
    return self.params

  @property
  def samples(self):
    """
    Returns the samples generated by the random circuit

    Returns:
        List: A list of length 'size' containing different samples generated by
              the quantum circuits
    """
    return self._samples

  @samples.setter
  def samples(self, samples):
    self._samples = samples

  def __str__(self):
    return str(self.samples)
```

```python
class Labels:
  """
  Creates a wrapper for labels for training the model

  Args:
      data (list(str)): A list of strings containing different states to be
                        generated by the model.

  Raises:
      TypeError: Raises an error, if any of the element is either not string or
                 a binary string
  """

  def __init__(self, *data):

    if isinstance(data[0], list):
      data = data[0]
    for label in data:
      if not isinstance(label, str):
        raise TypeError("Expected a string of strings but recieved a {}".
                        format(type(label).__name__))
      try:
        check = int(label,2)
      except TypeError:
        raise TypeError("The label {} is not a binary string".format(label))

    self._data = list(data)
    self.size = len(self.data)
    self.bit_len = len(self.data[0])

  @property
  def data(self):
    """
    Returns the labels

    Returns:
        List: List of binary labels for validation
    """
    return self._data

  @data.setter
  def data(self, data):
    self._data = data

  def __str__(self):
    return str(self._data)
```

```python
random_states = TrainingData(4, size=4)
labels = Labels('0011', '0101', '1010', '1100')
print(random_states)
```

    ['1000', '0011', '0001', '0000']

**Aim**: To map these newly random states to the predefined label states

For example -

* 0010 -> 0011
* 0011 -> 0101
* 0111 -> 1010
* 1000 -> 1100

**Proposed Solution**: Use Hybrid QGANs with Quantum Generators and Classical Discriminator to generate a required states for the random states.

#Design Quantum Circuit for QNN

The quantum circuit consists of alternating group of RY and RZ gates followed by full entanglement of CNOT gates. Taking the 32 theta and 4 label values, the quantum circuit can look like -

```python
class QNNCircuit:
  """
  Creates a Quantum Neural Network Circuit for Quantum Generator

  Args:
      kernel_size (int): The number of qubits in the Quantum Neural Network
                         Circuit.
      backend (AerBackend): The backend required to run the quantum simulation
                            for random state generation. If not set, the
                            QASM Simulator will be chosen as default.
      shots (int): The number of shots provided to the backend. Default value
                   is 1024.

  Raises:
      TypeError: Raises the error, if the type determined is not valid.
  """

  def __init__(self, kernel_size,
               backend=Aer.get_backend('qasm_simulator'),
               shots = 1024):

    if not isinstance(kernel_size, int):
      raise TypeError("The number of qubits must be an integer instead of {}.".
                      format(type(kernel_size).__name__))

    # --- Circuit definition ---
    self.kernel_size = kernel_size
    self.training_qubits = QuantumRegister(kernel_size, name="training")
    self.measuring_register = ClassicalRegister(kernel_size, "meas")
    self.circuit = QuantumCircuit(self.training_qubits,
                                  self.measuring_register)
    # ---------------------------
    self.backend = backend
    self.shots = shots

  def create_circuit(self, thetas, label):
    """
    Add the gates and assign parameters to Quantum Circuit.

    Args:
        thetas (list(float), ndarray(float)): Training parameters to assign to
                                              the quantum circuit.
        label (list(float), ndarray(float)): Parameters to initialize the
                                             state to the circuit.

    Raises:
        TypeError: Raises an error if the arguments are neither list nor
                   ndarray.
        ValueError: Raises an erorr if label size is not equal to kernel_size
    """

    if not isinstance(thetas, (list, np.ndarray)):
      raise TypeError("Expected a list of floating or integer values but \
      recieved a {} for thetas".format(type(thetas).__name__))
    if not isinstance(label, (list, np.ndarray)):
      raise TypeError("Expected a list of floating or integer values but \
      recieved a {} for label".format(type(label).__name__))
    if len(label) != self.kernel_size:
      raise ValueError("Expected a size of {} for label but recieved {}".
                       format(self.kernel_size, len(label)))

    if isinstance(thetas, np.ndarray):
      thetas = thetas.tolist()
    if isinstance(label, np.ndarray):
      label = label.tolist()

    #Initialising the Quantum state on the Quantum Circuit
    #RY(π/2)|0> = |1>
    #RY(0)|0> = |0>
    for i in range(len(label)):
      self.circuit.ry(label[i], self.training_qubits[i])

    for i in range(len(thetas)):
      #Fully entangled CNOT gates after each RY and RZ gates pair
      if (i%(2*self.kernel_size))==0 and i!=0:
        for j in range(self.kernel_size):
          for k in range(j+1,self.kernel_size):
            self.circuit.cnot(self.training_qubits[j],
                              self.training_qubits[k])

      #Alternating RY and RZ gates
      if (i//self.kernel_size)%2 == 0:
        self.circuit.ry(thetas[i],
                        self.training_qubits[i%self.kernel_size])
      else:
        self.circuit.rz(thetas[i],
                        self.training_qubits[i%self.kernel_size])

  def expval(self):
    """
    Calculates the expectation value of the created circuit.

    Expectation Value = Sum(state(i)*probability(i))

    Returns:
        Float: Expectation value of the quantum circuit.
    """
    expval = 0
    result = self.get_states()
    for key,val in result.items():
      expval += key*val
    return expval

  def get_states(self):
    """
    Measures the quantum circuit and outputs the probability of each state

    Returns:
        Dict(string: float): Probaility of different states measured by the
                             quantum circuit
    """

    self.circuit.measure(self.training_qubits, self.measuring_register)
    job = self.backend.run(self.circuit, shots=self.shots)
    counts = job.result().get_counts()
    output = {key: val/self.shots for key, val in counts.items()}
    return output
```

#Defining Hybrid Quantum GAN

The Hybrid Quantum GAN consists of Quantum Generator and Classical Discriminator

* **Quantum Generator** - It takes the random states and uses QNNCircuit to generate the probability of all the possible states.
* **Classical Discriminator** - It takes the probabilties generated by the generator to check if it is producing real or fake outputs.

```python
def get_accuracy(pred_list, actual):
  """
  Determines the average accuracy of the generated data.

  Args:
      pred_list (list(list(float))): Probabilty of different states of different
                                     random input states
      actual (list(str)): Correct labels which is used to determine the accuracy

  Returns:
      float: Returns the accuracy of the output generated by the generator
  """
  total = 0
  for i in range(len(pred_list)):
    total += pred_list[i][int(actual[i],2)]
  return total/len(pred_list)
```

```python
def hotshot(counts, bit_len):
  """
  Converts the different states and its probability into a list of all state
  relative probabilities wrt highest probable state.

  Args:
      counts (dict(str:float)): Dictionary with different states as the keys and
                                probailities as its values
      bit_len (int): The length of bits provided

  Returns:
      list(float): List of probabilities of state converted into the index of
                   the list
  """
  new_count = []
  max_val = max(list(counts.values()))
  for i in range(bit_len*bit_len):
    key = "{0:b}".format(i).zfill(bit_len)
    if key in counts:
      new_count.append(counts[key]/max_val)
    else:
      new_count.append(0.0)
  return new_count
```

```python
class QNNModel(nn.Module):
  """
  Quantum Nueral Network model to generate different states

  Args:
      kernel_size (int): The number of qubits in the Quantum Neural Network
                         Circuit.
      backend (AerBackend): The backend required to run the quantum simulation
                            for random state generation. If not set, the
                            QASM Simulator will be chosen as default.
      shots (int): The number of shots provided to the backend. Default value
                   is 1024.
      n_params (int): Number of parameters to be trained. If not set, 8 will be
                      chosen as deafult.

  Raises:
      ValueError: Raises the error, if the size is less than 1
  """

  def __init__(self,kernel_size,
               backend = Aer.get_backend('qasm_simulator'),
               shots = 1024, n_params = 8):

    super(QNNModel, self).__init__()

    if n_params < 1:
      raise ValueError("The minimum number of parameters must be 1.")

    self.backend = backend
    self.kernel_size = kernel_size
    self.shots = shots
    self.n_params = n_params
    self.param = nn.Parameter(torch.rand(self.n_params,
                                         requires_grad=True)*np.pi)

  def forward(self, data):
    """
    Performs forward propogation

    Args:
        data (TrainingData): Random training data to initialize the state

    Returns:
        tensor: Tensor of different probabilities of different input states
    """
    size = len(data.samples)
    self.circuits = [QNNCircuit(self.kernel_size) for i in range(size)]
    param = self.param.data.numpy()
    data_params = data.to_params()

    for i in range(size):
      self.circuits[i].create_circuit(param, data_params[i])

    output = [hotshot(self.circuits[i].get_states(),
              self.kernel_size) for i in range(size)]
    return torch.tensor(output)
```

```python
class Discriminator(nn.Module):
  """
  Creates a discriminator for the GANs. Contains a single Linear layer
  followed by the sigmoid activation function.

  Args:
      n (int): The number of neurons in the hidden layer.

  Returns:
      float: The probailility of given output state
  """
  def __init__(self, n):
        super(Discriminator, self).__init__()
        self.dense = nn.Linear(int(n), 1);
        self.activation = nn.Sigmoid()

  def forward(self, x):
        return self.activation(self.dense(x))
```

```python
def GAN_training(generator, discriminator, g_optimizer, d_optimizer,
                 loss_fn, g_loss_list = [], d_loss_list = [],
                 g_accuracy_list = [], d_accuracy_list = [], epochs = 15000):
  """
  Trains the GAN with given generator and discriminator

  Args:
      generator (torch.nn.Module): Generator for GAN
      discriminator (torch.nn.Module): Discriminator for GAN
      g_optimizer (torch.optim): Generator's Optimizer
      d_optimizer (torch.optim): Discriminator's Optimizer
      loss_fn (torch.nn.Loss): Loss function for Discriminator
      g_loss_list (list): List to store Generator's Loss
      d_loss_list (list): List to store Discriminator's Loss
      g_accuracy_list (list): List to store Generator's Accuracy
      d_accuracy_list (list): List to store Discriminator's Accuracy
      epochs (int): Number of iterations for GAN. Deafult value is 15000

  """

  true_labels = torch.full((labels.size,), 1, dtype=torch.float)
  true_data = torch.tensor([hotshot({state:1},labels.bit_len)
                            for state in labels.data])

  fake_labels = torch.zeros(labels.size)

  generator.train()
  discriminator.train()
  for epoch in range(1,epochs+1):

    # Forward pass
    g_optimizer.zero_grad()

    # Generate the data on the random states
    g_out = generator(random_states)
    g_accuracy = get_accuracy(g_out, labels.data)

    # Calculate the Discriminator's loss on output of random states and true
    # labels
    g_d_out = discriminator(g_out).view(-1)
    g_loss = loss_fn(g_d_out, true_labels)

    # Train the discriminator on the true data and true labels
    d_optimizer.zero_grad()
    true_d_out = discriminator(true_data).view(-1)
    true_d_loss = loss_fn(true_d_out, true_labels)

    # Train the discriminator on the output of random states and fake labels
    g_d_f_out = discriminator(g_out.detach()).view(-1)
    g_d_loss = loss_fn(g_d_f_out, fake_labels)
    d_loss = true_d_loss + g_d_loss
    d_loss.backward()
    d_optimizer.step()

    # Backpropogation of Generator
    g_loss.backward()
    g_optimizer.step()

    g_loss_list.append(g_loss.item())
    d_loss_list.append(d_loss.item())
    g_accuracy_list.append(g_accuracy)
    d_accuracy_list.append(true_d_out.mean(0).detach().numpy())

    if epoch%1000==0:
      print('Epoch {:03d}\tGenerator Loss: {:.4f}\tGenerator Accuracy: {:.4f}\t'.
              format(epoch, g_loss.item(), g_accuracy), end='')
      print('Discriminator Loss: {:.4f}\tDiscriminator Accuracy: {:.4f}'.
            format(d_loss.item(), d_accuracy_list[-1]))
```

#Train the Hybrid Quantum GAN

```python
qgenerator = QNNModel(kernel_size=4, n_params=32)
qdiscriminator = Discriminator(qgenerator.kernel_size**2)

qg_optimizer = optim.Adam(qgenerator.parameters())
qd_optimizer = optim.Adam(qdiscriminator.parameters(), lr=0.001)

qloss_fn = nn.BCELoss()
epochs = 15000

qg_loss_list = []
qd_loss_list = []

qg_accuracy_list = []
qd_accuracy_list = []

GAN_training(qgenerator, qdiscriminator, qg_optimizer, qd_optimizer,
             qloss_fn, qg_loss_list, qd_loss_list, qg_accuracy_list,
             qd_accuracy_list, epochs = 15000)
```

    Epoch 1000	Generator Loss: 1.9247	Generator Accuracy: 0.4301	Discriminator Loss: 0.4518	Discriminator Accuracy: 0.7560
    Epoch 2000	Generator Loss: 2.8220	Generator Accuracy: 0.4441	Discriminator Loss: 0.2039	Discriminator Accuracy: 0.8774
    Epoch 3000	Generator Loss: 3.2595	Generator Accuracy: 0.4309	Discriminator Loss: 0.1222	Discriminator Accuracy: 0.9318
    Epoch 4000	Generator Loss: 3.8882	Generator Accuracy: 0.4463	Discriminator Loss: 0.0738	Discriminator Accuracy: 0.9597
    Epoch 5000	Generator Loss: 5.1541	Generator Accuracy: 0.4210	Discriminator Loss: 0.0379	Discriminator Accuracy: 0.9755
    Epoch 6000	Generator Loss: 5.0866	Generator Accuracy: 0.4039	Discriminator Loss: 0.0249	Discriminator Accuracy: 0.9846
    Epoch 7000	Generator Loss: 6.6369	Generator Accuracy: 0.4491	Discriminator Loss: 0.0167	Discriminator Accuracy: 0.9903
    Epoch 8000	Generator Loss: 6.4657	Generator Accuracy: 0.4427	Discriminator Loss: 0.0092	Discriminator Accuracy: 0.9937
    Epoch 9000	Generator Loss: 7.1595	Generator Accuracy: 0.4641	Discriminator Loss: 0.0068	Discriminator Accuracy: 0.9960
    Epoch 10000	Generator Loss: 8.3733	Generator Accuracy: 0.4339	Discriminator Loss: 0.0040	Discriminator Accuracy: 0.9974
    Epoch 11000	Generator Loss: 9.6134	Generator Accuracy: 0.4483	Discriminator Loss: 0.0019	Discriminator Accuracy: 0.9983
    Epoch 12000	Generator Loss: 9.4173	Generator Accuracy: 0.4439	Discriminator Loss: 0.0016	Discriminator Accuracy: 0.9989
    Epoch 13000	Generator Loss: 9.5850	Generator Accuracy: 0.4326	Discriminator Loss: 0.0009	Discriminator Accuracy: 0.9993
    Epoch 14000	Generator Loss: 10.2748	Generator Accuracy: 0.4223	Discriminator Loss: 0.0007	Discriminator Accuracy: 0.9995
    Epoch 15000	Generator Loss: 10.7425	Generator Accuracy: 0.4345	Discriminator Loss: 0.0006	Discriminator Accuracy: 0.9997

#Results and Conclusions

```python
plt.figure(figsize=(10,5))
plt.title("Generator and Discriminator Loss During Hybrid Quantum \
GAN Training")
plt.plot(qg_loss_list, label='Generator')
plt.plot(qd_loss_list, label='Discriminator')
plt.xlabel('Training Iterations')
plt.ylabel('Training Loss')
plt.legend()
plt.show()
```

```python
plt.figure(figsize=(10,5))
plt.title("Generator and Discriminator Accuracy During Hybrid Quantum \
GAN Training")
plt.plot(qg_accuracy_list, label='Generator')
plt.plot(qd_accuracy_list, label='Discriminator')
plt.xlabel('Training Iterations')
plt.ylabel('Training Accuracy')
plt.legend()
plt.show()
```

##Conclusions

By analyzing the Generator's and Discriminator's Loss Graph of Hybrid Quantum GAN, we can see that the loss of the Generator is continously increasing while Discriminator's loss is decreasing. This indicates that the Hybrid QGAN model is reaching failure mode. The discriminator can easily identify that the random quantum states cannot produce the predefined output states. It can also be said that the generator is not able to produce the states with probable states.

This can also be verified by analysing the accuracy graph. It shows that the disciminator is becoming good at finding that the given outputs don't correspond to the actual labels. For the ideal GAN model,

* The accuracy of generator should reach 100%.
* The Discriminator's loss should be around 0.5 indicating it is facing hard time deciding if the generator is producing the correct results.

###Possible Solutions

* **Increasing the number of iterations**: Increasing the number of iterations in GANs usually help the generator to produce good data. However, in this case it seem unlikely it will help because with increasing iterations, the loss does not seem to rise to 0.5 or become constant.
* **Decreasing the learning rate of Discriminator's optimizer**: This ensures that the discriminator learns slowly so that generator could get enough time to generate good data. However, during my trails, it also does not seem to work much.
* **Change the Quantum Circuit**: The reason why the generator is not producing the convincing results is that the circuit is not good enough. However, I tried different quantum circuit but none of them seemed to work much good.

#Testing the Quantum Generator

```python
def GAN_test(generator, states, labels):
  """
  Tests the GAN generator.

  Args:

      generator (torch.nn.Module): GAN Generator
      states (TrainingData): Training Data to generate result
      labels (Labels): Output Labels
  """
  generator.eval()
  with torch.no_grad():
    output = generator(states)
    for i in range(len(labels.data)):
      output_dict = {"{0:b}".format(key).zfill(labels.bit_len):output[i][key]
                for key in range(len(output[i]))}

      out_state = max(output_dict, key = lambda x: output_dict[x])
      print("Random State = {}\tActual Label = {}\t".
              format(states.samples[i], labels.data[i]), end=' ')
      print("Output Label = {}\tActual Label Accuracy = {:.2f}%".
            format(out_state, output_dict[labels.data[i]]*100))
```

```python
GAN_test(qgenerator, random_states, labels)
```

    Random State = 1000	Actual Label = 0011	 Output Label = 1001	Actual Label Accuracy = 25.33%
    Random State = 0011	Actual Label = 0101	 Output Label = 0101	Actual Label Accuracy = 100.00%
    Random State = 0001	Actual Label = 1010	 Output Label = 1011	Actual Label Accuracy = 34.64%
    Random State = 0000	Actual Label = 1100	 Output Label = 1001	Actual Label Accuracy = 19.71%

```python
new_states = TrainingData(4, size=4)
GAN_test(qgenerator, new_states, labels)
```

    Random State = 1010	Actual Label = 0011	 Output Label = 1101	Actual Label Accuracy = 1.40%
    Random State = 1000	Actual Label = 0101	 Output Label = 1001	Actual Label Accuracy = 4.93%
    Random State = 1011	Actual Label = 1010	 Output Label = 1101	Actual Label Accuracy = 55.69%
    Random State = 1001	Actual Label = 1100	 Output Label = 1011	Actual Label Accuracy = 2.94%

```python

```


```python
%matplotlib inline

import os
import re
import urllib.request
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import matplotlib.pyplot as plt
import itertools

from torch.utils.data import Dataset, DataLoader
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence
device = torch.device("cuda:0") if torch.cuda.is_available() else torch.device("cpu")
```

In this notebook you will work with a generative language model that generates inflected wordforms one character at a time. We will use text data (we will work on the character level) in Spanish and pytorch.

The first section concerns data manipulation and data loading classes necessary for our implementation. You do not need to modify anything in this part of the code.

Let's first download the SIGMORPHON dataset that we will be using for this notebook: these are inflected Spanish words together with some morphosyntactic descriptors. For this notebook we will ignore the morphosyntactic descriptors.

```python
url = "https://raw.githubusercontent.com/ryancotterell/sigmorphon2016/master/data/"
train_file = "spanish-task1-train"
val_file = "spanish-task1-dev"
test_file = "spanish-task1-test"

print("Downloading data files...")
if not os.path.isfile(train_file):
    urllib.request.urlretrieve(url + train_file, filename=train_file)
if not os.path.isfile(val_file):
    urllib.request.urlretrieve(url + val_file, filename=val_file)
if not os.path.isfile(test_file):
    urllib.request.urlretrieve(url + test_file, filename=test_file)
print("Download complete.")
```

# Data

In order to work with text data, we need to transform the text into something that our algorithms can work with. The first step of this process is converting words into word ids. We do this by constructing a vocabulary from the data, assigning a new word id to each new word it encounters.

```python
UNK_TOKEN = "?"
PAD_TOKEN = "_"
SOW_TOKEN = ">"
EOW_TOKEN = "."

def extract_inflected_word(s):
    """
    Extracts the inflected words in the SIGMORPHON dataset.
    """
    return s.split()[-1]

class Vocabulary:

    def __init__(self):
        self.idx_to_char = {0: UNK_TOKEN, 1: PAD_TOKEN, 2: SOW_TOKEN, 3: EOW_TOKEN}
        self.char_to_idx = {UNK_TOKEN: 0, PAD_TOKEN: 1, SOW_TOKEN: 2, EOW_TOKEN: 3}
        self.word_freqs = {}

    def __getitem__(self, key):
        return self.char_to_idx[key] if key in self.char_to_idx else self.char_to_idx[UNK_TOKEN]

    def word(self, idx):
        return self.idx_to_char[idx]

    def size(self):
        return len(self.char_to_idx)

    @staticmethod
    def from_data(filenames):
        """
            Creates a vocabulary from a list of data files. It assumes that the data files have been
            tokenized and pre-processed beforehand.
        """
        vocab = Vocabulary()
        for filename in filenames:
            with open(filename) as f:
                for line in f:

                    # Strip whitespace and the newline symbol.
                    word = extract_inflected_word(line.strip())

                    # Split the words into characters and assign ids to each
                    # new character it encounters.
                    for char in list(word):
                        if char not in vocab.char_to_idx:
                            idx = len(vocab.char_to_idx)
                            vocab.char_to_idx[char] = idx
                            vocab.idx_to_char[idx] = char

        return vocab
```

```python
# Construct a vocabulary from the training and validation data.
print("Constructing vocabulary...")
vocab = Vocabulary.from_data([train_file, val_file])
print("Constructed a vocabulary of %d types" % vocab.size())
```

```python
# some examples
print('e', vocab['e'])
print('é', vocab['é'])
print('ș', vocab['ș'])  # something UNKNOWN
```

We also need to load the data files into memory. We create a simple class `TextDataset` that stores the data as a list of words:

```python
class TextDataset(Dataset):
    """
        A simple class that loads a list of words into memory from a text file,
        split by newlines. This does not do any memory optimisation,
        so if your dataset is very large, you might want to use an alternative
        class.
    """

    def __init__(self, text_file, max_len=30):
        self.data = []
        with open(text_file) as f:
            for line in f:
                word = extract_inflected_word(line.strip())
                if len(list(word)) <= max_len:
                    self.data.append(word)

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        return self.data[idx]
```

```python
# Load the training, validation, and test datasets into memory.
train_dataset = TextDataset(train_file)
val_dataset = TextDataset(val_file)
test_dataset = TextDataset(test_file)

# Print some samples from the data:
print("Sample from training data: \"%s\"" % train_dataset[np.random.choice(len(train_dataset))])
print("Sample from validation data: \"%s\"" % val_dataset[np.random.choice(len(val_dataset))])
print("Sample from test data: \"%s\"" % test_dataset[np.random.choice(len(test_dataset))])
```

Now it's time to write a function that converts a word into a list of character ids using the vocabulary we created before. This function is `create_batch` in the code cell below. This function creates a batch from a list of words, and makes sure that each word starts with a start-of-word symbol and ends with an end-of-word symbol. Because not all words are of equal length in a certain batch, words are padded with padding symbols so that they match the length of the largest word in the batch. The function returns an input batch, an output batch, a mask of 1s for words and 0s for padding symbols, and the sequence lengths of each word in the batch. The output batch is shifted by one character, to reflect the predictions that the model is expected to make. For example, for a word
\begin{align}
    \text{e s p e s e m o s}
\end{align}
the input sequence is
\begin{align}
    \text{SOW e s p e s e m o s}
\end{align}
and the output sequence is
\begin{align}
    \text{e s p e s e m o s EOW}
\end{align}

You can see the output is shifted wrt the input, that's because we will be computing a distribution for the next character in context of its prefix, and that's why we need to shift the sequence this way.

Lastly, we create an inverse function `batch_to_words` that recovers the list of words from a padded batch of character ids to use during test time.

```python
def create_batch(words, vocab, device, word_dropout=0.):
    """
    Converts a list of words to a padded batch of word ids. Returns
    an input batch, an output batch shifted by one, a sequence mask over
    the input batch, and a tensor containing the sequence length of each
    batch element.
    :param words: a list of words, each a list of token ids
    :param vocab: a Vocabulary object for this dataset
    :param device:
    :param word_dropout: rate at which we omit words from the context (input)
    :returns: a batch of padded inputs, a batch of padded outputs, mask, lengths
    """
    tok = np.array([[SOW_TOKEN] + list(w) + [EOW_TOKEN] for w in words])
    seq_lengths = [len(w)-1 for w in tok]
    max_len = max(seq_lengths)
    pad_id = vocab[PAD_TOKEN]
    pad_id_input = [
        [vocab[w[t]] if t < seq_lengths[idx] else pad_id for t in range(max_len)]
            for idx, w in enumerate(tok)]

    # Replace words of the input with <unk> with p = word_dropout.
    if word_dropout > 0.:
        unk_id = vocab[UNK_TOKEN]
        word_drop =  [
            [unk_id if (np.random.random() < word_dropout and t < seq_lengths[idx]) else word_ids[t] for t in range(max_len)]
                for idx, word_ids in enumerate(pad_id_input)]

    # The output batch is shifted by 1.
    pad_id_output = [
        [vocab[w[t+1]] if t < seq_lengths[idx] else pad_id for t in range(max_len)]
            for idx, w in enumerate(tok)]

    # Convert everything to PyTorch tensors.
    batch_input = torch.tensor(pad_id_input)
    batch_output = torch.tensor(pad_id_output)
    seq_mask = (batch_input != vocab[PAD_TOKEN])
    seq_length = torch.tensor(seq_lengths)

    # Move all tensors to the given device.
    batch_input = batch_input.to(device)
    batch_output = batch_output.to(device)
    seq_mask = seq_mask.to(device)
    seq_length = seq_length.to(device)

    return batch_input, batch_output, seq_mask, seq_length

def batch_to_words(tensors, vocab: Vocabulary):
    """
    Converts a batch of word ids back to words.
    :param tensors: [B, T] word ids
    :param vocab: a Vocabulary object for this dataset
    :returns: an array of strings (each a word).
    """
    words = []
    batch_size = tensors.size(0)
    for idx in range(batch_size):
        word = [vocab.word(t.item()) for t in tensors[idx,:]]

        # Filter out the start-of-word and padding tokens.
        word = list(filter(lambda t: t != PAD_TOKEN and t != SOW_TOKEN, word))

        # Remove the end-of-word token and all tokens following it.
        if EOW_TOKEN in word:
            word = word[:word.index(EOW_TOKEN)]

        words.append("".join(word))
    return np.array(words)
```

In PyTorch the RNN functions expect inputs to be sorted from long words to shorter ones. Therefore we create a simple wrapper class for the DataLoader class that sorts words from long to short:

```python
class SortingTextDataLoader:
    """
    A wrapper for the DataLoader class that sorts a list of words by their
    lengths in descending order.
    """

    def __init__(self, dataloader):
        self.dataloader = dataloader
        self.it = iter(dataloader)

    def __iter__(self):
        return self

    def __next__(self):
        words = None
        for s in self.it:
            words = s
            break

        if words is None:
            self.it = iter(self.dataloader)
            raise StopIteration

        words = np.array(words)
        sort_keys = sorted(range(len(words)),
                           key=lambda idx: len(list(words[idx])),
                           reverse=True)
        sorted_words = words[sort_keys]
        return sorted_words
```

# Model

## Deterministic language model

In language modelling, we model a word $x = \langle x_1, \ldots, x_n \rangle$  of length $n = |x|$ as a sequence of categorical draws:

\begin{align}
X_i|x_{<i} & \sim \text{Cat}(f(x_{<i}; \theta))
& i = 1, \ldots, n \\
\end{align}

where we use $x_{<i}$ to denote a (possibly empty) prefix string, and thus the model makes no Markov assumption. We map from the conditioning context, the prefix $x_{<i}$, to the categorical parameters (a $v$-dimensional probability vector, where $v$ denotes the size of the vocabulary, in this case, the size of the character set) using a fixed neural network architecture whose parameters we collectively denote by $\theta$.

This assigns the following likelihood to the word
\begin{align}
    P(x|\theta) &= \prod_{i=1}^n P(x_i|x_{<i}, \theta) \\
    &= \prod_{i=1}^n \text{Cat}(x_i|f(x_{<i}; \theta))
\end{align}
where the categorical pmf is $\text{Cat}(k|\pi) = \prod_{j=1}^v \pi_j^{[k=j]} = \pi_k$.

Suppose we have a dataset $\mathcal D = \{x^{(1)}, \ldots, x^{(N)}\}$ containing $N$ i.i.d. observations. Then we can use the log-likelihood function
\begin{align}
\mathcal L(\theta|\mathcal D) &= \sum_{k=1}^{N} \log P(x^{(k)}| \theta) \\
&= \sum_{k=1}^{N} \sum_{i=1}^{|x^{(k)}|} \log \text{Cat}(x^{(k)}_i|f(x^{(k)}_{<i}; \theta))
\end{align}
 to estimate $\theta$ by maximisation:
 \begin{align}
 \theta^\star = \arg\max_{\theta \in \Theta} \mathcal L(\theta|\mathcal D) ~ .
 \end{align}

We can use stochastic gradient-ascent to find a local optimum of $\mathcal L(\theta|\mathcal D)$, which only requires a gradient estimate:

\begin{align}
\nabla_\theta \mathcal L(\theta|\mathcal D) &= \sum_{k=1}^{|\mathcal D|} \nabla_\theta  \log P(x^{(k)}|\theta) \\
&= \sum_{k=1}^{|\mathcal D|} \frac{1}{N} N \nabla_\theta  \log P(x^{(k)}| \theta)  \\
&= \mathbb E_{\mathcal U(1/N)} \left[ N \nabla_\theta  \log P(x^{(K)}| \theta) \right]  \\
&\overset{\text{MC}}{\approx} \frac{N}{M} \sum_{m=1}^M \nabla_\theta  \log P(x^{(k_m)}|\theta) \\
&\text{where }K_m \sim \mathcal U(1/N)
\end{align}

This is a Monte Carlo (MC) estimate of the gradient computed on $M$ data points selected uniformly at random from $\mathcal D$.

For as long as $f$ remains differentiable wrt to its inputs and parameters, we can rely on automatic differentiation to obtain gradient estimates.

An example design for $f$ is:
\begin{align}
\mathbf x_i &= \text{emb}(x_i; \theta_{\text{emb}}) \\
\mathbf h_0 &= \mathbf 0 \\
\mathbf h_i &= \text{rnn}(\mathbf h_{i-1}, \mathbf x_{i-1}; \theta_{\text{rnn}}) \\
f(x_{<i}; \theta) &= \text{softmax}(\text{dense}_v(\mathbf h_{i};  \theta_{\text{out}}))
\end{align}
where
* $\text{emb}$ is a fixed embedding layer with parameters $\theta_{\text{emb}}$;
* $\text{rnn}$ is a recurrent architecture with parameters $\theta_{\text{rnn}}$, e.g. an LSTM or GRU, and $\mathbf h_0$ is part of the architecture's parameters;
* $\text{dense}_v$ is a dense layer with $v$ outputs (vocabulary size) and parameters $\theta_{\text{out}}$.

In follow up labs this model will be a component in a *deep generative model*. You may skip this lab if you are comfortable with the overall notion of having neural networks parameterise statistical/probabilistic models.

## Implementation

In our implementation we create a pytorch module with the NN blocks we need, namely, embedding layer, LSTM cell, and output layer. The forward method will use these layers to map inputs to categorical distributions.

You may want to check the documentation for

* `torch.nn.Embedding`
* `torch.nn.Linear`
* `torch.nn.LSTM`
* `torch.distributions.categorical.Categorical`

```python
from torch.distributions import Categorical

class CharLM(nn.Module):

    def __init__(self, vocab_size, emb_size, hidden_size,
                 pad_idx, dropout=0.):
        """
        :param vocab_size: size of the vocabulary of the language
        :param emb_size: dimensionality of embeddings
        :param hidden_size: dimensionality of recurrent cell
        :param pad_idx: the id reserved to the -PAD- token
        :param dropout: a dropout rate (you can ignore this for now)
        """
        super().__init__()  # necessary for pytorch
        # Construct your NN architecture here
        # - in pytorch your NN blocks must be attributes of self,
        #   e.g. self.linear_layer = nn.Linear(...)
        pass

    def step(self, prev_x, hidden):
        """
        Performs a single LSTM step for a given previous word and hidden state.
        Returns the unnormalized log probabilities (logits) over the vocabulary for this time step
        and the updated hidden state.

        :param prev_x: [B, 1] id of the previous token
        :param hidden:  hidden ([num_layers, B, H] state, [num_layers, B, H] cell)

        :returns: [B, 1, V], ([num_layers, B, H], [num_layers, B, H])
        """
        # output: [B, H]
        return None, None

    def forward(self, x) -> Categorical:
        """
        Performs an entire forward pass given a sequence of words x.
        Returns a collection of Categorical distributions over the vocabulary,
        one distribution per time step per sequence.

        :param x: [B, T] token ids

        Returns a [B,T] collection of Categorical distributions, each over V outcomes.
        """
        # Setting hidden=None implies a fixed initial hidden/cell state, namely, a vector of zeros.
        hidden = None
        outputs = []
        for t in range(x.size(1)):
            # Here we slice the t-th token
            # [B, 1]
            prev_x = x[:, t].unsqueeze(-1)
            # Here we update our hidden state for prediction of
            #  a categorical distribution for outcome at position t+1
            # logits: [B, 1, V]
            logits, hidden = self.step(prev_x, hidden)
            outputs.append(logits)
        # [B, T, V]
        outputs = torch.cat(outputs, dim=1)
        return Categorical(logits=outputs)

    def loss(self, output_distributions: Categorical, observations):
        """
        Computes the terms in the loss (negative log-likelihood) given a
        collection of Categorical output distributions and a collection of
        observations in context (i.e. x_i|x_{<i}).

        :params output_distributions: a [B, T] collection of V-dimensional
            categorical distributions.
        :params observations: [B, T] token ids (next tokens, with respect to
            the inputs that were used to parameterise the output distributions)
        :returns:
            negative log likelihood (scalar node), dict reserved for future use
        """
        return None, dict()
```

The code below is used to assess the model and also investigate what it learned. We implemented it for you. It's useful however to learn from this example: we do interesting things like computing perplexity and sampling novel words!

# Evaluation metrics

During training we'd like to keep track of some evaluation metrics on the validation data in order to keep track of how our model is doing and to perform early stopping.

A common metric to evaluate language models is the perplexity per word. The perplexity per word for a dataset is defined as:

\begin{align}
    \text{ppl}(\mathcal{D}|\theta) = \exp\left(-\frac{1}{\sum_{k=1}^{|\mathcal D|} n^{(k)}} \sum_{k=1}^{|\mathcal{D}|} \log P(x^{(k)}|\theta)\right)
\end{align}

where $n^{(k)} = |x^{(k)}|$ is the number of tokens in a word and $P(x^{(k)})$ is the probability that our model assigns to the datapoint $x^{(k)}$.

```python
def eval_perplexity(model, dataset, vocab, device, batch_size=128):
    """
    Return ppl per word as well as average negative log-likelihood
        given a model and a dataset.
    """
    dl = DataLoader(dataset, batch_size=batch_size)
    sorted_dl = SortingTextDataLoader(dl)

    # Make sure the model is in evaluation mode (i.e. disable dropout).
    model.eval()

    log_px = 0.
    num_predictions = 0
    num_words = 0

    # We don't need to compute gradients for this.
    with torch.no_grad():
        for words in sorted_dl:
            # [B, T], [B, T], [B, T], [B]
            x_in, x_out, seq_mask, seq_len = create_batch(words, vocab, device)

            # Compute output categoricals, i.e.
            #  X_i|theta, x_{<i} ~ Cat(f(x_{<i}; theta))
            p_x = model(x_in)

            # [B, T]
            log_prob = p_x.log_prob(x_out)
            # Let's mask positions that should not contribute to the likelihood
            log_prob = torch.where(seq_mask, log_prob, torch.zeros_like(log_prob))
            # Let's aggregate the log-probability of all observations
            # []
            log_prob = log_prob.sum()
            # And accumulate it over batches
            log_px += log_prob
            num_predictions += seq_len.sum()
            num_words += seq_len.size(0)

    # Compute and return the perplexity per word.
    perplexity = torch.exp(-log_px / num_predictions)
    NLL = -log_px / num_words
    return perplexity, NLL
```

Lastly, we want to occasionally qualitatively see the performance of the model during training, by letting it generate word forms from the trained model (a distribution over word forms). For this we use ancestral sampling, that's is we compute a categorical distribution for time step t, and draw a random outcome for that position, then include the random outcome in the available context for the next time step and repeat the process till we sample end of sequence or reach a pre-determined maximum length.

If the model works well, this should mostly generate valid word forms, which means our model managed to concentrate probability mass on the space of valid Spanish words (rather than the full space of strings made of characters in the Spanish alphabet, which is a large set full of nonsense).

```python
def ancestral_sampling(model, vocab, batch_size=None, max_len=50, device=None):
    """
    Greedily decodes a word by picking the character with
    maximum probability at each time step.
    :params model: CharLM
    :param vocab: a Vocabulary
    :param max_len: cap on length for performance (GPU memory is not unbounded)

    :return a batch of samples
    """

    # Disable dropout.
    model.eval()

    batch_size = 1 if batch_size is None else batch_size

    # Don't compute gradients.
    with torch.no_grad():

        # We feed the model the start-of-word symbol at the first time step.
        prev_x = torch.ones(batch_size, 1, dtype=torch.long).fill_(vocab[SOW_TOKEN]).to(device)

        # Initialize the hidden state from z.
        hidden = None

        samples = []
        for t in range(max_len):
            # logits: [B, V]
            logits, hidden = model.step(prev_x, hidden=hidden)
            # Cat(f(x_{<i}; theta))
            p_x = Categorical(logits=logits)
            # Random trial for X_i
            # [B]
            sample = p_x.sample()
            samples.append(sample)
            # [B, 1]
            prev_x = sample.view(batch_size, 1)
        # [B, T]
        return torch.cat(samples, dim=1)
```

# Training

Now it's time to train the model. We use early stopping on the validation perplexity for model selection.

```python
# Define the model hyperparameters.
emb_size = 256
hidden_size = 256
dropout = 0.6
batch_size = 64
learning_rate = 0.001
num_epochs = 20
########

# Create the training data loader.
dl = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
sorted_dl = SortingTextDataLoader(dl)

# Create the generative model.
model = CharLM(
    vocab_size=vocab.size(),
    emb_size=emb_size,
    hidden_size=hidden_size,
    pad_idx=vocab[PAD_TOKEN],
    dropout=dropout
)
model = model.to(device)

# Create the optimizer.
optimizer = optim.Adam(model.parameters(), lr=learning_rate)

# Save the best model (early stopping).
best_model = "./best_model.pt"
best_val_ppl = float("inf")
best_epoch = 0

# Keep track of some statistics to plot later.
train_NLLs = []
val_perplexities = []
val_NLLs = []

step = 0
training_loss = 0.
num_batches = 0
for epoch_num in range(1, num_epochs+1):
    for words in sorted_dl:

        # Make sure the model is in training mode (for dropout).
        model.train()

        # Transform the words to input, output, seq_len, seq_mask batches.
        x_in, x_out, seq_mask, seq_len = create_batch(words, vocab, device)

        # Output distributions X_i|x_{<i}
        p_x = model(x_in)
        loss, _ = model.loss(p_x, x_out)

        # Backpropagate and update the model weights.
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

        # Update some statistics to track for the training loss.
        training_loss += loss
        num_batches += 1

        # Every 100 steps we evaluate the model and report progress.
        if step % 100 == 0:
            val_ppl, val_NLL = eval_perplexity(model, val_dataset, vocab, device)
            print("(%d) step %d: training loss = %.2f --"
                  " validation PPL = %.2f "
                  " validation NLL = %.2f " %
                  (epoch_num, step, training_loss,
                   val_ppl, val_NLL))

            # Update some statistics for plotting later.
            train_NLLs.append((step, (training_loss/num_batches).item()))
            val_perplexities.append((step, val_ppl.item()))
            val_NLLs.append((step, val_NLL.item()))

            # Reset the training statistics.
            training_loss = 0.
            num_batches = 0

        step += 1

    # After an epoch we'll compute validation perplexity and save the model
    # for early stopping if it's better than previous models.
    print("Finished epoch %d" % (epoch_num))
    val_perplexity, val_NLL = eval_perplexity(model, val_dataset, vocab, device)

    # If validation perplexity is better, store this model for early stopping.
    if val_perplexity < best_val_ppl:
        best_val_ppl = val_perplexity
        best_epoch = epoch_num
        torch.save(model.state_dict(), best_model)

    # Print epoch statistics.
    print("Evaluation epoch %d:\n"
          " - validation perplexity: %.2f\n"
          " - validation NLL: %.2f"
          % (epoch_num, val_perplexity, val_NLL))

    # Also show some qualitative results by sampling wordforms from the model.
    samples = ancestral_sampling(model, vocab, batch_size=5, device=device)
    for sample in batch_to_words(samples, vocab):
        print("-- Model generation: \"%s\"" % sample)
```

# Let's plot the training and validation statistics:

```python
steps, training_NLL = list(zip(*train_NLLs))
_, val_ppl = list(zip(*val_perplexities))
_, val_NLL = list(zip(*val_NLLs))

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(18, 5))

# Plot training loss
ax1.set_title("Training loss")
ax1.plot(steps, training_NLL, "-o")
plt.show()

# Plot validation perplexities.
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(18, 5))
ax1.set_title("Validation perplexity")
ax1.plot(steps, val_ppl, "-o", color="orange")
ax2.set_title("Validation NLL")
ax2.plot(steps, val_NLL, "-o",  color="orange")
plt.show()
print()
```

Let's load the best model according to validation perplexity and compute its perplexity on the test data:

```python
# Load the best model from disk.
model = CharLM(
    vocab_size=vocab.size(),
    emb_size=emb_size,
    hidden_size=hidden_size,
    pad_idx=vocab[PAD_TOKEN],
    dropout=dropout)
model.load_state_dict(torch.load(best_model))
model = model.to(device)

# Compute test perplexity and ELBO.
test_perplexity, test_NLL = eval_perplexity(model, test_dataset, vocab, device)
print("test perplexity = %.2f -- test NLL = %.2f" %
      (test_perplexity, test_NLL))
```

# Qualitative analysis

Let's generate some wordforms from our model and inspect what our model assigns non-negligible probability mass to.

```python
# Generate 10 samples from the standard normal prior.
num_samples = 10

samples = ancestral_sampling(model, vocab, batch_size=num_samples, device=device)
for sample in batch_to_words(samples, vocab):
    print("-- Model generation: \"%s\"" % sample)
```

```python

```


# GaiaLab - Introductory infos

Toy model in which the snapshots are taken continuously, i.e. not at discrete time steps.

---

## Content:

- [#1.-Scanner-Object](#1.-Scanner-Object)
- [#2.-Source-detection](#2.-Source-detection)
    - Sirio
    - Vega
    - Proxima Centauri
- [#3.-For-more](#3.-For-more)

---

## Few definitions: (from wikipedia:)
* Attitude:   https://en.wikipedia.org/wiki/Attitude_control
* Parallax:   https://en.wikipedia.org/wiki/Parallax
* B-Spline:   https://en.wikipedia.org/wiki/B-spline
* Quaternion: https://en.wikipedia.org/wiki/Quaternion
* CCD (Charge-Coupled Device): https://en.wikipedia.org/wiki/Charge-coupled_device
* BCRS and GCRS (Barycentric/Geocentric Celestial Reference System): https://en.wikipedia.org/wiki/Barycentric_celestial_reference_system
* Some other things about references frames: https://www.gaia.ac.uk/science/astronomical-coordinate-systems
* TCB (Barycentric Coordinate Time): https://en.wikipedia.org/wiki/Barycentric_Coordinate_Time
* Topocentric function: https://en.wikipedia.org/wiki/Horizontal_coordinate_system

---
## Simplifications
* The real orbit in lissajous around L2, but in the simplified version we have a tilted orbit around the sun

```python

```

# Abstract

This notebook explains the wotking principle of the scanner object, together with auxiliary source and satellite objects

```python
# To use interact -- IPython widget
from __future__ import print_function
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets

# Generic module imports import
from IPython.display import Image
import sys
import numpy as np
from matplotlib import pyplot as plt

# append to path the folder that contains the analytic scanner and import local files
sys.path.append('../GaiaLab/scan/analytic_scanner')

import constants as const

from scanner import Scanner
from satellite import Satellite
from source import Source

import analytic_plots as plots
from agis_functions import scanning_direction
from agis_functions import get_angular_FFoV_PFoV
from agis_functions import *

# Ipython magics
%load_ext autoreload
%autoreload 2
# for pseudo interactive plots
# %matplotlib notebook

# Set some user-specific variables
MAIN_FOLDER = '../' # relative path to project folder
FIG_FOLDER = './figures/' # relative path to figure
```

**About the frames we will use**

Here is an image representing the frames:

Figure 1. Definition of angles $\epsilon$, $\lambda$, $\nu$, $\xi$, and $\Omega$ in the nominal scanning law.

The nominal scanning law of Gaia is described by the following angles:
* $\epsilon$: obliquity of equator. This is a $\it{constant}$ chosen to be 23º 26' 21.448''
* $\xi$: revolving angle. At any time the z axis is at this $\it{constant}$ angle from $\vec{s}$. For Gaia, the current choice is 55º.
* $\lambda_{s}(t)$: nominal longitud of the sun in the ecliptic plane. This is chosen to have a constant speed: $\dot{\lambda}(t) = \frac{2\pi}{365}$
* $\nu(t)$: revolving phase. Gives a nearly constant precession rate.
* $\Omega(t)$: spin phase.

Frames of referenced used:
* Interial Frame: $\textbf{G} = \big[ \textbf{l},\textbf{m},\textbf{n}\big]$ gaia non-turning wrt to sun reference system
* Celestial frame of reference (ort): $\textbf{N} = \big[ \textbf{l},\textbf{j},\textbf{k}\big]$ Sun reference
* Instrument triad frame (ort): $\textbf{Z} = \big[ \textbf{x},\textbf{y},\textbf{z}\big]$  Gaia reference turning
* Ecliptic triad $\textbf{E}= \big[ \textbf{l},\textbf{j},\textbf{k}\big]$

Given by:
$$
\textbf{k} = \textbf{n}cos(\epsilon) - \textbf{m} sin(\epsilon) \\
\textbf{j} = \textbf{m}cos(\epsilon) + \textbf{n} sin(\epsilon) \\ \textbf{s}=\textbf{l}cos(\lambda_{s}(t)) + \textbf{j} sin(\lambda_{s}(t)
$$

The total inertial rotation of the telescope is therefore:
$$
\begin{equation}
\vec{\omega} = \textbf{k} \dot{\lambda_{s}} + \textbf{s} \dot{\nu} + \textbf{z} \dot{\Omega}
\end{equation}
$$

The precessional motion of the z-axis is:

$$
\begin{equation}
\dot{z} = \omega \times z= \big( k \times z\big)\dot{\lambda_{s}} + \big(s \times z\big)\dot{\nu}
\end{equation}
$$

And if we consider the rate of change of the z-axis wrt time to be a constant:
$$
\begin{equation}
\frac{dz}{dt}\frac{dt}{d\lambda_{s}} = \frac{dz}{dt} \dot{\lambda_{s}}^{-1} = S
\end{equation}$$

And keeping the condition $ \|\big( k \times z \big)\|^{2} = 1 - sin^{2}(\xi)sin^{2}(\nu)$ then the following ODE for $\nu$ is obtained:

$$
\begin{equation}
\dot{\nu} = \dot{\lambda_{s}}\frac{\sqrt{S^{2} - cos^{2}(\nu)} + cos(\xi)sin(\nu)}{sin(\xi)}
\end{equation}
$$

Taking the positive sign of the square root implying increasing $\nu$.
Then, for constant inertial spin rate about the z axis using the equation for the inertial rotation $\omega$, we obtain:
$$
\begin{equation}
\dot{\Omega} = \omega_{z} - \dot{\nu}cos\xi - \dot\lambda_{s}sin\xi sin\nu
\end{equation}
$$

To see how does the longitude and latitude of the z axis changes with time, according to Figure 1, the geometry gives the following equations in the ecliptic plane:
$$
\begin{equation}
\lambda_{z}(t) = \lambda_{s}(t) + arctan\big[ tan\xi cos\nu(t) \big]. \\
\beta_{z}(t) = arcsin \big[ sin\xi sin\nu(t) \big].
\end{equation}
$$

Frames of referenced used:
* Celestial Frame: $\textbf{G} = \big[ \textbf{l},\textbf{m},\textbf{n}\big]$
* Instrument Frame: $\textbf{Z} = \big[ \textbf{x},\textbf{y},\textbf{z}\big]$
* Barycentric Reference System $\textbf{BCRS}= \big[ \textbf{l'},\textbf{m'},\textbf{n'}\big]$

The BCRS is a frame of reference that has its axis all parallel to G-Frame, but that is displaced by 1AU. That is to say, it is the frame centered at the sun.

# 1. Construct Object

```python
t_init = 0  # [days] initial time of satellite position wrt epoch J2000. This is also the time followed by the scanner.
t_end = 365*5  # [days] end time at which we compute satellite position.
my_dt = 1/24 # [days] time step for the computation of the satellite attitude

# Create the Satellite object
gaia = Satellite(t_init, t_end, my_dt)

# Create the Scanner object
scanner = Scanner()
```

### 1.1 Check properties of objects

##### 1.1.1 properties of the Satellite

Plotting the time evolution of the attitude components, begining at $t=0$ up to $t=80$ days.

```python
# plot the attitude of the satellite
plots.plot_attitude(gaia, ti=0, tf=80, n_points=1000, style='-')
```

 ##### 1.1.2 Scanner properties
 We just check some of the properties of the scanner:

```python
print('Size of the field of view of the scanner: ', np.degrees(scanner.zeta_limit),'°')
print('Is the scanner using two telescope in a gaia-like fascion? ', scanner.double_telescope)
```

    Size of the field of view of the scanner:  0.5 °
    Is the scanner using two telescope in a gaia-like fascion?  True

# 2. Source detection

In this section we create example stars from real data. The Source object takes as inputs:

> Source('name', $\alpha$, $\delta$, parallax, $\mu_{\alpha}$, $\mu_{\delta}$, $\mu_{r}$)

with units: [string, deg, deg, mas, mas/yr, mas/yr, km/s]

### 2.1 Create source

```python
# Comments / uncomment lines to play with different sources

# source = Source("sirio", 101.28, -16.7161, 379.21, -546.05, -1223.14, -7.6)
source = Source("vega", 279.2333, 38.78, 128.91, 201.03, 286.23, -13.9)
# source = Source("proxima",217.42, -62, 768.7, 3775.40, 769.33, 21.7)
```

##### 2.1.1 Visualize the evolution of the source in time

```python
# With respect to equatorial coordinates:
fig = plots.plot_star_trajectory(source, gaia);
```

```python
plots.plot_ICRS_coordinates_versus_time(source, gaia)
```

### 2.2 Scan source

```python
time_taken_for_scan = scanner.scan(gaia, source, t_init, t_end)
scanner.compute_angles_eta_zeta(gaia, source)  # computes the field angle associated to each observation
print('Scan took: ', time_taken_for_scan, 'seconds')
print('Source observed ', len(scanner.obs_times), 'times')
```

    Scan took:  2.1216583251953125 seconds
    Source observed  43 times

##### 2.2.1 Visualize scan

```python
fig = plots.plot_star_trajectory(source, gaia, obs_times=scanner.obs_times, equatorial=True,
                                 show_scanning_directions=False);
```

Where the red dots represent the position of the star at the time of observation.

##### 2.2.2 Quantitative results

Check the deviation from 0 in the along scan direction:

```python
print('Scan error in the along scan direction: ', scanner.scanner_error(), '[rads], \nor equivalently,',
       scanner.scanner_error()/const.rad_per_mas, '[mas]')
```

    Scan error in the along scan direction:  1.9494987634786976e-14 [rads],
    or equivalently, 4.021129847278875e-06 [mas]

Check that all the observations are inside the limitations of the Field of View (FoV) which is [-0.5°, 0.5°]:

```python
# Check that all the observations are in the field of view:
plt.figure()
plt.title('Across scan angle of the observations')
plt.plot(scanner.zeta_scanned, '+', label='Observation')
plt.hlines(np.radians([-0.5, 0.5]), xmin=0, xmax=len(scanner.obs_times), label='| 0.5°|')
plt.xlim(0, len(scanner.obs_times))
plt.xlabel('Index of the observation')
plt.ylabel('$\zeta$ [rads]')
plt.legend()
plt.show();
```

Where $\zeta$ is the Across scan angle of the observation.

# 3. Conclusion

Thank you for reading! We hope tha tthis notebook could help you understand the model and we hope you had fun discovering it!
For more about this project, visit the others notebooks as well as the gaia website.

_For any errors or remarks, don't hesitate to contact us!_

```python

```


# Cocco, Gomes & Maenhout

# "[Consumption and Portfolio Choice over the Life Cycle](https://academic.oup.com/rfs/article-abstract/18/2/491/1599892)"

- Notebook created by Mateo Velásquez-Giraldo
- All figures and tables were taken from the linked paper.

### Summary

This article uses dynamic structural model to analyze the optimal portfolio allocation between a risky and a risk-fee asset over the life cycle. It finds that human wealth acts as an imperfect substitute for the risk-free asset, generating an optimal risky asset share that decreases over the life cycle. The model is further used to show that portfolio choices arising from models without life-cycle considerations (e.g, Merton's) generate substantial welfare losses.

### The base model

The authors' aim is to represent the life cycle of a consumer that is exposed to uninsurable labor income risk and chooses how to allocate his savings between a risky and a safe asset, without the possibility to borrow or short sell.

##### Dynamic problem

The problem of an agent $i$ of age $t$ in the base model is recursively represented as

\begin{split}
V_{i,t} =& \max_{0\leq C_{i,t} \leq X_{i,t}, \alpha_{i,t}\in[0,1]} U(C_{i,t}) + \delta p_t E_t\{ V_{i,t+1} (X_{i,t+1}) \}\\
&\text{s.t}\\
&X_{i,t+1} = Y_{i,t+1} + (X_{i,t} - C_{i,t})(\alpha_{i,t} R_{t+1} + (1-\alpha_{i,t})\bar{R}_f)
\end{split}

where $C_{i,t}$ is consumption, $\alpha_{i,t}$ is the share of savings allocated to the risky asset, $Y_{i,t}$ is labor income, and $X_{i,t}$ represents wealth. The utility function $U(\cdot)$ is assumed to be CRRA in the base model. The discount factor is $\delta$ and $p_t$ is the probability of survival from $t$ to $t+1$. Death is certain at a maximum period $T$.

Note that the consumer can not borrow or short-sell.

#### Labor income

An important driver of the paper's results is the labor income process. It is specified as follows:

\begin{equation}
\ln Y_{i,t} = f(t,Z_{i,t}) + v_{i,t} + \epsilon_{i,t}, \quad \text{for } t\leq K.
\end{equation}

where $K$ is the (exogenous) age of retirement, $Z_{i,t}$ are demographic characteristics, $\epsilon_{i,t}\sim \mathcal{N}(0,\sigma^2_\epsilon)$ is a transitory shock, and  $v_{i,t}$ is a permanent component following a random walk

\begin{equation}
v_{i,t} = v_{i,t-1} + u_{i,t} = v_{i,t-1} + \xi_t + \omega_{i,t}
\end{equation}

in which the innovation is decomposed into an aggregate ($\xi_t$) and an idiosyncratic component ($\omega_{i,t}$), both following mean-0 normal distributions.

Post-retirement income is a constant fraction $\lambda$ of income in the last working year $K$.

A crucial aspect of the labor income process is that $f(\cdot,\cdot)$ is calibrated to match income profiles in the PSID, capturing the usual humped shape of income across lifetime.

#### Assets and their returns

There are two assets available for consumers to allocate their savings.

- Bonds: paying a risk-free return $\bar{R}_f$.

- Stocks: paying a stochastic return $R_t = \bar{R}_f + \mu + \eta_t$, where the stochastic component $\eta_t \sim \mathcal{N}(0, \sigma^2_\eta)$ is allowed to be correlated with the aggregate labor income innovation $\xi_t$.

### Key Results

I now report the main results of the article.

#### The optimal risky asset share

The next figure shows the policy function for the risky portfolio share as a function of wealth at different ages.

The optimal risky share is decreasing in wealth. The authors argue this is due to the fact that, at low levels of wealth, relatively safe human wealth represents a higher fraction of the consumer's wealth, so he shifts his tradeable wealth towards riskier alternatives.

Analyzing the policy rule by age also shows that the risky share increases from young to middle age, and decreases from middle to old age. This is consistent with the previous interpretation: shares trace the humped shape of labor earnings.

#### The welfare implications of different allocation rules

The authors next conduct a welfare analysis of different allocation rules, including popular heuristics. The rules are presented in the next figure.

The utility cost of each policy in terms of constant consumption streams with respect to the authors calculated optimal policy function is reported in the next table.

Interestingly, the "no-income" column corresponds to the usual portfolio choice result of the optimal share being the quotient of excess returns and risk times relative risk aversion, disregarding labor income. The experiment shows this allocation produces substantial welfare losses.

### Conclusion

This article provides a dynamic model with accurate lifetime income profiles in which labor income increases risky asset holdings, as it is seen as a closer substitute of risk-free assets. It finds an optimal risky asset share that decreases in wealth and with age, after middle age. The model is also used to show that ignoring labor income for portfolio allocation can generate substantial welfare losses.