Formalizing Informal Text using Natural Language Processing

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import joblib
import re
import os
from sklearn.model_selection import train_test_split
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.layers import Embedding, LSTM, Dense, Dropout
from tensorflow.keras.models import Model
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
from nltk.translate.bleu_score import sentence_bleu
import nlpaug.augmenter.word as naw
import nlpaug.augmenter.sentence as nas
title_font = {'family': 'serif', 'color': 'darkred', 'weight': 'bold', 'size': 18}
label_font = {'family': 'Arial', 'weight': 'normal', 'size': 16}
import warnings
warnings.filterwarnings("ignore")
%matplotlib inline
```

1. Attention Model:

While simple encoder decoder seq2seq model works well for shorter sequences, it badly struggles for longer sequences. This is because of the fact that output token at a particular timestep might be dependent on a token parsed by the encoder a while back. But decoder model only gets to know the output token of the current step. This is where Attention model comes in. It introduces a simple architecture between encoder and decoder to enable the decoder to consider weighted outputs of encoder of all the previous timesteps.

```python
from IPython.display import Image
Image(filename = "Attention.jpg")
```

The figure above shows the mechanism of attention based encoder decoder model. As you can see, the encoder model only differs from a simple encoder decoder model in that it generates an output for each timestep alongwith the hidden state. All the encoder outputs are then fed to the attention model, where the weights corresponding to all the encoder outputs are calculated to enable the decoder to focus on certain tokens while making predictions. The input to the decoder is then computed by weighing the ground truth token with exponent of a concatenated output of hidden state at previous timestep and attention weights to make a prediction.

The important part of attention model is to calculate the weights of output encoder tokens also known as attention weights. These weights are computed by using specific scoring functions. We will consider three types of scoring functions in this case study namely Dot, General and Concat.

\begin{align}
H_t\ &=\ \sum_{i}^{}\alpha\ .\ Y
\\
\alpha_t\ &=\ \frac{exp(E_t)}{\sum_{}^{}exp(E_t)}
\\
\\
E_t\ &=\ H_{t}^{\ T}\ Y_i\ &\text{Dot}
\\
&=\ H_{t}^{\ T}\ \ W\ Y_i\ &\text{General}
\\
&=\ v^{\ T}\ tanh(W\ [H_t,\ Y_i])\ &\text{Concat}
\end{align}

Hence, in total we will train three Attention based Encoder Decoder models using Dot, General and Concat scoring functions.

2. Loading and Preprocessing Data:

As we saw with simple encoder decoder model, the data scarcity is hampering the performance badly. Hence, here we will try data augmnetation techniques as well.

2.1. Loading data:

First, we will load the dataset and won't preprocess it here as we have to augment it as well.

```python
# Reading the file
f = open("en2cn-2k.en2nen2cn", "r", encoding = 'utf-8')
text = f.read()
# Removing last instance after splitting as it is empty string
text = text.split('\n')[:-1]
# Creating the pandas dataframe
data = [[text[i], text[i+1]] for i in list(range(0, 6000, 3))]
df = pd.DataFrame(data, columns = ['Informal text', 'Formal text'])
df.head()
```

      Informal text
      Formal text

      0
      U wan me to "chop" seat 4 u nt?
      Do you want me to reserve seat for you or not?

      1
      Yup. U reaching. We order some durian pastry a...
      Yeap. You reaching? We ordered some Durian pas...

      2
      They become more ex oredi... Mine is like 25.....
      They become more expensive already. Mine is li...

      3
      I'm thai. what do u do?
      I'm Thai. What do you do?

      4
      Hi! How did your week go? Haven heard from you...
      Hi! How did your week go? Haven't heard from y...

2.2. Augmenting the Dataset:

For data augmentation, we can use nlpaug library. For our purpose, synonym augmentation and spelling augmentation are the suitable techniques. For each formal sentence in the dataset, we will first add synonym augmented pairs to get 4000 instances. On top of that, we will apply spelling augmnetations to get 8000 instances in total.

```python
# Applying Synonym augmentation
aug = naw.SynonymAug(aug_src = 'wordnet')
for text in df['Formal text'].values:
    augmented = pd.DataFrame({"Informal text":[aug.augment(text)], "Formal text":[text]})
    df = df.append(augmented, ignore_index = True)
# Applying Spelling augmentation
aug = naw.SpellingAug()
for text in df['Formal text'].values:
    augmented = pd.DataFrame({"Informal text":[aug.augment(text)], "Formal text":[text]})
    df = df.append(augmented, ignore_index = True)
df.tail()
```

      Informal text
      Formal text

      7995
      Hmm. I think I uauslly boock on weenkends. It ...
      Hmm. I think I usually book on weekends. It de...

      7996
      Can uou ask them wheaher they have [[por any s...
      Can you ask them whether they have for any sms...

      7997
      We aru near Coca already.
      We are near Coca already.

      7998
      Hall ereven. Got lectures. And forgte about co...
      Hall eleven. Got lectures. And forget about co...

      7999
      I bring for u. I can not promess you 100% to w...
      I bring for you. I can not promise you 100% to...

2.3. Preprocessing the Dataset:

The models we will design are known as sequence to sequence models as we are providing a text sequence as input and expect the text sequence as output. For that, the input to the encoder should be encoded with start of sentence and end of sentence tokens as it will enable encoder to know span of each sentence. We can use '<' and '>' tokens for initiation and termination respectively. For decoder, input should be appended with '<' token at the beginning and output should be appended with '>' token at the end.

```python
# Creating encoder inp, decoder inp and decoder_out
encoder_inp = '<' + df['Informal text'].astype(str) + '>'
decoder_inp = '<' + df['Formal text'].astype(str)
decoder_out = df['Formal text'].astype(str) + '>'
# Creating the dataframe
preprocessed_data = pd.DataFrame()
preprocessed_data['encoder_inp'] = encoder_inp
preprocessed_data['decoder_inp'] = decoder_inp
preprocessed_data['decoder_out'] = decoder_out
preprocessed_data.head()
```

      encoder_inp
      decoder_inp
      decoder_out

      0
      <U wan me to "chop" seat 4 u nt?>
      <Do you want me to reserve seat for you or not?
      Do you want me to reserve seat for you or not?>

      1
      <Yup. U reaching. We order some durian pastry ...
      <Yeap. You reaching? We ordered some Durian pa...
      Yeap. You reaching? We ordered some Durian pas...

      2
      <They become more ex oredi... Mine is like 25....
      <They become more expensive already. Mine is l...
      They become more expensive already. Mine is li...

      3
      <I'm thai. what do u do?>
      <I'm Thai. What do you do?
      I'm Thai. What do you do?>

      4
      <Hi! How did your week go? Haven heard from yo...
      <Hi! How did your week go? Haven't heard from ...
      Hi! How did your week go? Haven't heard from y...

2.4. Splitting the data into training and validation sets:

Before we split the data, we will look at the distribution of lengths of encoder inp, decoder inp and decoder_out to get the idea of input shape we will need to embed our data into.

```python
# creating axes to draw plots
fig, ax = plt.subplots(1, 3)

# plotting the distributions
sns.distplot(preprocessed_data['encoder_inp'].apply(len).values, hist = False, kde = True, kde_kws = {'shade': True, 'linewidth': 2}, color = "green", ax = ax[0])
sns.distplot(preprocessed_data['decoder_inp'].apply(len).values, hist = False, kde = True, kde_kws = {'shade': True, 'linewidth': 2}, color = "indigo", ax = ax[1])
sns.distplot(preprocessed_data['decoder_out'].apply(len).values, hist = False, kde = True, kde_kws = {'shade': True, 'linewidth': 2}, color = "orange", ax = ax[2])

# adding titles to the subplots
ax[0].set_title("Lengths of Encoder Input", fontdict = title_font, pad = 20.0)
ax[1].set_title("Lengths of Decoder Input", fontdict = title_font, pad = 20.0)
ax[2].set_title("Lengths of Decoder Output", fontdict = title_font, pad = 20.0)

# rescaling the figure
fig.set_figheight(5)
fig.set_figwidth(15)
```

As we can see, most of the sentences are of length around 50 and almost all the sentences have lengths less than 200. Hence, we can filter out the sentences which are of length more than 200.

```python
# Filtering out sentences of length more than 200
preprocessed_data = preprocessed_data[preprocessed_data['encoder_inp'].apply(len) <= 200]
preprocessed_data = preprocessed_data[preprocessed_data['decoder_inp'].apply(len) <= 200]
preprocessed_data = preprocessed_data[preprocessed_data['decoder_out'].apply(len) <= 200]
preprocessed_data.head()
```

      encoder_inp
      decoder_inp
      decoder_out

      0
      <U wan me to "chop" seat 4 u nt?>
      <Do you want me to reserve seat for you or not?
      Do you want me to reserve seat for you or not?>

      1
      <Yup. U reaching. We order some durian pastry ...
      <Yeap. You reaching? We ordered some Durian pa...
      Yeap. You reaching? We ordered some Durian pas...

      2
      <They become more ex oredi... Mine is like 25....
      <They become more expensive already. Mine is l...
      They become more expensive already. Mine is li...

      3
      <I'm thai. what do u do?>
      <I'm Thai. What do you do?
      I'm Thai. What do you do?>

      4
      <Hi! How did your week go? Haven heard from yo...
      <Hi! How did your week go? Haven't heard from ...
      Hi! How did your week go? Haven't heard from y...

We can now split the data into train, validation and test sets. As we have less data, we will split with about 90:05:05 split to use more data to train the model.

```python
train, validation = train_test_split(preprocessed_data, test_size = 0.025, random_state = 859)
train, test = train_test_split(train, test_size = 0.025, random_state = 859)
joblib.dump(train, 'train.pkl')
joblib.dump(validation, 'validation.pkl')
joblib.dump(test, 'test.pkl')
print(f"Shape of Training set: {train.shape}")
print(f"Shape of Validation set: {validation.shape}")
print(f"Shape of Test set: {test.shape}")
# for one sentence we will be adding '>' token so that the tokanizer learns the word '>'
# with this we can use only one tokenizer for both encoder output and decoder output
train.iloc[0]['encoder_inp']= str(train.iloc[0]['encoder_inp']) + '>'
```

    Shape of Training set: (7507, 3)
    Shape of Validation set: (198, 3)
    Shape of Test set: (193, 3)

2.5. Tokenizing data:

Tokenizing the data means, encoding the sentences with numbers. The numbers are assigned by an unique id from the vocabulary. So, the particular sentence will be encoded by unique ids of words occuring in that sentence. We will create the two tokenizers each for informal and formal data.

```python
# Tokenizing informal data with case preservation and excluding common punctuations like .,?!:;
tknizer_informal = Tokenizer(filters = '"#$%&()*+-/=@[\\]^_`{|}~\t\n', lower = False, char_level = True)
tknizer_informal.fit_on_texts(train['encoder_inp'].values)
joblib.dump(tknizer_informal, 'tknizer_informal.pkl')
# Tokenizing formal data with case preservation and excluding common punctuations like .,?!:;
tknizer_formal = Tokenizer(filters = '"#$%&()*+-/=@[\\]^_`{|}~\t\n', lower = False, char_level = True)
# Introducing '<end>' token on first sentence so that vocabulary learns it
train['decoder_inp'].iloc[0] = train['decoder_inp'].iloc[0] + '>'
tknizer_formal.fit_on_texts(train['decoder_inp'].values)
joblib.dump(tknizer_formal, 'tknizer_formal.pkl')
# Printing sizes of vocabularies
vocab_size_informal = len(tknizer_informal.word_index.keys())
print(f"Vocab size of Informal text: {vocab_size_informal}")
vocab_size_formal = len(tknizer_formal.word_index.keys())
print(f"Vocab size of Formal text: {vocab_size_formal}")
```

    Vocab size of Informal text: 120
    Vocab size of Formal text: 92

2.6. Padding data:

Padding refers to appending a common id (i.e. generally 0) to make all the sentences of same length. As we saw earlier, we can make the sentence lengths as 200.

```python
# Encoding the sentences by numerical ids in place of words
encoder_seq = tknizer_informal.texts_to_sequences(train['encoder_inp'].values)
decoder_inp_seq = tknizer_formal.texts_to_sequences(train['decoder_inp'].values)
decoder_out_seq = tknizer_formal.texts_to_sequences(train['decoder_out'].values)
# Padding the sentences to make all the sentences of same length
encoder_seq = pad_sequences(encoder_seq, maxlen = 200, dtype='int32', padding='post')
decoder_inp_seq = pad_sequences(decoder_inp_seq, maxlen = 200, dtype='int32', padding='post')
decoder_out_seq = pad_sequences(decoder_out_seq, maxlen = 200, dtype='int32', padding='post')
```

3. Designing the Data Pipeline:

We have to build a data pipeline to train the model as model expects tuples of length batch size of preprocessed data at runtime. We will load the source and target tokenizers and pad the data into sequences. Then, feed it according to the batch size.

3.1. Preprocessing the Data:

We will first convert sentences into sequences by tokenizing and padding.

```python
class Dataset:
    '''
    Generic class used to preprocess the data
    '''
    def __init__(self, data, tknizer_informal, tknizer_formal, max_len):
        '''
        This method intializes input sequences and the tokenizers
        '''
        self.encoder_inps = data['encoder_inp'].values
        self.decoder_inps = data['decoder_inp'].values
        self.decoder_outs = data['decoder_out'].values
        self.tknizer_informal = tknizer_informal
        self.tknizer_formal = tknizer_formal
        self.max_len = max_len

    def __getitem__(self, i):
        '''
        This method tokenizes the data and pads it with zeros to make all the sequences of same length
        '''
        # Tokenizing the sequences by passing them in lists as required by tokenizer
        self.encoder_inp_seq = self.tknizer_informal.texts_to_sequences([self.encoder_inps[i]])
        self.decoder_inp_seq = self.tknizer_formal.texts_to_sequences([self.decoder_inps[i]])
        self.decoder_out_seq = self.tknizer_formal.texts_to_sequences([self.decoder_outs[i]])
        # Padding the sequences with zeros
        self.encoder_inp_seq = pad_sequences(self.encoder_inp_seq, maxlen = self.max_len, dtype = 'int32', padding = 'post')
        self.decoder_inp_seq = pad_sequences(self.decoder_inp_seq, maxlen = self.max_len, dtype = 'int32', padding = 'post')
        self.decoder_out_seq = pad_sequences(self.decoder_out_seq, maxlen = self.max_len, dtype = 'int32', padding = 'post')
        return self.encoder_inp_seq, self.decoder_inp_seq, self.decoder_out_seq

    def __len__(self):
        '''
        This method is required by model.fit method at runtime to keep logs
        '''
        return len(self.encoder_inps)
```

3.2. Creating Dataloader:

we will now design a dataloader which shuffles the preprocessed dataset and returns the tuple of form ([[encoder_inp], [decoder_inp]], decoder_out) at runtime

```python
class Dataloader(tf.keras.utils.Sequence):
    '''
    Keras Dataloader instance to feed the model with preprocessed data at runtime
    '''
    def __init__(self, dataset, batch_size = 1):
        '''
        this method initializes preprocessed dataset and batch size
        '''
        self.dataset = dataset
        self.batch_size = batch_size
        self.indexes = np.arange(len(self.dataset.encoder_inps))

    def __getitem__(self, i):
        '''
        This method is used to pack the input data in tuples of form ([[encoder_inp], [decoder_inp]], decoder_out)
        '''
        # Tracking indices of start and stop
        start = i * self.batch_size
        stop = (i + 1) * self.batch_size
        data = []
        for j in range(start, stop):
            data.append(self.dataset[j])
        # Creating data in tuples of form ([[encoder_inp], [decoder_inp]], decoder_out)
        batch = [np.squeeze(np.stack(samples, axis = 1), axis = 0) for samples in zip(*data)]
        return tuple([[batch[0],batch[1]],batch[2]])

    def __len__(self):
        '''
        This method is required by model.fit method at runtime to keep logs
        '''
        return len(self.indexes) // self.batch_size

    def on_epoch_end(self):
        '''
        This method is a callback to shuffle the indices of data on each epoch
        '''
        self.indexes = np.random.permutation(self.indexes)
```

```python
# Defining parameters
BATCH_SIZE = 128
MAX_LEN = 200
# Preprocessing data
train_dataset = Dataset(train, tknizer_informal, tknizer_formal, MAX_LEN)
validation_dataset  = Dataset(validation, tknizer_formal, tknizer_formal, MAX_LEN)
# Creating Dataloader
train_dataloader = Dataloader(train_dataset, batch_size = BATCH_SIZE)
validation_dataloader = Dataloader(validation_dataset, batch_size = BATCH_SIZE)
# Checking the dimensions
print(train_dataloader[0][0][0].shape, train_dataloader[0][0][1].shape, train_dataloader[0][1].shape)
```

    (128, 200) (128, 200) (128, 200)

4. Designing the Attention based Encoder Decoder Model:

4.1. Designing Encoder:

The encoder will take sequential word embeddings of the source sentences as input at each time step, and encode its information in encoded vector using current state and LSTM hidden state. Hence, at the output of encoder, we get an encoded vector of source sentence which can be thought of as latent information vector.

```python
class Encoder(tf.keras.Model):
    '''
    Encoder model takes a input sequence and returns Encoder outputs as encoder_final_hidden_state, encoder_final_current_state
    '''
    def __init__(self, inp_vocab_size, embedding_dim, lstm_size, input_length):
        '''
          This method intializes the Encoder model.
        '''
        super().__init__()
        # Initializing the parameters
        self.inp_vocab_size = inp_vocab_size
        self.embedding_dim = embedding_dim
        self.lstm_size = lstm_size
        self.input_length = input_length
        # Initializing Embedding layer
        self.embedding = Embedding(input_dim = self.inp_vocab_size, output_dim = self.embedding_dim,
                                   embeddings_initializer = tf.keras.initializers.RandomNormal(mean = 0, stddev = 1, seed = 859),
                                   input_length = self.input_length, mask_zero = True, name = "Encoder_Embedding")
        #Intializing Encoder LSTM layer
        self.lstm1 = LSTM(self.lstm_size, return_state = True, return_sequences = True,
                          kernel_initializer = tf.keras.initializers.glorot_uniform(seed = 859),
                          recurrent_initializer = tf.keras.initializers.orthogonal(seed = 859), name = "Encoder_LSTM1")
        self.lstm2 = LSTM(self.lstm_size, return_state = True, return_sequences = True,
                          kernel_initializer = tf.keras.initializers.glorot_uniform(seed = 859),
                          recurrent_initializer = tf.keras.initializers.orthogonal(seed = 859), name = "Encoder_LSTM2")

    def call(self, input):
        '''
          This method takes a sequence input and the initial states of the Encoder as inputs.
          Sequence input is passed to the Embedding layer and initial states are passed to the LSTM layer of the Encoder model.
          It returns Encoder outputs as last time step's hidden and current states.
        '''
        # Unpacking the input
        input_sequence, states = input[0], input[1]
        # Passing input sequence to embedding layer
        input_embedded = self.embedding(input_sequence)
        # Passing embedidng layer output to lstm layer
        self.enc_output, self.last_hidden_state, self.last_current_state = self.lstm1(input_embedded, initial_state = states)
        self.enc_output, self.last_hidden_state, self.last_current_state = self.lstm2(self.enc_output, [self.last_hidden_state, self.last_current_state])
        # Returning the outputs
        return self.enc_output, self.last_hidden_state, self.last_current_state

    def initialize_states(self, batch_size):
      '''
        Given a batch size this method will return intial hidden state and intial current state.
        If batch size is 32, Hidden state is zeros of size [32,lstm_units], current state is zeros is of size [32,lstm_units]
      '''
      self.first_hidden_state, self.first_current_state = tf.zeros([batch_size, self.lstm_size]), tf.zeros([batch_size, self.lstm_size])
      # Returning the initializations
      return self.first_hidden_state, self.first_current_state
```

4.2. Designing Attention Model:

Attention model takes two inputs in the form of decoder hidden state of previous timestep and encoder output and calculates attention weights.

```python
class Attention(tf.keras.Model):
    '''
        Attention model takes two inputs in the form of decoder_hidden_state, encoder_output and
        returns context vector and attention weights(softmax - scores).
    '''
    def __init__(self, lstm_size, scoring_function):
        super(Attention, self).__init__()
        # Initializing the parameters
        self.lstm_size = lstm_size
        self.scoring_function = scoring_function
        # Initializing weights for 'dot' scoring function
        if self.scoring_function=='dot':
            self.V = tf.keras.layers.Dense(1)
        # Initializing weights for 'general' scoring function
        if scoring_function == 'general':
            self.W = tf.keras.layers.Dense(lstm_size)
        # Initializing weights for 'concat' scoring function
        if scoring_function == 'concat':
            self.W1 = tf.keras.layers.Dense(lstm_size)
            self.W2 = tf.keras.layers.Dense(lstm_size)
            self.V = tf.keras.layers.Dense(1)

    def call(self,input):
        '''
            Attention model takes two inputs in the form of decoder_hidden_state and all the encoder_outputs.
            Based on the scoring function we will find the score or similarity between decoder_hidden_state and encoder_output.
            Multiply the score function with your encoder_outputs to get the context vector.
            Returns context vector and attention weights(softmax - scores)
        '''
        # Unpacking the input
        decoder_hidden_state, encoder_output = input[0], input[1]
        if self.scoring_function == 'dot':
            # Implement Dot score function here
            query_with_time_axis = tf.expand_dims(decoder_hidden_state, 1)
            score = self.V(tf.linalg.matmul(encoder_output, query_with_time_axis, transpose_a=False, transpose_b=True))

        elif self.scoring_function == 'general':
            # Implement General score function here
            decoder_hidden_state = tf.expand_dims(decoder_hidden_state, axis = 2)
            output = self.W(encoder_output)
            score = tf.keras.layers.Dot(axes=(2, 1))([output, decoder_hidden_state])
        if self.scoring_function == 'concat':
            # Implement General score function here
            decoder_hidden_state = tf.expand_dims(decoder_hidden_state, 1)
            score = self.V(tf.nn.tanh(self.W1(decoder_hidden_state) + self.W2(encoder_output)))
        # Calculating context vector and attention weights
        attention_weights = tf.nn.softmax(score, axis=1)
        context_vector = attention_weights * encoder_output
        context_vector = tf.reduce_sum(context_vector, axis=1)
        return context_vector, attention_weights
```

4.3. Designing Timestep Decoder:

For each time step, Timestep decoder will implement concatenation operation on output of previous timestep of decoder and attention weights computed by attention model.

```python
class Timestep_Decoder(tf.keras.Model):
    '''
        Timestep Decoder model takes one input token at a time and returns final hidden state of the LSTM unit
    '''
    def __init__(self, out_vocab_size, embedding_dim, input_length, lstm_size, scoring_function, embedding_matrix = None):
        # Initialize the parameters
        super().__init__()
        self.out_vocab_size = out_vocab_size
        self.embedding_dim = embedding_dim
        self.input_length = input_length
        self.lstm_size = lstm_size
        self.scoring_function = scoring_function
        self.attention = Attention(self.lstm_size, self.scoring_function)
        self.embedding_matrix = embedding_matrix
        # Initializing Embedding layer based on the availability of Embedding matrix
        if self.embedding_matrix is None:
            self.embedding = Embedding(input_dim = self.out_vocab_size, output_dim = self.embedding_dim,
                                       embeddings_initializer = tf.keras.initializers.RandomNormal(mean = 0, stddev = 1, seed = 859),
                                       input_length = self.input_length, mask_zero = True, name = "embedding_layer_decoder")
        else:
            self.embedding = Embedding(input_dim = self.out_vocab_size, output_dim = self.embedding_dim,
                                       input_length = self.input_length, mask_zero = True, name = "embedding_layer_decoder",
                                       embeddings_initializer = tf.keras.initializers.Constant(self.embedding_matrix), trainable = False)
        #Intialize Decoder LSTM layer
        self.lstm1 = LSTM(self.lstm_size, return_state = True, return_sequences = True,
                          kernel_initializer = tf.keras.initializers.glorot_uniform(seed = 859),
                          recurrent_initializer = tf.keras.initializers.orthogonal(seed = 859), name = "Timestep_Decoder_LSTM1")
        self.lstm2 = LSTM(self.lstm_size, return_state = True, return_sequences = True,
                          kernel_initializer = tf.keras.initializers.glorot_uniform(seed = 859),
                          recurrent_initializer = tf.keras.initializers.orthogonal(seed = 859), name = "Timestep_Decoder_LSTM2")
        #Intialize Dense layer(tar_vocab_size) without activation='softmax'
        self.dense = Dense(out_vocab_size)

    def call(self, input):
        '''
            Timestep decoder model generates final lstm unit hidden state depending on input token and earlier encoder states
        '''
        # Unpacking the input
        input_token, encoder_output, encoder_hidden, encoder_current = input[0], input[1], input[2], input[3]
        # Passing the decoder_input to the embedding layer
        embedded_token = self.embedding(input_token)
        # Passing encoder hidden and encoder output to attention model to get context vector
        context_vector, attention_weights = self.attention([encoder_hidden, encoder_output])
        # Reshaping context vector for concatenation
        query_with_time_axis = tf.expand_dims(context_vector, 1)
        # Concatenating context vector and embedded token
        out_concat = tf.concat([query_with_time_axis, embedded_token], axis = -1)
        # Getting final lstm hidden state
        dec_output, encoder_hidden, encoder_current = self.lstm1(out_concat, [encoder_hidden, encoder_current])
        dec_output, encoder_hidden, encoder_current = self.lstm2(dec_output, [encoder_hidden, encoder_current])
        # Weighing decoder output by output vocabulary size
        out = self.dense(tf.reshape(dec_output, (-1, dec_output.shape[2])))
        # Returning the output
        return out, encoder_hidden, encoder_current
```

4.4. Designing Decoder:

Decoder model simply calls timestep decoder at each timestep and generates the final output tokens.

```python
class Decoder(tf.keras.Model):
    '''
        Decoder model generates the final output tokens by passing it all the encoder states and decoder input sequence
    '''
    def __init__(self, out_vocab_size, embedding_dim, input_length, lstm_size, scoring_function, embedding_matrix = None):
        super().__init__()
        # Intializing the parameters
        self.out_vocab_size = out_vocab_size
        self.embedding_dim = embedding_dim
        self.input_length = input_length
        self.lstm_size = lstm_size
        self.scoring_function = scoring_function
        self.embedding_matrix = embedding_matrix
        # Initializing Timestep decoder instance
        self.timestepdecoder = Timestep_Decoder(self.out_vocab_size, self.embedding_dim, self.input_length,
                                                self.lstm_size, self.scoring_function, self.embedding_matrix)
    @tf.function
    def call(self, input):
        '''
            According to the length of the decoder input sequence, returns a tensor of sequence of decoded tokens as output
        '''
        # Unpacking the input
        decoder_input, encoder_output, encoder_hidden, encoder_current = input[0], input[1], input[2], input[3]
        # Initializing an empty Tensor array, that will store the outputs at each and every time step
        all_outputs = tf.TensorArray(tf.float32, size = tf.shape(decoder_input)[1], name = "output_array")
        # Iterating till the length of the decoder input
        loop = tf.shape(decoder_input)[1]
        for timestep in range(loop):
            # Calling the Timestep Decoder for each token in decoder input
            output, encoder_hidden, encoder_current = self.timestepdecoder([decoder_input[:, timestep:timestep+1], encoder_output, encoder_hidden, encoder_current])
            # Storing the output in tensorarray
            all_outputs = all_outputs.write(timestep, output)
        # Reshaping the tensor array
        all_outputs = tf.transpose(all_outputs.stack(), [1,0,2])
        # Returning the tensor array
        return all_outputs
```

4.5. Designing Final Model Architechture:

Attention based Encoder Decoder model gets the tuple of input sequences as input and implements the Encoder, Attention, Timestep Decoder and Decoder models using subclassing API.

```python
class Attention_Based_Encoder_Decoder(tf.keras.Model):
    '''
    The Attention_Based_Encoder_Decoder Model initializes both Encoder and Decoder Models and outputs normalized probabilties of the tokens in target vocabulary.
    '''
    def __init__(self, input_length, inp_vocab_size, out_vocab_size, lstm_size, scoring_function, batch_size, embedding_dim, embedding_matrix = None):
        '''
          This method intializes the both the Encoder and Decoder models
        '''
        super().__init__()
        # Initializing the parameters
        self.input_length = input_length
        self.inp_vocab_size = inp_vocab_size + 1
        self.out_vocab_size = out_vocab_size + 1
        self.lstm_size = lstm_size
        self.scoring_function = scoring_function
        self.batch_size = batch_size
        self.embedding_dim = embedding_dim
        self.embedding_matrix = embedding_matrix
        #Creating Encoder model object
        self.encoder = Encoder(inp_vocab_size = self.inp_vocab_size, embedding_dim = self.embedding_dim, lstm_size = self.lstm_size, input_length = self.input_length)
        #Creating Decoder model object
        self.decoder = Decoder(out_vocab_size = self.out_vocab_size, embedding_dim = self.embedding_dim, lstm_size = self.lstm_size,
                               scoring_function = self.scoring_function, input_length = self.input_length, embedding_matrix = self.embedding_matrix)

    def call(self, data):
        '''
        This method takes data from data pipeline in tuples of length 2, where first is encoder_inp and second is decoder_inp
        encoder_inp is fed to Encoder model object alongwith initial states whereas decoder_inp is fed to decoder model object alongwith
        Encoder last hidden and current states.
        The Model then returns normalized output probabilities of tokens in target vocabulary.
        '''
        # Unpacking data
        enc_inp, dec_inp = data[0], data[1]
        # Initializing Encoder initial states
        initial_state = self.encoder.initialize_states(self.batch_size)
        # Calling Encoder model object
        encoder_output, encoder_hidden, encoder_current = self.encoder([enc_inp, initial_state])
        # Calling Decoder model object
        final_output = self.decoder([dec_inp, encoder_output, encoder_hidden, encoder_current])
        return final_output
```

5. Designing the Model Pipeline:

5.1. Creating Custom Loss Function:

We will now create a custom loss Function that will mask the padded zeros while for more reliable loss calculation.

```python
loss_object = tf.keras.losses.SparseCategoricalCrossentropy(from_logits = True, reduction = 'none')

@tf.function
def loss_function(real, pred):
    # Custom loss function that will not consider the loss for padded zeros.
    # Refer https://www.tensorflow.org/tutorials/text/nmt_with_attention
    # optimizer = tf.keras.optimizers.Adam()
    mask = tf.math.logical_not(tf.math.equal(real, 0))
    loss_ = loss_object(real, pred)
    mask = tf.cast(mask, dtype=loss_.dtype)
    loss_ *= mask
    return tf.reduce_mean(loss_)
```

5.2. Creating Tensorboard Callback:

To keep track of loss while training the model, we will create a Tensorboard callback by providing log directory.

```python
def create_tensorboard_cb(model):
    '''
    Takes path string as input and returns tensorboard callback initialized in that path.
    '''
    import time
    root_logdir = os.path.join(os.curdir, model)
    run_id = time.strftime("run_%Y_%m_%d-%H_%M_%S")
    logdir = os.path.join(root_logdir, run_id)
    return tf.keras.callbacks.TensorBoard(logdir, histogram_freq = 1)
```

5.3. Creating Predict Function:

The predict function will take as informal input sentence and model instance with which to predict as input and return the output as prediction.

```python
def predict(input_sentence, model):
    '''
    Takes input sentence and model instance as inputs and predicts the output.
    The prediction is done by using following steps:
    Step A. Given input sentence, preprocess the punctuations, convert the sentence into integers using tokenizer used earlier.
    Step B. Pass the input_sequence to encoder. we get encoder_outputs, last time step hidden and cell state
    Step C. Initialize index of '<' as input to decoder. and encoder final states as input_states to decoder
    Step D. Till we reach max_length of decoder or till the model predicted word '>':
            pass the inputs to timestep decoder at each timestep, update the hidden states and get the output token
    Step E. Return the predicted sentence.
    '''
    # Tokenizing and Padding the sentence
    inputs = [tknizer_informal.word_index.get(i, 0) for i in input_sentence]
    inputs = tf.keras.preprocessing.sequence.pad_sequences([inputs], maxlen = MAX_LEN, padding = 'post')
    inputs = tf.convert_to_tensor(inputs)
    # Initializing result string and hidden states
    result = ''
    hidden = tf.zeros([1, UNITS]), tf.zeros([1, UNITS])
    # Getting Encoder outputs
    enc_out, state_h, state_c = model.encoder(inputs, hidden)
    dec_hidden = [state_h, state_c]
    dec_input = tf.expand_dims([tknizer_formal.word_index['<']], 0)
    # Running loop until max length or the prediction is '>' token
    for t in range(MAX_LEN):
        # Getting Decoder outputs fot timestep t
        output, state_h, state_c = model.decoder.timestepdecoder(dec_input, enc_out, state_h, state_c)
        # Getting token index having highest probability
        predicted_id = tf.argmax(output[0]).numpy()
        # Getting output token
        if tknizer_formal.index_word.get(predicted_id, '') == '>':
            break
        else:
            result += tknizer_formal.index_word.get(predicted_id, '')
            dec_input = tf.expand_dims([predicted_id], 0)
    # Postprocessing the result string to remove spaces between punctuations
    return result
```

6. Training the Model using Dot Scoring Function:

6.1. Compiling and Fitting the model:

We can now train the model by using model fit method.

```python
tf.random.set_seed(859)
# Defining model parameters
UNITS = 200
EPOCHS = 50
TRAIN_STEPS = train.shape[0]//BATCH_SIZE
VALID_STEPS = validation.shape[0]//BATCH_SIZE
# Defining model instance with 'dot' scoring function
model_dot  = Attention_Based_Encoder_Decoder(input_length = MAX_LEN, inp_vocab_size = vocab_size_informal,
                                             out_vocab_size = vocab_size_formal, lstm_size = UNITS,
                                             scoring_function = 'dot', batch_size = BATCH_SIZE,
                                             embedding_dim = vocab_size_formal, embedding_matrix = None)
# Compiling the model using 'adam' optimizer and custom loss function
optimizer = tf.keras.optimizers.Adam(0.01)
model_dot.compile(optimizer = optimizer, loss = loss_function)
# Creating callbacks to control model training
learning_rate_cb = tf.keras.callbacks.ReduceLROnPlateau(monitor = 'val_loss', factor = 0.5, verbose = 1, patience = 1, min_lr = 0.0001)
tensorboard_cb = create_tensorboard_cb("Model_Dot_logs")
stopper_cb = tf.keras.callbacks.EarlyStopping(monitor = 'val_loss', patience = 3, verbose = 1, restore_best_weights = True)
checkpoint_cb = tf.keras.callbacks.ModelCheckpoint("Model_Dot",
                                                    save_best_only = True, save_weights_only = False)
# Fitting the model on training data
model_dot.fit(train_dataloader, steps_per_epoch = TRAIN_STEPS, epochs = EPOCHS,
              callbacks = [learning_rate_cb, tensorboard_cb, stopper_cb, checkpoint_cb],
              validation_data = validation_dataloader, validation_steps = VALID_STEPS)
model_dot.summary()
```

    Epoch 1/50
    57/57 [==============================] - 122s 2s/step - loss: 1.1210 - val_loss: 1.0406
    Epoch 2/50
    57/57 [==============================] - 102s 2s/step - loss: 0.8188 - val_loss: 0.8296
    Epoch 3/50
    57/57 [==============================] - 101s 2s/step - loss: 0.6637 - val_loss: 0.6886
    Epoch 4/50
    57/57 [==============================] - 102s 2s/step - loss: 0.5683 - val_loss: 0.6106
    Epoch 5/50
    57/57 [==============================] - 102s 2s/step - loss: 0.5132 - val_loss: 0.5731
    Epoch 6/50
    57/57 [==============================] - 102s 2s/step - loss: 0.4758 - val_loss: 0.5366
    Epoch 7/50
    57/57 [==============================] - 105s 2s/step - loss: 0.4455 - val_loss: 0.5196
    Epoch 8/50
    57/57 [==============================] - 102s 2s/step - loss: 0.4224 - val_loss: 0.5007
    Epoch 9/50
    57/57 [==============================] - 101s 2s/step - loss: 0.4036 - val_loss: 0.4801
    Epoch 10/50
    57/57 [==============================] - 98s 2s/step - loss: 0.3874 - val_loss: 0.4739
    Epoch 11/50
    57/57 [==============================] - 99s 2s/step - loss: 0.3733 - val_loss: 0.4621
    Epoch 12/50
    57/57 [==============================] - 99s 2s/step - loss: 0.3595 - val_loss: 0.4504
    Epoch 13/50
    57/57 [==============================] - 99s 2s/step - loss: 0.3481 - val_loss: 0.4464
    Epoch 14/50
    57/57 [==============================] - 103s 2s/step - loss: 0.3384 - val_loss: 0.4411
    Epoch 15/50
    57/57 [==============================] - 100s 2s/step - loss: 0.3313 - val_loss: 0.4270
    Epoch 16/50
    57/57 [==============================] - 99s 2s/step - loss: 0.3221 - val_loss: 0.4309

    Epoch 00016: ReduceLROnPlateau reducing learning rate to 0.004999999888241291.
    Epoch 17/50
    57/57 [==============================] - 99s 2s/step - loss: 0.2907 - val_loss: 0.3985
    Epoch 18/50
    57/57 [==============================] - 99s 2s/step - loss: 0.2697 - val_loss: 0.3969
    Epoch 19/50
    57/57 [==============================] - 99s 2s/step - loss: 0.2599 - val_loss: 0.3947
    Epoch 20/50
    57/57 [==============================] - 98s 2s/step - loss: 0.2541 - val_loss: 0.3950

    Epoch 00020: ReduceLROnPlateau reducing learning rate to 0.0024999999441206455.
    Epoch 21/50
    57/57 [==============================] - 98s 2s/step - loss: 0.2340 - val_loss: 0.3792
    Epoch 22/50
    57/57 [==============================] - 98s 2s/step - loss: 0.2201 - val_loss: 0.3791

    Epoch 00022: ReduceLROnPlateau reducing learning rate to 0.0012499999720603228.
    Epoch 23/50
    57/57 [==============================] - 98s 2s/step - loss: 0.2079 - val_loss: 0.3774
    Epoch 24/50
    57/57 [==============================] - 98s 2s/step - loss: 0.2010 - val_loss: 0.3817

    Epoch 00024: ReduceLROnPlateau reducing learning rate to 0.0006249999860301614.
    Epoch 25/50
    57/57 [==============================] - 97s 2s/step - loss: 0.1944 - val_loss: 0.3799

    Epoch 00025: ReduceLROnPlateau reducing learning rate to 0.0003124999930150807.
    Restoring model weights from the end of the best epoch.
    Epoch 00025: early stopping
    Model: "attention__based__encoder__decoder"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    encoder (Encoder)            multiple                  566148
    _________________________________________________________________
    decoder (Decoder)            multiple                  742451
    =================================================================
    Total params: 1,308,599
    Trainable params: 1,308,599
    Non-trainable params: 0
    _________________________________________________________________

The model achieves the loss of 0.3774 on validation set which is better than that of simple encoder decoder model.

6.2. Calculating the BLEU Score:

We can now calculate the BLEU score to quantify the model performance.

```python
# Removing '<' and '>' tokens and postprocessing punctuations to make plain texts
def rem(s):
    if s.startswith('<'):
        s = s[1:]
    if s.endswith('>'):
        s = s[:-1]
    return s
test['informals'] = test['encoder_inp'].apply(rem)
test['formals'] = test['decoder_inp'].apply(rem)
def predictor(s):
    # Modifing predictor using dot scoring function
    result = predict(s, model_dot)
    return result
test['predictions'] = test['informals'].apply(predictor)
# Process inputs for Bleu score
def convert_formals(s):
    return [s.split()]
def convert_predictions(s):
    return s.split()
test['formals'] = test['formals'].apply(convert_formals)
test['predictions'] = test['predictions'].apply(convert_predictions)
bleu_scores_dot = [sentence_bleu(test['formals'].iloc[i], test['predictions'].iloc[i]) for i in range(len(test))]
print(f"Mean Bleu score of predictions: {np.mean(bleu_scores_dot)}")
```

    Mean Bleu score of predictions: 0.5053083753162261

The model achieves the BLEU score of 0.505  on test set which is significantly better than that of baseline encoder decoder model. Let us check the distribution of the bleu scores.

```python
plt.figure(figsize = (5, 5))
ax = sns.distplot(bleu_scores_dot, hist = False, kde = True, kde_kws = {'shade': True, 'linewidth': 2}, color = "green")
plt.title("Distribution of Bleu Scores by Dot Model", fontdict = title_font, pad = 20.0)
plt.xlabel("Bleu Score", fontdict = label_font)
plt.ylabel("Count", fontdict = label_font)
for label in (ax.get_xticklabels() + ax.get_yticklabels()):
    label.set_fontname('Arial')
    label.set_fontsize(14)
plt.show()
```

The distribution shows that the model with dot scoring function achieves the bleu score of around 0.7 for majority of the sentences. Let us generate a random prediction using this model.

```python
print("Informal Sentence: how r u")
print(f"Formal Prediction: {predict('how r u', model_dot)}")
```

    Informal Sentence: how r u
    Formal Sentence: How do going to your?

The model corrected the word 'how' to 'How' alongwith capitalizing the letter 'h'. It also tried to correct the word 'u' to 'you' but failed instead predicting 'your'. It also introduced the correct punctuation '?'. But more importantly, the prediction is not meaningful or convincing. This issue can be overcome by training the model on large dataset.

7. Training the Model using General Scoring Function:

7.1. Compiling and Fitting the model:

We can now train the model by using model fit method.

```python
tf.random.set_seed(859)
# Defining model instance with 'general' scoring function
model_general  = Attention_Based_Encoder_Decoder(input_length = MAX_LEN, inp_vocab_size = vocab_size_informal,
                                             out_vocab_size = vocab_size_formal, lstm_size = UNITS,
                                             scoring_function = 'general', batch_size = BATCH_SIZE,
                                             embedding_dim = vocab_size_formal, embedding_matrix = None)
# Compiling the model using 'adam' optimizer and custom loss function
optimizer = tf.keras.optimizers.Adam(0.01)
model_general.compile(optimizer = optimizer, loss = loss_function)
# Creating callbacks to control model training
tensorboard_cb = create_tensorboard_cb("Model_General_logs")
checkpoint_cb = tf.keras.callbacks.ModelCheckpoint("Model_General",
                                                    save_best_only = True, save_weights_only = False)
# Fitting the model on training data
model_general.fit(train_dataloader, steps_per_epoch = TRAIN_STEPS, epochs = EPOCHS,
              callbacks = [learning_rate_cb, tensorboard_cb, stopper_cb, checkpoint_cb],
              validation_data = validation_dataloader, validation_steps = VALID_STEPS)
model_general.summary()
```

    Epoch 1/50
    57/57 [==============================] - 129s 2s/step - loss: 1.1200 - val_loss: 1.0382
    Epoch 2/50
    57/57 [==============================] - 111s 2s/step - loss: 0.8096 - val_loss: 0.7968
    Epoch 3/50
    57/57 [==============================] - 110s 2s/step - loss: 0.6537 - val_loss: 0.6525
    Epoch 4/50
    57/57 [==============================] - 110s 2s/step - loss: 0.5507 - val_loss: 0.6274
    Epoch 5/50
    57/57 [==============================] - 110s 2s/step - loss: 0.4919 - val_loss: 0.5500
    Epoch 6/50
    57/57 [==============================] - 110s 2s/step - loss: 0.4539 - val_loss: 0.5078
    Epoch 7/50
    57/57 [==============================] - 109s 2s/step - loss: 0.4239 - val_loss: 0.4900
    Epoch 8/50
    57/57 [==============================] - 109s 2s/step - loss: 0.4014 - val_loss: 0.4712
    Epoch 9/50
    57/57 [==============================] - 109s 2s/step - loss: 0.3807 - val_loss: 0.4567
    Epoch 10/50
    57/57 [==============================] - 109s 2s/step - loss: 0.3664 - val_loss: 0.4445
    Epoch 11/50
    57/57 [==============================] - 109s 2s/step - loss: 0.3547 - val_loss: 0.4362
    Epoch 12/50
    57/57 [==============================] - 111s 2s/step - loss: 0.3449 - val_loss: 0.4233
    Epoch 13/50
    57/57 [==============================] - 110s 2s/step - loss: 0.3371 - val_loss: 0.4203
    Epoch 14/50
    57/57 [==============================] - 110s 2s/step - loss: 0.3281 - val_loss: 0.4101
    Epoch 15/50
    57/57 [==============================] - 110s 2s/step - loss: 0.3215 - val_loss: 0.4102
    Epoch 16/50
    57/57 [==============================] - 109s 2s/step - loss: 0.3175 - val_loss: 0.4020
    Epoch 17/50
    57/57 [==============================] - 109s 2s/step - loss: 0.3095 - val_loss: 0.4126
    Epoch 18/50
    57/57 [==============================] - 109s 2s/step - loss: 0.3135 - val_loss: 0.4118

    Epoch 00018: ReduceLROnPlateau reducing learning rate to 0.004999999888241291.
    Epoch 19/50
    57/57 [==============================] - 112s 2s/step - loss: 0.2767 - val_loss: 0.3613
    Epoch 20/50
    57/57 [==============================] - 112s 2s/step - loss: 0.2516 - val_loss: 0.3505
    Epoch 21/50
    57/57 [==============================] - 111s 2s/step - loss: 0.2419 - val_loss: 0.3466
    Epoch 22/50
    57/57 [==============================] - 110s 2s/step - loss: 0.2364 - val_loss: 0.3398
    Epoch 23/50
    57/57 [==============================] - 114s 2s/step - loss: 0.2353 - val_loss: 0.3473
    Epoch 24/50
    57/57 [==============================] - 112s 2s/step - loss: 0.2307 - val_loss: 0.3449

    Epoch 00024: ReduceLROnPlateau reducing learning rate to 0.0024999999441206455.
    Epoch 25/50
    57/57 [==============================] - 111s 2s/step - loss: 0.2088 - val_loss: 0.3082
    Epoch 26/50
    57/57 [==============================] - 109s 2s/step - loss: 0.1913 - val_loss: 0.2959
    Epoch 27/50
    57/57 [==============================] - 109s 2s/step - loss: 0.1842 - val_loss: 0.2935
    Epoch 28/50
    57/57 [==============================] - 109s 2s/step - loss: 0.1803 - val_loss: 0.2904
    Epoch 29/50
    57/57 [==============================] - 109s 2s/step - loss: 0.1780 - val_loss: 0.2919
    Epoch 30/50
    57/57 [==============================] - 111s 2s/step - loss: 0.1759 - val_loss: 0.2853
    Epoch 31/50
    57/57 [==============================] - 110s 2s/step - loss: 0.1726 - val_loss: 0.2886
    Epoch 32/50
    57/57 [==============================] - 109s 2s/step - loss: 0.1703 - val_loss: 0.2829
    Epoch 33/50
    57/57 [==============================] - 109s 2s/step - loss: 0.1667 - val_loss: 0.2790
    Epoch 34/50
    57/57 [==============================] - 109s 2s/step - loss: 0.1623 - val_loss: 0.2805
    Epoch 35/50
    57/57 [==============================] - 110s 2s/step - loss: 0.1639 - val_loss: 0.2845

    Epoch 00035: ReduceLROnPlateau reducing learning rate to 0.0012499999720603228.
    Epoch 36/50
    57/57 [==============================] - 109s 2s/step - loss: 0.1506 - val_loss: 0.2563
    Epoch 37/50
    57/57 [==============================] - 108s 2s/step - loss: 0.1379 - val_loss: 0.2512
    Epoch 38/50
    57/57 [==============================] - 108s 2s/step - loss: 0.1325 - val_loss: 0.2493
    Epoch 39/50
    57/57 [==============================] - 109s 2s/step - loss: 0.1307 - val_loss: 0.2484
    Epoch 40/50
    57/57 [==============================] - 109s 2s/step - loss: 0.1300 - val_loss: 0.2472
    Epoch 41/50
    57/57 [==============================] - 108s 2s/step - loss: 0.1270 - val_loss: 0.2427
    Epoch 42/50
    57/57 [==============================] - 108s 2s/step - loss: 0.1246 - val_loss: 0.2459
    Epoch 43/50
    57/57 [==============================] - 109s 2s/step - loss: 0.1259 - val_loss: 0.2400
    Epoch 44/50
    57/57 [==============================] - 109s 2s/step - loss: 0.1243 - val_loss: 0.2400
    Epoch 45/50
    57/57 [==============================] - 110s 2s/step - loss: 0.1217 - val_loss: 0.2392
    Epoch 46/50
    57/57 [==============================] - 109s 2s/step - loss: 0.1197 - val_loss: 0.2396
    Epoch 47/50
    57/57 [==============================] - 110s 2s/step - loss: 0.1166 - val_loss: 0.2358
    Epoch 48/50
    57/57 [==============================] - 110s 2s/step - loss: 0.1154 - val_loss: 0.2348
    Epoch 49/50
    57/57 [==============================] - 108s 2s/step - loss: 0.1152 - val_loss: 0.2373
    Epoch 50/50
    57/57 [==============================] - 108s 2s/step - loss: 0.1141 - val_loss: 0.2378

    Epoch 00050: ReduceLROnPlateau reducing learning rate to 0.0006249999860301614.
    Model: "attention__based__encoder__decoder_4"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    encoder_4 (Encoder)          multiple                  566148
    _________________________________________________________________
    decoder_4 (Decoder)          multiple                  782649
    =================================================================
    Total params: 1,348,797
    Trainable params: 1,348,797
    Non-trainable params: 0
    _________________________________________________________________

```python
# Fitting the model on training data
model_general.fit(train_dataloader, steps_per_epoch = TRAIN_STEPS, epochs = EPOCHS,
              callbacks = [learning_rate_cb, tensorboard_cb, stopper_cb, checkpoint_cb],
              validation_data = validation_dataloader, validation_steps = VALID_STEPS)
model_general.summary()
```

    Epoch 1/50
    57/57 [==============================] - 121s 2s/step - loss: 0.1078 - val_loss: 0.2256
    Epoch 2/50
    57/57 [==============================] - 114s 2s/step - loss: 0.1016 - val_loss: 0.2244
    Epoch 3/50
    57/57 [==============================] - 113s 2s/step - loss: 0.0984 - val_loss: 0.2227
    Epoch 4/50
    57/57 [==============================] - 114s 2s/step - loss: 0.0968 - val_loss: 0.2236
    Epoch 5/50
    57/57 [==============================] - 114s 2s/step - loss: 0.0957 - val_loss: 0.2241

    Epoch 00005: ReduceLROnPlateau reducing learning rate to 0.0003124999930150807.
    Epoch 6/50
    57/57 [==============================] - 113s 2s/step - loss: 0.0936 - val_loss: 0.2196
    Epoch 7/50
    57/57 [==============================] - 113s 2s/step - loss: 0.0917 - val_loss: 0.2214
    Epoch 8/50
    57/57 [==============================] - 112s 2s/step - loss: 0.0906 - val_loss: 0.2213

    Epoch 00008: ReduceLROnPlateau reducing learning rate to 0.00015624999650754035.
    Epoch 9/50
    57/57 [==============================] - 112s 2s/step - loss: 0.0893 - val_loss: 0.2209
    Restoring model weights from the end of the best epoch.
    Epoch 00009: early stopping
    Model: "attention__based__encoder__decoder_4"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    encoder_4 (Encoder)          multiple                  566148
    _________________________________________________________________
    decoder_4 (Decoder)          multiple                  782649
    =================================================================
    Total params: 1,348,797
    Trainable params: 1,348,797
    Non-trainable params: 0
    _________________________________________________________________

The model achieves the loss of 0.2196 on validation set which is better than the model using dot scoring function.

7.2. Calculating the BLEU Score:

We can now calculate the BLEU score to quantify the model performance.

```python
# Removing '<' and '>' tokens and postprocessing punctuations to make plain texts
def rem(s):
    if s.startswith('<'):
        s = s[1:]
    if s.endswith('>'):
        s = s[:-1]
    return s
test['informals'] = test['encoder_inp'].apply(rem)
test['formals'] = test['decoder_inp'].apply(rem)
def predictor(s):
    # Modifing predictor using general scoring function
    result = predict(s, model_general)
    return result
test['predictions'] = test['informals'].apply(predictor)
# Process inputs for Bleu score
def convert_formals(s):
    return [s.split()]
def convert_predictions(s):
    return s.split()
test['formals'] = test['formals'].apply(convert_formals)
test['predictions'] = test['predictions'].apply(convert_predictions)
bleu_scores_gen = [sentence_bleu(test['formals'].iloc[i], test['predictions'].iloc[i]) for i in range(len(test))]
print(f"Mean Bleu score of predictions: {np.mean(bleu_scores_gen)}")
```

    Mean Bleu score of predictions: 0.5096058330933692

The model achieves the BLEU score of 0.5096 on test set which is significantly better than that of baseline encoder decoder model. Let us check the distribution of the bleu scores.

```python
plt.figure(figsize = (5, 5))
ax = sns.distplot(bleu_scores_gen, hist = False, kde = True, kde_kws = {'shade': True, 'linewidth': 2}, color = "green")
plt.title("Distribution of Bleu Scores by General Model", fontdict = title_font, pad = 20.0)
plt.xlabel("Bleu Score", fontdict = label_font)
plt.ylabel("Count", fontdict = label_font)
for label in (ax.get_xticklabels() + ax.get_yticklabels()):
    label.set_fontname('Arial')
    label.set_fontsize(14)
plt.show()
```

The distribution shows that the model with general scoring function achieves the bleu score of around 0.6 for majority of the sentences but is somewhat better than the dot model. Let us generate a random prediction using this model.

```python
print("Informal Sentence: how r u")
print(f"Formal Prediction: {predict('how r u', model_general)}")
```

    Informal Sentence: how r u
    Formal Prediction: How are you?

As expected with the lower validation loss, unlike the model with dot scoring function, the model is predicting pretty accurately.

8. Training the Model using Concat Scoring Function:

8.1. Compiling and Fitting the model:

We can now train the model by using model fit method.

```python
# Defining model instance with 'concat' scoring function
model_concat  = Attention_Based_Encoder_Decoder(input_length = MAX_LEN, inp_vocab_size = vocab_size_informal,
                                             out_vocab_size = vocab_size_formal, lstm_size = UNITS,
                                             scoring_function = 'concat', batch_size = BATCH_SIZE,
                                             embedding_dim = vocab_size_formal, embedding_matrix = None)
# Compiling the model using 'adam' optimizer and custom loss function
optimizer = tf.keras.optimizers.Adam(0.01)
model_concat.compile(optimizer = optimizer, loss = loss_function)
# Creating callbacks to control model training
tensorboard_cb = create_tensorboard_cb("Model_Concat_logs")
checkpoint_cb = tf.keras.callbacks.ModelCheckpoint("Model_Concat",
                                                    save_best_only = True, save_weights_only = False)
# Fitting the model on training data
model_concat.fit(train_dataloader, steps_per_epoch = TRAIN_STEPS, epochs = EPOCHS,
              callbacks = [learning_rate_cb, tensorboard_cb, stopper_cb, checkpoint_cb],
              validation_data = validation_dataloader, validation_steps = VALID_STEPS)
model_concat.summary()
```

    Epoch 1/50
    57/57 [==============================] - 147s 2s/step - loss: 1.0973 - val_loss: 0.9655
    Epoch 2/50
    57/57 [==============================] - 126s 2s/step - loss: 0.7658 - val_loss: 0.7458
    Epoch 3/50
    57/57 [==============================] - 127s 2s/step - loss: 0.6057 - val_loss: 0.6164
    Epoch 4/50
    57/57 [==============================] - 127s 2s/step - loss: 0.5203 - val_loss: 0.5795
    Epoch 5/50
    57/57 [==============================] - 126s 2s/step - loss: 0.4742 - val_loss: 0.5523
    Epoch 6/50
    57/57 [==============================] - 125s 2s/step - loss: 0.4432 - val_loss: 0.5233
    Epoch 7/50
    57/57 [==============================] - 125s 2s/step - loss: 0.4190 - val_loss: 0.5048
    Epoch 8/50
    57/57 [==============================] - 125s 2s/step - loss: 0.3997 - val_loss: 0.4880
    Epoch 9/50
    57/57 [==============================] - 127s 2s/step - loss: 0.3838 - val_loss: 0.4823
    Epoch 10/50
    57/57 [==============================] - 127s 2s/step - loss: 0.3695 - val_loss: 0.4747
    Epoch 11/50
    57/57 [==============================] - 125s 2s/step - loss: 0.3569 - val_loss: 0.4574
    Epoch 12/50
    57/57 [==============================] - 125s 2s/step - loss: 0.3482 - val_loss: 0.4640

    Epoch 00012: ReduceLROnPlateau reducing learning rate to 0.004999999888241291.
    Epoch 13/50
    57/57 [==============================] - 127s 2s/step - loss: 0.3170 - val_loss: 0.4346
    Epoch 14/50
    57/57 [==============================] - 127s 2s/step - loss: 0.2972 - val_loss: 0.4300
    Epoch 15/50
    57/57 [==============================] - 126s 2s/step - loss: 0.2867 - val_loss: 0.4318

    Epoch 00015: ReduceLROnPlateau reducing learning rate to 0.0024999999441206455.
    Epoch 16/50
    57/57 [==============================] - 126s 2s/step - loss: 0.2663 - val_loss: 0.4175
    Epoch 17/50
    57/57 [==============================] - 126s 2s/step - loss: 0.2530 - val_loss: 0.4208

    Epoch 00017: ReduceLROnPlateau reducing learning rate to 0.0012499999720603228.
    Epoch 18/50
    57/57 [==============================] - 126s 2s/step - loss: 0.2393 - val_loss: 0.4160
    Epoch 19/50
    57/57 [==============================] - 126s 2s/step - loss: 0.2316 - val_loss: 0.4198

    Epoch 00019: ReduceLROnPlateau reducing learning rate to 0.0006249999860301614.
    Epoch 20/50
    57/57 [==============================] - 126s 2s/step - loss: 0.2238 - val_loss: 0.4179

    Epoch 00020: ReduceLROnPlateau reducing learning rate to 0.0003124999930150807.
    Restoring model weights from the end of the best epoch.
    Epoch 00020: early stopping
    Model: "attention__based__encoder__decoder_3"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    encoder_3 (Encoder)          multiple                  566148
    _________________________________________________________________
    decoder_3 (Decoder)          multiple                  823050
    =================================================================
    Total params: 1,389,198
    Trainable params: 1,389,198
    Non-trainable params: 0
    _________________________________________________________________

The model achieves the loss of 0.4160 on validation set which is not better than the model using general scoring function.

8.2. Calculating the BLEU Score:

We can now calculate the BLEU score to quantify the model performance.

```python
# Removing '<' and '>' tokens and postprocessing punctuations to make plain texts
def rem(s):
    if s.startswith('<'):
        s = s[1:]
    if s.endswith('>'):
        s = s[:-1]
    return s
test['informals'] = test['encoder_inp'].apply(rem)
test['formals'] = test['decoder_inp'].apply(rem)
def predictor(s):
    # Modifing predictor using concat scoring function
    result = predict(s, model_concat)
    return result
test['predictions'] = test['informals'].apply(predictor)
# Process inputs for Bleu score
def convert_formals(s):
    return [s.split()]
def convert_predictions(s):
    return s.split()
test['formals'] = test['formals'].apply(convert_formals)
test['predictions'] = test['predictions'].apply(convert_predictions)
bleu_scores_con = [sentence_bleu(test['formals'].iloc[i], test['predictions'].iloc[i]) for i in range(len(test))]
print(f"Mean Bleu score of predictions: {np.mean(bleu_scores_con)}")
```

    Mean Bleu score of predictions: 0.45978226278062007

The model achieves the BLEU score of 0.459  on test set which is significantly better than that of baseline encoder decoder model. Let us check the distribution of the bleu scores.

```python
plt.figure(figsize = (5, 5))
ax = sns.distplot(bleu_scores_con, hist = False, kde = True, kde_kws = {'shade': True, 'linewidth': 2}, color = "green")
plt.title("Distribution of Bleu Scores by Concat Model", fontdict = title_font, pad = 20.0)
plt.xlabel("Bleu Score", fontdict = label_font)
plt.ylabel("Count", fontdict = label_font)
for label in (ax.get_xticklabels() + ax.get_yticklabels()):
    label.set_fontname('Arial')
    label.set_fontsize(14)
plt.show()
```

The distribution shows that the model with concat scoring function achieves the bleu score of around 0.65 for majority of the sentences but is somewhat better than the general model. Let us generate a random prediction using this model.

```python
print("Informal Sentence: how r u")
print(f"Formal Prediction: {predict('wat r ya talkin abt', model_concat)}")
```

    Informal Sentence: how r u
    Formal Prediction: Have are it it it.

The model corrected the informal word 'r' to 'are'. It also corrected the capitalization. But more importantly, the prediction is not meaningful or convincing. This issue can be overcome by training the model on large dataset.

9. Error Analysis:

The model with general scoring function has performed best among the three models. Now, we will analyze the behaviour of this model on test dataset by checking the best and worst predictions made by the model. For that we will have to sort the bleu scores achieved by the model on the test set, and then print the corresponding predictions.

```python
# Sorting the indices by blue scores
scores = np.array(bleu_scores_gen)
indices = np.argsort(scores)
# Getting worst score indices
worst = indices[:5]
# Getting best score indices
best = indices[-5:][::-1]
print('Best Predictions:')
print("-"*50)
for i in best:
    print(f"Informal Input : {test['informals'].iloc[i]}")
    print(f"Expected Output : {' '.join(test['formals'].iloc[i][0])}")
    print(f"Predicted Output : {' '.join(test['predictions'].iloc[i])}")
    print(f"Bleu Score of Prediction : {scores[i]}")
    print("\n")
print('='*100)
print('Worst Predictions:')
print("-"*50)
for i in worst:
    print(f"Informal Input : {test['informals'].iloc[i]}")
    print(f"Expected Output : {' '.join(test['formals'].iloc[i][0])}")
    print(f"Predicted Output : {' '.join(test['predictions'].iloc[i])}")
    print(f"Bleu Score of Prediction : {scores[i]}")
    print("\n")
```

    Best Predictions:
    --------------------------------------------------
    Informal Input : Okay.
    Expected Output : Ok.
    Predicted Output : Ok.
    Bleu Score of Prediction : 1.0

    Informal Input : So what are you doing now? Can 1 pastime you to a walk? See? Then Iodin maybe pass in by where you bequeath later.
    Expected Output : So what are you doing now? Can I interest you to a walk? See? Then I maybe pass in by where you leave later.
    Predicted Output : No. I just want to watch that someone ask you a few dollars on Friday. So if you want to go for the party.
    Bleu Score of Prediction : 0.7071067811865476

    Informal Input : You prefer other day's onr lyou want Monday em Thursday soy you can comming directly? Or DO you have fire days? You choose, because you aire the only on studying.
    Expected Output : You prefer other days or you want Monday and Thursday so you can come directly? Or do you have free days? You choose, because you are the only one studying.
    Predicted Output : You are having a good time too. Thanks for the dinner. I just want to watch movies. I was thinking of asking you and have to do then going?
    Bleu Score of Prediction : 0.7001600475627672

    Informal Input : I got those Thank. Do you wented to chage?
    Expected Output : I got those tanks. Do you want to change?
    Predicted Output : I think it is not enough to buy something.
    Bleu Score of Prediction : 0.6865890479690392

    Informal Input : I ' ll pass you the notes nixst Monday because I gotten CCA tomorrom. Don't ' t know I possible lo hel.
    Expected Output : I'll pass you the notes next Monday because I got CCA tomorrow. Don't know if possible to meet.
    Predicted Output : I'll probably see you at 2:40. Then I message me and shop, got a place in about 1:30.
    Bleu Score of Prediction : 0.6865890479690392

    ====================================================================================================
    Worst Predictions:
    --------------------------------------------------
    Informal Input : Okeh, and so later call me.
    Expected Output : Ok, then later call me.
    Predicted Output : Ok. I want to there later.
    Bleu Score of Prediction : 0.0

    Informal Input : Okay. Then dont be late ah
    Expected Output : Okay. Then don't be late.
    Predicted Output : Ok. I am not crazy.
    Bleu Score of Prediction : 0.0

    Informal Input : How ' s they shopping?
    Expected Output : How's the shopping?
    Predicted Output : How about you and message me and sit with you then.
    Bleu Score of Prediction : 0.0

    Informal Input : OI, Zepol! Damn borring! Sigh.
    Expected Output : Hi, Zepol! Damn bored! Sigh.
    Predicted Output : Ok, then I will be going to be a bit.
    Bleu Score of Prediction : 0.0

    Informal Input : My friand says still got.
    Expected Output : My friend says still got.
    Predicted Output : Are you going to see a lot of shopping?
    Bleu Score of Prediction : 0.0

The important observation regarding the predictions is that be it best predictions or worst predictions, the model is capable of correcting the misspellings, capitalizations, and punctuations. The predictions with higher bleu score have more words overlapping with ground truth. The worst predictions however, are for the instances where there are lot of misspellings and incorrect capitalizations as the model is sensitive to it. Nevertheless, the model is trained on very little data and hence has lot of scope for improvement with large datasets like GYAFC corpus.

9. Summary:

The model with dot scoring function did not generate a satisfactory prediction. But model with general scoring function performed excepationally well in terms of meaning preservation also it achieved the lowest validation loss. Model with concat scoring function is performing well too. While data augmentation and introduction of attention model significantly improves the performance, it can be further improved by using large dataset. For now though, the model with general scoring function is suitable for our deployment purposes.

```python

```


 9. Markov chains

 Andrej Košir, Lucami, FE

 Kontakt: prof. dr. Andrej Košir, andrej.kosir@lucami.fe.uni-lj.si, skype=akosir_sid

 1

## ■ Intro

- Goals:
    - Markov chain modeling
    - Markov chain state analysis
- Stochastic process on finite states, most useful and used. We limit to:
    - Discrete times
    - Discrete state space (finite or countably many states)
- Case: user behavior
- Usage
    - Finite state machine analysis: software, TC system, ...
    - Models in economy
- What are questions:
    - Distributions of visits
    - Long term behavior
    - Absorption of states

 2

## ■ Sections

9.1. Definition, basic characteristics

■ Markov chain definition $\large{*}$

■ Transition matrix, distribution of states $\large{*}$

9.2. Classification of states

■ Communication of states and period $\large{*}$

■ Recurrent and transient state $\large{*}$

■ Ergodic Markov chain and return times

■ Stationary (limit) distribution $\large{*}$

■ Finite Markov chain $\large{*}$

9.3. Transition matrix estimation

■ Transition probability estimation

■ Confidence intervals and quantiles

■ Sample size determination

■ Numerical aspect

■ Case: telecommunication service user modeling

 3

## 9.1. Definition, basic characteristics

■ Markov chain definition

■ Transition matrix, distribution of states

 4

## ■ Markov chain definition

- State space $S$, $m=|S|$
    - Finite, countable, continuous

- Time $T$
    - Discrete, continuous

- Stochastic process:
$$ X_n : T \to S $$

- Discrete Markov chain is a discrete time discrete space Markov chain having Markov property
$$ P[X_{n+1}|_{X_n, X_{n-1}, \ldots, X_0}] = P[X_{n+1}|_{X_n}] = P[X_1|_{X_0}] $$

- Transition probability of one step
$$ p_{ij} = P[X_{n+1}=j|_{X_n=i}] $$

- Finite state space means there is a matrix  of transition probabilities
$$ P = [p_{ij}], \; i,j\in S. $$
    - Row sums are equal to 1.

 5

```python
import numpy as np
from sympy import Matrix
from sympy.functions import re

# Case 1
# Transition matrix
P = np.array([[0.7, 0.2, 0.1, 0.0],
              [0.2, 0.5, 0.2, 0.1],
              [0.05, 0.15, 0.6, 0.2],
              [0.0, 0.2, 0.0, 0.8]])
```

## ■ Transition matrix, distribution of states

- Transition matrix for $n$ steps
    - $p_{ij}^{(n)} = P[X_{k+n}|_{X_k=i}]$, $P^{(n)}=[p_{ij}^{(n)}]$
    - From total probability formula
    $$ P^{(n)} = P^n $$

- Distribution of states:
$$ \pi_i^n = P[X_n=i], \pi^n = [\pi_1^n, \ldots, \pi_i^n] $$
    - Initial distribution: $\pi^0$. If initial state $X_0=i_0$ is known
    $$ \pi^0=\delta_{i_0 i} $$
    - From total probability formula:
    $$ \pi^n = P^{(n)} \pi^0 $$
    - More on limit distribution later

 6

## 9.2. Classification of states

■ Communication of states and period

■ Recurrent and transient state

■ Ergodic Markov chain and return times

■ Stationary (limit) distribution

■ Finite Markov chain

 7

## ■ Communication of states and period

- Accessible state
    - State $j$ is accessible from state $i$ if $\exists n: p_{ij}^{(n)}>0$. Notation $i\leftrightarrow j$
    - Two states communicate if $i\rightarrow$ and $i \leftarrow j$, denoted by $i\leftrightarrow j$
    - „Communicate“ is equivalent relation

- Markov chain is irreducible if there are only one single communicating class

- Period of state $i$ ia denoted by $d(i)$ is greatest common divisor of all $n$ such that $P_{ii}^{(n)} > 0$:
    - If $i\leftrightarrow j$ then $d(i) = d(j)$;
    - State is aperiodic if $d(i)=1$;

- State $i$ is zero state if
$$ \lim_{n\to\infty} P_{ii}^{(n)} = 0. $$

 8

```python
# ---------------------------------------------------------------------------------------------------
# Classification of states
print ('Transition matrix P=\n', P)
print ('\nTransition matrix P^2=\n', np.linalg.matrix_power(P, 2))

# All entries of P^2 are non-zero
# => All atates are connected
```

    Transition matrix P=
     [[0.7  0.2  0.1  0.  ]
     [0.2  0.5  0.2  0.1 ]
     [0.05 0.15 0.6  0.2 ]
     [0.   0.2  0.   0.8 ]]

    Transition matrix P^2=
     [[0.535 0.255 0.17  0.04 ]
     [0.25  0.34  0.24  0.17 ]
     [0.095 0.215 0.395 0.295]
     [0.04  0.26  0.04  0.66 ]]

```python
# ---------------------------------------------------------------------------------------------------
# Decompose transition matrix
# Use Jordan decomposition
#mP = Matrix(P)
#mB, mJ = mP.jordan_form()
#B, J = np.array(mB), np.array(mJ)
B = np.array([[-0.5,-0.776313,0.659526,-0.413722],
              [-0.5,-0.212847,-0.11514,0.830469],
              [-0.5,0.283275,-0.73692,-0.123153],
              [-0.5,0.521335,0.0933625,-0.352121]])
J = np.array([[1.,0,0,0],
              [0,0.718345,0,0],
              [0,0,0.553349,0],
              [0,0,0,0.328305]])

# Test decomposition
err_mat = P - np.dot(B, np.dot(J, np.linalg.inv(B)))
print ('\nDecomposition error norm = ', np.linalg.norm(err_mat))
```

    Decomposition error norm =  7.249920892047682e-07

## ■ Recurrent and transient state

- Probability of return after exactly $n$ steps
$$ f_{ij}^n = P[X_n=j|_{X_{n-1}\ne j, \ldots, X_1\ne j, X_0=i}] $$

- Random variable of transitions from $i$ to $j$ is denoted by $T_{i}$ and its distribtion is
$$ T_{ij} \sim \left(\matrix{0 & 1 & 2 & \ldots \cr f_{ij}^0 & f_{ij}^1 & f_{ij}^2 & \ldots}\right) $$
    - Denote $T_i = T_{ii}$
    - Denote $\sum_{n=0}^\infty f_{ii}^n = f_{ii}^*$. Value
    $$ 1 - f_{ii}^* $$
    is a probability that Markov Chain newer returns to state 𝑖 after leaving it.

- State $i$ is **recurrent** if $f_{ii}^* = 1$. State is **transient** if it is not recurrent.
    - Recurrence and transience are communicating class properties

- We have: state $i$ is recurrent if and only if
$$ \sum_{n=1}^\infty P_{ii}^{(n)} = \infty $$

- Also: if $i\leftrightarrow j$ then both states are either transient or recurrent

 9

```python
# Recurent, transient?

# Eigenvalues
eigVals, eigVecs = np.linalg.eig(P.T)

# Get it sorted
idx = eigVals.argsort()[::-1]
eigVa = eigVals[idx]
eigVc = eigVecs[:,idx]

las_bf =  (1./(1-eigVa[1:4]))-1
print ('las_bf', las_bf)

sumJn = np.diag(np.insert(las_bf, 0, np.inf))
print ('\nSum J^n\n', sumJn)

sumPn = np.dot(B, np.dot(sumJn, np.linalg.inv(B)))
print ('\nSum P^n\n', sumPn)

# Since all diagonal elements of sum P^n are infinity
#  => all states are recurrent
```

    las_bf [2.55044912 1.23888596 0.48877142]

    Sum J^n
    [[       inf 0.         0.         0.        ]
     [0.         2.55044912 0.         0.        ]
     [0.         0.         1.23888596 0.        ]
     [0.         0.         0.         0.48877142]]

    Sum P^n
    [[inf inf inf inf]
     [inf inf inf inf]
     [inf inf inf inf]
     [inf inf inf inf]]

```python
def inf_sum(a_lst):
    return [(la/(1.0-la) if la < 1 else np.infty) for la in a_lst]

# ---------------------------------------------------------------------------------------------------
# Are states recurrent?
# Compute sum of P^n
sumJ = np.diag(inf_sum(np.diag(J)))
sumP = np.dot(B, np.dot(sumJ, np.linalg.inv(B)))

print ('\nSum of P^n = \n', sumP)
# => All states are reccurent
```

    Sum of P^n =
    [[inf inf inf inf]
     [inf inf inf inf]
     [inf inf inf inf]
     [inf inf inf inf]]

## ■ Ergodic Markov chain and return times

- Markov chain is **Ergodic** if it is
    - Irreducible
    - Recurrent (all states are recurrent)
    - Aperiodic (all states are aperiodic)
    - Nonzero (all states are positive)

- Expected return time:
$$ E(T_i) = \sum_{n=0}^\infty n f_{ii}^{(n)} $$

- In ergodic Markov chain
$$ \lim_{n\to\infty} P_{ii}^{(n)} = \frac{1}{E(T_i)} $$
$$ \lim_{n\to\infty} P_{ij}^{(n)} = \lim_{n\to\infty} P_{ii}^{(n)} $$

- Recurrent **zero** state $i$:
$$ \lim_{n\to\infty} P_{ii}^{(n)} = 0 \; \Leftrightarrow \;  E(T_i) = \infty  $$

- Recurrent **positive** = **ergodic** state $i$:
$$ \lim_{n\to\infty} P_{ii}^{(n)} > 0 \; \Leftrightarrow \; E(T_i) < \infty  $$

 10

```python
def inf_lim(a_lst):
    return [(0 if la < 1 else 1) for la in a_lst]

# ---------------------------------------------------------------------------------------------------
# Zero or positive states?
# Compute lim of P^n
limJ = np.diag(inf_lim(np.diag(J)))
limP = np.dot(B, np.dot(limJ, np.linalg.inv(B)))

print ('\nlim of P^n = \n', limP)

# Since all lim P^n are positive
# => all states are positive

# Expected times of visits
ETi = 1.0/np.diag(limP)

print ('\nExpected number of visits py states:\n', ETi)
```

    lim of P^n =
    [[0.21301781 0.27218919 0.18934915 0.32544385]
     [0.21301781 0.27218919 0.18934915 0.32544385]
     [0.21301781 0.27218919 0.18934915 0.32544385]
     [0.21301781 0.27218919 0.18934915 0.32544385]]

    Expected number of visits py states:
    [4.69444311 3.67391519 5.28124903 3.07272668]

## ■ Stationary (limit) distribution

- State distribution after $n$ steps
$$ \pi_i^n = P[X_n=i], \qquad \pi^n = [\pi_1^n, \ldots, \pi_i^n] $$

- Stationary (limit) distribution:
  $$ \pi = \lim_{n\to\infty} \pi^n $$
    - When uniquely defined: when all states are recurrent positive

- Ergodic MC: stationary distribution exists
- Stationary distribution of ergodic MC chain is left eigenvector of transition matrix:
  $$ \pi^\top = \pi^\top P. $$

- Note: in an ergodic MC a stationary distribution independent of initial distribution $\pi^0$.

 11

```python
# ---------------------------------------------------------------------------------------------------
# Limit distribuion
eigVals, eigVecs = np.linalg.eig(P.T)
pi = eigVecs[:, 0]/sum(eigVecs[:, 0])

print ('\neigVals=\n', eigVals)
#print '\neigVecs=\n', eigVecs
print ('\nLimit distribution =', pi)
```

    eigVals=
    [1.         0.32830522 0.71834549 0.55334929]

    Limit distribution = [0.21301775 0.27218935 0.18934911 0.32544379]

## ■ Probability of absorption

- Absorption: chain never leave a subset of states $S_0 \subset S$

- What is the probability of absorption?

- If a chain starts in transient state, then either:
    1. It leaves a set of transient states with probability of 1
    2. It enters a set of recurrent states with probability of 1

- Absorbing states $i$: the only state of an absorbing state
We have: $i$ absorbing if and only if
$P_{ii}=1$

- Let $i\in S$ be a transient state, $S_e$ a set of recurrent states:

Probability of absorption: $\pi_i(S_e)$ is the probability MC with $X_0=i$ gets absorbed into the set of states $S_e$

- We have
$$ \lim_{n\to\infty} P_{ij}^{(n)} = \pi_i(S_e)\pi_j. $$

 12

## ■ Finite Markov chain

- Finite number of states $|S|=m < \infty$.

- Each finite MC has recurrent states!

- All recurrent states are positive, that is ergodic and has finite expected return time
$$ \lim_{n\to\infty} P_{ii}^{(n)} > 0. $$

- Classification of states: all states are $S=S_m\cup S_e$, where
    - Transient states: $S_m$
    - Ergodic states: $S_e$

- Transient matrix is of a block form:
    - Ergodic states: $1, \ldots, m-K$
    - Transient state: $m-K+1, \ldots, m$
    $$ P = \left[\matrix{S & 0 \cr R & Q}\right] $$

- Fundamental matrix:
$$ N = (I - Q)^{-1}.  $$

 13

```python
# Case 2
c2P = np.array(
    [[0.7, 0.3, 0.0, 0.0],
     [0.2, 0.8, 0.0, 0.0],
     [0.05, 0.15, 0.6, 0.2],
     [0.0, 0.2, 0.0, 0.8]])

print ('Transition matrix P=\n', c2P)

```

    Transition matrix P=
     [[0.7  0.3  0.   0.  ]
     [0.2  0.8  0.   0.  ]
     [0.05 0.15 0.6  0.2 ]
     [0.   0.2  0.   0.8 ]]

```python
# ---------------------------------------------------------------------------------------------------
# Classification of states
print ('Transition matrix P=\n', c2P)
print ('\nTransition matrix P^2=\n', np.linalg.matrix_power(c2P, 2))
print ('\nTransition matrix P^3=\n', np.linalg.matrix_power(c2P, 3))

# Matrix multiplications by blocks
# => 2 x 2 block of zeros stayes the same
# => There are two connected classes of states
```

    Transition matrix P=
     [[0.7  0.3  0.   0.  ]
     [0.2  0.8  0.   0.  ]
     [0.05 0.15 0.6  0.2 ]
     [0.   0.2  0.   0.8 ]]

    Transition matrix P^2=
     [[0.55  0.45  0.    0.   ]
     [0.3   0.7   0.    0.   ]
     [0.095 0.265 0.36  0.28 ]
     [0.04  0.32  0.    0.64 ]]

    Transition matrix P^3=
     [[0.475  0.525  0.     0.    ]
     [0.35   0.65   0.     0.    ]
     [0.1375 0.3505 0.216  0.296 ]
     [0.092  0.396  0.     0.512 ]]

```python
# ---------------------------------------------------------------------------------------------------
# Limit distribuion
eigVals, eigVecs = np.linalg.eig(c2P.T)

# Get it sorted
idx = eigVals.argsort()[::-1]
eigenValues = eigVals[idx]
eigenVectors = eigVecs[:,idx]

# Normalize eigenvector
pi = eigenVectors[:, 0]/sum(eigenVectors[:, 0])

print ('\neigVals=\n', eigenValues)
#print '\neigVecs=\n', eigenVectors
print ('\nLimit distribution =', pi)
```

    eigVals=
     [1.  0.8 0.6 0.5]

    Limit distribution = [ 0.4  0.6 -0.  -0. ]

```python
# ---------------------------------------------------------------------------------------------------
# Decompose transition matrix
# Use Jordan decomposition
#mP = Matrix(c2P)
#mB, mJ = mP.jordan_form()
#B, J = np.array(mB), np.array(mJ)
c2B = np.array([[1.0, 0, 0, 9.0/4],
              [1.0, 0, 0, -3.0/2],
              [1.0, 1.0, 1.0, -7.0/8],
              [1.0, 1.0, 0, 1.0]])
c2J = np.array([[1.0,0,0,0],
              [0,4.0/5,0,0],
              [0,0,3.0/5,0],
              [0,0,0,1.0/2]])

# Test decomposition
err_mat = c2P - np.dot(c2B, np.dot(c2J, np.linalg.inv(c2B)))
print ('\nDecomposition error norm = ', np.linalg.norm(err_mat))
```

    Decomposition error norm =  1.9675159943996247e-16

```python
def prd(a,b):
    if np.isinf(a) and b == 0:
        return 0
    if np.isinf(b) and a == 0:
        return 0
    return a*b

def myDot(A,B):
    nA1, nA2, nB2 = A.shape[0], A.shape[1], B.shape[1]
    AB = np.zeros((nA1,nB2))
    for i in range(nA1):
        for j in range(nB2):
            curr = 0
            for k in range(nA2):
                curr += prd(A[i,k], B[k,j])
            AB[i, j] = curr
    return AB

# Classification of states
# Recurent, transient?

# Eigenvalues
eigVals, eigVecs = np.linalg.eig(c2P.T)

# Get it sorted
idx = eigVals.argsort()[::-1]
eigVa = eigVals[idx]
eigVc = eigVecs[:,idx]

las_bf =  (1./(1-eigVa[1:4]))-1
sumJn = np.diag(np.insert(las_bf, 0, np.inf))

sumPn = myDot(c2B, myDot(sumJn, np.linalg.inv(c2B)))
print ('\nSum P^n\n', sumPn)

# Since diagonal elements of sum P^n i=1 and i=2 are infinity
#  => states i=1 and i=2 are recurrent
# Since diagonal elements of sum P^n i=3 and i=4 are finite
#  => states i=3 and i=4 are transient

```

    Sum P^n
     [[inf inf 0.  0. ]
     [inf inf 0.  0. ]
     [inf inf 1.5 2.5]
     [inf inf 0.  4. ]]

```python
def inf_lim(a_lst):
    return [(0 if la < 1 else 1) for la in a_lst]

# ---------------------------------------------------------------------------------------------------
# Zero or positive states?
# Compute lim of P^n
limJ = np.diag(inf_lim(np.diag(c2J)))
limP = np.dot(c2B, np.dot(limJ, np.linalg.inv(c2B)))

print ('\nlim of P^n = \n', limP)

# Since all lim P^n are positive
# => all states are positive

# Expected times of visits
ETi = 1.0/np.diag(limP)

print ('\nExpected number of visits py states:\n', ETi)

```

    lim of P^n =
     [[0.4 0.6 0.  0. ]
     [0.4 0.6 0.  0. ]
     [0.4 0.6 0.  0. ]
     [0.4 0.6 0.  0. ]]

    Expected number of visits py states:
     [2.5        1.66666667        inf        inf]

    /home/nbuser/anaconda3_501/lib/python3.6/site-packages/ipykernel/__main__.py:16: RuntimeWarning: divide by zero encountered in true_divide

```python
# ---------------------------------------------------------------------------------------------------
# Limit distribuion
eigVals, eigVecs = np.linalg.eig(c2P.T)

# Get it sorted
idx = eigVals.argsort()[::-1]
eigenValues = eigVals[idx]
eigenVectors = eigVecs[:,idx]

# Normalize eigenvector
pi = eigenVectors[:, 0]/sum(eigenVectors[:, 0])

print ('\neigVals=\n', eigenValues)
#print '\neigVecs=\n', eigenVectors
print ('\nLimit distribution =', pi)
```

    eigVals=
    [1.  0.8 0.6 0.5]

    Limit distribution = [ 0.4  0.6 -0.  -0. ]

## ■ Number of visits of a transient state

- Random variable $n_i$: number of visits of a state $i$ in an arbitrary long time

- Expected number of visits of a state $j$ if initial state is $𝑖$:
 $$ E_i[n_j] < \infty $$

- We have: if state $i$ is transient:
$$ E_i[n_j] = \sum_{k=1}^\infty P_{ij}^{(k)} $$

- Also - fundamental matrix gives expected values:
$$ \left[E_i[n_j]\right] = N $$

 14

```python
# Expected number of visits of recurent states
m = 4 # |S|
K = 2 # number of recurent states

Q = c2P[K:m, K:m]
print ('Part of transition matrix Q = \n', Q)

N = np.linalg.inv(np.eye(m-K)-Q)
print ('\nFundamental matrix N =\n', N)
```

    Part of transition matrix Q =
    [[0.6 0.2]
     [0.  0.8]]

    Fundamental matrix N =
    [[2.5 2.5]
     [0.  5. ]]

## 9.3 Transition matrix estimation

■ Transition probability estimation

■ Confidence intervals and quantiles

■ Sample size determination

■ Numerical aspect

■ Case: telecommunication service user modeling

 15

## ■ Transition probability estimation

- Transition probability
$$ p_{ij} = \frac{n_k}{n} $$
is a statistical estimation;

- Parameter estimation
    - estimate and termine
        - expected value $\overline{x}$
        - standard deviation $\sigma$
        - for a selected risk level $\alpha$ compute $z_\alpha$ (if normal distribution $z_{0.05}=1.96$)
    - Confidence interval:
$$ CI = [\overline{x} - z_\alpha\frac{\sigma}{\sqrt{n}}, \overline{x} + z_\alpha\frac{\sigma}{\sqrt{n}}] $$
    - Sample size: determined from a preset confidence interval length
    $$ |CI| = 2 z_\alpha\frac{\sigma}{\sqrt{n}} $$

 16

## ■ Confidence intervals and quantiles

- Confidence interval visualization

 17

## ■ Transition matrix estimation sample size determination

- From confidence interval size

- Confidence interval for risk level $\alpha=0.05$ ($z_\alpha=1.96$)
$$ CI = [p_0 - z_\alpha\sqrt{p_0(1-p_0)/n}, p_0 + z_\alpha\sqrt{p_0(1-p_0)/n}] $$

For $p=0.65$ and $|CI|=\Delta p$ at $\alpha=0.05$
$$ n\geq \frac{4 z_\alpha^2}{\Delta p^2} p_0 (1-p_0) $$
we get
 - for $\Delta p=0.01:$ $n\geq 34959$,
 - for $\Delta p=0.02:$ $n\geq 175$,
 - for $\Delta p=0.04:$ $n\geq 88$.

 18

## ■ Numerical aspect of transition matrix computation

- We need matrix power
$$ P^n = P^{(n)} $$

- Jordan decomposition
$$ P = B J B^{-1},  $$
where $J$ is Jordan canonical form.

- For each class of states we have
$$ J = \lambda I + R $$
and
$$ P^n = B J^n B^{-1} $$

 19

## ■ Case: telecommunication service user modeling

- A set of telecommunication users $U$

- We split them into classes: $U=U_1 \cup U_2 \cup U_3 \cup U_4$
    - Very satisfied: $U_1$
    - Satisfied: $U_2$
    - Not satisfied: $U_3$
    - Churners: $U_4$

- Initial distribution: estimation of users

- Results:
    - Recurrent - transients states
    - Stationary distribution

 20

## ■ Conclusion

- Most useful: finite Markov chain
- Check the transition matrix assumption: stationarity

 21