<jupyter_start><jupyter_text>Tokenizer practiseColab Notebook containing code for basic NLP practise using Keras.* Topics covered:  - word index  - sequences  - out-of-vocabs  - padding* Class used: `Tokenizer`* Functions / methods used  - `fit_on_texts`  - `texts_to_sequences`  - `pad_sequences`  Import `tokenizer` from `keras`<jupyter_code>import tensorflow as tf
from tensorflow.keras.preprocessing.text import Tokenizer<jupyter_output><empty_output><jupyter_text>Generate word index<jupyter_code># consider the following list of sentences
sentences = [
    "winter is coming",
    "winter is here"
]

# create an instance of the Tokenizer class
tkz = Tokenizer(num_words = 100, oov_token="<missing>")<jupyter_output><empty_output><jupyter_text>Parameters used in the above object creation:- `num_words`: The maximum number of words the tokenizer would look for in the sentences- `oov_token`: replace unrecognized  ("out-of-vocabulary") words with this token<jupyter_code>tkz.fit_on_texts(sentences)  # creates a list with words and their indices
ind = tkz.word_index  # returns a dictionary with words and their indices
print(ind)<jupyter_output>{'<missing>': 1, 'winter': 2, 'is': 3, 'coming': 4, 'here': 5}<jupyter_text>Generate sequences<jupyter_code>seq = tkz.texts_to_sequences(sentences) # returns a list with each word replaced by its index from the word_index
seq<jupyter_output><empty_output><jupyter_text>Missing textHere is where the `oov_token`parameter will come into play. The sequence will replace any out-of-vocab word with the value of the above parameter.If the `oov_token` parameter was not provided above, the unrecognized words would have been ignored.<jupyter_code>test_text = [
    "when winter comes and",
    "here lies my grave",
    "the lone wolf dies but the pack survives"
]
seq = tkz.texts_to_sequences(test_text)
seq
tkz.fit_on_texts(test_text)
ind = tkz.word_index
print(ind)<jupyter_output>{'<missing>': 1, 'winter': 2, 'is': 3, 'here': 4, 'the': 5, 'coming': 6, 'when': 7, 'comes': 8, 'and': 9, 'lies': 10, 'my': 11, 'grave': 12, 'lone': 13, 'wolf': 14, 'dies': 15, 'but': 16, 'pack': 17, 'survives': 18}<jupyter_text>PaddingHere, extra info is added to the sequence sublists so that their lengths are uniform.<jupyter_code>from tensorflow.keras.preprocessing.sequence import pad_sequences
pads = pad_sequences(seq) 

pads<jupyter_output><empty_output><jupyter_text>Additional parameters that can be used:- use `padding="post"` for padding at the end of the matrix; - use `maxlen=5` for a maximum of 5 words; you will lose from the beginning- use `truncating="post"` along with maxlen to lose extra info from the end<jupyter_code><jupyter_output><empty_output>