title: keras understanding Word Embedding Layer
body: <p>From the <a href="https://machinelearningmastery.com/use-word-embedding-layers-deep-learning-keras/" rel="nofollow noreferrer">page</a> I got the below code:</p>
  
  <pre><code>from numpy import array
  from keras.preprocessing.text import one_hot
  from keras.preprocessing.sequence import pad_sequences
  from keras.models import Sequential
  from keras.layers import Dense
  from keras.layers import Flatten
  from keras.layers.embeddings import Embedding
  # define documents
  docs = ['Well done!',
          'Good work',
          'Great effort',
          'nice work',
          'Excellent!',
          'Weak',
          'Poor effort!',
          'not good',
          'poor work',
          'Could have done better.']
  # define class labels
  labels = array([1,1,1,1,1,0,0,0,0,0])
  # integer encode the documents
  vocab_size = 50
  encoded_docs = [one_hot(d, vocab_size) for d in docs]
  print(encoded_docs)
  # pad documents to a max length of 4 words
  max_length = 4
  padded_docs = pad_sequences(encoded_docs, maxlen=max_length, padding='post')
  print(padded_docs)
  # define the model
  model = Sequential()
  model.add(Embedding(vocab_size, 8, input_length=max_length))
  model.add(Flatten())
  model.add(Dense(1, activation='sigmoid'))
  # compile the model
  model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['acc'])
  # summarize the model
  print(model.summary())
  # fit the model
  model.fit(padded_docs, labels, epochs=50, verbose=0)
  # evaluate the model
  loss, accuracy = model.evaluate(padded_docs, labels, verbose=0)
  print('Accuracy: %f' % (accuracy*100))
  </code></pre>
  
  <ol>
  <li>I looked at <code>encoded_docs</code> and noticed that words <code>done</code> and <code>work</code> both have  one_hot encoding of 2, why? Is it because <code>unicity of word to index mapping non-guaranteed.</code> as per this <a href="https://keras.io/preprocessing/text/#one_hot" rel="nofollow noreferrer">page</a>?</li>
  <li>I got <code>embeddings</code> by command <code>embeddings = model.layers[0].get_weights()[0]</code>. in such case why do we get <code>embedding</code> object of size 50? Even though two words have same one_hot number, do they have different embedding?</li>
  <li>how could i understand which embedding is for which word i.e. <code>done</code> vs <code>work</code></li>
  <li><p>I also found below code at the <a href="https://stackoverflow.com/questions/51235118/how-to-get-word-vectors-from-keras-embedding-layer">page</a> that could help with finding embedding of each word. But i dont know how to create <code>word_to_index</code></p>
  
  <p><code>word_to_index</code> is a mapping (i.e. dict) from words to their index, e.g. <code>love</code>: 69
  words_embeddings = {w:embeddings[idx] for w, idx in word_to_index.items()}</p></li>
  <li><p>Please ensure that my understanding of <code>para #</code> is correct.</p></li>
  </ol>
  
  <p>The first layer has 400 parameters because total word count is 50 and embedding have 8 dimensions so 50*8=400. </p>
  
  <p>The last layer has 33 parameters because each sentence has 4 words max. So 4*8 due to dimensions of embedding and 1 for bias. 33 total</p>
  
  <pre><code>_________________________________________________________________
  Layer (type)                 Output Shape              Param#   
  =================================================================
  embedding_3 (Embedding)      (None, 4, 8)              400       
  _________________________________________________________________
  flatten_3 (Flatten)          (None, 32)                0         
  _________________________________________________________________
  dense_3 (Dense)              (None, 1)                 33        
  =================================================================
  </code></pre>
  
  <ol start="6">
  <li>Finally, if 1 above is correct, is there a better way to get embedding layer <code>model.add(Embedding(vocab_size, 8, input_length=max_length))
  </code> without doing one hot coding <code>encoded_docs = [one_hot(d, vocab_size) for d in docs]
  </code></li>
  </ol>
  
  <p>+++++++++++++++++++++++++++++++
  update - providing the updated code</p>
  
  <pre><code>from numpy import array
  from keras.preprocessing.text import one_hot
  from keras.preprocessing.sequence import pad_sequences
  from keras.models import Sequential
  from keras.layers import Dense
  from keras.layers import Flatten
  from keras.layers.embeddings import Embedding
  # define documents
  docs = ['Well done!',
          'Good work',
          'Great effort',
          'nice work',
          'Excellent!',
          'Weak',
          'Poor effort!',
          'not good',
          'poor work',
          'Could have done better.']
  # define class labels
  labels = array([1,1,1,1,1,0,0,0,0,0])
  
  
  from keras.preprocessing.text import Tokenizer
  
  tokenizer = Tokenizer()
  
  #this creates the dictionary
  #IMPORTANT: MUST HAVE ALL DATA - including Test data
  #IMPORTANT2: This method should be called only once!!!
  tokenizer.fit_on_texts(docs)
  
  #this transforms the texts in to sequences of indices
  encoded_docs2 = tokenizer.texts_to_sequences(docs)
  
  encoded_docs2
  
  max_length = 4
  padded_docs2 = pad_sequences(encoded_docs2, maxlen=max_length, padding='post')
  max_index = array(padded_docs2).reshape((-1,)).max()
  
  
  
  # define the model
  model = Sequential()
  model.add(Embedding(max_index+1, 8, input_length=max_length))# you cannot use just max_index 
  model.add(Flatten())
  model.add(Dense(1, activation='sigmoid'))
  # compile the model
  model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['acc'])
  # summarize the model
  print(model.summary())
  # fit the model
  model.fit(padded_docs2, labels, epochs=50, verbose=0)
  # evaluate the model
  loss, accuracy = model.evaluate(padded_docs2, labels, verbose=0)
  print('Accuracy: %f' % (accuracy*100))
  
  embeddings = model.layers[0].get_weights()[0]
  
  embeding_for_word_7 = embeddings[14]
  index = tokenizer.texts_to_sequences([['well']])[0][0]
  tokenizer.document_count
  tokenizer.word_index
  </code></pre>