title: Keras model with fasttext word embedding
body: <p>I am trying to learn a language model to predict the last word of a sentence given all the previous words using keras. I would like to embed my inputs using a learned fasttext embedding model.</p>
  <p>I managed to preprocess my text data and embed the using fasttext. My training data is comprised of sentences of 40 tokens each. I created 2 np arrays, X and y as inputs, with y what I want to predict.</p>
  <p>X is of shape (44317, 39, 300) with 44317 the number of example sentences, 39 the number of tokens in each sentence, and 300 the dimension of the word embedding.</p>
  <p>y is of shape (44317, 300) is for each example the embedding of the last token of the sentence.</p>
  <p>My code for the keras model goes as follow (inspired by <a href="https://machinelearningmastery.com/how-to-develop-a-word-level-neural-language-model-in-keras/?fbclid=IwAR3NyptI5uPN_8uOP69QWKRfpTfaqG-Y1XzUB2ciN0aTr-vQDxUhnfY4Ets" rel="nofollow noreferrer">this</a>)</p>
  <pre><code>#importing all the needed tensorflow.keras components
  model = Sequential()  
  model.add(InputLayer((None, 300)))
  model.add(LSTM(100, return_sequences=True))
  model.add(LSTM(100))
  model.add(Dense(100, activation='relu'))
  model.add(Dense(300, activation='softmax'))
  model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
  model.fit(X, y, batch_size=128, epochs=20)
  model.save('model.h5')
  </code></pre>
  <p>However, the accuracy I get while training on this model is extremely low (around 1.5%). I think there is some component of the keras model that I misundertood, as if I don't embed my inputs and add an extra embedding layer instead of the InputLayer I get an accuracy of about 60 percents.</p>
  <p>My main doubt is the value of &quot;300&quot; on my second Dense layer, as I read that this should correspond the vocabulary size of my word embedding model (which is 48000), however if I put anything else than 300 I get a dimension error. So I understand that I'm doing something wrong, but I can't find how to fix it.</p>
  <p>PS :
  I have also tried <code>y = to_categorical(y, num_classes=vocab_size)</code> with vocab_size the vocabulary size of my word embedding, and by changing 300 by this same value in the second Dense, however then it tries to create an array of shape(13295100, 48120) instead of what I expect : (44317, 48120).</p>