title: BERT details + how do you use BERT's sequence output with a Keras biLSTM model?
body: <p>I am experimenting with a biLSTM model and 2 different embedding techniques (FastText, BERT) applied at 2 different levels (word, sentence) all for a binary text classification task. I'm new to the BERT ecosystem and the nuances of complex deep learning in general and wanted some advice.</p>
  <p>My biLSTM model is in Keras:</p>
  <ol>
  <li>How exactly do I use BERT's sequence output here? (Similar to passing weights via an embedding matrix in an embedding layer.)</li>
  <li>Is it recommend to use a biLSTM model on top of the sequence output(or BERT) in the first place? Examples that I have seen show an average pooling and couple of dense layers. Is this what is meant by &quot;fine tuning&quot;?</li>
  <li>If the biLSTM model is necessary, would it then make sense to work <em>only</em> with BERT's 768 dimension embeddings or is the regular sequence output possible?</li>
  <li>When should I set trainable as False when using BERT?</li>
  </ol>
  <p>Will appreciate your guidance. Happy to share more details if needed. Thank you.</p>