title: How exactly should the input file be formatted for the language model finetuning (BERT through Huggingface Transformers)?
body: <p>I wanted to employ the <code>examples/run_lm_finetuning.py</code> from the <a href="https://github.com/huggingface/transformers" rel="nofollow noreferrer">Huggingface Transformers repository</a> on a pretrained Bert model. However, from following the documentation it is not evident how a corpus file should be structured (apart from referencing the Wiki-2 dataset). I've tried</p>
  
  <ul>
  <li>One document per line (multiple sentences)</li>
  <li>One sentence per line. Documents are separated by a blank line (this I found in some older pytorch-transformers documentation)</li>
  </ul>
  
  <p>By looking at the code of <code>examples/run_lm_finetuning.py</code> it is not directly evident how sequence pairs for the Next Sentence Prediction objective are formed. Would the <code>--line-by-line</code> option help here? I'd be grateful, if someone could give me some hints how a text corpus file should look like.</p>
  
  <p>Many thanks and cheers,</p>
  
  <p>nminds</p>