title: Fine tuning a pretrained language model with Simple Transformers
body: <p>In his article 'Language Model Fine-Tuning For Pre-Trained Transformers' Thilina Rajapakse (<a href="https://medium.com/skilai/language-model-fine-tuning-for-pre-trained-transformers-b7262774a7ee" rel="nofollow noreferrer">https://medium.com/skilai/language-model-fine-tuning-for-pre-trained-transformers-b7262774a7ee</a>)
  provides the following code snippet for fine-tuning a pre-trained model using the library <code>simpletransformers</code>:</p>
  
  <pre><code>from simpletransformers.language_modeling import LanguageModelingModel
  import logging
  
  
  logging.basicConfig(level=logging.INFO)
  transformers_logger = logging.getLogger("transformers")
  transformers_logger.setLevel(logging.WARNING)
  
  train_args = {
      "reprocess_input_data": True,
      "overwrite_output_dir": True,
  }
  
  model = LanguageModelingModel('bert', 'bert-base-cased', args=train_args)
  
  model.train_model("data/train.txt", eval_file="data/text.txt")
  
  model.eval_model("data/test.txt")
  </code></pre>
  
  <p>He then adds:</p>
  
  <blockquote>
    <p>We assume that you have combined all the text in your dataset into two
    text files train.txt and test.txt which can be found in the data/
    directory.</p>
  </blockquote>
  
  <p>I have 2 questions:</p>
  
  <p><strong>Question 1</strong></p>
  
  <p>Does the highlighted sentence above implies that the entire corpus will be merged into one text file?  So assuming that the Training Corpus is comprised of 1,000,000 text files, are we supposed to merge them all in one text file with code like this?</p>
  
  <pre><code>import fileinput
  with open(outfilename, 'w') as fout, fileinput.input(filenames) as fin:
      for line in fin:
          fout.write(line)
  </code></pre>
  
  <p><strong>Question 2</strong></p>
  
  <p>I presume that I can use the pretrained model: <code>bert-base-multilingual-cased</code>.  Correct?</p>