<p>The script you are talking about is the right one for continuing pre-training. The original BERT uses next-sentence prediction as an auxiliary objective. When it is provided a pair of sentences (separated by the <code>[SEP]</code> token), the embedding of the <code>[CLS]</code> (the very first one) token is used as an input to a classifier telling if the sentences are adjacent in a coherent text or not.</p>
<p>This what the empty lines are for: on a document boundary, the sentences cannot adjacent.</p>
<p>However, the contribution of the next-sentence objective is arguable. For instance, the <a href="https://arxiv.org/abs/1907.11692" rel="nofollow noreferrer">RoBERTa</a> considers it superfluous and only uses the masked-language-modeling objective a still gets better representation quality than the original BERT.</p>
Input format for BERT fine-tuning on corpus
<p>I want to fine-tune BERT on a specific language domain using the following git repo:</p>
<p><a href="https://github.com/cedrickchee/pytorch-pretrained-BERT/blob/master/examples/lm_finetuning/README.md" rel="nofollow noreferrer">https://github.com/cedrickchee/pytorch-pretrained-BERT/blob/master/examples/lm_finetuning/README.md</a></p>
<p>Regarding the input format, it says:</p>
<blockquote>
<p>The scripts in this folder expect a single file as input, consisting
of untokenized text, with one sentence per line, and one blank line
between documents. The reason for the sentence splitting is that part
of BERT's training involves a next sentence objective in which the
model must predict whether two sequences of text are contiguous text
from the same document or not, and to avoid making the task too easy,
the split point between the sequences is always at the end of a
sentence. The linebreaks in the file are therefore necessary to mark
the points where the text can be split.</p>
</blockquote>
<p>What do they mean with documents in this regard? From my understanding, the <code>.txt</code> file used for fine-tuning just contains a lot of domain specific text with one sentence per line. Just to be sure, is it the correct approach to use this repository if I want to fine tune BERT on a specific language domain?</p>
<p>Thank you for your help!</p>