<p>I have not heard of the pipeline you just mentioned. In order to construct an LM for your use-case, you have basically two options:</p>
<ol>
<li><p>Further training BERT (-base/-large) model on your own corpus. This process is called <em>domain-adaption</em> as also described in this <a href="https://arxiv.org/pdf/2004.10964.pdf" rel="nofollow noreferrer">recent paper</a>. This will adapt the learned parameters of BERT model to your specific domain (Bio/Medical text). Nonetheless, for this setting, you will need quite a large corpus to help BERT model better update its parameters.</p>
</li>
<li><p>Using a pre-trained language model that is pre-trained on a large amount of domain-specific text either from the scratch or fine-tuned on vanilla BERT model. As you might know, the vanilla BERT model released by Google has been trained on Wikipedia and BookCorpus text. After the vanilla BERT, researchers have tried to train the BERT architecture on other domains besides the initial data collections. You may be able to use these pre-trained models which have a deep understanding of domain-specific language. For your case, there are some models such as: <a href="https://github.com/dmis-lab/biobert" rel="nofollow noreferrer">BioBERT</a>, <a href="https://github.com/ncbi-nlp/bluebert" rel="nofollow noreferrer">BlueBERT</a>, and <a href="https://github.com/allenai/scibert" rel="nofollow noreferrer">SciBERT</a>.</p>
</li>
</ol>
<blockquote>
<p>Is it possible with hugging-face?</p>
</blockquote>
<p>I am not sure if huggingface developers have developed a robust approach for pre-training BERT model on custom corpora as claimed their code is still in progress, but if you are interested in doing this step, I suggest using <a href="https://github.com/google-research/bert" rel="nofollow noreferrer">Google research's bert</a> code which has been written in Tensorflow and is totally robust (released by BERT's authors). In their readme and under <code>Pre-training with BERT</code> section, the exact process has been declared. This will provide you with Tensorflow checkpoint, which can be easily converted to Pytorch checkpoint if you'd like to work with Pytorch/Transformers.</p>
Pretraining a language model on a small custom corpus
<p>I was curious if it is possible to use transfer learning in text generation, and re-train/pre-train it on a specific kind of text.   </p>

<p>For example, having a pre-trained BERT model and a small corpus of medical (or any "type") text, make a language model that is able to generate medical text. The assumption is that you do not have a huge amount of "medical texts" and that is why you have to use transfer learning.</p>

<p>Putting it as a pipeline, I would describe this as:  </p>

<ol>
<li>Using a pre-trained BERT tokenizer.  </li>
<li>Obtaining new tokens from my new text and adding them to the existing pre-trained language model (i.e., vanilla BERT).  </li>
<li>Re-training the pre-trained BERT model on the custom corpus with the combined tokenizer.  </li>
<li>Generating text that resembles the text within the small custom corpus.</li>
</ol>

<p>Does this sound familiar? Is it possible with hugging-face?</p>