<p>The Transformer model was <a href="https://arxiv.org/pdf/1706.03762.pdf" rel="nofollow noreferrer">originally proposed for machine translation</a> in 2017, where it was directly trained on translation tasks. The additional step of pretraining Transformer-based models via self-supervised learning came later with models like GPT and BERT. The post you are looking at is an example of the former kind of approach.</p>
Transformers training vs fine-tuning for a specific task
<p>I am looking at the below tensorflow transformers implementation.</p>
<p><a href="https://www.tensorflow.org/text/tutorials/transformer" rel="nofollow noreferrer">https://www.tensorflow.org/text/tutorials/transformer</a></p>
<p>I am not sure I understood correctly. When initialising a transformers model it need to be trained on a lot of raw text in an unsupervised way so that it learns the language and then you can fit it to a particular task.</p>
<p>In this example, I am not sure if the training data is used to train the transformers model itself? It look like there is only one &quot;fitting&quot; procedure. Is this correct?</p>