 The last decoder is followed by a final linear transformation and softmax layer, to produce the output probabilities over the vocabulary.Transformers typically undergo semi-supervised learning involving unsupervised pretraining followed by supervised fine-tuning.Pretraining is typically done on a much larger dataset than fine-tuning, due to the restricted availability of labeled training data.