 Since their introduction, Transformers have become the basic building block of most state-of-the-art architectures in NLP, replacing gated recurrent neural network models such as the long short-term memory (LSTM) in many cases.Since the Transformer architecture facilitates more parallelization during training computations, it has enabled training on much more data than was possible before it was introduced.This led to the development of pretrained systems such as BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer), which have been trained with huge amounts of general language data prior to being released, and can then be fine-tune trained to specific language tasks.