 The attention calculation for all tokens can be expressed as one large matrix calculation, which is useful for training due to computational matrix operation optimizations which make matrix operations fast to compute.The matrices formula_30, formula_31 and formula_32 are defined as the matrices where the formula_7th rows are vectors formula_15, formula_35, and formula_36 respectively.formula_37

One set of formula_38 matrices is called an attention head, and each layer in a Transformer model has multiple attention heads.