section_path: Training and architecture/Mixture of experts
section_name: Mixture of experts
paragraph_text: Mixture of experts
  
  The largest LLM may be too expensive to train and use directly. For such models, mixture of experts (MoE) can be applied, a line of research pursued by Google researchers since 2017 to train models reaching up to 1 trillion parameters.