section_path: Properties/Scaling laws
section_name: Scaling laws
paragraph_text: Scaling laws
  
  The performance of an LLM after pretraining largely depends on the:
  
  "Scaling laws" are empirical statistical laws that predict LLM performance based on such factors. One particular scaling law (" Chinchilla scaling ") for LLM autoregressively trained for one epoch, with a log-log learning rate schedule, states that: { C = C 0 N D L = A N α + B D β + L 0 {\displaystyle {\begin{cases}C=C_{0}ND\\L={\frac {A}{N^{\alpha }}}+{\frac {B}{D^{\beta }}}+L_{0}\end{cases}}} where the variables are
  
  and the statistical hyper-parameters are