section_path: Compression
section_name: Compression
paragraph_text: Compression
  
  Typically, LLMs are trained with single- or half-precision floating point numbers (float32 and float16). One float16 has 16 bits, or 2 bytes, and so one billion parameters require 2 gigabytes. The largest models typically have 100 billion parameters, requiring 200 gigabytes to load, which places them outside the range of most consumer electronics.
  
  Post-training quantization aims to decrease the space requirement by lowering precision of the parameters of a trained model, while preserving most of its performance. The simplest form of quantization simply truncates all numbers to a given number of bits. It can be improved by using a different quantization codebook per layer. Further improvement can be done by applying different precisions to different parameters, with higher precision for particularly important parameters ("outlier weights"). See the visual guide to quantization by Maarten Grootendorst for a visual depiction.
  
  While quantized models are typically frozen, and only pre-quantized models are fine-tuned, quantized models can still be fine-tuned.