Paul G. Allen School of Computer Science & Engineering, University of Washington; Language Technologies Institute, Carnegie Mellon University; Allen Institute for Artificial Intelligence
Tokenization;LLMs;under-resourced languages;equitable LLMs
Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models