section_path: Dataset preprocessing/Tokenization/Problems
section_name: Problems
paragraph_text: Problems
  
  A token vocabulary based on the frequencies extracted from mainly English corpora uses as few tokens as possible for an average English word. However, an average word in another language encoded by such an English-optimized tokenizer is split into a suboptimal amount of tokens. GPT-2 tokenizer can use up to 15 times more tokens per word for some languages, for example for the Shan language from Myanmar. Even more widespread languages such as Portuguese and German have "a premium of 50%" compared to English.
  
  Greedy tokenization also causes subtle problems with text completion.