Naver Labs Europe; University of Amsterdam
Nadezhda Chirkova, Sergei Troshin
source code processing;tokenization;byte-pair encoding
CodeBPE: Investigating Subtokenization Options for Large Language Model Pretraining on Source Code