NVIDIA; UIUC; University of Wisconsin, Madison
large language model;pretraining;retrieval augmentation;retro;knowledge retrieval
Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study