Technology Innovation Institute, Abu Dhabi
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, Julien Launay
web data;crawl;curated;deduplication;NLP;LLM
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data Only