Grumiaux was born in Villers-Perwin, Belgium. His grandfather urged him to begin music studies at the age of only 4. He trained on violin and piano with the Fernand Quintet at the Charleroi Conservatory. He took first prize at the age of 11. He received his first few major awards before reaching the age of 20. He took the Henry Vieuxtemps and François Prume prizes in 1939, and the Prix de Virtuosité from the Belgian government in 1940. During this time he also studied composition privately in Paris with the famous Romanian violinist George Enescu.