Carrouges was born in the late 1330s in the village of Saint-Marguerite-de-Carrouges as the eldest son of knight and minor noble Sir Jean de Carrouges III and his wife Nicole de Buchard. Carrouges III was an influential man in lower Normandy, being a vassal of the Count of Perche and a veteran soldier in his service. Carrouges III had been rewarded for his long military service in the Hundred Years War with a knighthood and the title of Viscount of Bellême, a rank that came with command of a vital hill castle overlooking the town as well as the role of sheriff in the vicinity, a post carrying significant financial and social rewards.