Carrouges IV grew up within his father's domain, centred around the village of Carrouges where the family maintained their own hereditary castle. He followed his father into the armed service of the Counts of Perche and served in several minor campaigns against the English and "routiers" in Normandy. Following his majority at age 21, he was given a parcel of the family lands to administer and became interested in solidifying and expanding the family holdings. In 1367 the family castle and the village of Carrouges were destroyed by English soldiers and a new castle was built on a hilltop nearby, under instructions from Charles V of France.