The French captured the castle in 1373 and 1381. The castle was probably last used militarily during the time of the French occupation of Jersey (1461–1468, when the French held the east of the island, but loyal Jersey folk held the western parishes). In 1483, after several years of petitioning King Edward IV, the Seigneur of St Ouen - recommended to the King's Grace by his father-in-law the governor, was granted a ″License to Crenelate″ - that is allowed to fortify his manor house. Governor Harliston approved the dismantling of the ″fort of refuge″ at Grosnez so that the already-worked stone could be re-used in fortifying St. Ouen's Manor. The financial savings were important to Seigneur De Carteret, as - when he came into his majority - ″trees grew in the Hall of the Manor because his Guardians had wasted the substance of the Rentes″ i.e. - embezzled the income - and he was investing heavily in renovating the dilapidated manor buildings. His shortage of money was both what influenced him to marry the daughter of the Plantagenet Governor Harliston, and also what led him into conflict with the second governor of the island after 1486. When Matthew Baker was sent to take Harliston's place in 1486, the Seigneur had not paid his taxes to the Exchequer for three years - since the death of Edward IV. De Carteret was already under some suspicion as a Plantagenet supporter - after all - the rebel Governor Harliston was his father-in-law. When King Henry VII ordered a general examination of outstanding accounts by the Exchequer clerks, it was inevitable that De Carteret would be required to pay the back taxes he owed the Crown. This, then, is the background to the dismantling of Grosnez Castle - and also of the later friction between De Carteret and Matthew Baker.