After the siege, Colonel John Carter became governor of the castle. He made more repairs to fix it. In 1655, the English Council of State was told, by the Parliament of England, to pull down the castle so people could not use it anymore. The Bakehouse Tower was partially torn apart during that time. The castle was given back to Edward Conway by Charles II as part of the Restoration. But, five years later Conway took all of the iron and lead that was on the castle and sell it to make money. A man named William Milward was in charge of taking the iron and lead off the castle. The people who lived in the town of Conwy protested the removal. Conway and Millward did not care and the castle became a complete ruin.