The estate was once again passed to various noble families until 1667, when Charles II granted it to Charles Paulet, 1st Duke of Bolton. He was followed successively by his son and grandson, but by the mid 18th century it was back in royal hands, being held successively by Prince William Henry (up to 1805) and Prince Frederick (until 1827). Subsequently, the manor was deemed "not important to be kept", and the copyholds of the manor, which included estates in Minstead, Burley, Bartley and Poulner, either became enfranchised or passed to the Crown.