The castle and manor were granted by Henry VIII to his Attorney General, William Whorwood, whose family owned the manor of Compton, also in Kinver. The castle then became a home of that family until the late 1650s. The present house was presumably built by Thomas Whorwood when he became entitled to the whole of the manor in the 1580s. His grandson John Whorwood was probably neutral in the Civil War, but the castle was taken and briefly held by the brother of Colonel Tinker Fox in 1644, and later surrendered to Sir Gilbert Gerard, the Governor of Worcester after he routed Fox's relief column in an action on Stourbridge Heath.