Castle Cary Manor was given to the Lovels by King Stephen of Blois, who reigned from 22 December 1135 to 25 October 1154 (18 years 308 days). The castle was besieged by King Stephen of Blois in 1138 in his struggle for the throne of England with his cousin Matilda. When the Lovels failed to produce a male heir during the reign of King Edward III, it passed through marriage to Nicholas de S. Maure, a Baron. It passed again through female heir during the time of King Henry V to Lord Zouches of Harringworth. King Henry VII attainting John Lord Zouch gave the Castle to Robert Willoughby Lord Brook and Lord Zouch's lands at Bridge-water to Lord Daubney. The Zouches had backed King Richard III, the wrong side of the struggle between Richard III and the Tudors.