William Berkeley, 1st Earl FitzHardinge refused to claim his right to the earldom. In 1831 William Berkeley was raised to the peerage as Baron Segrave, of Berkeley Castle in the County of Gloucester. Berkeley was returned to parliament as one of two representatives for Gloucestershire in 1810 (succeeding his uncle Sir George Cranfield-Berkeley), a seat he only held until 1811. In 1836 he was appointed Lord-Lieutenant of Gloucestershire, a post he retained until his death. In 1841 he was further honoured when he was made Earl FitzHardinge. Berkeley never married. He had several mistresses, and in 1821 John Waterhouse succeeded in an action for "criminal
Berkeley Castle received a royal charter from King Henry II giving him permission to rebuild the castle. Fitzharding built the circular shell keep during 1153–56, probably on the site of the former motte. The building of the curtain wall followed, probably during 1160–90 by Robert and then by his son Maurice. Much of the rest of the castle is 14th century and was built for Thomas de Berkeley, 3rd Baron Berkeley: Thorpe's Tower, to the north of the keep, the inner gatehouse to its southwest, and other buildings of the inner bailey. The castle was ransacked in 1326 by the forces of