William Berkeley, 1st Earl FitzHardinge refused to claim his right to the earldom. In 1831 William Berkeley was raised to the peerage as Baron Segrave, of Berkeley Castle in the County of Gloucester. Berkeley was returned to parliament as one of two representatives for Gloucestershire in 1810 (succeeding his uncle Sir George Cranfield-Berkeley), a seat he only held until 1811. In 1836 he was appointed Lord-Lieutenant of Gloucestershire, a post he retained until his death. In 1841 he was further honoured when he was made Earl FitzHardinge. Berkeley never married. He had several mistresses, and in 1821 John Waterhouse succeeded in an action for "criminal
Frederick Augustus Berkeley, 5th Earl of Berkeley Berkeley once had on the road:He had always declared that any one might without disgrace be overcome by superior numbers, but that he would never surrender to a single highwayman. As he was crossing Hounslow Heath one night, on his way from Berkeley Castle to London, his travelling carriage was stopped by a man on horseback, who put his head in at the window and said, ‘I believe you are Lord Berkeley?’ ‘I am.’ ‘I believe you have always boasted that you would never surrender to a single highwayman?’ ‘I have.’ ‘Well,’ presenting a pistol, ‘I am a single highwayman,