At the election of 1857 Sir William, who had meantime been defeated at Harwich, was again returned at Barnstaple. He was, however, defeated in 1859, but was elected in 1863 at Ludlow. This seat he held for only two years, when he was again defeated and did not re-enter parliament until 1874, when he was returned for Kidderminster, a constituency he represented for six years, when he retired. He was a familiar figure at the Carlton Club, always ready with a copious collection of anecdotes of Wellington, Benjamin Disraeli and Napoleon III.