Butler had an exceptionally long ministerial career and was one of only two British politicians (the other being John Simon, 1st Viscount Simon) to have served in three of the four Great Offices of State but never to have been Prime Minister, for which he was passed over in 1957 and 1963. At the time, the Conservative Leadership was decided by a process of private consultation rather than by a formal vote. After retiring from politics in 1965, Butler was appointed Master of Trinity College, Cambridge.