After leaving the Cabinet, Brooke stood unsuccessfully for the position of Speaker of the House of Commons. The House instead elected the Labour MP Betty Boothroyd to the role, with several Conservative MPs voting against Brooke on the grounds that he had too recently been in the Cabinet and was thus insufficiently close to the backbenches. Brooke then remained on the backbenches for a short time, before being brought back into the Cabinet later in the year as Secretary of State for National Heritage, a role he held until 1994. During his time as Heritage Secretary, he oversaw the restoration of Windsor Castle following the fire that had struck the State Apartments in 1992.