After the dismissal of the Whitlam government on 11 November 1975, Withers was appointed to Malcolm Fraser's first (caretaker) ministry, becoming Vice-President of the Executive Council as well as briefly holding the portfolios of Special Minister of State, Capital Territory, Media, and Tourism and Recreation during the period leading up to the December election. After the election, Withers became Minister for Administrative Services, and continued as Vice-President of the Executive Council until 7 August 1978. He was dismissed by Fraser in the wake of the findings of a Royal Commission into aspects of a redistribution of certain federal electorates in Queensland. The royal commission found that Withers had exercised his ministerial influence in an inappropriate way. At the time he commented about Fraser that "When the man who's carried the biggest knife in this country for the last ten years starts giving you a lecture about propriety, integrity and the need to resign, then he's either making a sick joke or playing you for a mug".