Killen was moved out of Defence in a 1982 reshuffle. He was made a Knight Commander of the Order of St Michael and St George, becoming "Sir James Killen KCMG", and appointed Vice-President of the Executive Council, a position he held until the defeat of the Fraser government in 1983 election by Labor under Bob Hawke. He became Father of the House of Representatives in April 1983, and resigned his seat of Moreton in August 1983 (the first Queensland Member of the House of Representatives to resign), and returned to his legal practice. He was a prominent figure at the Brisbane bar through the 1980s and 1990s.