Bryce lost his Waitotara seat in 1887. In 1889, Bryce was re-elected in 1889, this time to , and then in 1890 to Waikato. He was considered by some of his parliamentary supporters to be a possible replacement for Premier Harry Atkinson. Briefly he was Leader of the Opposition, but resigned from Parliament in 1891 for refusing to withdraw his criticism of the Premier. The Speaker, William Steward, ruled the criticism unparliamentary, and when Bryce refused to withdraw the words, "the House passed a vote of censure on him for not obeying the chair. He left the chamber, and never came back."