At the Colonial Conference held in London during 1887, Downer represented South Australia, but during his return journey to Australia his government was defeated. This ministry was responsible for a tariff imposing increased protective duties. Downer was not in office again for several years, but in October 1892 again became Premier, taking also the portfolio of Chief Secretary. In May 1893 he exchanged this for the position of Treasurer of South Australia, but was ousted at the 1893 election by liberal Protectionist Kingston with the support of the new Labor Party led by John McPherson. Downer remarked of this party: 'They are very clever fellows. I have great respect for the way they use either side for their purposes with absolute impartiality'. For most of the period until 1899 Downer led the Opposition.