In February 1869, he resigned that position and never held office again. Later on in the year, in response to a request that representatives of the colony should be sent to a conference on colonial affairs in London, Higinbotham moved and succeeded in carrying five resolutions declining to send representatives, and repeating his views that the internal affairs of a colony are its own concern and that the colonial office should only look after matters that effect the whole empire. A year later at the election for Brighton held in March 1871 Higinbotham was defeated by 14 votes. It was a contest between a realist and an idealist. His opponent, Thomas Bent, was a man who understood the art of looking after his own constituency. Higinbotham cared nothing for its special needs and thought only of the good of the whole colony. He welcomed his release from the bickerings of politics and for two years built up his position as a barrister.