In 1856 after a number of small arguments with his offices, Tolmer was removed as commissioner. He tried farming but this was unsuccessful. He tried to lead an exploring group to cross Australia from south to north, but this also failed before it left Adelaide. He became a ranger for crown lands in 1862 and retired in 1885. He died at Mitcham, South Australia, on 7 March 1890.