James Backhouse was born in 1794, the fourth child of James and Mary Backhouse a Quaker business family of Darlington, County Durham, England. He was the third after his father and grandfather to be called James Backhouse. His grandfather died as a Quaker prisoner and martyr at Lancaster Castle in 1697. His father, James, (together with his father and brother), founded the Backhouse's Bank in Darlington. His mother was Mary Dearman of Thorne, Yorkshire, also a devout Quaker. His father died when he was a child and his mother brought him up in a religious atmosphere. He was educated in Leeds and began work in a grocery, drug and chemical business, but he developed tuberculosis and became too delicate to pursue a sedentary occupation. Coming from a family of botanists, he was encouraged to spend time in the open air and study botany. An uncle helped him in this and he made botanical trips to Upper Teesdale with the Durham lead miner, John Binks (1766–1817) who is credited with the discovery of many of the area's rare plants. Binks was a major influence, as was the Newcastle botanist, Nathaniel John Winch, and William Jackson Hooker (1785–1865). James then spent two years near Norwich learning the nursery trade, and where he first conceived of 'a gospel errand into Australia' believing strongly that this was the will of God.