Nonetheless he did take over the company's factoring business with the help of a loan for £500 thanks to the backing of his father and uncle and for several years travelled the country selling goods. In 1839, he married Mary (b. 1813 – died 28 November 1881), eldest daughter of Benjamin Scholefield of Sheffield. Brown lived with his wife throughout much of his working career at Shirle Hill in Sheffield. He died without issue at Shortlands, the house of Mr. Barron, Bromley in Kent, on 27 December 1896, and was buried at Ecclesall on 31 December.