His brother did not take kindly to losing his flock, most of whom would join the Church, despite a late attempt to have Robert Aitken (an English priest and preacher) offer baptism to his congregation. By 21 September 1837, Heber C. Kimball was able to note of James Fielding that, "His church has left him and he is an object of pity." James was made something of a laughing stock in Preston, and referred to his brother as a 'sheep stealer.'; James later published pamphlets attacking the Church. But more lay behind his bitterness than just the loss of his congregation. At the date of the missionaries' arrival James was doing so well that his congregation had outgrown the 500 seat Vauxhall Chapel, and he had contracted to build a new, larger chapel a few streets away on Avenham Lane, which opened in January 1838 as the Primitive Episcopalian Church. It had cost £1,500 to build and the builders and contractors were pressing for payment of their bills. Without much of a congregation and therefore unable to meet the demands, James was compelled to sell the building. Ever anxious to eliminate 'dissenting' congregations, and in genuine need of more Church of England places of worship to accommodate Preston's growing population, Roger Carus Wilson, Vicar of Preston, bought the newly built chapel in April 1838 at the much reduced price of £1,000. The whole affair was of considerable embarrassment to James. By the beginning of January 1838, he became seriously ill. Many in Preston blamed Joseph, saying that he had broken his brother's heart by stealing his congregation. James eventually recovered his health but not his congregation. Joseph Fielding recorded that his brother finally left his church in March 1839. However, according to his own statement, James continued to preach elsewhere in the Preston area for a further 13 years.