He was the eldest son of John Foster, a small farmer, weaver and Baptist, living at Wadsworth Lane in the parish of Halifax, Yorkshire, born 17 September 1770. From a young age he assisted his parents in spinning and weaving wool. At age 17 he became a member of the Baptist congregation at Hebden Bridge; and soon after was "set apart" as minister by a special religious service, and went to reside at Brearley Hall with John Fawcett, who was directing the studies of some Baptist students. After three years here he entered the Baptist College, Bristol, in September 1791, remaining there till May 1792, and then entering on the regular work of a preacher.