Foster first took charge of a small Baptist society at Newcastle-on-Tyne for three months in 1792. In the beginning of 1793 he went to Dublin to minister at a meeting-house in Swift's Alley. He lost his congregation, a recurring feature of his life. He went home, but returned to Dublin in 1795 to take charge of the classical and mathematical school of John Walker, which after eight or nine months he gave up as a failure. He was close to some of the extreme Dublin democrats, exposing him to the danger of imprisonment.