Holder would spend much of his life in jail for his religious activism, being jailed as early as 1655 in Ilchester, Somerset for refusing to remove his hat. By the early 1650s Fox had been sending Quaker missionaries into Wales and Ireland, and by 1656 he deemed it time to send evangelists to the American colonies, an undertaking readily embraced by Holder. The first Quakers to arrive in the Massachusetts Bay Colony were Mary Fisher and Ann Austin, who landed in Boston in the early summer of 1656 aboard the "Swallow". Although there was no law barring their presence, they were jailed for three weeks and sent back to England.