In 1850, Garnet went to Great Britain at the invitation of Anna Richardson of the free produce movement, which opposed slavery by rejecting the use of products produced by slave labor. He was a popular lecturer, and spent two and a half years lecturing. At first, the work separated Garnet from his family, who remained back in New York State. While he was abroad, his seven-year-old son, James Crummell Garnet, died on March 1, 1851 while Garnet was abroad. His wife Julia, his young son Henry, and their adopted daughter Stella Weims joined Garnet in Great Britain later that year.