Henry Garnet was born into slavery in Chesterville (then New Market), Kent County, Maryland, on December 23, 1815. "is grandfather was an African chief and warrior, and in a tribal fight he was captured and sold to slave-traders who brought him to this continent where he was owned by Colonel William Spencer." According to James McCune Smith, Garnet's father was George Trusty and his enslaved mother was "a woman of extraordinary energy." In 1824, the family, which included a total of 11 members, secured permission to attend a funeral, and from there they all escaped in a covered wagon, via Wilmington, Delaware, where they were helped by the Quaker and Underground Railroad stationmaster Thomas Garrett.