Smith was born into slavery in 1813 in New York City and was set free on July 4, 1827, at age 14, by the Emancipation Act of New York. That was the final date when New York officially freed its remaining slaves. His mother was an enslaved woman named Lavinia who achieved her freedom later in life; in 1855, Smith described her as a "self-emancipated woman." She was born into slavery in South Carolina and had been brought to New York as a slave. His father was Samuel Smith, a white merchant and his mother's master, who had brought her with him to New York from South Carolina.