Smith served for nearly 20 years as the doctor at the Colored Orphan Asylum in New York. After it was burned down in July 1863 by a mob in the New York Draft Riots, in which nearly 100 blacks were killed, Smith moved his family and practice to Brooklyn for their safety. Many other blacks left Manhattan for Brooklyn at the same time. The parents stressed education for their children. In the 1870 census, his widow and children continued to be classified as white.