In 1846, Smith was appointed as the only doctor of the Colored Orphan Asylum (also known as the Free Negro Orphan Asylum), at 44th Street and Fifth Avenue. (Before that time, the directors had depended on "pro bono" services of doctors.) He worked there for nearly 20 years. The asylum was founded in 1836 by Anna and Hannah Shotwell and Mary Murray, Quaker philanthropists in New York. Trying to protect the children, Smith regularly gave vaccinations for smallpox. Leading causes of death were infectious diseases: measles (for which there was no vaccine), smallpox, and tuberculosis (for which there was no antibiotic at the time). In addition to caring for orphans, the home sometimes boarded children temporarily when their parents were unable to support them, as jobs were scarce for free blacks in New York. Waves of immigration from Ireland and Germany in the 1840s and 1850s meant that many new immigrants were competing for work.