In the mid-1850s, Smith joined James W.C. Pennington and other black leaders in establishing the Legal Rights Association (LRA) in New York City. A pioneering minority-rights association, the LRA waged a nearly ten-year campaign against segregated public transportation in New York City. This organization successfully defeated segregation in New York and served as a model for later rights organizations, including the National Equal Rights League and the National Association for the Advancement of Colored People (NAACP), founded in the early 20th century.