During the mid-1850s, Smith worked with Frederick Douglass to establish the National Council of Colored People, one of the first permanent black national organizations, beginning with a three-day convention in Rochester, New York. At the convention in Rochester, he and Douglass emphasized the importance of education for their race and urged the founding of more schools for black youth. Smith wanted choices available for both industrial and classical education. Douglass valued his rational approach and said that Smith was "the single most important influence on his life." Smith tempered the more radical people in the abolitionist movement and insisted on arguing from facts and analysis. He wrote a regular column in Douglass' paper, published under the pseudonym, 'Communipaw.'