Smith was an early member of the Republican Party, and danced with his wife at Abraham Lincoln's inaugural ball in 1861. He was also a founder of the Boston YMCA, and was involved in many reform efforts and schemes for public improvement in the course of his long life. He is buried in the Mount Auburn Cemetery in Cambridge, Massachusetts. One block east of Villa Zorayda is the largest building Smith constructed in St. Augustine, the Casa Monica Hotel (later purchased by Henry Flagler and renamed the Cordova Hotel).