The outbreak of the American Civil War and financial scandals caused great losses in the 1860s. He helped provide transportation for troops, and was meeting at Port Royal, South Carolina with Admiral William Reynolds whom he had known in Hawaii in 1840, when he heard Abraham Lincoln had been assassinated. He attended the funeral procession on his way home. In 1866 he invested in a venture to use freed slaves for a cotton plantation in Yazoo County, Mississippi, but that venture also failed. He had to sell his Beacon Street mansion to settle his debts.