Aldrich gave up on going to college after his father's death in 1849. At age 16, he joined his uncle's business office in New York in 1852 and started working on the newspapers and magazines. Aldrich made friends with other young poets, artists and wits of New York's "bohemia" of the early 1860s, including Edmund Clarence Stedman, Richard Henry Stoddard, Fitz Hugh Ludlow, Bayard Taylor and Walt Whitman. From 1856 to 1859, Aldrich worked for the "Home Journal", which was edited by Nathaniel Parker Willis at that time. During the Civil War, he was the editor of the "New York Illustrated News".