Aldrich married and had two sons. Mark Twain is said to hated Aldrich's wife Lilian. He wrote in 1893: "Lord, I loathe that woman so! She is an idiot—an absolute idiot—and does not know it ... and her husband, the sincerest man that walks ... tied for life to this vacant hellion, this clothes-rack, this twaddling, blethering, driveling blatherskite!"