Clement Clay and his wife Virginia were among Southerners imprisoned at Fort Monroe after the war; they were suspected of being involved in the assassination plot against President Abraham Lincoln. Also held at that prison was Jefferson Davis, their friend and former president of the Confederacy. They became even closer friends during this time. Varina Howell Davis used to visit her husband, bringing their youngest daughter. The Clays were released in 1866, but Davis was not released until 1867. The families had some continued contact after the Clays returned to Huntsville, Alabama.