There he began to write his memoir, "The Rise and Fall of the Confederate Government". Dorsey was instrumental in his success, organizing his day, motivating him to work, taking dictation, transcribing notes, editing and offering advice. Rumors quickly began to fly that the two were having an illicit affair, and it was nearly "an open scandal," but they refused to yield to it. Varina Davis became enraged and refused for a long time to set foot on Dorsey's property. Eventually she accepted Dorsey's invitation to live there and moved into one of the guest cottages at Beauvoir.