A series of tragedies struck the Weeks family following the completion of the house. David Weeks died in August 1834. His wife married John Moore. The household was dependent on slavery. Mary and John supported secession. During the American Civil War, Federal troops occupied the property. Union officers lived in the house. Mary Weeks died in December 1863. The house was still being used by Union troops. He son restored some of the family fortune during the Reconstruction years. Succeeding generations however were forced to sell much of the property to meet living expenses.