In 1804, Custis married Mary Lee Fitzhugh. They had four children. Only one, Mary Anna Randolph Custis, survived. In 1818, Custis completed Arlington House in Alexandria, Virginia. He displayed a large number of Mount Vernon relics there.