Most of the land occupied by Henderson Hall was originally owned by the Custis family and later the Syphax family. Maria Carter Syphax, the matriarch of the Syphax family, was rumored to be the mulatto daughter of George Washington Parke Custis, grandson of Martha Washington and founder of the Arlington Estate on the banks of the Potomac River (later the home of Robert E. Lee). The Spyhax family sold the land to John Dormoyle in 1901, who then sold it to Frederick Rice in 1924.