William's best-known brother was Captain Guy Ellis Baker (b. 1858), who married Louisa Irene Palma Di Cesnola, daughter of Civil War Medal of Honor recipient General Louis Palma di Cesnola. His other brothers included Benjamin Henry (b. 1869), George Clinton (b. 1872), and Ashley Bayeux (b. 1877). Baker spent much of his boyhood in the town of Ballston Spa, and his family discovered and purchased the property where he would build his summer studio, "The Castle", on the shores Ballston Lake.