Baker was born November 27, 1859, in New York City, the son of Yale alumnus Benjamin Franklin Baker (b. 1834) and Harriette Luisa Bayeux (married 1857). Harriette was descended from well-to-do Huguenots who moved from France to New York before the American Revolutionary War. Benjamin's father was Ellis Baker (1793–1873), director of the Albany City Bank, Albany Mutual Insurance Company, and People's Line Steamboats, as well as founder of Albany Rural Cemetery and Albany Hospital. He also operated stagecoach lines from Albany to Boston as well as north and west of Albany.