Vanderbilt was born on February 2, 1856 in New Dorp, Staten Island. He was the third son of eight children born to William Henry Vanderbilt (1821–1885) and Maria Louisa (née Kissam) Vanderbilt (1821–1896). His siblings were Cornelius Vanderbilt II, who married Alice Claypoole Gwynne; Margaret Louisa Vanderbilt, who married Elliott Fitch Shepard; William Kissam Vanderbilt, who married Alva Erskine Smith and Anne Harriman Sands Rutherfurd; Emily Thorn Vanderbilt, who married William Douglas Sloane and Henry White; Florence Adele Vanderbilt, who married Hamilton McKown Twombly; Eliza Osgood Vanderbilt, who married William Seward Webb; and George Washington Vanderbilt II, who married Edith Stuyvesant Dresser.