Olyphant is a descendant of the Vanderbilt family of New York. His paternal fourth great-grandfather was family patriarch Cornelius Vanderbilt; his third great-grandfather was William Henry Vanderbilt, who doubled the family's railroad fortune; his great-grandmother was socialite Emily Vanderbilt Sloane; and his great-uncle was music producer John Hammond. The surname Olyphant is of Scottish origin. Another paternal fourth great-grandfather, Dr. David Olyphant, was born in Scotland and served as director-general of the Southern hospitals during the American Revolutionary War. His third great-grandfather, David Olyphant, and great-great-grandfather, Robert Morrison Olyphant, were both prominent businessmen.