Elgort was born in New York City, New York. His father is Arthur Elgort, a photographer who has worked for "Vogue". His mother is Grethe Barrett Holby, an opera director. He has Russian Jewish, English, German and Norwegian ancestry. He has been dating his girlfriend, Violetta Komyshan, since high school.