Cavett was born on November 19, 1936 in Gibbon, Nebraska. He was raised in New York City, New York. He has German, French, British and Irish ancestry. Cavett studied at Yale University. He was married to Carrie Nye from 1964 until her death in 2006. He has been married to Martha Rogers since 2010. He has no children. 