Frost was married to Lynne Frederick from 1981 until they divorced in 1982. He had an 18-year intermittent affair with Carol Lynley. Then he was married to Lady Carina Fitzalan-Howard from 1983 until his death in 2013. He had three sons with Fitzalan-Howard. Diana, Princess of Wales was the godmother of Frost's youngest son. He was once briefly engaged to American actress Diahann Carroll.