 In May 1953 Sargent married Neltje Doubleday, who was 18.She was the granddaughter of the late Frank N. Doubleday, who founded the Doubleday publishing company in 1897.The couple had a daughter Ellen and son John Turner Sargent Jr.

After they divorced in 1965, Neltje Doubleday Sargent moved with their children to Wyoming.