 His paternal grandfather was botanist Charles Sprague Sargent, the first director of the Arnold Arboretum at Harvard University.His father became successful in finance as a partner in Hornblower & Weeks, a securities concern in New York.The young Sargent attended the private St. Mark's School and a year at Harvard College before enlisting in the Navy during World War II.