Born in Charlotte, North Carolina, he was the son of the American painter Robert Gwathmey and photographer Rosalie Gwathmey. He attended the High School of Music and Art in New York City, graduating in 1956. Charles Gwathmey attended the University of Pennsylvania and received his Master of Architecture degree in 1962 from Yale School of Architecture, where he won both the William Wirt Winchester Fellowship as the outstanding graduate and a Fulbright Grant. While at Yale, he studied under Paul Rudolph.