Mathews was born in New York City. He was raised in Massachusetts. He studied at Princeton University and at Harvard University. In 1949 he eloped with his childhood friend and artist Niki de Saint Phalle. After having two children together, they separated in 1960. He later married Marie Chaix. Their marriage would last until Mathews' death in 2017.