Following his marriage to Suzanne Morin with whom he had a son born in Europe and a daughter, Swing married his second wife, Betty Gram Swing (c. 1894 – 1969), a feminist and suffragist, whom he met in Berlin in 1919. Because Swing shared her views on gender equality, he adopted her surname and became known as Raymond Gram Swing. The couple had two sons and a daughter. They divorced in 1944 and he dropped the "Gram" from his name. He then married Mary Hartshorne. They lived in Washington, D.C. and spent time at her home in Jamaica where he composed music. He later married Meisung Loh, who survived him.