From February 12, 1969 until his death, Swain was married to Joye Raechel Boulton. While living in San Miguel de Allende, Mexico, they adopted Rocio Mendez Garcia (born 1959) and Antonia (born 1964). Later, in Costa Rica, they adopted Ronald, who would die of AIDS just two years after the death of his adoptive father.
Swain was married to Joye Raechel Boulton for a period of time, during which he had three children: Rocio Mendez Garcia, Antonia, and Ronald.