While studying at Rollins, Rogers met Sara Joanne Byrd who was from Oakland, Florida. They were married on June 9, 1952. They had two sons: James, in 1959, and John, in 1961. Joanne Rogers died in January 2021, aged 92.