O'Connor was married twice and had four children. His first marriage was in 1944 to Gwendolyn Carter, when he was 18 and she was 20. They married in Tijuana. Together they had 1 child, a daughter Donna. The couple divorced in 1954. During the turbulent nine year marriage, there was physical abuse toward O'Connor brought on by Carter and her frustration over the lack of an acting career. In the divorce Carter was given ownership of their home and custody of their daughter. According to reports at the time the couple split, O'Connor was left with only the dog and sought the help of multiple psychiatrists. Carter remarried in 1955 to actor Dan Dailey. O’Connor married his second wife, actress Gloria Noble, in 1956, remaining together until his death in 2003. They had 3 children, son Donald Frederick, daughter Alicia, and son Kevin. Noble died in 2013.