When O'Connor was only two years old, he and his sister Arlene, who was seven at the time, were in a car crash outside a theater in Hartford, Connecticut; O'Connor survived, but his sister did not. A few weeks later, his father died of a heart attack while dancing on stage in Brockton, Massachusetts. His brother Billy died a decade later from scarlet fever and his eldest sibling Jack died from alcoholism in 1959. His three other siblings died during childbirth. O'Connor said the tragedies "marred my childhood and it's still haunting."