Storms' father Pablo was a Cuban American jai alai player who was more popular for his unpredictability than his skill at the sport; he died when he hurled a "pelota" that rebounded and struck him forcefully in the head. This occurred when Serge was very young, and left his mother a widowed single mother. She had an affair with another jai alai player from Spain, which resulted in pregnancy. Since she could barely cope with one child, and had no support from the father (he went back to Spain), she gave up her second son for adoption.