Carroll Cole was born in Sioux City, Iowa, the second son of LaVerne (May 25, 1900 - February 5, 1975) and Vesta Cole (September 4, 1904 - January 27, 1984). His younger sister was born in 1939 and soon afterwards, his family moved to California, where LaVerne found work in a shipyard. Not long after that, LaVerne went to fight in World War II. While his father was away, his mother had several affairs and sometimes took Cole along to her rendezvous, threatening to beat him if he told his father. Vesta was emotionally abusive to Cole and dressed him as a girl. At school, he was teased about his "girl's name" by his peers.