Julian Carroll was born in West Paducah in McCracken County, Kentucky. He was the third of eleven children born to Elvie B. "Buster" and Eva (Heady) Carroll. His father was a tenant farmer, but shortly after the Ohio River flood of 1937, the family moved to Heath in McCracken County, where Buster Carroll sold tractor implements and in 1940 opened an automobile repair shop. Through his early teenage years, Carroll lived with his grandparents to help care for an ailing grandfather.