Robbins was born in Glendale, a suburb of Phoenix in Maricopa County, Arizona. His mother was mostly of Paiute Indian heritage. Robbins grew up in a difficult family situation. His father took odd jobs to support the family of 10 children; however, his hard drinking led to divorce in 1937. Among his warmer memories of his childhood, Robbins recalled having listened to stories of the American West told by his maternal grandfather, Texas Bob Heckle, who was a local medicine man. At 17, Robbins left his troubled home to serve in the United States Navy as an LCT coxswain during World War II. He was stationed in the Solomon Islands in the Pacific Ocean. To pass the time during the war, he learned to play the guitar, started writing songs, and came to love Hawaiian music.