Bailey and her siblings grew up near De Smet, Idaho, on the Coeur d'Alene Reservation. Her father played fiddle and called square dances. Her mother played piano every evening and taught her to play and sing. Her younger brothers included Miles, Al, a vocalist and composer, and Charles, a lyricist. The family often hosted Saturday night gatherings, creating small music and dance hall in the home for local ranchers with both Josephine and Mildred at the piano. Mildred also accompanied her mother to traditional native ceremony where she heard and practiced the traditional songs and lyrics that would later influence her unique singing voice.