Jimmy Baker was born some time around 1915. He was born at Malumpa, an old camping site near what is now Kaṉpi in north-western South Australia. His name at the time was Pintjutjara. He had a brother, Toby Ginger Baker, and a sister, Tjuwilya. He and his family lived a traditional, nomadic life in the desert, and had no contact with Euro-Australian society. Their first encounter with White people was in the early 1920s, with a group of Christian missionaries travelling from Ernabella to Warburton. On their way back, the missionaries tried to persuade the family to come with them to Ernabella, but Jimmy's father, Tjuwintjara, was not ready to give up life in the bush.