At Ernabella, Jimmy's first job was baking bread. Because of this, he was given the surname "Baker", and he chose the Christian name of "Jimmy" himself. Later, he got a job building fences on nearby stations, first at Kenmore Park and then Everard Park. In the late 1930s, Jimmy got married to a woman named Nyinmungka, who he met while working at Everard Park. The two of them moved back to Kenmore Park to live. They had one son and two daughters: Anton, Kay and Marita. All of Jimmy's children became artists and founding members of the Tjunga Palya centre.