James was born near Ernabella (now Pukatja) in central Australia, and was a member of the Pitjantjatjara people. He went to school at the Ooldea Mission. In 1945, he was wrongfully arrested and left the mission. James began his career as a tracker for police in 1948. He became well known for tracking criminals (including arsonists, poachers and escapees) and lost persons. He was awarded the Medal of the Order of Australia in 1984 and was the first South Australian Aboriginal Person of the Year in 1983. After his death, a memorial to Jimmy James was erected in Berri, South Australia.