Bailey was born in Devizes, Wiltshire, and raised from age 10 onwards on a farm in Nangeenan, Western Australia, without formal education. While growing up, he learned German, Italian, Spanish, Latin, and Greek from household books, and Russian from a neighbour. After he grew interested in the lettering on tea-chests from India, he acquired a book of Bible selections translated into languages with non-European scripts, including Tamil, Arabic, and Japanese. By the time he had left home, he was reading Avestan as well.