Le was born in Quảng Trị, South Vietnam. He grew up in Da Nang. He fled Vietnam from the new communist rule in November 1977. He came by boat to Darwin as a refugee. He is married to Lan. The couple's two sons, Don and Kim, are named after Australian cricketers Don Bradman and Kim Hughes.