The son of Leslie Keith Davidson and Hilda Aileen Clifton, Davidson grew up in Lisarow, New South Wales (NSW) near the city of Gosford on the NSW Central Coast. He learnt to play cricket on a pitch that he dug out of a hill on his family's rural property. By the age of nine, he was playing in the second division of the Gosford grade competition. Throughout his high school years, he represented Northern High Schools in the state combined public schools' competition where he played against his future Test captain Richie Benaud, who captained City High Schools. During his teenage years, Davidson kept fit by working on the family property, chopping wood and carrying farm produce.