Harvey was born in Fitzroy, a suburb of Melbourne, Victoria. He is one of six cricketing brothers. Neil and brothers Merv, Mick and Ray would go on to play for Victoria. During the 1949–50 tour of South Africa, Harvey met Iris Greenish. The couple later married and had three children.