Harvey was born into a working-class, devoutly Methodist family. His father was a keen amateur cricketer and an instilled a love of the game in his six sons, all of whom played for Fitzroy in Victorian Premier Cricket. Harvey made his first class debut in the middle of the 1947–48 season, and played his first two matches for Victoria alongside his more decorated brothers Merv and Neil. However, he performed poorly and did not play a match in the following summer and for the two subsequent seasons, he played in only three games but contributed little to his team's cause. In 1951–52 he made five appearances, gaining selection only when Victoria's Test players were busy representing Australia. The following season, he gained selection for the last three matches and broke through for his maiden first-class century, having never previously passed 50.