During the winter, they played baseball for the Fitzroy Baseball Club, often competing in matches played as curtain raisers to the elite Australian rules football competition, the Victorian Football League. In 1948, Harvey was selected at short stop in the 1948 All-Australian baseball team. Saturday night entertainment for the family typically consisted of dinner after the day’s cricket matches for Fitzroy and Plant, Liddicut and other club personnel were often invited. Under the influence of cricketing mentors Plant and Liddicut, the boys were taught to adopt an aggressive approach, using fast feet movement to attack spin bowling in particular. Harvey made his First XI debut for Fitzroy in 1941–42. In 1943–44, he scored a club record of 817 First XI runs in a season. In 1942–43, Neil was promoted to Fitzroy's First XI, joining Merv, Mick and Ray. During that season, the family held down the first four batting positions for Fitzroy; Merv and Mick opened the batting and Ray and Neil came in after them. Ray continued to play for Fitzroy over the next two decades when he was not required by Victoria. Outside cricket, Harvey was an electrical fitter and he worked for the Metropolitan Tramways Board. He and his wife Lorna and settled in the inner-northern suburb of Reservoir.