Born at Kapunda, South Australia, Horace "Horry" Harvey moved to Broken Hill, New South Wales where he worked for BHP, driving horse-drawn trailers. In 1914, he married Elsie May Bitmead and their first two children, daughter Rita and son Mervyn, were born in the mining town. The family relocated to Newcastle, a mining town and harbour in New South Wales, and had two more sons, Clarence (always known as "Mick", as he was born on Saint Patrick's Day) and Harold. In 1926, the Harveys shifted to the inner-Melbourne suburb of Fitzroy, a staunchly working-class, industrial area. During their relocation, Ray was born in Sydney. Horace secured a job at the confectionery company Life Savers (Australasia) Ltd, located next door to their house at 198 Argyle Street. The 19th century two-storey house was owned by the firm and was used as lodgings for the workers' families. It no longer exists, having been demolished to make way for a textile factory. The two youngest sons Neil and Brian were born in Fitzroy.