Ray Harvey's father, Horace "Horry" Harvey moved to Broken Hill, New South Wales where he worked for BHP driving horse-drawn trailers. In 1914, he married Elsie May Bitmead and their first two children, daughter Rita and son Mervyn, were born in the mining town. The family relocated to Newcastle, a mining town and harbour in New South Wales and had two more sons, "Mick" and Harold there. In 1926, the Harveys shifted to the inner-Melbourne suburb of Fitzroy, a staunchly working-class, industrial area. During their relocation, Ray was born in Sydney. Horace secured a job at the confectionery company Life Savers (Australasia) Ltd. located next door to their house at 198 Argyle Street. The 19th century two-storey house was owned by the firm and was used as lodgings for the workers' families. It no longer exists, having been demolished to make way for a textile factory. The two youngest sons Neil and Brian were born in Fitzroy.