Ballard was born in Toronto, Ontario, Canada as Edwin Harold Ballard. He later reversed the names and referred to himself as Harold E. Ballard. For six years before World War I, Ballard and his family lived in Norristown, Pennsylvania. They returned to Toronto where his father, English-born Sidney Eustace Ballard, founded Ballard Machinery Supplies Co., a sewing machine manufacturer, which at one point was one of Canada's leading manufacturers of ice skates (it got out of the business in the early 1930s, when the Canadian skate market was dominated by CCM). Harold attended Upper Canada College as a boarding student until dropping out in his third year in 1919.