Hartman was born Philip Edward Hartmann on September 24, 1948 in Brantford, Ontario, Canada. He was the fourth of eight children of Doris Marguerite (Wardell) and Rupert Loebig Hartmann, a salesman specializing in building materials. One of his brothers is named John Hartman. His parents were Catholic and raised their children in that faith. As a middle child, Hartman found affection hard to earn and stated: "I suppose I didn't get what I wanted out of my family life, so I started seeking love and attention elsewhere."