Dougherty was born in Texas. His parents were Ethel and Edward Dougherty. He was youngest of five children. His grandparents were immigrants from England and Ireland. When he was a child, he moved to Globe, Arizona with his parents.