Flagler was born in Hopewell, New York, the son of Isaac Flagler, a Presbyterian minister and his wife, the widowed Elizabeth Caldwell (Morrison) Harkness. She had brought two sons to the marriage with Flagler from her previous marriage to the widower Dr. David Harkness of Milan, Ohio. His son by his first marriage, Stephen V. Harkness, became Elizabeth's stepson. Together David and Elizabeth had a son Daniel M. Harkness before his death. He was of paternal German descent from the Palatinate region. The immigrant ancestor was Zacharra Flegler who first settled in Walworth, England and then left for America arriving in New York in 1710 eventually settling in Dutchess County. It was a grandson Solomon Flagler, who first used the different spelling of the surname. Solomon had eleven children including Isaac, Henry's father.