Hoyer was born on June 14, 1939 in New York City, New York. He is of Danish descent. Hoyer studied at University of Maryland, College Park and at Georgetown University. He was married to Judith Hoyer until her death in 1997. They had three children.