Specter was born on February 12, 1930 in Wichita, Kansas. He graduated from the University of Pennsylvania. He was with the United States Air Force during the Korean War. Specter later graduated from Yale Law School. He was married to Joan Specter. They had two children.