Alger was divorced in 1961 from Lucille "Lynn" Antoine, who said that politics caused an estrangement in the marriage to point that they had little in common except for a liking for gin rummy. The couple had three children, Jill Alger of The Villages in Sumter County in central Florida] and sons David and Steven, who died in 1964 and 2012, respectively. Alger's second wife, the former Priscilla Jones, also died in 2012, after thirty-six years of marriage. He had two step-children in Massachusetts, Robert Jones of Amherst and Laura Jones of Chatham. Alger had seven grandchildren and seven great-grandchildren. Alger retired in 1990 and spent ten years with his wife Priscilla traveling around the United States in a recreational vehicle. The couple settled in 2000 in Barefoot Bay in Brevard County near Melbourne on the central section of the Atlantic Coast of Florida.