Byrd enjoyed retirement at his home in Winchester, "Courtfield," and time spent with his nine grandchildren and later his twelve great grandchildren. Byrd's wife of 48 years, Gretchen, died in 1989. He continued to serve as Chairman of the Board of the "Star" for almost twenty years. In 2003 he was named to the Virginia Communications Hall of Fame. Byrd became a lecturer at Shenandoah University, and in 1984 the business program was renamed the Harry F. Byrd Jr. School of Business. His name was removed from the business school after a unanimous vote on June 10, 2020, due to the senator's legacy as a civil rights opponent. In 2007 Byrd completed a literary work, "Double Trouble: Vignettes From A Life of Politics and Newspapering". On October 20, 2009, with the death of retired U.S. Senator Clifford P. Hansen, a Wyoming Republican, Byrd became the oldest living former senator until his death at the age of 98.