Williams married Susan Delise in July 1978. They are the parents of one daughter, Rae, and two sons, Antonio ("Tony") and Raphael ("Raffi"). Tony, who was a Senate page and intern for GOP Senator Strom Thurmond from 1996 to 1997, was a speechwriter and legislative correspondent for Republican Senator Norm Coleman from 2004 to 2006; in 2006, Tony ran unsuccessfully for a seat on the Council of the District of Columbia, losing to Tommy Wells. The younger son, Raffi, studied anthropology and played lacrosse at Haverford College, his father's "alma mater" in Pennsylvania. He has worked for the House Rules Committee and as the communications director for Michigan Republican Dan Benishek's successful 2012 congressional campaign, and was a deputy press secretary for the Republican National Committee. Raffi served as press secretary to former Secretary of Housing and Urban Development Ben Carson.