On December 26, 1952 Williams wed Estelle Skolfield (1908–1999) of Baton Rouge, an LSU English professor. Estelle had received a bachelor of science degree from LSU in business administration and a master's in English. She joined the LSU faculty in 1938 and taught freshman composition and grammar for 25 years. The T. Harry Williams Center for Oral History established the Estelle Skolfield Williams Graduate Assistantship in her honor, a reflection of her work in helping her husband record oral interviews for the book "Huey Long".