Williams was raised in Clarksville, Texas. His grandparents were farmers; his stepfather was a janitor in a post office. Williams attended a local junior college for a year, and then worked in media before joining the war effort in early 1942 by enlisting in the United States Army Air Force. He spent two and a half years as a sergeant in India and Burma. During his enlistment in Calcutta, he wrote pages of a novel, which later became "Nothing But the Night," published in 1948 by Swallow Press and later reissued by New York Review Books Classics.