Born and raised in Pasadena, California, Fussell was the second of three children. His father, Paul Fussell (1895–1973), son of a widowed schoolteacher, became a corporate lawyer in Los Angeles with the firm of O'Melveny & Myers. His mother, Wilhma Wilson Sill (1893–1971), was the daughter of a carriage trimmer in Illinois. His brother, Edwin Sill Fussell, was an author, poet, and professor of American Studies at the University of California, San Diego; his sister Florence Fussell Lind lives in Berkeley, California.