Yerby was born in Augusta, Georgia, on September 5, 1916, the second of four children of Rufus Garvin Yerby (1886–1961) and Wilhelmina Ethel Yerby (née Smythe) (1888–1960). Rufus, a hotel doorman, was part African-American, part Seminole; Wilhelmina ("Willie") was Scots-Irish. Yerby would later refer to himself as "a young man whose list of ancestors read like a mini-United Nations." One of Yerby's siblings was Alonzo Yerby, Associate Dean of the Harvard School of Public Health in Boston and New York City Hospitals Commissioner.