Switzer was born on October 5, 1937, in Crossett, Arkansas, to parents Frank Mays Switzer and Mary Louise Switzer. Barry and his younger brother, Donnie, were at home in rural Ashley County, Arkansas with their mother and father when, in early February 1954, it was raided by the Arkansas Alcoholic Beverage Control Commission and the Arkansas State Police. The Commission and the State Police found untaxed contraband liquor in the home. Frank made bond but was later tried and convicted of illegal trafficking in alcohol for purposes of re-sale ("bootlegging"). He was sentenced to a term of five years in prison, but that conviction was reversed upon appeal. Frank did serve five months of that term, and as a result, missed seeing Barry play his senior season of high school football.