Townsend was born on October 6, 1917, in Montgomery County, Mississippi, the last of the 20 children of Ella and James Lee Townsend. After some of their animal stock was mysteriously poisoned, she suspected a local racist had done it; she said of this incident: "our stock got poisoned. We knowed this white man had done it ... That white man did it just because we were gettin' somewhere. White people never like to see Negroes get a little success. All of this stuff is no secret in the state of Mississippi."