Gibson was born on August 25, 1927, in the town of Silver, in Clarendon County, South Carolina, to Daniel and Annie Bell Gibson, who worked as sharecroppers on a cotton farm. The Great Depression hit rural southern farmers sooner than much of the rest of the country, so in 1930 the family moved to Harlem,as part of the Great Migration, where Althea's three sisters and brother were born. Their apartment was located on a stretch of 143rd Street (between Lenox Avenue and Seventh Avenue) that had been designated a Police Athletic League play area; during daylight hours it was barricaded so that neighborhood children could play organized sports. Gibson quickly became proficient in paddle tennis, and by 1939, at the age of 12, she was the New York City women's paddle tennis champion. Gibson quit school at the age of 13 and, using the boxing skills taught to her by her father, engaged in a life of what she would later refer to as "street fighting", girls basketball, and watching movies. Fearful of her father's violent behavior, after dropping out of school, she spent some time living in a protective shelter for abused children.