Fitzgerald was born on April 25, 1917, in Newport News, Virginia. She was the daughter of William Fitzgerald and Temperance "Tempie" Henry. Her parents were unmarried but lived together in the East End section of Newport News for at least two and a half years after she was born. In the early 1920s, Fitzgerald's mother and her new partner, a Portuguese immigrant named Joseph da Silva, moved to Yonkers, in Westchester County, New York. Her half-sister, Frances da Silva, was born in 1923. By 1925, Fitzgerald and her family had moved to nearby School Street, a poor Italian area. She began her formal education at the age of six and was an outstanding student, moving through a variety of schools before attending Benjamin Franklin Junior High School in 1929.