Fitzgerald was born on April 25, 1917, in Newport News, Virginia. She was the daughter of William Fitzgerald and Temperance "Tempie" Henry. Her parents were unmarried but lived together in the East End section of Newport News for at least two and a half years after she was born. In the early 1920s, Fitzgerald's mother and her new partner, a Portuguese immigrant named Joseph Da Silva, moved to Yonkers, in Westchester County, New York. Her half-sister, Frances Da Silva, was born in 1923. By 1925, Fitzgerald and her family had moved to nearby School Street, a poor Italian area. She began her formal education at the age of six and was an outstanding student, moving through a variety of schools before attending Benjamin Franklin Junior High School in 1929.
Starting in third grade, Fitzgerald loved dancing and admired Earl Snakehips Tucker. She performed for her peers on the way to school and at lunchtime. She and her family were Methodists and were active in the Bethany African Methodist Episcopal Church, where she attended worship services, Bible study, and Sunday school. The church provided Fitzgerald with her earliest experiences in music.