Born in Hartford, Connecticut, Driggs grew up in New Rochelle, a suburb of New York City, in a family that was supportive of her artistic interests. After a summer spent painting with her sister in New Mexico in her late teens, she felt she had found her life's calling. At twenty, she enrolled in classes at the Art Students League of New York, where she studied under George Luks and Maurice Sterne, both of whom were charismatic, inspirational figures in her early life. She also attended the evening criticism classes held at the home of painter John Sloan. Driggs spent fourteen months in Europe from late 1922 to early 1924, drawing and studying Italian art. There she met Leo Stein, first in Paris and later in Florence, who became an important intellectual influence, and who urged her to study Cézanne. He also introduced her to the works of Piero della Francesca, the Renaissance artist for whom she felt throughout her life the greatest admiration.