Kelley was raised in Spokane, Washington, the eldest child of Adele and William Vincent Kelley, a lawyer who served as president of the city's bar association. Growing up, Kelley helped take care of her five sisters, Mary Cary, Ellen, Margaret, Adele Monica and Madeleine Sophie, as well as her brother, John. The family vacationed in Europe and spent summers at their two lakeside cottages in western Idaho. Kelley graduated from St. Augustine's Elementary School and then attended the private prep school Holy Names Academy.