Alice Elvira Freeman was born in Colesville, New York, as the eldest child of four children of Elizabeth Josephine Higley and James Warren Freeman. From her father, she acquired her "moral beauty", as well as her height and red highlights in her hair. Palmer was particularly close to her mother during her childhood, partly because her mother was only 17 years-of-age when she was born, and also because of their shared responsibility caring for her younger siblings and performing household duties. Elizabeth was an advocate for children and women's rights and health care, as well as the temperance movement. Her father, a farmer in her early years, was interested in making direct changes to people's lives, rather than through reform movements, and shared his interest in education and the sciences with his daughter.