Potter was a member of several families that were prominent in the American South since the 17th century. She was a great-great-granddaughter of Francis Scott Key and a direct descendant of Pocahontas. His wife Mary Tayloe Lloyd Key was both a Lloyd, daughter of Edward Lloyd IV of Wye House and a Tayloe of Mount Airy on her mother, Elizabeth's side, the daughter of John Tayloe II and sister to John Tayloe III of The Octagon House, arguably the wealthiest American, if not Southern Planter. Her grand-aunts Jennie and Hetty Cary (wife of the Confederate general John Pegram) were well-known figures during the Civil War, known as the "Cary Invincibles" and considered heroines for sewing battle flags. It was Jennie Cary who put the words of James Ryder Randall's poem "Maryland, My Maryland" to the German folk song "Lauriger Horatius", thereby creating what would become the state song of Maryland. Her mother's cousin and sometime guardian Constance Cary Harrison was one of the United States' best-known women in the late 19th century, a prominent novelist and social reformer. Another cousin, Francis Burton Harrison, served as Governor General of the Philippines and was a Democratic candidate for the U.S. presidency.