Duke, born as Anne LaBarr Lederer, grew up in Concord, New Hampshire, where her father, writer and linguist Richard Lederer, taught English literature at St. Paul's School and her mother, Rhoda Lederer, taught at Concord High School. Her parents were both card players and Duke became interested in cards from an early age. Her siblings are professional poker player Howard Lederer and author/poet Katy Lederer, who published a memoir about the Lederer family. Her father is Jewish, while her mother was a gentile, and therefore she was not raised Jewish.