Born Harriet E. "Hattie" Adams in Milford, New Hampshire, she was the mixed-race daughter of Margaret Ann (or Adams) Smith, a washerwoman of Irish ancestry, and Joshua Green, an African-American "hooper of barrels" of mixed African and Indian ancestry. After her father died when Hattie was young, her mother abandoned Hattie at the farm of Nehemiah Hayward Jr., a well-to-do Milford farmer "connected to the Hutchinson Family Singers". As an orphan, Adams was bound by the courts as an indentured servant to the Hayward family, a customary way for society at the time to arrange support and education for orphans. The intention was that, in exchange for labor, the orphan child would be given room, board and training in life skills, so that she could later make her way in society.