Her parents died early. After this, her maternal grandmother, Marry Ludlow Hall (1843-1919) raised her. She spent her childhood in Tivoli, New York. Most of her mother’s family members tended to look down at her, perhaps because of her plain looks and six-foot tall frame. Even her Hyde Park Roosevelts, including her future mother-in-law, Sara Delano Roosevelt, would remark to her about the Manhattan Roosevelts: "we got all the looks and the money."