When Dolley was 15, Payne moved his family to Philadelphia, where he went into business as a starch merchant, but the business had failed by 1791. This was seen as a "weakness" at his Quaker meetings, for which he was expelled. He died in October 1792 and Mary Payne initially made ends meet by opening a boardinghouse, but the next year she took her two youngest children, Mary and John, and moved to western Virginia to live with her daughter Lucy and her new husband, George Steptoe Washington, a nephew of George Washington.