 On August 30, 1789, three months after his last trip to China, he married Anna Olcott (1765–1850), the daughter of Thomas Olcott.They moved to Hartford, then Philadelphia, and eventually in New York City.Together, they had ten children, two of whom died in infancy.