In 1808, Webster married Grace Fletcher, a schoolteacher and the daughter of a New Hampshire clergyman. Between 1810 and 1822, Daniel and Grace had five children: Grace, Daniel "Fletcher", Julia, Edward, and Charles. Grace and Charles died before reaching adulthood. Webster's wife, Grace, died in January 1828 due to a cancerous tumor, and Webster suffered another loss when his brother, Ezekiel, died in April 1829. In December 1829, Webster married Caroline LeRoy, the 32-year-old daughter of a New York merchant. They remained married until Webster's death, and she lived until 1882. After the death of his first wife, Webster was frequently the subject of rumors in Washington regarding his alleged promiscuity; many suspected that the painter Sarah Goodridge, with whom he had a close relationship, was his mistress.