Story's young wife, Mary F.L. Oliver, died in June 1805, shortly after their marriage and two months after the death of his beloved father. In August 1808, he married Sarah Waldo Wetmore, the daughter of Judge William Wetmore of Boston. They had seven children but only two, Mary and William Wetmore Story, would survive to adulthood. Their son became a noted poet and sculptor—his bust of his father was mounted in the Harvard Law School Library—who would later publish "The Life and Letters of Joseph Story" (2 vols., Boston and London, 1851).