Story was born at Marblehead, Massachusetts. His father was Dr. Elisha Story, a member of the Sons of Liberty who took part in the Boston Tea Party in 1773. Dr. Story moved from Boston to Marblehead during the American Revolutionary War. His first wife, Ruth (née Ruddock) died and Story remarried in November 1778, to Mehitable Pedrick, nineteen, the daughter of a wealthy shipping merchant who lost his fortune during the war. Joseph was the first-born of eleven children of the second marriage. (Story also fathered seven children from his first marriage.)