In 1754, Smith and his wife had a daughter called Hannah. Less than a month later, Smith was forcefully separated from his family when he was sold to Thomas Stanton in Stonington, Connecticut. They were reunited the following year when Stanton bought Meg and Hannah. After Smith had begun saving money that he had earned from working outside jobs and selling produce he grew, he hoped to buy freedom for his family. His time enslaved by the Stanton's began peacefully, but he was thrown into discord after Smith found his wife and the wife of Thomas Stanton in a heated dispute, with the latter beating upon the former with a switch. When Smith tried to break up the fight, Mrs. Stanton turned the switch upon him and so Smith took it from her grasp and threw it into the fire. In retaliation for this, Thomas Stanton attacked Smith seemingly out of the blue with a boat oar. After fending off this attack, a vexed Smith went to local authorities to complain about his abuse. Stanton and his brother, who had come to do violence unto Smith for his rebellion received a harsh warning and were publicly humiliated. In revenge for this, the brothers once again assaulted Smith once they were out of sight of the courthouse, and were once again overcome and repulsed by Smith.