At age 17, Smalls married Hannah Jones, an enslaved hotel maid, in Charleston on December 24, 1856. She was five years his senior and already had two daughters. Their own first child, Elizabeth Lydia Smalls, was born in February 1858. Three years later they had a son, Robert Jr., who later died at age two. Robert aimed to pay for their freedom by purchasing them outright, but the price was steep, $800 (). He had managed to save up only $100. It could take decades for him to reach $800.