Robert Smalls was born in 1839 to Lydia Polite, a slave owned by Henry McKee. She gave birth to him in a cabin behind McKee's house, at 511 Prince Street in Beaufort, South Carolina. He grew up in the city under the influence of the Lowcountry Gullah culture of his mother. His mother lived as a servant in the house, but she grew up in the fields. Robert was liked more than other slaves, so his mother worried that he might grow up not understanding the troubles of field slaves. She asked for him to be made to work in the fields and to see whipping.