With his first wife Hannah Jones Smalls, whom he married on December 24, 1856, Robert Smalls had three children: Elizabeth Lydia (1858–1959; m. Samuel Jones Bampfield, nine living children); Robert Jr. (1861–1863), who died at the age of two; and Sarah Voorhies (1863–1920, m. Dr. Jay Williams, no children). Hannah Jones Smalls had two daughters before she met and married Robert Smalls: Charlotte Jones (m. Willie Williams) and Clara Jones (m. James Rider). Smalls and his family were affiliated with the Baptist Church and attended Berean Baptist Church when living in Washington, D.C. Smalls was a Prince Hall mason as a member of Sons of Beaufort Lodge #36.