The Rocks became established shortly after the colony's formation in 1788. It was known as Tallawoladah by the Cadigal people. The original buildings were first traditional vernacular houses, of wattle and daub, with thatched roofs, and later of local sandstone, from which the area derives its name. From the earliest history of the settlement, the area had a reputation as a slum and the arriving convicts' side of town, often frequented by visiting sailors and prostitutes. After November 1790, many of the inhabitants were also aboriginals. In 1823, the district had a population of about 1,200. During the late nineteenth century, the area was dominated by a gang known as the Rocks Push. It maintained this rough reputation until approximately the 1870s.