In 1839 it was chosen as the site of prison by the Governor of the colony of New South Wales, Sir George Gipps. Between 1839 and 1869 the island was used as a convict prison. The first prisoners came from Norfolk Island. They cut silos into the rock to store the colony's grain supply. By 1842, about 140 tonnes of grain were stored on the island.