Stone was dug up on the island to build the prison as well as other buildings in Sydney, including the seawall for Circular Quay. Between 1847 and 1857, convicts were used to dig the Fitzroy Dock, Australia’s first dry dock, on the island.