They first started work to build the fort in 1840-42. It had a gun pit (special place for the cannon to sit and fire from) and firing wall (pictured in intro) that was built out of large blocks of sandstone and carved partly from the original rock that was already there. The first part of the building process around this time was done under the watchful eye of Major George Barney, a Commanding Royal Engineer who used hired workers provided by Governor Gipps.