then came that morning , a few days before we were due to depart , as i sat on the terrace of the old avery , trying to write my letter to caroline and passing the time of day with anne bonny , when we heard the sound of carriage-gun fire from the harbour .