The fortress sits out on the breakwater and you will walk past the city's fishing fleet to reach it.
walk past the fishing fleet and you will get to the fortress on the breakwater.
The fortress is before the fishing fleet.