The club was unable to build upon their relatively competitive start to life in the Highland League, and gradually, season-by-season, began to struggle to the point where they had finished bottom in 14 of the 18 seasons between 1996–97 and 2013–14. Their struggles included a record 17–0 away defeat in 1998, to now Scottish League One side Peterhead. Despite this, "The Fort" did not finish bottom that season, with the wooden spoon going to Nairn County. In the 2008–09 season, they accumulated only a solitary point from 28 matches, making it the Highland League's record lowest points total since its inception.