The Category A listed ruined Old St Peter's Church (St. Peter's Kirk) is one of the older churches in Scotland, dated to at least 1125, and at one time it was the principal church for the county, administered by the Bishops of Caithness. The church held hearings against criminal activity and determined how those caught should be punished. In 1701, a woman who had a relationship with a Dutch sailor had her head shaved and was publicly shamed, paraded through the town by the local hangman.