More than 10,000 fewer people travelled by sea to the island in the first six months of the year - a 20% drop compared to the previous year.
The figures have been described as "very disappointing" by Guernsey's marketing and tourism boss.
Mike Hopkins said the result of the EU referendum was partly to blame.
"Overall visitor numbers to May have been very disappointing, with the month of May being particularly tough for everyone; the decline in the number of visitors departing by sea, having made the biggest impact," he said.
Mr Hopkins said the poor May figures followed "both uncertainty around the Brexit Referendum and a strong May performance in the previous year."
But he added that June 2016 had shown an overall increase in visitors.
Since the beginning of 2016 ferry passengers numbers have gone down by 12,500 and there has been a fall in visitors from cruise ships and private yachts of 7,000.
Day visitors from neighbouring Jersey and France have also gone down considerably - 3,300 fewer people visited Guernsey compared to 2015.
Guernsey's Committee for Economic Development said it was taking action to put pressure on the island's air and ferry providers to supply a more consistent service.
The committee's predecessor, the Commerce and Employment department, set a target in 2015 to increase visitors to the island by 30% by 2025.
An island tourism boss has said he partly blames the EU referendum for a big fall in the number of visitors to Guernsey by sea.