Farms have been hit with a shortage of the migrant workers that Britain relies on to bring in the fruit and vegetable harvests, according to a series of new reports.

There was a 17% shortfall in May, leaving some farms critically short of pickers, according to a new National Farmers Union (NFU) survey. The decline is blamed on Brexit, with the vote to depart the EU leaving the UK seen as “xenophobic” and “racist” by overseas workers, according to the director of a major agricultural recruitment company.

The UK requires about 80,000 seasonal workers to pick the vegetable and fruit harvest and virtually all come from eastern Europe. Just 14 of the 13,400 workers recruited between January and May this year were British, the NFU survey found. Three-quarters of the workers came from Bulgaria and Romania, and almost all the rest from other eastern European countries.

The authoritative NFU analysis showed the shortfall in May 2016, before the Brexit referendum, was just 4%. It also showed the proportion of overseas workers choosing to return to the UK has plummeted from two-thirds to one-third since January, meaning valuable experience is being lost.

“The grim reality is that the perception from overseas is we are xenophobic, we’re racist, and the pound has plummeted too,” said John Hardman, director at Hops Labour Solutions, who also estimates a 20% shortage of workers. “We’ve gone with Brexit and that makes us look unfriendly.

“The immediate impact is that there will be crops left in the fields,” he said. “[The warning] couldn’t be more timely with Wimbledon around the corner as 99.9% of Wimbledon strawberries are picked by eastern Europeans.”

Another large survey of salad and fruit farmers, released by the BBC on Thursday, found that 21% reported fewer workers than they needed in late May. Almost 80% of the growers said recruitment had been more difficult than last year, with 20% saying it had been the hardest for years.

Further evidence of the labour crisis comes from a recent survey by the CLA, which represents rural land and business owners. It found that 44% of its members said they had experienced a reduction in the availability of migrant labour over the past year.

“Farmers and growers need to know how the government will deal with the need from industries that rely on seasonal workers,” said Ali Capper, chair of the NFU horticulture board. “The NFU is calling for reassurance that farmers will be able to source a reliable and competent workforce both now and in the future. Without that, this trend is likely to continue and will hit hard.”

A previous seasonal agricultural workers scheme was scrapped by Theresa May when she was home secretary, after Bulgaria and Romania joined the EU. Farmers have been calling for a new scheme since the Brexit vote.

But in March, the immigration minister, Robert Goodwill, told MPs: “We do not believe there is sufficient evidence to justify a seasonal agricultural workers’ scheme in 2017.” The NFU’s concerns about food rotting in the fields due to a lack of workers were “a little bit of a scare story”, he said.

A government spokesperson said: “We are determined to get the best deal for the UK in our negotiations to leave the EU, not least for our world-leading food and farming industry, which is a key part of our nation’s economic success.”

Hardman said people who thought the shortage of farm labour could be filled by UK workers were “delusional”. He said: “There is no appetite in the UK labour pool for seasonal agricultural work.” The hospitality industry was more attractive for temporary work and unemployment is low in key areas, like Kent, he said.

Hardman said he understood why the government has not committed to a new migrant worker scheme: “Firstly, it’s political suicide and secondly they are working on hard historical data. Eighteen months ago we did not have an issue, but the tide of shortages is on its way.”

Another report on Thursday, from the industry group British Summer Fruits, estimates that strawberry and raspberry prices will jump 35%-50% if British growers do not have access to seasonal overseas workers after Brexit.

In August 2016, some of the Britain’s biggest fruit and vegetable growers said that without seasonal migrant workers they would switch their production to other countries, where many already have large operations, or would switch to cereals which are harvested by machines.