It said uncertainty about the referendum's result and "paralysis" in government contributed to the negative mood.
The FSB claimed increasing numbers of companies were planning job cuts.
Firms prepared to invest were down by almost a third compared to a year ago.
Mike Cherry, FSB chairman, said the increase in labour and administrative costs also played a role.
More than a thousand of his members were questioned for the survey between April and May.
He called on the government to ensure continued access to the single market in "whatever form that takes".
"We're calling, along with others, for spending on physical infrastructure particularly in the north, for the trans-Pennine route to be done, digital infrastructure to be advanced and the announcement last week to put air capacity on hold by holding back on Heathrow, on Gatwick or anywhere else frankly was not helpful," he added.
Mr Cherry said for the first time since 2009, the UK faced a real chance of recession.
"To head this off, we need to do everything we can to support small firms to grow, create jobs and weather the harsh economic headwinds."
Confidence among small firms was at a four-year low even before the EU referendum, according to a survey from the Federation of Small Businesses (FSB).