A referendum on whether the UK should stay in the EU will be held on 23 June.
If the UK votes to leave there will be an initial two years in which a new trade deal with the EU can be struck.
A report by Davy stockbrokers suggests that even if a deal is done there could be "non-tariff barriers".
Those could include regulatory differences and customs checks.
The report says that as the Republic of Ireland accounts for a third of Northern Ireland's goods exports non-tariff barriers "could be particularly costly" for Northern Ireland.
It cites Norway as an example, which has access to the UK single market but still faces rules of origin requirements and anti-dumping duties.
The report, written by economist Conall Mac Coille, takes the view that the UK would ultimately maintain its membership of the single market with a "fudge" allowing some autonomy on migration.
It says the worst case scenario is the failure to secure a free trade deal, which would mean UK exporters would have to pay tariffs on products going to the EU.
It describes the EU tariffs on agricultural products as "exceptionally high".
It also emphasises that the UK and the Republic of Ireland could not do a bilateral trade deal as Ireland would be bound by wider EU policy.
Proponents of leaving the EU say it would leave the UK free to make free trade deals with fast growing economies around the world.
One pro-leave economist, Dr Graham Gudgin, has said leaving the EU would not have "much effect" on the Northern Ireland economy.
He said studies that have been done so far suggest the impacts on the UK economy would be in the range of plus or minus 2% of gross domestic product.
He described that impact as "pretty small" and "within the margin of error".
Northern Ireland could face barriers to trade with the Republic of Ireland even with the agreement of a free trade deal in the wake of a UK exit from the European Union, a new report suggests.