Prof Michael Dougan said that it would mean "an enormous amount of paperwork" for Northern Ireland businesses.
The UK will have to decide if it will leave the customs union when it leaves the EU.
Remaining in the union would mean the UK could not negotiate its own trade agreements.
However leaving will mean that goods moving between the UK and the EU will be subject to customs procedures and probably tariffs.
That would mean the Irish border becoming a customs frontier which could have significant impacts on businesses trading across the border.
Professor Dougan, an expert in European law at University of Liverpool, said there would have to be some form of customs enforcement.
In evidence to the Northern Ireland Affairs Committee, Mr Dougan said that, aside from tariffs, the customs border is about "regulatory enforcement" for things like product standards.
"There does have to be a frontier, it does have to involve checks and does have to involve pre and post border formalities." he added.
His University of Liverpool colleague, Dr Stephanie Reynolds, explained that EU customs controls are based on an "electronic risk management system" which identifies "high risk" cargo before it reaches the border.
There has also been a move towards paperless customs with the vast majority of declarations being filed electronically.
A system called Authorised Economic Operator status also gives quicker access to certain simplified customs procedures and in some cases the right to 'fast-track' shipments.
She added that most inspections of cargos generally focus on "non-intrusive methods" such as x-rays.
They gave the example of the Norway/ Sweden border as an example of a "well-managed, fully co-operative" border.
Sweden is a member of the customs union but Norway is not.
BBC Northern Ireland has been to the main crossing between the two Nordic countries to see how traffic and business if affected and to see if there are any lessons for Northern Ireland.
There are reports across all platforms on this subject including The View on BBC One Northern Ireland at 22.45 GMT on Thursday.
The UK leaving the European customs union would not create an "iron curtain" at the Irish border a leading academic has said.