Lord Mervyn King predicted the long-term effect of a Brexit on GDP would turn out to be "a bit of a fuss about nothing".
In a BBC interview, he also accused the Remain campaign of treating people considering voting Leave like "idiots".
He said voters had not been impressed by "scaremongering tactics".
"If you say to someone 'you're an idiot if you don't agree with me' you're not likely to bring them in your direction," he added.
UK shares remain volatile in the wake of Thursday's vote, with some stocks temporarily suspended.
It came as sterling hit a 31-year low, and yields on 10-year government bonds sank below 1% for the first time.
Lord King, who was Bank of England governor for 10 years until 2013, said: "I don't think people should be particularly worried, markets move up, markets move down.
"We don't yet know where they will find their level and the whole aspect of volatility is that there is a trial and error process going on before markets discover what the right level of stock markets and exchange rates actually are.
"What we need is a bit of calm now, there's no reason for any of us to panic."
He said uncertainty would affect investment in the short term, but predicted the long-term impact to be "much smaller than either side pretend", predicting that "in 25 years' time we'll look back and say a little bit, that at least in economic terms, maybe that was a bit of a fuss about nothing".
Lord King's successor, Mark Carney, has said "extensive contingency planning" makes the UK "well-prepared" for the consequences of the vote to leave.
In a statement aimed a calming markets, Chancellor George Osborne said the UK was ready to face the future "from a position of strength".
But his predecessor, Labour's Alistair Darling, warned that a "vacuum" before a new prime minister is appointed could affect investment in the UK and said he was more worried now than during the financial crisis in 2008.
Volatility in UK shares following the vote to leave the EU is "no reason for any of us to panic", the former governor of the Bank of England says.