With most results declared, Mr Iohannis, mayor of Sibiu, had 54.5% of the vote to Mr Ponta's 45.5%.
Mr Iohannis, 55 and an ethnic German, vowed to change politics and said "another kind of Romania is beginning".
Romania is one of the most corrupt EU states, something the centre-right Mr Iohannis has vowed to tackle.
Despite the election result, Mr Ponta told a local TV channel that he had "no reason to resign" as prime minister.
He had hoped to become the country's youngest president, replacing the incumbent Traian Basescu who cannot stand for re-election after serving two terms.
Klaus Iohannis - Romania's president-elect
Profile of Klaus Iohannis
Romania's large diaspora of up to four million people played a key role in the election. Many expat voters were said to be disillusioned with Mr Ponta.
After the first round of voting there were protests at polling stations in Paris, London and other cities. In some places voters had to queue for hours - with some unable to vote - leading to the resignation of the foreign minister last week.
The number of expatriate voters on Sunday more than doubled to 379,000 and large queues thronged polling stations at embassies and consulates across Europe, from Milan and Munich to London and Portsmouth.
Police in Paris fired tear gas on Sunday evening to disperse voters angry that they had been unable to cast their ballots. The new foreign minister had suggested that voters in France should travel instead to the eastern city of Nancy.
Romanian media praise election result
Expatriate voters overshadow presidential poll
Mr Ponta had been leading in the opinion polls and had beaten Mr Iohannis, the mayor of Sibiu in Transylvania, in the first round of the presidential election.
"We are a democratic country," Mr Ponta said outside the headquarters of his Social Democratic Party on Sunday. "The people are always right."
Mr Ponta, 42, had promised to reduce the budget deficit, increase pensions and the minimum wage.
As prime minister, he oversaw economic growth and political stability in Romania, the EU's second-poorest state after Bulgaria.
Aside from tackling corruption, Mr Iohannis, 55, promised in his election campaign to strengthen the independence of the judicial system.
Opposition candidate Klaus Iohannis has won a surprise victory in Romania's presidential election, defeating PM Victor Ponta after a tight race.