The eurozone is expected to grow by 1.6% this year and 1.4% in 2017. Before the referendum the IMF had predicted growth of 1.7% for both years.
The IMF also revised down its 2018 growth forecast to 1.6% from 1.7%.
It said medium-term growth prospects for the 19-member bloc were "mediocre" due to high unemployment and debt.
Mahmood Pradhan, deputy director of the IMF's European Department, said the outlook could worsen if drawn-out negotiations between the UK and the EU led to a continuation of recent trends in financial markets - where investors have shunned riskier assets.
"If that risk aversion is prolonged, we think the growth impact could be larger and at this point, it is very difficult to tell how long that period lasts," he said in a conference call.
The revised 2017 figure was the IMF's "best case" scenario, assuming a deal was struck that allowed the UK to retain its access to the EU's single market, Mr Pradhan said.
However, if the UK decided not to maintain close ties with the EU and chose to rely on World Trade Organization rules, there could be "major disruptions," he said.
Mr Pradhan added it was "very, very early days to have any strong sense of confidence" about what the eventual relationship between the UK and EU would be.
In the medium-term, challenges such as high unemployment and persistent structural weaknesses in the euro area would continue to weigh on growth, the IMF said.
"As a result, growth five years ahead is expected to be about 1.5%, with headline inflation reaching only 1.7%," the report said.
It also said that as the euro area was such a big player in world trade, any slowdown could have an impact on other economies, including emerging markets, but it expected this to be "limited".
The International Monetary Fund has cuts its economic growth forecasts for the eurozone in the wake of the UK's vote to leave the European Union.