Iranian Culture Minister Ali Jannati blamed "obstacles raised by the Saudis". Saudi Arabia blamed "unacceptable" Iranian conditions.
Hundreds of pilgrims died - many of them Iranians - in a stampede at the Muslim pilgrimage last year.
But the rivals are split on many other issues, with relations cut in January.
Mr Jannati said that "after two series of negotiations without any results because of obstacles raised by the Saudis, Iranian pilgrims will unfortunately not be able to take part in the Hajj" in September.
The Iranian Hajj Organisation said: "Saudi Arabia is opposing the absolute right of Iranians to go on the Hajj and is blocking the path leading to Allah."
It blamed the Saudis for failing to meet Iranian demands for "the security and respect" of Iranian pilgrims.
The gulf between them - Understanding the Saudi-Iranian rift
The Saudi-US-Iran triangle: How crisis reflects deeply fractured Middle East
Great rivalry explained: Why don't Iran and Saudi Arabia get along?
Sheikh Nimr al-Nimr: Who was leading Saudi Shia cleric?
Visa issues and flights to Saudi Arabia were thought to be key problems.
Some 60,000 Iranians attended last year.
The Saudi Hajj ministry insisted it had offered "many solutions" to Iran's demands during two days of talks that ended on Friday.
The Saudi foreign ministry said Iran's Hajj conditions were "unacceptable".
Islam requires devout Muslims to perform the Hajj pilgrimage at least once if they are able.
In January, Saudi Arabia broke off diplomatic ties with Iran, amid a row over the Saudi execution of a prominent Shia Muslim cleric.
The Saudis executed Sheikh Nimr al-Nimr and 46 others after they were convicted of terror-related offences.
In response, Iranian protesters stormed the Saudi embassy in Tehran, setting fire to the building.
Shia-dominated Iran and predominately Sunni Saudi Arabia are long-standing regional rivals who back opposing sides in the conflicts in Syrian and Yemen.
Iran and Saudi Arabia have failed to resolve a row over the Hajj pilgrimage and Iranian citizens will not travel to Mecca this year, Tehran says.