Iran had agreed to provide information to help allay concerns about a military dimension to its programme, but had failed to do so, the IAEA reported.
These include suspected work on detonators which can set off nuclear bombs and high explosive charges.
It comes weeks before a deadline for Iran and world powers to reach a final deal to resolve the nuclear issue.
The US, EU and other powers suspect Iran is secretly seeking to develop nuclear weapons. Iran has strongly denied the charge, insisting its programme is purely for peaceful purposes.
Since the election of President Hassan Rouhani last year, Iran has promised to further co-operate with the IAEA.
Last November, Iran and the IAEA reached an agreement which outlined how Iran would co-operate with the agency's investigations into Iran's past nuclear activities and address the agency's concerns.
In an IAEA report released on Friday, the agency said that some of five steps that were meant to be completed by 25 August had been missed.
It also said more activity had been conducted at a military base in Parchin, where research on developing nuclear weapons is alleged to have taken place.
"These activities are likely to have further undermined the agency's ability to conduct effective verification," the AFP news agency quotes the report as saying.
The IAEA negotiations are separate from a parallel diplomatic process between Iran and the five permanent members of the UN Security Council - the US, Russia, China, Britain and France - plus Germany, known at the 5+1 group.
Two months ago, they agreed to extend the deadline from 20 July to 24 November to reach a comprehensive agreement under which Iran would curb its nuclear activities in exchange for the easing of economic sanctions.
US and Iranian officials are currently meeting in Geneva to hammer out a deal.
Iran has missed a deadline to answer questions about its nuclear programme, the IAEA global nuclear watchdog says.