The Vatican has tried to gain entry to a so-called "white list" of countries that are recognised globally as financially transparent.
The report said the Vatican's measures for tackling money laundering and financing of terrorism were inadequate.
However, the bank had "come a long way" in addressing financial transparency.
The report by Moneyval - the European body that vets banks - graded the secretive bank in 16 key areas.
The Vatican bank was found to be falling short on seven of them and given a negative grade.
Moneyval said the reporting of suspicious transactions was deficient and found the Vatican police to be ill equipped to deal with financial crime.
While offences such as financing terrorism had been criminalised by the Holy See - the central administration of the Vatican - methods used to prevent these crimes had not been implemented, Moneyval said.
The Moneyval report is not a qualification for entry on the "white list", but its approval influences a country's ability to gain international recognition for good financial practices.
The Vatican bank needs more reform in order to show it is effective at preventing financial crime, a report by a European banking watchdog has said.