It found the total spending in 2015 on agency midwives, overtime and the NHS's flexible "bank" midwives was £72.7m.
The RCM described it as an "incredibly expensive and wasteful way to staff maternity units".
An NHS spokesman said it was committed to cutting agency staff costs so patients "get the right care".
Some 123 NHS trusts in England with maternity units responded to the RCM's Freedom of Information requests, which revealed that 46 had used agency midwives last year to fill gaps on hospital rotas.
In 2015 the government introduced a cap on NHS trusts' spending on agency staff, but the figures show that for many trusts some costs continue to rise.
The £25m spent on agency midwives represented a 40% increase compared with 2014, and a 146% increase over four years.
The report also said that on average agency midwives were paid £41 an hour, about half of which went to the agency.
Additionally, overtime pay for NHS midwives cost an average of £23 per hour, while bank staff - in-house workers who want to work flexibly across health trusts - cost £25.63 per hour on average.
Jon Skewes, from the RCM, said spending on this scale "simply cannot continue".
He went on: "An over-reliance on temporary staff is clearly more expensive than employing the correct number of permanent staff and needs to be corrected sooner rather than later."
But a spokesman from NHS Improvement - which combines a number of health monitoring and improvement bodies - said trusts "have made good progress" and saved over £600m on agency staff since last year.
"We are committed to helping the NHS cut the cost of agency midwives and all agency staff, so that patients get the right care, from the right staff, at the right time," he added.
The overall budget for NHS England in 2015-16 year was £101.3bn.
The NHS in England spent £25m on agency midwives last year - more than double the figure for 2013, the Royal College of Midwives has reported.