In a report on literacy standards of 11 to 14-year-olds, education watchdog Estyn blamed a lack of time and support for schools.
The provision to develop pupils' literacy skills in a majority of schools was only "adequate".
Education Minister Huw Lewis said the policy was having an impact.
The Literacy and Numeracy Framework (LNF) was introduced in 2013 for pupils aged five to 14 following concerns they were falling behind.
It aimed to improve how pupils read, write and express themselves in all subjects.
The LNF includes annual tests to see how they are progressing.
But two years after its introduction, the report said the LNF was not producing results.
Standards in general were judged excellent in about one in nine secondary schools, inspectors found.
They were good in close to two in five, which is similar to the previous two years.
"The LNF was introduced quickly, but progress in implementing the framework has been slower than expected," Estyn said.
The watchdog noted:
Inspectors have said the Welsh government should "provide guidance and support" for teachers to help them implement the LNF and "develop literacy skills across the curriculum".
Last week, a study conducted by the Association of Teachers and Lecturers Cymru found almost two-thirds of teachers in Wales think the national reading and numeracy tests are "not fit for purpose".
Fewer than half (49.2%) of those questioned said all their pupils were able to access the tests and only 37% said they were fit for purpose.
Amid concerns blanket test papers are not suitable for younger children, the union said "disadvantaged learners" were among those finding them most difficult to use.
The Welsh government said developing the literacy skills of young people remained "a critical issue for Welsh education and a key government priority" and there was "still much to do".
Mr Lewis told BBC Wales it was a "profound shift" and he said he would not apologise for reform taking place with a sense of urgency.
"Estyn is talking about events 12 months ago and a very great deal has happened since then," he said.
"I'm confident we've moved a great distance."
A key policy to drive up standards in numeracy and literacy in Wales has made "modest" progress, partly due to a lack of support from the Welsh government.