Estyn's annual report urged head teachers to "focus on the basics" and take a fresh look at teaching methods.
It said economic deprivation was "not a reason for lower standards" as many of the best schools were in poor areas.
Chief inspector Meilyr Rowlands called on teachers and leaders to be "imaginative in the classroom" and "share and learn from the best".
A growing number of secondary schools showed some excellence, inspectors said, up from 23% in 2010 to 38% in 2015.
For primary schools, that figure rose from 8% to 18%.
However, the report said the proportion of schools judged good or better had "not changed much".
Struggling schools "need to be identified and supported as early as possible", it added, calling on the relevant councils and agencies to "intervene quickly where necessary".
A Welsh government spokeswoman said: "We remain committed to ensuring the delivery of quality of provision for all our learners, wherever in Wales they are, and we are working closely with the profession to make this a reality."
The gap between the best schools and those that are struggling is still too wide, inspectors have said.