Vaughan Gething said "institutional abuse" found at Glan Clwyd Hospital's Tawel Fan unit was not typical of the NHS care received by patients.
Families said patients at Tawel Fan were treated like animals in a zoo.
But a north Wales AM said people lacked confidence that Betsi Cadwaladr Health Board was being brought to account.
The health board has apologised for the "inexcusable and unacceptable" treatment.
Meanwhile, the Conservatives have repeated their call for an inquiry into the NHS in Wales.
Welsh Tory leader Andrew RT Davies told BBC Sunday Politics Wales there had been "a long line of failures on behalf of Betsi Cadwaladr health board".
He said: "It's important now that Welsh government, instead of wringing their hands, actually step in and take decisive action to address these shortfalls."
'It was probably a mistake keeping him in there'
Asked if the findings were damaging for Labour, Mr Gething said: "I think that families affected by what happened in Tawel Fan will take real exception to people trying to use what happened there as somehow a party political failing or a party political triumph on one side or another.
"This is about families who've been let down by a service that could and should have served them and their loved ones properly and in this case it did not.
"That stands in stark contrast to the regular pattern of care that people receive day in and day out right across Wales.
"I think any minister in this department of any political party would be appalled as I am as to what happened and would have a clear expectation that the health board properly address in a timely manner the failings that have been revealed and laid bare for the wider public to see."
Liberal Democrat AM for north Wales Aled Roberts said people in the region "were getting pretty alarmed with regard to the regularity of these reports with regard to failings within our health system up here".
He added: "I don't think at the moment the people of north Wales have sufficient confidence that the health inspectorate and the Welsh government in particular are actually holding this health board to account."
The deputy health minister has warned opponents not to treat a damning report into a Denbighshire mental health ward as a "party political triumph".