Carl Sargeant was updating the assembly on action being taken in Wales since the Grenfell Tower fire in west London.
He said the UK government's expert group may alternatively recommend that "full system" tests are done first.
Mr Sargeant said the decisions are "not political" and he would follow the advice of the UK government's experts.
He told AMs that he visited Swansea on Monday to meet tenants living in blocks which had failed fire safety tests.
Three blocks in Clyne Court, Sketty, and one block in Jeffreys Court, Penlan, became the first in Wales to fail tests on cladding which contains Aluminium Composite Material (ACM), carried out by the British Research Establishment (BRE).
Tests are underway on three high-rise blocks in Newport which also use ACM.    No other high-rise social housing in Wales is thought to use that material.
Mr Sargeant said he was taking advice from the independent advisory group set up by the UK Government's Department for Communities and Local Government (DCLG).
"These are professionals and I don't think this is a political decision," he said.
"This should be based on evidence and advice.
"If they suggest the removal of cladding is the best option then we will seek to remove that cladding and we'll talk about the implications on finances after that."
Mr Sargeant added that if the group recommends "full system" testing of the buildings, he would support that too.
That would involve testing a building's overall fire safety, rather than just the combustibility of its cladding.
"I think that gives actually a more true result on the effects of fire safety on a building when you have a whole system test," he said.
"But if they don't recommend that, they recommend something else, that route I will take also.
"I will not be making a decision other than what the recommendations are from DCLG and the advisory group."
Meanwhile the deputy leader of Swansea council said a decision on what to do about tower blocks which have failed fire safety tests needs to be made "quickly".
"We are still confident that we meet current building regs standards and our blocks are safe but if there does need to be further testing it should be a full system test as we have stated all along,"  Clive Lloyd said.
Cladding that has failed fire safety tests will be removed from tower blocks if the UK Government recommends it, the communities secretary has said.