Welsh Tory leader Andrew RT Davies called Lesley Griffiths's comments "highly offensive".
They called into question whether Ms Griffiths was the right person for the job, he said.
Welsh ministers said she was referring to the "diverse range of skills needed to be successful in modern farming".
Ms Griffiths said her remarks were based on her early discussions with farmers after being appointed as the cabinet minister responsible for agriculture by First Minister Carwyn Jones following May's assembly election.
Answering questions in the Senedd on 22 June she said: "I also think we need to look at how we can help them [farmers] with the sustainability and the resilience of their businesses, and to look at the business side of it, because I think farmers, certainly in my very early discussions with them, are perhaps not the best people to run a business.
"They haven't that kind of business perspective also, and they want to work with us in relation to that.
"This is just some of the farmers I've spoken to early in the portfolio. I'm not saying all farmers at all."
Ms Griffiths will have one of her first big engagements in the job at the Royal Welsh Show later this month.
Mr Davies said Ms Griffiths should apologise to the farming community.
"To say that 'farmers are not the best people to run a business' is not only highly offensive but calls into question whether she is in fact the right person to take this brief forward.
"It simply betrays her ignorance of this vital artery of the Welsh economy."
Challenged with the official record of her comments in the chamber, on Tuesday, the first minister said: "That's not what she said."
Mr Davies said he was "dumb-founded that the first minister has refused to acknowledge the extremely offensive comments".
A Welsh government spokesman said: "These comments from plenary a fortnight ago have been taken completely out of context.
"The cabinet secretary was referring to the very diverse range of skills needed to be successful in modern farming."
The spokesman added that Ms Griffiths was "already forging positive relations with Wales' farming community and had hosted a "highly constructive round table discussion with key partners" on Tuesday.
The rural affairs secretary should apologise for saying farmers are "perhaps not the best people to run a business", the Conservatives have said.