Mark Drakeford said the devolved administrations had "anxieties" about the joint ministerial committee (JMC) set up by the UK government.
It met on Thursday for the first time since Theresa May said the UK should leave the European single market.
Welsh Secretary Alun Cairns said there was "common ground" on free trade.
The UK government's Brexit Secretary, David Davis, said: "As the prime minister made clear in her speech, Brexit is a chance for us to build a stronger UK and strengthen the Union between our four nations.
"It is only by us coming together that we can fully grasp the opportunities that our departure presents.
"We will use the latest thinking and analysis shared at today's meeting to inform our work to deliver an exit that works for the whole of the UK."
Mr Drakeford said Wales would continue to push for "full and unfettered" access to the single market, and wanted to be sure the forum was "having an impact" on the way the UK government was shaping its thinking on Brexit negotiations with the remaining EU member states.
"This is meant to be a fundamental forum," said Mr Drakeford.
"It was mentioned by the prime minister in her speech and I feel we need to be given confidence by the UK government that we are not simply here to express our views but that those views go on and make a difference."
While Sinn Fein said its ministers in the Northern Ireland executive may walk away from the JMC process, Mr Drakeford said the Welsh Government would continue to engage with it to make sure Welsh interests were "fully and significantly expressed".
Mr Cairns said it had been a "constructive" gathering, adding: "Even the Welsh Government accepted today that the language that the prime minister used earlier this week about a free trade agreement with the European Union is not inconsistent with the language that they have talked about.
"So this demonstrates that there is a lot of common ground but I'm not underestimating either the challenges that are there."
The meeting - described by Mr Drakeford as "robust" - discussed papers from the Scottish government and Northern Ireland executive.
He said a Welsh Government paper would be published on Monday and discussed at the next meeting of the JMC in February.
Ministers in Wales need to be given confidence their views on Brexit are making a difference, the finance secretary has said.