Mr Crabb, who represents the area as MP for Preseli Pembrokeshire, was talking to workers and their families at an open surgery.
He was joined by his assembly counterpart Paul Davies at the discussions in Milford Haven.
All but 60 of the plant's 400 jobs are set to go after a deal to save it from closure collapsed.
Speaking after the surgery, assembly member Mr Davies said: "Understandably people were very concerned about their futures and were questioning why the sale process had collapsed so late in the day.
"Naturally the people we met were gutted and clearly very concerned about their prospects.
"Of course, it would be very helpful to understand from the potential buyer, the Klesch Group, why the sale did not go through.
"I believe that the company has a moral obligation to explain the rationale behind the sale not proceeding given that our communities' expectations had been raised.
"The news is devastating not only for Murco employees, but also those that indirectly rely on the refinery.
"It's absolutely crucial that every support possible is now given to the people affected by this devastating news."
A task force will meet next week to discuss bringing new jobs to the area.
It will be led by Wales' Economy Minister Edwina Hart and comes after Mr Crabb said both the UK and Welsh governments will work to "minimise the significant economic impact the closure will have on our county".
Local councillor Huw George said the task force would be up and running next week and Jobcentre Plus would hold a jobs fair.
On Thursday, Plaid Cymru MEP Jill Evans called for European Union funding to be sought to help affected workers.
She said money should be available under the EU's Globalisation Adjustment Fund which has benefitted many workers in other European countries to retrain, look for new work or set up their own businesses.
The oil refinery has been up for sale for four years without finding a buyer.
The sale of the firm to the Swiss-based Klesch Group was called off on Tuesday night.
In addition to its own workforce, the refinery employs a further 200 contractors and is understood to support a further 4,200 jobs across south west Wales.
Welsh Secretary Stephen Crabb has met with workers hit by the closure of the Murco oil refinery in Pembrokeshire.