Finlay Macleod of the Fishermen's Mission said all four reported missing had made it to their life raft.
But two of them tried to swim for the shore, which was in sight.
One of the four was found safe, while another is still missing.
Two of the four men on the raft were later found to have died, whilst another is still missing.
Mr Macleod told the BBC: "Two of them decided to swim to the shore, which they could see.
"And during that swimming, one of them drifted away from the other and he was lost."
He added: "The other one made it to the shore, and after clinging to the rocks for some considerable time he was rescued by the boat from the RNLI."
The boat, the Louisa, is understood to have taken on water in the early hours of yesterday morning, forcing the crew to abandon ship.
The leader of Western Isles Council, Angus Campbell,  said the entire community was grieving.
He said: "I think it's an absolutely tragic story, the loss of these fishermen. The communities right from the Butt (of Lewis) to Barra will be thinking of the families of these poor people.
"I know our thoughts and condolences will go out to the families. And I think everybody in the Western Isles will be mourning over this weekend. "
The police have confirmed that the search for the missing crew member has continued on Sunday.
RNLI and coastguard volunteers have been assisted by the Police Scotland helicopter.
Ch Insp Alastair Garrow said: "All of the partners are determined to do everything they can to help the family and the community deal with this tragedy.
"Police Scotland have an enquiry team working alongside other agencies and the known circumstances will be reported to the Procurator Fiscal. There will be further investigations necessary to fully establish what has happened."
Two of the men stranded in the sea after a fishing vessel sank off Mingulay in the Western Isles in the early hours of Saturday morning left their life raft to swim for shore.