Worshippers had gathered at the Torbay Islamic Centre for early prayers at 03:30 when the men approached.
Imam Abdi from the centre said the "shocked" congregation "stayed inside" the building as the men were hurling abuse from outside.
Police are treating the incident, in which no-one was hurt, as a suspected hate crime.
Imam Abdul Kader Abdi said: "They were throwing stones at the mosque and shouting, 'we don't want Isis in here', and 'you know what happened in Manchester'."
The group, who had gathered in the mosque in Avenue Road, Torquay, began their early morning prayers, despite the ongoing attack.
"No-one went out of the mosque. They stayed for a while and after the prayers they left.", Imam Abdul said.
"We are very shocked, we didn't expect it, we haven't had this problem before."
He added: "Many people are traumatised and scared."
Devon and Cornwall Police said the three men were believed to be in their 20s, and said officers would be reviewing CCTV footage as part of the investigation.
Iman Atta OBE from the support group Tell MAMA (Measuring Anti-Muslim Attacks) said the smaller number of Muslim communities in the region made them more "visible" and "an easy target".
She added the organisation had seen a rise in reports of Islamophobia in the aftermath of the terrorist attacks on Westminster in March, and earlier this week in Manchester.
Ms Atta also said that the religious significance of the day, being the first day of the Muslim festival Ramadan, may have been a factor in the minds of the perpetrators.
Three men hurled stones and used abusive language to attack a congregation at a mosque.