The disturbance broke out at Liverpool's Titanic Hotel during Friday night's event, which was hosted by mayor Joe Anderson.
The group, who broke a window and kicked doors, are understood to have been protesting against Mr Anderson.
A spokeswoman for Love Activists said they regretted any upset caused.
Stuart Fergus, married to James's mother Denise, called the protestors "vile".
Mr Fergus said one member of the group used a rolled-up banner bearing James's image as a battering ram during the incident.
The toddler's mother was left in tears as about 40 protestors from the group were moved away from the building by security staff.
Mr Fergus said: "We heard them say 'get Joe out' and that was when we realised what it was all about.
"But this wasn't a political night. There was nothing political about it at all, so we are surprised.
"Denise adores the people of Merseyside, who have always stood by her and helped her in James's name.
"But to have a small group of people behave like this...whatever they say they were doing, they must have known what the event was for."
Merseyside Police said about 12 members of the group managed to enter the reception area of the hotel, causing damage to "a number of items".
A spokeswoman for Love Activists said the protest was peaceful and claimed members of the group were assaulted by security staff.
"As far as we are concerned, both prior to the event and at the event itself, there was no indication that the event was anything to do with James Bulger," she said.
"Of course we would not want to offend the name of James Bulger."
Asked if the group would like to apologise to the toddler's family, the spokeswoman said: "We regret any upset caused."
The family of murdered toddler James Bulger have condemned protesters who broke into a hotel during a charity dinner in memory of the youngster.