Surely there are limits on free expression.  The question is where we draw that line.  If this group attended the memorial wearing swastika armbands, and holding Nazi flag and proceeded to tear Jewish and Islamic prayer books and the memorial, would that be ok?
When this group walked in to the library, were others frightened.  Did knowing this group and their history prevent others from comfortable using the library that evening.  These must all be taken into consideration.