But we as a society have not decided such a thing.  Only the group that thinks like you.  There are other groups who think other things.  For example the religious right would likely wish to ban speakers whose message promoted homosexuality.  Recall that popular vote banned gay marriage in many states where it would still be banned save for judicial activism.  Point is, your view is not the universal view, but as a liberal you think it should be.