The reason why feelings alone CANNOT  be the basis for official action is simple. I am not responsible for your paranoia. 
Note that I am not suggesting that black fellow students of the poster in California are paranoid; they have objectively reasonable grounds to believe they are being threatened. However there are people, most notably American policemen,  who will feel threatened without reasonable grounds; their feelings are not and cannot be my responsibility, so a right not to feel threatened is meaningless.
This is not merely hypothetical, it's personal. I am quite pacifist by nature but I am very large; 195 cm., 150 kg. There are people who feel threatened by my simple presence in their vicinity. Do you believe that I violate those people's rights simply by being out in public.?