He has the first amendment right to refuse service to anyONE, not any GROUP*. That's the difference.

(*Except in cases where individuals belong to a group that poses a reasonable threat to the service provider, like a jewish baker refusing service to a neo-nazi.)