If a gay person is traveling and a hotel or B&B refuses to accommodate them, it's harmful discrimination.   It can kill.  If a doctor refuses care because you are gay causes real harm.   An exterminator who allows a gay couple to live in filth but not a straight couple is discriminating.  When you open a business, you agree to follow the rule not to discriminate.

"Refusing to associate" is different from discriminating.  Religious people don't have the right to use their religion as an excuse to harm others who only want to do business with them.  Making a cake is a baker's job.  Marching in a pride parade or having a beer with someone is "associating".  Just  to be clear that discrimination, not refusing to associate with them, is illegal.