In 1964, the US Congress passed the 1964 Civil Rights Act. The US Supreme Court upheld it as a legitimate exercise of congressional power -- citing the Commerce Clause, I believe.
I bet that, after that SCOTUS ruling, whites opposed to serving either black people or interracial couples tried to use freedom of religion to justify their discrimination. And the US Supreme Court said "no".
Why did the current SCOTUS take this case? To create a "gay exception"?