Many of the arguments of pseudo-Christians in support of race-based separation, segregation and white supremacy are based on the so-called "Curse of Ham" (Gen. 9:20-27).  The Wikipedia article provides a decent review of the relevant issues.

<<I think a modern society has to have some rules about how far someone with strongly held religious beliefs can go in expressing those beliefs when that impinges on the freedoms of others.>>
Absolutely! Religious faith cannot be a trump card (pun acknowledged but not intended) that permits the abuse or harm of another person, or the denial of their rights.