I would suggest that the ban on polygamy is rarely applied to "mainstream" society because mainstream society rarely practices polygamy. From time to time we hear of a man (it's always a man) who travels a lot on business and has two wives in different cities who don't know about each other, and those cases are prosecuted, but it is extremely unusual for a mainstream male to actually cohabit with two women, and unheard of for a mainstream woman to actually cohabit with two men.

Note that having both a wife and a mistress (or a husband and a lover) does not offend the polygamy ban as written, as long as only one of them is held out as a spouse.