The same way you turn down any other invitation... "Thanks for asking, but I think I'll pass" You don't owe any explanation,and in this case, explaining your reason might be taken as a comment on their taste, and not a very nice comment, even though you don't mean it that way.