"Non sequitur" in logic is an invalid argument where the conclusion could be either true or false (because there is a disconnect between the premises and the conclusion), but the argument nonetheless asserts the conclusion to be true and is thus fallacious.

"Your point is irrelevant whether it stands or not." is clearly not a non sequitur, it is an assertion that without regard to the poll numbers,  the President has made a decision.

If your point had to with minority views, "Fact: Most Americans are not against  distribution of contraceptives overseas." would have been appropriate rather than "aren't right-wing Catholics".