The OED reflects usage, i.e. this is the meaning attributed to the term by the Left. It uses our horror of "irrational fears" as a way of discrediting, dismissing and dismissing the expression of what is actually intense dislike, or disapproval, or fear of Islam, "esp. as a political force", by attributing it to "prejudice" and "irrationality", rather than rebutting  the actual arguments or dealing with the conditions that provoke the fears.