It seems to me that the Church latched on to just one of the 'sayings' about 'marriage' and made that the absolute rule, without considering other sayings.

What did Jesus actually say about divorce? Matthew's version?  Matthew 19:9-10 "whoever divorces his wife, except for sexual immorality, and marries another, commits adultery.” or Mark's, no allowance for exceptions. 
Or Paul's 1 Corinthians 7:15 (BBE)"But if the one who is not a Christian has a desire to go away, let it be so: the brother or the sister in such a position is not forced to do one thing or the other: but it is God's pleasure that we may be at peace with one another."
 Is "God's pleasure that we may be at peace with one another."grounds for divorce?

And what exactly is this abhorrence of 'adultery' all about? Is that an unpardonable sin? Why?

If the church has this 'power to bind and loose' then the church's rules ought to look more to 'loosening' and 'forgiving' than to 'binding' and 'retaining'.