You say:  "Those grounds of divorce would apply in many marital situations."

Precisely.  

But Jesus did not give us a rule that provided several grounds for divorce.  Knowing well that there are many marital situations where divorce would seem permissible, he still allowed only one --- adultery.  Along comes Paul, and widens the rule written by God.  By what authority?  Paul was a great saint, but he was not God.  Paul gave us a more palatable rule, but he was not God.