It was not the Pharisees, but His own disciples who expressed that view.
And Jesus replied: "It is not everyone who can accept what I have said, but only those to whom it is granted."
His disciples had also misunderstood.