It is said the reason was that Naoyasu was so loyal to Ieyasu that he became too strict with his vassals, and this worked against him.
その理由は直政が家康に対して忠義を篤くしすぎて、あまりにも家臣達に厳しくなってしまったことが裏目に出てしまったのではないかと言われている。