His inconstancy between the Minamoto clan and the Taira clan caused Yoritomo's distrust, and he was granted a reward far less than those of the Shoni clan, the Otomo clan and the Shimazu clan, which were appointed Shugo (provincial governors), and on the contrary, Takanao came to be checked by the gokenin from Kanto provinces, who were placed around his home base.
源平の間を揺れ動いたことで頼朝の疑念を招き、隆直への恩賞は守護に任じられた少弐氏や大友氏・島津氏に遠く及ばず、逆に多くの関東系御家人を本拠地周囲に配置され、その牽制を受けた。