The cause of the conflicts with Yoritomo is described in the article of August 17, 1184 of "Azuma Kagami," and according to this, he invoked the wrath of Yoritomo for having accepted the title from the Imperial court, without getting his brother's permission, and then he was excluded from the mission of pursuit on August 6 of the same year (old lunar calendar).
頼朝との対立の原因については、確かに、『吾妻鏡』元暦元年（1184年）八月十七日条には、同年8月6日_(旧暦)、兄の許可を得ることなく官位を受けたことで頼朝の怒りを買い、追討使を猶予されたと書かれている。