However, the Oe clan family, who in turn was appointed as the shokan of the manor also came into conflict with the Todai-ji Temple by not paying the annual tributes, called themselves kugonin (people serving the Imperial Palace and offering food and crafts to the Emperor) to try to directly connect with the Imperial Palace, and managed to practically rule Kuroda no sho by uniting with the shugo and gokenin of Iga Province who were supposed to suppress them.
しかし、代わって同荘荘官職についた大江氏一族もまた、年貢納入を行わないなど東大寺との対立を深め、供御人と称して朝廷と直接結ぼうとし、これを鎮圧するはずの伊賀国守護、同御家人らと結んで、黒田荘を実質的に支配するに至った。