In the Eisho era (1504 - 1521), Nobutaka made a member of the clan Takanao UENO enter Kimurayama-jo Castle to make him the lord and appointed his son Yorihisa UENO to the lord of Bicchu Matsuyama-jo Castle, and he himself went back to Kyoto and served in the Muromachi bakufu again.
永正年間に、信孝は鬼邑山城を一門の上野高直を入れて城主となし、子の上野頼久をして備中松山城主に封じ自らは帰洛し再び室町幕府に近侍した。