Because of this, he was oppressed by FUJIWARA no Yorinaga, who held the title of Sadaijin (minister of the left), and was removed as azukaridokoro (a deputy of "Shoen" manor lord) in the manor of sekke where Nobunori was designated (because Yorinaga was Toshi choja (head of the Fujiwara clan) on this occasion).
このため、左大臣藤原頼長から圧迫を受け、信範が任じられていた摂家の荘園の預所を解任されている（当時、頼長が藤氏長者であったため）。