However, in 807, during the era of Emperor Heijo, FUJIWARA no Otomo (son of the Korekimi, or Dainagon (chief councilor of state)) and FUJIWARA no otoei/Tadatoshi (son of the Tsugutada, or Chunagon (vice-councilor of state)), who were in the second and third positions of the political power at the time, fell from power as a result of Iyo Shinno no Hen (the Conspiracy of Imperial Prince Iyo), whereby the family lines of Toyonari and Otomaro declined as the central nobility, followed by the fall of Nakamaro.
しかしながら、続く平城天皇朝の大同2年（807年）に当時政権№2,3の座にあった、藤原雄友（是公の子：大納言）・藤原乙叡（継縄の子：中納言）が伊予親王の変により失脚し、仲麻呂に続き、豊成・乙麻呂の系統も中央貴族としては衰退した。