Due to the investiture of Prince Oi (Emperor Junnin) as Crown Prince supported by FUJIWARA no Nakamaro in 756, he was rather excluded from the government, and then demoted to Dazai gon no sochi (Provisional Governor-General of the Dazai-fu offices) for complicity in the Rebellion of TACHIBANA no Naramaro occurring the following year.
天平勝宝8年（756年）、大炊王（淳仁天皇）が弟藤原仲麻呂の後押しで立太子したことから、政権の外に押し出され気味となり、翌年の橘奈良麻呂の乱に連座して大宰権帥に落とされた。