According to Manyoshu (the oldest anthology of tanka), it can be inferred that he was once cast down because it was revealed that he had an extramarital affair with Imperial Princess Tajima no himemiko (FUJIWARA no Fuhito's niece) from the bloodline of Fujiwara clan, the emerging clan which were to supersede the Soga clan.
また、万葉集によれば、蘇我氏に代わって台頭する事となる、藤原氏の血を引く但馬皇女（藤原不比等の姪にあたる）との密通がばれ、一時左遷されていたと推測される。