The relationship between Emperor Kotoku and Naka no Oe no Oji became inharmonious. An incident happened where Naka no Oe no Oji left Naniwanomiya Palace and returned to Asuka, with a large number of the retainers following him and Emperor Kotoku being left in complete isolation.
孝徳天皇と中大兄皇子は不和となり、白雉4年（653年）に中大兄皇子が難波宮を引き払って飛鳥へ戻り、群臣もこれに従い、孝徳天皇は全く孤立した。