However, the darkness between Emperor Montoku and Yoshifusa continued, and it is said while Emperor Montoku was in power he moved to the togu (a palace for a crown prince) located at the end of the palace, or Reizen-in where the Retired Emperor Saga's palace was located far from the palace; he never lived or spent time in the main building of the palace because Yoshifusa had put political pressure on him.
だが、その後も天皇と良房の暗闘は続き、良房の圧力の前に在位中には内裏の外れにある東宮や嵯峨上皇の離宮だった冷然院などに居住して、遂に一度も内裏正殿を居住の間として生活を送ることはなかったという。