In 1594, Hideyoshi began construction of Fushimi-jo Castle for his residence after his retirement, and in 1595, because of an adverse relationship with Hideyoshi, Hidetsugu was sent to Mt. Koya-san and then ordered to commit suicide, thereafter Jurakudai was demolished.
秀吉は隠居後の居所として1594年には伏見城の築城を始めるが、翌1595年に関係が悪化していた秀次を高野山に追放した後切腹を命じるに及んで、聚楽第も破却された。