After the death of Yoshimitsu, the Kitayama villa became Rokuon-ji Temple (the Kinkaku-ji Temple), and then in 1422, the it became a Zen temple, but a large part of the major buildings were reconstructed in a new location, and at the time, the garden seemed suffered considerable damage, it is said that broken garden stones were left there for a long time.
義満の死後、鹿苑寺（金閣寺）となり1422年には禅寺になっていくが、主要な建築物の大部分は移築されその際に庭園も相当な影響をうけたらしく破壊された庭石は長らく放置されていたという。