It was destroyed by fire in 1196, and went into a drastic decline during the Middle Ages so that, in the Edo period, only a temporary hall remained.
建久7年（1196年）の火災による焼失後、中世以降の衰退は著しく江戸時代には仮堂一宇を残すのみであった。