In January 1186, on the occasion of Emperor Gotoba's Imperial visit for a Katatagae gyoko (to stay the night in a place situated at another point of the compass when one's destination from home lies in an ill-starred direction), as Hachijoin's residence had been seriously damaged by an earthquake and the repairs had not been completed, Yorimori's residence adjacent to it was chosen; however, Yorimori flatly declined to allow the use of his residence, stating that there were impurities there (see the entries for January 5, 6 and 7 in the "Gyokuyo").
文治2年（1186年）正月に後鳥羽天皇の方違行幸が行われた時、八条院の邸宅が地震により破損が著しく修復も済んでいなかったため、隣の頼盛邸に白羽の矢が立ったが頼盛は「家に穢れがある」と称して固辞した（『玉葉』正月5、6、7日条）。