In 1340, when Iinoya-jo Castle fell under the attack of KO no Moroyasu and Yoshinaga NIKI,he stayed in Teradomari, Echigo Province (currently Nagaoka City, Niigata Prefecture), or Hojozu, Ecchu Province (currently Imizu City, Toyama Prefecture); after that, he went to Okawara (currently Oshika-mura, Nagano Prefecture) at the invitation of Takamune KOSAKA (a member of the Mochizuki clan, who were a branch of the Shigeno clan) of Ina County, Shinano Province (currently Nagano Prefecture) in 1344.
1340年（暦応3年、興国元年）に足利方の高師泰・新木義長らに攻められて井伊谷城が落城した後、越後国（新潟県）の寺泊（現、新潟県長岡市）や、越中国（富山県の放生津（現、富山県射水市）などに滞在した後、1344年（興国5年/康永3年）に信濃国（長野県）伊那郡の豪族香坂高宗（滋野氏支流望月氏の一族）に招かれ、大河原（現、長野県大鹿村）に入った。