However, his daughter who was the wife of Yoshimoto IMAGAWA passed away in 1550 and Nobutora hesitated to continue living in Suruga, and decided to leave.
しかし天文19年（1550年）には今川義元の室になっている娘が他界し、今川家に居づらくなってしまい駿河を出る事になる。