He lost his eldest son Saburo (his real son) early, his sons Tsunashige and Nagatoshi in the battle of Kanbara-jo Castle in Suruga Province (Shizuoka Prefecture) against Shingen TAKEDA in 1569, therefore, he adopted Kagetora UESUGI, the seventh son of Ujiyasu HOJO, giving him the reigns of the family, and retired, calling himself Genan Sotetsu.
長男の三郎（実子）は夭折し、1569年に武田信玄との駿河国（静岡県）の蒲原城の戦いにおいて子の綱重・長順を失い、北条氏康の7男上杉景虎を養子に迎えて家督を譲り、隠居して幻庵宗哲と号した。