However, soon after Yoshitaka succeeded the headship of the family, their stronghold, Hamamatsu-jo Castle, was surrendered to Ujichika IMAGAWA on September 4, 1517, and the deputies both died in the battle.
しかし、義堯が家督を継いで間もない永正14年（1517年）8月19日、今川氏親により拠点である浜松城を落とされ揃って戦死。