Apparently Yoshiie retired to Toishi-jo Castle in 1531 following the death in battle of lord Muramune, when defeated at the Battle of Daimotsu by the combined forces of Harumoto HOSOKAWA and Motonaga MIYOSHI.
享禄4年（1531年）、主君の村宗が大物崩れで細川晴元と三好元長の連合軍に敗れて戦死すると、それを機に砥石城で隠居した。