However such energetic activity gradually became a load for sickly Yoshiyuki, and he resigned from Shugoshiki of the three provinces including Tanba in 1381 and left to Hoki Province.
しかし、これらの活躍は病弱な義幸の体には徐々に負担となり、永徳元年（1381年）には丹波を始めとする3ヵ国の守護職を辞めて伯耆国に下向した。