Later, Ujiharu was defeated by backlash of the bakufu (Japanese feudal government headed by a shogun), and moved to Sanuki province by boat with Kiyouji to attempt to raise an army again in 1362, but Kiyouji was killed in battle attacked by Yoriyuki HOSOKAWA who was ordered by the bakufu, and therefore, Ujiharu surrendered and returned to allegiance to the bakufu, and was appointed Awaji shugo again.
その後、幕府の巻き返しにあって敗北し、貞治元年/正平17年（1362年）に清氏と共に讃岐国に渡海して再挙を図ったが、ここで幕命を受けた細川頼之に攻撃されて清氏は戦死し、氏春は降伏して幕府に帰順し、淡路守護に再任された。