Yoshiaki and his family left the Kinugasa-jo Castle, in which he resided, to join MINAMOTO no Yoritomo, when Yoritomo, the bereaved child of Yoshitomo, raised his army in 1180.
治承4年に義朝の遺児・源頼朝が挙兵すると、一族挙げてこれに合流しようと居城の衣笠城を出撃する。