While Yasutsune soon returned to his previous post that time, he was subsequently removed from office for the third time and was also banished to Izu Province under suspicion of involving in the rebellion by MINAMOTO no Yoshitsune and MINAMOTO no Yukiie in 1185.
このたびもまもなく復帰するが、今度は文治元年（1185年）源義経・源行家の謀叛への係累を疑われて、三度目の解官、さらには伊豆国への流罪となった。