In July 1160, he was finally arrested together with Mitsumune for having attempted to kill Retired Emperor Goshirakawa, and was exiled to Satsuma Province.
永暦元年（1160年）6月、後白河の命を狙ったという罪状でついに光宗とともに逮捕され、薩摩国に配流となった。