Even after that, he served as a vassal close aide of the retired emperor Goshirakawa, but was dismissed from his post and arrested for the reason that he cursed the emperor in 1161, and was transported to the Suo Province in the following year.
その後も後白河院の近臣として仕えるが、院と二条天皇との対立が深まる中で、応保元年（1161年）に天皇を呪詛したとして解官・逮捕され、翌年周防国へと配流された。