However, he lost in the battle against the local lords including the Daijo clan, which originally gripped strong influences in the province, and punished by Imperial order, he was exiled to Ichikawa no sho estate, Kai Province, with his first son Kiyomitsu.
しかし同国内に元来強い勢力を持つ大掾氏らと争って敗れ、勅勘を蒙って長男清光とともに甲斐国市河荘に配流となった。