During that time, he held positions such as Sagami no kuni no suke (Assistant governor of Sagami Province), but when the direct descendant of the Northern House of the Fujiwara clan instigated 'Anna Incident' in 969 to expel other clans from the Imperial Court, he was hunted and captured by MINAMOTO no Mitsusue, who was a member of the opposing clan Seiwa-Genji (Minamoto clan), and was overthrown and sentenced to banishment to Oki Province.
この間、相模国介などを務めるが、安和2年（969年）に起きた藤原北家嫡流による他氏排斥事件「安和の変」で対立する清和源氏の一族・源満季に追捕されて失脚、隠岐国に流罪となった。