He was assigned to Sashoben (third-ranked officer of the left) in 1165, but removed from office in the following year, 1166, and he had dark days for a long time since then.
長寛3年(1165年)に左少弁に任官するが、翌永万2年(1166年)に解官され、以後長年に渡って不遇の時を過ごす。