Afterwards, he took charge of Hokumen no bushi (the Imperial Palace Guards for the North side) of Emperor Toba, taking over the elder brother's position, and when his own daughter (Tosa no Tsubone) became a retired Emperor's favorite concubine, he was taken up to be the retired Emperor's courtier.
その後、兄に代わり鳥羽天皇の北面武士を務めるが、娘（土佐局）が法皇の寵妃となったことを契機としてその近臣となる。