In 1158, Yoritomo was appointed to Kogo no miya gon no shoshin (Junior Office Secretary of the Empress's Household) serving Imperial Princess Muneko who became Kisai no miya (Empress) as Emperor Goshirakawa's junbo (a woman who was given the status equivalent to the Emperor's birth mother).
保元3年（1158年）には後白河天皇准母として皇后宮となった統子内親王に仕え皇后宮権少進に補任。