After the death of Emperor Gosanjo, he continued his service as kinshu (an attendant) to Gosanjo's son, Emperor Shirakawa, and after Shirakawa abdicated, Toshiaki also enjoyed the confidence of the retired Emperor as Betto (chief administrator) of the Retired Emperor, and played an important role at the Imperial court with his sober character.
後三条天皇崩御後も、引き続きその息子の白河天皇の近習として重用され、退位後の上皇後院の別当としても厚く信頼され、また剛直な人柄は廟堂に重きをなした。