He was elevated to the title of hakushaku (count) in 1884 because of his achievements since the Restoration, and in the following year, with the commencement of the cabinet system, left the cabinet to work as an imperial court councillor and later privy councillor where he educated the Emperor Taisho, Imperial Princess Masako and Imperial Princess Fusako.
明治17年（1884年）、維新以来の功績によって伯爵を授かり、翌年からは内閣制度開始とともに閣外に去って宮中顧問官、ついで枢密顧問官をつとめ、この間、大正天皇・恒久王妃昌子内親王・北白川房子の養育係主任でもあった。