He came back and served as Kanpaku (chief adviser to the Emperor) and Nairan (a preliminary inspection of official documents submitted from the Great Council of State to the Emperor) in 1862; he resigned as Kanpaku in the following year due to the emergence of Sonno Joi (19th century slogan advocating reverence for the Emperor and the expulsion of foreigners).
文久2年（1862年）に復帰して関白内覧を務めるが、翌年尊王攘夷派の台頭により関白を辞した。