He was appointed to Gon chunagon (Provisional vice-councilor) in 1858, and trusted, however, he was obliged to leave the capital to Choshu in a conspiracy by those who advocated kobu-gattai (integration of the imperial court and the shogunate), and thereby became a man of so-called Shichikyo ochi (the exile of the Seven nobles from Kyoto) in 1863.
安政5年（1858年）、権中納言となり重んじられるが、文久3年（1863年）公武合体派の策略により、三条実美らと長州へ下向、いわゆる七卿落ちの一人となる。