In 1857, he was appointed as the minister of the center, but in 1859 he had a conflict with Kanpaku (chief adviser to the Emperor) Hisatada KUJO over the Imperial consent on the Treaty of Amity and Commerce between the United States and Japan, and on April 30, 1858, together with the Minister of the Left Tadahiro KONOE, he was ordered to refrain from entering the Imperial Palace.
1857年（安政4年）内大臣となるが、1859年（安政5年）には日米修好通商条約への勅許を巡り関白九条尚忠と対立して、3月7日には左大臣近衛忠煕とともに参内停止を命じられる。