Although he served in important positions of the new government, such as Sanyo (councilor), chairman of the Council of the Left and Sangi (councilor), he left the new government with Taisuke ITAGAKI and Takamori SAIGO after the defeat in Seikanron (debate on subjugation of Korea) of 1873.
新政府では参与、左院議長、参議などの要職に就くが、明治6年（1873年）の征韓論に敗れ、板垣退助・西郷隆盛らと共に下野する。