He thereby accompanied Kazunomiya on the journey down to Edo in the following year, 1861, which angered a section of radical Sonno Joi ha (supporters of the doctrine of restoring the emperor and expelling the barbarians).
その経緯から翌文久元年（1861年）、和宮の江戸下向に随行するが、これが一部の過激な尊皇攘夷派からの憤激を生んだ。