However, this provoked rage from those who advocated restoring power to the emperor, because the bakufu government still tried to use the emperor to fortify the power of the government, and roju Ando was attacked outside Sakashita-mon Gate, got severely injured, and eventually lost his power (the Sakashitamongai incident).
しかし、幕権強化のために朝廷を利用することは尊王派の怒りを買い、文久2年（1862年）正月、老中安藤は江戸城坂下門外で襲撃され、一命は取り留めたが後に失脚した（坂下門外の変）。