On the other hand, some views state that he caused a loss of morale and talent from the Shogun's retainers, as his methods through Ansei no Taigoku resulted in the expulsion of many Joi-ha (supporters of foreigner expulsion) as well as Kaimei-ha (supporters of enlightenment) bureaucrats, which led him to become a remote cause of the fall of the bakufu.
一方、その強権的な手腕で断行した安政の大獄では攘夷派のみならず開国を推し進めた開明派官僚まで大量に追放したため、幕臣からモラルや人材が失われ、幕府滅亡の遠因になったという見方もある。