The heavy punishments implemented through Ansei no Taigoku were detested by many, such that Naosuke came to be called 'Ii no Akaoni' ('The Red Ogre of Ii'), like Naomasa II, founder of the Hikone Domain.
安政の大獄における過酷な処分は多くの人々から恨まれ、彦根藩祖である井伊直政同様に「井伊の赤鬼」といわれた。