Around that time, political cartoons of Inoue drawn with a deformed crab-like face frequently appeared in newspapers, and a rumor spread that posting a slip of paper with the phrase 'Mr. Kakugoro INOUE is staying here' on one's house entrance could prevent smallpox; such treatment ensured he was one of the most well-known men of the age.
この頃の新聞などに、蟹の顔をした井上の漫画がよく書かれたり、天然痘が流行した頃、家の入口に「井上角五郎様御宿」と書いた紙を貼れば、天然痘除けになるという噂が広るなどこの時代の名物男であった。