It is said that the reason of his downfall was 'because he sent a love letter to the wife of his colleague Kojiro TORII' or 'because of his younger brother Hanroku's problems with women.'
失脚の理由は、「同僚鳥居幸次郎の妻に艶書を遣わした」とも「弟の半六の女性問題」ともいわれている。