However, his eldest son, Kyuzo, who showed artistic talent and would have followed in his father's footsteps, died early (it is said that he was killed by a member of the Kano School who feared the power being gained by the Hasegawa School), and in his late life, an accident caused him to lose the use of his right arm.
しかし、画才に恵まれ跡継ぎと見込んでいた長男・久蔵に先立たれ（長谷川派の勢力伸長を恐れた、狩野派による「暗殺」とも言われている）、晩年には事故により利き腕である右手の自由を失う。