In 1852, he married Suga IJUIN (who was called Toshi (Toshiko) as well) who was Kanehiro IJUIN's sister on his parents recommendation, but his grandfather Yuzan died in July, his father Yoshibee died in September and his mother Masa died in November one after another, so he had to support a family by himself.
嘉永5年（1852年）、父母の勧めで伊集院兼寛の姉伊集院須賀（敏（敏子）であったとも云われる）と結婚したが、7月に祖父遊山、9月に父吉兵衛、11月に母マサが相次いで死去し、一人で一家を支えなければならなくなった。