Although he moved to Shizuoka with the Tokugawa family in 1869, he was in a wretched condition; he returned to Tokyo in 1877 and lived in his acquaintance's house near the Yushima-tenjin Shrine for a while.
1869年（明治2年）徳川家と共に静岡へ移るも窮状を極め、1877年（明治10年）東京へ戻り、しばらくは湯島天神下の知り合いの元に身を寄せる。