When he was eight years old, his father was placed in disinheritance and deported in the aftermath of Tempo Reforms; therefore his uncle Sukekata SHIBUKSWA succeeded to the family estate.
8歳の時に天保の改革の反動によって父が廃嫡・流刑となったために、家督は叔父の渋川佑賢が継ぐことになる。