In 1848, his father Yoshihira was enlisted as the domain's ashigaru (common foot soldier) to guard the seaside against ships from overseas that appeared off the coast of Tosa; he continued to live in Shichiken machi inside the castle town, and Izo himself succeeded the status as ashigaru.
嘉永元年（1848年）、土佐沖に現れた外国船に対する海岸防備のために父義平が藩の足軽として徴募され、そのまま城下の七軒町に住むようになり、以蔵自身はこの足軽の身分を継いでいる。