While his father was once adopted by Kimura family in Ujitawara (present Ujitawara-cho, Tsuzuki County, Kyoto Prefecture) along with his wife, later both returned to the original Uebayashi family for some reason; on this occasion, only Rokuzo was left under the custody of the Kimura family.
父が宇治田原（現在の京都府綴喜郡宇治田原町）の木村家に夫婦養子に入った後、理由あって上林家に戻ったが、そのとき鹿蔵のみ木村家に残されたという。