Daizen had a son called Rokuro who became the first doctor in his line of the Inoue family, whose profession was successively followed by sons including Kinga's grandfather Kian INOUE and Kinga's father Kansai INOUE, who both worked as Han-i (Edo-period doctor working at a public clinic for the whole clan) under Masayuki INOUE of Kasama Domain in Hitachi Province (present-day Kasama City, Ibaraki Prefecture).
この大膳の子六朗から代々医者となり、祖父井上喜庵、父井上寛斎ともに藩医として常陸国笠間藩（現茨城県笠間市）の井上正之に仕えた。