He was born into the Iida family in Higo Province, a vassal of a chief retainer of the Kumamoto Domain, Korekata NAGAOKA, and he was adopted by Shigesaburo INOUE afterwards.
肥後国に熊本藩家老長岡是容の家臣飯田家に生まれ井上茂三郎の養子になる。