He was born in 1501 as the third son of Munenobu INOUE who was the chief retainer of the Hosokawa clan, kanrei (shogunal deputy) of the Muromachi bakufu (Japanese feudal government headed by a shogun), in Tango Province.
文亀元年（1501年）、室町幕府管領細川氏の家老井上宗信の三男として丹波国に生まれる。