His family came from a line of the Hosokawa clan, a branch family of the Ashikaga clan, which belonged to the Kawachi-Genji (Minamoto clan) as a branch of Seiwa-Genji (Minamoto clan).
家系は清和源氏のひとつ・河内源氏の流れを汲む足利氏の庶流・細川氏。