Afterward, when Harumoto became hostile to Nagayoshi, Nobuyoshi was placed under house arrest, together with Ujitsuna HOSOKAWA, a kanrei, in Yodo-kojo Castle and in 1558 celebrated his attainment of manhood under Nagayoshi's guidance.
その後、父が長慶と敵対すると管領細川氏綱と共に淀古城に軟禁され、1558年に長慶の下で元服した。