That Tadaoki served many lords including Ashikaga, Oda and Tokugawa while ensuring that the Hosokawa clan continued to prosper is testament to his political skills; however, he was not a man given to forgive, even when it was someone close to him, and this led him to fall out with his father Yusai HOSOKAWA after he (Yusai) surrendered a castle to the enemy during the Battle of Sekigahara (for details refer to the paragraph on the Battle of Tanabe Castle).
足利氏、織田氏、豊臣氏、徳川氏と多くの主君に仕えながら細川氏を生き延びさせた政治手腕の反面、身内の者にも容赦を加えない苛烈な側面もあり、関ヶ原の合戦中、実父の細川幽斎が居城を敵に明け渡した（詳細は田辺城の戦いの項を参照のこと）ことから一時不和になっている。