Yoshitaka's wife was a daughter of Tadakatsu TORII (son of Mototada TORII, who fought at Fushimi Castle during the Battle of Sekigahara, and a vassal of his brother Tadamasa TORII)
妻に鳥居忠勝（関ヶ原の戦いで伏見城攻防戦にあたった鳥居元忠の子。兄鳥居忠政の家臣）の娘を迎えた。