It seems that this story was just a tradition based on the fact that, later, Yoshiaki KIRA fought with and defeated by Ieyasu TOKUGAWA and Ietada MATSUDAIRA, who was Yoshiharu's grandson, entered the Tojo-jo Castle.
後年、吉良義昭が徳川家康と戦って敗れ、東条城に義春の孫の松平家忠が入ったことから創られた伝説と思われる。