However, there was no way he could get along with Tomiko HINO, who had been so opposed to him in the Onin War, and Yoshimi gradually became opposed to Tomiko, destroying the Ogawadai manor (Tomiko's residence) and seizing her territory.
しかし応仁の乱であれほど敵対していた日野富子と反りが合うわけもなく、次第に対立して義視は富子の邸宅である小川第を破壊したり、領地を差し押さえたりした。