It is said that he incurred Yoritomo's displeasure even more since he refused a proposal to take his widowed daughter (the lawful wife of MINAMOTO no Yoshihira) as Yoritomo's concubine.
さらに、未亡人となっていた娘（源義平室）を頼朝の側室にと乞われるもこれを拒否し、より一層頼朝の不興を買ったと伝えられている。