MINAMOTO no Toshifusa first caused a disturbance by having an affair with Imperial Princess Kenshi of Emperor Gosuzaku, who became an official wife for him, and running away with her without Emperor Gosuzaku's approval, that infuriated Imperial Prince Takahito (later, Emperor Gosanjo), Imperial Princess Kenshi's brother, who did not have a child with her.
正室の後朱雀天皇皇女娟子内親王とは始め許しもなく密通・駆け落ち騒動を引き起こし、内親王の弟尊仁親王（後三条天皇）の怒りを買ったが、彼女との間に子はできなかった。