He officially married with the daughter of Shigeyori KAWAGOE (Sato Gozen) through Yoritomo's good offices, and also had concubines, one is a beloved Japanese traditional dancer, Shizuka Gozen, who is famous for her dance at Tsurugaoka Hachiman-gu Shrine, and the other is the daughter of Tokitada, and she was said to have been forwarded to him by her father, Tokitada, who wanted to secure his own neck after the extinction of the Taira family.
妻には頼朝の媒酌による正室の河越重頼の娘（郷御前）、鶴岡八幡宮の舞で有名な愛妾の白拍子・静御前、平家滅亡後に平時忠が保身の為に差し出したとされる時忠の娘がある。